Every agency's pitch deck now says the same things. The differences that predict whether a project succeeds are not in the deck — they are in how the team answers six specific questions, and in what the contract says about data, ownership and exit. Here is what to ask and how to read the answers.
1. What have they actually put in production?
A demo proves a model can produce an impressive output once. Production proves the team handled authentication, rate limiting, cost control, evaluation, monitoring, and the two a.m. failure — which is where the real engineering is.
Ask
- Which of these systems is running in production right now, and who operates it?
- What broke after launch, and what did you change as a result?
- What is the p95 latency and cost per query, and how do you know?
- Show me the evaluation report for one of these, redacted as needed.
Worry if
- Every reference is a pilot, a proof of concept, or 'under NDA' without exception.
- They cannot describe a post-launch failure. Every real system has one; not having a story means not having operated it.
- Accuracy is quoted with no mention of the dataset it was measured on.
2. How do they define and measure 'working'?
This is the highest-signal question on the list. An agency that builds an evaluation set before writing model code is running an engineering process. One that demos and asks whether it feels right is running a design process wearing engineering clothes.
Ask
- When in the project do you build the evaluation set, and who supplies the examples?
- Which metrics gate a release, and what happens when one regresses?
- How would you detect that quality degraded three months after launch?
- How do you measure whether the system correctly refuses questions it cannot answer?
Worry if
- Evaluation is described as a phase near the end rather than an input at the start.
- The only metric is a single accuracy number with no baseline to beat.
- There is no answer for detecting post-launch drift.
3. Where does your data go, and who owns the result?
Get this in writing before technical discussions go far, because it can disqualify a vendor regardless of engineering quality. The questions are specific and the answers should be too.
Ask
- Does the system run in our cloud account or yours? If yours, what exactly is stored, where, and for how long?
- Which third-party model providers see our data, under what data-processing terms, and is training on our data contractually disabled?
- Who owns the trained weights, the code, the prompts and the evaluation datasets at the end?
- Is there any licence-back to you on artefacts we paid to create?
- What happens to our data if we terminate?
Worry if
- Ownership is 'shared' or unspecified in the master agreement.
- The system depends on a proprietary runtime or platform only they can operate.
- They cannot name every third party that will process your data.
4. Who is actually going to do the work?
The gap between the people in the pitch and the people on the commits is the oldest failure mode in the industry, and it has not improved. Name the team in the contract.
Ask
- Name the individuals who will be on this project and their allocation percentage.
- Which parts, if any, are subcontracted?
- What happens if a named person leaves mid-project?
- Can we meet the engineer who will lead it, not only the account lead?
Worry if
- The team is described only by role and seniority, never by name.
- Allocations are vague, or one 'lead' is spread across many concurrent clients.
- The technical lead is absent from every call.
5. Is the cost model honest about what scales?
Build cost is knowable and quotable. Run cost is where projects get killed at month four, and it is dominated by inference volume, retrieval infrastructure and the size of the model chosen.
Ask
- What is the estimated monthly run cost at our expected volume, and what drives it?
- Are model-provider and cloud costs billed to our own accounts, or marked up through you?
- What is the plan if usage is ten times the estimate?
- Which optimisations are in scope, and at what point are they worth doing?
Worry if
- Run cost is not discussed until after the build quote is signed.
- Provider costs are opaquely bundled with a margin you cannot see.
- There is no answer for what happens at ten times volume.
6. What does handover look like?
The best outcome is that you could continue without them. Ask what that would take, and listen for whether the answer sounds like a plan or a threat.
Ask
- What documentation exists at the end, and can we see an example from a past project?
- Could our own team run the retraining and re-indexing jobs unaided?
- What is the support tier structure and the response-time commitment in each?
- If we ended the engagement in month six, what would break?
Worry if
- Documentation is 'the code is the documentation'.
- Critical operations run on the vendor's infrastructure with no path to yours.
- SLA commitments are described verbally and absent from the contract.
Signals that outrank the pitch
- They tell you a part of your request is a bad idea, with a reason, before you have signed anything.
- They ask about your data quality early and specifically, because they know that is what determines the schedule.
- They scope discovery separately and hand you its output regardless of whether you continue.
- They quantify uncertainty instead of quoting a single confident number.
- They decline work that does not fit. An agency that says yes to everything is optimising for the contract, not the outcome.
The short version
- Ask what is in production, who operates it, and what broke.
- Ask when the evaluation set is built. Start-of-project is the answer you want.
- Get data residency, provider terms and IP ownership in writing before the technical deep-dive.
- Get the individual engineers named in the contract with allocations.
- Get a run-cost estimate at your real volume, with provider costs billed transparently.
- Get a handover definition that would let you continue without them.