Text-to-SQL benchmark scores do not predict accuracy on production databases because benchmark schemas are small, consistently named and self-documenting. Real schemas are large, inconsistent, and carry meaning that exists nowhere in the catalog — so the only useful evaluation is a set of your own questions against your own database.
Public text-to-SQL benchmarks did the field a real service: they made progress measurable at a point when everyone was demoing on their own cherry-picked examples. They are also close to useless for deciding which tool to buy, and the reason is not that they are badly built.
What benchmark schemas have that yours does not
- A sensible size. Benchmark databases have tens of tables. Production warehouses have hundreds to thousands, which changes the problem from comprehension to retrieval.
- Consistent naming. Benchmark columns are called `student_id`. Yours has `cust_id`, `customerId` and `fk_cust` in three tables written in three different years.
- Self-documenting semantics. A benchmark schema means what it says. Yours has a boolean called `flag_2` that gates refund eligibility, and everybody who knew that has left.
- No tribal knowledge. In your database, "active" means `status_id = 3` and everyone knows to exclude `is_internal`. Nothing in the catalog says so.
A system tuned to score well on tidy schemas is being optimised on a distribution your database is not drawn from. That is not cheating — it is just a different problem wearing the same name.
The other reason the number misleads
Execution accuracy scores a query as right or wrong. In practice the cost of those two failures differs by orders of magnitude. A query that errors costs a retry. A query that runs, returns a plausible number and is silently four times too large because of a fan-out costs a decision. A single percentage collapses both into one figure, and the second is the one that should keep you up.
What to measure instead
Build your own benchmark. It takes an afternoon and it is the only number that will predict anything.
- Collect twenty questions people actually asked your data team in the last month. Real ones, in the words they used.
- Have someone who knows the schema write the correct SQL and record the correct answer.
- Run all twenty through each candidate tool.
- Score three buckets, not one: right, obviously wrong, and plausibly wrong. The third bucket is the one that decides.
- Repeat on your worst schema, not your cleanest. The cleanest one is where every tool looks fine.
Twenty questions is enough to separate tools decisively, because on a real schema the gap between a system that retrieves your foreign keys and one that does not is not subtle.
Why we do not publish an accuracy number
We could run the benchmarks and quote a figure — everyone does, and the figures are all high. It would tell you nothing about your schema, and it would sit on a marketing page next to a claim that we show the query with every answer, which is the actual answer to the accuracy question. If you can read the SQL that produced a number, its correctness is something you check in five seconds rather than something you take on our word.
The free plan has no card and no expiry for the same reason. Twenty of your own questions against your own database is a better evaluation than anything we could publish about ourselves.