Insights

Text-to-SQL in the enterprise: Lessons from the BIRD Benchmark

By Dr. Adnan Masood and Nagur Shareef Shaik

UST

Text-to-SQL is software that takes a question written in plain English, generates the corresponding SQL query, runs it against a database, and returns the answer. Our team at UST recently ranked 12th in the world on BIRD-SQL, the most demanding public benchmark for this technology, through a research program conducted with Professor Tatsunori Hashimoto's group at Stanford and led by Nagur Shareef Shaik on our team. The study is published as AxisSQL, with the code, configurations, and per-run outputs released alongside the paper. The findings apply directly to how enterprises should build, buy, and evaluate natural language analytics, and this piece walks through the technology, the research, and the implementation lessons in that order.

What enterprises use this for

The most common application is self-service analytics. A finance manager asks for revenue by product line over the last six quarters, a supply chain planner asks which distribution centers are below safety stock, and each gets an answer in seconds without filing a ticket with the BI team. The value is less about any single query and more about removing the analyst queue as the bottleneck between a business question and its answer.

The same capability shows up across industries in more specific forms. A health plan analyst asks which members with diabetes have no claim for an eye exam this year. An insurance auditor pulls every policy that matches a treaty condition during a review. A retail category manager asks which stores discounted a product below the approved floor last month. A bank investigator lists transactions that triggered a specific exception rule. Each of these is a database query that a business user can now phrase in English, and each previously waited days for someone fluent in SQL.

A second category is acceleration of the data teams themselves. Analysts use these systems to draft complex joins and window functions they then review, which shortens development time without removing human review. During warehouse migrations, teams use them to generate validation queries that compare old and new platforms table by table.

The third and fastest-growing category is agentic systems, meaning AI applications that complete multi-step tasks by calling tools. An agent that reconciles invoices, monitors a supply chain, or prepares a regulatory filing needs to read structured data, and text-to-SQL is the interface between that agent and the warehouse. Every agent that touches a database embeds a text-to-SQL problem, whether anyone calls it that or not.

In all of these settings, production deployments run read-only, inherit the user's row-level security, and log every generated query for review. The technology extends access to data, and the existing access controls still decide who sees what.

DIVIDER

Why the problem is harder than the demonstrations suggest

BIRD, the benchmark, gives a realistic picture of the difficulty. Introduced by Li et al. at NeurIPS 2023, it contains 12,751 question-and-query across 95 large databases in 37 professional domains, and it deliberately preserves the dirty values of real operational data: inconsistent formats, cryptic column names, encoded categories. Performance is scored by execution accuracy, meaning the generated query returns the same results as a correct one. Human data engineers score 92.96 percent. The best AI system in the world currently scores just under 82.

A typical enterprise warehouse is harder than BIRD. It has more tables, staler documentation, and column names that made sense to someone in 2009. A vendor quoting near-perfect accuracy has measured something easier than your data.

DIVIDER

What we measured

Recent gains on benchmarks like BIRD come mostly from inference-time scaling, meaning spending more compute when the question is asked: generating more candidate queries, voting among them, running more verification rounds, or using larger models. Teams usually turn several of these dials at once, making it impossible to know which one drove the improvement.

Our study, AxisSQL, held a standard five-stage pipeline fixed (DeepEye-SQL, covering value retrieval, schema linking to identify the relevant tables and columns, query generation, automated repair, and final selection) and varied five scaling axes independently under a strict compute budget: the number of candidates generated, the number of verification rounds, which stage receives a stronger model, the presence of database metadata, and the base model itself. On BIRD's hidden test set, the optimized configuration reached 76.86 percent execution accuracy, a controlled gain of 1.79 points over the identical pipeline running the identical model without our changes. Three findings matter for practitioners.

DIVIDER

Finding one: more candidates stop helping at about twelve

The intuitive way to improve an AI system is to let it try more times, and prior work shows that repeated sampling does improve coverage in a predictable way (Brown et al., 2024) while eventually saturating against the limits of the verifier (Cobbe et al., 2021). We measured what that buys in a text-to-SQL pipeline. The oracle ceiling, also written Pass@N, is the share of questions for which at least one correct query exists in the candidate pool. As the pool grows from 3 to 24 candidates, the oracle ceiling climbs from 75.5 to 83.6 percent, but realized accuracy flattens near 12 candidates. Doubling from 12 to 24 costs 3.6 times the tokens and returns three-tenths of a point, within measurement noise. Across four different models, roughly seven points of accuracy sit in the selection gap, the difference between the ceiling and the realized score. In those cases, the pool already contained the correct SQL and the system chose a wrong one.

The efficient fix, adapted from work on agentic aggregation by Lee et al. (2026), is execution-grounded aggregation: run the top candidates against the live database, show a reasoning model each query alongside its actual results, let it pick a winner or compose a corrected query from the fragments that returned sensible data, then verify the choice once by executing it. On identical candidate pools, this raised accuracy from 69.8 to 73.8 percent for roughly 25 percent more tokens, ahead of a tuned baseline built on self-consistency voting at 71.4 percent. The gains concentrate on moderate-difficulty questions, the joins, filters, and grouping operations that make up most real business queries. Compute spent on selecting far outperformed compute spent on generating.

DIVIDER

Finding two: metadata was the largest single lever

The biggest effect in the study came from documentation. With no column metadata, the pipeline scored 66.5 percent. With it, 73.8. Metadata generated automatically, by profiling each database and having a language model write a short description of every column, following the approach of Shkapenyuk et al. (2025), outperformed the human-written documentation that ships with the benchmark, 72.2 percent to 71.0, and combining both sources performed best. The extraction runs once per database, and its cost amortizes across every future query.

For enterprises, the implication is direct: data documentation is now a measurable input to model accuracy. After years of asking teams to document their columns with limited success, the request finally comes with a number attached.

DIVIDER

Finding three: inference scaling substitutes for about one model size tier

We tested one model family at four sizes, from 4 billion to 31 billion parameters. A 12B model running the optimized pipeline scored 69.0 percent, above both the 27B and 31B models running single-shot, which scored 62.0 and 63.5. Test-time compute and parameter count act as complementary resources, and disciplined inference scaling delivered accuracy equivalent to moving up roughly one model size.

For regulated industries, this changes the deployment math. Organizations evaluating enterprise AI deployments must balance model capability, governance requirements, and operational cost. A healthcare or financial services organization can run an open-weight model inside its own security perimeter, at a fraction of frontier API pricing, and reach accuracy that previously required a much larger model. Our leaderboard submission uses a disclosed 31B open model with published code, configurations, and a reproducible evaluation harness, so a client's model risk team can audit the full system rather than accept an assurance.

DIVIDER

Where frontier models such as Claude fit

The study used open-weight models deliberately, because controlled experiments require disclosed, reproducible components. The findings themselves are about architecture, and they transfer to any model choice. One result points the way: when we upgraded a single pipeline stage to a stronger model, upgrading the aggregator returned 0.89 accuracy points per thousand additional tokens, against 0.23 for the generation stage and 0.36 for schema linking. The strongest reasoning should sit at the judgment stage, where the model weighs candidate queries against their execution results.

In client deployments, UST applies that principle as an Anthropic partner by pairing efficient open models for candidate generation with a frontier model such as Claude at the aggregation and verification stage, where its reasoning over schemas, queries, and result sets earns the cost. Claude's long context window also helps at schema linking, where enterprise warehouses with thousands of columns exceed what smaller models can consider at once. The architecture is vendor-neutral; the allocation principle, frontier reasoning, where the pipeline exercises judgment and efficient models where it produces volume, is what the measurements support.

DIVIDER

Implementation guidance

Several practices follow from the data. Treat text-to-SQL as a system, because a single model call will not survive contact with a real warehouse; production accuracy comes from retrieval, schema linking, generation, repair, and selection working together. Track both realized accuracy and the oracle ceiling in your evaluations, since the gap between them tells you whether to invest in generation or in selection, and in our measurements it was consistently selection. Fund metadata extraction before larger models, because it delivered the highest return of any axis tested. Gate the expensive reasoning: our aggregator activates only when candidates disagree, roughly a third of queries, so cost tracks uncertainty and routine questions stay cheap.

Sequence the rollout by difficulty. On the hidden test set, our system scored 85.4 percent on simple queries, 72.6 on moderate, and 56.8 on challenging ones, where failures happen early, at schema linking. Deploy first for the workloads where accuracy is already strong, keep an analyst in the loop for the hard cases, and tell users which is which.

DIVIDER

What the benchmark result establishes

The BIRD leaderboard is populated almost entirely by research groups at large technology companies, including teams from Google, Amazon, Alibaba, Huawei, Tencent, and AT&T, which makes it an unusual place for a digital transformation company to appear. The practical value of the ranking is verifiability: a client evaluating our text-to-SQL work can check a public scoreboard with a hidden test set, and the paper, code, and configurations are public alongside it.

The study's final observation is worth carrying into any deployment. Accuracy in these systems follows a saturating curve as compute increases, and the curve's ceiling is set by the selector and the metadata, while additional sampling only moves performance along the existing curve toward that ceiling. Knowing which of the two you are paying for is what separates an engineered system from an expensive one.

Looking to deploy natural-language analytics, agentic data access, or AI-powered decision support on enterprise data? Explore how UST helps organizations operationalize AI responsibly and at scale.