Skip to content
All work

An agent that does the data science, not just the SQL

71% on BIRD, eight points above GPT-4o

At UCLA Trustworthy AI Lab I built an agent that plans and executes multi-step analyses from a plain-language prompt. It completed 71% of BIRD benchmark tasks, eight points above GPT-4o.

71%
Task completion on the BIRD benchmark
+8 pts
Above GPT-4o on the same benchmark
400+
Test sessions with zero state loss
34
Table schemas in the eval harness
A bar chart of task completion on the BIRD benchmark. This agent completes 71 percent of tasks; GPT-4o completes 63 percent on the same tasks, a gap of eight percentage points. task completion · bird benchmark 0% 25% 50% 75% 100% This agent 71% GPT-4o 63% +8 pts same tasks · same schemas · n = 400+ sessions
Fig. 1Task completion on BIRD against GPT-4o on the same tasks and schemas.
Three dot grids showing the evaluation harness's coverage: 34 table schemas, over 120 question types, and over 20 tool-calling functions. Every observed failure attributes to a cell in that space rather than to the agent as a whole. 34 table schemas 120+ question types 20+ tool functions every failure attributes to a cell: tool × schema × question class, not "the agent was wrong"
Fig. 2The harness's coverage. Scoring end-to-end pass/fail tells you the agent was wrong; scoring across this grid tells you which tool, schema shape, or question class lost, which is what the three architectural fixes came from.
Requests enter through API Gateway to a Dockerized Lambda. Each step appends an immutable event to a log stored in PostgreSQL and S3. Current state is a fold over that log, which makes any session replayable. API Gateway concurrent requests Lambda Docker · CDK append-only event log plan query observe revise answer append fold PostgreSQL events S3 artifacts state = fold(events) · any session replays exactly 400+ sessions · zero data loss · concurrent steps cannot clobber
Fig. 3State as a fold over an append-only log: concurrent steps can't clobber each other, and a failure leaves a record rather than a corrupted document.

Context

Most “talk to your data” systems are text-to-SQL: one prompt in, one query out. Real analysis isn’t shaped like that. It’s plan, query, look at what came back, revise, query again, and the interesting failures happen between the steps, not inside them.

BIRD is a good measuring stick because its questions require reasoning over messy, realistic schemas rather than clean toy tables.

The problem

Planning. Decompose a vague request into ordered steps, pick the right tool for each, and recognize when a result means the plan was wrong.

State, which people underestimate. A multi-step agent on serverless has no natural place to keep its own state: steps land on different Lambda invocations, run concurrently, and can fail partway. Losing state mid-analysis doesn’t just fail the task. It fails it silently, leaving a plausible partial answer.

What I built

A serverless backend sized for concurrent inference. Dockerized Lambda behind API Gateway, provisioned with CDK. Containerized because the dependency surface for analysis tooling is far too large to zip; CDK because the infrastructure had to be reproducible for anyone rerunning the experiments.

An event-sourced persistence layer. Every step appends an immutable event to PostgreSQL, with larger artifacts in S3; current state is a fold over the log. That solved distributed state outright: concurrent steps can’t clobber each other, a failed step leaves a record rather than corruption, and any session replays exactly. 400+ sessions, zero data loss.

A data-centric evaluation harness, which is what actually produced the result (Fig. 2). Instead of end-to-end pass/fail, it spans 34 schemas, 120+ question types, and 20+ tool-calling functions, so a failure attributes to a specific tool, schema shape, or question class. That attribution drove three architectural changes worth 11% accuracy, findable only because the harness said where the losses were.

The hard part

Resisting the urge to fix the prompt. Every failure looks like a prompting problem until you have per-tool attribution. Once the harness could say “this question class fails on this tool,” the fixes turned out to be architectural, not verbal: tool granularity, and the shape of what got returned to the planner.

Outcome

71% on BIRD, eight points above GPT-4o on the same tasks (Fig. 1). Beating a frontier model wasn’t a modeling win: the leverage was evaluation and architecture. Knowing precisely where the agent lost, and having state guarantees strong enough for multi-step plans to run without silently degrading.

What I’d do differently

Build the harness first. I built it third, and every improvement before it was guesswork I couldn’t measure. On an agent project the harness isn’t test infrastructure. It’s the instrument you’re doing the work with.