Songbird answers football questions in plain English on MockingBoard Stats (opens in a new tab). It writes SQL against every NFL season since 1999, 1.28 million plays in all, and shows you the query it ran. Everything else follows from one rule: the database computes and the model translates. Songbird never states a number that didn't come back from a query.
One question, end to end
A question is first classified into one of four difficulty classes, or as out of scope. The class decides which model writes the SQL. Direct lookups go to a fast, cheap model; joins, windows, and multi-step reasoning go to progressively stronger ones; the hardest class goes to Claude Opus. The classifier is a small model with a two-second timeout and a fallback, and it's never allowed to refuse on its own. A suspected out-of-scope question goes to the first-class model to confirm or rescue. At around 1% label noise, a hard refusal from the classifier would break the easiest class's accuracy gate all by itself.
The SQL passes guardrails before it touches data. It must be a single SELECT over whitelisted views, and a keyword scan runs on a copy with comments stripped and string literals blanked. File and metadata functions are banned, as are comma and cross joins. Rows, runtime, and query length are all capped. It runs on server-side DuckDB with a fixed memory ceiling. If the query fails, the error goes back to the model for one repair attempt. If that fails too, Songbird says it stopped instead of guessing.
A separate call writes the answer, and it sees only the question, the SQL, and up to 50 result rows. Its instructions are blunt: every number you state must appear in the rows; never compute new statistics; never fill gaps from memory. The model can also propose a chart as a small typed spec, which is validated and compiled to SQL against the same views. An invalid spec means no chart, never an error, and every chart opens in the workbench's chart builder so you can keep going.
Cost is metered in credits by class. Credits are reserved at the maximum before any model call and settled in one transaction alongside a trace of the run. The hardest questions can escalate to a bounded deep dive, capped at five queries and ninety seconds.
The eval is the product
Every model or prompt change runs against a gold set of 170 questions, now on its twentieth version, spread across the four classes plus out-of-scope ones. Scoring is execution match, not a model judge. Gold and candidate both run; values are rounded and column names ignored; every gold row has to appear in the candidate's result, in order for ranked questions and with an exact row count for sets. Each class has its own gate (98% for the easiest, 90% and 80% for the next two) and every run also reports refusal precision, p50 and p95 latency, routing agreement, and cost. Songbird was gate-checked on a frozen set before launch, and the model lineup is picked by these runs, not by leaderboards.
The misses taught more than the passes:
- The first judge was broken and scored everything 0%.
- Many "model errors" were mine: sibling views with mismatched columns, a gold answer that counted only offensive snaps, and a prompt example that taught the wrong answer.
- Quarterback passing stats were quietly dropping scrambles. Josh Allen's 2024 had 497 recorded dropbacks against 540 real ones, and the fix meant a new gold version.
- Extending the data back to 1999 silently changed 20 frozen answers, so the gold set now has a freshness check that re-executes it against current data.
I wrote both the gold set and the conventions it tests, so it's a regression gate for this product, not a public benchmark.
Three products, one question
Before launch I asked the same question verbatim to Songbird and to two dedicated consumer NFL assistants, NFL Pro's Fantasy AI Assistant and SumerBrain. It wasn't a scored run, just a head-to-head on a hard question:
Puka Nacua was a college receiver with middling production and the draft consensus ignored him. He then broke out in the NFL as the most productive rookie WR in history. What hints about Puka were in the data that were missed, and who from the 2025 college season fits that same profile?
Answering it means holding two datasets and computing across them. Songbird routed it to the top class and answered both halves in one query. It anchored Puka's 2022 BYU season as a baseline and returned his full college arc, where the efficiency signal was there in 2020 and 2021. Then it screened 2025 college receivers against his own numbers. NFL Pro's assistant answered with unsourced narrative, and its top "2025 college" match had played 2025 in the NFL. SumerBrain honestly declined both halves for lack of college data.
Stack
TypeScript and Next.js, zod at every boundary, DuckDB's Node API on the server (DuckDB-WASM powers the rest of the workbench in the browser), and Firestore for quotas and traces. Models come from Anthropic and OpenAI directly and from DeepSeek through the Vercel AI Gateway. It all lives in the MockingBoard monorepo, and Songbird is also embedded in the draft and fantasy apps.