AVAEvaluating AVA’s ability to turn data questions into reliable answers. View evaluation reports.
Context
| Evaluation | Context | Correct | Accuracy | Tokens / question | Time / question |
|---|---|---|---|---|---|
| Tabular Q&A DataBench[1] | Raw data (header + first 3 rows) | 1,494 | 83.51% (baseline) | 991 | 1.35 s |
| With Schema (Schema + Profile) | 1,506 | 84.18% (+0.67 pp) | 1,990 | 2.08 s | |
| With Semantic | 1,664 | ★ 93.01% (+9.50 pp) | 3,288 | 1.66 s | |
| Complex SQL · SQLite Spider2-Lite[3] | Raw data (header + first 3 rows) | 35 | 25.93% (baseline) | 4,057 | 12.93 s |
| With Schema (Schema + Profile) | 45 | 33.33% (+7.41 pp) | 20,945 | 11.89 s | |
| With Semantic | 109 | ★ 80.74% (+54.81 pp) | 23,705 | 13.33 s |
DataBench: 1,789 single-table Q&A questions covering filtering, aggregation, and sorting. Spider2-Lite: 135 complex SQL questions across multiple tables, using supplied external knowledge; raw data includes headers and the first 3 rows per table. With Schema adds structure and statistics; With Semantic adds business definitions. All queries use complete data.
Strategy
| Evaluation | Model | Strategy | Correct | Accuracy | Tokens / question | Time / question |
|---|---|---|---|---|---|---|
| Tabular Q&A DataBench[1] | GLM-5.1 | Direct | 1,550 | ★ 86.64% | 2,315 | 20.48 s |
| Loop | 1,504 | 84.07% | 5,089 | 82.52 s | ||
| Database queries BIRD New Dev[2] | GLM-5.1 | Direct | 968 | 63.10% | 6,780 | — |
| Loop | 1,010 | 65.84% | 35,272 | — | ||
| GLM-5.2 | Direct | 1,003 | 65.38% | 6,575 | — | |
| Loop | 1,030 | 67.14% | 33,545 | — | ||
| GLM-5.3 | Direct | 1,045 | ★ 68.12% | ≥ 9,529 | — | |
| Loop | 1,016 | 66.23% | ≥ 32,976 | — | ||
| Kimi-K2.5 | Direct | 941 | 61.34% | 7,220 | — | |
| Loop | 905 | 59.00% | 51,631 | — | ||
| Kimi-K3 | Direct | 1,014 | 66.10% | 6,696 | — | |
| Loop | 993 | 64.73% | 43,512 | — |
AVA Workflow. DataBench: 1,789 questions; historical runs with different context configurations. BIRD New Dev: 1,534 questions with Schema, evidence, and database descriptions.
Execution
| Evaluation | Model | Execution | Correct | Accuracy | Tokens / question | Time / question |
|---|---|---|---|---|---|---|
| Tabular Q&A DataBench[1] | GLM-5.1 | SQL (DuckDB) | 1,550 | ★ 86.64% | 2,315 | 20.48 s |
| Python | 1,528 | 85.41% | 2,002 | 22.03 s |
DataBench: 1,789 questions · AVA Workflow · Direct. Historical SQL and Python runs use different execution configurations.
Integration
| Evaluation | Model | Integration | Correct | Accuracy | Tokens / question | Time / question |
|---|---|---|---|---|---|---|
| Tabular Q&A DataBench[1] | GLM-5.1 | SDK | 1,550 | ★ 86.64% | 2,315 | 20.48 s |
| Agent Skill | 1,511 | 84.46% | 23,384 | 40.93 s |
DataBench: 1,789 questions. SDK uses AVA Workflow with Direct; Agent Skill lets the agent choose tools and execution steps. These historical runs use different execution configurations.
Dataset notes
-
DataBench. 1,789 questions across 79 full tables, excluding
080_Books. Tables average 46,911 rows and 22.8 columns. Measures tabular answer correctness using reference answers and type-specific matching. Details. -
BIRD New Dev. 1,534 questions across 11 databases, version
dev_20251106. Databases average 6.8 tables; tables average 52,436 rows and 10.6 columns. Measures Text-to-SQL correctness by comparing query results with reference SQL results. Details. -
Spider2-Lite SQLite subset. 135 complex SQL questions across 30 databases. Databases average 13.7 tables; tables average 43,009 rows and 7.0 columns. Scored by the official execution-result scorer; missing answers count as incorrect. Covers the SQLite subset only. Details.
Scale averages are calculated over unique base tables, excluding views and internal tables. Datasets use different scoring protocols, so their scores are not directly comparable. These evaluations measure answer correctness, not visualization quality.