arxiv:2608.03451

DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces

Published on Aug 4

· Submitted by

Boyan Li on Aug 7

HKUSTDial

Upvote

Authors:

Boyan Li ,

Abstract

Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia. Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis, leaving heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation insufficiently unified. We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces. It contains 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video. DataSpace also served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition. Each agent receives only a question and workspace and returns the complete requested tabular result. We construct DataSpace with DataSpace-Builder, an execution-grounded framework comprising cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, and human review and task repair by 11 domain experts. A deterministic evaluator performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison. Across six recently released frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%, while harness choice creates a 15.36-point spread with the backbone fixed. Multimodal evidence integration and joins consistently reduce accuracy across all six backbones. These results show that DataSpace remains unsaturated and identify key challenges for improving data-agent reliability.

View arXiv page View PDF Project page GitHub 14 Add to collection

Community

BugMaker-Boyan

Paper author Paper submitter 2 days ago

Can a data agent do more than answer a question—can it navigate an entire workspace and deliver a result you can actually verify?

Real-world analytics rarely begins with a single clean table. The evidence is often scattered across databases, structured files, long documents, and videos—sometimes even across languages.

We are excited to introduce DataSpace, a benchmark that challenges data agents to become true workspace solvers. Given only a natural-language question and a task-local heterogeneous workspace, an agent must autonomously discover the relevant evidence, connect information across formats, perform multi-step analysis, and return the complete, verifiable tabular result.

📊 DataSpace at a glance:

410 cross-language analytical tasks
7,439 artifacts totaling 15.01 GB
6 formats: CSV, JSON, SQLite, Markdown, PDF, and video
Financial, macroeconomic, and healthcare scenarios
Execution-grounded construction and review by 11 domain experts
Deterministic, model-free evaluation of complete tabular results

DataSpace also served as the official evaluation benchmark for the KDD Cup 2026: Data Agents for Complex Data Analysis, bringing the challenge to 703 valid teams and 1,307 participants worldwide.

Our evaluation across 6 frontier multimodal models and 5 widely used agent harnesses reveals how much remains unsolved. The strongest controlled setting reaches only 66.34% Task Accuracy, while 76 tasks are missed by every tested model. Even with the backbone fixed, changing the agent harness creates a striking 15.36-point accuracy gap. Multimodal evidence integration and cross-source joins remain persistent bottlenecks.

The message is clear: today’s data agents are promising, but reliable workspace-scale analytics is far from solved.

We are releasing the benchmark, dataset, evaluator, baselines, and leaderboard to support the next generation of reliable and verifiable data agents. We warmly invite the community to test new models, harnesses, and agent designs—and push beyond today’s frontier. 🚀

📄 Paper: https://arxiv.org/abs/2608.03451
🌐 Project & KDD Cup: https://dataagent.top/
💻 Code & Evaluator: https://github.com/HKUSTDial/DataSpace
🤗 Dataset: https://huggingface.co/datasets/HKUSTDial/DataSpace
🏆 Leaderboard: https://dataspace-bench.github.io/

Questions, feedback, results, and contributions are all highly welcome!

librarian-bot

1 day ago

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

O96a

about 4 hours ago

Every data agent I've shipped eventually hits the same wall: the answer lives across a database, a CSV, and a doc that contradicts both, and there's no way to prove the agent actually found it. This benchmark finally makes that the point instead of the footnote. I like that it forces complete tabular outputs and deterministic evaluation — "verifiable" is the word that matters, because the failure mode isn't wrong answers, it's confident answers with no evidence trail. The 410 cross-language tasks over 7,439 artifacts is the kind of mess that resembles a real org workspace, not a cleaned-up toy. My one hesitation is whether the evaluation rewards the agent for showing its work or just for landing on the right cell — those are different skills and I'd want to see the rubric. Still, this is the benchmark I'd actually run my RAG stack against before claiming it works.

BugMaker-Boyan

Paper author 10 minutes ago

Thanks for the thoughtful comment, Aamer — you have identified an important distinction. In the current DataSpace release, the official score is outcome-based: an agent receives credit only when its complete submitted table—not merely an individual cell—matches the reference under a frozen, deterministic configuration. This includes type- and precision-aware normalization, header-invariant column alignment, and order-aware row comparison; missing or extra rows or columns fail the task.

So, in this version, “verifiable” means that the final analytical result can be checked exactly, reproducibly, and without an LLM judge. It does not yet mean that the agent’s source selection and evidence trail are themselves scored. We retain execution traces and use them in our human-confirmed trace-level failure analysis, but that is diagnostic rather than part of Task Accuracy. We agree that outcome correctness and faithful evidence trails are distinct capabilities, and we see process-level verification as an important direction.

The full scoring protocol and evaluator are available here: https://github.com/HKUSTDial/DataSpace/tree/main/evaluation. We would love to see how your RAG stack performs on DataSpace—and would especially value your thoughts on what a useful evidence-trail metric should require.