From the Research Universities edition of November 10, 2026
Agents Propose, Code Verifies, Humans Decide: Verified Agentic Ingestion of Heterogeneous Scientific Tables into Relational Databases
Colorado State University and The Ohio State University · Research data management (biomedical dataset ingestion) · United States (authors' institutions); four public biomedical datasets
- Publisher
- arXiv
- Original publication
- 8 November 2026 (arXiv 2611.11045v1, cs.DB)
- Source retrieved
- 2026-11-11
What happened
The authors describe an ingestion pipeline that turns messy public scientific tables (wide matrices, long tables, per-cohort files) into a PostgreSQL database under one rule: LLM agents propose typed schemas from compact evidence about the files, a deterministic verifier with 39 check types accepts or rejects each proposal in a structure loop and a meaning loop, and a human approves the schema and the load. On four public biomedical datasets, one held out from development, 9 of 20 first proposals failed a check and every failure was caught; the loops repaired eight and a one-line reviewer edit fixed the ninth. All 45.1 million values loaded with zero reconciliation gap, structure discovery took 4-7 LLM calls and data preparation at most one more, at under $0.50 per dataset. In a separate test, each of nine planted data errors was quarantined with a reason or recorded in an audit log. The paper illustrates silent failure with GSE73072, where keying on a subject column alone would keep 30 subjects instead of 151. Included now because it was submitted 8 Nov, appeared in the 10 Nov arXiv listings and was not archived.
Why it matters
Relevant to university data-management, library and research-computing groups that help investigators reuse public datasets: it shows a controlled pattern for letting agents do the structural work while code and a reviewer own correctness, with measured cost on four datasets.
Evidence and measured results
arXiv preprint reporting results on four public biomedical datasets (one held out from development), 45.1 million values, per-dataset call counts and cost, and a planted-error test with nine errors. No comparison against an unconstrained coding agent. Not peer reviewed.
Limitations and uncertainty
Preprint; four biomedical datasets and one model family; no baseline against an unconstrained agent (the subject-ID case is an illustration, not a failure rate); kept values on repeated keys depend on file order; large wide matrices enlarge prompts; costs are LLM-call costs only, not engineering or reviewer time.
Put this evidence to work
Lighthouse Advisory interpretation, grounded in this source. Enriched 2026-11-10; this does not change the original publication date. Each role follows the same checklist so takeaways compare across sources.
Sales
Role takeaway
- Customer problem
- Reanalysis of public scientific datasets requires reconnecting inconsistent files, and automated loaders can succeed silently with wrong keys.
- Stakeholders
- Research data services lead
- Bioinformatics or data core director
- Library data curators
- Principal investigators reusing public cohorts
- Discovery questions
- Which public or collaborator datasets are reloaded most often?
- How are ingestion errors detected today?
- Who approves a schema before load?
- Do any datasets carry controlled-access restrictions?
- Value hypothesis
- Agent-proposed schemas gated by deterministic checks and human approval may reduce manual ingestion work while preserving an audit trail; the paper measures LLM calls and cost, not staff time.
- Potential engagement
- A pilot ingesting two or three local datasets with the propose-verify-approve pattern, comparing errors caught, reviewer effort and reconciliation results with the current process.
- Unsupported claims
- Staff-time or total cost savings
- Performance on non-biomedical or controlled-access data
- Superiority over other coding agents
- That verification removes the need for human approval
Pre-sales engineering
Role takeaway
- Fit
- Good for table-based scientific data with identifiable keys; evidence covers four biomedical datasets and one model family.
- Architecture and integration
- Profile files into compact evidence, call narrow proposal agents, run a 39-check verifier in structure and meaning loops, compile DDL, load via batched COPY, quarantine and reconcile.
- Prerequisites
- PostgreSQL target
- File profiling that produces citable evidence
- LLM endpoint
- Reviewer interface for schema and load approval
- Constraints
- Kept value on duplicate keys depends on file order
- Descriptive-column placement is unverified
- Wide matrices enlarge prompts
- No unconstrained-agent baseline
- Security
- Only compact evidence goes to the model, but controlled or identifiable data needs its own hosting, access and retention review.
- Proof of value
- Run local datasets through the pipeline and record first-proposal check failures
- Plant known key and encoding errors and confirm each is quarantined or logged
- Reconcile staged and loaded row counts
Delivery
Role takeaway
- Work
- Select pilot datasets, adapt check types to local conventions, set reviewer checkpoints and run the load with reconciliation.
- Dependencies
- Reviewer time
- Database environment
- Dataset documentation
- Ownership
- The research data service owns the pipeline and verifier rules; the dataset's investigator approves the schema and load.
- Skills and adoption
- Data stewards learn to review typed schema proposals and quarantine reasons rather than write loaders by hand.
- Governance checkpoints
- Schema approval before any table is created
- Load approval before rows are inserted
- Review of quarantined rows
- Acceptance criteria
- Proposed: every load reconciles to zero gap
- Proposed: planted errors are quarantined or logged
- Proposed: reviewer hours per dataset recorded
- Risks
- Rubber-stamp approvals
- Verifier rules missing local modeling issues
- Unreviewed quarantine backlog
- Prompt growth on wide matrices
Implementation considerations
Lighthouse Advisory interpretation across the operating dimensions a public-sector buyer must settle before this evidence becomes a design. Each note answers the question under its heading for this specific source.
Architecture and integration
What must connect, and where does the AI sit in the workflow?
Narrow agents receive measured evidence, never the full data or database access; a verifier compares typed proposals with evidence; code compiles DDL, reshapes data, quarantines unplaceable rows, records provenance and reconciles every staged row; two human checkpoints precede schema creation and load.
Governance
Who approves, reviews and stays accountable for outcomes?
Human approval of the schema and of the load, provenance records and quarantine reasons give an audit trail. Reviewers remain accountable; the verifier does not check where descriptive columns belong, which varied across runs on one dataset.
Security and privacy
What data, permissions and controls need testing?
Agents see compact evidence rather than raw data, which limits exposure to the model provider; the datasets studied were public, so controlled or identifiable research data would need separate hosting and access decisions that the paper does not address.
Accessibility and workforce
Who is affected, and what skills or accommodations follow?
Moves data-wrangling effort toward review and rule-writing; the paper offers no evidence on workload or training needs.
Procurement
What should contracts, pricing and exit terms secure?
When evaluating agentic data tools, ask for deterministic verification, provenance and reconciliation features and measured first-pass failure rates, not only success rates.
Operating model
Which teams own the service once it runs?
Supports a shared research-data service where agents draft schemas, code enforces rules and named humans accept each load.
Publication history
- 2026-11-10Research Universities · Issue 633 resources
Stable resource ID: verified-agentic-ingestion-scientific-tables-agents-propose-code-verifies-humans-decide-20261108