Datalab Releases Open-Source Omni Extract Bench for Structured-Document Extraction Evaluation
Summary
Datalab releases Omni Extract Bench, an Apache-2.0 open-source benchmark that evaluates structured-document extraction from Datalab, Reducto, Extend and LlamaExtract with normalized JSON scoring, Hungarian array matching and provider-paid API runs.
Key Points
- Datalab releases Omni Extract Bench, an Apache-2.0 benchmark for fair structured-document extraction evaluation.
- The benchmark runs Datalab, Reducto, Extend and LlamaExtract through a resumable CLI harness that requires API keys and incurs provider costs.
- Its scorer normalizes and flattens JSON outputs, uses Hungarian matching for arrays, and reports accuracy, precision, recall, F1 and detailed error verdicts.