State-of-the-art models offer high performance, but document analysis is even more difficult.

The latest generation of Frontier models achieves things that were impossible a year ago. In benchmarks involving complex inference, they achieve scores at or near the level of human experts. They handle context windows on the scale of 1 million tokens.
We write code, analyze financial models, process documentation, and produce output that can withstand scrutiny by real experts. Progress is genuine, and benchmarks reflect that.

If you're building pipelines that process real-world business documents like invoices, financial statements, contracts, and technical manuals, you've probably looked at these models and wondered, "Just how far can a good model go?"

The benchmark results are good, and the performance looks promising.
So, what actually happens if we use a simple extraction prompt to point one of those models at a large amount of real-world documents and ask it to do its job?

That's exactly what we wanted to know.

How to conduct the test

We used SCORE-Bench, a publicly available benchmark for document analysis. SCORE-Bench consists of 224 real-world documents that reflect the documents actually handled in enterprise environments.
Examples include scanned invoices with tilted text, financial reports containing deeply hierarchical tables, medical forms, legal contracts, and technical documents with complex multi-column layouts. The complete dataset is publicly available on Hugging Face.
We tested Claude Opus 4.6, GPT-5.2, Claude Sonnet 4, GPT-5-mini, and Gemini 2.5 Pro. For each model, we provided only a simple extraction prompt and made no additional configurations. For the baseline comparison, we used our VLM Partitioner pipeline, which, in addition to Claude Opus 4.5, incorporates optimized extraction prompts, post-processing, and mechanisms to enforce output structure.

The difference between these two conditions is what this result measures.

result

In all five models, the overall accuracy, as measured by the combined metrics, was 4 to 16 points lower than our pipeline. The overall picture is as follows:

The differences are not uniform across all metrics. For some metrics, particularly hallucination suppression, some models show results remarkably close to our pipeline. On the other hand, for metrics such as table extraction and document structure understanding, the differences are considerably larger. It is precisely in these areas that we find the more interesting findings.

Points where the gap widens

Hallucination rate

This was the most surprising discovery for us.
The hallucination rate is an indicator that measures how many of the tokens generated by the model are not present in the original document.
In this metric, Opus 4.6 stood out. Opus 4.6's value was 0.044, while our pipeline was 0.043. In practical terms, they are almost equivalent.

GPT-5.2 was 0.167, which was about four times higher than our pipeline's value.
The GPT-5-mini had a value of 0.161, while the Gemini 2.5 Pro had a value of 0.257.

What does this mean for your pipeline?
Hallucinated content doesn't tell you "this is hallucinated." It looks just like regular extracted text.
The data flows directly into the vector store, becomes searchable, and is used as input to the LLM as if it were real information.
As a result, in the later stages, there is a possibility that "confidently incorrect answers" may be generated in a way that is difficult to detect.
Opus 4.6's fundamental capabilities in this regard are genuine. It reads documents and does not arbitrarily create content that does not exist.
On the other hand, Opus 4.6's weakness lies in its structure. This is a completely different issue.

The opposite of hallucination is coverage.
In other words, it's an indicator of how much of the original document's content is actually included in the output.
In this respect, Opus 4.6 scored 0.737, the lowest result among all the models tested.
In other words, Opus 4.6 hardly causes hallucination, but it misses about a quarter of the content of the original document.

It's worth understanding this trade-off.
Models that find more content tend to create content that doesn't actually exist.
GPT-5.2 and GPT-5-mini achieve higher coverage scores, but they also have higher hallucination rates.
Gemini has a lot of information, but it shows the highest hallucination rate among this group.
High recall, meaning the ability to extract a lot of necessary information, and low hallucination rate, meaning not creating information that doesn't exist.
Whether or not both of these conditions are being met simultaneously is reflected in the numbers at the top of the graph.

For your pipeline, missing content means a gap in search results.
When an LLM receives a question, if the relevant information that is actually needed has not been extracted in the first place, the answer will be incomplete or incorrect.
This is a less conspicuous failure pattern than hallucination, but it is equally a real problem in terms of its impact on subsequent outcomes.

Table Extraction

The table contains a great deal of information that is extremely valuable to companies.
Examples include financial figures, price lists, and specification lists.
Mis-extracting these is not simply a matter of minor precision.
This is a data integrity issue.
Regarding table extraction accuracy, all five models evaluated performed worse than the Company pipeline, with a maximum accuracy difference of 23 points.

The most important metric here is not simply whether the model can correctly read the text within the cell.
The question is whether you understand which row and column the text belongs to.
For example, even if the model can perfectly read the text "4.2 million," it might still place it in the wrong column.
At that point, even if the numbers themselves are correct, the meaning they convey is incorrect.
The failure pattern you'll encounter in the later stages of processing isn't garbled text.
This data appears correct on the surface, but is structurally incorrectly positioned.

Document structure

This is the item that showed the biggest difference in this data, and at the same time, it is the most difficult problem.
Element mapping is a measure of whether a model correctly identifies headings, paragraphs, tables, figures, etc., and arranges them in the correct order.
In all the models we evaluated, this metric underperformed the Company pipeline, with the difference reaching as much as 16 points.

This is a serious problem because failures in element mapping are often difficult to detect.
The text itself exists. The document is also being processed.
However, the structural relationships between the elements are incorrect.
Paragraphs may be linked to the wrong section, figures may be separated from their captions, and subheadings may be treated as part of the main text.
When such chunks are submitted to the RAG pipeline, the context of those chunks becomes broken.
LLM uses the correct words, but processes them in the wrong order and relationships.
The resulting answers will reflect this effect, but there's no clear sign that something was wrong.
This is where simple prompts lag the most, and where the difference between raw model calls and optimized pipelines becomes most apparent.

Output consistency

Aside from accuracy, there are other concerns that should be raised regarding production operation.
Gemini 2.5 Pro returned inconsistent output formats for 12 out of 224 documents, or approximately 5% of the entire dataset.
Therefore, those outputs required post-processing before scoring could even begin.

At that percentage, it's no longer simply a matter of some documents having low accuracy.
This problem arises because some documents cannot be directly imported into the pipeline.
When implemented on a large scale, this means additional engineering work to handle exceptions.
Furthermore, unless a fallback layer is incorporated, the pipeline will not be fully automated.

The true nature of the gap

All of the models we tested this time possess high capabilities.
The hallucination results clearly demonstrate this. Opus 4.6 is able to read documents without creating content that is not present in the document.
In other words, this difference is not a problem with the model's capabilities themselves, but rather a problem with its settings and configuration.
A simple extraction prompt does not instruct the model on how to handle joined table headers, the order in which document elements should be arranged, how much content to extract, or what output format to use.
We simply request that the model "extract" the information, and we're hoping that the model will handle the rest.
For many tasks, that's perfectly fine.
However, when parsing documents on an enterprise scale, the details that are not specified in simple prompts are precisely where the process can break down.
There are three layers between raw model calls and the production-ready pipeline.
The first is an optimized prompt.
This provides the model with specific instructions regarding the structural requirements for document parsing. For example, it specifies how to handle headings, how to deal with irregular table layouts, how to arrange document elements, and how to balance coverage with accuracy.
The second is post-processing.
Post-processing normalizes the output and addresses edge cases that may occur even with properly designed prompts.
The third is the enforcement of the output structure.
This is a mechanism that defines what constitutes a valid response before the model generates an answer.
Each of these layers corresponds to a specific failure pattern observed in these results.
The difference in Element Alignment is a problem with prompt design.
The recall trade-off observed in Opus 4.6 is also a problem with prompt design.
The output format mismatch observed in Gemini is a problem with the output structure.
If even one of these layers is missing, the impact will be reflected in the numbers.
Performance improvements don't come from the model alone.
It is created by what you build around that model.

Please try it with your own document.

All the models we tested were powerful, and the results reflect that.
At the same time, these results demonstrate that general-purpose model performance and document parsing performance are two different things.
There is a definite difference between calling a raw model and having a production-ready pipeline.
And that gap can be bridged.
However, that requires more than just a prompt.