On 6 October, Togal.AI published Construction’s Last Exam, a public benchmark for whether an AI model can do a quantity takeoff, with a per-model scoreboard and a downloadable score file. Its headline result is that general-purpose frontier models are nowhere near: the best scores 34 out of 100, and most score under 20.
Eleven weeks earlier, a different vendor published a different takeoff benchmark, on which frontier models score in the 40s and 50s. Neither company is wrong. Working out why they disagree is the most useful thing a contractor can take from either.
The benchmark describes itself as “a multimodal benchmark for quantity takeoff — wall centerlines, areas, and symbol search on construction drawings.” It contains 33 task instances, split 12 Areas, 12 Counting and 9 Linear. The overall score, per the page, is “mean of Areas, Linear, and Counting.”
Togal.AI is a Miami takeoff-software company, which is both why the benchmark exists at all and the first thing a reader should know about it. Both facts are addressed below.
The Scoreboard
The benchmark publishes its raw numbers as a CSV at progress-chart/scores.csv, which this publication fetched directly rather than reading off the chart. It carries 24 general-purpose model entries plus one entry for Togal’s own product, dated between January 2025 and 13 October 2026.
The highest-scoring general-purpose model on the file is OpenAI’s GPT-6 Sol at 34.0, dated 22 September 2026. Google’s Gemini 4 Argon, the most recent entry on the file at 13 October 2026, scores 25.7. GPT-6 Luna scores 21.6. GPT-6 Astra and Gemini 3.8 Flash both score 19.2. Gemini 3.7 Flash scores 16.6 and Anthropic’s Claude Opus 5.5 scores 16.2. DeepSeek Flash 4.1 scores 12.3.
At the bottom, Zhipu’s GLM 4.6V and Alibaba’s Qwen 3.8 Max both score 1.7, with Qwen 2.5 VL 72B and Claude Sonnet 4.5 just above at 2.2 — the range behind the trade-press framing of “as low as 2%”, which names no model. Eight of the 24 entries score under 5.
The one entry not from a model developer is Togal’s own, at 97.7. The benchmark page does not present this as a model score. It labels it explicitly: “Togal.ai is a reference mark (human-in-the-loop, max. 30 seconds).” It is a different kind of measurement from the rest of the file — a person working inside purpose-built takeoff software, time-capped — and the page says so. Readers comparing 97.7 against 34.0 should carry that label with them, because the page itself does.
How the Tasks Are Run
Per Construction Briefing’s 13 October report by Andy Brown, models were handed raw PDF drawings and allowed to use their own built-in vision and coding capabilities, but not third-party tools or agentic workflows. Outputs were scored against ground truth using standard computer-vision metrics — intersection over union, F1 and mean average precision. That report names Frank Gabel, an AI engineer, as the benchmark’s developer, and says Togal tested API-accessible frontier models from OpenAI, Anthropic, Google, xAI and selected Chinese providers.
The task definitions on the benchmark page are specific about what is being measured. Linear tasks “extract wall runs, centerlines, building outlines, openings, and boundary geometry.” Areas tasks “trace room, floor, finish, material, paving, landscape, hatch, and pattern regions.” Counting tasks “detect, count, locate, and classify doors, fixtures, devices, plants, and repeated symbols.”
That is geometry, and the page says so: it describes this as a “geometry-only release,” with a fuller suite covering reasoning and end-to-end tasks previewed separately. The page also publishes a cost column, noting that “API cost scales published rates so GPT-6 Astra ≈ $14.0 for all tasks” — a useful reminder that the comparison is not only about accuracy.
The no-tools constraint is the most important thing about the setup and the easiest to lose. These scores measure a raw model reading a PDF. They do not measure what a model does inside a product built around it, with vectorisation, a drawing parser, a symbol library and retrieval over past projects in front of it. No production takeoff product this publication is aware of works that way, which means the benchmark is a floor rather than a forecast.
Who Built It, Stated Plainly
Togal.AI sells takeoff software. Its own site advertises “up to 98% accuracy” on drawings and “98% accuracy on floor plans.” Its benchmark page credits the “Togal.ai ML Team” with “benchmark design, vectorization workflows, and task authoring,” and says “internal Togal.ai estimators validate ground truth and review architectural drawing analysis tasks,” with “room for future community contributors.”
So the benchmark’s tasks were authored by a vendor, its ground truth was validated by that vendor’s own estimators, and its top line is that vendor’s product. None of that is concealed — all of it is printed on the page — and none of it makes the frontier-model scores wrong. It does mean the scores carry one organisation’s judgement about what a correct answer looks like on 33 drawings, with no external review of that judgement yet published.
The benchmark page points to a public repository for the task set: git clone https://github.com/Togal-ai-Team/tle-bench.git, with drawing PDFs and prediction dumps in a linked Google Drive folder. This publication tried to open that repository and received a 404. Because an unreachable page is weak evidence on its own, the control was the same organisation’s own GitHub page, which loads normally and lists six public repositories — bdr_bear, forks of mmdetection and mmsegmentation, eslint-config-togal, and archived copies of cvat and click — none of them tle-bench. So the 404 is a real response rather than a blocked session. It is still not evidence of much: a private, renamed or not-yet-published repository returns exactly the same thing. The task set could not be inspected, and nothing in this piece rests on having read it.
The “First” Claim, and the Benchmark That Got There First
Togal’s 6 October launch release calls Construction’s Last Exam “the first AI benchmark built for the construction industry.” That is the company’s characterisation, and it does not survive a search.
Several construction benchmarks predate it, and one of them is a takeoff benchmark with a public per-model table. Handoff published the Construction Blueprint Takeoff Benchmark (TAKEOFFBENCH-V1) on 21 July 2026, with the accompanying paper posted to arXiv on 15 August 2026. It uses 10 residential blueprint sets from real permissioned projects, with 2,009 expert-validated line items across nine trades, of which 1,348 primary-tier materials are scored on coverage and quantity precision and combined into a composite.
Separately on the academic side, AECV-Bench was posted to arXiv on 8 January 2026 by Aleksei Kondratenko, Mussie Birhane, Houssame E. Hsain and Guido Maciocci, evaluating multimodal models on 120 floor plans for object counting and 192 question-answer pairs for drawing-grounded document QA. Its findings rhyme with Togal’s: models did well on text extraction, reaching up to 0.95 accuracy, and poorly on symbol-centric understanding, with door and window counting at roughly 0.40 to 0.55. Its authors conclude that current systems work as document assistants but lack robust drawing literacy.
So the superlative is reported here as Togal’s wording, not as fact. What CLE does appear to have that the others do not is a continuously updated, dated scoreboard — a trend line, with a new row added as recently as 13 October — which is a narrower claim and, for a reader trying to work out which way this is going, the more useful property.
The Two Takeoff Benchmarks Disagree, and That Is the Story
Put the two side by side and they do not say the same thing about frontier models, which is more interesting than either in isolation.
On CLE, the best general-purpose model scores 34.0 and most score under 20. On TAKEOFFBENCH-V1, the published table runs: Handoff-H1 at 81.6 composite, independent human estimators at 77.6, then claude-fable-5 at 61.4, gpt-5.6 at 58.6, claude-opus-4-8 at 54.6, Kimi-K2.6 at 54.5, claude-sonnet-4-6 at 53.9, GLM-5.2 at 41.6 and DeepSeek-V4-Pro at 34.7. Frontier models in the 40s and 50s on one benchmark; the low teens and 20s on the other.
The analysis from here is this publication’s own, and the gap is not evidence that either party is wrong. The two measure different things under different rules. CLE scores takeoff geometry — wall centerlines, traced areas, symbol counts — against ground truth using computer-vision metrics, with models reading raw PDFs and explicitly barred from tools and agentic workflows. TAKEOFFBENCH scores material line items for coverage and quantity precision, judged by an LLM, on residential sets. Tracing a polygon to a given intersection-over-union is a harder thing to do with a raw vision model than naming the materials a drawing implies and getting the quantity within 25%.
Two vendor benchmarks, two vendors at the top of their own tables, two different numbers for how good the frontier is. That is not a scandal; it is what an industry looks like before it has agreed a yardstick, and it is the strongest argument for the thing neither benchmark has yet: tasks authored and ground truth validated by somebody who is not selling takeoff software.
What the Trend Line Says
The analysis from here is this publication’s own.
The single number everyone will quote is 34. The more informative thing in the file is its shape over time. The earliest entry, from January 2025, scores 2.2. The highest, from September 2026, scores 34.0. Twenty months of frontier model releases moved raw PDF takeoff from roughly nothing to roughly a third — on a 33-task geometry set, with no tooling allowed.
Two readings are available and both are defensible. The first: 34 is unusable. The score is a mean of computer-vision agreement metrics rather than a percentage of a takeoff done right, but no reading of it describes work an estimator could hand to a bid team, and a number built on that would be a liability event. On that reading the benchmark is a corrective to a sales pitch, which is roughly how the trade press has used it.
The second reading: a fifteen-fold improvement in under two years on a task nobody was optimising for is not a plateau, and the models were reading raw PDFs with their hands tied. The gap between 34 and the high-90s a production tool claims is currently filled by engineering — vectorisation, parsers, symbol libraries, human review — and the open question is how much of that scaffolding survives as the base models get better at reading a drawing.
This is consistent with what contractors have actually been reporting. Our June look at AI estimating accuracy found that the high accuracy figures hold up on clean, well-formatted drawings with consistent symbols and fall apart on marked-up, scanned or inconsistent sets — which is the same finding this benchmark reaches from the other direction, by taking the scaffolding away. It also sits beside the adoption picture: Bluebeam’s 2027 outlook put 28% of US AEC firms running AI in production, against 27% a year earlier, and UpCodes’ pass over 1,300 plan sets showed what AI review of drawings catches when it is pointed at compliance rather than measurement.
What the industry still has not got is a shared yardstick. It now has at least two vendor-built ones that disagree, which is a step forward from having none and a long way from settled. The useful next step is not a higher score on either; it is tasks authored and ground truth validated by somebody who does not sell takeoff software — something CLE’s own credits leave room for, listing “room for future community contributors.” Until that happens, both scoreboards are genuinely informative artefacts with one organisation’s hand on the questions and the answer key, and the honest way to read either is as a floor on a narrow task rather than a verdict on what AI can do with a drawing.
