Choosing the right AI model is an operating decision. It requires understanding how models fail, what a practitioner needs to review or correct, and how much time and money models take to complete tasks.
Today, we’re sharing our first set of benchmarks drawn from work at Current and Shield, our accounting and IT services platforms. We believe the field needs more benchmarks tied to economically valuable work, so builders can understand where models are useful in real-world environments and where they can still improve.
These benchmarks measure model performance against practitioner-defined standards. At Holdings, these tests help us compare model configurations and choose which models our systems run on for specific tasks.
Owning and operating these businesses gives us a direct view into what is required for each task. Our engineers develop these tests with our practitioners, using their decisions and corrections to define what successful work requires. This follows a founding principle of Holdings: build with the people who understand the work.
What the first results show
Notably, more spending can improve performance but the size of the gain varies considerably by task and model. To assess what additional spending can achieve, we use GPT-6 Luna set to medium reasoning as a baseline across both benchmarks. This configuration ranked second on ‘score per dollar’ across all tested configurations and achieved higher accuracy than its lowest-cost setting, making it a useful reference point for comparing the cost and performance of other configurations.
For tax, Claude Opus 5.5 scored the highest accuracy overall, 76.28%, at $5.04 per return. Meanwhile, GPT-6.1 Sol medium achieved the highest score per dollar among tested configurations. It scored 74.18% at $1.37 per return compared to Luna medium’s 67.61% at approximately $0.11. That is a 6.57 percentage-point improvement in exchange for $1.26 per return. Switching from Sol medium to Opus high adds 2.11 percentage points for $3.67 per return.
For IT, DeepSeek V4.1 Flash low scored the largest pass-rate improvement per additional dollar over the baseline. It passed 25 of 52 tickets, compared with 14 for Luna medium. For about six cents more per attempt, the observed pass rate increased from 26.92% to 48.08%. GPT-6 Astra xhigh achieved the highest pass rate overall, passing 32 of 52 tickets (61.54%) at $7.63 per ticket, compared with approximately $0.13 for DeepSeek low.
We also examined the specific errors behind the scores. In a tax example, a model missed a documented repayment and overstated taxable retirement income. In an IT support example, an agent replied to a customer despite an explicit requirement to let the engineer respond. These examples show two challenges: properly connecting the evidence needed for a correct answer and respecting the boundaries of a task. Understanding both helps us improve our production systems and identify where in our processes practitioner judgment remains necessary.
Learning from the work and where we go from here
At Holdings, we use this work to continuously improve the systems our practitioners rely on and the service our customers receive. When a practitioner’s corrections reveal a recurring problem, we turn this into an eval. We then test whether changes to the model and the harness improve performance on the task.
We’re sharing the results so other builders, researchers, and business owners can examine the work and learn from our approach. We will update and expand our benchmarks as we build new products and expand to additional industries.