Finance Agent

Key Takeaways

• Gemini 3.8 Flash leads Finance Agent v2 with 61.44% accuracy. Muse Spark 1.2 follows at 60.60%, with Muse Spark 1.3 Max at 59.96%. The top three sit within 1.5 points of each other, while the two Muse Spark models cost about 60% less per test than Gemini 3.8 Flash.
• Models are able to handle simple retrieval tasks, but still struggle to perform reliably on harder, multi-step financial work that relies on precise numbers and specific industry convention.
• Only two models clear 60% with Partial Credit, and the order flips under stricter All-Pass scoring: Muse Spark 1.2 leads at 50.88%, ahead of Gemini 3.8 Flash at 49.69%, and no model reaches 51%.
• There is substantial room for improvement: no single model leads all nine question categories, and the hardest categories remain Financial Modeling and Precedents, where the category leaders reach only 34.52% and 36.37%, respectively.
• One of Opus 5’s 1,350 run-task results used Claude Opus 4.8 as a refusal fallback. That task already scored 0.0, so the published accuracy is unchanged.

Background

Finance Agent v2 builds on Finance Agent v1.1 with 927 expert-reviewed questions across public, private validation, and held-out test splits. Dataset design, splits, and access are covered in Methodology.

The benchmark tests a model’s ability to perform the work of entry-level financial analysts — answering difficult questions on public company filings.

Automating analyst-grade financial research is valuable because the work is expensive, repetitive, and time-sensitive: analysts often need to move from filings and transcripts to a defensible model or investment memo under deadline. The work is also difficult because the answer depends on finding the right source, applying the right finance convention, and carrying precise intermediate numbers through several steps.

Finance Agent v2 measures how well AI can aid analysts in automating busy work so they can focus on higher-leverage tasks.

Results

The Pareto chart above illustrates how model accuracy trades off against cost and latency. Gemini 3.8 Flash leads at 61.44% at $2.00 per test. Muse Spark 1.2 follows at 60.60% and Muse Spark 1.3 Max at 59.96%, each at roughly $0.77 per test, less than 40% of the leader’s cost. Muse Spark 1.3 Max is also the fastest of the three at about 3 minutes 19 seconds per task, narrowly ahead of Gemini 3.8 Flash, while Muse Spark 1.2 takes about 5 minutes.

The mid-tier models sit only a few points below the leaders at a fraction of the cost, suggesting the accuracy premium at the frontier is modest relative to the price gap.

Where Models Stand

The retrieval and summarization categories cluster at the top: General Quantitative, Earnings Analysis, General Qualitative, Market Analysis, and Disclosure Analysis all have category leaders above 70%. Adjustments tops out at 56.29%, while Comparables reaches 52.00% as cross-document reconciliation gets harder. Financial Modeling and Precedents remain the hardest categories, topping out at 34.52% and 36.37%, respectively.

When measuring models on their ability to perfectly answer questions, scores fall substantially across the leaderboard compared with allowing partially complete responses.

The strongest model reaches 50.88% All-Pass accuracy, the next group sits in the mid-to-high 40s, and mid-tier models land in the low 30s. Models are still unable to execute financial tasks perfectly, often missing core details that real analysts would include.

The tool call analysis reveals how many steps the models take to produce an answer. Interestingly, the top performers also register relatively high numbers of tool calls. This suggests that their comparatively strong performance is partly due to leveraging the tools provided.

The radar graph shows the per-model breakdown of tool calls across the six tools.

Calculator is the dominant tool for most frontier models. Higher-performing models tend to spend more of their tool budget on numerical resolution after finding the relevant source material. Lower-performing models lean harder on web_search and retrieve_information, making more exploratory calls that do not translate into correct answers. Across the models shown, price_history accounts for at most 4.22% of tool calls, matching its narrower role in Market Analysis questions.

Model Output Examples

The question above is a General Quantitative Analysis task: deciding whether Centene owed a rebate to policyholders in either fiscal year based on the medical loss ratio threshold, and reporting the underlying MLRs.

Trajectory Comparison

The visualizations below show how a frontier model and a smaller, cheaper model approached the same Financial Modeling question of building a DCF model. Each row is one agent turn; bar width is wall-clock time per turn (model reasoning + tool execution).

Claude Opus 4.7 trajectory (passed most checks):

The frontier model worked through the question in 12 turns: pull market data, fetch and parse the two filings, run retrieve_information to extract the inputs needed for the DCF, then step through bursts of parallel calculator calls to compute projections, terminal value, and discounted cash flows before submitting.

GPT 5.4 Nano trajectory (zeroed on most checks):

The smaller model needed 34 turns to reach an answer of similar shape — nearly 3x as many. After the retrieval phase, it hammers the calculator one operation at a time for the next 27 turns rather than fanning out parallel calls, and the resulting numbers land close to the rubric’s targets but not exactly enough to be rewarded.

Methodology

Agents are all evaluated on a shared default harness with access to six tools: edgar_search (SEC EDGAR API), web_search, parse_html_page (download an HTML page), retrieve_information (query over fetched HTML), calculator, and price_history.

Time Limit

Each task has a two-hour time limit. An answer must be produced within the time limit, otherwise the task is scored as a zero. The time limit was chosen empirically by measuring how long current frontier models need to converge on these tasks and setting the limit comfortably above that ceiling. A fixed time budget also allows us to normalize and compare performance across different agent implementations.

Grading

Each question is composed of weighted checks, and a subset are flagged as dealbreakers, load-bearing facts or numbers, that are required for a satisfactory answer. Failing any dealbreaker means the answer receives no credit for that question, regardless of the remaining content in the response. Two metrics are reported:

• Partial Credit (primary): the dealbreaker-gated, severity-weighted average of per-check scores. A response with a correct dealbreaker but a few peripheral misses scores below 100%; a response with any failed dealbreaker scores 0%.
• All-Pass (secondary): 100% only if every check passes, 0% otherwise.

All responses are graded by a three-judge LLM jury consisting of three frontier models: GPT-5.4, Gemini-3.1-Pro, and Claude Sonnet 4.6.

Changes from v1.1

• Step up in difficulty. Even the best models reach only 61.44% with Partial Credit and 50.88% under All-Pass grading. Questions require connecting data across multiple documents, tighter numeric precision, and answers are expected to include insights that skilled analysts would include but are not explicitly stated in the question.
• Expanded taxonomy. The taxonomy was reorganized around real analyst workflows rather than retrieval tiers. v1.1’s easier retrieval-focused buckets (Quantitative Retrieval, Qualitative Retrieval, Numerical Reasoning, Complex Retrieval, Beat or Miss, Trends) are replaced with Comparables, Precedents, Earnings Analysis, Disclosure Analysis, and a split between General Qualitative and General Quantitative analysis.
• New grading mechanism. Dealbreaker-gated Partial Credit replaces v1.1’s flat per-question score, and All-Pass is reported alongside it as a strict secondary metric.
• Stricter numeric tolerances. Tighter thresholds on rounding and precision drift. Answers that previously passed under v1.1’s looser tolerance now fail.
• Expanded harness. Adds calculator and price_history on top of the v1.1 tools.
• Multi-run aggregation. Every model is run three times; reported scores are mean-of-runs with standard error of the mean.
• Expanded test set. Larger held-out test split for tighter measurement.

Question Design

Questions target the analytical depth expected of a 2nd or 3rd-year investment banking analyst. Each question was designed to satisfy four criteria:

• Determinism. A single, unambiguous correct answer with no room for competing interpretations.
• Multi-source synthesis. Answers require chaining information across multiple filings or data sources rather than a single lookup.
• Domain specificity. Questions require implicit industry knowledge that follows sector convention rather than explicit instruction.
• Forensic precision. Critical information is frequently buried in footnotes, MD&A caveats, or accounting policy disclosures.

Dataset

The dataset is divided into three parts: Public (27 open-source samples), Private Validation (450 samples available for license), and Test (450 samples).

• The Public set and agent harness are fully open and can be accessed here.
• The Private Validation set is available for license. Interested parties are encouraged to contact us directly for access.
• The Test set will remain private. All results reported on this page are based solely on the Test set to prevent overfitting.

The dataset splits were sampled to preserve the distribution of question categories and difficulty.

Question Taxonomy

Finance Agent v2 organizes questions into nine analytical categories reflecting real equity-research workflows.

General Qualitative Analysis

Summarization and comparison of fundamental filing sections: business model, risk factors, MD&A, and standard disclosures across companies.

Compare Walmart, Costco, and Target’s capital allocation priorities across capex, dividends, share repurchases, and debt management.

General Quantitative Analysis

Extraction and calculation of reported financials such as revenue growth, CAGR, leverage ratios, and executive compensation — often requiring verification against restated historicals.

Compare Home Depot and Lowe’s FY2024 inventory efficiency and calculate the difference in days inventory outstanding.

Market Analysis

Relative trading performance, total shareholder return, and how news cycles or guidance shifts drive stock volatility relative to sector indices.

Measure Sun Communities’ stock reaction after the announced sale of Safe Harbor Marinas, then relate the move to the company’s stated use of proceeds.

Comparables

Building trading comps tables, calculating EV multiples, and normalizing enterprise value across peers by adjusting for off-balance-sheet items buried in footnotes.

Rank major U.S. banks by excess CET1 ratio relative to their regulatory minimums.

Precedents

Analyzing M&A transaction multiples from S-4 filings and target financials, with industry-specific EBITDA normalization (e.g. exploration expense add-backs in Oil & Gas).

Extract enterprise values and EV/EBITDA multiples for recent industrial distribution acquisitions and rank the transactions by pre-synergy multiple.

Adjustments

Bridging GAAP to non-GAAP or pro forma figures by reconciling SBC, acquired intangible amortization, and other non-cash items across the P&L and cash flow statement.

Reconcile Honeywell’s GAAP operating income to segment profit across annual releases and identify newly introduced adjustment categories.

Earnings Analysis

Comparing reported results against consensus estimates and prior guidance, including non-GAAP reconciliations across consecutive quarterly reporting cycles.

Compare Rapid7’s Q3 2025 actuals against prior revenue, non-GAAP operating income, and ARR guidance.

Disclosure Analysis

Tracking shifts in MD&A language, KPI definitions, and segment reporting methodology across multiple annual filings, then restating prior periods to reflect the new format.

Track Boeing’s segment reporting and 787 cost-recovery disclosures across FY2022-FY2024 10-K filings.

Financial Modeling

Multi-step frameworks including DCF/NPV, LBO, and M&A accretion/dilution models built from historical ratios extracted from primary filings.

Assess whether Ralph Lauren could justify a distressed acquisition of Capri under stated synergy, margin, and valuation assumptions.

Acknowledgements

We would like to thank Andrew Schettino and all of the financial experts who worked on Finance Agent v2.

Citation

If you use this benchmark in your research, please cite the paper.

@misc{bigeard2025fab,
title        = {Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks},
author       = {Bigeard, Antoine and Nashold, Langston and Krishnan, Rayan and Wu, Shirley},
year         = {2025},
month        = may,
howpublished = {Vals AI},
url          = {https://arxiv.org/abs/2508.00828},
}