FinVault: Benchmarking Financial Agent Safety in Execution-Grounded Environments

An anonymized benchmark release for stress-testing financial agents before they touch real money, real customers, or real compliance workflows.

Data | Examples | Usage | Validation | Citation

FinVault is designed for a failure mode that ordinary Q&A safety tests often miss: a financial agent can sound safe while still taking an unsafe action through tools, databases, or workflow state transitions. FinVault evaluates agents in isolated sandbox environments where attacks are judged by observable execution outcomes rather than by text alone.

• Execution-grounded safety evaluation: every scenario is backed by a sandbox environment, tool interface, state model, and task-specific vulnerability checks.
• Finance-native risk coverage: 31 regulatory-style scenarios span credit, insurance, securities, payments, AML/compliance, and risk management.
• Attack and benign parity: paired attack and normal datasets let users measure both attack success and over-refusal risk.
• Reproducible release checks: packaged tests validate dataset structure and scan for common secret and PII patterns.
• Anonymized public package: raw private data, logs, model outputs, API credentials, paper build artifacts, and working notes are excluded.

• Financial agents require tool-aware safety evaluation; response-level moderation alone cannot verify whether money movement, account changes, or compliance decisions were actually safe.
• Social and authority-based attacks remain important because they target operational trust boundaries rather than only model instruction-following.
• Defensive prompting can be evaluated alongside external safeguard models through the same sandbox harness, making false positives and blocked unsafe actions directly comparable.

• 2026-06-04: FinVault anonymized benchmark release is published with sandbox code, datasets, release checks, and documentation.
• 2026-06-04: The release package includes 31 scenarios, 107 attack cases, 107 normal cases, and 856 synthesized attacks.

FinVault contains six financial domains and eight synthesized attack families. The public package keeps the benchmark executable while replacing direct private identifiers with neutral placeholders.

FinVault examples are stored as structured JSON files under:

sandbox/attack_datasets/
sandbox/normal_datasets/
sandbox/attack_datasets_synthesis/

Each sandbox scenario contains its own environment, state, tools, prompt template, and vulnerability checks under sandbox/sandbox_00 through sandbox/sandbox_30.

pip install -r requirements.txt

cd sandbox
python run_attack_test.py --scenario 00
python run_attack_test.py --all --concurrency 8

cd sandbox
python run_llama_guard3_test.py --scenario 00
python run_llama_guard4_test.py --scenario 00
python run_gpt_oss_safeguard_test.py --scenario 00

Model configuration lives in:

sandbox/config/agents_config.yaml

Use environment variables or local untracked files for real credentials. Do not commit .env files or API keys.

Run the release checks before using or redistributing the package:

python scripts/check_release.py
python -m unittest discover -s tests
python -m py_compile sandbox/run_attack_test.py sandbox/run_llama_guard3_test.py sandbox/run_llama_guard4_test.py sandbox/run_gpt_oss_safeguard_test.py sandbox/defense_evaluation.py

Expected release-check output:

{
  "status": "ok",
  "dataset_json_files": 310
}

sandbox/
  attack_datasets/               # anonymized attack cases
  normal_datasets/               # anonymized normal business cases
  attack_datasets_synthesis/     # anonymized synthesized attack cases
  sandbox_00 ... sandbox_30/     # execution-grounded financial environments
  base/                          # shared abstractions
  defense/                       # safeguard integrations
  attack_testing/                # LLM-agent test harness
  config/                        # public placeholder configuration
  prompts/                       # scenario prompt templates
finvault_dataset/                # loading, validation, and privacy helpers
scripts/check_release.py         # release integrity and privacy scan
tests/test_release_integrity.py  # unittest checks
static/                          # README figures and visual assets

This release is intentionally anonymized. It excludes original non-anonymized data, private working notes, API keys, .env files, generated logs, model outputs, paper source files, and review materials. The anonymization summary is provided in anonymization_summary.json.

@misc{finvault2026,
  title        = {FinVault: Benchmarking Financial Agent Safety in Execution-Grounded Environments},
  author       = {Anonymous Authors},
  year         = {2026},
  howpublished = {GitHub repository},
  note         = {Anonymized benchmark release}
}

The code and data are provided for research use. Users are responsible for complying with applicable institutional, legal, and model-provider policies when running evaluations.

FinVault is released as an anonymized research artifact for studying financial-agent safety, execution-grounded evaluation, and operational risk in tool-using AI systems.