Dependency-Aware Chain-of-Thought Compression for Financial Reasoning

We employ a bidirectional LSTM network augmented with conditional random fields to identify segment boundaries. Given the token sequence $\mathbf{c}=(c_{1},c_{2},\ldots,c_{T})$, the network first computes contextual representations:

The concatenated representation $\mathbf{h}_{t}=[\overrightarrow{\mathbf{h}}_{t};\overleftarrow{\mathbf{h}}_{t}]$ captures both preceding and following context, which proves essential for detecting boundaries that depend on what comes next. Early experiments with unidirectional models showed poor performance on boundaries preceding numerical computations, where the boundary significance only becomes apparent from the subsequent calculation.

The boundary probability at each position is computed through a feedforward layer:

where $\mathbf{e}_{t}\in\mathbb{R}^{2}$ represents emission scores for boundary and non-boundary labels.

To ensure globally consistent segmentation, we apply a linear-chain CRF layer that models label transitions:

where $\mathbf{A}\in\mathbb{R}^{2\times 2}$ is the transition matrix and $Z(\mathbf{c})$ is the partition function computed via the forward algorithm. The CRF layer prevents degenerate solutions such as consecutive boundaries or excessively long segments, which frequently occurred with independent classification.

Financial reasoning text presents unique segmentation challenges that required specific adaptations. Numerical expressions spanning multiple tokens, such as percentage changes or currency amounts, must remain intact within segments. We address this by incorporating a numerical span detector that identifies contiguous numerical expressions:

During CRF decoding, we modify transition scores to prohibit boundaries within detected numerical spans by setting
$\mathbf{A}[y_{t-1},{\scriptstyle\text{BOUNDARY}}]=-\infty$ when $\text{NumSpan}(t)=1$ and the span continues from position $t-1$.

Additionally, financial text frequently contains table references and structured data mentions that should not be split. We found that simply expanding the context window was insufficient; instead, we pretrain the boundary detector on a auxiliary task of table cell boundary detection, which transfers effectively to reasoning text segmentation.

Each segment $s_{k}$ becomes a node in the graph. We compute node embeddings by mean-pooling token representations from a frozen language model encoder:

To capture segment-level semantics beyond token averaging, we apply a learned projection with residual connection:

This additional projection layer allows the model to learn task-specific representations while leveraging the pretrained encoder’s semantic knowledge. We experimented with fine-tuning the encoder but found that freezing it and adding the projection layer yielded comparable performance with substantially reduced memory requirements during training.

Edges represent logical dependencies where the source segment provides information required by the target segment. We predict edges using a biaffine attention mechanism that has proven effective for dependency parsing:

where $\mathbf{h}_{i}=\text{MLP}_{\text{head}}(\tilde{\mathbf{v}}_{i})$ and $\mathbf{h}_{j}=\text{MLP}_{\text{dep}}(\tilde{\mathbf{v}}_{j})$ are head and dependent representations computed through separate multilayer perceptrons.

The edge probability is obtained via sigmoid activation:

A critical challenge we encountered was handling long-range dependencies that span many intermediate segments. The biaffine scorer tends to underestimate such dependencies due to the difficulty of directly relating distant segments. Our solution introduces a path-augmented scoring term:

where $\Phi_{ij}=\max_{p\in\text{Paths}(i,j)}\prod_{(a,b)\in p}P(e_{ab}=1)$ captures transitive dependency strength.

The dependency graph must be acyclic to represent valid logical flow. We enforce this through a differentiable acyclicity regularization term based on the matrix exponential characterization:

where $\mathbf{A}\in[0,1]^{K\times K}$ is the adjacency matrix with $A_{ij}=P(e_{ij}=1)$, and $\odot$ denotes element-wise multiplication. This regularizer equals zero if and only if the expected graph is acyclic.

During inference, we apply a greedy pruning procedure that removes the lowest-probability edge from any detected cycle, iterating until the graph is acyclic. In practice, the regularization during training ensures cycles are rare, typically requiring fewer than two pruning iterations.

The importance scorer employs a dual-encoder architecture that separately encodes the question and each reasoning segment, then computes relevance through cross-attention. This design allows efficient scoring of all segments with a single question encoding pass.

The question encoder applies self-attention layers to produce a contextualized representation:

where $n$ is the question length and $L$ is the number of layers.

For each segment, we compute cross-attention scores against the question:

The attended question representation $\mathbf{q}_{k}$ captures which aspects of the question segment $s_{k}$ addresses. The importance score combines this relevance signal with segment-intrinsic features:

where $f_{k}$ is a feature vector containing segment length, position, numerical content indicators, and graph centrality measures.

An important trick we discovered is incorporating the segment’s graph centrality into the importance score. Segments with high out-degree, meaning many other segments depend on them, tend to contain foundational information that should be preserved. We compute PageRank scores on the dependency graph and include them in $f_{k}$:

where $d=0.85$ is the damping factor.

Selecting the optimal subset of segments is formulated as a constrained optimization problem:

subject to:

The first constraint enforces the length budget, while the second ensures that if a segment is selected, all its dependencies are also selected. This problem generalizes the knapsack problem with precedence constraints.

We solve it via dynamic programming on the topologically sorted graph. Let $\text{dp}[k][l]$ denote the maximum importance achievable considering segments $1,\ldots,k$ with total length exactly $l$. The recurrence is:

where the second case requires $l\geq|s_{k}|$ and $\text{DepsSelected}(k)=\text{true}$.

The time complexity is $O(K\cdot L_{\text{budget}})$, which is efficient for typical reasoning chain lengths. We implement the dependency check through bit manipulation, maintaining a bitmask of selected segments and verifying parent inclusion in constant time after preprocessing parent masks.

For each pair of consecutively selected segments $(s_{i},s_{j})$ where intermediate segments were removed, we extract boundary context:

where Suffix and Prefix extract $w=15$ tokens from segment ends and beginnings respectively.

A lightweight transformer-based sequence-to-sequence model takes boundary context as input and generates a smoothed transition:

To prevent hallucination, the decoder employs a copy mechanism that biases generation toward input tokens:

where $\lambda_{t}$ is a learned gate and $\alpha_{ti}$ are copy attention weights. The rewriter is trained on synthetic boundary pairs created by randomly removing segments from correct reasoning chains, avoiding error propagation from the full pipeline.

We verify factual consistency by computing embedding similarity between the original boundary region and the rewritten version:

If similarity falls below threshold $\tau$, we fall back to simple concatenation with a generic connective phrase.

The segmentation network is pretrained on manually annotated reasoning chains using CRF negative log-likelihood:

The dependency graph predictor is pretrained on synthetic dependency data generated by prompting a large language model to annotate reasoning chain dependencies.

The importance scorer and graph predictor are trained jointly using a multi-task objective:

The scoring loss uses ground-truth labels derived from oracle compression identifying the minimal segment subset producing correct answers:

The complete pipeline is fine-tuned using reinforcement learning with a reward combining accuracy and compression:

We employ REINFORCE with baseline subtraction to reduce gradient variance:

where $b$ is an exponential moving average of recent rewards.

We use a frozen Qwen3-4B model as a semantic feature extractor, accessing its intermediate layer representations to initialize segment embeddings. Specifically, we extract features from layer 16 of 32, which empirically balances semantic abstraction with surface-level detail:

The frozen encoder provides rich pretrained representations without the computational cost of fine-tuning or the risk of catastrophic forgetting.

After compression, the retained segments are concatenated with minimal rewriting and fed to the language model for final answer generation:

We apply standard decoding with temperature $\tau=0.7$ and nucleus sampling with $p=0.9$ to generate diverse answer candidates for the best-of-5 evaluation protocol.