Large Language Model-Guided Evolutionary Discovery of Native Neural Architectures for Spiking Sequence Modeling

Spiking neurons combine binary outputs with continuous membrane states that retain temporal information. Architectural choices determine how these signals are transformed and propagated. In vision models, Spikformer uses spike-based attention without softmax (Zhou et al., 2023), while Spike-driven Transformer places residual connections before spiking activations to preserve binary communication (Yao et al., 2023). These designs show how adapting information pathways to spiking computation can support performance with less computation.

For sequence modeling, SpikeGPT combines recurrent processing with binary spiking activations for language generation (Zhu et al., 2024). The SpikingSSMs architecture applies leaky integrate-and-fire (LIF) dynamics to state-space outputs, combining temporal memory with sparse synaptic computation (Shen et al., 2025). Dyn-SSM incorporates refractory LIF neurons with soft reset, preserving residual membrane potential after firing (Zhong et al., 2026).

Automated methods also demonstrate the value of SNN-specific architectural exploration. SNASNet searches forward and temporal feedback connections, while MSE-NAS explores neuron operations and multiscale connectivity (Kim et al., 2022; Pan et al., 2025). AutoSNN considers accuracy and spike count and reports improvements over its handcrafted baselines (Na et al., 2022). For spiking language models, EQ-SpikeLM combines evolutionary channel pruning with subsequent post-training quantization (Zhang et al., 2026). Further couplings among spike activity, sequence-state updates, and residual pathways offer opportunities to extend this accumulated design knowledge.

Evolutionary neural architecture search (ENAS) uses population-based variation and selection to optimize neural architectures (Liu et al., 2023; Li et al., 2024). Most methods vary operators, connections, widths, and depths within predefined representations. For example, EvoCNN evolves variable-length sequences of convolutional, pooling, and fully connected layers (Sun et al., 2020), while NSGA-Net evolves CNN blocks through predefined operation and connection choices (Lu et al., 2021). These encodings support new combinations of existing building blocks, but a mechanism outside their permitted operations and composition rules requires redesigning the search representation.

Genetic programming and grammar-based NAS support variable computational structures assembled from reusable primitives. CGP-CNN evolves convolutional architectures from layer-level components, while einspace uses typed primitives to define a broader compositional space (Suganuma et al., 2017; Ericsson et al., 2024). AutoML-Zero further explores model computation and learning rules constructed from basic mathematical operations (Real et al., 2020). High-level primitives supply architectural priors through predefined computational forms; finer primitives expose more computational choices but can make useful designs harder to find. Even when composition obeys type and interface constraints, finding useful architectures can require many expensive evaluations. This motivates using architectural knowledge to guide the generation and modification of candidate architecture code.

LLM-guided evolutionary search has become a general approach to automated algorithm design (Wu et al., 2025; Ma et al., 2026), evolving heuristics, metaheuristics, and scientific programs (Romera-Paredes et al., 2024; Liu et al., 2024; van Stein & Bäck, 2025; Novikov et al., 2025); ShinkaEvolve, for example, combines code-embedding similarity screening with an LLM novelty judge to reject redundant candidate programs (Lange et al., 2026). For neural architectures, LLMs use pretrained knowledge and natural-language design goals, constraints, and principles to guide configuration search or generate architecture code. LLMENAS adapts fitness functions within a predefined cell search space (Lai et al., 2026), while Design Principle Transfer uses natural-language principles to narrow subsequent searches (Zhou et al., 2025). EvoPrompting evolves complete classifiers and, for graph networks, selected computations within a fixed processor (Chen et al., 2023). LLMatic combines network-code evolution with quality-diversity search for image classification (Nasir et al., 2024). ASI-Arch evolves attention-layer implementations under interface and computation constraints; it checks novelty through rationale retrieval and LLM judgments before training, and includes LLM-assessed architectural quality in its fitness (Liu et al., 2025); Genesys discovers language-model architectures with LLM agents on a genetic-programming backbone (Cheng et al., 2025). These code-based approaches establish architecture discovery with mechanisms for diversity and novelty. Our work combines population-relative novelty and surrogate-predicted performance to select SNN architecture programs for training; Section 2.3 details the representations supporting these decisions.

A search encoding specifies how architectures can be constructed; candidate descriptors summarize the resulting architectures for comparison and prediction. Implementation-level comparisons include the lexical, syntactic, and data-flow matches measured by CodeBLEU (Ren et al., 2020). Natural-language descriptions expose design intent, as in Evolution of Heuristics (EoH), which jointly evolves heuristic ideas and executable implementations (Liu et al., 2024). Execution-based descriptions capture observed behavior: phenotypic characterization records program responses to probe cases and supports surrogate modeling (Hildebrandt & Branke, 2015), while BehaveSim compares intermediate solution trajectories (Zhang & Lu, 2026). These views provide complementary, partial evidence: similar code need not implement the same computation, stated design intent need not match the implementation, and finite probes reveal only part of a program’s behavior.

For neural architectures, structural statistics and initialization-time forward and backward probes provide information before candidate training. Training-free proxies estimate trained performance from initialization-time activations or gradients without optimizing candidate weights (Abdelfattah et al., 2021). DCL-ENAS pretrains an architecture encoder without performance labels, then contrastively fine-tunes a predictor on evaluated architectures to predict their relative performance (Zhang et al., 2026b). In expressive grammar-based NAS, Transferrable Surrogates builds transferable predictors from training-free proxies and neural graph features, or from a fine-tuned language model, and uses them to filter candidates or serve directly as search objectives (Qin et al., 2025). Accurate performance prediction alone does not validate distances between architecture representations as measures of novelty.

Descriptors also determine which differences evolutionary search rewards. Novelty search rewards distance from previously observed behaviors, while MAP-Elites retains high-performing solutions within descriptor-defined regions (Lehman & Stanley, 2011; Mouret & Clune, 2015). In LLMatic, discussed in Section 2.2, the network archive uses width-to-depth ratio and floating-point operations (FLOPs) as descriptors (Nasir et al., 2024). BOP-Elites models both quality and descriptors with surrogates to select evaluations when these quantities are expensive to obtain (Kent et al., 2025). For SNNs, architectures with similar size and computational cost can nevertheless differ in how spike activity interacts with state updates and gating. We estimate novelty relative to a reference population by comparing code, design rationale, and a behavioral fingerprint combining architecture statistics with initialization-time probes. The same fingerprint supports performance prediction, and both estimates guide the selection of candidates for expensive training evaluations.

An SNN processes information through interconnected spiking neurons, whose internal states evolve over time and whose outputs are discrete binary events (Neftci et al., 2019). In a discrete-time description, a neuron’s binary output records whether it fires at each step, while its membrane potential remains continuous-valued. A common example is the leaky integrate-and-fire (LIF) neuron. In its hard-reset form, the pre-reset potential $\tilde{v}_{\tau}$, spike $s_{\tau}$, and membrane state $v_{\tau}$ at step $\tau$ satisfy

where $I_{\tau}$, $\lambda$, and $\theta$ denote input current, leakage factor, and firing threshold, and $\mathbb{I}[\cdot]$ denotes the indicator function. The neuron integrates the current with its decayed membrane state, emits a spike when the resulting potential reaches or exceeds the threshold, and resets its state to zero after firing.

Within the network, incoming spikes contribute to a neuron’s input current through weighted synaptic connections. The resulting spike response depends on both the incoming signals and the neuron’s preceding membrane state. Binary synaptic inputs also permit weighted sums to be evaluated by accumulating the weights associated with active spikes, providing opportunities for sparse computation (Davies et al., 2018; Horowitz, 2014).

Let $\mathcal{A}$ denote the search space of admissible SNN block architectures, each represented by executable code and feeding binary spikes to its parameter-dominant feature projections (the spiking-projection constraint; Appendix A.2). Architecture performance is task-dependent, so our main search combines WikiText-2 language modeling and ListOps hierarchical expression evaluation (Merity et al., 2017; Nangia & Bowman, 2018) to assess each design beyond a single task. Their weak rank agreement in our cross-task comparison supports their use as complementary evaluation signals (Appendix A.1). For $a\in\mathcal{A}$, let $w$ collect the separate trainable parameter vectors of the full models instantiated with $a$ for the evaluation tasks, and let $\mathcal{L}_{\mathrm{train}}(a,w)$ sum their training losses. We aggregate their evaluation scores into a single performance objective $y(a,w)$, with larger values indicating better performance. Let $\mathcal{R}$ be a finite set of reference architectures, held fixed for each ranking, and let $\mathcal{N}(a;\mathcal{R})$ denote the estimated architectural novelty of $a$ relative to $\mathcal{R}$. To encourage distinct architectural alternatives alongside performance, we model discovery as a multiobjective optimization problem following NAS formulations (Lu et al., 2021; Lu et al., 2024) and novelty-based multiobjectivization (Mouret & Doncieux, 2012), mathematically as follows:

In practice, fixed, finite training protocols approximate the lower-level optimization. Measuring a candidate’s fitness in the main search requires separate training on both datasets, increasing evaluation cost relative to training the same candidate on either task alone. Novelty can instead be estimated before weight training. To limit evaluation cost, each search run evaluates at most $B$ additional architectures under its prescribed task and training protocol, excluding those evaluated initially.

To guide the discovery of useful native SNN architectures, we need to estimate architectural differences among generated candidates before committing to weight training. Source-code differences alone need not reflect changes in architectural mechanisms. We therefore propose a three-view representation for SNN architectures (Fig. 3): code describes implemented operations and dataflow; the design rationale states the intended architectural idea, following EoH’s idea–code pairing (Liu et al., 2024); and a numerical behavioral fingerprint records structural attributes and initialization-time responses. Together, these views provide partial evidence for comparing candidates generated in the code space.

Inspired by response-based program characterization (Hildebrandt & Branke, 2015; Zhang & Lu, 2026), we probe the instantiated SNN before weight training. Probe inputs, sequence shape, initialization procedure, and seed are fixed across candidate comparisons. We adapt SWAP’s sample-wise pattern count (Peng et al., 2024b) to binary spike outputs, yielding sample-wise spiking patterns (SWSP): each recorded neuron–sequence-position response forms a pattern across probe samples, and SWSP counts the distinct patterns. Alongside SWSP, we use mean spike activity across spiking layers (FireRate) and five established activation- and gradient-based proxies (Abdelfattah et al., 2021; Mellor et al., 2021), giving seven initialization-time measurements. We supplement them with four module statistics describing components and ten structural statistics describing dimension flow, computational organization, and resource allocation. We construct the behavioral fingerprint $\mathbf{x}(a)$ as a 21-dimensional feature vector by concatenating these measurements. Feature definitions are given in Appendix C.1.

For architectures $a_{1}$ and $a_{2}$, we adopt CodeBLEU (Ren et al., 2020) and average its two comparison directions to obtain code similarity $s_{\mathrm{code}}(a_{1},a_{2})$. For rationale similarity $s_{\mathrm{rat}}(a_{1},a_{2})$, we encode each LLM-generated design rationale with the pretrained text-embedding model text-embedding-v4 (Alibaba Cloud, 2025) and compute cosine similarity between the resulting vectors. Fingerprint similarity $s_{\mathrm{fp}}(a_{1},a_{2})$ uses cosine similarity after fixed feature-wise affine scaling of $\mathbf{x}(a_{1})$ and $\mathbf{x}(a_{2})$ (Appendix C.2). We define pairwise dissimilarity using fixed, untuned equal weights:

Inspired by novelty search (Lehman & Stanley, 2011), we estimate novelty as the mean dissimilarity to all other reference architectures:

The surrogate in Section 3.4 uses evaluated architectures’ fingerprints and measured task outcomes to predict task performance from an untrained candidate’s $\mathbf{x}(a)$.

To limit costly weight training, we use an inner loop for surrogate-guided architecture evolution and an outer loop for training allocation and archive updates, i.e., surrogate model management (Jin, 2011; Zhang et al., 2010). Let $\mathcal{D}_{0}$ denote the initial archive of evaluated expert architectures, and $\mathcal{D}_{t}$ the archive after outer iteration $t$. Each archived architecture has an associated fingerprint and measured task outcomes. We adopt TabPFN-2.5 (Hollmann et al., 2025; Grinsztajn et al., 2025) as the surrogate, using these records from $\mathcal{D}_{t-1}$ as labeled context. Given $\mathbf{x}(a)$, the surrogate predicts task metrics without training candidate $a$. The same fitness mapping used for measured outcomes aggregates these predictions into $\hat{y}_{t}(a)$; hereafter, $y(a)$ denotes fitness measured after the prescribed training protocol.

We initialize the inner-loop populations from $\mathcal{D}_{t-1}$ using NSGA-II’s nondominated sorting and crowding-distance truncation (Deb et al., 2002), based on measured fitness $y(a)$ and novelty $\mathcal{N}(a;\mathcal{D}_{t-1})$. The selected architectures seed separate island populations, from which parents are sampled within or across islands. The LLM revises or recombines their code and design rationales. Feasible offspring pass fingerprint-based near-duplicate screening before surrogate evaluation. Predicted fitness guides subsequent parent selection and survival within islands, while accepted candidates accumulate in a pool for possible training.

Before selecting architectures for training, we shortlist candidates to avoid comparing every pair of generated architectures. We retain a subset prioritized by predicted performance and supplement it with candidates having high mean three-view dissimilarity to that subset. After screening against the evaluated archive for near-duplicates, we apply the same selection procedure using predicted fitness and novelty relative to the filtered pool, which remains fixed throughout selection. The resulting batch contains previously unevaluated architectures, limited by the maximum batch size $K$ and remaining budget. The selected architectures are trained under the prescribed task protocols and added to $\mathcal{D}_{t}$ with their measured outcomes, supplying additional surrogate context and parent candidates for the next iteration. Across the prescribed iterations, at most $B$ additional architectures are evaluated under these protocols. Search returns the evaluated archive and its nondominated subset under measured fitness and novelty relative to the final archive. Appendix C gives the complete procedures and budget constraints.

Candidate blocks share the sequence-model wrapper described in Appendix D.1. During the main search, each selected candidate is trained separately for 30 epochs on WikiText-2 (WT2) language modeling and 25 epochs on one-third of the ListOps training set (Merity et al., 2017; Tay et al., 2021), producing distinct model weights for the two tasks. We instantiate the composite fitness as

where the task transforms use SpikingDeltaNet as the $0.500$ fitness anchor (Appendix D.2).

The initial archive contains 20 expert SNN adaptations of linear recurrent and state-space architectures (Appendix C.3). Gemini-2.5-Flash (Comanici et al., 2025), selected after a comparison of candidate LLMs (Appendix C.5), is used as the LLM to generate candidates, with prompts in Appendix G, and the surrogate specified in Section 3.4 predicts their task metrics. The main search runs for 10 outer iterations with a batch limit of $K=16$ and a budget of $B=160$ architecture evaluations, each comprising both task-training runs; candidates that duplicate archived architectures are removed, so 151 new architectures are trained. Candidate weight training costs an estimated 132 V100 GPU-days (3,171 GPU-hours). Appendix C.4 gives the search configuration and resource accounting. The code, generated programs, and evaluated archive will be released.

To evaluate the selected architectures at full scale, we retrain them on WikiText-103 (WT103) (Merity et al., 2017) and the six Long Range Arena (LRA) tasks, whose sequences span 1K–16K tokens (Tay et al., 2021), following the task-specific configurations of SpikingSSMs (Shen et al., 2025); the WT103 configuration matches that of Dyn-SSM (Zhong et al., 2026). Unlike the search-time setting, ListOps is trained on the full training split and evaluated on the same test split, whereas WT103 and the other five LRA tasks are not used during search (Appendix D.1). The LRA average covers all six tasks. The ANN reference, DeltaNet, is among the strongest of the 20 experts and 97 ASI-Arch models trained under our common protocol. S4D-Lin and S6-based SNN results are quoted from Shen et al. (2025) and Dyn-SSM results from Zhong et al. (2026); all other models, including our spiking adaptations SpikingDeltaNet and SpikingMamba2, are trained under our protocol. The main search comprises one run; Sections 4.4 and 4.5 report repeated smaller-scale studies.

Figure 4 traces the main search. Candidates that surpass the strongest expert appear in every iteration from the second onward, and the best fitness improves in five separate iterations, reaching $0.872$ at iteration 9. Progress is therefore sustained rather than confined to an early proposal, consistent with the archive supplying better parents and surrogate context as it grows. The nondominated front also advances in the two task metrics, eventually containing candidates that surpass SpikingDeltaNet on both WT2 and ListOps (Appendix C.6).

Using search-time measurements only, we select three architectures that represent distinct regions of the final Pareto front in composite fitness and novelty (Fig. 10): NeuroGate, the extreme solution with the highest fitness; HomeoResSSM, the knee point, which lies farthest from the line joining the two extreme solutions after normalization (Das, 1999; Zhang et al., 2015); and LoopMem, the most novel member that retains nontrivial performance on both tasks.

The selected architectures remain strong after full-scale retraining (Table 1). On WT103, all three outperform the listed SNN baselines, and NeuroGate also surpasses the ANN DeltaNet trained under our protocol (26.4 versus 27.5 perplexity), although its parameter-dominant projections receive binary spikes. On LRA, NeuroGate attains the highest average accuracy among the SNNs, between the two ANN baselines, and remains the highest among the SNNs without ListOps (90.5 versus 89.0), while HomeoResSSM gives the best Retrieval accuracy among all listed models. The discovered architectures thus combine competitive accuracy with distinct task profiles.

These results come with substantially lower estimated energy. Counting spike-driven projections as accumulate operations and continuous computation as multiply–accumulate operations, with each model’s measured firing rate and a shared model wrapper, the discovered architectures reduce the arithmetic energy of a dense Transformer of the same size by $17$–$51\times$ (Appendix F, which also details the Dyn-SSM estimate). The spiking feed-forward network (FFN) and output head of the shared wrapper account for most spike-driven operations, so differences among the SNNs mainly reflect their firing rates and token-mixer arithmetic. NeuroGate fires about half as often as SpikingDeltaNet, so its additional control computation still yields a larger reduction, and LoopMem attains the largest reduction among all listed models. We next examine which evolved elements distinguish the discovered architectures and whether their effects depend on spiking computation.

The discovered architectures mainly contain two kinds of elements that we regard as native to SNNs. ① Spike-activity-dependent computation takes spike activity itself as an input. NeuroGate maps the mean input spike activity of its query, key, and value projections at each position to gains on the delta-rule update coefficient and the output gate (Fig. 1(c)), and HomeoResSSM uses the spike activity of its state-space and channel-mixing branches to scale their residual contributions (Fig. 5(a)); we test this class on HomeoResSSM. Because each neuron integrates its input across tokens and resets on firing, this activity reflects membrane dynamics that have no counterpart in ANN units, so the control reads a signal that exists only in spiking units. ② Spiking-specific design choices do not read spikes explicitly, but depart from ANN practice in either direction: components that are uncommon in ANN design are added when they benefit the SNN, and components that are standard in ANNs are removed when they degrade it. Direct ANN-to-SNN adaptation introduces neither kind of change. LoopMem illustrates the first direction, feeding a normalized summary of its recurrent state back to the forget gate (state-norm feedback) (Fig. 5(b)); NeuroGate illustrates the second, omitting the sigmoid linear unit (SiLU) activation that DeltaNet applies after its query, key, and value convolutions.

Table 2 examines these elements on WT2. All variants, including the discovered architectures, are trained with the fixed protocol applied to every search candidate, and we compare the effect of each element within an implementation rather than perplexities across implementations. For ①, removing HomeoResSSM’s control raises perplexity by 6.9. Driving the same controller with the channel mean of the continuous inputs, from which the spikes are generated, recovers most of this benefit, yet the spike-driven control still attains 1.7 lower perplexity. For ②, restoring SiLU raises NeuroGate’s perplexity by 5.9. Removing LoopMem’s state-norm feedback raises perplexity by 8.8 in the SNN, whereas the same removal lowers the perplexity of the non-spiking variant by 1.6. LoopMem’s feedback thus has opposite effects in the two implementations, and the ANN-standard SiLU degrades the SNN.

From the search-space perspective, each control element combines several decisions: which signal to summarize, how to transform it, and which update, branch, or gate it modulates. In OpenArchEvo, the LLM can propose these decisions within one code revision of a parent program. Among seven published encodings, covering SNN cell and block spaces, recurrent-cell spaces, and a recursive architecture grammar, none can select a spike-statistic or state-summary input, a learned controller, and its target together; directly representing these elements requires extending the encoding (Appendix B, Table 5). Open-code search can express such relations, but the best architecture of our adapted ASI-Arch run, SpikingCondFuse, conditions its gate on hidden-state statistics rather than spike activity (Appendix D.4). Appendix E gives the equations of the evolved elements.

Inner-loop comparisons. We compare our inner-loop search with adaptations of EoH (Liu et al., 2024) and FunSearch (Romera-Paredes et al., 2024), which replace the inner loop while sharing our outer evaluation and surrogate-selection loop. Their original designs evaluate every generated program, which is impractical when each evaluation requires training an architecture. The EoH adaptation uses thought–code evolution in a single population, whereas the FunSearch adaptation uses code-based best-shot prompting with islands. Our inner loop features population management and redundancy control guided by the three-view representation and novelty. In the main search, the trained expert archive supplies the initial surrogate context and parents. Each WT2-only search instead begins with a warm-start stage that trains a first batch of diversified candidates. Each configuration then runs six outer iterations of WT2-only search, scored by a WT2-only fitness of the same form (Appendices D.2 and D.4). OpenArchEvo attains the highest final best fitness and hypervolume (Fig. 6). All configurations use the same total training budget.

End-to-end program-search comparison. We retain the agent workflow of ASI-Arch (Liu et al., 2025), adapt its evaluation tasks, and impose the spiking constraints. Table 3 reports the resulting search outcomes and representative architectures. The ASI-Arch and OpenArchEvo runs train 39 and 151 architectures over six and four weeks, respectively. OpenArchEvo uses one program generator; the API expenditure excludes candidate-training compute. The best and Top-10 mean fitness are higher in our completed run. Appendix D.4 provides the adaptation details.

Near-duplicate screening. Fingerprint-based near-duplicate screening removes 30% of raw proposals (Fig. 7). The shared initialization-time probes and architecture statistics identify these candidates before surrogate-guided selection.

Surrogate prediction. The surrogate studies were conducted before the main search and guided the choice of its surrogate and fingerprint (Appendix D.3). Initialization-time probes capture network responses and form the starting point for our surrogate-input study (Fig. 8(b)). Adding SWSP and firing rate to the classical proxies improves ranking on WT2, with a smaller change on ListOps. Supplementing these probes with module and structural statistics gives the strongest ranking along the tested sequence. With the complete fingerprint, TabPFN-2.5 ranks best among the tested regressors on both tasks (Fig. 8(a)). In these studies, prediction generally improves as the labeled context grows (Fig. 8(c)). Early in search, candidates are selected for training by a surrogate with little labeled context. In WT2-only searches without the warm-start stage, which start from the expert archive alone, the best fitness after six iterations is 0.56 instead of 0.68, and the gap persists across iterations (Appendix D.4).

Search ablations. We compare candidate-generation and selection variants in six-iteration WT2-only searches, each repeated three times. Besides the full configuration, which evolves candidates on multiple islands and selects them with the surrogate, the variants evolve a single island, sample candidates from the LLM without evolution before surrogate selection, or sample and select without the surrogate. The full configuration has the lowest Top-1 and Top-5 PPL and the highest novelty, while the single-island variant has the lowest fitness variance (Table 4).