Improve HomotopySweep adaptive step control: success expansion, quality-gated growth, trust-monitored secant predictor

[ChrisRackauckas-Claude]

Note

This PR should be ignored until reviewed by @ChrisRackauckas.

Follow-up to #962 (cf. this comment): improves the adaptive algorithm in HomotopySweep.

What was there before

The adaptive logic was failure-only: a rejected step halved dλ and the increment never grew back, so one hard region permanently degraded the resolution of the rest of the sweep. Warm starts were zeroth-order (reuse previous u).

What this adds

Three mechanisms from the predictor-corrector path-tracking literature, sized for a natural-parameter sweep whose corrector is a generic NonlinearSolve algorithm (no polynomial structure, no tangent/Jacobian access from the sweep itself):

1. Success expansion (the classic success/failure heuristic, e.g. Timme, Mixed precision path tracking for polynomial homotopy continuation, Adv. Comput. Math. 47 (2021)): after expand_threshold consecutive accepted steps, dλ grows by expand_factor (default ×2), capped at max_step_factor of the span width.

2. Quality-gated growth (a Deuflhard-style local error estimate): the known failure mode of the pure success/failure heuristic is expensive trial-and-error in hard regions (cf. Hao & Zheng 2020) — the step balloons on the easy stretch before a sharp turn, and each oversized rejection costs a full inner-solver iteration budget (measured: one rejection through the default polyalgorithm ≈ 600 residual evaluations). Expansion therefore additionally requires evidence of corrector headroom: either the secant prediction error is small relative to the recent solution movement (expand_quality), or the corrector converged within 2 iterations (which covers exponentially flattening path tails, where the relative error measure stays at a constant mediocre value while the absolute corrections become negligible).

3. Trust-monitored secant predictor: warm starts extrapolate linearly through the last two accepted points, so the corrector starts on the path tangent. Right after a sharp turn a stale tangent predicts far off the path, so the secant's measured prediction error is compared against the trivial constant prediction each accepted step, with hysteresis: two consecutive good measurements to (re-)engage, any bad measurement or rejected step to disengage and fall back to the constant warm start. predictor = :constant restores the old warm-start behavior.

All defaults preserve the old semantics where it matters: expand_factor = 1 disables growth entirely, expand_quality = Inf disables the gate, adaptive = false fixed-step mode is untouched.

Measured behavior (local runs, Julia 1.11)

Sharp-turn path u*(λ) = 2tanh(20(λ-1/2)) with corrector residual x + 2sin(x) (spurious merit-stationary traps for warm-start error > ~2.09), default polyalgorithm, maxiters = 100:

controller	residual evals	rejected steps
naive growth (no gate, no trust)	1406	2
+ quality gate	808	1
+ trust monitoring + hysteresis (this PR's default)	98	0
old behavior (no growth), secant	142	0

Smooth cubic path u*(λ) = 1 + λ: secant predictor 36 evals vs 64 constant; easy sweeps finish in 5 steps instead of 11 (growth doubling to the cap).

Tests

Four new test items (registered in test/runtests.jl):

• item15: constructor defaults + validation for the new kwargs
• item16: expansion takes strictly fewer steps; max_step_factor cap binds, including over an nsteps-derived initial step
• item17: secant predictor spends strictly fewer residual evaluations than :constant on a linear path
• item18: sharp-turn problem — sweep bisects through the turn, the increment regrows on the far shoulder (≥2× within the post-rejection accepted steps), and takes strictly fewer steps than expand_factor = 1

Local test results: all 18 homotopy sweep items pass (86 assertions), NonlinearSolveBase QA (Aqua + ExplicitImports) passes 14/14, Runic check clean. One pre-existing test (item7, aliasing) caught a real bug in an intermediate version of this PR — the θ measurement must not reuse the guess buffer the inner solver may have mutated in place — which is fixed and the test passes.

NonlinearSolveBase bumped 2.31.0 → 2.32.0.

Possible follow-ups (out of scope here)

• Pseudo-arclength reparameterization for folds (the item6 fold test currently — correctly — reports failure; arclength continuation could follow the path around the turning point)
• Hao & Zheng (2020)-style augmented systems tracking the minimum Jacobian eigenvalue near bifurcations
• Higher-order (Padé / polynomial) predictors once more accepted points are retained

🤖 Generated with Claude Code

[ChrisRackauckas-Claude]

Two follow-up commits since the initial push:

• Spell out journal abbreviation flagged by typos CI — the Spell Check workflow flagged "Comput." in the docstring's journal citation; spelled out the journal name.
• Fix unresolvable @ref to SciMLBase.HomotopyProblem in HomotopySweep docstring — fixes Docs build: new unresolvable @ref to SciMLBase.HomotopyProblem from HomotopySweep docstring (#962) #968 (the one genuinely new master CI failure introduced by Add HomotopySweep: natural-parameter continuation solver for HomotopyProblem #962): the docs build cannot resolve @ref to a SciMLBase docstring that is not included on any docs page, so it is now a plain code span. Included here because this PR already edits that docstring.

For context when reading this PR's CI: master is currently red for unrelated, pre-existing reasons — Runic format-check failures on six sublibrary test/runtests.jl files from #961 (#969), pre-existing :example_block docs errors (large_systems.md, snes_ex2.md), a stale DiffEqBase = "6.213.0" compat in test/trim/Project.toml, a long-standing Downgrade Sublibraries FastClosures lower-bound conflict, and self-hosted-runner infra flakes (gitconfig lock races, lost runners). The signal jobs for this PR are the Core test group (homotopy sweep items 1–18) and the NonlinearSolveBase sublibrary CI.

[ChrisRackauckas-Claude]

Pushed Develop in-repo sources for base-env test groups on Julia < 1.11, fixing the only PR-attributable CI failure (tests / Core (julia lts) on both platforms).

Root cause: the root test env's [sources] table is ignored on Julia < 1.11, and unlike the sublibrary and dep-adding group paths, the base-env group path in test/runtests.jl never developed the in-repo paths. So the lts Core jobs resolved NonlinearSolveBase 2.31.0 from the registry and tested that instead of the PR branch — failing on this PR's new kwargs with a kwerr, and silently testing stale sublibrary code on lts for any PR before it. This is also the same mechanism behind the registry-lag Core (julia lts, macos) failure on master at 633d93a (the #962 merge ran CI before 2.31.0 had propagated to the pkg servers).

Verification: ran the full root GROUP=Core suite via Pkg.test() on a fresh Julia 1.10.11 resolve locally — all 49 testsets pass, including the previously-failing HomotopySweep step-control kwargs validation + defaults (18/18), exit 0.

Status of the remaining red checks on the previous run, none PR-caused:

• tests / Core julia 1 + pre, all sublibraries / NonlinearSolveBase (Core+QA, 1/lts/pre), Spell Check, Runic Suggestions: pass
• Runic Format Check: pre-existing from test: uniformize monorepo onto OrdinaryDiffEq canonical structure #961 (format-check failing on master since #961: sublibrary test/runtests.jl not Runic-formatted #969)
• Documentation: now fails only with the pre-existing :example_block errors — the :cross_references error from Add HomotopySweep: natural-parameter continuation solver for HomotopyProblem #962 is gone with this PR's docstring fix (Docs build: new unresolvable @ref to SciMLBase.HomotopyProblem from HomotopySweep docstring (#962) #968)
• Trim, downgrade-sublibraries: pre-existing compat staleness (DiffEqBase pin / FastClosures lower bound)
• BoundaryValueDiffEq downstream: red on BVDE's own master; OrdinaryDiffEq Interface: lost self-hosted runner

[ChrisRackauckas-Claude]

Final CI status — all PR-relevant jobs green

Green (caused by / exercising this PR's code):

• tests / Core — all 6 variants (julia 1 / lts / pre × ubuntu / macos), including the lts jobs that the _develop_inrepo_sources commit fixed. The homotopy sweep test items 1–18 run here.
• sublibraries / lib/NonlinearSolveBase — Core (julia 1 / lts / pre) and QA (julia 1 / lts), all pass.
• tests / Adjoint, Bounds, NoPre, QA, Verbosity, Wrappers, Downstream — all variants pass.
• Downgrade / Downgrade Tests - Core, Spell Check, runic-suggestions, all other sublibrary jobs — pass.
• OrdinaryDiffEq.jl Interface + Regression downstream — pass.

Red, none caused by this PR:

• Documentation — now only the pre-existing :example_block errors (large_systems.md, snes_ex2.md); the :cross_references error Add HomotopySweep: natural-parameter continuation solver for HomotopyProblem #962 introduced is fixed here (Docs build: new unresolvable @ref to SciMLBase.HomotopyProblem from HomotopySweep docstring (#962) #968).
• Runic Format Check — pre-existing from test: uniformize monorepo onto OrdinaryDiffEq canonical structure #961 (format-check failing on master since #961: sublibrary test/runtests.jl not Runic-formatted #969).
• Trim (DiffEqBase 6/7 compat), downgrade-sublibraries (FastClosures lower-bound) — pre-existing compat staleness, red on master.
• tests / CUDA, SimpleNonlinearSolve [CUDA] — tests pass; job fails in the Codecov step on a ~/.gitconfig lock race on the shared self-hosted runner. Infra.
• NonlinearSolveHomotopyContinuation sublibrary — pre-existing InteractiveUtils miss; fix in Add missing InteractiveUtils test dep to NonlinearSolveHomotopyContinuation #970.
• ModelingToolkit.jl Initialization / InterfaceI / InterfaceII downstream — all three fail with the same pre-existing UndefKeywordError: keyword argument P not assigned from MTK's BifurcationKit extension, unrelated to NonlinearSolve (NonlinearSolveBase 2.32.0 from this branch installs and precompiles cleanly in all three).

Net: every job that compiles or tests this PR's changes is green. Remaining reds are pre-existing (#968 fixed in-PR, #969/#970/#971 filed) or self-hosted-runner infra.