Calibrating mechanistic models of complex systems
The hardest scientific models are executable programs with known structure and dozens to hundreds of continuous constants: rate coefficients, couplings, and response terms that must be fit to sparse, noisy observations. Gradient-free calibration scales poorly as that parameter count grows. Exact gradients through the model, kept as a program, let an outer search propose structure while the whole parameter vector is calibrated at once.
Climate and Earth systems
Ocean, atmosphere, and land-surface models encode fluid dynamics and biogeochemistry, yet carry large vectors of uncertain constants (eddy diffusivities, carbon-pool and forcing terms). Co-search calibrates them against sparse observations where gradient-free methods stall, as in this program's El Nino inverse-model study.
advection-diffusion field · fitting eddy diffusivity and forcing
Epidemic and ecological spread
Spatial models of contagion track susceptible, infected, and recovered populations moving under region-specific transmission, seasonality, and mobility. The outer loop proposes the structural form; exact gradients fit the continuous rates to case data, with honest scoring on held-out signal.
agent-based SIR spread · fitting transmission and recovery rates
Digital twins of coupled systems
A digital twin mirrors a real system, a power grid, a reactor, a sensor network, as an executable model whose coupling and response constants are fit to live telemetry and kept in sync. Many parameters, stiff dynamics, and structure that changes as the system is re-specified: exactly the co-search regime.
coupled oscillators · fitting coupling strength to telemetry
In every case the model is a program with many continuous constants. Co-search proposes the structure; exact gradients, delivered fast enough to run inside the search loop, calibrate the parameters. The three papers below build that capability from the ground up.
Differentiate the program, then the evaluator, then make the evaluator fast
Three papers form one arc. First compile individual Scheme programs into exact differentiable graphs. Then compile the interpreter once, so programs become runtime data and gradients flow through interpretation. Then rebuild that interpreter as a native runtime, so calibrating a whole population of candidate programs is cheap.
Neural Compiler
Added: a compiler from a first-order Scheme subset (a few dozen operations, including vector and matrix primitives) to frozen, exact, differentiable PyTorch modules. The program contributes zero approximation error; only its constants are learned.
Removed: the need to re-implement a known scientific model inside an ML framework before gradients can reach its unknowns.
Result: on ODE benchmarks, a two-constant compiled model beats MLPs thousands of times its size, with error hundreds of times lower in distribution and thousands of times lower under extrapolation.
Limitation kept: first-order programs only, and one compiled graph per program.
Read about the Neural Compiler →DMCI
Added: compile a self-hosting Scheme interpreter once into a differentiable graph. Every program it runs is data, and exact gradients flow through interpretation to the constants inside the program. The model is not differentiable; the interpreter is.
Removed: per-program compilation and the first-order restriction; closures and recursion run as interpreted data.
Result: interpreted gradients match direct compilation to numerical precision on every tested program-seed pair, across recursive and higher-order programs.
Limitation kept: the eager PyTorch backend is representation-bound; value boxing and evaluator walking consume nearly all of the forward pass, and raw arithmetic only about a hundredth of it.
Read about DMCI →NDVM
Added: a native C++ CPU runtime that splits discrete structure (scalar tags, addresses, interned symbols, closures, environments) from differentiable numbers (dense payload buffers carrying the batch axis), keeps control flow as the exact realized trace with no branch smoothing, and records a compact payload-only reverse tape.
Removed: the representation overhead, so calibrating each candidate in a co-search stops being the bottleneck.
Result: single calibrations run thousands of times faster at bit-identical fit quality, and amortizing one interpreter walk over a population cuts per-candidate cost by another order of magnitude.
Scope: CPU-native by design; a GPU port is a forward-only proof of concept in an appendix.
Read about NDVM →Program-and-parameter co-search
A non-differentiable outer loop searches the discrete space of program structures; an exact-gradient inner loop searches each candidate's continuous parameters. Because every runtime-supplied program inherits differentiability from one frozen evaluator, the two close into a single joint discovery loop.
Why gradient-free search is not enough
A candidate program can only be judged after its continuous constants are fit, and gradient-free optimizers fit them poorly as parameter counts grow. In the DMCI LIM-ENSO case study, gradient-free differential evolution failed to recover a stable operator on the same compiled Kalman-filter likelihood that exact-gradient optimizers solved cleanly. A structure can look bad simply because its constants were never found.
Exact gradients through programs-as-data close the gap. One evaluator is differentiated once; every proposed structure inherits exact reverse-mode gradients with no per-candidate reimplementation, so each candidate is scored at its calibrated best.
Where the outer loop comes from
The outer loop is exactly the machinery of LLM-guided evolutionary search: AlphaEvolve (Novikov et al., 2025, DeepMind), OpenEvolve (Sharma, 2025, an open-source implementation of the AlphaEvolve approach), and earlier FunSearch (Romera-Paredes et al., Nature 2024). These systems already select programs on a fitness score; plugging in an exact-gradient calibration step turns that score from "how good is this structure with guessed constants" into "how good is this structure at its best".
NDVM makes the inner loop cheap
One Kalman MLE calibration drops from 211,277.2 ms on the eager DMCI backend to 25.94 ms with a bit-identical NLL: 8144.1× per calibration, or 17 versus 138,769 calibrations per CPU-hour. In a fixed 900-second budget over LLM-proposed programs (offline-cached proposal streams, so the model is out of the timed loop), the discovery frontier shifts 23.8× on a scalar task and 339.8× on a recurrence-heavy task, where the PyTorch/DMCI backend never reaches a successful fit.
Honest bound: with a live LLM at 0.5 s per candidate the end-to-end gain is 405× (108× at 2.0 s); the loop becomes LLM-bound and proposal is the next floor.
The NDVM co-search results →Co-search recovers real models
In the DMCI paper's case studies, co-search starting from a smooth square-root seed rediscovered a knee-capable battery capacity-fade law and improved held-out extrapolation on real lithium-ion cells; and exact gradients through an interpreted Kalman-filter likelihood (LIM-ENSO) recovered stable operators where gradient-free search failed.
Caution, also from the DMCI paper: the FluZoo influenza study is an honest negative result on fitness fidelity. An under-converged inner fit flattered a more flexible program, and the advantage vanished under rigorous re-scoring. Co-search is only as trustworthy as its inner calibration.
The DMCI case studies →Three papers, one program
Each paper has a companion page on this site with interactive figures drawn from its measured results.
The Neural Compiler
Compiles a first-order Scheme subset into frozen, exact, differentiable PyTorch modules, so a known model's unknown constants are fit by gradient descent through its own structure.
Compile Once, Differentiate Everywhere
Compiles a self-hosting Scheme interpreter once into a differentiable graph; any program it runs is data, and exact gradients flow through interpretation to the constants inside the program.