1.36B parameter model,
validated results.
AERO benchmarks, methodology, and research white papers.
Measurable edge across every training metric.
AERO requires at least 2× fewer training steps to reach the same model quality, on any hardware — directly measured, not a projection.
Combines 2× fewer steps with faster multi-node communication. In our testing, AERO achieved the same loss target in 27% the wall-clock time.
WHITE PAPER · 1.36B SWIGLU C4 PRETRAINING · JUNE 2026
AERO Training Loop — 2× Tokens, 3.7× Wall-Clock
Apples-to-apples vs well-tuned AdamW at 1×–8× Chinchilla, benchmarked against Stanford's published optimizer landscape
Read AERO Training Loop — 2× Tokens, 3.7× Wall-Clock (PDF)BACKGROUND
- –AERO is a drop-in training loop from DevOpt Labs — full forward/backward/step replacement, not just an optimizer swap.
- –Same code path on H100/A100, single- and multi-node fabrics.
- –Methodology follows Stanford's Fantastic Pretraining Optimizers (Wen et al., 2025).
METHODS
- –1.36B SwiGLU transformer; packed C4-EN, GPT-2 BPE; seq 2,048; bf16 mixed precision.
- –Cosine LR (peak 1.9e–3, 8k warmup); 1× C = 27B tokens, ~415k steps.
- –Test 1: 1×8×H100 (token efficiency). Test 2: 4×8×A100 (wall-clock, comms-bound).
- –AdamW tuned to Stanford-consistent slope α=0.32 — well-tuned, not under-tuned.
KEY FINDINGS
- –AERO@1×C = 2.55 = AdamW@2×C — 2.0× tokens-to-target, directly measured.
- –Cluster wall-clock: 47 h vs 175 h — 3.7× faster, 73% GPU-hours saved.
- –Per-step time on 32×A100: 408 ms vs 760 ms (Lever 2 throughput advantage).
- –130M proxy reproduces gap to two decimals across seeds.
IMPLICATIONS
- –~10× the best published optimizer gain (Muon/Soap/NAdamW ~1.0–1.1×).
- –Levers compound — wall-clock gain grows on slower fabrics.
- –Stable single-config across 415k steps; no spikes or restarts.
- –C4-EN at 1×C favors AdamW — 2× headline is a lower bound.
HEADLINEAERO halves training tokens to a given loss and cuts cluster wall-clock by ~3.7× — measured, seed-invariant, and a lower bound on production-typical gains.
Source: DevOpt Labs — AERO Performance White Paper, 1.36B SwiGLU C4 Pretraining, June 2026
RESEARCH BRIEF · PRETRAINING OPTIMIZERS · 2026-06-05
AERO at 1×C beats AdamW at 2×C
130M-parameter LLaMA-style pretraining on SlimPajama-6B and OpenWebMath, one RTX 5090 per run
Read AERO at 1×C beats AdamW at 2×C (PDF)BACKGROUND
- –Extends the 1×-Chinchilla study by doubling AdamW's compute (278k steps, ~4.55B tokens).
- –Decisive question: can AERO at 1×C still beat AdamW given 2×C?
- –All else held fixed — model, data, tokenizer, batch, schedule, seed.
METHODS
- –130M LLaMA-style decoder; SwiGLU, RoPE, RMSNorm; GPT-2 BPE, seq 2,048.
- –Batch 16,384 tok/step; cosine LR → 0 with 1k warmup.
- –Budgets: 1×C = 139k steps; 2×C = 278k steps.
KEY FINDINGS
- –SlimPajama: AERO 1×C 3.259 vs AdamW 2×C 3.385 — AERO wins by 0.126 nat at half the compute.
- –OpenWebMath: AERO 1×C 2.205 vs AdamW 2×C 2.264 — AERO wins by 0.059 nat at half the compute.
- –At equal 2×C compute, gap widens to 0.242 (SPJ) and 0.166 nat (OWM).
IMPLICATIONS
- –AERO delivers better loss for half the compute against a fully LR-tuned AdamW baseline.
- –Lead grows with data entropy — larger margin on the noisier corpus.
- –Unused stability headroom suggests reported margins are a lower bound.
HEADLINEAERO at 1×C beats AdamW at 2×C on both corpora — better loss for half the compute, lead widens at matched 2×C budget.
Source: AERO at 1×C vs AdamW at 2×C — SlimPajama-6B & OpenWebMath pretraining study, June 5 2026
DIAGNOSTIC STUDY · GRADIENT-CLIP INTERNALS · 2026-06-04
Untapped AERO Potential — Gradient-Norm Headroom
AERO sits 4–5× below AdamW's gradient norm and almost never clips — a saturated AdamW vs an unsaturated AERO
Read Untapped AERO Potential — Gradient-Norm Headroom (PDF)BACKGROUND
- –Diagnostic from SlimPajama, OpenWebMath, and C4 pretraining runs.
- –Same model, data, schedule, seed; only the optimizer differs.
- –AdamW clips at norm 1.0; AERO at 4.0 — pre-clip grad_norm_raw compared.
METHODS
- –TensorBoard scalars: grad_norm_raw, clip_coef, clipped.
- –Rolling-mean smoothing (window 800 at ebsz 8, 2000 at ebsz 32).
- –Runs span ebsz 8 (1-GPU) and ebsz 32 (8-GPU) across all three corpora.
- –% clipped = rolling fraction of steps with clip_coef < 1.
KEY FINDINGS
- –AdamW clips ~100% of steps by mid-training (86% even at ebsz 32).
- –AERO clips ≈0% — raw norm stays ~4–5× below AdamW at matched batch.
- –Both follow the 1/√B noise-floor law; AERO sits in a much lower band.
IMPLICATIONS
- –AdamW is saturated; AERO retains large unused stability margin.
- –Low end-of-run norm is a flat-minimum signature — matches AERO's lower val loss.
- –Reported AERO margins are likely a lower bound.
HEADLINEAdamW runs pinned at its clip bound; AERO never clips and sits 4–5× lower — large stability headroom left to spend for more performance.
Source: Untapped AERO potential — gradient-norm headroom diagnostic, June 4 2026
EARLY RESULTS · DIRECTLY MEASURED VISION BENCHMARKS · FIVE TASKS
AERO Across Five Vision Tasks
The training-loop advantage carries across five public vision benchmarks — medical and natural images alike — with only the optimizer changed in every comparison
Read AERO Across Five Vision Tasks (PDF)BACKGROUND
- –First proven on large-language-model pretraining; now measured directly on vision.
- –Five public benchmarks: medical segmentation, medical classification, and CIFAR.
- –Proxy-scale backbones — SAM ViT-B and ViT-Ti.
METHODS
- –Only the optimizer changes — data, model, seed, budget, schedule all identical.
- –Both sides tuned: AERO vs AdamW's own swept best.
- –Four seeds most tasks (three on NCT-CRC); public splits.
KEY FINDINGS
- –Wins 5 / 5 vision tasks vs tuned AdamW.
- –+1.0 to +3.3 pp higher accuracy / Dice at peak.
- –2–4× fewer steps to reach AdamW's peak.
- –CIFAR wins unanimous across all four seeds.
IMPLICATIONS
- –Signals a general training-loop effect, not a dataset artifact.
- –Margin grows on the harder task (CIFAR-10 → 100: +3.12 → +3.32 pp).
- –Still early — public, proxy-scale; clinical and RL not yet measured.
HEADLINEAERO wins all five vision benchmarks vs a tuned AdamW — +1.0 to +3.3 pp higher accuracy and 2–4× less compute to reach AdamW's peak, in every head-to-head so far.
Source: DevOpt Labs — AERO × Vision early results across VerSe'20, MURA, NCT-CRC-HE, and CIFAR-10/100. Datasets and backbones openly licensed; AERO configuration proprietary.