Oracle and gate
The oracle
The reference is gfortran 15.2.0 run natively (GCC releases/gcc-15.2.0):
it is the compiler LFortran’s and llvm-test-suite’s reference outputs are made
with. It is GPL and is only ever run as a program, never linked or ported (see
NOTICE). “Correct” means the same exit status and the same stdout and stderr
bytes, except where the output is processor-dependent (bib section 8).
go install github.com/go-embedded-fortran/fortran/cmd/foracle@latest
export FORACLE_HOME=$HOME/foracle-home
foracle toolchain # installs gfortran 15.2.0 into $FORACLE_HOME, no root
foracle version
foracle run prog.f90 # compile with the oracle, run, relay stdout/stderr/statusgfortran is obtained as the conda-forge package gfortran==15.2.0, installed by
a pinned micromamba (release 2.10.0-0, sha256 checked) into
$FORACLE_HOME/env. foracle checks gfortran --version and refuses anything
but 15.2.0. macOS arm64 and Linux amd64 are supported. flang 23.1.4 is optional
and Linux only; that path is written but not exercised by CI yet.
Programs are compiled with -O0 -w -fno-backtrace. Runs are bounded: a
wall-clock timeout (10 s), a cap on stdout and stderr (1 MiB), a fixed
environment and an address-space limit (1 GiB). The memory limit is enforced on
Linux; macOS accepts RLIMIT_AS but does not enforce it.
| command | what it does |
|---|---|
foracle golden | regenerates testdata/programs.golden from the 14 programs in testdata/programs/ |
foracle corpus --dir D | compiles and runs every Fortran file below D; one line per file |
foracle bench | the reference column of the gate: the four kernels with gfortran -O2, 7 rounds, median and spread; refuses above 1-minute load 8 |
foracle gate [--smoke] [--wait-load D] | the whole gate: the contenders on the kernels, checksum bounds, the decision rule; refuses above 1-minute load 8, or with --wait-load waits and reruns a round the load spoiled |
foracle parse-rate --dir D | parses every Fortran file below D and compares with gfortran -fsyntax-only |
foracle parse-bench --dir D | parse throughput against gfortran -fsyntax-only; refused above load 8 |
foracle toolchain --home H [--flang] | installs the pinned toolchain |
foracle version | prints the pinned versions and whether the toolchain was found |
The 14 golden programs (testdata/programs/*.f90) are original: list-directed
and formatted output, integer and real kinds, arrays and sections, column-major
order, modules, derived types, recursion, strings and internal I/O, stop,
error stop and stderr. CI regenerates the golden file on linux/amd64 and
darwin/arm64 and requires no difference.
No corpus is committed (the gcc testsuite is GPL; FCVS has its own terms). CI
fetches them by commit: gcc gfortran.dg and gfortran.fortran-torture (gcc
15.2.0, 5115c7e447fc), llvm-test-suite FCVS (19b349c5aa39) and LFortran
integration_tests (4f2cd049c87f).
The gate (bib section 4.2)
foracle gate decides between (A) a pure-Go evaluator and (B) compiling to
wasm32. The decision rule is fixed in bib section 4.2 and implemented once, in
internal/gate/rule.go, with its thresholds (5, 2, 1.2) pinned by a test.
| contender | what it is |
|---|---|
| A | package eval (prototype, the seed of M3-A): the typed tree compiled once into closures over typed slots; real(8) arrays of rank 1 and 2, column-major, bounds-checked; whole-array expressions and matmul handed to go-ndarray v0.40.3 |
| B-near, B-upper | the C transliterations in testdata/gate/c/ compiled by clang 23.1.4 to wasm32-wasip1 at -O0 and -O2, run on wazero v1.12.0 (the version go-embedded-c pins) |
| gfortran | 15.2.0 -O2, native: the reference |
| direct | go-ndarray called straight from Go, no evaluator, no copy: the baseline of A’s array kernels |
Every contender times its kernel with a clock around the kernel itself, one core
(GOMAXPROCS=1), in interleaved rounds (7 rounds, median and spread). A
checksum out of its bound is a failure, not a time. The bound is computed with
math/big and a contender is within 1.01 * m * 2^-53 * exact, m being the
number of rounding steps on the longest chain of the kernel (nonnegative data,
so the bound holds for any summation order, FMA or vectorisation).
Only the desktop home is implemented: the playground (headless Chromium) needs a browser lane that does not exist yet (#5). The rule needs both homes to pick A, so with the desktop alone it can only answer B or “A first”.
Figures
2026-11-15, v0.21.0, foracle gate --wait-load 14h, built from a clean
checkout of the tag. Desktop darwin/arm64, one core, 7 interleaved rounds;
every kept round is bounded by 1-minute load samples between 7.1 and 8.0; 3
rounds were discarded for load after 3 h 12 min of waiting. Seconds, median and
spread (max-min):
| kernel | A | B-near | B-upper | gfortran -O2 | direct go-ndarray |
|---|---|---|---|---|---|
| K1 | 0.1535 +0.0059 | 1.4778 +0.0283 | 0.7068 +0.0254 | 0.0605 +0.0019 | - |
| K2 | 0.1850 +0.0059 | 3.2997 +0.0509 | 1.7110 +0.0637 | 0.0372 +0.0099 | - |
| K3L | 0.1157 +0.0033 | 2.1937 +0.2551 | 1.0939 +0.0576 | 0.0193 +0.0010 | - |
| K3M | 0.0070 +0.0032 | 0.0084 +0.0027 | 0.0084 +0.0042 | 0.0058 +0.0017 | 0.0079 +0.0030 |
| K4 | 0.0750 +0.0287 | 0.3017 +0.0534 | 0.2798 +0.0481 | 0.0367 +0.0119 | 0.0907 +0.0166 |
Time over gfortran on the loop kernels: K1 A 2.54, B-near 24.42, B-upper 11.68; K2 4.97, 88.71, 46.00; K3L 5.99, 113.62, 56.66. Inputs to the rule: g(rA) = 4.23, g(rB0) = 62.67, g(rA)/g(rB0) = 0.067; K4 A / direct = 0.827, K3-matmul A / direct = 0.880.
Verdict: A first (between; re-gate after go-embedded-c M3). The desktop ratio is at most 2 and both array kernels are within 1.2 of direct go-ndarray, but the playground home is not measured (#5), so the rule cannot pick A outright.
The first run (2026-11-14, v0.10.0) had A at 8.33, 93.31 and 108.06 times gfortran on K1, K2 and K3L. The loop work of v0.18.0 to v0.21.0 puts A 4 to 10 times ahead of B-upper on every loop kernel. The gap to gfortran (2.5x to 6x) remains an open defect: #17.
Created on by tannevaled
Modified on by tannevaled