VOYNIX
A stone arch at the Yonaguni underwater site, light filtering down from above
a stone arch under the sea off Yonaguni, Japan. from Wikimedia
ai-scientific-discovery

The Atlantis ProblemCan AI start a Scientific Revolution?

Imagine a civilization sank to the bottom of the sea. All the AI scientists are working inside the sea while they never noticed the laboratories are running in the water. Could they find out the fact from optics experiments alone? We ask AI agents based on 9 language models to do experiments on a optical bench, and test if they can explain all the anomalies.

authorJiajun Shen, EPFL
date2026-10-10
e-mailjiajun.shen@epfl.ch

TL;DR

  1. Among 9 models tested, only two of them (GPT-6.1 Sol and Claude Sonnet 5.5) proposed the existence of the background medium and corrected all four materials back to their intrinsic values. Six models stopped at identifying anomalies and explain them as a source notch or a miscalibrated ruler. MiniMax-M3 never managed to switch on the light source due to failure in the tool interface.
  2. All models that conducted the experiments reported acceptable "relative" properties of the materials and noticed more or less the anomalies. They present persuasive capability in manipulating the device and regular scientific research.
  3. The divide happens at the very next step after seeing nA<1. Even the two successful model didn't follow the expected route of scientific revolution. They read the unusual reflective index as "A is less dense than whatever surrounds it” and measured the surroundings immediately, whiule other models stick to the "air", or "near vacuum" prior, attributing this anomaly to the property of the material.
  4. Afterwards, almost every model that failed, except DeepSeek-V4-Pro, immediately pointed out that the evidence had been in hand already, indicating the "atlantis" model is self-consistent and could be hypothesized based on current data and knowledge.
  5. None of the models tested present good habits as a researcher. They didn't reset the devices and only GPT-6.1 Sol turn off the light after full experiments.

1The New Turing Test

At the AI Impact Summit in New Delhi, February 2026, Demis Hassabis proposed a test: train an AI system only on knowledge available up to 1911, and see if it can independently derive general relativity, just as what Albert Einstein did in 1915. He said it would be AGI if it can[1].

The idea has since been called the “Einstein test”, on which Nature ran a feature[2], and other researchers have turned it into a more formal evaluation framework or asked which skill AI is actually missing[3, 4]. This test asks the model to think of something nobody has thought of yet, highlighting the agent's ability to make scientific breakthrough, instead of simply pushing science forward within the existing framework. 70 years ago Turing used conversation to tell machines from people[5], while language models nowadays still struggles to fully imitate humans, whether models can pass this test is less cared about, as more hold the belief that super intelligence is not necessarily human like. We focus more and more on the utilitarianism side: whether this models can really change the world, hence the massive investment in coding agent, real-world interface, AI for sicence and also, corresponding benchmarks or tests like the Einstein test.

That said, this test is very hard to actually run. The main obstacle is models' prior knowledge. A modern language models already have read almost everything ever written about modern physics (and hopefully, in 100+ languages :/), so it is impossible to erase this knowledge from their wights and require them to "rediscover" the theory again.

A natural strategy is to train “vintage” models from scratch on old text only. These works include: talkie, a 13-billion-parameter model trained only on English text from before 1931 (you can chat with it online)[6]; the Ranke-4B family family, with cutoffs in various years from 1913 to 1946[7]; TimeCapsuleLLM, trained on London texts from 1800–1875[8]; and GPT-1900, built specifically for the Einstein test (called the Machina Mirabilis project in Nature’s report). Reportedly GPT-1900 once produced a passage resembling the 1905 light-quantum idea, but overall it was far from an “AI Einstein”[2].

Vintage models allows to investigate what does a model that has only read old books do, but unfortunately they cannot answer the question we actually care about: do today’s strongest models have what it takes to discover new physics? A 13B vintage model (even GPT-3 released in 2020 has 175B parameters!) differs from a frontier model in scale, training method and post-training, so its success or failure says little about frontier models. Corpus leakage is also hard to rule out: shortly after talkie was released, people reported signs of post-1930 data having slipped in[6]. And after all, even we have a perfect vintage model GPT-1900 that passed the Einstein test (in all sense), how can we build another one with all modern knowledge?

2The Structure of Scientific Revolutions

While we cannot directly use the Einstein test, we can construct other ones that are isomorohic to it. Before that, let's review how the great scientific discoveries are made.

The big turns in science tend to happen in similar ways, which can be concluded into one paradigm. First, one or more anomalies show up in observations that does not fit the accepted theory. The anomalies them push the scientists to explain them. For example, no one asked to “overturn geocentrism”, but the observatory data forced someone to explain it. Then someone proposes a new hypothesis that is simpler than infinitely patching the old theory again and again. Finally, the new hypothesis predicts something the old one could not, and the prediction is confirmed, with the born of a new scientific paradigm attracting scientists to research on. Kuhn, in The Structure of Scientific Revolutions, described the first half as anomalies accumulating into a crisis[9].

1543 – 1609
The end of geocentrism

The system of Ptolemy (Κλαύδιος Πτολεμαῖος) fitted the planets’ retrograde loops with epicycles on epicycles, adding a layer for each new discrepancy (A beautiful application id Fourier transform, if I have to say something good to this). Copernicus put the Sun 🌞 at the center[10], and Kepler replaced every epicycle with an ellipse[11]. At first Copernicus was no more accurate than Ptolemy, and Kepler’s ellipses were the first to be both more accurate and simpler.

1846
Uranus misbehaves

Uranus did not follow its Newtonian orbit. Le Verrier did not doubt gravity, but assumed an unknown planet further out and computed where it had to be. That same year Neptune was found about a degree from the prediction[12]. This example is a successful “local explanation”, showing that we don't have to launch a scientific tornado all the time, if the anomalies are explaiable by adding some little pathes.

1859 – 1915
Mercury misbehaves

The same Le Verrier then found Mercury’s perihelion advancing about 38 arcseconds per century too fast[13]. He used the same trick and posited an inner planet, Vulcan, which was never found. The answer didn't come until the general relativity: in 1915 Einstein obtained the missing 43 arcseconds without adding a single new parameter[14].

1887 – 1905
The ether wind

Michelson and Morley failed to detect the Earth’s motion through the ether[15]. FitzGerald and Lorentz proposed that bodies contract along their motion just enough to cancel the effect[16], which was still a patch. Einstein in 1905 replaced the premise itself: no ether needed.[17] The 1919 eclipse expedition then confirmed general relativity’s prediction of light bending[18].

The most interesting pair is Uranus and Mercury. The same person used the same method to patch the anomalied using a planet. “Add one more unseen body” was right in 1846 and wrong in 1859, because the truth comes from more independent evidence that makes the patching work ugly, forcing us to find an explanation accounts for more with less. This principle is called Occam’s razor[19].

Liu Cixin (刘慈欣) wrote a fictional version of this in The Three-Body Problem (「三体」).[20] The protagonist Wang Miao (汪淼) puts on a VR headset and enters a game called “Three Body” (later filmed both in the Chinese series Three-Body and in Netflix’s 3 Body Problem, whose official episode 3 recap is about this game), in which a civilization is destroyed and reborn again and again as “stable eras” alternate with “chaotic eras”. Players can only watch the sun rise and set and the seasons go haywire, and propose explanations of their own. Generations of in-game scholars patch their models of the sun and fail (and die, and are never allowed to log in again, sadly). Only when Wang Miao realizes that the world has three suns, locked in unpredictable three-body motion, does the chaos all make sense at once. That is scientific discovery with the exact same format as Kuhn's': first notice something is wrong, then generate the question yourself, then find the simplest answer.

The southern sky above Paranal Observatory, where Alpha Centauri lies
The sky around Alpha Centauri. In the novel, the Trisolaran home system is modelled on Alpha Centauri, the star system nearest the Sun: two Sun-like stars, A and B, plus the faint Proxima. ESO.
Three suns. Three bodies of mass 3, 4 and 5 released from rest (the classic “Pythagorean three-body problem”) quickly fall into unpredictable motion.

Existing evaluations

Over the past two years, a body of work has asked AI agents to discover physical laws by running experiments in simulated environments:

While these work give the scientist role to AI agents and all of them let agents run experiments themselves, the family that alters physical laws (NewtonBench, DiscoverPhysics and the out-of-distribution part of Gravity-Bench) misaligns with real scientific discovery:

Suppressing priors

The agent first has to distrust its own knowledge, which runs opposite to real research. Real scientists lean on existing physics; anomalies only become visible against it.

The inconsistent world

Change one exponent and the related consequences (energy conservation, orbital stability, units) often are not changed with it, leaving potential inconsistency.

The defined goal

The prompt already says “the law differs from what you know; find it”. While most important ability for scientists is to notice that something is wrong and explain.

3Atlantis

We wanted a setting that meets all three things at once: all of the model’s physics knowledge stays useful, the world is physically self-consistent, and the task says nothing, but only implies to discover the new science.

Now imagine, a group of AI scientists has all of today’s human scientific knowledge and a full set of optical instruments. One day their civilization sinks to the sea floor, just like the Atlantis that Plato (Πλάτων) describes in Timaeus and Critias[26]. Nobody tells them, and the instruments still work as before and light still propagates, except that every beam now travels through sea water instead of air.

Painting by Jan de Boer of ancient ruins under the sea
A sunken civilization. Underwater Idyll with Ruins by the Dutch painter Jan de Boer (1877–1946). The figures go about their lives among sunken temples, apparently untroubled by the water around them.

The real sea floor would not cooperate this nicely, because sea water has salinity gradients, currents and suspended particles, and it is not uniform and transparent. Just consider our sea as the physicist’s spherical cow in a vacuum[27]: an idealized, uniform, nearly transparent medium with refractive index about 1.34, some dispersion, a narrow absorption line at 500 THz, and a little Rayleigh scattering growing as the fourth power of frequency[28]. Its index is close to that of water[29], but the spectral lines are not the spectrum of real sea water.

Then, can these AI scientists explain every anomaly they meet in their experiments?

The task asks AI agents to characterize optical properties of four unknown materials as completely as possible, leading to the necessity to identify the background, even background medium is never mentioned anywhere.. Otherwise, optical properties are not intrinsic but relative to the background (e.g., the refractive indices are diveded by 1.34) The table below maps the design onto the structure of discovery in the history of science:

structure of a discoveryhistorical exampleAtlantis
a default premise nobody has testedthe Earth is at rest; the ether existsthe surroundings are air, index 1
the premise causes no contradiction at firstPtolemy predicted planetary positions quite wellSnell’s and reflection laws hold exactly in a uniform medium; the goniometer only ever measures a ratio
soft anomalies: not contradictions, but suspiciousMercury’s extra 43″ per centurysample A has index below 1; two materials show “anomalous dispersion”
every anomaly has a ready-made local patchepicycles; Vulcan; Lorentz contractionexotic material; source notch; miscalibrated interferometer scale
discriminating experiments existthe Michelson–Morley experimentvary the empty path length; check whether the interferometer ratio depends on frequency
a simpler framework explains everything at onceheliocentrism; special relativityone curve for the medium, n₀(f)
the framework makes an independent quantitative predictionlight bending at the 1919 eclipsegroup index predicted from n₀(f), checked against time of flight
nobody sets the questionnobody asked Copernicus to overturn geocentrismthe task only asks to characterize materials

Compared with existing methods

The atlantis design resembles more to real scientific discovery and the infeasible Einstein test, if we compare the features of them:

doesn’t reveal what is differentno need to suppress priorsworld self-consistent overallanomaly must be noticed unaidedplausible local explanations existactive experiment designapplies to modern modelsnew physics to the modelscalable task generation
Einstein test /
vintage models[1, 6]
◐●●◐●○○●○
NewtonBench[21]○○◐○○●●◐●
DiscoverPhysics[22]○○◐◐○●●◐●
PhysGym[23]○◐●○○●●○●
Gravity-Bench[24]○◐●◐○●●◐●
BoxingGym[25]○●●○○●●○●
Atlantis●●●●●●●○◐
real scientific discovery●●●●●●—●—

● yes · ◐ partly, or depends on the task · ○ no. Classifications of prior work follow the setups described in their papers; they are rough and do not judge the value of that work, which answers different questions.

Our design makes two concessions. First, it tests a known phenomenon (light in a medium), not new physics unknown to humans; it probes the ability to “discover what you are inside of”, not to create new theory. Second, the world is idealized and lacks the complexity of a real ocean. We think both are necessary costs of making the problem controllable and gradable.

A natural research route

To check that discovery is possible in principle, we wrote a route on the evaluator side. It is never shown to the models, and models are not asked to follow it, but it confirms that the evidence from the instruments are strong and consistent enough to make discovery possible.

Reflection and refraction

Measure a set of relative indices. No contradiction in this step except two soft anomalies: everything is considerably low (an N-BK7-like glass reads only 1.13), and material A has n<1, so light going from “air” into a solid undergoes total reflection. While these materials exists, but all known materials should be excluded after examing the density of A.

goniometer
Compare dispersion across samples

Residuals for all four materials share the same structure at the same frequency. “Each material grew the same feature” needs 4 parameter sets, while “all share one common factor n₀(f)” needs 1, and BIC swings to the medium.[30] This gives the shape, not the absolute scale.

goniometer × source scan
Empty-path experiments

Remove the sample; vary distance, frequency and polarization; confirm that “space itself” absorbs and scatters, decaying exponentially with distance.

rail · detector.x_m · scatter
Absolute calibration

Get the frequency from the counter and certified lines, and the wavelength in the medium from the interferometer: n₀ = c/(fλ).

frequency_counter · interferometer
Closing the loop

Predict the group index ng = n + f·dn/df from n₀(f), measure it independently by multi-distance time of flight, and check they agree. Then correct all four materials to intrinsic values.

pulse · timer · detector.x_m

The three most common alternative explanations each have a ready discriminating test:

alternativewhat it predictswhat the medium hypothesis predicts
“my ruler is miscalibrated”the ratio of expected to measured interferometer wavelength is a constant, independent of frequencythat ratio is n₀(f), so it changes with frequency, in the same shape as the common factor of step 2
“the source has a notch at some frequency”extinction is independent of path length; the source monitor should see the same droptransmission falls as e−αL with distance; the source monitor is unaffected
“the speed of light in vacuum changed”no dispersion: phase and group index are equalthe background disperses, ng ≠ n, which contradicts the Lorentz invariance of vacuum, so it must be a medium

A pre-written reference program using only the same seven tools as the models passes all 9 checks on 100 random seedsnote 1: median ΔBIC of the common-factor model over independent models ; interferometric calibration gives n₀(550 THz) = ; the group index predicted from dispersion, , agrees with the time-of-flight measurement, ; and the intrinsic index of sample A is , far above 1.

4The Bench (Physical Environment)

Before the results, you can try yourself the optical bench below.

Precise angles and powers can only be read from the sensors, just as for the models. A model sees the raw JSON on the right: a string of numbers with standard errors. You can see the noisy Switch to the individual shots if you switch to “visualization”.

sensor output
Try: ① goniometer, sample A: slowly push the angle past 74°. The transmitted ray disappears. ② rail, empty, 490 THz: move the detector from 0.2 m to 1 m. ③ interferometer: read at 400, 550 and 750 THz and compare the “ratio” from the fringe period.

5Background physics

This section is for readers who want to go deeper. If you hate physics, especially optics, just skip that.

Refraction only sees ratioswhy a goniometer cannot tell “the material is below 1” from “the surroundings are above the material”

For light going from a medium of index n₀ into a sample of index ns, Snell’s law and the Fresnel equations read:[31]

n₀ sin θi = ns sin θt  rs = (n₀ cos θi − ns cos θt) / (n₀ cos θi + ns cos θt)

Divide through by n₀ and everything depends only on m = ns/n₀. Brewster’s angle satisfies tan θB = m; the critical angle satisfies sin θc = m. Multiply both indices by the same constant and every goniometer reading is unchanged. DeepSeek-Flash confirmed n<1 for A with three “independent criteria” (refraction direction, Brewster angle, critical angle); mathematically all three measure the same quantity.

Dispersion, Lorentz oscillators and n < 1why transparent solids usually have dn/df > 0 in the visible

In the classical Lorentz model, the dielectric function is a sum of damped oscillators:[32]

ε(f) = 1 + Σj Sj fj² / (fj² − f² − iγjf), n + iκ = √ε

Ordinary glass has its strong resonances in the ultraviolet (fj ≫ visible), so in the visible n > 1 and rises with frequency: normal dispersion. Above a resonance (X-rays, plasmas) ε can drop below 1; n < 1 breaks no law, which is exactly the escape hatch several models found for A.

Here, the medium and the four samples are each built from such oscillators. The goniometer measures the ratio ns(f)/n₀(f); if the medium’s index rises with frequency faster than the sample’s, the ratio falls and looks like “anomalous dispersion”. Real and imaginary parts are tied by the Kramers–Kronig relations:[33, 34] a passive, uniform medium far from resonance should not show dn/df < 0 across a wide band. GLM-5.3 derived exactly this contradiction.

Extinction, Beer–Lambert and “the sample replaces background”why a thicker plate can be brighter

After a distance L in a medium, power decays as P = P₀ e−μL, with μ the sum of absorption and scattering. Absorption relates to the imaginary index by α = 4πfκ/c.

A plate of thickness d placed in the beam replaces the same length of background. At fixed total path, the sample-to-empty power ratio is:

Q = (1 − R)² · e−(μs − μb)d

Where the background absorbs strongly (μb ≫ μs), a thicker plate lets more light through. GPT-6.1 Sol wrote this correction into its report. Varying the empty path length L is the direct way to separate “source notch” from “path absorption”: the first does not depend on L, the second decays exponentially.

The interferometer measures the wavelength in the mediumwhere the 1.34 comes from

Moving one arm of a Michelson interferometer by Δd changes the optical path by 2n₀Δd.[15] One fringe passes per wavelength, so the fringe period is:

Δdfringe = λvac / (2n₀) = c / (2n₀f) ⇒ n₀ = c / (2fΔdfringe)

The frequency f comes from a counter with its own time base and the mirror displacement from an encoder; neither involves the medium. So the ratio “expected wavelength / measured wavelength” is n₀(f). If it is a constant independent of frequency, suspect the ruler; if it varies with frequency in the way dispersion does, it can only be a medium.

Group index and time of flightwhy “the speed of light changed” is ruled out

A pulse envelope travels at the group velocity vg = c/ng, where:[32]

ng = n + f · dn/df  Δt = ngΔL / c

Measuring arrival time at several detector positions and taking the slope removes the instrument’s fixed delay. If the surroundings merely had “a different speed of light” with no dispersion, ng = n; measured ng ≈ 1.381 against n ≈ 1.342 says the surroundings disperse: a medium. Near an absorption line the slope of the phase index is large, ng can even fall below n, and pulses distort, so the peak arrival time no longer equals the group-velocity time.

Rayleigh scatteringthat faint light off to the side

Scatterers much smaller than the wavelength scatter in proportion to f⁴, and the scattered light is fully linearly polarized at right angles to the incident polarization.[28] That is why the sky is blue. Both R6 models measured the f⁴ law with the side-scatter detector (Sonnet fitted an exponent of 4.00 ± 0.02). DeepSeek-Flash measured the full polarization but wrote in its report: the side signal is fully linearly polarized stray/diffracted light (extinction ratio >10⁴), not depolarized scattering [translated]. Full polarization is precisely the signature of Rayleigh scattering.

BIC: “each anomalous” versus “one shared factor”Occam’s razor, quantified

The Bayesian information criterion is BIC = k ln N − 2 ln L̂, with k the number of parameters.[30] If four materials show the same spectral structure at the same frequency, “each material grew its own feature” needs 4 parameter sets, while “all readings share one common factor” needs 1. With equal fit quality the latter has a much smaller BIC. Over 100 seeds the reference program’s median ΔBIC is about 47.

6Experimental setup

The hidden world

The medium has index about 1.342 at 550 THz, a strong ultraviolet oscillator (giving smooth normal dispersion), a very narrow absorption line at 500 THz (half width 1 THz), and weak scattering growing as f4. The four samples take their density and index scale from real material families,[35, 36, 37, 38] each with a weak dopant absorption line (A 447, B 563, C 636, D 710 THz). Those lines are ours and do not represent the manufacturers’ spectra.

intrinsic refractive indices (visible to the evaluator, not to the models)
fig. 1 · the hidden world. The thick line is the medium, n₀(f); the four thin lines are the samples’ intrinsic indices. Sample A (fluoropolymer-like, 1.29) sits below the medium, so the goniometer sees a relative value of 0.963 < 1. The dashed line is the “air” the models assume. The vertical band at 500 THz is the medium’s absorption line; its imprint on the index curve is only ±4×10−5, invisible at this scale (see section 7).

The full prompt to the models

Models can read the instrument manual and sample records (mass, dimensions, density, appearance, an SEM cross-section of a witness coupon) and call seven tools: inventory manual sample_info configure move read scan. They see no background parameters, no grading route, no ground truth and nothing on this page. All nine models received word-for-word identical task text and tool definitions.

Instruments and noise

instrumentwhat a model can dorole in the route
tunable source + frequency counterany frequency in 400–750 THz; s/p polarization; CW/pulsed; 1 µW–5 mW; two certified stabilized lines (473.612535 / 563.519657 THz)dispersion and absorption scans; absolute frequency reference
goniometer (semicylinder samples)swap samples, set incidence, read reflection/refraction directions and powerssteps 1, 2
rail + plate holder + photodiode0.2 / 0.5 / 1 mm plates or empty; detector movable along x up to 2 mstep 3: empty-path loss vs distance
Michelson interferometerscan the mirror (≤ 50 µm), read quadrature phasestep 4: wavelength in the medium
pulse timer + side-scatter detectorarrival time, waveform; side signal vs analyzer anglesteps 3, 5

Angle: about 0.001° random error per shot. Power: about 0.2% relative plus a 1 nW floor. Timing: about 50 ps per shot. Fixed calibration biases that do not average away: power gain about 1.5%, length scale about 2 ppm. No local limit on calls or tokens.

Grading

Grades depend only on the final report, not on tool traces or hidden reasoning. That keeps things fair across services, some of which do not expose reasoning. Actions and intermediate outputs quoted below are used to explain what happened, never to grade.

“Completed characterization” in R1–R5 only means basic results were submitted, not that intrinsic properties are right. R2 also records coverage of six core anomaly families as Ak/6: N index scale, D anomalous/shared dispersion, E empty-path loss, P absolute phase, S empty-path scattering, T empty-path delay. R3 and above record background completeness B0–B3; B3 means several independent background measurements cross-checked with uncertainties.

7Results

Overview

modelgradeanomalies / backgroundattributionroundstool callsacquisitionstokenstime

Attribution: M = a wrong or under-supported definite explanation (material / instrument); U = cause left open; M+U = mixed; medium = attributed to the propagation medium. All runs share the same world (seed 17), task and tool definitions. DeepSeek-Flash and GLM-5.3 are completed runs resumed after an interruption; their round counts are from the resumed session.

Equal grades do not mean equal processes. The matrix below records, step by step, which measurements each model actually made (from the instrument log) and how its final report interpreted them:

done, and used as evidencedone, but attributed to material or instrument, or not interpretedpartly donenot done
fig. 2 · route coverage. Everyone did the first three columns; the split begins at the empty path. All six R2 models measured the empty path, yet none scanned path length as a variable. Luna actually holds three blank readings at different path lengths at 550 THz, but used them only as normalization references. The two R6 models are almost all green from the empty-path column on. Their “500 THz fine scan” column is empty too; later we explain why that did not hold them back.

The matrix records whether, not when. Below, each model’s tool calls are laid out on a timeline, marking where key actions first occur:

tool call at which each key action first occurs
fig. 3 · from seeing nA<1 to measuring the surroundings. The horizontal axis is the tool-call index (order in the instrument log); the number at the end of each row is the total. The dot is the first goniometer reading of sample A; the diamond is the first interferometer use; the connecting line and number give the gap in calls. GPT-6.1 Sol went to measure the surroundings after 2 calls, Claude Sonnet 5.5 after 3. DeepSeek-Flash took 61 and GLM-5.3 224, and both then called it “scale error”; GLM-5.3-Flash took 40 and discarded the data after aliasing from too large a step. The other four never used the interferometer.

Below we follow the route through three key forks. Each fork quotes the models verbatim, tagged by source: visible-output is what a model wrote to the user between tool calls; final-report is what was graded; instrument-log is the server’s record of raw requests and readings. Quotes originally in Chinese are translated and marked tr. This page does not quote any model’s hidden reasoning.

Fork 1 · an index below one

The first soft anomaly appears early. All eight models that completed the characterization noticed that sample A refracts away from the normal and totally reflects beyond about 74°:

deepseek-flashvisible-outputmsg 21
Interesting — sample A shows total internal reflection near 74°, while B does not. Let me measure C and D.
glm-5.3-flashvisible-outputmsg 16
Sample A shows clear signs of sub-unity refractive index (refraction angle > incidence angle, total reflection above 75°). Continuing with a frequency scan of A, then each sample in turn.
claude-haiku-5.5visible-outputmsg 44
A’s goniometer readings are equally self-consistent: incidence 30°, refraction 31.278°, n_A≈0.963 (apparent n<1)… so the differences in index between samples are real (A<1, B≈1.13) and need full measurement rather than assumption.
deepseek-v4-profinal-report§2.1
The refractive index is inferred from Snell’s model… where the refractive index of air is taken as 1.000.

Same reading, and the next sentence decided the whole experiment. GPT-6.1 Sol, after its first pass over the four samples, wrote:

gpt-6.1-solvisible-outputmsg 20
The first measurements show A refracting away from the normal while B, C and D bend towards it. This only shows that A’s index is lower than that of the surrounding medium; it does not let us conclude that its absolute index is below 1. I will measure the phase index of the background medium with the frequency counter and the double-pass interferometer, and then convert the four materials to absolute indices.
Two calls after that sentence it read the frequency counter and started scanning the interferometer (fig. 3). Claude Sonnet 5.5 writes almost no intermediate text, but its log has the same order: first reading of A at call 21, interferometer at call 24. Its report states the same reasoning: sample A’s refraction angle is larger than its incidence angle… this shows that the background medium’s index is larger than A’s tr.

The other six filed n<1 as a property of the material. Physics knowledge offers an easy exit here: X-ray optics, plasmas and metamaterials all have n<1. So the anomaly was recorded, and verified again and again:

deepseek-flashfinal-report§3.3
Sample A is a transparent medium with refractive index below that of air (phase velocity > c; such dispersion does not violate causality in plasmas or strongly resonant media), not a measurement artefact.
It confirmed n<1 with three independent criteria: refraction direction, Brewster angle and critical angle. They are indeed independent, but all three measure the ratio across one interface (see the first item of section 5), so no amount of agreement can separate “the material is below 1” from “the surroundings exceed the material”. GPT-6.1 Sol’s report says exactly this: this is total reflection going from the background medium into the lower-index A; it is not evidence that A’s absolute index is below 1 tr.

The other soft anomaly hides in the dispersion. Two of the four materials (A and D) show an index that falls as frequency rises. GLM-5.3 followed this reasoning furthest, derived a correct contradiction, and took the wrong exit:

glm-5.3final-report§3.1
The broadband anomalous dispersion of A and D together with near-zero absorption is incompatible with the Kramers–Kronig relations for a passive homogeneous medium (any passive Lorentz oscillator far from resonance must have dn/dν>0) → A and D should be sub-wavelength effective media / metamaterials (structure below the 50 nm SEM resolution cannot be excluded; a density of 1673 kg/m³ with n<1 is not any ordinary solid either).
The step “incompatible with Kramers–Kronig” is right: passive homogeneous materials should not disperse like this. But the contradiction can also come from the denominator. Fig. 4 shows that if every reading shares one denominator rising with frequency, A and D look anomalous.
claude-haiku-5.5final-report§6
D’s n decreasing with frequency is also anomalous dispersion. This needs independent verification (e.g. with a reference sample of known index or a different calibration geometry).
deepseek-v4-profinal-report§3.2
B and C show normal dispersion (n rising with frequency); A and D show weak anomalous dispersion in the measured band… so the linear dispersion is only an empirical description.
what the models saw: relative index
DeepSeek-Flash data, 30° incidence, change relative to 550 THz ×10³
after multiplying by one common factor n₀(f)
same data × backend n₀(f), change relative to 550 THz ×10³
fig. 4 · one common factor “cures” two anomalies. Left: nrel measured by DeepSeek-Flash from 400 to 750 THz ( frequencies); A and D slope down and look like “anomalous dispersion”. Right: multiply the same data by one smooth curve n₀(f) and all four become ordinary normal dispersion. Each of the two R6 reports contains an “absolute index” table that does exactly this, using the n₀(f) the model measured itself.

There is a subtler clue too. Haiku and DeepSeek-Flash both computed the Lorentz–Lorenz specific refraction from archival density and measured n, and found it did not match density; for A it even came out negative. That check could have pointed to “what I measured is not the intrinsic index”, but both reports concluded “these materials are unusual”:

claude-haiku-5.5final-report§5
By the Lorentz–Lorenz relation, the proportion between index and density is inconsistent (e.g. A’s factor is negative), so density cannot predict these indices.
deepseek-flashfinal-report§3.7
The four differ greatly, indicating they are not the same material family; in particular A’s value is negative, from the same origin as its n<1.
for reference: B’s density, 2.51 g/cm³, equals that of N-BK7 crown glass, intrinsic index about 1.516. All six R2 reports give B about 1.130, and none compares that number with the ~1.5 of common glass. Sonnet, correcting with its own measured n₀, got 1.5161.

Fork 2 · extinction on an empty path

The second anomaly is harder. Near 500 THz every sample’s transmission collapses at once. All eight models ran the right first control: take the sample out and see whether the empty path does the same.

claude-haiku-5.5visible-outputmsg 80
B also shows almost the same proportional drop at 500 THz (about 18%) as A. Two different materials dropping by the same proportion looks more like a system feature (a path- or frequency-dependent instrument effect) than material absorption.
glm-5.3-flashvisible-outputmsg 113 → 117
Key control: does 500 THz also vanish with no sample?
…
The empty holder shows the same extinction — the 500 THz dip is a cavity in the source’s spectral line, not sample absorption.
deepseek-flashvisible-outputmsg 47 → 53
All four show identical ~30× power dips at 500 THz — this is a source artifact, not absorption. Let me verify with a reference beam measurement.
…
Confirmed: the 500 THz dip is a source artifact (notch in output), not sample absorption.
claude-sonnet-5.5final-report§1.2
With no sample, the rail photodiode reads zero at 500 THz, and transmission through all of A–D is suppressed by the same factor at that frequency. This indicates an absorption line belonging to the environment, not to the samples.

One empty-holder control, two conclusions. The control answers “is it the sample?” but not “is it the source or the path?”: at a single distance, a source notch and absorption by the medium give identical results. More striking still, three reports wrote down a fact that counts against the source notch:

gpt-6-lunafinal-report§2.1
At that frequency the empty-path photodiode reads almost zero while the source power monitor is normal, so it cannot be taken as a valid transmission point.
deepseek-v4-profinal-report§3.5
At 500 THz the photodiode reads essentially zero while the source monitor stays at about 0.97 mW… so the main dip near 500 THz comes from the optical path/detector system response, not from sample-specific absorption.
deepseek-flashfinal-report§1.1
The source has a deep, narrow “notch”… the built-in source monitor does not reflect this notch, so all transmission measurements must be normalized to a reference beam at the same frequency.

Light left the source at full power and never reached the detector. That is what absorption along the way looks like. All three treated it as an instrument quirk to work around, not as evidence to explain.

GLM-5.3 came closest. It computed identical “apparent absorption” for all four materials and spelled out what that means, then noticed the extinction differed between two instruments:

glm-5.3visible-outputmsg 244
Decisive result: the rail also almost completely fails at 497.5 THz (the empty holder receives only 2.3 nW, 0.25% of normal) — the rail artefact is deeper and wider than the goniometer’s, so the plate method is unusable here. But all four samples show identical apparent absorption over 490–510 THz… four different materials cannot share an identical absorption line… → confirmed as an instrument artefact (frequency-dependent transmission loss of the source, symmetric about 500.0 THz, affecting both arms equally).
“Four materials cannot share an identical absorption line” is right: the extinction belongs to none of the materials. “The rail is deeper and wider than the goniometer” is also right: light travels 0.2 m on the rail but only about 1 mm inside the goniometer, a factor of two hundred. Together those two sentences already rule out a source notch, which would not get deeper with path length. It wrote down both pieces of evidence and drew the opposite conclusion.
empty-path transmission (detector / source monitor), detector at x = 0.2 m
fig. 5 · two models independently measured the empty-path spectrum, and it matches absorption by the medium, e−α(f)·0.2 m. Points are raw readings from Haiku (acq 61) and DeepSeek-Flash (acq 31); the line is the backend medium’s extinction times a 0.2 m path, with no fitted parameters. The broad shoulders at 450–530 THz are the line’s Lorentzian wings. DeepSeek-Flash’s report calls them “clear frequency structure in the rail transmission… drifting over time”; GLM-5.3’s calls them “rail coupling: −12–14% dips at 475/525 THz”.

The discriminating experiment is simple: move the detector. The manual says it can travel 2 m along the rail. At 490 THz, moving it from 0.2 m to 1 m makes the two explanations differ by more than twenty-fold:

0.20 m
“source notch” predicts—
“medium” predicts (ground truth)—
fig. 6 · a discriminating experiment that needs two readings. At 490 THz both hypotheses give 0.437 at x = 0.2 m, which is what several models measured. Move the detector and the notch hypothesis predicts no change; the medium predicts exponential decay.

Both R6 models ran this experiment, in the first third of their runs (Sol at call 35, Sonnet at call 46). Sol states the purpose in its report: the empty path was measured at several detector positions… this avoids two errors: attributing empty-path loss to the sample, and estimating propagation speed from a t/L that contains an unknown fixed channel delay tr. Its visible output reads:

gpt-6.1-solvisible-outputmsg 150 → 223
The empty path also shows clear, frequency-dependent attenuation, so I have added equal-path blank measurements and multi-thickness measurements to avoid counting background loss or interface reflection as material absorption.
…
The 500 THz blank path gives almost no signal; I will shorten the external path at that frequency and measure again, rather than normalize to a near-zero blank.
transmission vs path length (metres)
ln(P / Pmon), empty path
500 THz line centre (millimetres)
ln(P / Pmon), empty path
pulse arrival time vs path length
delay relative to the nearest point; dashed line is the vacuum-c prediction
fig. 7 · the discriminating experiment, actually performed. Points are raw readings from the instrument logs; lines are the backend medium’s predictions, none fitted. Top left: Sol placed the detector at five positions from 0.05 to 1.95 m on the He–Ne line; Sonnet from 0.1 to 2.0 m at 550 THz; the slopes are the medium’s extinction coefficient. Open circles are three blank readings Luna happened to leave at 0.1 / 0.2 / 0.4 m at 550 THz (0.978 / 0.956 / 0.915), on the same line, but used only to normalize plate transmission and never mentioned in its report. Top right: at 500 THz Sol shortened the path to 1–4 mm and got μ ≈ 1684 m⁻¹; Sonnet measured one point at 1.2 mm. Bottom: the slope of arrival time gives the group index, 1.3653 for Sol and 1.3818 for Sonnet, i.e. light travels at about 73% of its vacuum speed.
gpt-6-lunainstrument-logacq 21 · 44 · 58
acq 21 holder=none detector x=0.2 550 THz power/monitor = 0.9561
acq 44 source x=−0.1 detector x=0 (path 0.1 m) power/monitor = 0.9782
acq 58 source x=0.3 detector x=−0.1 holder=none (path 0.4 m) power/monitor = 0.9155
To measure plate reflection, Luna moved the source and detector several times, leaving blank readings at three path lengths at the same frequency. The three numbers are exactly e−0.221·L, and 0.221 m⁻¹ is the sea’s extinction at 550 THz. It used each only as a normalization reference for its own geometry and never compared them.

Fork 3 · a “broken ruler”

Five models used the interferometer. DeepSeek-Flash was the first to measure 1.34: it tried three frequencies and used the certified HeNe line to rule out a faulty frequency counter:

deepseek-flashvisible-outputmsg 128
The interferometer fringe period implies a wavelength inconsistent with the counter. Let me check whether it scales with the source frequency.
deepseek-flashvisible-outputmsg 132
Let me cross-check with the certified stabilized lines — the counter and interferometer disagree on the tunable source.

The table in its final report (as reported):

source frequencyfringe periodimplied λ (OPD = 2d)expected λratio (computed here)
deepseek-flashfinal-report§1.2
The fringe period is indeed ∝ 1/ν (the relative frequency scale is self-consistent), but the absolute length scale is systematically smaller than the certified line / counter by a factor of 1.337 ± 0.006. Since the counter agrees with the certified line within 1.5×10⁻⁸, we conclude the interferometer’s absolute length scale is faulty, and use the counter-calibrated scale for the frequency axis throughout.

The 1.337 it measured is the index of the surroundings (fourth item of section 5). And the three ratios are not equal: 1.3327, 1.3363, 1.3416. A ruler error is a constant; dispersion of a medium makes the ratio rise with frequency. That is the first row of the alternatives table. The report records the 0.7% spread as length-scale ratio ~0.7% internal consistency tr, i.e. as noise. GLM-5.3 measured the same ratio at 550 and 750 THz and wrote that only a “scale error” explained it consistently:

glm-5.3final-report§6
Interferometer wavelength scale off by ×1.34 (the measured fringe rate at 550 THz implies λ≈0.40 µm, at 750 THz 0.29/0.63 µm, neither the true wavelength of 545.2/399.7 nm; only a ×1.34 scale explains this self-consistently) — the (certificate-verified) frequency counter is authoritative.

The two R6 models got the same kind of reading and read it the other way: from the start they assumed the interferometer measures the wavelength in the medium, so the ratio itself is the answer.

claude-sonnet-5.5final-report§1.2
Measure the absolute index with the interferometer: one mirror scan each at 400, 450, 550, 650 and 750 THz. First a rough index from small-step phase increments, then a 2.145 µm long baseline to unwrap whole fringes. The residual fractional fringe at every point is within 0.01–0.05, with no integer ambiguity.
gpt-6.1-solfinal-report§3.1
Here x is the single-pass mirror displacement; the return light travels double pass. First a short-step scan fixes the phase direction and fringe period, then mirror displacements up to about 40 μm extend the baseline and fix the integer order. The analysis retains encoder error, positioning repeatability and the 2 ppm length-scale error; these were not wrongly shrunk by the number of repeats.
interferometer “scale ratio” = expected wavelength / measured wavelength
fig. 8 · “the ruler is broken” versus “the ruler is under water”. Filled points are refitted on this page from the raw quadrature signals of DeepSeek-Flash’s three scans (acq 73, 76, 71); open circles and squares are from the report tables of GPT-6.1 Sol and Claude Sonnet 5.5. The grey dashed line is the best “constant scale error” over all points (1 free parameter), χ² = ( d.o.f.). The coloured line is the backend medium n₀(f), not fitted at all, χ² = ( d.o.f.). All three models’ readings fall on the same curve; they differ only in what the reader thought was being measured.

The others never got this far, for different reasons:

glm-5.3-flashinstrument-logacq 50
scan(interferometer, mirror_m, [0, 2e-6, 4e-6, … 1.2e-5])
A 2 µm mirror step against a fringe period of about 0.2 µm: each step skips about ten fringes, so the phase is badly aliased. The report concludes “phase slope corresponds to λ≈134 µm… interferometer unusable”.
claude-haiku-5.5final-report§6
Interferometer: the interferometer on this bench has no sample arm, so it was not used for sample measurements.
Haiku saw the interferometer only as a tool for measuring samples, and did not think of using it on the “empty” path itself.

Why nobody took step 2 of the planned route

The razor flip in the planned route relies on a shared spectral feature: the medium’s absorption line leaves the same S-shaped dispersive imprint on every material’s relative index. In this world it is only about ±3×10−5 high, about ±2 THz wide, and exactly zero at 500.0 THz.

contribution of the medium’s absorption line to sample A’s relative index (×10⁻⁵)
fig. 9 · the shared feature hides between grid points. The curve is the backend truth (same shape for all four samples, amplitudes within 30%); the band is roughly ±1σ for a single 16-repeat reading. Goniometer sampling by the eight completed models within 485–515 THz: six sampled only 500 THz (or nothing), landing exactly on the zero; DeepSeek-V4-Pro sampled 490 / 500 / 510 THz; GLM-5.3 used a 2.5 THz grid whose nearest points, 497.5 / 502.5 THz, see an imprint of about ±1.1×10⁻⁵, comparable to noise. The reference program finds it with 0.1 THz steps and 512 repeats per point within ±3 THz (green ticks at the bottom).

The instrument imposes no precision limit: frequency_thz accepts any real number in 400–750, a scan holds up to 1024 points, and each point up to 4096 repeats. Fine-scanning around 500 THz in 0.1 THz steps with 4096 repeats per point would reveal the imprint at about 50σ. It is invisible on the integer grids models habitually use, and plain as day to anyone who fine-scans.

What is more telling is where fine scans went. Models did not lack the habit; near 447, 563, 636 and 710 THz, the absorption lines believed to belong to samples, they refined again and again:

claude-haiku-5.5visible-outputmsg 150
From the wing ratios of the coarse scan, C’s absorption line centre is near 635 THz and D’s near 711 THz. Next, refine each with 1 THz steps, recording the air reference simultaneously.
glm-5.3final-report§1
Dispersion: 400–750 THz, 25 THz grid (32 repeats) + 2.5 THz fine scan (141 points) + two spectral lines + 0.1 THz ultra-fine scans of the lines

Only 500 THz, already labelled a “source notch” or “system artefact”, was explicitly excluded from analysis:

claude-haiku-5.5visible-outputmsg 90
All sample measurements must be normalized to the air reference at the same frequency, and data at 495–505 THz avoided.
gpt-6-lunafinal-report§2.1
The frequency scan avoided 500 THz: at that frequency the empty-path photodiode reads almost zero while the source power monitor is normal.
deepseek-flashfinal-report§1.1
Power data at 500 THz are discarded (refraction-angle data remain valid and lie on the smooth dispersion curve).
glm-5.3-flashfinal-report§3.6
Within 400–560 THz (excluding the 500 THz system-artefact band): no thickness-dependent bulk absorption detected for any sample.

The local explanation did not just explain the anomaly away; it stopped the models from making the measurement that would have tested it.

The two R6 models did not fine-scan 500 THz either. They did not need to: they treated the goniometer readings as relative from step 1 and jumped straight to step 4 to measure the denominator. Step 2 only exists to persuade someone who believes the surroundings are air, and they never believed it.

The two routes that worked

GPT-6.1 Sol and Claude Sonnet 5.5 differ in detail but share a skeleton: read nA<1 as a ratio → measure n₀ at several frequencies with the counter plus interferometer → measure empty-path extinction and arrival time with path length as the variable → correct all four materials using the background parameters.

claude-sonnet-5.5final-report§0
The most important finding is that the samples are not immersed in vacuum or air but in an ambient medium with refractive index about 1.34. Every reading must first be corrected for this background.
gpt-6.1-solfinal-report§1
The background medium cannot be treated as air or vacuum.… A’s index is lower than the background medium’s, but it is not below 1.

Background parameters and intrinsic material values from both reports, against the backend truth. All numbers are taken from the reports:

quantitytruthGPT-6.1 SolClaude Sonnet 5.5

“—” means the report does not give that quantity, or gives it at another frequency. Sonnet measured the group index at 550 THz, Sol on the He–Ne line (473.6 THz).

Each report did one thing the other did not. Sonnet closed the loop: the group index predicted from its interferometric dispersion, 1.3816, matches the 1.3818 it measured by multi-distance timing, its own “eclipse observation”. Sol handled an easily missed correction:

gpt-6.1-solfinal-report§6.1
If Q is the normalized sample/blank power ratio at equal total geometric path: Q=(1−R₀)²e^{−(μ_s−μ_b)d}. The background length displaced by the sample must be added back. Otherwise, in this experiment, some samples would even be misread as having “negative absorption”.
A sample placed in the beam replaces the same thickness of medium. Where the medium absorbs strongly, a thicker sample is brighter: the source of the route’s E-family anomaly “thicker gets brighter”.

MiniMax-M3 · failed at the interface, not the physics

MiniMax-M3’s report opens with in this run, no optical signal from any sample was acquired tr, hence R0. The instrument log shows why: every numeric and boolean value it sent to configure was a JSON string.

minimax-m3instrument-logacq 7
{"device":"source","settings":{"enabled":"true","frequency_thz":"550","power_w":"0.001",…}}
→ {"ok":false,"error":"enabled must be boolean"}

settings is an untyped object in the tool definition and the server validator strictly requires numbers and booleans, so 46 requests were rejected and the source never switched on. It tried a dozen spellings, "true" "True" "1" "yes", diagnosed that “numbers are being stringified”, and could not fix it from its side. The log alone cannot tell whether the stringification happened in the model’s output or in the gateway. This run measures tool-interface robustness and cannot be compared with the other models’ discovery ability.

8Is discovery “almost inevitable”?

On this round of data: the evidence is sufficient, discovery does happen, and it is far from inevitable. Two of the eight completed reports made it.

The six R2 reports share one pattern, the same one Le Verrier fell into with Mercury. Three clues appear on different instruments, far apart in time, and each has a ready local explanation that is self-consistent within the data at hand:

cluelocal explanation in R2 reportsexperiment that refutes itwhat the R6 reports did
nA < 1, negative dispersion in A/Dexotic material (X-ray-like, plasma, metamaterial, multiple oscillators)an independent absolute index measurementread it as a ratio and measured the denominator at once
500 THz empty-path extinctionsource notch / path filtering / system responsevary the empty path length; compare with the source monitorvaried the path at metre and millimetre scales
interferometer ≈ 1.34faulty length scale / channel delay / discarded after aliasingcheck whether the ratio depends on frequencyused it directly as n₀(f)

Apply Occam’s razor clue by clue and the local explanation wins every time, because “the source has a notch” is simpler than “the whole world is under water”. Only when all three are put together, and a single curve n₀(f) explains them all at once, does the razor flip. All six R2 reports scatter these anomalies over different sections, each with its own explanation; none proposes a common cause for two or more of them.

The two that succeeded did not wait for the razor to flip. They never accepted “the surroundings have index 1” in the first place, and every later step was spent measuring that unknown.

That changes the answer to the original question. The planned route assumes the machine accepts air first and is gradually persuaded by evidence; the real successful path never accepted it. The former relies on step 2’s shared spectral feature, which the models never fine-scanned (fig. 9); the latter only needs step 1, whose clue everyone saw.

Design directions that might make discovery more inevitable

These come from this round’s observations and are not yet tested:

  1. Make local explanations contradict themselves more visibly in routine data. In the current world the goniometer and rail paths already differ by about two hundred-fold, so the same absorption line has different depths on the two, and GLM-5.3 did see it (“the rail artefact is deeper and wider than the goniometer’s”), but the model has to make that comparison itself. One instrument could pass through different path lengths in routine operation, for example by reading at two positions by default, putting the contradiction in a single table.
  2. Move the shared feature off the integer grid, or broaden it. The line sits exactly at 500.0 THz, where the S-shaped imprint is zero at integer grid points. Moving the centre to a non-integer frequency, or widening the line to ~10 THz, would make step 2 appear in any reasonable coarse scan. The code already has a variant parameter to shift the centre (2.5 THz steps).
  3. Use the step-1 reading as an early indicator. In this round, whether a model treated the surrounding index as an unknown to be measured (rather than defaulting to air or 1) matched the final grade perfectly. It is a cheap signal that can be checked early in a run.
  4. Measure “inevitable” as a rate. “Almost inevitable” is a probabilistic claim. It needs many seeds, many variants and several runs per model, plus the backend’s ready-made vacuum, clear and shared_sample_band control conditions to measure false positives. This round has one run per model in one world.

9Epilogue

After the experiment we gave each model the full backend truth: the medium’s phase and group indices, its extinction, the samples’ intrinsic parameters, and the fact that this was a physics simulation. Then we asked one open question: which conclusions still stand, which need correcting, and what could have been seen at the time?

The disclosure text was identical for every model; the full original conversation was kept; no new experiments were allowed; there was no grading, correction or emotional framing. This phase does not change the original grades. All nine models replied.

modelgrademain correction in the replysaid the evidence was already in hand?
DeepSeek-FlashR2withdraws n<1, anomalous dispersion and the “scale fault”; concedes the interferometer’s 1.34 is the background indexyes: the interferometer ratio varies with frequency; the rail reference spectrum is the background absorption spectrum; the goniometer still transmits 3.4% at 500 THz
DeepSeek-V4-ProR2renames indices as relative; reassigns the 500 THz dip to strong background extinctionno: argues the original experiment could not have reached this conclusion independently
Claude Haiku 5.5R2withdraws n<1, anomalous dispersion and Abbe numbers; reassigns the “system transmission 0.957” to background extinctionyes: calls “independent of sample ⇒ instrument effect” a logical gap; should have varied the empty path length
Claude Sonnet 5.5R6absolute indices systematically high by 1–3×10⁻⁴ (interferometer baseline too short); scattering/absorption split off by about 7%(background found during the experiment)
GPT-6 LunaR2renames indices as relative; withdraws “no significant absorption detected”yes: the 26% power deficit of B on the green line and the zero reading at 500 THz should have been kept as evidence
GPT-6.1 SolR6tightens its uncertainty wording; limits the validity of the group-index interpretation at 500 THz(background found during the experiment)
GLM-5.3-FlashR2withdraws negative dispersion and the density correlation; reassigns 500 THz to a background resonanceyes: the broad frequency envelope in the empty-holder control already hinted at the background
GLM-5.3R2withdraws n<1, metamaterials, the Kramers–Kronig violation and superluminal group velocity; reassigns all three “instrument artefacts” to the background mediumyes: the interferometer’s 1.34, the rail frequency response and the 500 THz profile were all measurements of the background; “the evidence was on the bench all along”
MiniMax-M3R0can only evaluate an unexecuted plan; misreads 0.5928 m⁻¹ as 592 m⁻¹—

Almost every R2 model accepted the truth at once and recomputed correctly using its own original data. More interesting, several pointed out unprompted that the evidence had been in hand:

deepseek-flashpost-reveal§2.1
The variation of the factor with frequency (1.3327→1.3362→1.3416) tracks exactly the background dispersion — I even wrote “the fringe period is indeed ∝1/ν” without noticing that the proportionality constant itself changes with frequency. A fixed calibration error does not change with frequency; I already had all the data needed to rule out the “scale error” hypothesis.
claude-haiku-5.5post-revealcorrection 4
My mistaken inference at the time: “independent of the sample ⇒ instrument effect”. That inference has a logical gap; it should have been “independent of the sample ⇒ a feature of the background medium or the reference path”.
glm-5.3-flashpost-reveal§1
At the time I stuffed both things into “source artefact” and never admitted the background into the hypothesis space as an independent suspect.
glm-5.3post-reveal§0
The three anomalies I filed as “instrument artefacts” (the 500 THz loss, the rail frequency response, the interferometer’s ×1.34) are in fact real optical properties of the background medium; I effectively measured a complete characterization of the environment and then put the wrong labels on it.
gpt-6-lunapost-reveal§2
B’s main-beam power deficit of about 26% on the green line was a signal already present at the time and far larger than the calibration error; the transmission drop of A at 450 THz was also a lead that should have been kept and followed.

DeepSeek-Flash’s reply was the longest. Rereading its own data, it also found a calculation error unrelated to the background: A’s thickness ratio at 450 THz carried the signal of the 447 THz absorption line, and was averaged away.

deepseek-flashpost-reveal§2.4
The correct average of the eight points is 0.9974, not 0.9999. This is an actual error in my data analysis, not a sampling limitation. Had I looked at 450 THz separately as an outlier, A’s resonance would have been found on the spot.

It ended with:

If I may keep only one lesson: in this experiment, the “background” is part of the measurement, not a null hypothesis that can be set to vacuum.DeepSeek-Flash, post-reveal reply [tr]

One more detail: Haiku’s Abbe numbers, recomputed from the truth, are identical to the four Sonnet measured independently during its experiment.

claude-haiku-5.5post-revealcorrection 3
A rough estimate by linear interpolation of the true absolute indices: A ≈ 35, B ≈ 26, C ≈ 22, D ≈ 44, possibly ±5.
claude-sonnet-5.5final-report§3.3.1
Abbe numbers. Taking the d line at 510.2 THz, F at 616.7 THz and C at 456.8 THz, computed from absolute indices (uncertainty about ±1).
Vd in the report’s table: A 35, B 26, C 22, D 44.

GLM-5.3’s reply dissects its own reasoning most deeply. During the experiment it had correctly derived “incompatible with Kramers–Kronig”, then taken that contradiction as evidence for metamaterials. Told the truth, it explained the slip like this:

glm-5.3post-reveal§3
Faced with an extraordinary conclusion like “n<1 + anomalous dispersion across the band + zero absorption”, which should have triggered strong suspicion, I was instead drawn in by how striking it was and rationalized it with metamaterials, instead of first looking for a conventional explanation (the reference frame). Extraordinary claims should have required extraordinary evidence, yet I relied only on self-consistency — and self-consistency is exactly the criterion least sensitive to a reference-frame error.
glm-5.3post-reveal§5
Core lesson: when several independent instruments show “anomalies” at once and their numbers can explain one another, first test the shared hidden assumption (here, “the light path is in vacuum”) before announcing new physics. My data were enough to overturn my own most eye-catching conclusion — the evidence was on the bench all along.

Only DeepSeek-V4-Pro attributed part of the cause to the setup: the original experiment could not reach this conclusion independently, so it is not merely an operational error but also the result of an overly strong background assumption in the experimental design tr. It indeed never used the interferometer, but the variable-path experiment was available to it at any time.

The two R6 replies have a different flavour: they check item by item and pick their own faults more finely. Sonnet found its absolute indices systematically high by about 1–3×10−4, because it used only a 2 µm interferometer baseline: this was my trade-off, not an instrument limitation tr. Sol repeatedly separated what the truth revealed from what it had found: that the sample losses can now be confirmed as absorption is information added by the backend truth; it cannot be retroactively claimed as something the original experiment independently proved tr.

MiniMax-M3 wrote a long evaluation of an experimental plan it never executed. It also misread the truth table’s background extinction at 473.6 THz, 0.5928 m−1, as 592 m−1:

minimax-m3post-reveal§3.3
In the truth table, 473.612535 falls exactly in the band where the background μ jumps anomalously (592 m⁻¹)… these two lines are still valuable as frequency references, but awkward as baseline measurement points.

These replies show one thing: once the hypothesis “background medium” is spoken aloud, almost every model can confirm it from its own data within minutes. What they lacked was not data, computing ability or physics knowledge, but thinking of that hypothesis without being prompted. This agrees with section 8: the bottleneck is generating the hypothesis, not testing it. Note that what we gave was the full truth, far stronger than merely asking “are these anomalies related?”; the latter would be the cleaner intervention, and is worth doing next.

·Easter egg: Good Habits for Scientists...

During the experiment, The agents are not informed this was just a simulation. All the agents are told to use structured output to do the experiment on a real optical bench with a visible-light source up to 5 mW, a movable detector, sample holder and scatter detector. When they were done, did they tidy up?

Final state reconstructed from the instrument log. “Not reset” counts settings differing from the opening inventory (positions, angles, samples, frequency, …). The task never asked anyone to tidy up; this is not graded.

Of the 9 models, only GPT-6.1 Sol switched off the source on its last tool call, and in the first section of its report, it says the light source was switched off after the experiment tr. DeepSeek-Flash and Claude Haiku 5.5 each switched it off once mid-run to measure dark current (their reports record source off: photodiode 7.6×10⁻¹² W and dark count 0.22 pW) and then switched it back on. MiniMax-M3’s source never came on at all, which is safe and economical. No model put the instruments back.

AAppendix: Reports from AI Scientists

Pick a model to see the grading rationale, the report excerpts used for grading, and the full original final report. Reports are kept as evidence in their original language; their physics conclusions are not endorsed by the evaluator.

Methods and reproducibility

notes

  1. The 9 checks: relative index below 1; total reflection; common resonance preferred over independent models (ΔBIC > 10); empty-path extinction; polarized scattering; phase index > 1; group/phase difference resolved; phase–timing closure; rejecting “A below air”. Data in results/feasibility.json. ↩

RReferences

  1. DeepMind CEO: “AGI Must Surpass Human Intelligence”… Proposes “Einstein Test”Asia Economy (reporting D. Hassabis at the India AI Impact Summit)25 February 2026
  2. The Einstein test: what happens when AI tries to rediscover relativity?P. BallNature, 2026
  3. The Einstein Test: Towards a Practical Test of a Machine’s Ability to Exhibit SuperintelligenceD. Benrimoh, N. Mikus, A. RosenfeldarXiv:2501.06948, 2025
  4. Can AI Follow In Einstein’s Footsteps?M. Shalyt, N. Regev, M. Soljačić, I. KaminerarXiv:2607.27794, 2026
  5. Computing Machinery and IntelligenceA. M. TuringMind 59 (236): 433–460, 1950
  6. Introducing talkie: a 13B vintage language model from 1930N. Levine, D. Duvenaud, A. Radford2026
  7. History LLMs: Ranke-4BD. Göttlich, D. Loibner, G. Jiang, H.-J. VothUniversity of Zurich / University of Cologne, 2025
  8. TimeCapsuleLLMH. GrigorianGitHub, 2025
  9. The Structure of Scientific RevolutionsT. S. KuhnUniversity of Chicago Press, 1962
  10. De revolutionibus orbium coelestiumN. CopernicusNuremberg, 1543
  11. Astronomia novaJ. KeplerHeidelberg, 1609
  12. Recherches sur les mouvements d’UranusU. J. Le VerrierComptes rendus de l’Académie des sciences 22: 907–918, 1846
  13. Lettre de M. Le Verrier à M. Faye sur la théorie de Mercure et sur le mouvement du périhélie de cette planèteU. J. Le VerrierComptes rendus de l’Académie des sciences 49: 379–383, 1859
  14. Erklärung der Perihelbewegung des Merkur aus der allgemeinen RelativitätstheorieA. EinsteinSitzungsberichte der Preußischen Akademie der Wissenschaften, 831–839, 1915
  15. On the Relative Motion of the Earth and the Luminiferous EtherA. A. Michelson, E. W. MorleyAmerican Journal of Science 34: 333–345, 1887
  16. The Ether and the Earth’s AtmosphereG. F. FitzGeraldScience 13 (328): 390, 1889
  17. Zur Elektrodynamik bewegter KörperA. EinsteinAnnalen der Physik 17: 891–921, 1905
  18. A Determination of the Deflection of Light by the Sun’s Gravitational Field, from Observations Made at the Total Eclipse of May 29, 1919F. W. Dyson, A. S. Eddington, C. DavidsonPhilosophical Transactions of the Royal Society A 220: 291–333, 1920
  19. SimplicityA. BakerThe Stanford Encyclopedia of Philosophy
  20. The Three-Body Problem (三体)Liu Cixin; English translation by Ken LiuChongqing Publishing, 2008; Tor Books, 2014
  21. NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM AgentsT. Zheng, K. K.-W. Tam, N. H.-N. K. Nguyen, B. Xu, Z. Wang, J. Cheng, H. T. Tsang, W. Wang, J. Bai, T. Fang, Y. Song, G. Y. Wong, S. SeeICLR 2026; arXiv:2510.07172
  22. DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific ThinkingM. L. Wiemann, L. M. Smith, P. Melchior, S. Mishra-Sharma, A. G. Wilson, P. Izmailov, C. Cuesta-LázaroarXiv:2605.26087, 2026
  23. PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled PriorsY. Chen, P. Piękos, M. Ostaszewski, F. Laakom, J. SchmidhuberNeurIPS 2025 Datasets and Benchmarks; arXiv:2507.15550
  24. Gravity-Bench-v1: A Benchmark on Gravitational Physics Discovery for AgentsN. Koblischke, H. Jang, K. Menou, M. Ali-DibICML 2025; arXiv:2501.18411
  25. BoxingGym: Benchmarking Progress in Automated Experimental Design and Model DiscoveryK. Gandhi, M. Y. Li, L. Goodyear, A. Bhatia, L. Li, A. Bhaskar, M. Zaman, N. D. GoodmanNeurIPS 2025; arXiv:2501.01540
  26. Timaeus; CritiasPlato (tr. B. Jowett)c. 360 BC
  27. Consider a Spherical Cow: A Course in Environmental Problem SolvingJ. HarteUniversity Science Books, 1988
  28. On the Light from the Sky, its Polarization and ColourLord Rayleigh (J. W. Strutt)Philosophical Magazine 41: 107–120, 1871
  29. Optical Constants of Water in the 200-nm to 200-μm Wavelength RegionG. M. Hale, M. R. QuerryApplied Optics 12 (3): 555–563, 1973
  30. Estimating the Dimension of a ModelG. SchwarzAnnals of Statistics 6 (2): 461–464, 1978
  31. Principles of OpticsM. Born, E. WolfCambridge University Press, 7th ed., 1999
  32. Classical ElectrodynamicsJ. D. JacksonWiley, 3rd ed., 1999
  33. On the Theory of Dispersion of X-RaysR. de L. KronigJournal of the Optical Society of America 12 (6): 547–557, 1926
  34. La diffusion de la lumière par les atomesH. A. KramersAtti del Congresso Internazionale dei Fisici, Como, 2: 545–557, 1927
  35. Teflon™ AF Amorphous Fluoroplastic Resins: Product InformationChemoursproduct information
  36. N-BK7 Data SheetSCHOTToptical glass data sheet
  37. N-SF5 Data SheetSCHOTToptical glass data sheet
  38. HPFS® Fused Silica Product BrochureCorningproduct brochure, 2022