Comoving Systems
← AI enablement for research

Research · Monte Carlo example

Could this rise occur by chance?

Synthetic laboratory record

A reference reading starts to climb.

A laboratory measures a reference standard each week to check for instrument drift. This synthetic record has 44 readings over 48 weeks, expressed in baseline standard deviations (SD). Nearby readings tend to move together, even when the instrument is stable.

Weekly readings of the synthetic lab reference standard Deviations from the reference standard’s baseline mean over 48 calendar weeks. Weeks 8, 19, 33 and 42 have no reading. The dashed line is the fitted trend in the strongest 12-week window, weeks 15–26. 1 12 24 36 48 −2 0 2 4 Deviation (baseline SD) Calendar week Weeks 15–26
The trace breaks at missing weeks. The dashed line fits the 11 readings from weeks 15–26. Its slope is 0.448 baseline SD per week.

The protocol specifies an upward trend over 12 calendar weeks. With missing readings, selecting 12 consecutive rows would silently extend some windows. The script keeps the original dates and tests all 37 windows. Weeks 15–26 show the strongest standardized rise.

Monte Carlo test

How often does noise produce a rise this strong?

19 of 9,999

The simulation generates 9,999 records under the no-drift model and searches all 37 windows in each record. In 19 records, the strongest rise scores at least as high as the one shown here (3.74, rounded).

Distribution of the maximum standardized slope under the no-trend model Count per half-unit bin, from a maximum standardized slope of 0 to 4.5: 3, 476, 2,592, 3,585, 2,231, 825, 230, 49 and 8. The dashed rule marks the observed score of 3.74. Nineteen of 9,999 simulated maxima reached or exceeded the unrounded score. 0 1 2 3 4 1 10 100 1,000 10,000 Count per bin (log scale) Maximum standardized slope Observed 3.74
Each step spans a half-unit bin. The dashed rule marks the example’s maximum. The exceedance count uses the individual simulated maxima before binning.

A single window chosen in advance would give a one-sided p-value of 0.000091 at this score. Here the scan selected the window, so the relevant Monte Carlo p-value is 0.002. The model that generated this unusual record contains no drift.

Comoving could give a lab’s AI agent access to the approved protocol, dated readings and instrument notes. The agent would quote the rule for missing weeks, flag any conflict with the notes and show the researcher the proposed scan settings. Once approved, those settings go to the same analysis script the lab can run on its own.

A second analyst can reproduce the winning window from the dated readings and approved protocol. A fresh simulation may give a different exceedance count. The researcher decides whether to repeat the measurement.

Method and assumptions

Record. The no-drift model is a stationary Gaussian AR(1) process with mean zero, unit variance and a known one-week correlation of 0.5. The simulation generates all 48 weeks and omits weeks 8, 19, 33 and 42 from every fit.

Scan. Fit an ordinary least-squares line with an intercept in each window. The correlation between readings k weeks apart is 0.5k. Using that covariance, the slope in weeks 15–26 has a null standard deviation of 0.1196 baseline SD per week. Dividing each slope by its own null standard deviation gives a standard normal score for any fixed window. The scan takes the largest score.

Calculation. For a single record drawn under the specified model, its maximum and the independent simulated maxima are exchangeable. Including the observed record gives its upper-tail rank among 10,000 records. This rank accounts for choosing a window within the illustrated record. Selecting the most striking of several records would require calibrating that additional search.

pscan=19+19,999+1=0.002

At the observed threshold, 19 exceedances in 9,999 independent simulations give a 95% Clopper–Pearson interval of 0.0011–0.0030 for the model’s tail probability. This interval measures Monte Carlo uncertainty conditional on the threshold and the model.

Applying the test. Check the noise model on independent baseline data. If its parameters are estimated, the calibration must account for that uncertainty. If missingness depends on the readings, simulate that mechanism. Any search across additional directions, window lengths or records belongs in the simulated procedure.

Run python reproduce.py with Python 3 to regenerate the readings, fitted slope, 37 window scores, histogram counts and interval. The script uses fixed random seeds and the standard library.

Download the Python simulation

Statistical methods and a published application

Calibrating a scan. Kulldorff compares the strongest candidate cluster with the strongest in each simulated dataset. We use the same principle to search over time windows. Kulldorff (1997), A spatial scan statistic

Ranking the observed result. Hope describes significance tests that compare an observed statistic with draws from a specified null model. Hope (1968), A Simplified Monte Carlo Significance Test Procedure

Finite simulation counts. Phipson and Smyth explain how ranks account for the observed statistic in Monte Carlo and random permutation tests. Phipson and Smyth (2010), Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn

Astronomy. To test a correlation between fast radio bursts and galaxies, Rafiei-Ravandi and colleagues randomized burst right ascensions while holding declinations fixed. Their significance calculation accounted for the search over angular scale and galaxy redshift cuts. Rafiei-Ravandi et al. (2021), CHIME/FRB Catalog 1 Results: Statistical Cross-correlations with Large-scale Structure

Particle physics. Gross and Vitells estimate the look-elsewhere effect in resonance searches using an asymptotic approximation. Gross and Vitells (2010), Trial factors for the look elsewhere effect in high energy physics

Discuss a research workflow