Research · Monte Carlo example
Could this rise occur by chance?
Synthetic laboratory record
A reference reading starts to climb.
A laboratory measures a reference standard each week to check for instrument drift. This synthetic record has 44 readings over 48 weeks, expressed in baseline standard deviations (SD). Nearby readings tend to move together, even when the instrument is stable.
The protocol specifies an upward trend over 12 calendar weeks. With missing readings, selecting 12 consecutive rows would silently extend some windows. The script keeps the original dates and tests all 37 windows. Weeks 15–26 show the strongest standardized rise.
Monte Carlo test
How often does noise produce a rise this strong?
19 of 9,999
The simulation generates 9,999 records under the no-drift model and searches all 37 windows in each record. In 19 records, the strongest rise scores at least as high as the one shown here (3.74, rounded).
A single window chosen in advance would give a one-sided -value of 0.000091 at this score. Here the scan selected the window, so the relevant Monte Carlo -value is 0.002. The model that generated this unusual record contains no drift.
Comoving could give a lab’s AI agent access to the approved protocol, dated readings and instrument notes. The agent would quote the rule for missing weeks, flag any conflict with the notes and show the researcher the proposed scan settings. Once approved, those settings go to the same analysis script the lab can run on its own.
A second analyst can reproduce the winning window from the dated readings and approved protocol. A fresh simulation may give a different exceedance count. The researcher decides whether to repeat the measurement.
Method and assumptions
Record. The no-drift model is a stationary Gaussian AR(1) process with mean zero, unit variance and a known one-week correlation of 0.5. The simulation generates all 48 weeks and omits weeks 8, 19, 33 and 42 from every fit.
Scan. Fit an ordinary least-squares line with an intercept in each window. The correlation between readings weeks apart is . Using that covariance, the slope in weeks 15–26 has a null standard deviation of 0.1196 baseline SD per week. Dividing each slope by its own null standard deviation gives a standard normal score for any fixed window. The scan takes the largest score.
Calculation. For a single record drawn under the specified model, its maximum and the independent simulated maxima are exchangeable. Including the observed record gives its upper-tail rank among 10,000 records. This rank accounts for choosing a window within the illustrated record. Selecting the most striking of several records would require calibrating that additional search.
At the observed threshold, 19 exceedances in 9,999 independent simulations give a 95% Clopper–Pearson interval of 0.0011–0.0030 for the model’s tail probability. This interval measures Monte Carlo uncertainty conditional on the threshold and the model.
Applying the test. Check the noise model on independent baseline data. If its parameters are estimated, the calibration must account for that uncertainty. If missingness depends on the readings, simulate that mechanism. Any search across additional directions, window lengths or records belongs in the simulated procedure.
Run python reproduce.py with Python 3 to regenerate the readings, fitted slope, 37 window scores, histogram counts and interval. The script uses fixed random seeds and the standard library.
Statistical methods and a published application
Calibrating a scan. Kulldorff compares the strongest candidate cluster with the strongest in each simulated dataset. We use the same principle to search over time windows. Kulldorff (1997), A spatial scan statistic
Ranking the observed result. Hope describes significance tests that compare an observed statistic with draws from a specified null model. Hope (1968), A Simplified Monte Carlo Significance Test Procedure
Finite simulation counts. Phipson and Smyth explain how ranks account for the observed statistic in Monte Carlo and random permutation tests. Phipson and Smyth (2010), Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn
Astronomy. To test a correlation between fast radio bursts and galaxies, Rafiei-Ravandi and colleagues randomized burst right ascensions while holding declinations fixed. Their significance calculation accounted for the search over angular scale and galaxy redshift cuts. Rafiei-Ravandi et al. (2021), CHIME/FRB Catalog 1 Results: Statistical Cross-correlations with Large-scale Structure
Particle physics. Gross and Vitells estimate the look-elsewhere effect in resonance searches using an asymptotic approximation. Gross and Vitells (2010), Trial factors for the look elsewhere effect in high energy physics