Issue 002 methodology appendix

The Sample Size of One: Why founders can't measure what worked

Every experimental figure in Issue 002 comes from controlled runs of the BayesBrain measurement engine against synthetic data with known planted ground truth. This appendix gives the sample frames, generator parameters, planted effect sizes, run counts, test definitions, thresholds, exclusions, and the file each published number came from. The per-number map is in the source ledger that accompanies this issue.

---

1. What kind of evidence this is

Everything measured for this issue is a simulation. The harness generates synthetic series, plants effects of known size at known dates, calls the production detection and measurement code, and scores the output against the truth it planted. It reimplements no statistics of its own.

This design can establish that an estimator recovers, or fails to recover, a known truth inside a world we wrote. It cannot establish that any real company's metrics behave like our archetypes, that real interventions produce effects of the shapes we plant, or that acting on any of this improves outcomes. A simulation that finds its own planted effects is a closed loop, and every number in the issue inherits that boundary.

Results are reported as measured, including the ones that argue against the issue's thesis. Section 8 lists those explicitly.

---

2. Runs and sample frames

Master seed for the validation study: 20260712. Mode: full.

---

3. Generator parameters

Archetypes. Five synthetic daily shapes: a bounded rate (logit random walk), compound growth (the MRR shape), a weekly seasonal series, an AR(1) series at persistence 0.0, 0.3, 0.6 and 0.9, and pure noise.

Planted events. Trend break, level shift, or transient anomaly, injected at a known day at a known size. Detection sweep: sizes swept from 5% to 60% of series level. Measurement runs: planted relative effects of 5%, 10%, 20% and 30%; the control-series run used 5%, 10% and 20% plus a placebo cell.

Measurement windows. Pre-period 120 points, measurement horizon 30 days.

Metric types in the measurement runs. Two: a compounding level on the MRR pattern (multiplicative scale, fitted on a log scale with a local linear trend) and a bounded mean-reverting rate (additive scale, fitted with a local level model).

Placebo cells. A quiet date with no planted effect, run through the identical pipeline.

Cadence runs. AR(1) noise generator with no trend for the false-positive cells; a planted linear slope expressed in standard deviations per observation for the power cells (0.05, 0.10 and 0.20 sd/obs at the reported cells). Persistence values 0.0, 0.5 and 0.8. Sample sizes 24, 90 and 180.

---

4. Tests, thresholds and decision rules

The 1% practical-significance floor was chosen on principle, as a "we do not report sub-1% moves" threshold, and not tuned to hit a target false-positive rate. The alternative candidates and their measured rates are reported in the source document rather than suppressed: no floor gives 40.8%, a 0.5% floor gives about 5.0%, the shipped 1% floor gives 0.83%, and an interval-inflation alternative reaches about 5% but with a factor fitted to this simulation's autocorrelation, which is why it was rejected.

---

5. Two false-positive tables, measured under different rules

The issue quotes trend false-positive rates from the cadence runs (34.7% at 24 points and high persistence). A second measured table in the integration research reports 16.2% for the same sample size and persistence. Both are correct and they are not the same measurement:

• The cadence table applies Mann-Kendall at a nominal alpha of 0.05, which is the test in isolation.
• The integration table applies the detector's full production gate: p <= 0.01 and an effect-size floor of absolute tau >= 0.25.

The floor is what does the work. The same research measured that the floor rescues the daily case (16.5% down to 0.8% at 365 points) and cannot rescue the low-n case, because with few points a large tau arises by chance. Issue 002 quotes the cadence table because the article's claim is about the trend test founders and tools actually apply, and it names the alpha inline.

---

6. Exclusions: numbers deliberately not published

No single-fit magnitude from a multiplicative-scale fit is quoted anywhere in the issue. A variance study fit the same cells 20 times each on the same data, unseeded, 120 fits total. Verdicts never flipped (0 of 120, which bounds the per-fit flip probability at roughly 2.5% by the rule of three), so every rate in this issue stands. Individual magnitudes on the log path did not: the near-threshold cell's point estimate spanned 12.7 percentage points across repeats of the identical fit, with run-to-run standard deviation of 0.051 on the estimate and 0.72 on the upper interval bound. The additive path was effectively deterministic (standard deviation at or below 0.0003).

Consequently the issue publishes aggregate rates, power fractions with their counts, and the magnitude ratios that were computed as medians across cells. It does not publish any individual fit's effect size, interval width, or E-value from a compounding-metric fit.

Also excluded: the E-value distribution (median 1.68 across 46 detected effects) is measured and sound as an aggregate but was left out of the article for space; it is in the ledger. The belief-store scenarios are used for one figure only (the +7.34% recovery case) and the rest are not published here.

Production note. The runs in this issue predate a change on 2026-07-27 that seeds the production fit from the measurement's identity, so a re-run of the same measurement is now bit-identical. The numbers are deliberately left as measured rather than quietly re-run.

---

7. Reproducibility, and one gap

The validation study regenerates with python scripts/run_validation.py --full from the recorded seed, writing scripts/validation/artifacts/results.json. The estimator-variance study regenerates with services/api/scripts/estimator_variance_study.py. The monthly-granularity facts are executable assertions in services/api/tests/test_monthly_granularity.py, including the anomaly ceiling, which the test verifies both algebraically and empirically.

The gap, disclosed: the cadence calibration document names scripts/cadence_calibration.py as its regeneration script, and that script is not present in the repository. Its figures are recorded in docs/DETECTOR-CADENCE.md with their date (2026-08-14), trial counts and parameters, and Issue 002 quotes them from that document, but they cannot currently be re-run from source the way the other runs can. Every number sourced only to that document is flagged in the ledger.

---

8. Where the data complicates the thesis

Reported here rather than smoothed into the article.

1. Not everything is unmeasurable. On a mean-reverting daily rate, single-company measurement detected every planted effect from 5% to 30% (10 of 10 per cell) and sized them within 3% of truth. The thesis is scoped to compounding levels, small effects, and low-cadence series, and the article says so.
2. The interval is the weak point even where power is fine. That same path's 95% intervals covered the true null 59.2% of the time (n=120). The shipped floor cuts the visible false-positive rate to 0.83% without repairing coverage.
3. "Monthly is underpowered" is false as usually stated. 24 monthly points detect a smaller annual drift (8% per year at 5% noise) than 90 daily points do (30% per year). The blocker is the false-positive rate under persistence and the arithmetic ceilings, not the sample count.
4. The control-series result is not a product. Power gains of 0.00 to 0.70 come with a 30% placebo false-positive rate, which is 6 of 20 and carries wide uncertainty at that sample size.
5. Small per-cell samples. Power cells in the core run are n=10; control-series cells are n=20. The issue prints counts rather than percentages wherever the denominator is 10 or 20, for that reason.
6. Vendor restatement windows are not re-verified. Integration research recorded per-source restatement windows from vendor documentation. Those vendor claims were not re-verified for this issue and no vendor-specific window is quoted; the article makes only the general claim that integrated sources restate, which is supported by the schema-level finding that a restatement overwrites history in place with no audit trail.

---

9. What this issue does not claim

• No claim that any real company's measurements are wrong.
• No claim that the synthetic archetypes resemble any particular company's metrics.
• No claim that the planted effects resemble the effects real interventions produce.
• No claim that cross-company control series, as measured here, are ready to be trusted. The naive version's 30% placebo rate is the reason.
• No prediction. Every rate describes the estimator we ran, on the generator we wrote, at the parameters listed above.

Questions about a specific number, or a discrepancy against your own runs: reach out through the survey link in the article closing.