forecast) surfaces forecasts produced by WorldMonitor’s forecasting pipeline. Each forecast is a structured prediction with a probability, a domain classification, a time horizon, and supporting signals; the panel exposes two axes of filtering (domain and macro-region) so you can home in on the slice that matters to you.
Reading a forecast
- The percentage is the chance that the forecast’s event happens within its time horizon. Fixed signal rules set it, and it can be blended with the price of a matching prediction market. No LLM chooses the number. For state-derived market and supply-chain forecasts, one input to the rules comes from an LLM that extracts signals from the news (issue #4963). An LLM also writes the scenario and case-file text.
- Every forecast is checked. A forecast is published with the rule that will decide it. If a rule is corrected while the forecast is still pending, the original is kept on the record. After the deadline, the resolver marks the forecast YES, NO or VOID and keeps the evidence.
- VOID means the forecast could not be checked fairly, for example because the feed that decides it could not be read or the evidence could not answer the question. A VOID is not scored as a hit or a miss. The share of VOIDs is published beside the scores.
- The track record is on /accuracy/. It has been under audit since 2026-10-07, and its scores are withdrawn until the record holds at least 30 forecast families, with at least 5 that came true and 5 that did not. A family is one forecast id, so a forecast seen in many runs counts once. Once the minimum is met on a fresh scorecard, the audit lifts without a code change, and /accuracy/ follows at its next weekly refresh. Until then there is no measured skill, so read a percentage as the estimate of the signal rules, not as proven odds.
- Empty filters. Cyber forecasts are withheld until a check exists that could decide them. Political forecasts have so far come only from prediction markets, which are withheld for the same reason, so the Political filter is usually empty. No detector has produced infrastructure forecasts since issue #5334, so the Infra filter is empty. See Resolution & track record below.
What the panel shows
A list of active forecasts, filtered by two pill rows:Domain filter
All / Conflict / Market / Supply Chain / Political / Military / Cyber / Infra
The Cyber and Political filters can be empty while cyber and prediction-market forecasts are withheld from publication (see Resolution & track record below). Every published political forecast came from prediction markets, so the Political filter stays empty until political forecasts have a question that can be checked.
Region filter
All Regions / MENA / East Asia / Europe / South Asia / Africa / LatAm / N. America
Forecasts whose region does not classify (unknown or global) only appear under All Regions. The panel’s minimum-probability floor is 0.1 (10%) — predictions below that threshold are suppressed from the panel view.
Each forecast row shows the prediction text, the event probability, the domain (color-coded), the time horizon, and a compact provenance trail. Deep-simulation path confidence is shown separately as confidence, not as event probability.
Calibration & Projection Notes
Forecast probabilities come from deterministic, rule-based signal detectors inscripts/seed-forecasts.mjs, with an optional prediction-market blend when a matching market anchor exists. LLM calls do not set the numeric probability. They generate the narrative scenario, case-file, branches, and perspective text around the already-scored forecast. The default source-backed LLM routes in the seeder are OpenRouter deepseek/deepseek-v4-flash (primary, reasoning disabled), then fixed free OpenRouter fallbacks google/gemma-4-26b-a4b-it:free and nvidia/nemotron-3-super-120b-a12b:free. One tracked exception: the critical-signals extraction stage contributes signal strength/confidence into the deterministic scoring of state-derived (market/supply-chain) forecasts, so that stage stays pinned to OpenRouter google/gemini-2.5-flash (see issue #4963).
Forecast probabilities are market-calibrated only when the seeder can match the forecast to a prediction-market anchor that asks the same kind of question. In those cases the calibration object includes the market title, market price, drift, source, the pre-blend internalProbability, and the post-blend marketBlendedProbability, and the displayed probability is blended with that market anchor. Forecasts with calibration: null are internal/legacy estimates derived from WorldMonitor signal rules rather than externally market-calibrated probabilities.
The seeder also computes horizon projections (24h, 7d, 30d) for each forecast. They are not probabilities, and they are not published: payloads generated after 2026-10-07 (issue #8967) omit the projections field, so the API and MCP stop returning it once the first such run replaces the cached forecasts. The seeder keeps them only in its internal forecast history, where horizon scoring reads them. Each value comes from a fixed, hand-set domain curve in scripts/seed-forecasts.mjs (computeProjections). The forecast probability is divided by the curve’s multiplier at an anchor horizon, multiplied by the multiplier for the target horizon, and then clamped to a 1% floor and 95% cap (0.01 / 0.95). The anchor is the forecast’s own emitted horizon, except for market forecasts, which anchor on the curve’s strongest multiplier so a 30-day market emission does not inflate the 24-hour value. For a forecast whose own resolution spec is a point-in-time hard contract (at-deadline, today the supply-chain chokepoint and GPS-jamming families), each projection other than the forecast’s own horizon gets a resolution window in the outcome ledger, keyed <id>@<deadline>@<horizon>, frozen at the probability it was first emitted with, and read from the nearest feed sample inside the stored tolerance (one resolver cycle, capped at half the horizon) around the horizon deadline. The resolutions seeder reports them per horizon in a separate projections section of the stored scorecard, never pooled into the headline. Each horizon row counts registered, matured, scored, UNOBSERVED and VOID windows and the forecast families behind the scored ones, and gives the realized YES rate with a 95% Wilson interval. The Wilson interval treats each window as independent, and one forecast can hold several windows, so read it beside the family count. The Brier score is withheld until a horizon reaches the minimum sample. The same rows are repeated per domain (byDomain) and per projection-curve version (byCurvesVersion). A horizon window keeps only the parent’s domain, region, title, origin and state bucket. The parent’s calibration, uncalibrated probability, base rate, probability source, ensemble passes and market slug describe the parent’s probability, so the window does not copy them, and each resolver run removes them from windows registered before this rule in the live ledger. Receipts already archived keep them. Every other projection (judged, within-horizon, or settlement-dated specs) is unscored with a stated reason. Each published forecast’s scoredHorizons field names the horizons that have a registered scoring window. The projection values and scoring windows are not published. The per-horizon grades are published under the minimum-sample rule described below.
The panel names the horizons listed in scoredHorizons on an “Also scored at” line under the forecast title. The headline probability is graded at the forecast’s own horizon, so the line lists only the extra horizons. The panel shows no projection values, and the narrative prompt does not receive them. The prompt still carries the forecast probability, which can equal an anchor-horizon projection, so a matching percentage in scenario or case text is not evidence that a projection was used. Point-in-time horizon scoring shipped under issue #7075. Judged and cumulative horizons are tracked in issue #8969.
The scorecard also writes a public horizonGrades block (issue #9057). It has one row per projection-curve version and horizon, cut from the byCurvesVersion slices, so a grade never mixes curve versions. A row carries a Brier score with its family-bootstrap 95% interval, and the realized YES rate with a 95% Wilson interval, only when its scored windows meet the headline’s family minimums: at least 30 forecast families, with at least 5 YES families and 5 NO families. The scorecard and /accuracy/ both apply this rule through meetsFamilyOutcomeMinimums in scripts/_forecast-scorecard.mjs, the predicate behind the headline. Below the minimums, the row carries its window and family counts and measurable: false, and /accuracy/ prints “Not yet measurable” beside those counts. Windows without a version stamp are counted in unversionedScored and never graded. A projection is not a probability, so each grade scores the projection value as if it were one, and /accuracy/ says so and names the curve version beside every grade. The MCP get_forecast_scorecard tool serves the block. The REST endpoint does not, because the public OpenAPI document has no room left under its size cap. The weekly /accuracy/ freeze reads the block through the MCP tool on the canonical MCP host (https://worldmonitor.app/mcp; the www alias answers 410) with its service key. If that read fails, the freeze keeps an earlier read of the same scoring run, when one exists, and logs a warning. The page and its scorecard.json download use the block only when its generatedAt equals the REST scorecard’s. The MCP tool and the page both check the minimums again from each row’s counts instead of trusting measurable. While the accuracy audit is open, the page shows no grades, like every other score, and the download keeps the rows marked as under audit.
The GET /api/forecast/v1/get-forecasts response includes degraded, stale, and error fields so clients can distinguish a backend/cache outage from a healthy empty forecast set.
Panel id is forecast; canonical component is src/components/ForecastPanel.ts.
How you reach it
- Cmd+K: type forecast, ai forecast, or predictions ai. The command palette label is “AI Forecasts” even though the internal panel id is
forecast. - Availability by variant: registered and enabled by default in the full/geopolitical variant only. Not present in the tech, finance, commodity, or happy variants. Source:
FULL_PANELSinsrc/config/panels.ts.
Data sources
Primary RPC:GET /api/forecast/v1/get-forecasts. The panel also consumes adjacent RPCs for the richer deep-simulation surface:
GET /api/forecast/v1/get-simulation-package— the most recent simulation inputs bundle.GET /api/forecast/v1/get-simulation-outcome— the most recent simulation result. Outcomes auto-refresh on the deep-forecast cron cadence (~hourly); each cycle enqueues a worker task after writing the simulation package.POST /api/forecast/v1/trigger-simulation— starts a fresh simulation run outside the cached cron output. Usually unnecessary — outcomes already auto-run each cron cycle.
forecast:predictions:v2 in Redis. Macro-region classification comes from shared/forecast-macro-regions.js.
Scoring and calibration
Published forecasts are capped and deduplicated before they reach the panel:
A market anchors a forecast only when all of these hold:
- The market title names the forecast’s own subject, read from the same subject table the judged resolver uses (
scripts/shared/judged-subject-terms.json, loaded byscripts/_forecast-subject.mjs). A country matches its names, demonyms and places, so a Ukraine forecast can anchor to a market about Odesa. A region matches its own terms (“Bab el-Mandeb” for the Red Sea, “Hormuz” for the Persian Gulf, “Baltic” for Northern Europe) and only the member countries its label names (“Israel/Gaza”, “Iran Theater”). A market about one Middle East country asks a narrower question than a Middle East forecast, so it does not anchor one; the judged resolver still reads news about any member country as evidence about the region. Terms match as whole words (“Baltic” never matches “Baltimore”), a name that is also a common word (“Jordan”, “Georgia”) counts only beside one of its co-terms, and excluded phrases (“Odessa, Texas”) never count. A shared macro-region tag or a neighbouring country from the entity graph does not count, so a Syria forecast does not anchor to a market about Iran (issue #9010). - The market title resolves on the event class that
MARKET_ANCHOR_EVENT_CLASSESinscripts/seed-forecasts.mjsdeclares for the forecast’s domain. Conflict escalation forecasts need a strike, attack, invasion, offensive, clash, or war market. A trade war or tariff war does not count. De-escalation forecasts need a ceasefire, truce, or peace-deal market. Political instability forecasts need a resignation, ouster, impeachment, coup, snap election, or unrest market; a market that only asks who will hold an office does not qualify. Supply-chain forecasts need a strait, canal, blockade, shipping, or transit market. Infrastructure forecasts need an outage, blackout, grid, or pipeline market. Market forecasts need a commodity or macro price market that also shares a title term with the forecast. - The market settles inside the forecast’s own window. The forecast deadline is emission time plus its horizon (
24h,7d, or30d). The market’sendDatemust fall between emission and the deadline plus the larger of 7 days and the horizon (MARKET_ANCHOR_MIN_SETTLEMENT_SLACK_MS). A 7-day forecast therefore accepts a market settling within 14 days of emission, and a 30-day forecast accepts one settling within 60 days. A market that settles much later prices risk over a longer period than the forecast claims, so its price is not an estimate of the same event. A market with no settlement date does not anchor. On the 2026-10-06 market universe every same-subject, same-event-class market settled at the end of 2026 or later, so no forecast was market-calibrated.
prediction_market signal) gets no anchor: it already asks its own market’s question, and any other market on its subject asks a different one. A domain or title family without a registry entry gets no anchor. Cyber, military, and GPS interference forecasts are excluded this way. A cyber forecast asks about threat activity in its country, and no market asks that question. A military forecast asserts an observed force posture or an airlift or air-activity surge, which no market prices; a strike or invasion market is at best a sub-event of it. The existing price, liquidity (volume >= 5,000), YES-direction, and region-tag filters still apply. This rule exists because region overlap alone let unrelated long-dated markets, such as a Taiwan invasion market or a presidential succession market, calibrate cyber-concentration forecasts (issue #7071).
When a market anchor applies, the final probability is blended as 0.4 * market_probability + 0.6 * internal_probability and then capped at the domain ceiling. The calibration object records the market title, market price, source, drift (internal probability minus market price), internalProbability (the pre-blend value), and marketBlendedProbability (the post-blend, post-cap value, equal to the published probability). The history snapshots and the resolution ledger carry the same fields, so a later audit can recompute the blend without reconstructing it from drift.
Market-bucket scenario calibration is an editorial calibration layer in scripts/seed-forecasts.mjs, not a learned market model. It biases how market-context simulation states contribute to forecast pressure and confidence so that direct energy/freight transmission receives more lift, broad macro buckets stay moderate, and defense repricing is damped unless the evidence is strong.
The conflict and UCDP conflict-zone rows are base detector caps. UCDP conflict-zone counts begin at the 10-event publish gate with a
0.35 base probability and ramp to the 0.85 base cap at 100 events. When the matching EMA risk score has velocitySpike, the seeder adds a +0.08 probability override after the base cap and clamps the result to 0.99.
Defense state calibration has additional direct-confirmation terms outside the table: each unit of direct defense_repricing confirmation adds +0.12 pressure and +0.08 confidence. When that direct defense confirmation is absent, pressure subtracts the table-driven dampener (0.12) and confidence subtracts a separate 0.04 absence penalty.
The unscored horizon projections are expanded to 24h, 7d, and 30d with these hand-set domain curves. The multipliers are editorial priors, not fitted values. Market projections use the curve’s peak multiplier as the anchor; other domains use the forecast’s emitted horizon:
The curves, the anchor rule, the floor and the cap together carry a version,
PROJECTION_CURVES_VERSION (currently 1). Each projected forecast and each horizon window it registers is stamped with it, and the scorecard reports each version apart, so a later curve change is not pooled with results from the old curves. A window registered from a history emission that carries no version stamp has no version and falls in its own slice (curvesVersion: null). That covers windows registered before the stamp shipped and windows registered after the deploy from older history emissions that lack it. The curves did not change between the start of horizon scoring on 2026-10-06 and the stamp.
Trend is serialized as a string, not an enum. The current values are rising, falling, and stable, based on a +/- 0.05 probability delta versus the prior forecast snapshot.
Shadow bet-engine evaluation lane
World Monitor also runs a shadow-only forecasting lane atforecast:bets:history:v1. It never writes forecast:predictions:v2, so its bets do not appear in the AI Forecasts panel. The purpose is to measure candidate generation methods against resolved outcomes before allowing them into the user-facing forecast stream.
The lane now includes a dedicated geopolitical prediction-market family:
- It selects liquid geopolitical markets whose settlement is 2–210 days away. The general market family retains the shorter 2–45 day window.
- The two families form a disjoint partition by market slug, so the same venue market cannot become two separate bets.
- Geopolitical candidates rank first for the optional ensemble stage, but the attempt budget reserves room for faster-resolving energy, commodity, general-market, and macro families.
- Identity and settlement use the venue slug rather than the mutable title. The hard resolution spec reads the settlement feed at the market deadline and resolves whether the YES price crossed 50%.
- A Polymarket event can list several markets in turn under one slug. When one closes, the bootstrap feed carries the event’s next market, with its own title and end date. A market bet’s id is the slug, so the resolver also keys a market-settlement window on its market’s title. The slug’s next market is its own question: it opens its own window and never joins, or moves the deadline of, the earlier market’s window. An emission of the same market joins its window, and its end date is how a pending window follows the venue’s close.
- The settlement loader records one result per market, keyed by slug and title, and a window reads only the record whose market title matches its question. A record for another market of the slug never grades it. Titles compare after case, spacing, curly apostrophes and a trailing
?are normalized; a different year is a different market. The bootstrap feed carries no market id, so a Polymarket market retitled beyond that reads as a new market: its later bets open their own window, and the first window keeps its last deadline. A Kalshi ticker is one market, so Kalshi bets skip the title check. - Results remain tagged
generationOrigin: "bet_engine"so the scorecard can compare that lane with the canonical forecast generator instead of mixing their records.
baselineProbability. When FORECAST_BETS_ENSEMBLE=1 is explicitly enabled, a bounded three-pass LLM ensemble may replace the working probability for top-ranked new attempts while leaving the baseline attached for Brier-score comparison. The ensemble is off by default; partial passes are labeled ensemble_partial and are eligible for a later upgrade rather than being treated as a completed ensemble. A bet that keeps the base rate, because it fell outside the top K or the budget or its ensemble call failed, carries probabilitySource: 'base_rate'. That base rate is a placeholder, not a forecast, so the bet opens no ledger window and is never scored. Its question stays open, so a later run can ensemble it, and the window opens on that run’s probability. With the ensemble off, the lane scores nothing.
This separation is deliberate: prediction-market liquidity is useful calibration evidence, but shipping a new family directly into the panel before it accumulates settlement history would confuse implementation completeness with demonstrated forecast skill.
Resolution spec
Every published forecast carries a resolution spec — a machine-checkable contract that defines what “came true” means at resolution time. The spec is eitherhard (auto-resolvable by comparing a metric from the same WorldMonitor feed the detector scored from) or judged (a resolution question for a later LLM judge, still with a hard deadline). The deadline is always present, encoded in epoch milliseconds: emission time plus the forecast’s stated horizon — except prediction-market forecasts, which resolve at the market’s own end date when it has one.
Hard specs apply to forecasts with clean metric backing:
- Conflict: UCDP conflict-event counts in the forecast’s region.
- UCDP zones: UCDP conflict-zone event counts.
- Market: commodity-future price moves measured from the emission-time baseline.
- Prediction-market: the spec still carries the market’s end date, but the resolver seals it as VOID with reason
market_price_not_outcome. Its feed,prediction:markets-bootstrap:v1, lists only open markets and clipsyesPriceto 10 to 90, so a read at the end date is the crowd’s price, not the outcome (issue #5233). These forecasts are no longer published (issue #8990). - Supply chain: the chokepoint’s disruption score on the deadline reading. A route whose standing threat score is below the feed’s red boundary resolves YES at a score of at least 50 (
CHOKEPOINT_DISRUPTED_MIN_SCORE, the boundaryscoreToStatuscalls red). The supply-chain detector emits at that boundary, recorded asrule: disruptedwith aruleVersion. A state-derived freight forecast attaches the same contract at any score, on the most disrupted chokepoint in its sea. Hormuz and Kerch carry a fixed war-zone base of 70, which is already red, so a reading cannot fall through 50. Their contract is a score above that base (rule: above_fixed_base). Pending rows move to the current contract before they resolve and keep the earlier threshold assupersededThreshold. Resolved rows keep the threshold they were graded on (issues #8990 and #9033). - GPS interference: the number of gpsjam.org hexes inside the shipping-zone box on the snapshot dated on or after the deadline day. The forecast resolves YES when the zone still holds at least 3 hexes, the same floor at which the detector emits it. The detector emits a zone only while it holds 9 hexes or fewer, at a probability of 0.58. Over 120 days of snapshots, zones in that range still held the floor on the day-7 snapshot in 32 of 55 windows, while the Eastern Mediterranean, Persian Gulf, Black Sea and Baltic never held fewer than 28 hexes and never fell to the floor, so a forecast there could not resolve NO (issue #9012). The spec records this as
rule: persistencewith aruleVersion. Pending rows written under the earlier emission-count threshold move to this rule before they resolve, and keep the old value assupersededThreshold(issue #8990). - Infrastructure (legacy rows only): outage events on the retired
infra:outages:v1feed. No detector has emitted infrastructure forecasts since issue #5334, and new infrastructure specs would be judged.
- Political: regime stability, negotiations, policy shifts.
- Military: posture transitions, deployment patterns.
- Cyber: attribution and impact severity. Cyber forecasts are no longer published (issue #8990).
- State-derived forecasts whose bucket has no feed metric at emission. Energy forecasts resolve on the
CL=Fprice, and supply-chain freight forecasts resolve on a chokepoint score. - Hard-family forecasts that cannot derive a finite threshold despite their domain (e.g., a market forecast with no tradeable anchor).
Emission-time extraction gate
Each hourly run dry-runs the resolver’s extractor on every published hard spec, against the same resolver-shaped feed the resolver will read (issue #7067). The run logs[ExtractionGate] shadow counters per outcome, family and domain, and a would_downgrade line for each spec whose metric cannot be extracted. The counter line names the deploy revision, so counts from before and after a spec change (the GPS-jamming changes in #9003 and #9047) can be read apart. In shadow the gate changes no spec.
Enforcement is off. The switch is the constant EXTRACTION_GATE_ENFORCED in scripts/_forecast-resolution.mjs. A reviewed change flips it only after the shadow cohort from 7 daily runs is posted on #7067 and the judged lane has stayed below 20 pending entries without growing for 7 runs. When it is on:
- Every published spec records
specOrigin:hard,judgedorhard_downgraded_unextractable. A hard spec also records its dispatch family asspecFamily. Ledger rows copyspecOrigin. Rows without it are legacy. - A hard spec whose metric cannot be extracted becomes a judged spec with
specOrigin: hard_downgraded_unextractable. It keepsoriginalFamily,originalMetricKey,originalSourceFeedanddowngradeReason, and its hard horizon contracts become unscored (parent_unextractable). The downgrade runs only while the stored scorecard is at most 48 hours old (EXTRACTION_GATE_SCORECARD_MAX_AGE_MS, the calibration gate’s bar on the same key), its judged lane holds fewer than 20 pending entries (EXTRACTION_GATE_MAX_PENDING_JUDGE), and the lane scores at least half of its instrumented resolutions within SLA (EXTRACTION_GATE_MIN_SCORED_WITHIN_SLA_RATE, 0.5). Below half, a downgraded row is more likely to end VOID or late than scored on time, which moves the hard-path failure into another lane. Otherwise the spec stays hard and the run logs the reason. - A hard spec and its downgrade are one question in the ledger. The extraction verdict follows the hourly feed snapshot, so one forecast can pass, fail and pass again. Whichever kind opens the window first keeps it, and emissions of the other kind count as sightings of that window. Two downgrades of one forecast are also one question when the same metric failed, even if the title or region in the judged question changed. A hard emission absorbed by a judged window registers no horizon windows. A ledger that already holds both keeps the earlier one and voids the other as
duplicate_window. - A judged spec whose only question would be the generic
Will "<title>" resolve YEStemplate (market, military and supply-chain forecasts) becomesunscoredwith reasongeneric_judged_question. The resolver opens no window for an unscored spec, so it never reaches the judged lane. The rule also applies to a downgraded spec, so a downgraded GPS-jamming or chokepoint forecast is withheld until its domain has a specific question.
specOrigins block, which the public endpoint does not serve. For each origin and hard family it reports entries, resolved, scored, VOID, pending, scored within SLA and median resolution latency. Every row is held to the judged-lane clock (deadline plus 18 hours plus 2 days), so a downgraded family is compared with its own hard path. The family is originalFamily or specFamily when the row stores one. A legacy row’s family is read back from the exact feed its spec read, and every market-input feed counts as market. Rows on the retired infrastructure outage feed (infra:outages:v1, issue #5334) and bet-engine rows are counted apart and excluded.
A withheld forecast is still published but opens no ledger window, so it would otherwise drop out of the record and lower the VOID rate by deletion. The stored scorecard’s internal withheldAtEmission block counts them the way the ledger counts windows: an hourly re-emission inside the span of an earlier counted emission of the same id is the same forecast. It counts each forecast once, by reason and by the hard family it lost (or its domain), over the forecast history the resolver read, whose oldest snapshot is since.
Refresh cadence
The seeder runs approximately hourly. The key is allowed up to 90 minutes inapi/health.js (maxStaleMin: 90) before the health surface escalates.
Resolution & track record
Published forecasts carry machine-checkable resolution specs. A daily resolver pre-registers those specs intoforecast:resolutions:v1, samples hard-source metrics until their deadlines, then resolves each published forecast to YES, NO, or VOID with evidence. The derived forecast:scorecard:v1 reports Brier score, log score, calibration buckets, VOID rate, the evaluation corpus described below, and market-anchor skill. Judged specs are tracked as pending for a fast-follow judge and are excluded from the first hard-source scorecard until resolved.
The VOID-share target is under 15% of resolved published forecasts (GO_FORWARD_VOID_SHARE_TARGET), measured only over windows first published on or after 2026-10-08 (GO_FORWARD_SINCE, issue #4930). A window is placed by its firstSeenAt, or by generatedAt when it has no sighting. The cohort is the ledger totals’ population, published forecasts only, cut to those windows: shadow bets (bet_engine) are left out until promoted, because users never see them; state-derived energy and freight forecasts are published and count. Earlier windows hold the voids relabelled in the issue #8990 audit, so the rolling VOID rate is not the target measure. The scorecard reports this cohort under goForward: entries, resolved, voids, the void share with its 95% Wilson interval, and voids by reason. The public endpoint does not serve that block. The methodology states the share, and /accuracy/ prints the methodology under the ledger totals, or in its Methodology section while an audit withdraws the scores. While nothing in the cohort has resolved, the sentence says so instead of printing 0%. Below GO_FORWARD_MIN_RESOLVED (30) resolved forecasts, the same minimum as the skill score’s family gate, the sentence states the share but does not compare it with the target. Judged forecasts resolve days after hard ones and have voided more often, so early readings run low. Once 2026-10-08 is older than the 180-day window, forecasts resolved before the window start drop out of the count, and the sentence names the window.
The scorecard also reports skill against the actual rate: always forecasting how often the same forecasts came true in the window. The Brier skill score is 1 - Brier / p(1-p), where p is that rate, computed on exactly the same rows. Its 95% interval comes from a bootstrap that resamples whole forecast families, and /accuracy/ calls a cohort better or worse only when that interval lies wholly above or below 0. A forecast family is one forecast id; its windows share a question and an outcome, so a family counts once. That only approximates independence: on 2026-10-07, 42 of 603 question texts in the ledger appeared under more than one forecast id. A cohort or domain is scored once it has at least 30 forecast families, with at least 5 that came true and 5 that did not; a family with both outcomes counts on both sides. The calibration refit (#9027) needs 12 of each, because a fitted map must beat the identity map, while a score only has to be stable. Below the minimum the scorecard still reports the reading, flagged as a small sample, and publishes no domain score.
The public accuracy record has been under audit since 2026-10-07 (issue #8990). While an audit holds, /accuracy/, its scorecard.json download, llms-full.txt, the REST scorecard’s underAudit field, the forecast panel and the MCP forecast tools withdraw every score and show a dated notice. The standing audit (STANDING_ACCURACY_AUDIT in shared/forecast-accuracy-audit.js) lifts by itself once the scorecard reports the headline cohort measurable: at least 30 forecast families, with at least 5 that came true and 5 that did not. No constant is edited to lift it. Every surface calls forecastAccuracyAudit() on the scorecard it shows, so all of them agree for the same scorecard. The REST handler and MCP read the seeder’s skill.measurable. The public contract has no room for that field, so /accuracy/ reads the same verdict from the insufficientSample flag of the skill Brier interval in its frozen capture; the seeder computes both with the same minimums over the same rows. A missing, degraded or unreadable scorecard is not measurable, so the audit holds. A stale scorecard holds the audit even when it is measurable: the REST response marked stale (seed older than 36 hours), the MCP tools when the seed-meta:forecast:scorecard clock is older than 36 hours or unknown, and /accuracy/ when its capture is stale, failed, or retained from an earlier week. The weekly capture also keeps the live API’s underAudit, and /accuracy/ holds the audit while that flag is set, whatever its own reading says. /accuracy/, scorecard.json and llms-full.txt change state only when the weekly refresh PR that carries a new capture is merged. Without an override, the panel waits for the scorecard before it shows either the notice or the scores.
To hold an audit while the headline is measurable, for a future incident, set FORECAST_ACCURACY_AUDIT_OVERRIDE to a { since, issue, reason } record and rebuild the generated outputs. since must be a calendar date, issue a positive integer and reason non-empty; any other truthy value holds the standing notice instead. The override can only add an audit. No setting lifts the audit while the headline is not measurable. The scorecard methodology also states how many headline forecasts were published after a blend toward a market price chosen by the matcher used before issue #7071. Those rows are scored on what was published. The methodology sentence publishes the count; the structured preLineageAnchorCount field stays off the API, MCP and /accuracy/.
State-derived market forecasts in the sovereign_risk, rates_inflation and fx_stress buckets are not published, because no feed can check them (issue #5234). Their earlier rows stay in forecast:resolutions:v1, but the scorecard leaves them out of every figure it reports, including the VOID rate. The resolver seals their open rows as VOID with reason withheld_unpublished and calls no judge. State-derived energy and freight forecasts are still published. They resolve on a price or chokepoint feed when that feed has a matching value at emission, and go to the judges otherwise.
The same rule withholds 2 more families, which had no path to a score (issue #8990). Both are listed in WITHHELD_PUBLISH_FAMILIES in scripts/seed-forecasts.mjs:
- Cyber (every forecast in the
cyberdomain). WhileCYBER_JUDGING_HELDholds the judges, every cyber row is sealed as VOID with reasonjudged_evidence_unreliable. - Prediction-market (every forecast that carries a
prediction_marketsignal, which the prediction-market detector attaches). These read the crowd price onprediction:markets-bootstrap:v1, and every row is sealed as VOID with reasonmarket_price_not_outcome.
withheld_family in the publish telemetry. Selection fills its target from the other domains, and its target is computed without them. They do not reach forecast:predictions:v2, the panel, the API or MCP, so the resolver opens no new rows for them. Rows opened before the change resolve as before and stay in the scorecard as VOIDs until they leave the rolling window. To restore a family, remove it from WITHHELD_PUBLISH_FAMILIES in the same change that gives it a checkable question. For cyber, that change is a question the news archive can answer, with the CYBER_JUDGING_HELD hold lifted. For prediction markets, it is a spec that resolves by market slug on the settlement feed prediction:markets-resolution:v1, as shadow bets already do.
Withholding the 2 families also removes the political domain: every political forecast in the forecast history read on 2026-10-08 came from prediction markets. In that history, 61% of runs would publish forecasts in only 3 of the domains, and no run in fewer. The funnel-diversity check (forecast:funnel:health:v1) still counts published domains and the synthetic share. Its domain floor, DEFAULT_MIN_DISTINCT_DOMAINS in scripts/_forecast-funnel.mjs, is 3 while the families are withheld, so that expected state logs no warning. The floor returns to 4 when both families are published again. The generator logs a warning below the floor. It records the result as information only and never marks health as failed, because health reports whether the generator ran, not the quality of its output.
Before the fix for issue #5233, the resolver read cyber:threats-bootstrap:v2 and infra:outages:v1 as raw seed envelopes ({_seed, data}), found no records in them, and scored every cyber count and every outage presence check as 0. Those NO outcomes measured the reader, not the world. The resolver now unwraps the envelope before it reads any feed, and the emission-time extraction check reads the same unwrapped view. Each run seals the rows resolved before the fix with that zero reading as VOID with reason resolver_envelope_bug. It keeps their original outcome and evidence under supersededOutcome and supersededEvidence, and records when it voided them in voidedAt. A row already corrected is left unchanged, and a zero read by the fixed resolver keeps its outcome. While voided rows are in the rolling window, the scorecard methodology states how many there are. Price checks on prediction:markets-bootstrap:v1 hit the same reader and sealed as VOID with no_establishable_metric.
The unwrap exposed a second problem in the cyber feed. Only AbuseIPDB and URLhaus records carry a country, and the feed lists each for less than a day. A 7-day count read once a day therefore depends on how many hours before the read the deadline fell. One claim read 112 threats 1 hour after its deadline and 0 at 18 hours. Cyber forecasts are now judged, like infrastructure and military, and each run moves pending hard cyber rows to the judged lane. The news archive held few items on these subjects, so judged cyber rows are held out too. When one comes due, the resolver seals it as VOID with reason judged_evidence_unreliable and calls no judge. The judged-evidence fixes in #8995 kept the CYBER_JUDGING_HELD hold, because the archive still rarely names a cyber subject together with its country. Cyber forecasts are withheld from publication until a cyber question can be checked (#8990). The calibration map’s code version moved to forecast-calibration-pav-v2, so the map is refitted without the voided rows.
Windows and the scored probability
The resolver keeps one ledger window per question. A window is keyed<id>@<deadline> and covers emissions from its generatedAt up to its deadline. A market-settlement bet window is the market’s own question and covers every later emission of that market, because its deadline follows the venue’s close date and can move back before the window’s emission once the market closes. A question is the forecast id plus what it asks: the region, and for a judged spec the question text, for a hard spec the metric, operator, window and direction. A bet also counts its threshold, because each bet run asks a new threshold. A detector re-derives its threshold from the live value on every run, so its threshold stays frozen at the window’s first emission. When questions of one id share a deadline, the later one gets a ~<hash> suffix on its key. Cyber and military judged questions are rendered from live state, so for those domains (FROZEN_JUDGED_QUESTION_DOMAINS) a judged question is the forecast itself: the id, region and horizon, not the text (issue #9067). The cyber question names a count of threat indicators derived again from the live tally on every run, and a cyber count migrated from a hard spec asks a different template. A military forecast is one per theater, and its title follows the live surge type (posture, air activity, airlift or fighter surge) and the dominant operator country. The window keeps the question of its first emission, and later runs count as sightings. Other judged questions stay keyed on their text. A detector’s title is part of its id, and a state-derived title names its state unit, whose label changes together with the region, which the key already holds. Before this fix, each new count or title opened another window. The correction below seals those extra windows as duplicate_window and keeps the earliest.
The resolver reads the last 200 forecast-history runs and the bet history on every run. An emission that falls inside an existing window for the same question joins that window, whatever the window’s status, so re-reading history cannot reopen a resolved window. The resolver does not open a window whose deadline has already passed when it first sees the emission, because it could only read that window from later data.
A window is scored on the probability published at its first emission (firstSeenProbability), with the calibration, provenance and ensemble passes that came with it. A later emission of the same question records its probability in lastSeenProbability and changes nothing scored. The bet seeder does not re-run the ensemble for a bet whose question already has an open window that covers the current run. A bet emission on the base-rate placeholder (probabilitySource: 'base_rate', or no probabilitySource, which predates the ensemble stage) never opens a window, so every scored bet window opened on an ensemble. The betEngine slice’s base-rate, market and deviation comparisons read only those windows (ensembleCount).
Before issue #8990, re-reading history reopened windows that had already resolved, and each open window was scored on its last sighting. Each run corrects the rows written before the fix, after the resolver_envelope_bug voids. A window whose emission falls inside an earlier window of the same question is a duplicate. It and its horizon windows are sealed as VOID with reason duplicate_window, name the window they duplicate in duplicateOf, and keep their original status or outcome and evidence under supersededStatus, supersededOutcome and supersededEvidence. A duplicate that was already VOID keeps its reason and gains duplicateOf. A later correction can void the window a duplicate names, or leave it naming a window whose span no longer holds its emission (issue #9067). Such a duplicate is pointed at the live window that covers its emission, or else at the end of the chain, and records duplicateRepointedAt. A duplicate was never a question, so no scorecard count, receipt or card chip includes it. A window whose scored probability differs from its first-seen one, or that the old code saw again, is restored to its first emission with reason last_seen_probability, and the values it replaces move under rescore. When that emission is still in the history read, the window’s provenance, base rate, ensemble passes, market anchor and post-blend lineage come back exactly (restoredFromHistory); every bet window’s first emission is in the bet history. Otherwise the fields are inferred. The later sighting’s passes, anchor and lineage move, and the base rate stays. A first-seen value equal to the base rate marks a window that opened on the placeholder (probabilitySource: 'base_rate', with rescore.inferredBaseRate). An ensemble can land on the base rate, so an inferred label keeps the row out of the skill comparisons but does not void it. A re-sighted row whose probability did not move keeps its provenance and passes. A row that its first emission confirms is stamped openingVerifiedAt, so a later call can no longer mistake it for an overwritten row. Only a call that read history rescores; the resolver’s second ingest of a run passes no history and leaves rows as they are. Each run applies the correction again after it opens new windows, because a new window can precede an existing window of its question. A rescored row has no recorded raw value for its first-seen probability, so the calibration fit leaves it out. A corrected row is left unchanged, so re-running the correction changes nothing.
Before issue #8995, the judged lane picked archive items by matching the words of the question template, with no cutoff at the deadline. An audit of the 27 judged YES and NO verdicts sealed that way found 14 unsupported by their own citations and 1 probably wrong. Re-judging needs the archive to cover the last 7 days before the deadline, and every one of those windows closed by 2026-08-23, before the oldest evidence the 15-day archive still holds. Each run therefore seals every judged YES or NO resolved before #8995 reached production (SUBJECT_GATED_SELECTION_SINCE_MS, 2026-10-07T14:39Z) as VOID with reason judged_old_selection. The row keeps its resolvedAt, its original outcome and evidence under supersededOutcome and supersededEvidence, and records voidedAt. The step runs after the window correction, so a duplicate window keeps duplicate_window. A voided row is no longer YES or NO, so re-running changes nothing.
Since issue #9011 every judged verdict records the evidence selection it was judged under as evidence.selectionVersion (JUDGED_EVIDENCE_SELECTION_VERSION, now 3: 1 is the stock-word selection, 2 is #8995 and 3 is #8999 with #9002). Every judged seal carries it, VOIDs included, and a correction that wraps the evidence keeps it under supersededEvidence. The step voids every judged YES or NO below JUDGED_MIN_TRUSTED_SELECTION_VERSION (2). A row sealed before the stamp existed is dated by the cutoff above: version 1 before it, version 3 after it, because no judged YES or NO was sealed between the #8995 and #8999 deploys. A future change to evidence selection bumps the version, and raising the minimum then reaches exactly the rows sealed under the older selections.
Before issue #8990 a bet window opened on its first emission whatever that emission carried, often the base-rate placeholder. The window kept the placeholder for its whole life, and the seeder never ran the ensemble for its question again. Each run seals every bet window that opened on the placeholder as VOID with reason base_rate_placeholder (“Opened on a base-rate placeholder, not a model forecast”). A pending window keeps its status under supersededStatus; a resolved window keeps its resolvedAt, outcome and evidence under supersededOutcome and supersededEvidence. A placeholder window that was already VOID keeps its reason. Like a duplicate, a placeholder window was never a question, so no scorecard count includes it, VOID included, and the scorecard methodology states how many were left out. The bet_engine counts by generation origin therefore shrink rather than gaining VOIDs. A placeholder window no longer owns its question, so a later window that the duplicate_window correction had voided as its duplicate is restored to the status or outcome it had, records duplicateRestore (formerDuplicateOf, restoredAt), and goes through the window correction again. The question is then scored from its first ensemble window. A duplicate whose deadline or settlement read precedes its own emission stays a duplicate: it was graded on an outcome known before it was forecast, typically a later market of a slug that inherited the earlier market’s close date and settlement. More generally, a duplicate the old slug-only key created for another market of the slug stays VOID duplicate_window and names that other market’s window, whether or not its read preceded its emission. An example is the Rhine “before September 1” market, which the old key folded into the slug’s “between September 1 and September 14” window. The market title is now part of the key, so no new such duplicate forms, and none of these rows is scored. The correction can rescore a restored window back to the placeholder it opened on, which voids it and restores the windows it had absorbed, so the two steps repeat until no window is restored, and the ledger converges in one run. Re-running changes nothing.
A row corrected after its receipt reached R2 (resolver_envelope_bug, duplicate_window, judged_old_selection, late_read, resolver_could_not_read_feed, base_rate_placeholder, a restored duplicate, a rescore, a late-read re-grade or a re-pointed duplicate) has its receipt written again. R2 writes run one at a time, at about 450 ms each, inside the run’s 150-second fetch phase, so each run rewrites at most RECEIPT_REARCHIVE_PER_RUN (50) of these receipts, oldest first. The backlog drains over later runs, and pruning keeps a row until its corrected receipt is written.
Evaluation corpus
Each window records 3 provenance fields when it opens (issue #7072):codeVersionis the deploy commit of the seeder that emitted the window. Each forecast-history and bet-history run carries it once. A window opened from history written before that has nocodeVersion.slais the service level the window is held to, in milliseconds after its deadline. A judged window gets{ lane: 'judged', ms }withDEFAULT_JUDGED_SLA_MS: 2 days, counted from the deadline plus the 18-hour reporting grace. A hard window gets{ lane: 'hard', ms }: the longest its feed makes the resolver wait before it seals or voids the window while its data is missing (hardResolutionBoundMsinscripts/_forecast-resolution-eval.mjs, the bounds described under Late reads below), plusRESOLVER_CYCLE_MSfor the run that applies the seal.RESOLVER_CYCLE_MSis 1 day 1 hour: one daily run, plus an hour for its start time, asLATE_READ_MAX_LAG_MSallows. A test drives the resolver hour by hour with each feed down and checks it seals each kind of window within its bound. The levels are:
A count or value window seals as soon as its data arrives, and the level is the bound past which a feed that is down is voided, so a window resolved inside it is on time. A count whose feed is up but has not reached the deadline yet pends with no bound; it stays past its service level until the feed catches up. A horizon window is held to its sample tolerance plus
RESOLVER_CYCLE_MS, because it grades on the nearest sample within that tolerance.
emissionHashis the SHA-256 of the JSON array[id, deadline, generatedAt, probability, resolution spec]of the opening emission. Anyone holding the history snapshot can recompute it and check that the window was scored on what was published.
receiptHash, the SHA-256 of the stored receipt bytes. The hash can be published; the object path in receiptArchiveKey cannot.
The scorecard’s corpus block counts the corpus. The block is internal. The API, MCP and /accuracy/ select their fields by name and leave it out, because the public OpenAPI document is at its size cap. Its fields cover 2 different scopes, which the block names in registrationScope and slaScope.
The registration fields cover the history the resolver read on its run (registrationScope: 'history_read'): the last 200 forecast runs, about 8 days, and the last 200 bet runs, about 200 days.
publishedCountcounts the distinct<id>@<deadline>forecast windows in that history.ledgerRegisteredCountcounts those that a ledger window covers at every emission, andregistrationCoverageis their share. Only runs and emissions from the last 180 days count. The ledger keeps a window until 180 days after it resolves, and a window resolves after the emissions it covers, so a counted emission never needs a window that a run has already pruned. The exception is a hard spec the resolver voids on its first read (unsupported_metric_keyormissing_threshold): that window resolves at once and keeps covering later emissions of its question. Those emissions count as registered even after the window is pruned.historySnapshots,historyFromandhistoryTodescribe the runs counted.- A ledger window absorbs later deadlines of the same question. An absorbed
<id>@<deadline>counts as registered, but it is scored once, at the absorbing window’s deadline, never at its own.registeredLedgerWindowscounts the ledger windows behind the registered windows, so it is smaller thanledgerRegisteredCount. unregisteredByReasonnames each gap:no_resolution_spec,withheld_at_emission(the extraction gate withheld the forecast from scoring, counted inwithheldAtEmission),base_rate_placeholder(a shadow bet emitted only on the seeder’s base-rate placeholder, which opens no window),no_deadline,deadline_passed_before_registration(the deadline fell before the first resolver run after the emission, which the resolver skips on purpose) orunregistered(the resolver had a run to open the window and did not).
slaScope: 'rolling_window'), without duplicate, withheld and horizon windows. The top-level counts are the attributable published forecasts (cohort: 'published_origin'), the headline population, which leaves out state-derived and unattributed rows that the ledger totals count. byGenerationOrigin repeats every count for each origin, so shadow bets and synthetic rows stay apart from the published figures.
resolvedWithinSlaCount,resolvedLateCountandslaUnmeasurablesplit every resolved window, VOIDs included, by the service level stamped on it. A window opened before the stamp takes its level from its spec.scoredWithinSlaCountandvoidWithinSlaCountsplit the on-time windows, because a lane that voids everything at once would resolve everything on time.pendingPastSlaCountcounts open windows already past their service level.byLanesplits each count into the hard and judged lanes.voidByReasoncounts VOIDs by reason across both lanes.judgedLane.voidByReasoncovers only the judged lane.byLane.<lane>.latencygives the median and the 90th percentile (nearest rank) of resolution time minus deadline, in milliseconds. A market that settles before its deadline gives a negative value.
uncertainty block also carries internal intervals that the public contract has no room for. Each one resamples whole forecast families:
overallLogScoreandskillLogScoregive the log score with its interval.byDomainandbyGenerationOrigingive each group’s Brier and log score with their intervals, the group’s family count and its small-sample flag. Like the public groups, they pool every origin.vsMarketcompares forecasts with markets on the published-origin cohort.wmMinusMarketBrieris the forecast Brier minus the market Brier, negative when the forecast did better. Its paired interval keeps each window’s 2 losses together. The publicvsMarketSkillstill pools every origin.
totals and funnel count published forecasts only (issue #8990, from schemaVersion 3). Unpromoted shadow bets (bet_engine) are scored for evidence but never shown, so both blocks leave them out. A promoted bet is published and counts, as in the headline. State-derived and unattributed (unknown) forecasts were shown to users, so they count. The scorecard uses 2 populations that are both published. Published forecasts (isPublishedEntry) are everything users were shown: totals, funnel and goForward count them. Attributable published forecasts (isPublishedOriginEntry) also leave out state-derived and unattributed rows, because those are not attributable model forecasts: the headline skill score, publishedByDomain, receipts, familyOutcomes, corpus and the calibration fit count them. Shadow bets keep their own reporting in byGenerationOrigin and the betEngine slice, and overall, calibration and vsMarketSkill still pool every origin. /accuracy/ labels each pooled figure “of every origin, unpublished shadow bets included”. Settled bets almost never void, so pooling them hid the published VOID rate. On the ledger written by the 2026-10-08 resolver run, the rolling VOID rate was 71.1% (356 of 501) over every origin and 94.9% (356 of 375) over published forecasts. The methodology states the population. /accuracy/ names it in the totals and funnel captions and in its verdict from the capture’s schemaVersion, so a capture taken before the change still reads as pooling every origin, and the dataset download records it in ledgerPopulation.
totals.publicationCoverage is deprecated. It divides scored windows by every ledger entry, immature ones included, so it measures neither publication nor coverage. The scorecard still emits it, so API and MCP readers do not break. /accuracy/ computes the same scored share of the ledger from the counts, rounded as the field was, and prints it with a 95% Wilson interval. The actual rates in its verdict, how often the headline and all graded forecasts came true, carry one too. Each interval needs only the 2 counts printed beside it, so none costs the public contract a byte.
The run-diff tool (scripts/diff-forecast-runs.mjs) reports structural differences between 2 forecast runs. Its candidateBrierDelta is always unavailable with reason emission_snapshot_required. A candidate’s Brier can be compared with a baseline’s only when both runs forecast from one frozen emission snapshot and are scored on the same <id>@<deadline> windows. No run archives the inputs it forecast from yet (issue #7073).
Late reads and missing feeds
A live feed (commodity quotes, chokepoint risk) reports the value at the moment it was fetched, so a point-in-time question (at-deadline or at-endDate) must be read close to its deadline. The resolver runs once a day (cron 0 6 * * *), and its runs start between 06:00 and 06:05 UTC, so the first run after a deadline reads the feed within one cycle of it. LATE_READ_MAX_LAG_MS is that cycle plus an hour for start jitter. The resolver grades on the first finite reading taken at or after the deadline and within that bound. When no such reading exists once the bound has passed, the row is sealed as VOID: feed_unavailable when the feed is missing, late_read when the feed holds the metric but read it too late, and value_source_never_settled (a price or value) or no_establishable_metric (any other metric) when the feed no longer carries the metric. A missed run therefore voids the questions due in the gap instead of grading them on a later price. Feeds that date each reading by its own period or by the venue’s adjudication (EIA, FRED, prediction:markets-resolution:v1) keep their settlement grace and are not bound by it; when one of them stays missing past that grace, the row is sealed as VOID feed_unavailable. A count feed that stays missing for its settlement lag plus VALUE_SETTLEMENT_MAX_LAG_MS (10 days) is sealed the same way.
intelligence:gpsjam:v2 publishes one snapshot per UTC day, dated by its date field. A GPS-jamming question grades on the snapshot dated the deadline’s UTC day. seed-gpsjam runs every 8 hours and day D’s snapshot first lands at its 08:00 tick on D+1, so the resolver’s 06:00 run on D+2 is the first to hold it. GPS_JAM_SNAPSHOT_MAX_LAG_MS is 3 days and 7 hours from the start of the deadline day: that first run, one more cycle for a snapshot published late, and an hour of start jitter. A snapshot dated after the deadline day means the deadline day was replaced before the resolver read it, and the row is sealed as VOID late_read. When the bound passes with only older snapshots, the row is sealed as VOID value_source_never_settled, or feed_unavailable when the feed is missing.
Before issue #8990 no bound applied, and windows the resolver reached days late were graded on that day’s price. Each run corrects those rows after the duplicate_window voids. A YES or NO row on a live feed whose read landed past the bound is graded again on the first on-time reading that any window of the same metric sampled, or sealed as VOID late_read when the ledger holds none. The replaced outcome and evidence move under supersededOutcome and supersededEvidence. A re-graded row records regradedAt, a voided row records voidedAt, and resolvedAt is unchanged. A corrected row reads on time or is VOID, so re-running the correction changes nothing.
VOIDs from feeds the resolver could not read
Some feeds held readings while the resolver could not read them, and the rows it sealed in that period carryno_establishable_metric, which reads “The feed had no reading for this question”. That blames the feed for a resolver fault (issue #9013). Each run relabels those rows resolver_could_not_read_feed (“Our resolver could not read this feed correctly”). A row is relabelled when it is a resolved hard VOID with no_establishable_metric on one of these feeds, sealed before the first deploy of the resolver that could read it:
The outcome and
resolvedAt are unchanged. The earlier evidence moves under supersededEvidence, and voidedAt records the relabel, so the R2 receipt is written again. A relabelled row no longer carries no_establishable_metric, so re-running changes nothing. VOIDs are outside the accuracy math, so no score changes.
A period feed (EIA, FRED) dates each sample by its observation day, a UTC midnight. A point read on one of them grades on the first sample dated on or after the deadline’s UTC day, the same day boundary the settlement gate accepts. Before issue #8990 the selection required a sample at or after the deadline time, so the deadline day’s own observation was skipped and the resolver read the live value instead.
Judged-lane evidence horizon
A judged entry is decided by two independent LLM judges reading the dedicated forecast evidence archive (forecast:evidence:v1), and each must cite an archive item by ID with a quote that appears in that item — a judgment with no valid citation can never seal as YES or NO. The judges see up to 32 items, and every item must name the forecast’s subject: its region or one of the region’s aliases (for example Damascus or Idlib for Syria). The alias table is generated for every country and every region label the forecast emitter can produce (scripts/build-judged-subject-terms.mjs). Names that are also common words or places elsewhere (Georgia, Jordan, Chad, Niger, Turkey, Congo) count only beside a disambiguating word such as a capital. Matching is on whole words. The words of the question template never select an item. Items published by the deadline come first, then items that name the subject in the headline, then items that use the forecast domain’s event words, then the newest. The evidence window ends 18 hours after the deadline, so reports of deadline-day events can arrive, and an entry is not judged before that. Citing an item published later is impossible, because such items are never shown. The window starts when the forecast was generated, clipped to the oldest evidence the archive still holds, so a 30-day question sees up to 14 days of its horizon. The prompt states the window as served. A NO may rest on absence as well as on proof: both judges may answer NO with basis absence, citing the on-subject items they read. Two kinds of check apply. If the archive’s coverage bounds do not span the last 7 days before the deadline through the end of the window, the entry stays pending (archive_incomplete) and no judge is called, until it reaches its recovery horizon, after which it seals as VOID with beyond_archive_horizon (described below). Once coverage holds, each absence-based NO is checked in code, and a judgment that fails is recorded as VOID with the failing condition as its reason: the archive must declare coverage bounds; at least JUDGED_ABSENCE_MIN_ARCHIVE_ITEMS of the items shown must name the subject, use one of the domain’s event words, and be published by the deadline (an item about the subject’s sport or culture is not coverage of a conflict); the judges must have been shown every on-subject item in the window, including reports from the 18-hour grace, because an item cap that dropped one could have dropped the report confirming the event; and the judgment must cite at least one of those coverage items. Subjects with more than 32 on-subject items in the window cannot seal an absence-based NO; agreement requires the same basis from both judges, and the sealed evidence records basis so absence-based NOs can be counted apart from proven ones. The archive that backs those citations retains 14 days, while a judged entry requires evidence reaching back 7 days before its deadline. An entry therefore has a hard recovery window: once now passes 14 days after the start of its required evidence, the evidence has aged out of the archive and no later run can retrieve it. The required evidence starts 7 days before the deadline, or at generation for a forecast generated later than that, so for 7-day and 30-day forecasts the window closes at deadline + 7 days.
Rather than retry such an entry until its attempt budget is exhausted, the resolver seals it deterministically as VOID with reason beyond_archive_horizon, and warns while entries are still inside the window so the backlog can be drained before they strand. Every judge attempt persists a structured record naming the stage it failed at (archive, judge_a/judge_b, normalize, agreement, terminal) and the failure class, which the scorecard rolls up under judgedLane — an internal operator block that is not served on the public scorecard endpoint — alongside first-attempt seal rate, VOID-by-reason and attempts per resolved entry. The SLA window is two days measured from the deadline plus the 18-hour reporting grace, the first moment the lane may judge, so it always holds a first attempt and one retry on the daily schedule. The SLA metric counts scored resolutions within the window rather than resolutions of any kind, so a lane that seals everything as an instant VOID reports a rate of zero rather than a perfect one; the VOIDs it produced are reported beside it. A beyond_archive_horizon VOID is cost control, not a resolution-quality result: it is counted as VOID, never as a scored resolution.
Calibration map (shadow)
Each resolver run also maintains a calibration map inforecast:calibration-map:v1. The map is fitted by isotonic regression (pool-adjacent-violators) on individual resolved YES/NO published-origin forecasts from the 180-day window. Each row is weighted 1 divided by the number of rows in its forecast family, so every family carries the same total weight and a question resolved in 10 windows counts once, not 10 times (issue #9034). In a simulation with 1 to 12 rows per family and within-family outcome correlation 0.7, family weights beat row weights by 0.001 to 0.003 expected Brier in every scenario tried. The published Brier score still counts rows. The fit cohort excludes bet_engine and state_derived entries, and entries with no recorded origin (filed as unknown), the same population as the headline skill score. Entries with no recorded origin predate origin tagging and cannot be attributed to a generator. It also excludes entries whose market anchor has no marketBlendedProbability lineage field. The matcher before issue #7071 chose those anchors, and it often paired forecasts with unrelated markets. The fit cohort uses the scorecard’s own predicates, so it never includes a row the scorecard excludes: VOID rows (late reads and old-selection judged verdicts among them), duplicate windows, withheld state buckets, and rows rescored to their first-seen probability.
Eligibility counts forecast families, as defined for the skill score above, not rows. A family’s windows share a generator, a region and much of an outcome history. A domain receives a non-identity map only when its cohort has at least 30 forecast families (CALIBRATION_MIN_FAMILIES), at least 12 of them with a YES outcome and at least 12 with a NO outcome (CALIBRATION_MIN_OUTCOME_FAMILIES). A family whose windows resolved both ways counts on both sides. A one-sided domain, such as cyber with only NO outcomes, can never fit. In a simulation on the live published-origin probabilities, isotonic regression lost to identity on expected Brier in every miscalibration tried until each outcome had at least 12 families. At 12 it first wins, for a biased forecaster. Each domain records its family counts and, when it is identity, the ineligibleReason. Every other domain, and any domain the map has not seen, passes probabilities through unchanged. Fitted values are bounded to the range 1% to 99%.
The map records its version, fit time, fit window, cohort filter, per-domain sample and positives, and the probability stage it was fitted on (marketBlendedProbability, read from calibration.marketBlendedProbability when an anchor applied and from the stored probability otherwise). The map stays unchanged between runs so that its forward cohort can grow. Every resolver run recomputes eligibility and refits only for one of these causes, recorded as refitReason:
fit_input_withdrawn: a row a fitted domain used was later voided, marked duplicate, rescored, withheld, re-graded, or changed in probability or domain. Each fitted domain stores a fingerprint of each input row (outcome, fitted probability and domain) underinputsfor this check. A row pruned from the ledger, or one that only left the rolling window, is not a withdrawal.fit_invalidated: a fitted domain fell below half of each minimum (CALIBRATION_RETAIN_FRACTION): fewer than 15 families, or fewer than 6 YES or 6 NO families.domain_became_eligible: an identity domain reached the family minimums.family_growth: an eligible fitted domain’s families first resolved since the fit equal the families the fit used (CALIBRATION_REFIT_FAMILY_GROWTHis 2). A new window of a family the fit already used does not count. A domain inside the retain band but under the minimums does not trigger growth, because a refit would only carry its fit unchanged.fit_aged_out: none of the rows a fitted domain used is still in the rolling window. When the domain is no longer eligible, the refit makes it identity.absent,invalidorcode_version_changed: the stored map is missing, unreadable, or from another fitting rule.
dataVersion and keeps the code version. Each refit starts a new forward cohort, so the triggers wait for new evidence rather than a change in the window’s count. The window holds about the arrival rate times 26 weeks, so its count plateaus. Growth therefore counts families that first resolved after the fit. When the arrival rate is too low for that count to reach the fitted one, the map refits once the fit’s own rows have all left the window. A fitted domain keeps its fit down to half the minimums, so a domain at the floor does not refit each time one family ages out and another resolves. When another domain’s trigger refits the map, a fit inside that band whose rows are still in the window and unwithdrawn is carried into the new map unchanged, with carriedFrom naming the map it was fitted in.
While the activation gate passes for the stored map, every trigger except fit_input_withdrawn is held, with no time limit. A refit would discard the forward evidence and revert a validated publication. The gate scores only forward outcomes resolved inside the scorecard’s 180-day window, so an old map stays held only while recent outcomes still validate it. When they stop, because the map no longer beats raw or because too few recent outcomes remain, the gate fails, the held trigger refits, and publication returns to raw on the next seeder run. family_growth and domain_became_eligible are also held while the gate lacks its forward sample (insufficient_forward_total or insufficient_forward_domain). Growth can arrive before the gate has its 60 forward families: 30 new families since the fit are half of what the gate needs. A refit then would empty the forward cohort again, and the map would never reach a verdict. On a partial sample the bootstrap interval is wide, so overall_not_non_inferior or domain_not_non_inferior alone does not release this hold. A map that helps usually carries them until its sample is complete. The hold ends early only when inferiority is demonstrated: the lower bound of the 95% interval on calibrated minus raw Brier exceeds the 0.005 margin, overall or in an activated domain, so raw is better even at the favourable end of the interval. Any other gate failure, such as no_non_identity_domain, also releases it. fit_aged_out and fit_invalidated are not held while the gate fails, so a map that never earns publication is refitted within about 180 days of its fit. Every pending trigger is checked in order, and the first one the gate does not hold refits. The resolver logs a held trigger, and the internal scorecard is built at the same run time the map was resolved at, so the hold and the publication read one verdict. A withdrawn input refits at once, and publication also returns to raw, because the gate for the new map has no forward outcomes yet. A failed read keeps the stored map instead of refitting.
The map changes published probabilities only while the activation gate below is eligible. The scorecard carries an internal calibrationShadow block for the forward cohort, which is the set of forecasts emitted after the map was fitted, less outcomes resolved before the scorecard’s 180-day rolling window. Like judgedLane, the public scorecard endpoint does not serve it. For the forward cohort the block reports row and family counts, raw and calibrated Brier, the paired family-cluster bootstrap 95% interval on their difference, expected calibration error with bootstrap intervals, and reliability buckets with Wilson intervals. Every figure shows its sample size. The evaluator refuses any entry emitted at or before the end of the fit window, so the map is never scored on the outcomes it was fitted on.
The block scores 3 stages on the same forward rows. In calibrationShadow, raw is the post-blend probability, the stage the map is fitted on, and calibrated is the map’s output. internal is the pre-blend internalProbability, the forecaster’s own value before the market anchor. It is read from the market-anchor lineage frozen when the window opened. A window that opened without a market anchor was never blended, so its pre-blend value is its post-blend value. Overall and in each domain, the internal block reports Brier, the post-blend Brier on the same rows, the paired family-cluster bootstrap 95% interval on internal minus post-blend Brier, expected calibration error with its bootstrap interval, and reliability buckets. A positive difference means the market blend and its domain cap lowered Brier. A row whose pre-blend value is unknown is counted in internal.missing and is not scored on this stage; its value is never imputed. Such rows are anchors recorded before the #7071 lineage and windows rescored by inference rather than restored from their first emission, because the old overwrite could have dropped the opening anchor. The internal stage is reported only. The activation gate below still compares calibrated with post-blend raw, and its verdict does not read this stage.
Activation gate
calibrationShadow.activationGate reports eligible and lists every reason it is not eligible. The gate is eligible only when all of these conditions hold:
- The forward cohort has resolved outcomes from at least 60 distinct forecast families (
ACTIVATION_MIN_FORWARD_FAMILIES), and from at least 30 families in every domain with a non-identity map (ACTIVATION_MIN_FORWARD_DOMAIN_FAMILIES). - The upper bound of the paired family-cluster bootstrap 95% interval on calibrated minus raw Brier is at most 0.005, overall and in every non-identity domain.
- The map is monotone and has at least one non-identity domain.
forwardCount (rows) and forwardFamilies, and each activated domain’s count and families; the minimums read the family counts.
Forward coverage (registered, matured, scored, VOID by reason, pending) and the generation-origin mix are reported beside the verdict, so a change in which forecasts resolve cannot read as a skill gain. The bootstrap draws 2,000 resamples of whole families from a fixed seed: each resample draws as many families as the cohort holds, with replacement, and pools all their rows. A rerun on the same ledger gives the same intervals. Repeated windows of one family therefore cannot narrow the interval.
Published calibration
Everyseed-forecasts run applies the same rule. The run reads the map and the latest calibrationShadow.activationGate. It publishes calibrated probabilities only when the gate is eligible, the gate evaluated the current map version, and the kill switch is off. In every other case the run publishes raw probabilities. The rule does not read the previous mode, so a gate that fails on a later resolver run reverts publication on the next seeder run. No manual step is needed in either direction.
The map applies to the post-blend probability, after the market anchor and the domain cap. This is the stage the map is fitted on. The map applies before confidence, projections, trends, case files, narratives, and publish selection, so every surface in a run shows the same number. Trends and change summaries compare against the prior run re-expressed on the current scale, so a flip or a refit does not read as a probability move. Only published-origin forecasts move. State-derived forecasts keep their raw value, because the fit excludes them. Calibrated values are bounded to 1% and 99%, not to the market-blend domain caps, because the map is fitted on outcomes after those caps.
Publish selection ranks on the published probability, so the forward cohort that the gate scores can change in composition after a flip. The gate still compares raw and calibrated values on the same entries. Coverage and origin mix beside the verdict, and the lastFlip time, show when the cohort’s selection rule changed.
A forecast that moves keeps its post-blend value as uncalibratedProbability. The value goes to the forecast history and the resolution ledger, but not to the public API. The ledger’s probability is the published value, so the public scorecard scores what readers saw. The fit and the shadow read uncalibratedProbability as the raw value. Without it, the shadow would compare the published calibrated value with itself, and the gate would stay eligible after any flip.
The public API does not mark which probabilities are calibrated. No proto field carries the mode, so readers cannot tell from a forecast alone. The scorecard methodology text states this. The public scorecard scores each window on the probability published when the window opened, so after a flip its rolling-window Brier mixes forecasts published raw and forecasts published calibrated until the raw-era windows leave the window.
A map written by another code version publishes raw, as does a gate verdict whose scorecard is more than 48 hours old.
Set FORECAST_CALIBRATION_FORCE_RAW=1 on the seed-forecasts service to publish raw probabilities whatever the gate says. A failed read of the map or the scorecard also publishes raw. A failed read of the previous record leaves the publication decision unchanged and skips the record write, so the stored flip history is kept.
Each run logs its mode, its reason, the map version, and the gate numbers: eligibility, forward count, the overall Brier-delta upper bound, and each activated domain’s count and upper bound. A mode change logs a FLIP line. The run writes its decision to forecast:calibration-publication:v1 after the publish succeeds. The record holds the mode, the reason, the gate numbers, the decision time, and lastFlip, which gives the time, direction, reason, and gate numbers of the most recent mode change. The resolver copies the record into the internal scorecard as calibrationShadow.publication. The reasons are gate_eligible, gate_ineligible, gate_map_mismatch, gate_stale, map_code_version, no_gate, no_map, read_failed, and force_raw.
Rollback to identity
Identity is the fallback at every level. An unknown domain, an identity domain, and a missing map all return the input probability unchanged. Raw publication is the fallback for the activated path: a failing gate, a missing map, or the kill switch publishes the post-blend probability on the next run. A calibrated publication keeps its post-blend value asuncalibratedProbability, so a return to raw loses no lineage.
Frozen evaluation cohorts
A frozen cohort follows one set of forecast windows from emission to outcome. A later change to the resolver, the scorecard filters or the rolling window therefore cannot drop an entry from the count (issue #7066).scripts/forecast-cohort.mjs is an operator tool, not a Railway service. It only reads production Redis and writes a local file.
node scripts/forecast-cohort.mjs freeze --id <id> --start <iso> [--end <iso>] [--origin a,b] [--domain a,b] reads these keys:
forecast:resolutions:v1- the newest 200 runs of forecast history and of bet history, the same read the resolver makes
- the forecast trace pointers in
forecast:trace:runs:v1
ingestHistory and emission helpers over them in memory. Every window opened by an emission inside the range is frozen under the id@deadline key the resolver gives it, including windows the next resolver run will register. Some emissions map to no window, for example when the deadline passed before any resolver run read them. Those are frozen too, with unregisteredReason, along with the missing horizon windows of a cohort window. Whether an emission counts therefore never depends on when the freeze ran.
Each entry records its origin, domain, lane (hard, judged or horizon), deadline, whether it is of published origin, its emissionHash, its service level, and the code version of the run that opened it. The service level is the sla the resolver stamps on the row when it opens it. A row without a stamp takes its lane’s level from its spec, as slaDueAt does: hardSlaMs for a hard or horizon window, DEFAULT_JUDGED_SLA_MS for a judged one. A horizon window the ledger never registered is frozen with the level the resolver would have stamped: its sample tolerance plus RESOLVER_CYCLE_MS. The code version comes from the first of these sources that has one:
- the ledger row’s stamp
- the forecast or bet history snapshot’s stamp
- the trace pointer of the same run
- an earlier manifest named with
--carry-code-versions
unknown. Both history lists stamp each snapshot since #9059, so the code version survives as long as the history does. Older snapshots fall back to the trace pointer list, which keeps 50 entries (about 25 runs), or to an earlier manifest of the same windows. --carry-code-versions checks the earlier manifest’s schema and manifestSha256, and carries only versions that freeze read first-hand, never one it carried itself.
Every emission in the range is accounted for in the counts:
- in a cohort window
- joined a window opened before the range, and is scored on that earlier emission
- left out by a filter
- skipped by ingest, with a reason from the resolver’s
REGISTRATION_GAP_REASONS; for example,base_rate_placeholdermarks a bet on the seeder’s base-rate placeholder, which opens no window
--origin legacy_detector), because the projection must read every emission the resolver reads. It also refuses a range that starts at or before the oldest snapshot read, because runs at the start of the range may already have been trimmed. Run the freeze within about 8 days of the range start.
The manifest is committed under data/forecast-cohorts/. Its manifestSha256 covers everything in it, including the range, the filters and every entry with its service level. A freeze never replaces an existing manifest: a later freeze would read a later ledger and history, so a new freeze needs a new --id.
node scripts/forecast-cohort.mjs reconcile --manifest data/forecast-cohorts/<id>.json walks every frozen entry and gives it exactly 1 class:
The resolver reads history when its run starts and writes the ledger minutes later. A ledger write soon after an emission therefore does not show that the resolver read that emission.
RESOLVER_CYCLE_MS (1 day and 1 hour, the scorecard’s resolver cycle) is the wait.
Reconciliation applies each entry’s frozen service level to the deadline the ledger holds now, through the scorecard’s slaDueAt. A judged window’s level starts after the 18-hour reporting grace. A market-settlement window is judged by its current deadline. A hard window waits as long as its feed is designed to make the resolver wait: for example, 2 days and 2 hours for a live point read, 15 days and 1 hour for EIA or a market settlement, and 76 days and 1 hour for monthly FRED.
The verdict covers published-origin entries, the population the epic’s acceptance names:
met: every entry is terminal within the service levelbreached: any breach is listedopen: otherwise
bet_engine) and synthetic windows get their own verdict beside it. The report counts each class by lane, origin and code version, tallies VOIDs by reason, and lists every breach by key. Reconciliation refuses a manifest that no longer matches its hash.
The resolver prunes resolved rows 180 days after resolution. A cohort entry reconciled after that reads missing_from_ledger, so reconcile a cohort before then.
Run archive and replay
Eachseed-forecasts run writes its inputs to the private forecast trace bucket in R2 as deep-snapshot.json. The snapshot holds the raw feeds the detectors read, the recorded critical-signal bundle, every forecast the run produced, and the code revision in triggerContext.deployRevision. Since issue #9058 it also holds calibrationPublication: the calibration mode, reason, map version and decision time the run scored with, and the map’s per-domain knots.
The run computes snapshotSha256, the SHA-256 of the exact bytes it writes, before it appends its history entry. The history entry carries runId and snapshotSha256, and the trace manifest and trace pointer carry the digest. When the resolver opens a ledger window, it copies runId and snapshotSha256 onto the row, and the row’s R2 receipt keeps them. The link is set when the window opens, like the opening probability, and a later sighting does not change it. Rows opened before #9058 have no link. The public scorecard receipts do not include either field. If the snapshot write fails after the history append, the digest names an object that does not exist, and the replay tool reports the snapshot as missing.
Each seed-forecast-bets run writes bets-input-snapshot.json in the same dated layout, in a run directory named <generatedAt>-bets. It holds the bet feeds left after the freshness filter, the rolling series before the run updates it, and, when the ensemble runs, the open-question tokens, the news titles, a digest of the ensemble prompts, and the provider and model of each LLM call. The bet seeder writes the snapshot before it appends its history entry, so a bet row’s snapshotSha256 always names a written object. Passing the archived feeds, generatedAt and series to buildBetsSnapshot reproduces every baselineProbability. Ensemble probabilities come from an LLM, so they are kept as recorded outputs and are not replayed.
scripts/replay-forecast-detectors.mjs replays one run. It takes a ledger key (--entry-key), a run id (--run-id), or a local file (--snapshot). It reads the snapshot with the R2 trace credentials and checks it against a recorded digest: the ledger row’s, the run’s trace manifest’s, or one given with --sha256. A snapshot with no recorded digest, such as one from before #9058, needs --trust-unverified. The tool runs only a revision on origin/main, so a commit reachable only from a fork’s pull request never runs. It checks out the recorded revision into a scratch git worktree, then reruns the detectors and scoring, or buildBetsSnapshot for a bet run, in a child process. The child gets only PATH, TZ and a scratch HOME, runs outside the repository, has fetch disabled, and has its clock frozen at the snapshot’s generatedAt. It aborts if any other environment variable appears. This keeps credentials away from trusted code; it is not a sandbox for untrusted code. It reports exact, differing, missing and extra counts against the recorded forecasts and their horizon projections. For a ledger row, it compares the replay with firstSeenProbability, the value the window is scored on. --candidate adds a second pass on the current code, labelled separately, so a code change is not read as a snapshot failure. An interrupted replay stops the child and removes its scratch files. A row from before #9058 is matched to the run whose id was minted up to 15 minutes before the row’s generatedAt and whose snapshot carries that same generatedAt.
Snapshots contain licensed third-party data, including ACLED events and news, and must stay private. They are never published, and their contents are never posted. The replay tool keeps a snapshot only in a temporary file readable by the current user, deletes it afterwards, and prints forecast ids and numbers only. Snapshots must be kept for at least the longest horizon, 210 days, plus the resolution delay. Since 2026-10-09, the dated run directories under the trace prefix are deleted after 400 days, with one lifecycle rule per year (2024 to 2027). The rules are set in the bucket configuration, not in this repository, and a rule for 2028 must be added before 2028-01-01. The rules must not cover the resolution receipts, and the resolver refuses to write receipts at or under the trace prefix.
Tier & gating
On web, AI Forecasts is currently in trial — free to anyone, including unauthenticated readers. On desktop, the same panel ships withpremium: 'locked' in src/config/panels.ts so it is gated there. This split reflects the product’s current trial positioning; expect the web gating to tighten over time.
API reference
- Forecast service — covers
get-forecasts,get-forecast-scorecard,get-simulation-outcome,get-simulation-package, andtrigger-simulation. - For programmatic prediction generation (fresh probabilities outside the cached cron output), see the MCP
generate_forecaststool.