/api/health
Primary health endpoint. Checks all Redis-backed data keys and seed freshness metadata in a single pipeline call.
Authentication: Compact health (?compact=1) is public for uptime and keyword monitors. Detailed health (/api/health without compact=1) and the operator history view (?history=1) require a valid operator/enterprise API key because they expose canonical Redis key names, record counts, and freshness thresholds. Browser origins must still pass the CORS allowlist in api/_cors.js; requests with no Origin header, such as server-side monitors, are allowed only for compact health unless they include an operator key. Health responses are never cached (Cache-Control: private, no-store, max-age=0 and CDN-Cache-Control: no-store).
HTTP Method: GET
Query Parameters
Response Status Codes
The overall health verdict lives in the JSONThe top-levelstatusfield, not the HTTP code. Every state exceptREDIS_DOWNreturns200so warn-level seed jitter doesn’t flap HTTP-status monitors (see PR #2699).REDIS_DOWNreturns503because with Redis unreachable the endpoint can assess nothing, so a plain HTTP probe must see a failure.
status describes user-visible platform availability, not perfect data-source health. HEALTHY can therefore coexist with warnings in summary and problems when those warnings are contained by usable last-good data.
Response Body
summary.total counts probed keys. warn is the complete actionable warning census and excludes on-demand empty keys, which appear in onDemandWarn. containedWarn is a subset of warn, not an additional bucket. The containment-eligible statuses are STALE_SEED, SEED_ERROR, STALE_CONTENT, COVERAGE_PARTIAL, COVERAGE_DEGRADED, and CHINA_DEGRADED. A warning is contained only when the current health sweep finds the served payload, metadata proves a positive record count, and every required reader diagnostic is structurally usable. The warning’s stale age remains visible but does not by itself mean that the platform stopped serving data. Availability remains HEALTHY while all actionable warnings are contained and the contained cohort is at or below 3% of probed keys.
Containment checks all required diagnostics independently of the selected status and has no source-specific allowlist or denylist. Missing or unusable data, malformed or unknown evidence, incompatible reader policies or cache state, REDIS_PARTIAL, ROLLOUT_PENDING, any uncontained warning, a contained cohort above 3%, and every critical failure remain availability-affecting. EMPTY_ON_DEMAND and pending diagnostics retain their existing semantics. staleContent counts all STALE_CONTENT diagnostics, including entries inside their finite three-hour grace period. Active grace entries count in ok and the optional pending sub-count, not warn. rolloutPending is a subset of warn. Only crit drives DEGRADED or UNHEALTHY.
With ?compact=1, problems contains every actionable failure and on-demand empty entry, including warnings contained from the availability verdict. pending contains diagnostics inside an active grace period. Empty maps are omitted. Full checks retain the same diagnosis. Cached verdicts still expire at active grace deadlines, and the scheduled monitor rechecks them before suppressing an alert. Strict data-quality monitoring must inspect summary.warn and problems; it must not use the top-level availability status alone.
Cable cache continuity
The cable-health handler recomputes its response after 30 minutes but retains the last valid payload for at most 90 minutes, matching its existing health budget. A cache hit does not advanceseed-meta:cable-health.fetchedAt. Slow or failed refreshes leave the prior payload and its original success time intact; repairing an evicted payload does not reset its expiry or overwrite a concurrent fresh publication. A confirmed zero-cable result is valid data. An unavailable source with no usable last-good result is not published as a successful empty observation.
Cache hits migrate legacy short TTLs to the snapshot’s original 90-minute deadline and retry missing or incorrect metadata. The repair checks the current payload atomically before changing its expiry or metadata, so an older reader cannot alter a newer publication. Concurrent requests within one runtime share the full refresh, including cache publication when NGA data is already cached.
Contracts Finder refresh failures
The hourly Global Tenders member runs within the half-hourlyseed-bundle-relay-backup scheduler. Contracts Finder uses two 45-second attempts for its documented OCDS query; a valid response was observed taking 23.6 seconds to first byte, beyond the previous 20-second limit. This stays within the member’s 180-second timeout.
Failed refreshes retain only validated open Contracts Finder records less than 180 minutes after that source’s last success. Attempts never advance the success clock, and successful sibling sources cannot renew this limit. Malformed releases, including active notices without a usable closing date, fail validation instead of replacing last-good data with an empty success. An actual successful response, including a verified empty result, clears the failure episode. Legacy failure metadata with no count does not qualify as a first failure.
globalTendersContractsFinder keeps SEED_ERROR, the warning count, source error, attempt time, success time, and consecutive failure count visible. Availability can contain the first failed refresh only after reading and checking its records in the canonical tender snapshot. containmentUntil is the earliest of 90 minutes after the first failure, 180 minutes after source success, or a retained tender’s closing time. The second failed refresh, missing or unreadable canonical data, malformed records, inconsistent metadata, or an expired deadline cannot qualify. Full and compact verdict caches expire at the same deadline; this does not change other sources’ health policy.
A failed refresh after a verified empty success can also qualify. The producer records confirmedEmpty only when the prior canonical envelope has an explicit zero source count, no source rows, and recent source success evidence. Health requires matching canonical and seed metadata, a fresh usable aggregate, and the same first-failure limits. Its deadline also includes aggregate freshness. Zero records without this proof, or records removed by expiry or validation, cannot qualify. The source remains error with SEED_ERROR; containment does not report a successful refresh.
BC active evacuation lists
canadaAlertsBcSource uses successful active-list ingestion with a 45-minute freshness budget. BC orders remain active until their status changes or they are removed. Their event modification age is not an expiry rule. See BC evacuation guidance.
The seeder validates every page before it replaces the snapshot. A verified empty list is valid. Malformed or failed responses retain the previous snapshot and success time. Event updatedAt, publishedAt, and onset retain the publisher’s dates. The old three-day content-age fields disappear after the first successful v2 publication. Existing missing-data, source-failure, and stale-ingestion checks still apply.
Bundle tick heartbeats
_bundle-runner.mjs writes bundle:heartbeat:<label> on every container start, including skip-all ticks. Member seed-meta cannot see a daily cron that never fires when its sections are weekly or monthly. /api/health probes staticRefBundleTick (bundle:heartbeat:static-ref) and, as of #6806, staticRefHeavyBundleTick (bundle:heartbeat:static-ref-heavy) — one heartbeat for the consolidated heavy bundle carrying Arms-Suppliers, Military-Bases and Mineral-Production. A 48h silence is STALE_SEED. Missing keys before the first tick are EMPTY under an expiring acknowledgement, not a page-out.
Consumer-price source coverage
The consumer-prices-core service exposesGET /wm/consumer-prices/v1/coverage?market=ae (the service API key is required). Its response includes attempted, completed, failed, and validator-rejected page counts for the market and each active retailer. status is healthy, partial, or degraded; a partial result remains publishable, while a rejected observation is never admitted merely to improve the coverage count. The publisher writes the same snapshot to consumer-prices:coverage:<market> and includes its aggregate completion ratio in seed-meta, so /api/health can distinguish a fresh partial run from a stopped producer.
This operational contract complements, rather than replaces, the source-quality work in #5445 and #5811.
Relay ingestion telemetry
The relay’s publicGET /health remains HTTP 200 for Railway/process-liveness probes, but its top-level JSON status now mirrors the aggregate ingestion verdict and can be degraded; ingestion.status carries the same application-level result. It becomes degraded when aviation or RSS served coverage falls below its configured floor, or when configured AIS is disconnected, has no usable current-connection position data, or serves zero snapshots after they are requested. AIS WebSocket upgrades time out after 30 seconds by default, and position readiness expires when an open stream stops delivering accepted position reports for five minutes by default; either failure re-enters the bounded reconnect schedule, while retained vessel data remains a stale fallback. Deployments intentionally missing AIS credentials report enabled: false and status: "disabled"; that optional adapter does not degrade aggregate health.
The curated X feed uses one newest-page request from a verified public List in each fixed 15-minute UTC slot. The adapter is enabled only when the AIS relay has both X_BEARER_TOKEN and X_CURATED_LIST_ID. Before activation, run node scripts/verify-x-accounts.mjs: it requires the List to be public, readable in one page, and an exact immutable-ID match for the 64 enabled entries in data/x-accounts.json. The verifier reads List and User resources only; the first natural relay slot owns the paid Post contract check. Public xFeed freshness advances only after a valid settled List page, including an explicit empty page, is committed with its seed metadata. Its freshness timestamp stays bound to the original provider success if publication replays later. The daily deletion audit is ancillary maintenance: its failure remains visible in authenticated status but does not make an accepted List page unhealthy.
Failed List attempts keep the last-good feed and its original freshness clock. They refresh failure metadata with sourceState: degraded, a bounded errorCode, and the attempt time, so health reports SEED_ERROR with the retained record count and age instead of losing those diagnostics when metadata expires. The 45-minute freshness threshold is unchanged. One completed HTTP 408/500/502/503/504 response can retry after 1–5 seconds, using a new shared-budget admission from spare capacity; future scheduled coverage stays reserved. Longer Retry-After values, auth/credit/rate-limit failures, malformed successful payloads, and ambiguous paid transport or settlement outcomes are not retried. The relay wakes at a shared backoff deadline within the current slot, so a 30-minute backoff that ends milliseconds after a UTC boundary does not add another 15-minute delay. This path uses the official X API directly with one configured bearer and List; it has no account or proxy failover.
The authenticated relay GET /status separates attempt, provider-success, and accepted-publication clocks and reports whether the next five-Post request is admissible. Note that the pre-existing lastPollAt field changed meaning in this release under the same name: it previously advanced on every completed cycle and now advances only with an accepted publication, matching lastAcceptedPublicationAt. The attempt clock it used to carry is lastAttemptAt, which is served on both GET /status and the public GET /health; monitoring built on the old attempt-freshness semantics should read lastAttemptAt.
The cutover order is strict because the old AIS relay and company-monitoring worker used a different daily coverage hold. Create and verify the List first, then set X_CURATED_LIST_ID on the running AIS relay before deploying this code; the old relay ignores that variable. Deploy both ais-relay and company-monitoring-worker from the new head no earlier than 30 minutes before 00:00 UTC. The new relay starts with both X prerequisites and keeps the last-good feed while its limiter rejects the old unversioned coverage hold. Confirm that all old instances of both services have exited before the boundary, because either old service can initialize the legacy hold after midnight. At or after 00:00 UTC, verify the List again and observe the first natural List slot. Do not reset or rewrite the live budget counters. If any old instance does not exit before the boundary, stop the cutover and roll back the new services instead of leaving X disabled or waiting through another full day.
GET /health also adds upstream attempt, throttle, failure, cooldown, position age, and position-freshness threshold details to explain the current verdict. A run of consecutive HTTP 429 upgrade rejections is reported as consecutiveThrottles, and once it reaches AIS_THROTTLE_ESCALATE_AFTER the reconnect ceiling switches from AIS_RECONNECT_MAX_MS to the much longer AIS_THROTTLE_RECONNECT_MAX_MS and throttleEscalated becomes true. A 429 arrives before the API key is sent, so it identifies an egress-IP rate limit rather than a transient stream fault; escalating stops the relay contributing refused requests to the provider’s sliding window, which can otherwise sustain the block. Any non-throttle outcome — a different handshake error, a close without a recorded error, or an accepted frame — resets the counter and restores the ordinary ceiling, so ordinary disconnects keep the responsive schedule. GET /metrics remains the operational counter surface: it exposes rolling-window per-route counters plus process-lifetime AIS connection attempts, successful streams, throttles, terminal failures, reconnect state, and the same position-freshness telemetry. It also reports timeout, authentication rejection, fallback, served coverage, and RSS feed backoff. AIS snapshot unauthorizedClient is tracked separately from upstream authentication failures so client traffic cannot be mistaken for provider health. The theaterPosture section attributes each successful theater-posture publication cycle to the upstream that actually fed it (opensky, adsb.lol, wingbits, or vessel-only), so healthy publication through the fallback chain is never mistaken for OpenSky recovery; the same source is written as sourceVersion on seed-meta:theater-posture (with a producer field distinguishing the relay loop from the seed-military-flights writer). Unlike the rolling-window route counters, theaterPosture.sourceCountsSinceBoot and emptyRejectionsSinceBoot are process-lifetime because the seed cadence exceeds the metrics window. The aviation section reports which OpenSky route is live — openskyRoute is the effective route (direct or proxy, derived from whether a proxy is actually in the request path), and openskyRouteRequested is what OPENSKY_ROUTE asked for. The two differing is itself the signal that OPENSKY_ROUTE=proxy was set without a usable OPENSKY_PROXY_AUTH/PROXY_URL, so requests silently fell back to direct. Those sit next to the two “blocked” signals that look alike and mean opposite things: openskyRouteRejection counts HTTP 403/451 — the origin refused this exit IP, so switching OPENSKY_ROUTE is the fix — while openskyProviderBlocked reflects a 429 cooldown, which is the account’s 4,000 credit/day quota and cannot be relieved by any route change. Ordinary 401s stay in openskyAuthRejection (rejected credentials); every other relay route still folds 403 into its own authRejection counter. lastRun carries the latest cycle’s source and counts. Published cycles require both the canonical envelope and seed metadata to succeed and include published: true, seededAt, and write outcomes; a replica-envelope failure is exposed through redisOk without hiding a successful canonical publication. Failed canonical or seed-metadata writes include published: false, attemptedAt, and reason: "write-failed", and do not increment the source counter. Rejected zero-input cycles also include published: false and attemptedAt, use reason: "no-input-records", and do not replace the last-known-good envelopes or seed-meta:theater-posture.
Key Classifications
Keys are grouped into three tiers that determine alert severity:Per-Key Statuses
Optional IMD Configuration
IMD is optional whenIMD_API_KEY is absent, including after intentional removal. Its disabled snapshot reports NOT_CONFIGURED, not an all-clear. Invalid configuration after activation and failed requests with credentials remain actionable.
The v2 activation marker requires a successful documented-product response. Valid quiet responses qualify. Total failures and carried-only snapshots do not. The older marker is ignored because failed product batches could write it.
Producers That Serve Last-Known-Good
Taiwan MND records failed source attempts in its source metadata. One failed attempt is pending only while positive last-good records remain fresh. The same error on the next distinct attempt warns. The pending deadline is 210 minutes after the first failure, or the last-good freshness deadline if earlier. A different error does not extend that deadline. Health reads do not advance the failure count. Natural events apply the same bounded source-failure contract to the 15 required NHC forecast-point layers. One transient failed NHC slice is pending only while the aggregate still has positive records and a complete, validated NHC snapshot remains within its original 540-minute retention window. The next failed slice warns, and a changed failure reason does not reset the episode. Missing, expired, malformed, or future snapshot state warns immediately. Complete valid empty NHC coverage replaces the previous storm snapshot. A complete empty result from every provider is healthy, but an empty aggregate with a provider failure is not published. Optional forecast-cone and past-point failures do not remove a storm confirmed by the required point layers. MND retries a list timeout, detail timeout, or missing-publication response once within the existing request and time limits. After a first failed source attempt with usable last-good records, a completed retained-data publication allows one early scheduled recovery attempt after 30 minutes. Before starting that child, the runner atomically claims the existing completion marker withsourceRetryClaimedFor. Its timestamp and TTL stay unchanged. A crash cannot renew that allowance. A completed second failure also blocks renewal, even if the error changes. Normal completion-based admission then applies. This does not change the source success timestamp or extend the health pending deadline. Missing or inconsistent attempt metadata does not grant an early retry. A failed claim keeps normal admission. Incomplete publication retains its existing retry behavior.
Unknown predecessor state stays actionable until a successful source run establishes a new baseline. Compact health retains the SEED_ERROR diagnosis under pending with sourceFailurePendingUntil until it becomes actionable. Dashboard freshness badges retain the diagnosis and data age from both compact maps. Pending does not mean newly published data.
The weather relay records each provider’s success time and consecutive failures in sourceHealth. A first partial failure is pending only when every failed provider has known last-good state and unexpired alerts in the selected payload. The previous envelope and source metadata must have matching publication timestamps; an incomplete pair cannot establish source history. The deadline is the earliest of 20 minutes after the first failure, 45 minutes after the provider’s last success, or the earliest retained selected alert expiry. Aggregate fetchedAt does not renew source freshness. A second failure, total source outage, missing payload, or unusable source evidence warns. Successful empty responses still purge that provider’s old alerts. Failed-source retention drops expired alerts before selection and notifications. The planned standalone writer has no source history and receives no pending allowance.
Some producers are expected to miss individual runs — an LLM synthesis stage
whose provider times out, for example — while the payload they last published
stays useful for hours. For these keys a single miss is not an outage, so the
producer does not write status: "error". Instead it holds fetchedAt at
the vintage it is still serving, reports recordCount for those served
records, and records the miss as diagnostics:
Each such key declares thresholds sized to its own cadence, and health reports
SEED_ERROR once either is crossed: warnAfterConsecutive misses in a row, or
warnAfterAgeMin since the last attempt with a miss on record. Because
fetchedAt is not advanced by a miss, the ordinary maxStaleMin age gate
keeps escalating independently — a producer that stops entirely still ages into
STALE_SEED. A producer that stops after recording a miss therefore reports
SEED_ERROR while retained data is still available. A miss with nothing left to
serve skips all of this and writes status: "error" immediately. What happens
once the canonical data key itself disappears is not specific to this contract —
see the fleet-wide rule below.
Two keys use this contract today: newsInsights and marketImplications.
Producer Faults vs. Missing Data
seed-meta outlives the data key it describes — 7 days against hours for most
canonical keys — so a producer that faulted once and then stopped leaves health
holding two true statements at the same time: the producer is unhappy and
nothing is being served. Every fault signal is affected, not just the
last-known-good contract above: status: "error", a non-ok sourceState, a
blocked source, and a crossed failure streak.
The stronger verdict wins, and ties go to the fault — it is the only one of
the pair that carries a cause. Concretely:
- A blank key whose absence is critical reports
EMPTY(crit), neverSEED_ERROR(warn). Without this rule a vanished homepage panel would report a warning for the seed-meta’s full 7-day life. - A blank key whose absence is not critical keeps
SEED_ERROR. This covers keys where an empty payload is a valid state (OK/STALE_SEED), keys covered by a cascade sibling (OK_CASCADE), on-demand keys (EMPTY_ON_DEMAND), and keys inside a rollout window (ROLLOUT_PENDING) — none of which should be able to silence a fault the producer actually reported.
errorCode is published whenever a fault of the SEED_ERROR kind fired and the
producer recorded a code — including when the missing-data verdict outranks that
fault and the status ends up EMPTY. An escalation to crit never costs you the
reason. (SOURCE_BLOCKED publishes no errorCode; its cause is the status
itself.) lastSynthesisFailureCode is broader still: like the other
last-known-good diagnostics above, it is published on every status whenever the
producer recorded one.
The resilienceStaticIndex check also publishes failedDatasets when its
static-index seed metadata names failed source adapters. This is a validated,
deduplicated list of at most 50 adapter keys. A non-empty list makes the check
SEED_ERROR and remains visible in the public ?compact=1 problem projection.
The sibling resilienceStaticFao check shares the producer heartbeat but does
not inherit failures for unrelated adapters.
JODI China Row
jodiOil, jodiGas and lngVulnerability publish an extra chinaRow block
whenever their producer recorded one:
The block never moves the status. China’s rows going unusable is an upstream
fact no retry clears — every China row in both JODI files has carried
ASSESSMENT_CODE 3 (“null/uncertain”) since at least 2026-08 — so grading it
would be a warning nobody can act on. What it must not be is silent: the
seeders publish the other 50-57 countries regardless of China, and this block
is what names the country that dropped out.
It describes the producer’s last completed publish, like every other
seed-meta-derived field: on a run that refused or failed, the previous
block stays in place and ages alongside seedAgeMin. A producer that reported
status: "error" publishes no block at all rather than relaying a verdict it
never reached this run. The block is also operator-only — it names a
country, so ?compact=1 strips it under the same rule as contentFreshness
and the decision-group breakdown.
Whether the dataset is still advancing is a separate, graded signal: both
seeders declare newestItemAt/maxContentAgeMin from the newest month a
quorum of countries reports a measurement for, so a JODI file that stops
publishing reads STALE_CONTENT rather than passing as fresh, and one
fast-reporting country cannot vouch for a frozen file. Countries whose every
field parsed to null are not published at all, so the seeders’ country floors
count coverage rather than rows.
Do not confuse checks.jodiGas.chinaRow with the checks.chinaCoverage entry
below: the first is one country’s row inside one source, the second is the
fleet-wide China coverage summary.
China Coverage Projection
chinaCoverage projects the hourly Railway summary at
health:china-coverage:v1. The evaluator checks each launched China contract
for both a fresh producer heartbeat and fresh, substantive China content; a
fresh seed cannot hide stale or missing source content. CHINA_DEGRADED is a
warning projection for partial or stale coverage, while CHINA_UNAVAILABLE is
critical when the summary is invalid or the launched content is unavailable.
The final public composition is monitored separately as
chinaDecisionSignals. Its canonical payload must contain all six stable
groups even when individual groups are explicitly unavailable. Health requires
six group records and a seed no older than 60 minutes. Per-source transport
details for policy, exchanges, and cross-Strait publishers remain visible only
in this authenticated operator view; they are not copied into the public
country summary or Pro MCP result.
The six required records count operationally covered groups, which is not
the same as populated groups. A group whose state is unavailable with the
cause healthy_quiet_window is covered: the upstream answered and simply had
nothing qualifying to report, which is not a source failure and needs no
operator action. Its public state and zero-item payload are unchanged — no
event is invented to fill it. Every other unavailable cause
(insufficient_data, provenance_rejected, upstream_unavailable,
unknown) is a real failure and stays uncovered, and a cause that is absent or
malformed fails closed as uncovered.
Both health surfaces publish a group breakdown so a shortfall names its own
work item instead of reading as a bare 4/6: quietGroups (nothing to do),
staleGroups (chase the source’s content), and unavailableGroups, each
carrying the group id and its unavailableCause.
Blocked China contracts remain visible in the audit with their stable reason
code but are excluded from the strict launched-entry health count.
The launched cross-Strait activity contract audits the durable archive
military:cross-strait-activity:v1 independently for producer transport and
latest Taiwan MND reporting-window freshness. A fresh seed with a stale
official report therefore remains degraded. Japan Joint Staff reviewed
observations are regional augmentation and do not satisfy the Taiwan MND
content requirement. /api/health separately monitors
military:cross-strait-activity-bootstrap:v1; fresh canonical data cannot hide
a missing compact UI projection. It also exposes dedicated MND and Japan Joint
Staff transport records. Japan Joint Staff alone reports SOURCE_BLOCKED, and
only when retained reviewed records exist and one of two evidenced conditions
holds. HTTP_403 means the direct request and an upstream response received
after a successful proxy CONNECT both returned HTTP 403 — the publisher itself
refused both paths. PROXY_TARGET_FORBIDDEN means the direct request returned
HTTP 403 and the proxy refused CONNECT for the target while a control CONNECT
to a different contracted host succeeded in the same run through the same
credentials — the proxy provider forbids this destination specifically, so no
configured transport path exists. An uncorroborated CONNECT refusal, a control
tunnel that also fails, PROXY_AUTH_FAILED, stale metadata, a missing source
record, or any other source using the blocked state still fail closed through
the existing STALE_SEED, EMPTY, or SEED_ERROR statuses. The distinction
matters operationally: PROXY_TARGET_FORBIDDEN is durable and needs a
different egress to change, whereas a bare CONNECT refusal is a proxy fault to
remediate. Other current fetch failures
report SEED_ERROR while the last-good archive remains available.
The bundle freshness gate advances only after the archive, projection, and
both source-health records publish successfully. The public bootstrap retains
the bounded reason codes used for disclosure but omits proxy response
diagnostics; full sanitized diagnostics remain in the authenticated operator
source record.
Operators can obtain the same sanitized, read-only audit with
node scripts/audit-china-coverage.mjs --json; add --strict to return a
nonzero exit code unless every launched entry is healthy. The audit reads only
the compact Redis contracts and emits status, age, and reason-code summaries—
never credentials or raw upstream payloads.
Cascade Groups
Some keys use fallback chains. If any sibling has data, empty siblings reportOK_CASCADE:
- Theater Posture:
theaterPostureLive->theaterPosture(stale) ->theaterPostureBackup - Military Flights:
militaryFlights->militaryFlightsStale - Displacement:
displacement(current UTC year) ->displacementPrev(prior year, covers the Jan-1 window before the new-year seed runs)
riskScores is intentionally stricter than a raw feed heartbeat. Its
recordCount is realtime signal-density coverage: the count of score-relevant
Tier-1 conflict, news, and cyber signal families present during the CII refresh.
The conflict family is satisfied by either the ACLED path or the UCDP event
feed, matching the CII v8 scorer. When those feeds are reachable but quiet,
riskScores can still report
COVERAGE_PARTIAL; underlying feed freshness is tracked by the source-specific
health entries where those feeds publish seed metadata.
portwatchPortActivity also uses minRecordCount. A fresh
seed-meta:supply_chain:portwatch-ports record below 174 countries reports
COVERAGE_PARTIAL instead of OK; partial runs may still refresh per-country
PortWatch cache entries, but the canonical country list and healthy seed-meta
signal do not advance until full 174-country coverage returns.
Port activity uses a rolling refresh: at most 30 countries per run, twice daily.
Complete coverage can include retained country payloads strictly below the
seven-day cache expiry; it does not require all countries to match the latest
upstream date in one run. Country timestamps and content clocks remain unchanged
when retained. Coverage diagnostics distinguish currentCountryCount from
retainedCountryCount and expose oldestCountryCacheWrittenAt. Missing or expired
countries and unresolved refresh failures still block a healthy publication.
Unchanged upstream dates also enter the refresh queue before cache expiry. With
174 countries, the lead reserves seven 12-hour runs after allowing two recurring
critical-country slots, plus one missed run. Thus full refreshes become due at
three days, while validated retained data remains usable strictly below seven
days. This can increase activity downloads for unchanged data, within the same
30-country run cap. An already expired backlog still requires successful bounded
refreshes before healthy publication resumes.
portwatchPortActivity additionally requires per-country content
freshness, which is a different question from transport freshness and country
cardinality. The seeder can reuse a cached country payload when upstream
max(date) has not advanced, so a run can report a fresh heartbeat and a
complete 174/174 country list while an individual country’s observation is
days old. The producer therefore publishes a contentFreshness block, and the
check verdict is:
The verdict keys on the decision-critical countries — currently
CN and
HK, the two the China corridor control towers read — rather than on all 174.
Health pins both that set and the 144-hour budget in its own config rather than
accepting whatever the producer declares, so a producer-side change cannot
narrow the alarm scope or widen its threshold silently: dropping CN from the
seeder’s list, or publishing a 30-day budget, would otherwise report OK with
China days stale — the exact failure this check exists to catch. A producer set
that covers the pinned countries and adds more is accepted.
The producer’s counts are a measurement taken at seeder-run time, and seed
metadata is only rewritten on a canonical-advancing 12-hour run. Health
therefore re-ages the oldest decision-critical observation against the
current time rather than trusting the count, so an observation that was inside
budget when the seeder measured it still alarms once it crosses the boundary
between runs. criticalOldestAgeMinutes on the wire is the recomputed age, not
the producer’s. A block claiming every critical country is fresh but carrying no
usable observation timestamp cannot be re-aged, so it reads as unusable.
unusableReasons names which condition failed (declared_scope_narrowed,
fresh_exceeds_covered, critical_observation_time_unusable, …) so a consumer
never has to reconstruct the verdict from the raw counts; expectedCriticalCountries
publishes the pinned scope alongside the declared one.
Because the edge function redeploys within minutes of a merge while the producer
is a 12-hour cron, the absent-block case had a bounded deployment-order grace.
The durable marker
seed-activated:supply_chain:portwatch-ports:content-freshness could grant it
only inside the compiled window 2026-08-03T10:24:42Z →
2026-08-04T06:00:00Z: one complete producer interval plus six hours of
scheduling slack after the schema shipped.
That window has closed, and the PortWatch grace is now permanently spent.
A clean EXISTS=0 no longer softens anything: a missing PortWatch content block
is COVERAGE_DEGRADED, unconditionally. The window is kept in source as the
audit record of what was granted and until when — it is deliberately not
reopened, because the producer has since published the block and re-granting the
softening would undo exactly the bound
#6111 asked for.
In steady state, therefore, contentFreshnessPendingUntil is not emitted at
all. It appears only while some key’s window is open, which today means only
if a NEW entry is added to CONTENT_FRESHNESS_ROLLOUT in
api/_content-freshness.js for a newly deployed content schema. Add one there
and the deadline is published automatically on every surface below; do not
extend the PortWatch entry.
While a window is open, the grace needs positive proof, not merely the
lack of a marker. The EXISTS read is three-valued: read-and-present revokes
the softening, read-and-absent grants it while the window is open, and a read
that failed or returned a malformed pipeline entry is unknown state and grants
nothing. A pending health entry publishes contentFreshnessPendingUntil;
compact health responses repeat the same per-key deadlines in
summary.contentFreshnessPendingUntil, so the bound is auditable without
reading the implementation. /api/seed-health publishes the same deadline on
its entry, and MCP applies the same shared window to its stale boolean while
exposing the optional top-level contentFreshnessPendingUntil field in
cache-tool output. The cache refresh path also refuses to serve a warm snapshot
past this deadline. The softening covers absence only — a block that is
present is always evaluated, and once the marker exists a block that disappears
fails closed.
The cfg.activationKey pending-activation path in /api/seed-health (and the
matching ON_DEMAND policy in /api/health) intentionally remains separate and
does not reuse this PortWatch window. Those markers describe optional or
operator-triggered producers for which “no metadata because the producer has
never run” is the expected state, not a newly deployed data schema; there is no
single producer cadence from which to derive a safe deadline. This path never
softens a meta-bearing content-freshness failure. A scheduled content schema
must use contentFreshnessActivation and its reviewed window instead. This
closes the clean-absent activation gap in
#6111.
Any check whose verdict rested on an unreadable marker carries
activationUnknown: true, on both endpoints and in every status. Without it the
payload is identical whether the marker read failed or the producer genuinely
never published — two different remediations (check Upstash for per-command
errors, versus check why the producer stopped writing). The flag reports which
evidence the verdict used; it never softens or hardens anything by itself.
MCP cache tools publish the same flag on their envelope for the same reason. A
tool that consults an activation marker and cannot read it returns
activationUnknown: true alongside stale, so a caller can tell an unreadable
marker from a producer that regressed — previously that failure was reported
only to Sentry and the two were indistinguishable on the wire. Like
contentFreshnessPendingUntil the field is optional: it is declared on every
cache envelope, but only a tool whose freshness check names a
contentFreshnessActivationKey can ever populate it.
The seeder reserves cold-fetch slots for the decision-critical countries so they
refresh every run they are a cache miss. The content budget deliberately spans
two full producer rotations: ceil(174 / 30) runs at a 12-hour cadence is
about 72 hours, leaving 72 hours of headroom before the 144-hour alarm.
The reservation still matters because it keeps CN/HK from waiting behind the
rest of the bounded queue when many countries become misses together.
The refresh deadline uses the content clock, not the retrieval timestamp. A
successful refetch of unchanged upstream data updates fetchedAt but carries
contentAsOfChangedAt forward, so a frozen feed remains due for another
decision-critical refresh and cannot hide behind a recent retrieval. If the
upstream still does not advance, the check stays STALE_CONTENT: the content
is genuinely old, the seeder cannot fix it, and the alarm reports an upstream
outage rather than an internal rotation delay.
Fleet-wide content staleness is normal by construction: the seeder refreshes at
most 30 countries per 12-hour run, so a full sweep takes about 72 hours; the
two-rotation budget leaves normal tail lag visible without making it an
actionable failure. Alarming on the fleet-wide age would produce a warning
that is always lit and never actionable. The fleet-wide counts
(coveredCount, freshCount, staleCount, unknownCount, a bounded
staleCountries list, and the oldest observation with its age) stay published
for visibility; only the critical subset moves the status.
The 144-hour budget is deliberately the same one
china-corridor-source-adapters.ts applies to a PortWatch observation, so this
alarm explains the China activity-nowcast’s marked_stale exclusion rather
than contradicting it. Health is inclusive at the boundary and treats a
future-dated observation as stale, both one step more conservative than the
data gate — an alarm should fire no later than the contract it protects.
The MCP freshness envelope over the same seed-meta key carries this dimension
too. get_chokepoint_status declares the identical pinned scope, budget, and
activation marker, and calls the same assessor — so it ages the same
contentAsOfChangedAt clock described above, and an MCP consumer cannot read
stale: false for a key this endpoint calls STALE_CONTENT. The parity is
asserted field-for-field against this config rather than left to a comment, and
both surfaces read the activation marker as an EXISTS, so its stored value
can never make them disagree.
/api/seed-health mirrors the same contract on its own supply_chain:portwatch-ports
entry, reporting coverage_degraded where this endpoint reports
COVERAGE_DEGRADED and stale_content where it reports STALE_CONTENT. All
three surfaces read the marker three-valued and grant the grace only on a
read-and-absent result. One test loop drives every marker outcome — read-present,
read-absent, and unreadable — against the block shapes that decide the verdict
(fresh, content-stale, absent, and present-but-unusable) through all three at
once, pinning the expected verdict and the status-name mapping above rather than
mere agreement between them.
An MCP caller still sees a single boolean: stale does not say which
dimension failed, so /api/health remains the surface that names the stale
country.
Both sides age the same clock, and it is not the retrieval timestamp. They read
contentAsOfChangedAt, which advances only when the upstream’s own max(date)
advances, falling back to fetchedAt only for payloads written before that
field existed. This matters because the seeder force-refetches every country
once its cache passes MAX_CACHE_AGE_MS, and that refetch resets fetchedAt
while returning unchanged content. Ageing fetchedAt would therefore have
admitted a frozen observation as current for one budget window out of every
cache lifetime — the same transport-for-content substitution this check exists
to end, one layer below it.
Named entities are operator-only. contentFreshness and the
chinaDecisionSignals group breakdown are stripped from the anonymous
?compact=1 projection, which keeps the status but not which source is
degraded.
predictionMarkets also requires at least one published market in each
geopolitical, tech, and finance pool. Its seed metadata publishes
poolCounts; missing, malformed, or under-floor counts report
COVERAGE_PARTIAL even when the aggregate market count is healthy.
Staleness Thresholds (maxStaleMin)
Earthquake ingestion retries transient USGS or NRCan failures once. The internalseismology:earthquakes:providers:v1 snapshot keeps each provider’s parsed observations and original success clock separately. A failed provider can contribute last-good coverage for less than 30 minutes, provided its content is still within the existing 48-hour budget. Fresh events from the other provider continue to publish.
The first exhausted transient failure remains visible as pending only with usable coverage from both providers. Its deadline is the earlier of 10 minutes after the first failed run or 30 minutes after the oldest provider success. A second failed run for that provider warns immediately. Missing, malformed, expired, or permanently failed coverage earns no pending period. Polling health cannot reset the deadline, and a successful provider fetch resets only that provider’s failure streak. The five-minute cron and existing freshness thresholds are unchanged.
Wildfire ingestion retries a transient CWFIS request once, unless the provider’s Retry-After exceeds the retry budget. The internal wildfire:cwfis-source:v1 snapshot preserves the complete Canadian source and its original success time. If the active layer fails, source data younger than three hours at the source attempt can be reused while fresh FIRMS and BC data continue to publish. A complete empty CWFIS response clears previous records.
The first two consecutive exhausted transient CWFIS failures are pending for at most three hours, bounded by the source’s three-hour retention limit. The third failed run warns. Missing, malformed, expired, or permanently failed coverage earns no pending period. Recovery state must persist before either public wildfire payload changes. Internal snapshot rows are excluded from both public payloads. The ten-minute cron, six-hour stale threshold, and existing FIRMS partial-coverage policy are unchanged.
Selected thresholds from SEED_META:
These are illustrative;SEED_METAinapi/health.jsis the source of truth and each entry documents its own cadence rationale.
Scheduled monitor results
The Seed Freshness Monitor publishes a production acceptance summary on each workflow run and retains its JSON observation for seven days. The summary lists active incidents, acknowledged degradation, stale-content grace deadlines, and baseline entries that no longer appear in health. A successful workflow run can still have active source incidents. The workflow alerts on new or changed incidents and suppresses repeat alerts for unchanged failures. Read the production acceptance summary to determine whether the data passed. An unavailable observation is Unverified, never healthy. Generate the same report locally without publishing GitHub statuses:Example Requests
/api/seed-health
Focused endpoint for seed loop freshness. Checks only seed-meta:* keys without fetching actual data payloads.
Authentication: Requires valid API key or allowed origin.
HTTP Method: GET
Response Status Codes
Response Body
Staleness Logic
A seed is considered stale when its age exceeds 2x the configured interval. This accounts for normal jitter in cron/relay timing. Seeds below an aggregateminRecordCount report coverage_partial and stale: true. Seeds below a subgroup floor such as prediction-market minPoolCounts also report coverage_partial, but retain stale: false while their producer heartbeat remains fresh so freshness and coverage stay distinct.
The commodity-vulnerability cohort is stricter than a non-empty check. The authoritative cohort pointer read by every vulnerability RPC must exist. Its seed metadata must clear the same dimensional floors the producer enforces before publication: country-specific import evidence for at least 110 countries, global production evidence for at least one commodity, at least 110 rankable countries, at least 220 rankable records (two scored commodities per country), at least one fresh rankable record, and the full reviewed commodity, HS4, and transit-HS2 input sets. Registry constants alone do not satisfy these measured coverage checks. The completeCountryCount field (countries with evidence for every reviewed commodity) is published as a diagnostic and does not move the status. The inverse projection must meet its configured chokepoint floor. Any shortfall or missing coverage field reports COVERAGE_PARTIAL; before the first atomic cohort activation, both checks remain rollout-pending.
Consumers: treat status and overall as authoritative for coverage. Do not rely on stale alone — pool shortfalls keep stale: false by design. When either coverage floor fails, the entry also sets coveragePartial: true so clients that only inspect booleans still see the shortfall.
History ingestion (intel-history:*)
Domains prefixed intel-history: do not describe a seeder’s canonical publish. They track whether that collector’s post-publish append to the historical intelligence store is still landing:
The append is fail-open by design — the canonical publish has already committed when it runs, so a failure must never fail the run. That means the collector’s own entry (
conflict:acled-intel, …) stays ok while history silently stops accumulating. These entries are the separate signal:
fetchedAtis the last healthy append, never the last attempt. A run that reached the relay advances it; a run that delivered nothing (every chunk rejected, or the wall-clock budget died before the first request) does not. So a broken relay freezes it and the entry goesstaleon the ordinary 2x-interval rule.status: "error"means the append failed on two consecutive runs — or, on the very first tick, that a relay credential present during an earlier successful append has since been removed.lastErrorCodenames the cause when there is one:http_401,budget_exhausted,all_chunks_failed,config_removed, or a clamped error-class name. Absent when the ingest has never failed.status: "not_configured"means this deployment has never had relay credentials. It is visible but never an alarm — no operator action clears it except provisioning the relay. Losing credentials after a successful append is not this state; it reportserrorwithlastErrorCode: "config_removed".recordCountis the volume the relay accepted on the last successful append. Zero is valid: a run whose records were all deduped still proves the pipeline works.
lastErrorReason, consecutiveFailures, missingConfig, and the inserted/deduped/abandoned counts — is not returned by either endpoint. It lives in the Redis record intel-history:ingest-health:<domain>:<resource>:v1, which the endpoints project from.
A stale or error here alongside an ok collector means canonical data is fine and the history store is the thing to investigate.
Example Request
Integration with Monitoring Tools
UptimeRobot
Use/api/health?compact=1 as the public monitor URL. The HTTP status code only distinguishes a total Redis outage from everything else:
503=REDIS_DOWN(Redis unreachable — a true hard outage)200= every other state, includingDEGRADEDandUNHEALTHY
https://api.worldmonitor.app/api/health?compact=1 and alert when the compact token "status":"HEALTHY" (no space after the colon) is absent from the response body. Compact mode serializes with no indentation, so this exact token is stable regardless of formatting.
This keyword monitor measures availability. A HEALTHY response can still carry contained warnings. A strict data-quality monitor must also alert when summary.warn is greater than zero and inspect problems for the affected sources.
The bare/api/healthURL is now an operator view and returns401without an API key. Public monitoring should always use?compact=1.
Custom Alerting
Parse the JSON response to build granular alerts:Differences Between Endpoints
Capture a source repair
scripts/capture-seed-recovery.mjs records read-only recovery evidence for wildfires, physicalDivergence, and crossStraitActivityTaiwanMnd. It writes a local JSON checkpoint and a Markdown view. It does not run a seeder or change Railway or Redis.
Use Node.js 24, an authenticated Railway CLI, WM_API_KEY (or WORLDMONITOR_API_KEY), and the existing Upstash REST credentials. The command uses loadEnvFile() and reads only the registered source metadata keys. Run it from a checkout that contains the required and deployed Git commits; missing ancestry is unverified.
Use the repair’s merge commit on main, then start before the first run you intend to count. A PR branch SHA can lose its ancestry after a squash merge. Set a concrete UTC deadline once:
Wildfire acceptance targets the primary-retry repair in PR #7701. Its producer emits
trying primary retry. An older producer’s trying secondary message does not prove that repair. Use #7701’s merge commit as the required revision after it merges.
Each observation checks the active image, commit ancestry, and configured cron. Source success must advance after both the baseline and the first observation that verifies that image. Deployment changes reset the run sequence. A source failure or a missed member observation also breaks consecutive proof. Polls with the same success time do not add runs.
The command attributes execution from a container start near its configured cron tick and a matching source completion. Railway documents timing variation of a few minutes. This check allows less than five minutes for the wildfire and macro schedules, and less than two minutes for the more frequent five-minute bundle schedule. This is schedule-alignment evidence; the operator must ensure no manual seeds occur in the window. A parent cron or bundle OK alone is insufficient. The exact next member due time remains unknown until execution evidence exists. Missing, truncated, or ambiguous log evidence cannot pass. Sample after each source run so metadata for an intermediate run is not lost.
Exit codes are 0 for passed, 1 for failed or unverified, and 2 for waiting or observing. An expired deadline remains failed on resume. A passed checkpoint records acceptance at that time; it is not an ongoing health guarantee. These are local report labels and do not change the public health statuses.
For physical divergence, initial recovery does not complete the separate 48-hour schedule check. That check must include two 09:00 UTC ticks and the Sunday 08:00 ordering when applicable. Normal interval-gate skips do not count as successful publications. Keep the JSON checkpoint local: it contains operational identifiers, but no credentials, raw logs, or source payloads.