ADR-017: Checkpoint Rendezvous and Clock Synchronization for Distributed Vital Channels

Status: Draft Date: 2026-08-05 Applies to: rte_checkpoint (new), rte_clocksync (new), and how both plug into the existing rte_channel / rte_checksum / rte_ipc / rte_watchdog modules.

1. Context

The vital channels that vote on a result (rte_channel) can run on separate physical machines, potentially in separate geographic locations (the repo already has a 2oo2-geographic-redundancy example for active/standby site failover). This ADR addresses a different problem: within a single voting cycle**, how do independently-clocked, independently-scheduled channels agree on which cycle's results they're comparing, and what happens if a peer's result doesn't show up?

True wall-clock simultaneity across a network is not achievable and is the wrong safety argument to depend on. What this ADR actually delivers is a bounded-time rendezvous on a logical checkpoint ID, which is the standard way distributed safety systems make this work: two channels are treated as "at the same point" not because their clocks agree, but because they both reached checkpoint N and confirmed it to each other within a fixed timeout — and if that confirmation doesn't happen in time, that's a fault, handled the same way any other vital fault is (safe-state / watchdog action), not silently ignored.

This closes three things already flagged elsewhere in this repo as designed-but-not-built:

• docs/REDUNDANCY_ARCHITECTURE.dox already specifies a rte_channel_checkpoint() / rte_checkpoint_config_t API (checkpoint ID, max delay, expected node count) in prose and example code, but grep -rn "rte_channel_checkpoint\|rte_checkpoint_config_t" include src returns nothing — it was never implemented.
• include/rte/watchdog/rte_watchdog.h already has a RTE_WATCHDOG_CHECKPOINT enumerator, unused by any real checkpoint logic.
• ADR-001's layered diagram names an empty "L1 Safety Communication Layer (future ADR)" slot between the OAL (rte_ipc/rte_timer) and the application layer (rte_channel lives here); this ADR is that future ADR.
• ADR-008 (rte_channel, a local two-channel comparator written before rte_channel's fuller 2oo2/2oo3/NMR implementation was known about) explicitly deferred "EN 50159-style message integrity if channels run on separate physical nodes... expected to build on <tt>rte_ipc</tt> in a future ADR." This is that ADR. rte_checkpoint and rte_clocksync are designed to plug into rte_channel (the more complete, actively-developed implementation), not rte_channel.

2. Decision

2.1 Reuse the EN 50159 envelope that already exists — don't rebuild it

rte_checksum.h's rte_vital_message_t + rte_checksum_vital_message_create() / rte_checksum_vital_message_verify() already implement most of what an EN 50159-style message envelope needs: a monotonic sequence_number (defends against repetition, deletion, insertion, re-sequencing), a sender_id, a timestamp_ms, and CRC-64 (defends against corruption). rte_checkpoint (below) uses this wire format directly for the checkpoint-arrival handshake instead of inventing a second envelope format — the earlier plan to add a standalone rte_synclink module is dropped for exactly this reason.

2.2 rte_checkpoint: bounded rendezvous, no new transport backend

rte_checkpoint adds no new backend of its own. It operates on an already-initialized rte_channel_t * and reuses that instance's already-registered backend_send/backend_recv callbacks (which is how "pluggable" is achieved — whatever transport the integrator registered for voting, e.g. rte_ipc over POSIX/RTOS/a real network link, is what checkpoint messages travel over too). This keeps the whole design consistent with ADR-005's established pattern instead of adding a parallel plug-in point.

rte_status_t rte_channel_checkpoint(rte_channel_t *handle,
                                       const rte_checkpoint_config_t *config);

rte_checkpoint_config_t keeps the field names already used in REDUNDANCY_ARCHITECTURE.dox's examples (checkpoint_id, max_delay_ms, expected_node_count) so those examples become literally true instead of needing to be rewritten, plus an optional watchdog handle (rte_watchdog_t, may be NULL) that gets kicked on a successful checkpoint.

Algorithm: broadcast a small checkpoint-arrival message (checkpoint_id as both the rte_vital_message_t sequence number and payload) to every channel via the vital channel's own backend_send, then backend_recv-poll each channel bounded by max_delay_ms, verifying each reply with rte_checksum_vital_message_verify() and discarding any that fail CRC or carry the wrong checkpoint_id (defends against corruption and cross-checkpoint mixups — a corrupted or stale reply must never count toward quorum). If at least expected_node_count valid replies arrive in time: kick the optional watchdog and return RTE_STATUS_OK. Otherwise: call rte_safestate_enter(RTE_SAFESTATE_LEVEL_SAFE, ...) directly (the same pattern rte_channel_receive() already uses on a voting disagreement — see its own doc comment) and return RTE_STATUS_TIMEOUT. RTE_WATCHDOG_CHECKPOINT now has real behavior behind it: an integrator can additionally configure a watchdog of that type with its own independent action (log/safe-state/reboot/failover) for liveness monitoring across many checkpoints, on top of the immediate safe-state entry a single failed checkpoint already triggers.

2.3 rte_clocksync: diagnostic only, never the basis of correctness

A small pluggable backend (rte_osadapter_clocksync_t + rte_osadapter_clocksync_register(), matching rte_timer's convention exactly) exposing rte_clocksync_get_offset_ms() and rte_clocksync_get_quality(). An integrator implements this against whatever real synchronization mechanism their hardware has (PTP, GPS discipline, NTP, or a custom link). Its purpose is diagnostics (log correlation across sites) and sizing max_delay_ms for a given deployment's known clock/network jitter — not determining whether two channels are "at the same point in time." That correctness argument comes entirely from section 2.2's checkpoint-ID rendezvous, which works even with zero clock synchronization. This is stated explicitly in the header so nobody later assumes "clocks report low offset" implies "results are simultaneous" — they don't; only a confirmed matching checkpoint ID within the timeout does.

3. Consequences

• Positive: no new envelope format, no new transport backend — both new modules are thin, and reuse everything already built and tested (rte_checksum, rte_ipc's backend registration, rte_watchdog, rte_safestate).
• Positive: REDUNDANCY_ARCHITECTURE.dox's existing usage examples (checkpoint_id/max_delay_ms/expected_node_count) become real rather than illustrative-only.
• Negative / scope limit: rte_clocksync provides no protocol implementation (no bundled PTP/NTP client) — only the pluggable interface and the explicit warning against misusing it for correctness. A real clock-sync protocol implementation is integrator-supplied, same as every other OAL backend.
• Negative / accepted risk: rte_checksum's sender_id is not cryptographically verified (no MAC/signature); on an open transmission system (untrusted network) this is a real masquerade-defense gap, not fully closed by this ADR. Flagged here rather than silently assumed away — a future ADR should address authenticated messaging for open transmission deployments per EN 50159 §5 if that deployment case is needed.
• Deferred: rte_checkpoint's relationship to rte_channel (ADR-008) is not resolved by this ADR — rte_channel remains a working, simpler same-build two-channel comparator; whether it should be deprecated in favor of rte_channel + rte_checkpoint is a separate decision for the team, not made here.

4. Location

include/rte/checkpoint/rte_checkpoint.h + src/checkpoint/rte_checkpoint.c (target rte::checkpoint, links rte::vital_channel, rte::checksum, rte::watchdog, rte::safestate). include/rte/clocksync/rte_clocksync.h + src/clocksync/rte_clocksync.c (target rte::clocksync, links rte::status/rte::types only — no dependency on checkpoint or vital_channel).