RTE RBC 2oo2 Reference Application 0.1.0 build 0006685
SIL 4 2oo2 Dual-Site Fault-Tolerant ERTMS Radio Block Centre Reference Architecture
Loading...
Searching...
No Matches
Timing Drift and Data-Loss Analysis

This document answers two specific questions about Platform_RTE + RBC_GP, raised after the full reboot-recovery work landed:

  1. Is Platform_RTE's timing resistant to clock drift between independently free-running channels (A, B, C, SITE)?
  2. Can a cyclic executive that blocks waiting for data from a dual channel silently miss data?

Both are answered with a derivation grounded in the actual POSIX OSAdapter and application code (not assumptions), with worked numeric examples using this project's real configured constants. Findings are graded Proven issue (a mathematical certainty given the current design/config), Latent risk (possible, not yet certain, depends on runtime conditions), or Not an issue (checked and ruled out).

Everything below was found by reading the actual source, not by pattern matching for "suspicious-looking" code - each finding traces to a specific file and line-level mechanism, cited inline.

1. Timing model: what actually drives a "cycle"

Every role (A, B, C, SITE) is a rte_appmanager_run() loop wrapping a single rte_timer (POSIX OSAdapter: src/posix_osadapter/rte_posix_osadapter_timer.c) that fires a callback which only counts one tick (or the equivalent heartbeat_ticks++ in site.c). The role's own execute() polls that counter in a sleep_ms(10) loop and, once non-zero, does the real work. In A/B the count is ab_gp_cycle_ticks_t since ISS-051 (2026-10-04): the callback calls ab_gp_cycle_ticks_post() and the wait loop ab_gp_cycle_ticks_take(), both under one RTE mutex (com/ab_gp_cycle_ticks.h); before that it was an unsynchronised volatile counter (ctx->cycle_ticks++ against a read followed by = 0U), a data race between the timer thread and the cycle thread. The Gateway services (role C, gateway_c_common.c) still use the plain counter.

Two properties of this timer OSAdapter matter a great deal and are both stated plainly in the file's own header comment - this is not a hidden bug, but its consequences for a long-running system are not spelled out anywhere, which is the gap this document closes:

/* Each period is slept as a fresh relative duration (not measured
* against an absolute deadline), so a concurrent system clock
* adjustment is not compensated for here */
for (;;) {
struct timespec remaining = period_ts;
do { rc = nanosleep(&remaining, &remaining); }
while ((rc != 0) && (errno == EINTR) && (state->should_stop == 0));
...
state->config.callback(state->self_handle, state->config.user_ctx);
...
}

The loop sleeps period_ms, then runs the callback, then loops back to sleep period_ms again - never period_ms minus however long the callback (or the OS scheduler) took. This is the crux of the drift question.

Update 2026-10-04 (Platform_OS_POSIX OSA audit item 8): the loop quoted above was the non-Linux branch; the Linux branch (the Docker target) already slept to an absolute CLOCK_MONOTONIC deadline. Since this date both hosts do: the timer thread reads CLOCK_MONOTONIC once when it starts and moves that deadline one period later before every expiry (src/rte_posix_osadapter_period.c; Linux clock_nanosleep(TIMER_ABSTIME), other hosts nanosleep() of the time left to the deadline, the clock read again after every wake-up). A failed clock read or sleep is reported (DEGRADED, "timer monotonic clock failed - timer stopped") and stops the timer instead of pacing from time 0. See Finding 2 for what this changes.

Update 2026-10-04 (Platform_OS_POSIX OSA audit item 7b-1): the wait to the deadline is now a timed condition-variable wait on one process-wide wake object (macOS pthread_cond_timedwait_relative_np() of the time left, other hosts pthread_cond_timedwait() on CLOCK_MONOTONIC; src/rte_posix_osadapter_period.c, semantics and limits in the file comment of src/rte_posix_osadapter_period.h), so a stop, suspend or destroy wakes the sleeping thread at once instead of after up to one period.

2. Finding: per-cycle timer drift is real and monotonic (not just jitter)

Verdict: Proven issue, currently negligible in magnitude, unbounded in principle. FIXED 2026-10-04 (Platform_OS_POSIX OSA audit item 8): the timer is paced by an absolute CLOCK_MONOTONIC deadline on every host, so tick k is due at t_0 + k*T and neither C_i nor J_i accumulates: a late wake-up or a long callback delays that one tick, and the next deadline is still t_0 + (k+1)*T (if a tick is more than T late, the next one follows at once - late, never skipped). On Linux (the Docker target) this was already the case; the derivation below described the non-Linux branch (development hosts, e.g. macOS) and is kept as the record of the defect. RBC_GP/tests/posix_osadapter/test_rte_posix_osadapter.c checks it on every host (20 expiries of a 40 ms timer with a 20 ms callback end in less than 1100 ms; the drifting loop needs at least 1200 ms). What remains is Finding 3 (scheduling latency itself) and the per-tick jitter J_k, which no longer adds up.

Let T be the configured period, C_k the wall-clock time the k-th callback invocation itself takes to run (including OS dispatch latency to wake the sleeping thread), and J_k any additional scheduler-induced overshoot of nanosleep() itself (always >= 0 - a sleep can be delayed, never shortened). The wall-clock timestamp of tick k is:

t_k = t_0 + sum_{i=1}^{k} (T + C_i + J_i)
= t_0 + k*T + sum_{i=1}^{k} C_i + sum_{i=1}^{k} J_i

Both extra sums are non-negative and never corrected (no absolute deadline is ever recomputed from t_0). This means the process's own tick times drift strictly later relative to an ideal free-running clock ticking at exactly T - a one-directional bias, not mean-zero jitter that averages out. Over a bounded demo run this is invisible (C_k here is a single uint32_t++, sub-microsecond); over the now-intentionally- infinite run this project was just changed to support ("run until Ctrl+C"), the bias accumulates without bound, in principle forever.

In practice, for this codebase, C_k is tiny for the timer callbacks themselves (just a counter increment), so the dominant term is J_k - OS scheduling jitter, not callback cost. That leads directly to the next finding.

3. Finding: no real-time scheduling guarantee in the environment that actually matters

Verdict: Latent risk - correctness on a quiet machine, unverified on a loaded one.

rte_posix_osadapter_task.c's own header says it plainly:

* Priority: applied via pthread_setschedparam() with SCHED_FIFO when the
* process holds the privilege to do so (typically CAP_SYS_NICE or root).
* If that fails, the task still runs - just under the default scheduling
* policy - rather than failing task creation outright, since most
* development/CI environments cannot grant real-time scheduling privilege.

Every background task and every timer thread in this project runs under SCHED_OTHER (best-effort, preemptible, no bounded latency guarantee) unless launched with elevated privilege - which the demo launcher (etc/run_all.sh) does not request. On an idle machine, J_k is sub-millisecond and invisible. Under load (CPU contention from other processes, or - concretely - eight rte_example processes all competing on one host, which is exactly this demo's own topology), the OS scheduler can delay a ready thread by tens or hundreds of milliseconds with no bound enforced or even measured anywhere in this codebase. This directly feeds into Finding 4.

4. Finding: the cross-compare skew tolerance is a heuristic, not a proven bound - and has no self-healing path outside a reconnect

Verdict: Proven design gap.

channel_ab.c tolerates a peer sample up to RTE_EXAMPLE_AB_CYCLE_SKEW_TOLERANCE (3) cycles stale before treating it as too old to compare:

else if (((peer_cycle > cycle) ? (peer_cycle - cycle) : (cycle - peer_cycle))
{ /* skip cross-compare - too stale */ }
#define RTE_EXAMPLE_AB_CYCLE_SKEW_TOLERANCE

At T = 500ms, 3 cycles is a 1500ms window. The file header's own justification is that A and B "start their cycle timers at two different wall-clock moments" and "stay at a roughly fixed relative skew for the rest of the run." That claim depends on an unstated assumption: that A's and B's respective drift rates (Finding 2/3) stay close to each other for the life of the process. Nothing in the code enforces this. If the two processes experience different scheduling pressure (plausible on a loaded host, or simply because A is LISTEN and B is CONNECT-with-retry, so their thread creation/scheduling history already differs), their J_k sums diverge over time and the skew grows monotonically, not just once at startup - contradicting the "fixed skew" assumption the tolerance value was sized against.

Once the skew exceeds 3 cycles, cross-compare is skipped every single cycle thereafter (the condition re-evaluates the same growing gap every time) - this is a permanent, silent loss of cross-compare, not a transient blip, and nothing in this codebase would ever notice or log it as an ongoing condition (see Finding 6 on watchdog blindness).

This session's own reboot-recovery work already had to solve one specific instance of exactly this failure mode: after a peer reconnects post-reboot, its cycle counter restarts at 1 while the survivor's counter is still in the hundreds, permanently exceeding tolerance. The fix (cycle_resync_requested, channel_ab.c) resets the cycle counter to 1 on every successful (re)connection event. That fix is correct for the reconnect case, but it is not a general answer to gradual drift: if the skew grows past tolerance without an intervening disconnect/reconnect (pure scheduling drift over a long quiet run), there is currently no mechanism anywhere that detects or corrects it. The system would silently stop cross-comparing forever, while every other symptom (links up, watchdog kicked, M136 flowing) looks completely healthy.

2026-10-04: cycle_resync_requested was dead since RBC_GP 13e56c6 (no setter) and is removed; a peer reboot now relies on the 60-skip REBOOT (ab_gp_channel_crosscompare.c).

Update 2026-10-07 (state of the code): the tolerance is 20 cycles (ADR-029); option (b) below exists: 60 consecutive skew skips end in react_to_fault(RTE_EXAMPLE_REASON_AB_PEER_LINK_DOWN) (REBOOT), and missed ticks advance the cycle number (Finding 5), so gradual drift no longer silently stops cross-compare for good. The current skip rules are in Site redundancy and A/B agreement, section 6.3.

This is the most important finding relative to your question: Platform_RTE's current skew-tolerance mechanism handles bounded, one-time skew (the startup case, and now the reconnect case) but has no proof, and no runtime enforcement, against unbounded, gradual skew. A rigorous fix would need either (a) a periodic resync signal independent of reconnection (e.g., stamping cycles against rte_timer_now() wall-clock time rather than a free-running local counter, as channel_ab.c's own file header already suggests as the "tighter design" not implemented here), or (b) an explicit alarm when N consecutive cycles have been skew-skipped.

5. Finding: the "wait for tick" pattern silently coalesces missed cycles - proof of guaranteed loss under this project's own configured timeouts

Verdict: Proven issue.

All three cyclic roles share the identical shape (from channel_ab.c, identical in monitor_c.c, and effectively identical in site.c's two-counter variant):

while (ticks == 0U) { sleep_ms(10U); ... } /* ticks: the count the timer callback increments */
ticks = 0U; /* <-- resets to 0, regardless of its value */

(The shape as analysed. A/B no longer reads this way: since ISS-051 the wait loop of ab_gp_channel_wait_cycle_tick() calls ab_gp_cycle_ticks_take() once per 10 ms pass, which returns the pending ticks and clears the count in one critical section, and the ticks beyond the first advance the cycle number by up to 20 instead of being dropped. Role C still has the plain counter.)

If the timer callback fires N times while execute() is still busy with a previous cycle (blocked in a rte_netlink_send()/receive() call, for instance), cycle_ticks reaches N, but the very next check only tests "is it non-zero" and then unconditionally zeroes it. All evidence that N-1 ticks were missed is discarded, with no counter, no log message, anywhere. site.c uses two counters (heartbeat_ticks/heartbeat_ticks_consumed) instead of a reset, but the consuming line ctx->heartbeat_ticks_consumed = ctx->heartbeat_ticks; collapses the same way - it jumps straight to the latest count instead of processing the missed beats one at a time.

This is not a hypothetical edge case for this specific configuration. Compare the numbers already in common_config.h:

A single slow-but-not-yet-dead link (e.g., a peer that's accepted the TCP connection but is momentarily not draining its receive buffer - fully within RTE_STATUS_TIMEOUT's normal, non-error semantics) can make one execute() call take up to 12x the nominal period. During that time the timer thread (running independently) has fired roughly 12 more ticks, every one of which gets silently discarded when execute() finally returns and resets the counter to 0. This is a deterministic consequence of the configured constants, not a probabilistic risk.

6. Finding: watchdog liveness is decoupled from cross-compare success

Verdict: Proven gap (design, not a bug).

channel_ab_execute() kicks the watchdog whenever the send to the peer succeeds (phase 7), independent of whether phase 5 (cross-compare) produced AGREE, a skew-skip, or nothing at all:

status = rte_netlink_send(peer_handle_now, out_bytes, ...);
if (status != RTE_STATUS_OK) { ...; return RTE_STATUS_OK; }
if (ctx->watchdog != NULL) { (void)rte_watchdog_kick(ctx->watchdog); }

So a channel stuck in the Finding 4 "permanently skew-skipping" state still kicks its watchdog every cycle (the link is fine, only the comparison is starved) - the watchdog proves transport liveness, not cross-compare liveness. Combined with RTE_WATCHDOG_ACTION_LOG being the configured action everywhere in this demo (already an acknowledged, documented limitation in the README), a permanently-desynced pair of channels would run "successfully" forever from every externally-visible health signal, while never producing another AGREE or forwarding another answer to C.

7. The core question: can data be lost while waiting on dual channels in the same thread/process? — proof via a counting argument

This is answered precisely, not just qualitatively, because every inter-thread handoff in this codebase uses the same primitive: a depth-1 "latest value wins" mailbox (rx_slot_t in channel_ab.c, answer_slot_t in monitor_c.c). A background rte_task writes into it continuously (as fast as the network delivers); the cyclic execute() reads it once per cycle. This is not a queue - a second write before the first is read destroys the first, unconditionally:

static void rx_slot_write(rx_slot_t *slot, const uint8_t *bytes, size_t len)
{
slot->seq++;
memcpy(slot->bytes, bytes, len); /* overwrites whatever was here */
slot->seq++;
}

Claim. For a depth-1 mailbox with a writer at long-run average rate 1/Tw and a reader at long-run average rate 1/Tr, the fraction of writes that are never observed by any read is exactly:

loss_fraction = max(0, 1 - Tw/Tr)

Proof. Over an interval of length L, the number of writes is W ≈ L/Tw and the number of reads is R ≈ L/Tr. Each read can witness at most one distinct value (whatever is in the mailbox at that instant), so the number of distinct writes ever observed is at most R (pigeonhole - R reads cannot witness more than R distinct values, even if every read happens to catch a different one). Hence the number of writes guaranteed lost is at least W - R, and the loss fraction is at least (W - R)/W = 1 - R/W = 1 - (L/Tr)/(L/Tw) = 1 - Tw/Tr. When Tw >= Tr (writer no faster than reader), R >= W is possible and the bound is vacuous (0) - consistent with "no *guaranteed* loss," though see the jitter caveat below. ∎

This single formula answers the question for every link in the system, because it only needs the two periods:

7a. C → A/B (m136_slot): Tw = 600ms (C's send period), Tr = 500ms (A/B's read period)

Tw < Tr is false (600 > 500), so loss_fraction = max(0, 1 - 600/500) = max(0, -0.2) = 0. No systematic loss possible in this direction - the reader samples faster than the writer produces, so every M136 is guaranteed to be witnessed by at least one A/B cycle (some cycles will re-read the same value twice in a row instead - wasted work, not lost data; visible today as A/B occasionally logging the same D_LRBG for two consecutive cycles).

7b. A → C (rx_a_slot / answer_slot_t): Tw ≈ 500ms (A's send period while AGREEing), Tr = 600ms (C's read period)

loss_fraction = 1 - Tw/Tr = 1 - 500/600 = 1/6 ≈ 16.7%

This is a guaranteed, not probabilistic, loss of roughly one in every six of A's forwarded answers, under ordinary healthy operation (A AGREEing and forwarding every cycle). It is a direct, provable consequence of RTE_EXAMPLE_C_REPORT_PERIOD_MS (600) being larger than RTE_EXAMPLE_AB_CYCLE_PERIOD_MS (500) combined with the depth-1-mailbox design - not a rare race condition, and not something jitter can fix (jitter changes which 1/6 are lost, not whether 1/6 are lost). It is directly falsifiable by comparing consecutive "A's answer (for its own cycle N)" lines in a C log: the printed N should occasionally jump by 2 instead of advancing by 1, each such jump being one silently-dropped intermediate answer.

Whether this matters depends entirely on the intended semantics: for "report the latest known status" (which is arguably what C's ACTIVE/ SUPPRESSED tagging is doing today), losing an intermediate value between two reports is harmless - the next read still gets the current truth. For a system where every individual M24/M15 decision must be durably observed (e.g., an authority that was briefly granted then revoked within that ~1/6-lost window), this would be a genuine, safety-relevant gap - a depth-1 mailbox is the wrong primitive for "every value must be seen," and this project's own doc comments already flag the seqlock as "demo-grade, not formally verified" for a different reason (synchronization safety, not this loss property) - this specific lossy-under-rate-mismatch property was not previously identified anywhere in the docs.

7c. A ↔ B peer link (rx_slot_t): Tw ≈ Tr ≈ 500ms (same nominal rate, independent processes)

With matched long-run rates, the bound above gives loss_fraction = 0 in expectation - over any sufficiently long window, writes and reads balance, so short-term "two writes before a read" events must be balanced by later "read finds nothing new" events elsewhere; the mailbox never falls permanently behind. Individual cycles can still see a transient miss (bounded by jitter, not by a structural rate mismatch), which is exactly the scenario RTE_EXAMPLE_AB_CYCLE_SKEW_TOLERANCE was designed to paper over for the cross-compare logic specifically (Finding 4) - but unlike 7b, this is not a guaranteed, compounding loss.

8. Summary table

# Finding Verdict Root cause
2 Timer drift is monotonic, not self-correcting FIXED 2026-10-04 (was: proven, currently tiny; non-Linux hosts only) Was: nanosleep() relative-duration loop, no absolute deadline; now an absolute CLOCK_MONOTONIC deadline on every host
3 No real-time scheduling guarantee under load Latent risk SCHED_OTHER fallback, no privilege requested
4 Skew tolerance has no defense against gradual (non-reconnect) drift Proven design gap; mitigated (20-cycle tolerance, REBOOT after 60 consecutive skips, missed ticks counted) Was: fixed 3-cycle window sized for one-time skew only
5 Missed timer ticks are silently coalesced/discarded Proven, deterministic given current timeouts cycle_ticks = 0 reset pattern vs. LINK_TIMEOUT_MS >> period
6 Watchdog proves link liveness, not cross-compare liveness Proven design gap Kick point is send-success, not AGREE
7a C→A/B M136 delivery Not an issue Reader faster than writer
7b A→C answer delivery Proven ~16.7% loss Writer (500ms) faster than reader (600ms), depth-1 mailbox
7c A↔B peer sample delivery Not a systematic issue Matched rates; only bounded jitter risk

9. What would actually close these gaps (not implemented here - analysis only, per your request)

  • Replace free-running local cycle counters with wall-clock timestamps (rte_timer_now()) for the cross-compare comparison, removing the dependence on both sides' drift rates staying close (closes Finding 4 for the gradual-drift case, not just the reconnect case already fixed).
  • Track and log/alarm on consecutive skew-skip or stale-tick counts, rather than only reacting to a hard disconnect (closes Finding 4/6's observability gap).
  • Change answer_slot_t/rx_slot_t from depth-1 to a small ring buffer where "every value must be seen" is actually the intended contract (closes 7b) - or, if latest-value-wins is the intended semantic, document that tradeoff explicitly next to the slot definitions the way the seqlock's own synchronization caveat already is.
  • Log (not silently discard) the count when cycle_ticks/heartbeat_ticks is found > 1 at consumption time, so a slow cycle is at least visible (closes Finding 5's observability gap; doesn't by itself fix the underlying coalescing).

10. Finding: the built-in checkpoint rendezvous could not complete within its own steady-state budget on a jittery host — FIXED (lookback)

Verdict: proven; root-caused and fixed 2026-08-28.

The AppManager built-in checkpoint (ADR-019/ADR-034) is a synchronous A→B→A rendezvous run as the LAST per-cycle stage (rte_appmanager.c stage 4). ab_gp_channel_checkpoint_backend_send() puts a CHECKPOINT_REQUEST on the A↔B peer link; the peer can only answer it from its own next pre_execute() pump, so a fresh round trip needs ≥ one full RTE_EXAMPLE_AB_CYCLE_PERIOD_MS (500 ms) — longer than RTE_EXAMPLE_AB_CHECKPOINT_MAX_DELAY_MS's 400 ms steady-state budget. It only ever completed in-window when host scheduling jitter was low enough to hold A's and B's cycle phases aligned so the peer's pump-sent reply happened to land inside the ~133 ms receive sub-window. On a loaded CI runner / VM the phase drifts, the reply lands outside every sub-window, rte_channel_checkpoint() returns RTE_STATUS_TIMEOUT → RTE_SAFESTATE_LEVEL_SAFE → this project's handler reboots (ADR-029 §2.6) → post-reboot negotiation mismatch → reboot again, ~every 20 s (surviving only the 15 s post-reconnect relaxed-budget window). This is the failure mode common_config.h's long RTE_EXAMPLE_AB_CYCLE_PERIOD_MS / ..._CHECKPOINT_MAX_DELAY_MS revert history was fighting; widening the constants was tried repeatedly and reverted (regressed local stability 11/11 → 6/11).

Fix (ab_gp_channel_checkpoint.c, RTE_EXAMPLE_AB_CHECKPOINT_LOOKBACK_CYCLES): the rendezvous now confirms this cycle when EITHER a real matching peer frame is on hand (retained by the per-cycle pump, or live-polled), OR the peer exchanged any checkpoint frame with us within the last N cycles ("lookback" — ctx->last_peer_checkpoint_cycle, a local cycle-counter check, immune to sub-cycle phase). A genuinely silent/partitioned peer still fails the rendezvous after N+1 checkpoint-quiet cycles (~1.5 s at N=2), so REQ-CHECKPOINT-003's bounded reaction is preserved, just measured in cycles. This rendezvous was always predominantly a liveness barrier in practice (the peer always answers a request); same-program-path divergence of actual train/route state is caught independently and every cycle by the cross-comparator (ab_gp_channel_crosscompare.c), not here. Verified: A/B WEST + EAST ran a 5-minute soak and the full rbc_scenario Robot suite with 0 SAFE-states / 0 reboots (previously ~1 every 20 s).

What the rendezvous does and does not prove today, and how a timeout is handled (GP's SAFE handler turns RTE_SAFESTATE_REASON_CHECKPOINT_TIMEOUT into a REBOOT): Site redundancy and A/B agreement, section 6.6.

11. Finding: GP-exe / GA-.so site-route table symbol collision — FIXED

Verdict: proven; Linux-only, macOS-masked; fixed 2026-08-28.

When the old shared ab_site.h was split into GP's full-RailML ab_siteGP.h and GA's reduced ab_siteGA.h, both kept the legacy external symbol names ab_site_track_layout[] / ab_site_track_layout_count for their route tables — with different ab_site_route_layout_t struct definitions (GP's ~9-field RailML entry vs GA's 4-field {id,start_signal,end_signal,length_m}). The GP executable links sa::site::gp and also loads librte_rbc2oo2_ga.{so,dylib} (which links sa::site::ga). Under ELF's flat symbol namespace the GP executable's ab_site_track_layout[] interposes the shared library's copy at load time, so GA's ab_ga_il_site_has_route_by_signals() iterated GP's RailML array through GA's 4-field struct view — every field read at the wrong offset, every route lookup failed "ROUTE_ADD for unknown route … dropping", no MA was ever granted, and the whole IL→RBC→train MA chain (7 rbc_scenario tests) failed. macOS' two-level namespace binds a dylib's own symbols first, so this did not reproduce there — a genuine "works on Mac, broken on Linux" defect.

Fix: GA's table and its element type are GA-namespaced — ab_site_ga_track_layout[], ab_site_ga_track_layout_count, ab_site_ga_route_layout_t (ab_siteGA.h, ab_siteGA.c, ab_ga_il.[ch], ab_ga_ma_logic_adaptation.c). GP's own ab_site_track_layout[] is unchanged and is now internal to the executable (no .so references it). Verified: nm -D shows the .so self-contained for site data; the rbc_scenario suite went 8/16 → 15/16 (the remaining failure is an unrelated pre-existing missing-keyword bug in 16_*.robot).