- Generated by
1.18.0
|
RTE RBC 2oo2 Reference Application 0.1.0 build 0006685
SIL 4 2oo2 Dual-Site Fault-Tolerant ERTMS Radio Block Centre Reference Architecture
|
The detailed narrative of the reference RBC executable: the A/B cyclic executive, cross-compare, West/East negotiation, live failover and its honest limitations. It was written before the GP/GA/SA split and before ADR-037 split role C into per-domain gateway processes, and it is partly outdated. Where this file and ../CLAUDE.md disagree on the current process and binary shape, CLAUDE.md and the contents of dist/<platform>/bin/ win. The current binaries are rte_example (roles A/B) and rte_gateway_c_{train,il,ctc,status} (role C, one c-<site> container runs all four).
The negotiation, failover, cross-compare and A/B cycle descriptions below are historical** (they name files such as channel_ab_negotiate.c, a negotiation-link watchdog and an M24/M15 decision that no longer exist, and they predate the site-role service of Platform_RTE ADR-040). The current, end-to-end description is Site redundancy and A/B agreement; where the two differ, that page and the source win. Main differences: the site role is decided by Platform_RTE rte_site_role (rules 1-7 plus a readiness gate); a silent counterpart never promotes a site (a dispatcher token is needed); A and B agree their startup timestamp over the A/B peer link; the A/B role check is rte_site_role's REQ-SITEROLE-014 check; cross-compare compares the train-session table on its own ab-crosscompare link; both A and B stage outputs.
Running the full stack (Docker Compose, Robot Framework, simulators) is described by RBC_Test_Env (test/RBC_Test_Env/doc/BUILD_GUIDE.dox), not here.
A real Linux/macOS application built on Platform_RTE (Platform_RTE), demonstrating a multi-process, dual-site (West/East) A/B/C redundancy architecture running on a real POSIX OAL OSAdapter — now provided by the sibling Platform_OS_POSIX project, not owned by this repo. This is a separate project from Platform_RTE itself: it consumes Platform_RTE as a dependency, the way a real downstream integrator would, per ADR-005's philosophy that an OSAdapter is an integrator-supplied implementation, not part of the reusable safety framework.
flowchart TB
subgraph WEST["WEST site"]
AW["A"] <-->|cross-compare| BW["B"]
AW <-->|"M136 down, M24/15 up"| CW["C (ACTIVE)"]
BW <-->|"M136 down, M24/15 up"| CW
end
subgraph EAST["EAST site"]
AE["A"] <-->|cross-compare| BE["B"]
AE <-->|"M136 down, M24/15 up"| CE["C (SUPPRESSED)"]
BE <-->|"M136 down, M24/15 up"| CE
end
AW <-->|"negotiate (this channel's own link)"| AE
BW <-->|"negotiate (this channel's own, SEPARATE link)"| BE
Six containers, one per (role, site) pair - no standalone SITE role. West/East negotiation is folded directly into A and B themselves: each of A and B owns its own independent negotiation link to its counterpart on the other site (AB/com/channel_ab_negotiate.c), so site-level negotiation is never a single point of failure the way one shared process would be. A and B cross-check their two independently-negotiated outcomes over their existing same-site peer link - see "West/East negotiation" below.
This project simulates a small slice of an RBC (Radio Block Centre) - A and B are the RBC itself**, a 2oo2 redundant pair. Neither the Interlocking (IL) nor the train talks to the RBC directly in this simulation; C plays that role - the interface to "the rest of the ERTMS system" - originating the train's own Position Report and receiving back the movement authority the RBC hands to IL/the train. Concretely: C periodically generates a simulated M136 (Position Report, D_LRBG = x % RTE_EXAMPLE_DLRBG_MODULUS) and forwards it to A and B; A and B each independently decide M24 (movement authority - D_LRBG < RTE_EXAMPLE_DLRBG_THRESHOLD) or M15 (the more restrictive SR Authorisation - D_LRBG >= threshold), cross-compare that decision (via the framework's rte_cross_comparator, ADR-025 - see "Framework feature adoption" below), and only A forwards the agreed answer back to C, standing in for the movement authority reaching IL/the train. A genuine cross-compare disagreement (not just a link-integrity fault) now escalates all the way to RTE_SAFESTATE_LEVEL_REBOOT and triggers a live ONLINE/STANDBY failover between sites, with the faulted site's own A/B genuinely resuming afterward, not just the site-level tag flipping - see "RBC simulation & live failover" below.
Two binaries, three roles, two sites - 6 containers total.** A and B share one binary/source tree/main() (rte_example, started as rte_example A WEST / rte_example B WEST / etc. - src/application/AB/main/main.c parses argv[1] (role: A or B) and argv[2] (site)); C has its own separate binary (rte_monitor_c WEST, src/application/C/main_c.c, argv[1] is just the site - this binary only ever is C). Both binaries share their common process startup (src/application/app_main_common.c). Each (role, site) pair runs as its own OS process/container (not a thread within one process), communicating over real TCP sockets (rte_netlink) - this is the actual redundancy boundary a real deployment would have, not a simulation of one. docker-compose.yml runs the full 6-container topology with each service on its own container, real Docker networking, and Compose's own DNS resolving one container's target host to another by service name - see "Running" below. The same binaries also still run as plain local OS processes (the etc/run_all.sh/.claude/skills/run-RBC_GP/smoke.sh path), all on 127.0.0.1, on distinct ports (src/application/site_config.h) to avoid colliding with each other - every CONNECT-role link's target host is resolved through site_config_host(), which defaults to 127.0.0.1 when the matching RTE_EXAMPLE_*_HOST env var is unset (the local/manual path) and Compose sets to the peer container's own service name (the container path) - see common_config.h's own doc on those env vars.
A and B (src/application/AB/main/channel_ab.c), per site: a real cyclic executive - rte_appmanager_run() loops forever (max_iterations = 0), one cycle per simulated 500ms cycle, paced by a real periodic rte::timer (not an ad hoc sleep loop). As of the framework's ADR-019, the eight phases below are split across three rte_appmanager hooks instead of one undivided execute(), so each hook's own job stays visible in its name: pre_execute() covers read channels (read the most recently received M136 from C and decide this cycle's own M24/M15 answer), check timer expiration (rte_watchdog_timer_tick()), check tasks (read whatever a background rte::task has most recently received from the peer - see below), and process data (CRC-64 decode/verify it); execute() covers only cross-compare (AGREE/DISAGREE against the local decision); post_execute() covers prepare data for sending (encode this cycle's own sample, plus the C-forwarding answer if AGREE and role A), dual transfer (send it to the peer over the same full-duplex link the background task receives on), and send to C (only on AGREE, and only from role A - see below). Results are threaded between the three hooks via ctx->cyc_* scratch fields on channel_ab_context_t since they no longer share one function's stack frame. See channel_ab.c's file header for the full phase-by-phase breakdown. A single-channel fault is deliberately injected on role A only, at exactly D_LRBG == RTE_EXAMPLE_DLRBG_FAULT_VALUE (300): A computes the wrong answer (M15 instead of the correct M24) even though its own CRC-64 still verifies fine.
Also as of ADR-019, A/B's rte_appmanager_config_t::checkpoint is wired in: before pre_execute() each cycle, rte_appmanager_run() calls rte_channel_checkpoint() (ADR-017) against a rte_vital_channel_t wrapping the same peer TCP link cross-compare already uses - not a second connection. Because that link's message_size is fixed and a background task already owns its only reader, checkpoint traffic is multiplexed onto it: every frame now carries a 1-byte kind tag (PEER_MSG_KIND_AB_SAMPLE/ _CHECKPOINT_REQUEST/_CHECKPOINT_REPLY), and the background reader (peer_rx_task_entry()) also auto-responds to the peer's own _REQUEST with a _REPLY (nothing in rte_checkpoint generates that reply automatically). A pthread_mutex_t (peer_send_mutex) guards every send on that link now that both the main thread and the background thread can send on it. rte_vital_channel_init()'s channel_count floor was relaxed (framework-side, ADR-019 addendum) to >= 1 to fit this single-physical-link topology. checkpoint is paused (vital_channel set to NULL) whenever the peer link itself is known down, deliberately not treated as a checkpoint-timeout SAFE-halt - see docs/architecture/ADR-019-...md §5.3 in the framework repo for the full reasoning. Live-verified: the 8-process WEST+EAST demo runs cycle-over-cycle with AGREE/0 errors and no checkpoint timeouts.
Structured message-trail logging:** every A/B/C application log - AB_SAMPLE sends/receives, M136 reports, checkpoint REQUEST/REPLY, AGREE/DISAGREE decisions, answers forwarded to C, each channel's own inter-site negotiation traffic, and every generic diagnostic (init/shutdown, link/task/timer/watchdog failures, role/mode transitions) - goes through the framework's rte_log_write_event() instead of the free-text rte_log_write(), producing a fixed, space-separated Key=Value line per event - Site=<site> Timestamp=<ms> Level=<LEVEL> Cycle=<n> Source=<src> Destination=<dst> Type=<type> Info=<info>[ <extra_fields>] (e.g. Site=WEST Timestamp=100022194 Level=INFO Cycle=3 Source=A/WEST Destination=B Type=AGREE Info=M24). Site is this process's own WEST/EAST identity (site_config_name(ctx->site)), first in the line so a log consumer merging output from all 6 processes can always tell them apart at a glance; Cycle=0 marks an event with no natural cycle (link/task/timer lifecycle events). Events with no natural cycle still carry a Type category (LINK, TASK, TIMER, WATCHDOG, MODE, ROLE, LIFECYCLE, NEGOTIATION, NEGOTIATION_MISMATCH, CHECKPOINT, XCOMPARE, TRANSFER) so they remain filterable even without a cycle number. Both the console printf trail and this structured OSAdapter line are still emitted side by side - the former for a human watching the terminal, the latter for a machine parsing the Platform_RTE-side log output. Framework-internal diagnostics (e.g. rte_watchdog.c's own "watchdog created"/"watchdog timeout" lines) are outside this application's log calls and are unaffected.
Since cross-compare already confirms A and B agree, only A forwards the agreed answer to C - B's own agreeing answer would be redundant. A genuine CRC-clean logical DISAGREE (the M24/M15 decision itself differs, not link corruption) is judged more severe than an ordinary value mismatch: both A's and B's own channel independently detect it and both set ctx->own_faulted (in-memory, no shared filesystem between containers), flush that fault to their own inter-site counterpart one last time before their own links close, and enter RTE_SAFESTATE_LEVEL_REBOOT - see "RBC simulation & live failover" below for what happens next.
The peer receive, and the M136 receive from C, are both asynchronous: background rte::task workers continuously receive from each link and hand each message to the cycle loop through a small lock-free seqlock-style slot (rx_slot_t in channel_ab.c) rather than blocking the 500ms cycle on the network. Because A and B each run their own independent, unsynchronized 500ms timer (no shared clock between them - rte::clocksync exists in the framework and would be the natural next step) - and, on top of that, A (LISTEN) and B (CONNECT-with-retry) start their own cycle timers at two different wall-clock moments during init() - their cycle counts settle into a roughly fixed relative skew for the rest of the run, not just occasional jitter. Cross-compare tolerates a peer report within RTE_EXAMPLE_AB_CYCLE_SKEW_TOLERANCE cycles of the local one (rather than requiring an exact match) so this fixed skew doesn't permanently starve one side of any comparable data; right at the D_LRBG=100 or D_LRBG=300 boundary, this tolerance can in principle produce a spurious DISAGREE (or fail to reproduce a real one) purely from that skew - an honest, documented edge case, not a bug (see channel_ab.c's file header).
Separately from the logical decision compare, the A<->B peer link itself carries a CRC-64 (rte::checksum) over each sample - a transmission integrity* check. If the CRC-64 doesn't match what actually arrived (link corruption, not a legitimate decision disagreement), the receiving side calls RTE_SAFESTATE(RTE_SAFESTATE_LEVEL_SAFE, ...) (ADR-004) and halts - no handler is registered for SAFE (only for REBOOT - see below), so this is the framework's default defensive infinite-loop halt, not a clean abort()/exit. Only the A<->B link is CRC-64 protected; the A/B<->C links are not (see channel_ab.c's file header for the scope note). Every cycle that completes a send to the peer kicks a real rte::watchdog armed in init().
Because A/B now run forever, something has to be able to stop them: main.c calls rte_appmanager_install_default_signal_handlers() - Platform_RTE's own SIGINT/SIGTERM -> rte_appmanager_request_shutdown() convenience - for a graceful stop (all links, the watchdog, the cycle timer, and the background tasks are closed down properly, not just killed) - see Running below for how the launcher uses this. Each role module (channel_ab.c/monitor_c.c/site.c) only builds what it does each cycle via a *_make_config() function; main.c is the one place that actually calls rte_appmanager_run(), for whichever role was selected. main.c also registers a RTE_SAFESTATE_LEVEL_REBOOT handler once, for every role, that calls rte_reboot_request().
There is no standalone SITE role anymore - West/East negotiation is folded directly into A and B themselves, see "West/East negotiation" below.
A LISTENs for B on the A/B link; both A and B CONNECT to C. Since container/process start order across the 6 independently-launched (role, site) pairs isn't guaranteed, the CONNECT side retries on connection refusal for a bounded window (RTE_EXAMPLE_CONNECT_TIMEOUT_MS in src/application/common_config.h) rather than treating an early refusal as fatal - the same pattern each channel's own EAST-side inter-site negotiation link uses (below).
Historical; current behaviour: Site redundancy and A/B agreement, section 5.*
No standalone SITE role: each of A and B owns its own, independent inter-site negotiation link to its counterpart on the other site - A-WEST<->A-EAST and B-WEST<->B-EAST are two SEPARATE TCP links, each wrapped in a real rte_dual_channel_t + rte_dual_negotiator_t (ADR-020), so site-level negotiation is never a single point of failure the way one shared process/link would be (src/application/AB/com/channel_ab_negotiate.c). WEST always LISTENs on its own negotiation port (RTE_EXAMPLE_A_NEGOTIATION_PORT/ _B_NEGOTIATION_PORT - fixed ports, not per-site offsets); EAST always CONNECTs (a fixed technical role, unrelated to which site ends up online), retrying for up to RTE_EXAMPLE_CONNECT_TIMEOUT_MS since container start order isn't guaranteed. The negotiator's own older-startup-timestamp-wins tie-break (own_id/peer_id as the deterministic fallback on an exact tie) settles each link's own startup ONLINE/STANDBY outcome independently.
Because A's and B's own negotiation links are genuinely independent (different TCP connections, different negotiator instances, started at slightly different wall-clock moments), they can occasionally settle on different* outcomes for the same site - a real, if narrow, race, not a bug. A and B guard against acting on a disagreed site-role by cross-checking their own independently-negotiated {is_online, single_mode} every cycle over their EXISTING same-site peer link (PEER_MSG_KIND_SITE_STATE - channel_ab_crosscompare.c): a mismatch is itself treated as a fault (own_faulted, RTE_SAFESTATE_LEVEL_REBOOT, RTE_EXAMPLE_REASON_AB_SITE_STATE_MISMATCH) - a deliberate judgment call trading a little availability (occasionally both channels reboot and re-negotiate before settling) for never letting A and B act on a disagreed site-role. See "Honest limitations" below for how often this is actually observed.
Both sites always run their full A/B/C group regardless of the negotiated outcome. Only each site's own C tags its reported output ACTIVE (this site is ONLINE) or SUPPRESSED (STANDBY, or FAULTED - see below); A/B's cross-compare logic is identical either way. C infers this live, every cycle** from whether its own site's A/B are actually forwarding answers at all (see C's own description above), starting from a fail-safe SUPPRESSED default until the first real answer (or lack of one) arrives - this is what makes a live failover (below) visible without restarting C, and needs no shared filesystem between C and its own site's A/B containers.
Two Platform_RTE primitives that used to have no consumer in this example are now load-bearing here, replacing hand-rolled logic that used to duplicate what they already do:
Historical; current behaviour: Site redundancy and A/B agreement, sections 5 to 7. In particular the negotiation-link watchdog of point 6 no longer exists: a lost inter-site link never reboots or promotes a channel by itself.*
The one-time negotiation above only decides who starts ONLINE. Live failover afterward - "current online goes to safe state and reboots, the other site becomes the new online and continues, and the faulted site itself recovers back into the mix" - works through six cooperating pieces, all now living directly inside channel_ab.c/ channel_ab_negotiate.c (no separate SITE process to relay any of it):
x (C's D_LRBG report counter) only reaches RTE_EXAMPLE_DLRBG_FAULT_VALUE once every RTE_EXAMPLE_DLRBG_MODULUS reports; set RTE_EXAMPLE_C_POSITION_START (an environment variable, e.g. a value close to the fault value) to start C's own counter closer to it for a quick demo/verification run, without changing the generation formula itself.
Each (role, site) pair is also just a plain OS process; run six instances across the two binaries, in any order, in separate terminals. Every role now runs forever (its own real 500ms/600ms cyclic executive - see Architecture above) until sent SIGINT/SIGTERM (Ctrl+C, or kill):
(<platform> is CMAKE_SYSTEM_NAME-CMAKE_SYSTEM_PROCESSOR, e.g. Darwin-arm64.)
Or use the launcher, which starts all six and then runs indefinitely - useful for watching the RBC fault/failover trigger on its own (see "RBC simulation & live failover" above), or forcing it quickly with RTE_EXAMPLE_C_POSITION_START. Ctrl+C (or kill/SIGTERM) stops everything gracefully via the script's own trap:
To see the RBC fault/failover scenario without waiting for x % RTE_EXAMPLE_DLRBG_MODULUS to wrap around to the fault value on its own, set RTE_EXAMPLE_C_POSITION_START close to it, e.g. via etc/run_all.sh (applies to both sites' C, since both share the launcher's environment):
Starting both sites' C at the same offset faults both sites within a cycle or two of each other - this is the realistic long-run case (see "RBC simulation & live failover" above for why) and exercises the full one-side-faults / other-side-faults / both-recover cycle, not just a single-site failover. To see only a single site fault in isolation instead, start just that site's C process directly with the same environment variable set, leaving the other site's C at its default.
This is a demonstration, not a certified safety application:
See ../../platform/Platform_RTE/docs/architecture/ADR-018-posix-oal-osadapter.dox (kept there as a historical record; its content now applies to src/posix_osadapter/ in this repository - see that file's own header for the relocation note) for what the POSIX OSAdapter itself does and does not guarantee.
1.18.0