|
Safe API Framework 0.1.0 build 384e6ae
Layered API framework for safety-related applications (ERTMS RBC reference targeting CENELEC EN 50128 SIL 4)
|
How Platform_RTE supports redundant, voted vital computation: the concepts, what is implemented today, and how the pieces fit in a 2oo2 deployment. This document merges the earlier REDUNDANCY_ARCHITECTURE.dox (a design proposal) and 2OO2_ARCHITECTURE_GUIDE.md (a generic 2oo2 guide). Both described APIs that were never built (rte_vital_channel_*, rte_nonvital_*, rte_cluster_*) or used other names for what is now rte_voter, rte_cross_comparator and rte_channel_link. Their API sketches, the per-CPU hardware walk-throughs and the illustrative timing tables were dropped because they no longer describe the code; the git history of both files keeps them. Real APIs are in the headers under include/rte/redundancy/ and authoritative over anything here. Hardware selection guidance is in HARDWARE_PATTERNS_GUIDE.dox.
| Term | Meaning here |
|---|---|
| Vital channel | A redundant computation whose outputs are compared or voted before they may leave the system |
| 2oo2 | Two channels must agree; any disagreement enters the safe state. Detects a single fault, does not mask it |
| 2oo3 / NMR | Majority vote among 3 or N channels; masks a minority fault |
| 2oo2 with redundancy | Two sites, each a 2oo2 pair; one site is ONLINE, the other STANDBY |
| Hot standby | The standby computes independently on the same inputs and needs no state transfer |
| Warm standby | The standby is partly prepared and receives periodic state; short catch-up on failover |
| Cold standby | State is transferred only at promotion |
The older text used "hot standby" for a standby that only replicates state. In the terms above that is warm.
| Concept | Module | Header |
|---|---|---|
| One redundant link with health tracking | rte_channel (channel link) | redundancy/channel_link/rte_channel.h |
| N-way voting (2oo2, 2oo3, NMR) across registered links | rte_voter | redundancy/voter/rte_voter.h |
| 2-way cross-comparison of two peers, channel or buffer form | rte_cross_comparator | redundancy/cross_comparator/rte_cross_comparator.h |
| Checkpoint rendezvous across channels | rte_checkpoint (rte_channel_checkpoint()) and the app manager's checkpoint stage | redundancy/checkpoint/, app/appmanager/ |
| Data integrity and the vital message envelope (sequence, sender, CRC-64) | rte_checksum | redundancy/checksum/ |
| Fault detection and recovery actions (log, safe state, reboot, failover, custom) | rte_watchdog | redundancy/watchdog/ |
| Site role (ONLINE/STANDBY): inter-site link, start-up negotiation, per-cycle role rules, partner supervision, takeover token check, A/B role agreement, warm-standby sync | rte_site_role, rte_site_role_policy (ADR-040) | redundancy/site_role/ |
| Start-up tie-break and the acknowledged link under the site role | rte_dual_negotiator, rte_dual_channel | redundancy/dual/ |
| State carried across a promotion, declared by the application | rte_state_transfer | redundancy/state_transfer/ |
| Topology, replicas, quorum, standby_mode, capability check | rte_redundancy_config | redundancy/config/ |
| Transition to SAFE or REBOOT | rte_safestate | utils/safestate/ |
| Named channels over pluggable OSAdapters | rte_channel_service | redundancy/channel_service/ |
Design decisions: ADR-008 (common-cause mitigation), ADR-017 (checkpoint and clock sync), ADR-019 (app manager hooks and checkpoint), ADR-020 (dual transfer negotiation), ADR-025 (voter and cross-comparator split), ADR-034 (staged sends and checkpoint signature marks), ADR-040 (site role service).
Supported today.** 2oo2 per site, and 2oo2 with redundancy across two sites with cold or warm standby. rte_voter votes N inputs, but registering an N-way mesh of channels between replicas is not implemented, so a real 2oo3 deployment is not possible yet. Hot standby needs platform-controlled invocation cadence in the app manager and is not implemented. Which combinations an application accepts is decided by its registered capability callback; the Platform rejects the rest at load time.
The reference RBC's cycle, stage by stage, with what is compared and what bypasses the gate: RBC_GP: Site redundancy and A/B agreement, sections 4 and 6. See also docs/architecture/ADR-034-checkpoint-signature-marks.dox.
| Strategy | Rule | Single-fault detection | Fault tolerance | On failure | Typical use |
|---|---|---|---|---|---|
| 2oo2 | Both agree, else fault | Yes | 0 (no masking) | Safe state | Dual channel, dual site pair |
| 2oo3 | At least two agree | Yes | 1 channel | Isolate the minority | Triple channel |
| NMR | Majority of N | Yes | up to (N-1)/2 | Isolate minority | Four or more replicas |
| Non-vital, single | No voting | No | None | Application decides | Logging, diagnostics |
2oo2 is simple, deterministic and biased to the safe side, and it is a good fit for binary decisions such as granting or withholding an authority. Its costs: no masking (a fault means a safe state, so availability suffers), the failed channel must be identified separately, hardware is doubled, and the channels must agree on timing. Use 2oo3 when availability matters more than the cost of a third channel, or add site redundancy to a 2oo2 pair to recover availability.
| Pattern | When it fits | Note |
|---|---|---|
| Shared memory, cache-coherent | Two cores on one die | Lowest jitter; a shared fault domain, so weaker independence |
| Message passing, symmetric | Separate processes or hosts | What the reference RBC uses (ab-peer); the transport is an OSAdapter |
| Asymmetric, one coordinator | A local channel plus a remote one | The coordinator becomes a single point to argue about |
Whichever is used, the comparison must be over data that both channels derived from the same input at the same logical point. That is what the checkpoint provides.
| Mode | Condition | Reaction in Platform_RTE |
|---|---|---|
| Match | Results equal | Continue; outputs are committed at the checkpoint |
| Mismatch | Results differ although both are valid | Safe state; in the reference RBC a REBOOT with own_faulted flushed to the counterpart site |
| Timeout | A peer sends nothing for the configured time | Reference RBC: a checkpoint timeout (SAFE, REQ-CHECKPOINT-003) is turned into a REBOOT by the application's SAFE handler, usually within about 3 cycles; an ONLINE channel that gets no cross-compare sample for 20 cycles while its checkpoint still confirms drops to single mode |
| Corruption | CRC-64 mismatch on a peer sample | Safe state (halt); this is a transmission fault, not a logical disagreement |
Common-cause failure limits all of this: identical CPUs and identical builds can fail together. Mitigation and its limits are in ADR-008 and safety/CCF_ANALYSIS.dox.
Before channels compare data they must be at the same logical point, and nothing may leave until that is established.
Properties this gives: temporal consistency (same point in time), logical consistency (same input state), no contradictory outputs, fail-safe behaviour at every stage, and an audit trail (checkpoint reached, timing, comparison result, commit). The checkpoint is correct without a shared wall clock (ADR-017); clock synchronization is diagnostic only. In the reference RBC the checkpoint budget is shorter than a cycle, so a "lookback" rule also accepts a peer that exchanged a checkpoint frame within the last few cycles, while a silent peer still trips the safe state. On that path the rendezvous proves liveness only; the data comparison is the cross-comparator's (see RBC_GP's site redundancy page, section 6.6).
| Model | Description | In Platform_RTE |
|---|---|---|
| Online (active-active) | All nodes process the same inputs and must agree | Within a site this is the 2oo2 pair. Across sites it would be an active-active design, which is not implemented |
| Standby | One site is ONLINE and outputs; the other is ready to take over | Implemented as cold and warm. Roles are ONLINE, STANDBY and FAULTED, negotiated by rte_site_role (ADR-040) over rte_dual_* |
Behaviour of the reference implementation:
| Requirement area | Implementation |
|---|---|
| Fail-safe architecture | 2oo2 enters the safe state on disagreement |
| Periodic and continuous self-checking | CRC-64 on inter-channel data, watchdogs on links |
| Deterministic timing | Bounded comparison and bounded waits; staged commit |
| Fault tolerance and detection | Two channels, single-fault detection, site failover |
| Traceability | Requirement IDs in code; SRS in docs/requirements/SRS.dox |
| Common-cause failure | ADR-008 and safety/CCF_ANALYSIS.dox |
Standards: EN 50128:2011 (fail-safe architecture, redundancy patterns), EN 50129:2018 (redundancy and 2oo2 voting), CENELEC TR 50128.