Status: current description, 2026-10-07, checked against the source of RBC_GP at G1 step 5b-2 and Platform_RTE at G1 step 5b-3a (ADR-040). This page is the one end-to-end description of how a site becomes ONLINE or STANDBY and of how channels A and B of a site check each other. Other pages link here instead of repeating it. The generic service (rte_site_role, its API and its rules as code) is described from the Platform_RTE side in ADR-040, "Service behaviour as built"; the rule list in rte_site_role_policy.h is authoritative where the two differ. Section 10 keeps the 2026-09-24 design notes (link phase problem, options, standby rules) for the record.
Written for: the RBC product owner, reviewers and anyone changing ab_gp_channel_negotiate.c, ab_gp_channel_crosscompare.c, ab_gp_channel_checkpoint.c, pre_execute.c or post_execute.c.
1. The system in one picture
Two sites, WEST (site id 0) and EAST (site id 1). Each site runs two vital channels A and B (2oo2) plus the non-vital Gateway (role C) services. One site is ONLINE and serves the trains, the other is STANDBY.
flowchart LR
subgraph EXT[Outside world]
TR[Trains]
IL[Interlocking]
CTC[CTC / dispatcher]
end
subgraph WEST[Site WEST]
CW[Gateway C]
AW[Channel A]
BW[Channel B]
end
subgraph EAST[Site EAST]
CE[Gateway C]
AE[Channel A]
BE[Channel B]
end
TR --- CW
IL --- CW
CTC --- CW
TR --- CE
IL --- CE
CTC --- CE
CW <-->|ab-relay-*| AW
CW <-->|ab-relay-*| BW
CE <-->|ab-relay-*| AE
CE <-->|ab-relay-*| BE
AW <-->|ab-peer + ab-crosscompare| BW
AE <-->|ab-peer + ab-crosscompare| BE
AW <==>|inter-site link A| AE
BW <==>|inter-site link B| BE
| Link | Between | Carries | Opened by |
| ab-peer | A and B of one site | SITE_STATE (3 bytes, every cycle), CHECKPOINT_REQUEST/_REPLY, STARTUP_TS, SAF_RB, KM114_DIGEST (PEER_MSG_KIND_*, com/ab_gp_channel_types.h) | ab_gp_channel_open_link(), main.c step 1 |
| ab-crosscompare | A and B of one site | AB_SAMPLE only (the cross-compare sample) | same |
| inter-site link | A/WEST and A/EAST; B/WEST and B/EAST (two independent links) | rte_dual_negotiator STATE beacons and the site-role DATA frame (section 5.4) | rte_site_role_start(); WEST listens, EAST connects; fixed ports RTE_EXAMPLE_A_NEGOTIATION_PORT (15021) and _B_ (15022) from common_config.h. The ab-negotiate entry of etc/config/*.json is not used to open it |
| ab-relay-* | A, B and their own Gateway | train, IL, CTC and UTIL traffic; the dispatcher's takeover token arrives on UTIL | ab_gp_channel_open_link() |
2. Who does what
| Concern | Owner | Where |
| Inter-site link, start-up tie-break, per-cycle role decision, partner supervision, reconnect, takeover token check, warm-standby sync, FAULTED flush | Platform_RTE rte_site_role and rte_site_role_policy (ADR-040) | src/redundancy/site_role/ |
| A/B role agreement check (REQ-SITEROLE-014/016): the rule, the 3-strike count, the NEGOTIATION_MISMATCH log | Platform_RTE rte_site_role | check_sibling_after_decision(), rte_site_role_check_sibling() |
| Startup timestamp agreed between A and B | RBC_GP | exchange_startup_timestamp() (com/ab_gp_channel_negotiate.c), rule in com/ab_gp_startup_timestamp.c |
| Transport of the sibling's site state to the service | RBC_GP | send_site_state() (main/post_execute.c), ab_gp_sibling_state_store() / _take() (com/ab_gp_sibling_state.c) |
| Readiness of this channel | RBC_GP | ab_gp_site_status_ready_for_online() (com/ab_gp_site_status.c) |
| What a promotion takes over (sessions, route db, cycle number) | RBC_GP | on_role_change(), apply_state_transfer() (com/ab_gp_channel_negotiate.c) |
| Reaction to an A/B role disagreement (reboot) | RBC_GP | ab_gp_channel_crosscompare_check_site_state() -> react_to_fault() |
| A/B cross-comparison: what is compared and when | RBC_GP | com/ab_gp_channel_crosscompare.c |
| The comparison itself and the REBOOT on DISAGREE | Platform_RTE rte_cross_comparator (ADR-025) | rte_cross_comparator_execute_buffers() |
| Checkpoint rendezvous, commit or discard of staged output | Platform_RTE app manager and rte_checkpoint (ADR-017/019/034); RBC_GP supplies the transport | com/ab_gp_channel_checkpoint.c, ab_gp_channel_on_checkpoint_result() |
3. Start-up of one channel
rte_gp_run() (main/main.c) runs the setup steps in this order; a FATAL step that fails ends the process (exit 1).
- ab_gp_channel_open_link() (FATAL): ab-peer, ab-crosscompare and the relay links.
- Train codec, GA, relay queues, train-link SaF (FATAL except the queues).
- ab_gp_channel_negotiate_setup() (FATAL), i.e. ab_gp_channel_negotiate_init():
- Startup timestamp exchange with the sibling (exchange_startup_timestamp(), ADR-040 step 5a), section 5.2. It is the only reader of ab-peer at this point; it runs before the watchdog exists, so it is bounded by construction (AB_GP_STARTUP_TS_MAX_SLICES read slices).
- rte_site_role_init() with GP's configuration: own id = site, WEST listens / EAST connects, A port 15021, B port 15022, ack_timeout_ms = neg_ack_timeout_ms (default 50 ms), send_miss_threshold 3, per-cycle reconnect attempt 50 ms (AB_GP_NEG_RECONNECT_ATTEMPT_MS), start-up bound RTE_EXAMPLE_CONNECT_TIMEOUT_MS (10 s), connect retry 100 ms with 500 ms per attempt, standby_sync = standby_mode is warm or hot, the agreed startup timestamp, the sibling source (sibling_source() -> ab_gp_sibling_state_take(), reach window 0 = RTE default 3 cycles) and, with RTE_TAKEOVER_KEY_FILE, the takeover MAC verifier (an unusable key file refuses to start).
- rte_site_role_start(): inter-site start-up negotiation, section 5.3. Not settled within the bound: the step fails and the process exits. A channel therefore cannot start while the other site is unreachable; in Docker the container is restarted (root ISSUES.dox ISS-009 describes this during a partition).
- ctx->is_online = the start-up role; an ONLINE start opens the 40-cycle join window (AB_GP_JOIN_ADOPT_CYCLES, section 6.3).
- Checkpoint, dual-transfer and watchdog setup (non-fatal: a failure runs the channel without them, root ISSUES.dox ISS-070). The checkpoint starts with the relaxed budget (RTE_EXAMPLE_AB_CHECKPOINT_RELAXED_MAX_DELAY_MS, 10 s) for RTE_EXAMPLE_CHECKPOINT_RELAX_DURATION_MS (15 s), then 400 ms.
- ab_gp_channel_crosscompare_setup() (FATAL): an rte_cross_comparator configured with RTE_SAFESTATE_LEVEL_REBOOT and reason RTE_EXAMPLE_REASON_AB_MTYPE_DISAGREE.
- Cycle timer (FATAL), then rte_appmanager_run().
4. One cycle, stage by stage (500 ms, RTE_EXAMPLE_AB_CYCLE_PERIOD_MS)
The app manager runs pre_execute, execute, post_execute, then the checkpoint stage and on_checkpoint_result. Every step below is in this order in the source.
Wait for the tick** (ab_gp_channel_wait_cycle_tick(), inside pre_execute): sleeps in 10 ms slices. Each slice calls ab_gp_channel_negotiate_service_link() -> rte_site_role_service_link(), which drains up to 4 inter-site frames so their acknowledgement does not depend on the other site's cycle phase. Missed ticks advance the cycle number (at most 20 at once). An ONLINE channel whose previous tick is more than 5 s back (ab_gp_stall_lost_contact()) calls rte_site_role_note_lost_contact() (section 5.7).
pre_execute** (main/pre_execute.c, ab_gp_channel_pre_execute()):
- ab_gp_channel_stage_reset(), then the tick wait above, then rte_watchdog_timer_tick().
- A/B peer link drained before the site-role step: the frames the checkpoint rendezvous deferred last cycle are replayed, then ab_gp_channel_poll_peer() reads every queued ab-peer frame into on_peer_frame_received(): SITE_STATE -> ab_gp_sibling_state_store() (raw 3 bytes, no cycle stamp); CHECKPOINT_REQUEST -> cached and answered with a CHECKPOINT_REPLY; CHECKPOINT_REPLY -> cached; SAF_RB, KM114_DIGEST; STARTUP_TS -> answered with this channel's agreed timestamp.
- Site-role step: ab_gp_channel_negotiate_execute() -> rte_site_role_execute(), section 5.5. It takes the sibling frame stored in step 2, so the decision and the A/B check of cycle N use the frame received in cycle N.
- log_role_transition() (ROLE event on a change of ctx->is_online).
- Train-link SaF supervision.
- ab_gp_channel_poll_relays(): Gateway envelopes are processed only while ONLINE (a STANDBY channel drains and drops them). ERTMS and SaF replies are staged here, in pre_execute (section 6.4). Any drained frame updates last_relay_rx_cycle, the readiness input.
- ab_gp_channel_poll_crosscompare(): every queued AB_SAMPLE from the sibling; the last one wins.
- ab_gp_channel_crosscompare_check_site_state(): reads rte_site_role_sibling_verdict() of step 3; FAULT -> react_to_fault() (section 5.8).
- decode_and_apply_peer_sample(): CRC-64 check of the sample; corruption -> SAFE (section 6.3).
check_dual_transfer_timeout(): ONLINE, not yet single, no sample for dual_transfer_timeout_ms (default 20 cycles) -> SINGLE mode (ab_gp_channel_dual_on_transfer_lost()).
execute** (main/execute.c): ERTMS engine step, route/train connect sweep, then ab_gp_channel_crosscompare_execute() (section 6), which sets ctx->cyc_agree.
post_execute** (main/post_execute.c):
- ONLINE: AB_SAMPLE of this cycle's session table staged for ab-crosscompare (RTE_CHECKPOINT_MARK()).
- ONLINE: IL keepalive, sent immediately.
- Train, CTC and IL outputs built; staged only when cyc_agree, or in SINGLE mode with fresh channel data (one RTE_CHECKPOINT_MARK() per staged output).
- send_site_state(): is_online(1) | single_mode(1) | ready_for_online(1) to the sibling, sent immediately, every cycle, ONLINE or STANDBY.
- ab_gp_channel_negotiate_sync_standby() -> rte_site_role_sync_standby(): warm/hot only, an ONLINE channel sends this cycle's post-compute snapshot to the other site before the commit (section 5.9).
Watchdog kick; status tick (status frame with site_wait= / site_why=).
Checkpoint stage** (Platform_RTE rte_appmanager_run() stage 4): rte_channel_checkpoint() with checkpoint_id = this cycle's folded RTE_CHECKPOINT_MARK() signature (ADR-034), over GP's transport backend (section 6.6). on_checkpoint_result: committed -> ab_gp_channel_commit_pending_sends() sends everything staged this cycle; not committed -> discarded.
sequenceDiagram
participant T as tick wait
participant P as pre_execute
participant S as rte_site_role
participant E as execute
participant Q as post_execute
participant K as checkpoint
T->>T: 10 ms slices: service inter-site link
T->>P: tick
P->>P: drain ab-peer (sibling SITE_STATE stored)
P->>S: rte_site_role_execute(cycle, local facts)
S->>S: poll sibling source, frame exchange, decide, A/B check
P->>P: relays (ONLINE only), AB_SAMPLE, site-state verdict, CRC
P->>E: inputs ready
E->>E: engine step, cross-compare -> cyc_agree
E->>Q: decisions
Q->>Q: stage AB_SAMPLE and agreed outputs, send SITE_STATE, standby sync
Q->>K: rendezvous on the folded signature
K->>K: commit or discard staged sends
5. ONLINE/STANDBY negotiation, end to end
5.1 Model
- Each channel negotiates on its own inter-site link: A/WEST with A/EAST, B/WEST with B/EAST. Neither link is a single point of failure, and the two links decide independently.
- The site role is the role both channels of a site hold. They must agree (2oo2, section 5.8); the start-up timestamp agreement (5.2) makes the two independent start-up decisions give the same answer, and the per-cycle rules let a channel follow its ONLINE sibling (rules 6 and 7).
- Three roles on the wire: ONLINE, STANDBY, FAULTED. FAULTED is only ever announced by the final frame a channel flushes before it reboots (section 5.7); a running GP channel never reports itself FAULTED in a normal cycle.
- After start-up, the role changes only through rte_site_role_decide() (REQ-SITEROLE-011). The rte_dual_negotiator keeps running as a link-health diagnostic and can legitimately disagree with the role.
5.2 Startup timestamp, agreed between A and B (G1 step 5a)
Both channels of a site present the same startup timestamp to their inter-site tie-break, so that the two independent tie-breaks (REQ-DUAL-NEGOTIATOR-003) give A and B the same verdict, which REQ-SITEROLE-014 then checks. The former /tmp/rte_site_<site>_ts.dat file is gone (it did not work across hosts).
- exchange_startup_timestamp() takes own_now = rte_timer_now(), sends a PEER_MSG_KIND_STARTUP_TS request (payload is_reply(1) | timestamp_ms(8, LE), carrying own_now) every 100 ms and reads ab-peer in 20 ms slices for at most AB_GP_STARTUP_TS_WAIT_MS (3000 ms). A's listen side gets RTE_STATUS_NOT_CONNECTED until B has connected and simply retries. SAF_RB / KM114_DIGEST frames seen meanwhile are deferred for pre_execute; other kinds are dropped (a SITE_STATE frame is dropped here too, see 5.3).
- The first STARTUP_TS frame ends the wait. ab_gp_startup_timestamp_agree(own, sibling): the sibling's value is adopted when it is not newer than the own one and less than AB_GP_STARTUP_TS_FRESH_MS (3000 ms) older; otherwise the own value is kept. The rule is symmetric, so a joint start gives both sides the same value. A request is answered once with a reply carrying the agreed value.
- No frame within 3000 ms: the own clock is used (WARNING event). Clock failure at the start: no exchange, the timestamp stays 0 and rte_site_role_start() takes its own clock (ERROR event). Clock failure during the exchange: the own clock (ERROR event).
- Log: one console line site startup timestamp <agreed> ms (own <own>, sibling <value|none>, adopted sibling | own
kept | sibling not reached within 3000 ms) and one NEGOTIATION event.
- Later requests (the sibling restarted) are answered by the running channel from on_peer_frame_received() and poll_for_checkpoint_frame() (ab_gp_channel_negotiate_on_startup_ts_frame()) with its agreed value, which never changes (rte_site_role resumes it across reconnects). A channel whose clock failed at start holds 0 and does not answer.
Consequence: a channel restarting next to a long-running sibling keeps a fresh timestamp (the sibling's is more than 3 s older). Its tie-break against the other site is then decided by that fresh value; see 5.3 for what follows.
5.3 Inter-site start-up decision (rte_site_role_start())
- Start-up role from the A/B sibling (REQ-SITEROLE-017, Platform_RTE step 5b-3a). The service polls the sibling source once before opening the link. A known sibling frame decides the role: sibling ONLINE -> this channel starts ONLINE, sibling STANDBY -> STANDBY, with the negotiator seeded to that state, so the tie-break below is not evaluated. RBC_GP does not feed this yet: nothing has stored a sibling frame before rte_site_role_start() (the per-cycle peer pump does not run yet and exchange_startup_timestamp() drops SITE_STATE frames), so the source returns nothing, the service logs start-up: no A/B sibling state, negotiating from timestamps and the tie-break always runs. Delivering the frame at start-up is G1 step 5b-3b (root TODO.dox).
- Link. WEST opens its listen side once; EAST retries the connect every 100 ms (500 ms per attempt) up to the 10 s bound. Failure: LINK ERROR event, the setup step fails.
- Tie-break (REQ-DUAL-NEGOTIATOR-003). The negotiators exchange STATE beacons carrying each side's startup timestamp until this side leaves IDLE (bound 10 s, link_poll_ms 300 ms per round): the older (smaller) timestamp wins ONLINE; on an exact tie the smaller site id wins (WEST = 0). Both links of a site compare the same two agreed timestamps, so A and B normally get the same verdict.
- Result: negotiation result -> ONLINE|STANDBY (own_ts=..., peer_ts=...) and a NEGOTIATION event.
What this gives today (tie-break only, 5b-3b not done):
| Situation | Start-up result |
| Both sites start together | the site whose agreed timestamp is older is ONLINE on both links; the other STANDBY |
| One whole site restarts while the other runs | the restarted site has the fresher timestamp: STANDBY |
| One channel restarts next to its running sibling | fresh own timestamp (5.2), so it loses the tie-break and starts STANDBY even when its site is ONLINE. If the site is ONLINE, rule 6 (sibling ONLINE, 1 cycle; rule 7 after 2 cycles while its link is not answering yet) promotes it as soon as it is ready (a relay frame from its Gateway has arrived); until then the A/B check counts mismatches, and a readiness WAIT that lasts 3 checks ends in an AB_SITE_STATE_MISMATCH reboot. REQ-SITEROLE-017 with step 5b-3b removes this window |
5.4 The site-role frame on the inter-site link
Sent every cycle by rte_site_role_execute() (and by the warm-standby sync), one rte_dual_channel DATA frame of RTE_EXAMPLE_SITE_EXTRA_PAYLOAD_SIZE bytes:
faulted(1) | single_mode(1) | cycle(4, LE) | state region | valid(1)
- The state region is GP's rte_state_transfer encoding of the train-session table (RTE_EXAMPLE_MAX_TRAINS x 19 bytes) and the route database. It carries live state and valid = 0xA5 only while the sender is ONLINE; otherwise an empty state, cycle 0 and valid = 0.
- The receiver treats "valid and not faulted" as the counterpart is ONLINE now and keeps a valid frame as the snapshot it would take over on promotion (receive_state() -> ctx->neg_peer_extra).
- Any decoded frame clears peer_unresponsive and a pending takeover confirmation (the counterpart is talking again).
- Known open defect: the received length is not checked (root ISSUES.dox ISS-038).
5.5 The per-cycle site-role step (rte_site_role_execute())
- Take the local facts from GP (local_facts()): faulted = ctx->own_faulted, single_mode = ctx->single_mode, own_ready = ab_gp_site_status_ready_for_online() (a Gateway relay frame within the last 10 cycles; the inter-site link is deliberately not part of readiness). The sibling_* facts are passed as false and ignored: the service derives them from the sibling source.
- Poll the sibling source once (ab_gp_sibling_state_take()): a frame stored since the last poll is decoded (online = byte 0 != 0, single_mode = byte 1 == single, ready = byte 2 != 0) and stamped with this cycle. Derived: sibling online / ready (last known frame), sibling reachable = a frame within the last 3 cycles (RTE_SITE_ROLE_SIBLING_REACH_DEFAULT_CYCLES; a cycle number that went back counts as not reachable).
- Refresh the takeover challenge (with a verifier configured).
- Link down: one reconnect attempt bounded by 50 ms (REQ-SITEROLE-012). A reconnect resumes this channel's own startup timestamp and, when ONLINE, its ONLINE state, instead of re-running the tie-break. If the attempt fails, the decision (steps 8 and 9) still runs, because the dispatcher rule needs no link, and the step ends.
- rte_dual_negotiator_execute() (non-blocking beacon; diagnostic).
- Send this cycle's frame; receive the counterpart's latest frame.
- Warm/hot only: a held snapshot silent for more than 3 cycles is dropped (STANDBY goes cold).
- Decision: rte_site_role_decide() over the facts (5.6), one action applied.
- A/B agreement check against the role just decided (5.8).
- Supervision: 3 consecutive unacknowledged sends, or a receive failure other than a timeout, close the link for reconnect (REQ-SITEROLE-013). If ONLINE: lost_peer_while_online is set. If STANDBY: peer_unresponsive is set, a TAKEOVER warning counterpart not responding - dispatcher confirmation required is logged and the channel stays STANDBY** (REQ-SITEROLE-003).
5.6 The decision rules
rte_site_role_decide() (Platform_RTE rte_site_role_policy.c) evaluates the rules in this order; the first that applies wins (REQ-SITEROLE-001). "Answering" = a counterpart frame within the last 2 cycles. Every scenario has a row in Platform_RTE's tests/site_role/test_rte_site_role_policy.c.
| # | Condition (as coded) | Action | Reason token |
| 1 | own FAULTED and counterpart FAULTED | RENEGOTIATE: WEST ONLINE, EAST STANDBY | both_faulted |
| 2 | not FAULTED, STANDBY, counterpart FAULTED | PROMOTE | peer_faulted |
| 3 | own FAULTED, counterpart not FAULTED, a valid counterpart snapshot held | DEMOTE | recovered |
| 4 | STANDBY, not FAULTED, counterpart unresponsive, dispatcher confirmation at most 120 cycles old (rte_site_role_takeover_allowed()) | PROMOTE | dispatcher |
| 5 | ONLINE, nobody FAULTED, counterpart ONLINE now and answering | DEMOTE if this channel lost contact while ONLINE, else ALARM (logged once, not resolved) | yield / dual_online |
| 6 | STANDBY, not FAULTED, link up, counterpart not unresponsive, answering, not ONLINE, not FAULTED: counts consecutive cycles; PROMOTE at 1 cycle when the A/B sibling is ONLINE, else at 6 cycles (RTE_SITE_ROLE_BOTH_STANDBY_CYCLES) and only on WEST | PROMOTE | both_standby_sibling / both_standby_west |
| 7 | STANDBY, not FAULTED, A/B sibling ONLINE, counterpart not FAULTED and not (ONLINE now and answering): counts consecutive cycles; PROMOTE at 2 | PROMOTE (follow the sibling) | both_standby_sibling |
| - | anything else | none (the counter goes back to 0) | - |
Readiness gate (REQ-SITEROLE-002).** Every PROMOTE becomes WAIT (waiting_ready, status site_wait=ready) unless this channel is ready and its sibling is ONLINE, ready, or not reachable. A WAIT keeps the rule 6/7 counter counting.
How a channel becomes ONLINE**, therefore, only by: the start-up decision; the counterpart's FAULTED notice (rule 2); a dispatcher confirmation while the counterpart is silent (rule 4); the both-STANDBY tie-break, a negotiation outcome both sites compute from the same link (rule 6); following its ONLINE sibling (rules 6 and 7). Silence alone never promotes. Rules 1 and 3 need own_faulted, which GP sets only in the fault reaction right before it reboots (section 5.8), so in RBC_GP they can only fire in that last cycle; they stay for the generic service.
Applying an action.** PROMOTE: role ONLINE, peer_unresponsive and the confirmation cleared, change reported with apply_state; GP takes over the counterpart's snapshot (sessions and route db always, REQ-RBC-006; the cycle number under RTE_EXAMPLE_TRANSFER_POLICY_ENV: continue (default), restart, ask) or starts cold (cycle 1, no sessions) without one, clears single_mode and opens the 40-cycle join window. DEMOTE: role STANDBY, lost_peer_while_online cleared. RENEGOTIATE: WEST ONLINE, EAST STANDBY. Every change gives a ROLE warning event and the console line *** TAKEOVER: ... *** or *** ... settling into STANDBY ***.
5.7 Lost contact, takeover, reconnect
- Counterpart silent while this channel is STANDBY: link closed (5.5 step 10), peer_unresponsive, status site_wait=dispatcher. The channel waits for an operator. Takeover token (REQ-SITEROLE-015, ADR-040 step 4): an operator logged in to RBC_Maintenance_Web sends POST /api/takeover/confirm; the Gateway builds one token per channel - that channel's current challenge (tko= in its status line) MACed with the site's takeover key (RTE_TAKEOVER_KEY_FILE) - and sends it as RBC_UTIL_TOPIC_TAKEOVER_CONFIRM over the UTIL relay; ab_gp_channel_negotiate_on_takeover_confirm() hands it to rte_site_role_on_takeover_token(), which checks version, site id, channel id, MAC and the current challenge. The challenge is random, single use and lives 240 cycles (120 s / cycle period). An accepted token is valid for 120 cycles; rule 4 then promotes (subject to the readiness gate). Without a key file the unauthenticated RTE_TAKEOVER_CONFIRM_FILE trigger still works (test fallback only). Known limits and open defects: the key sits on the non-vital Gateway until gap G5; ISS-017, ISS-018, ISS-019, ISS-022 (root ISSUES.dox).
- Contact lost while ONLINE: link closed, lost_peer_while_online set (also set by the 5 s stall detector of the tick wait). The channel stays ONLINE. If it later sees the counterpart ONLINE and answering, it yields (rule 5, DEMOTE); a dual ONLINE without that history is an ALARM, logged as DUAL ONLINE detected and not resolved.
- Reconnect: one bounded attempt per cycle; on success the dual channel and negotiator are re-initialised with the resumed timestamp (and ONLINE state if ONLINE) and link (re)established is logged. A STANDBY reconnect does not resume a state; its negotiator re-decides from timestamps, which only affects the diagnostic.
- Counterpart FAULTED: a channel that reboots on a fault flushes one final frame with faulted = 1 (rte_site_role_flush_faulted()); the counterpart's rule 2 promotes it. Only the fault reactions of sections 5.8, 6.3 (persistent cycle skew) and 6.5 flush; the checkpoint-timeout reboot (6.6) does not, so the counterpart then only sees silence.
5.8 What A and B must agree on, and the reaction
- What: {online, single_mode} of a channel and of its sibling (REQ-SITEROLE-014). ready is carried but only feeds the readiness gate.
- Transport: every post_execute sends SITE_STATE = is_online | single_mode | ready_for_online on ab-peer (immediate, not staged). The receiver stores the raw bytes (pre_execute peer drain, or the checkpoint poll), the service takes them once per cycle (REQ-SITEROLE-016).
- Check: in rte_site_role_execute(), after the decision of the same cycle, rte_site_role_check_sibling() compares this channel's role and single_mode with the last known sibling frame. No frame yet this run: no verdict (UNKNOWN). Equal: AGREE, the count resets. Different: the count rises; below 3 the verdict is PENDING and the console shows site state mismatch with local peer (mismatch count n/3 ...); at 3 consecutive disagreeing checks the verdict is FAULT with the console line NEGOTIATION MISMATCH ... entering REBOOT and an ERROR event NEGOTIATION_MISMATCH. The reachability window does not gate the check: a sibling that has stopped sending is still compared by its last frame.
- Reaction: in the same pre_execute, ab_gp_channel_crosscompare_check_site_state() reads rte_site_role_sibling_verdict(); FAULT -> react_to_fault(): own_faulted = true, FAULTED flushed to the inter-site counterpart, CTC alarm, ab_gp_channel_shutdown(), then RTE_SAFESTATE(REBOOT,
RTE_EXAMPLE_REASON_AB_SITE_STATE_MISMATCH). The process re-execs as the same role and site (5 s mocked reboot delay).
- Timing: the sibling frame of cycle N was sent in the sibling's previous post_execute, so an ordinary one-cycle promotion skew between A and B costs one strike; the peer drain before the site-role step (5b-2 review fix) keeps it at one.
5.9 Standby state: cold and warm
standby_mode (redundancy config) selects it; cold is the default.
- All modes: the ONLINE channel's frame (5.4) carries its live state every cycle; STANDBY keeps the latest valid one. In cold mode the held snapshot is never aged: a promotion adopts the last one received, however old.
- Warm/hot (standby_sync): after compute, before the commit, the ONLINE channel sends a fresh snapshot (rte_site_role_sync_standby()) and marks STANDBY "not working" when it is not acknowledged within ack_timeout_ms; it then retries every 10 cycles and never withholds output. STANDBY drops a snapshot silent for more than 3 cycles (goes cold). Hot independent computation is not built (HOT_STANDBY_DESIGN.dox).
- Status: ab_gp_site_status_collect() (com/ab_gp_site_status.c) appends site_wait=nothing|dispatcher|link|ready and site_why=<reason token> to the periodic status frame, shown by RBC_Maintenance_Web.
6. A/B cross-comparison, end to end
6.1 What is compared
The AB_SAMPLE (ab_gp_channel_payload_encode_ab_sample()) is cycle(4) | session table | CRC-64(8). The compared payload is the wire-encoded session table only: RTE_EXAMPLE_MAX_TRAINS (2) slots x 19 bytes (RTE_EXAMPLE_TRAIN_SESSION_WIRE_SIZE), per slot:
| Field | Bytes |
| in_use | 1 |
| train_id | 1 |
| cycle | 4 |
| d_lrbg | 4 |
| granted_length | 4 |
| ma_seq | 4 |
| ma_acked | 1 |
Not compared: v_train, m_mode, ma_pending_send, the CTC notification flags, the ERTMS per-train context (s_train_contexts: position, MA packets, session state) and the route database. The encoded form is compared (no struct padding); rte_cross_comparator_execute_buffers() does a full memcmp() (REQ-CROSSCOMPARATOR-004).
6.2 When
- The sample is staged in every ONLINE post_execute and sent at the commit of that cycle's checkpoint, on ab-crosscompare. A STANDBY pair sends and compares nothing.
- The receiver drains ab-crosscompare in pre_execute (last sample wins) and compares in execute: this channel's table now against the sibling's table as of its last committed sample, at least one cycle older. Differences around every update are therefore expected; the skips below absorb them.
6.3 Skips, tolerances and their limits (ab_gp_channel_crosscompare_execute())
cyc_agree starts false every cycle and is set only by an AGREE.
| Check, in order | Outcome |
| Not ONLINE, or no sample arrived this cycle | skip (DEBUG XCOMPARE event) |
| Sample CRC-64 corrupt (RTE_STATUS_DATA_CORRUPTION, in pre_execute) | RTE_SAFESTATE(SAFE, RTE_EXAMPLE_REASON_AB_CRC_MISMATCH): GP's SAFE handler logs SAFE_HALT, raises a CTC alarm and the process halts |
| Sample malformed | skip |
| abs(peer_cycle - cycle) > RTE_EXAMPLE_AB_CYCLE_SKEW_TOLERANCE (20) | skip; 60 consecutive skew skips -> react_to_fault(RTE_EXAMPLE_REASON_AB_PEER_LINK_DOWN) (REBOOT, as in 5.8) |
| Per-train gate: a slot in use on either side whose 7 fields differ | skip the whole compare this cycle, at most RTE_EXAMPLE_XCOMPARE_SYNC_SKIP_LIMIT (10) consecutive times per slot; then the compare runs and DISAGREEs (REQ-RBC-008) |
| Per-train gate: slot free here, in use at the sibling for 2 cycles in a row, inside the join window (40 cycles after promotion or an ONLINE start) | adopt the sibling's session once (ab_gp_session_adopt_allowed()), WARNING event |
| otherwise | compare: AGREE -> cyc_agree = true (INFO AGREE event); DISAGREE -> 6.5 |
6.4 The agreement gate on outputs, and what bypasses it
post_execute stages train (flat scheme), CTC and IL outputs only when should_forward_to_c = cyc_agree || (single_mode && cyc_have_channel_data) (post_execute.c). An output that is not staged is rebuilt the next cycle (its pending flags are cleared only when it is staged). Both A and B stage; the Gateway forwards the first copy and drops the identical duplicate.
Not behind cyc_agree:
- ERTMS replies, including the Msg 3 Movement Authority, and SaF handshake frames: staged in pre_execute from the relay poll (ERTMS: build_and_stage_reply() in ab_gp_proc_dispatch.c; SaF: saf_stage_to_train() in pre_execute.c), committed by the checkpoint alone. A channel with a diverging LRBG or route table can send its Msg 3 in the same cycle; the divergence shows only later through granted_length / ma_seq (up to 10 skips, about 5 s), then DISAGREE. Open, vital: root ISSUES.dox ISS-048 (owner decision: stage ERTMS decision outputs behind the gate, or add the MA content to the compared payload).
- IL keepalives, SITE_STATE, checkpoint replies, STARTUP_TS, SAF_RB: link and protocol traffic, sent immediately.
- SINGLE mode: outputs are staged without a comparison whenever fresh channel data arrived.
6.5 DISAGREE
rte_cross_comparator_execute_buffers() calls GP's xcompare_on_disagreement() (console DISAGREE (first differing
train index=n) ... entering REBOOT, ERROR event DISAGREE, then fault_cleanup(): own_faulted, FAULTED flushed to the inter-site counterpart, CTC alarm, shutdown of the links) and then itself enters RTE_SAFESTATE_LEVEL_REBOOT with RTE_EXAMPLE_REASON_AB_MTYPE_DISAGREE (the comparator always enters its configured level, ADR-025; a zero-filled level would be DEGRADED and non-halting, root ISSUES.dox ISS-041 - GP configures REBOOT). Each channel compares independently, so normally both channels of the site reboot; the other site's channels see the FAULTED notice and promote (rule 2) once ready. For testing, RTE_EXAMPLE_XCOMPARE_FAULT_CYCLE makes role A corrupt its own compared payload from that cycle on.
6.6 The checkpoint rendezvous and how it relates
- What it checks: checkpoint_id is the CRC-64 fold of every RTE_CHECKPOINT_MARK() passed this cycle (the AB_SAMPLE stage and one per staged output), so equal ids mean the same program path (ADR-034). It does not compare data. Its result commits or discards everything staged this cycle, including the ERTMS replies of 6.4.
- Transport (ab_gp_channel_checkpoint.c): backend_send() sends CHECKPOINT_REQUEST on ab-peer; backend_recv() returns, in order: a cached CHECKPOINT_REPLY (echoes this channel's id), a cached sibling CHECKPOINT_REQUEST (the sibling's id: confirms only on the same path), then the lookback: the sibling exchanged any checkpoint frame within the last RTE_EXAMPLE_AB_CHECKPOINT_LOOKBACK_CYCLES (2) cycles -> this channel's own arrival is returned, which matches by construction; last, a live poll in 20 ms slices for the rest of the budget, which also services the inter-site link and stores a SITE_STATE frame for the sibling source. A stale or wrong id does not count (REQ-CHECKPOINT-002).
- What it really guarantees: on the lookback path only that the sibling ran its checkpoint stage within 2 cycles (liveness). Divergence of data is the cross-comparator's job, which gates only the outputs of 6.4 (TIMING_AND_DATA_LOSS_ANALYSIS.dox finding 10).
- Budget and failure: 400 ms (10 s for the first 15 s). No confirmation in time -> Platform_RTE enters SAFE with RTE_SAFESTATE_REASON_CHECKPOINT_TIMEOUT (REQ-CHECKPOINT-003); GP's SAFE handler (ab_gp_channel.c) treats that reason as a lost peer link: own_faulted, shutdown, RTE_SAFESTATE(REBOOT, RTE_EXAMPLE_REASON_AB_PEER_LINK_DOWN). It does not flush FAULTED to the inter-site counterpart. A sibling that dies is therefore detected here within about 3 cycles, long before the 20-cycle dual-transfer timeout that would enter SINGLE mode.
- Known open defects: a failed pre_execute / execute skips the checkpoint and on_checkpoint_result for that cycle (ISS-031); a checkpoint setup failure runs without a rendezvous (ISS-070); the checkpoint ignores a failed rte_timer_now() (ISS-036, the watchdog part is fixed).
7. Failure reactions at a glance
| Event | Detected by | Reaction | Counterpart site sees |
| A/B role disagreement 3 checks in a row | rte_site_role (REQ-SITEROLE-014) | REBOOT (AB_SITE_STATE_MISMATCH), FAULTED flushed | FAULTED -> rule 2 promotion |
| Session tables DISAGREE | rte_cross_comparator | REBOOT (AB_MTYPE_DISAGREE), FAULTED flushed | FAULTED -> rule 2 |
| A/B cycle numbers > 20 apart for 60 cycles | ab_gp_channel_crosscompare_execute() | REBOOT (AB_PEER_LINK_DOWN), FAULTED flushed | FAULTED -> rule 2 |
| Checkpoint not confirmed in budget (sibling dead or silent) | rte_channel_checkpoint() | SAFE -> GP handler -> REBOOT (AB_PEER_LINK_DOWN), no flush | silence -> unresponsive, waits for the dispatcher |
| CRC-64 corruption on AB_SAMPLE | ab_gp_channel_payload_decode_ab_sample() | SAFE: log, CTC alarm, halt | silence |
| No AB_SAMPLE for 20 cycles while ONLINE | check_dual_transfer_timeout() | SINGLE mode | - |
| Inter-site link: 3 unacknowledged sends or a hard receive error | rte_site_role (REQ-SITEROLE-013) | close, reconnect each cycle; STANDBY: unresponsive; ONLINE: lost-contact history | - |
| ONLINE channel stalled more than 5 s | tick wait stall detector | lost-contact history (yield if the counterpart took over) | - |
| Inter-site link cannot be opened at start-up | rte_site_role_start() | setup step fails, process exits | - |
8. Known open points
- ISS-048: ERTMS replies bypass the cross-compare gate (6.4).
- G1 step 5b-3b: GP does not deliver the sibling's frame before rte_site_role_start(), so REQ-SITEROLE-017 never applies in RBC_GP and a restarting channel goes through the STANDBY start of 5.3.
- G1 step 5c: the site link and the A/B peer link move to rte_flow; rte_site_role_check_sibling() (no longer called by GP) is to be deleted from the API.
- ISS-036 (negotiator, dual channel and checkpoint ignore a failed clock), ISS-037 (the AB_SAMPLE encoder can store a CRC of 0 before CRC-64 init; latent, GP initialises first), ISS-038, ISS-042 (start-up bound), ISS-043 (dual channel overwrite; benign for the one-link site role), ISS-009 and ISS-022 (Docker observations): root ISSUES.dox.
9. Tests
- Platform_RTE: tests/site_role/test_rte_site_role_policy.c (decision table), tests/site_role/test_rte_site_role.c (service: start-up, sibling source, REQ-SITEROLE-017, token, teardown).
- RBC_GP: tests/gateway/test_ab_gp_startup_timestamp.c, test_ab_gp_sibling_state.c, test_ab_gp_standby_policy.c (stall rule, session adoption).
- Live: .claude/skills/run-RBC_GP/smoke.sh (start order, failover via RTE_EXAMPLE_XCOMPARE_FAULT_CYCLE), the RBC_Test_Env Docker suites (takeover, partitions, reboots).
10. Design history (working document of 2026-09-24)
Kept as written then, with notes where the code has moved on. Everything above supersedes it.
10.1 What was found broken: acknowledgement depends on cycle phase
The link is rte_dual_channel. A sender waits ack_timeout_ms (50 ms) for an acknowledgement. The receiver produces that acknowledgement only from inside its own receive() call, and the RBC called receive() once per cycle in pre_execute. The two sites' cycles are independent and not phase-locked.
sequenceDiagram
participant W as A/WEST (cycle phase 0 ms)
participant E as A/EAST (cycle phase 240 ms)
Note over W: pre_execute: negotiate
W->>E: DATA snapshot
Note over E: busy in execute/checkpoint,<br/>not calling receive()
Note over W: waits 50 ms ... TIMEOUT (no ACK)
Note over E: 240 ms later: pre_execute, receive()
E-->>W: ACK (too late, nobody waiting)
Note over W,E: 3 consecutive misses =><br/>"link appears dead" => tear down => (2026-09-24 rule) both promote to ONLINE
With aligned phases both sides were inside their send at the same moment and polled each other, so it worked. When phases drifted it never recovered by itself. Observed in the Docker stack: both sites role=ONLINE,
counterpart=ONLINE (split brain), 100+ send timeouts and ~40 "link appears dead" per site within a minute. Since 2026-09-25 a dead link never promotes** (rule 4 needs the dispatcher), and Option A below is built.
10.2 Option A: fast internal service tick (built)
The wait for the next cycle tick wakes every 10 ms and calls ab_gp_channel_negotiate_service_link(), which drains and acknowledges the link; the checkpoint live poll does the same.
| |
| Measured | Docker, cold mode, 60 s after start, all four channels (A/B x WEST/EAST): stable ONLINE/STANDBY in 117 of 117 negotiation lines each, 0 "link appears dead", 1 send failure (at startup). Before the change: both sites ONLINE, 38 "appears dead" and 115 send failures per site. |
| Good | Single thread, deterministic, no new process, no locks around vital state. Acknowledgement latency drops from up to 500 ms to about 10 ms whenever the RBC is waiting. |
| Weak | The RBC is not polling while it computes. An acknowledgement can still be late during those stages. |
10.3 Option B: site-link agent (proposed, not built)
A small non-vital process on the same hardware would own the site-to-site UDP link, acknowledge immediately and keep the newest snapshot from the other site for A/B to read locally.
| |
| Good | No acknowledgement wait inside the RBC cycle; timing independent of the other site's phase. |
| Cost | An agent acknowledgement proves the agent got the snapshot, not that STANDBY applied it; "standby updated before output" becomes asynchronous. |
| Cost | The agent is not vital, so every snapshot needs end-to-end CRC/authentication that A and B verify. |
| Cost | New failure mode (agent dies = link down), supervision, packaging, a new product. |
ADR-040 decision 2 (2026-10-02) keeps the drive points in the application cycle (option A); there is no RTE site-link thread.
10.4 Decisions that were open on 2026-09-24
- Accept Option A first, then B only if needed: Option A built; B not started.
- If B: is the weaker "agent received it" acknowledgement acceptable? Open, only relevant if B is built.
- Ship default standby_mode: cold (unchanged) until the warm-mode Docker runs pass.