- Generated by
1.18.0
|
Safe API Framework
Layered API framework for safety-related applications (ERTMS RBC reference targeting CENELEC EN 50128 SIL 4)
|
Status: Accepted, with a known open issue (§2.5) - the transport decision and Phase 1 are done and verified; Phase 2's raw-link liveness/ordering work is verified for most restart/partition scenarios but has a reproducible reconnect livelock specific to a-east restarting, not yet fixed. See §2.5 before treating this migration as fully closed. Date: 2026-08-18 Applies to: include/safeapi/netlink/sapi_netlink.h (doc contract only - no API/ABI change), safeAPIRBC2oo2's src/posix_backend/sapi_posix_backend_netlink.c, src/application/AB/channel_ab_negotiate.c/channel_ab_io.c/ channel_ab_types.h, src/application/C/monitor_c_io.c/ monitor_c_types.h, src/application/common_config.h, tests/robot/fault_injection.robot.
safeAPIRBC2oo2's 6-container RBC topology communicated over real TCP via sapi_netlink. A live field bug (a container whose peer restarted with a fresh TCP connection was never detected as dead, because a framework-level bug in sapi_dual_channel.c was masking real transport failures as generic timeouts - fixed separately, see the dual-channel hard-fault propagation fix committed immediately before this ADR) prompted a broader question: should this link's reliability semantics keep depending on TCP's own connection-oriented guarantees at all?
Decision, made directly by the framework's integrator: move sapi_netlink to UDP, with message ordering, deduplication, and liveness detection becoming explicitly the application/sapi_dual_* layer's responsibility rather than something the transport provides implicitly.
This ADR was executed in two phases, both covered here:
Per ADR-005/ADR-018 (concrete backends live in the consumer, not the framework), safeAPIRBC2oo2/src/posix_backend/sapi_posix_backend_netlink.c was rewritten in place - same sapi_posix_backend_netlink() accessor, so registration and every include site needed zero changes. This is an outright migration, not an opt-in toggle; no second backend file was kept.
HELLO/HELLO_ACK handshake. UDP's connect() performs no network I/O and gives no signal the peer is reachable, unlike TCP's connect()/ accept(). Without a handshake, sapi_netlink_open() would return SAPI_STATUS_OK for a CONNECT role even if nothing were listening yet, silently breaking its own documented contract ("OK means usable"). Two fixed 1-byte magic datagrams (HELLO/HELLO_ACK) are exchanged entirely before sapi_netlink_open() returns a handle - LISTEN binds and waits for the first HELLO (learning its peer's address from it, the UDP analogue of accept(), then connect()s to lock that peer), CONNECT dials immediately and retries HELLO on a short period until it sees HELLO_ACK, bounded by connect_timeout_ms either way.
One datagram per call, no byte accumulation. The old TCP backend's transfer_all() looped poll()+recv()/send() until exactly message_size bytes transferred - correct for a byte stream, actively wrong for UDP (a recv() call returns one whole datagram; looping to "finish" a supposedly-partial receive would incorrectly wait for bytes that will never arrive from a different datagram). backend_send()/ backend_receive() now do a single send()/recv() per call. backend_receive() uses MSG_TRUNC so an oversized datagram is detectable (real length reported even past buffer_size) rather than silently truncated, and any length mismatch against the link's fixed message_size is reported as SAPI_STATUS_DATA_CORRUPTION - a wire protocol violation, not a value to accept partial-length.
SAPI_STATUS_HARDWARE_FAULT becomes opportunistic, not primary. TCP's recv()==0/ECONNRESET/EPIPE gave every caller a reliable "peer is gone" signal; UDP has no such thing. The one exception kept is ECONNREFUSED, which a connected UDP socket can surface on a subsequent send()/recv() after the kernel receives an ICMP Port-Unreachable - real when it fires, but best-effort (silently suppressed by NAT/firewalls, never fires for a merely slow/partitioned peer). The header doc for sapi_netlink_send()/_receive() was updated to state this plainly rather than imply a guarantee no UDP backend can make (new REQ-OAL-NETLINK-014: this service provides no ordering/ dedup/delivery guarantee of its own - any such guarantee is the caller's job).
channel_ab_negotiate.c's existing teardown trigger already closed the link on a plain SAPI_STATUS_TIMEOUT from the ACK wait - it never depended on HARDWARE_FAULT specifically. This means sapi_dual_channel/ sapi_dual_msgchannel's existing sequence+CRC+ACK machinery (ADR-020) required zero functional changes to keep working correctly under UDP; this is a validation of that design, not a coincidence.
Added hardening (small, safeAPIRBC2oo2-only, not a framework change): a single dropped datagram is a routine, expected event under UDP even on a healthy link (no transport-level retransmission), unlike a TCP ack_timeout_ms genuinely meaning something was wrong. Tearing the negotiation link down on the very first miss would reset sapi_dual_negotiator_t's tie-break state machine (back to SAPI_DUAL_STATE_IDLE) far more often than a real fault warrants. A new ctx->neg_consecutive_send_miss counter (SAFEAPI_EXAMPLE_NEG_SEND_MISS_THRESHOLD = 3, common_config.h) requires several consecutive misses before teardown fires; a receive-side hard fault (a status that is neither OK nor TIMEOUT) still tears down immediately, no hysteresis - that is never a routine event.
Initial end-to-end testing (smoke.sh, the real 6-process topology) surfaced that a UDP handshake, even a fast one, is not free the way TCP's near-instant accept()/connect() was, and this interacted badly with existing tuning:
Real 6-container Docker testing of the Phase-1-only build (docker restart on a-west mid-run - the exact scenario that originally motivated this whole ADR) reproduced a genuine regression: b-west's raw peer link had no way to detect that a-west's process had restarted (under UDP there is no transport-level disconnect signal the way TCP's recv()==0/ECONNRESET gave it), so it never redid the HELLO handshake, and a-west was left waiting forever for a HELLO that never came - a-west crash-looped, b-west went silent. Under the old TCP backend this exact scenario self-healed automatically; under UDP-with-Phase-1-only it did not. Closed the same session rather than shipped as a known gap, once demonstrated empirically (see the "Recommended follow-up" pattern this framework already uses elsewhere - a theoretical gap noted in an ADR is not the same evidentiary bar as a reproduced failure).
Staleness-timeout reconnect trigger, mirroring §2.2's negotiation-link pattern but for links with no ACK/sequence layer of their own: each raw link's rx task (channel_ab_io_peer_rx_task_entry(), channel_ab_io_m136_rx_task_entry(), monitor_c_io_rx_a_task_entry()/_rx_b_task_entry()) now counts consecutive SAPI_STATUS_TIMEOUT results from its own bounded sapi_netlink_receive() call. SAFEAPI_EXAMPLE_LINK_STALE_TIMEOUT_COUNT (common_config.h, = 2) consecutive timeouts - meaning SAFEAPI_EXAMPLE_LINK_TIMEOUT_MS (3000ms) of total silence - closes the link for reconnect, the same as an outright hard-fault status always did. A single successful receive resets the counter to 0.
Duplicate/replay rejection reusing existing wire fields, not a new framework primitive. An earlier design sketch (Phase 2, before implementation) proposed a new generic-capacity sequence+CRC framework module (sapi_dual_seqframe), reasoning from sapi_vital_message_t's 248-byte payload cap being too small for the peer link's 273-byte frame. That reasoning does not survive contact with what actually needs ordering: AB_SAMPLE/M136/ANSWER already carry their own cycle counter in their existing wire format (channel_ab_wire.c/ monitor_c_wire.c), which every consumer already decodes - no new bytes on the wire, no new framework module, are needed to detect a stale or duplicate frame. Each rx task now rejects (does not write into its rx_slot_t/answer_slot_t) an incoming frame whose decoded cycle is not strictly newer than the last one accepted ((int32_t)(decoded_cycle - last_accepted_cycle) <= 0). A frame that fails its own existing integrity check (AB_SAMPLE's CRC-64; M136/ ANSWER have none, pre-existing and unrelated to this migration) is left to pass through unchanged, preserving whatever existing corruption handling already covers it - this check only ever suppresses a validly-decoded but stale/duplicate frame. CHECKPOINT_REQUEST/_REPLY (already self-checking via a full sapi_vital_message_t) and SITE_STATE (a level-triggered current-state broadcast with no natural sequence field, self-correcting next cycle if a stale one is used - not worth wrapping) are both left as-is.
A second, empirically-found instance of the §2.3 budget problem - and why a bigger budget alone wasn't the real fix. Retesting in Docker after the staleness-reconnect fix surfaced the same class of issue one level up: the reconnecting side re-enabled the built-in checkpoint (ctx->checkpoint_cfg.voter) the instant its own link's handshake completed, immediately racing the first real round trip on a link that had just come back - under real Docker bridge-network conditions (not loopback), this occasionally still exceeded SAFEAPI_EXAMPLE_AB_CHECKPOINT_MAX_DELAY_MS, triggering the same intentional SAPI_SAFESTATE_LEVEL_SAFE halt as §2.3, this time on the reconnecting side rather than at startup. The budget was raised once more (300ms -> 450ms, 90% of the 500ms cycle period) as a first attempt, but a docker restart of a-east still reproduced the same halt on b-east afterward - proving this was not (only) a margin problem: a handshake completing proves the socket is usable, not that the peer's own side has resumed sending, and no fixed budget reliably bridges that gap under real jitter.
First redesign: checkpoint_cfg.voter no longer re-enabled at handshake completion at all - a new checkpoint_pending_reenable flag deferred it until the peer rx task's own first successful receive after reconnect, i.e. until the link concretely proved it was carrying data. This traded one bug for another: sapi_channel_checkpoint(NULL, ...) returns SAPI_STATUS_INVALID_PARAM gracefully rather than SAFE-halting (by design - see sapi_appmanager_run()'s own comment on this), which sapi_appmanager_pace_failed_checkpoint() retries in a busy-wait paced by the very budget this fix meant to stop racing. Gating checkpoint's own send behind "wait for incoming data" turned out to be circular: if both sides of a link reconnect around the same time, each one's checkpoint stays paused waiting for the other to send first, and neither's checkpoint (the only thing that would send) is enabled to do so - a genuine mutual deadlock, reproduced live (docker restart a-east, both a-east and b-east spinning at 100% CPU in the pacing busy-wait, neither progressing). Fixed by adding a bounded fallback: a checkpoint_reenable_deadline_ms set at reconnect time (SAFEAPI_EXAMPLE_CHECKPOINT_REENABLE_GRACE_MS = 600ms) that re-arms checkpoint on its own if the deadline passes with nothing received - re-enable now fires on whichever of the two signals (data arrived, or deadline elapsed) comes first, breaking the circularity while keeping the stronger data-arrived signal as the fast path. Verified: the deadlock no longer reproduces on the scenario that found it.
Retesting the same a-east restart scenario after the deadlock fix surfaced a further, deeper, and still-unresolved issue: a-east and b-east both settle into a perpetual cycle of "peer link stale -> close -> reconnect -> (re)established -> stale again" every few seconds, indefinitely (confirmed by direct log observation over 30+ seconds of real wall time, not a test-harness artifact - RestartCount stays flat, so this is not a crash loop, but cross-compare/checkpoint never stabilizes either). This did not reproduce on the equivalent a-west restart earlier in the same session, so it is not simply "any container reboot" - there is some asymmetry or timing correlation specific to this scenario not yet isolated.
Working theory, not yet confirmed: open_connect_udp() mints a fresh local ephemeral UDP port on every reconnect attempt (a plain socket() call, no SO_REUSEPORT/fixed local port). If both sides end up reconnecting on independent, unsynchronized cadences (each driven by its own SAFEAPI_EXAMPLE_LINK_STALE_TIMEOUT_COUNT-based detection, with no coordination between them), the LISTEN side can connect() to (lock onto) a peer address:port that is already stale by the time the CONNECT side's next reconnect attempt fires from a different ephemeral port
Status: open. tests/robot/fault_injection.robot's "Container Reboot Recovers - A East" test case (and by the same mechanism, "Network Partition Recovers - A East") reliably reproduce it and will fail until this is fixed - left failing deliberately rather than adjusted to pass, so the suite keeps surfacing this until it's actually resolved. Do not treat a green run of just the a-west/b-west cases as evidence this class of issue is closed.
sapi_safechannel.c's voter channels: confirmed zero consumers in safeAPIRBC2oo2 today, and already has no reconnect capability regardless of transport. Not touched by either phase.
1.18.0