Watchdog Mechanism

Detect hung systems/tasks and trigger recovery. More...

Files
file	src/redundancy/watchdog/rte_watchdog.c
	Real watchdog implementation: a fixed-size pool of watchdog slots, timed via the already-portable rte_timer_now() OAL primitive rather than any new OS-specific timing code of its own (REQ-OAL-COMMON-010: no dynamic allocation).

Data Structures
struct	rte_watchdog_config_t
	Watchdog configuration. More...
struct	rte_watchdog_status_t
	Watchdog health/status information. More...

Enumerations
enum	rte_watchdog_type_t { RTE_WATCHDOG_SYSTEM , RTE_WATCHDOG_TASK , RTE_WATCHDOG_CHANNEL , RTE_WATCHDOG_CHECKPOINT }
	Watchdog type. More...
enum	rte_watchdog_action_t {   RTE_WATCHDOG_ACTION_LOG , RTE_WATCHDOG_ACTION_SAFESTATE , RTE_WATCHDOG_ACTION_REBOOT , RTE_WATCHDOG_ACTION_FAILOVER ,   RTE_WATCHDOG_ACTION_CUSTOM }
	Recovery action when watchdog fires. More...

Functions
rte_status_t	rte_watchdog_create (rte_watchdog_t *handle_out, const rte_watchdog_config_t *config)
	Create a watchdog.
rte_status_t	rte_watchdog_start (rte_watchdog_t watchdog)
	Start watchdog timer.
rte_status_t	rte_watchdog_stop (rte_watchdog_t watchdog)
	Stop watchdog timer.
rte_status_t	rte_watchdog_kick (rte_watchdog_t watchdog)
	Kick (pet) watchdog - prove liveness.
rte_status_t	rte_watchdog_get_status (rte_watchdog_t watchdog, rte_watchdog_status_t *status_out)
	Get watchdog status.
rte_status_t	rte_watchdog_destroy (rte_watchdog_t watchdog)
	Destroy watchdog.
rte_status_t	rte_watchdog_manager_initialize (void)
	Initialize watchdog manager (call once at startup).
rte_status_t	rte_watchdog_manager_shutdown (void)
	Shutdown watchdog manager (call once at shutdown).
void	rte_watchdog_timeout_handler (uint32_t watchdog_id)
	Watchdog timeout handler (INTERNAL - called by framework).
void	rte_watchdog_timer_tick (void)
	Poll all active watchdogs for expiry (call periodically).

Detailed Description

Detect hung systems/tasks and trigger recovery.

Enumeration Type Documentation

rte_watchdog_type_t

enum rte_watchdog_type_t

Watchdog type.

Defines what the watchdog is monitoring.

Enumerator
RTE_WATCHDOG_SYSTEM	Entire system liveness
RTE_WATCHDOG_TASK	Specific task/thread
RTE_WATCHDOG_CHANNEL	IPC/redundancy channel
RTE_WATCHDOG_CHECKPOINT	Checkpoint barrier

Definition at line 41 of file rte_watchdog.h.

rte_watchdog_action_t

enum rte_watchdog_action_t

Recovery action when watchdog fires.

Defines what happens when watchdog timeout occurs.

Enumerator
RTE_WATCHDOG_ACTION_LOG	Log event only
RTE_WATCHDOG_ACTION_SAFESTATE	Trigger safe-state
RTE_WATCHDOG_ACTION_REBOOT	System reboot
RTE_WATCHDOG_ACTION_FAILOVER	Loss of a redundant peer/channel detected - invokes config->custom_action (same dispatch as RTE_WATCHDOG_ACTION_CUSTOM; custom_action is required for this action too - see rte_watchdog_create()). Use this over CUSTOM when the reason a watchdog exists is specifically "my redundant partner stopped responding" (e.g. a dual-channel cross-compare link) - the distinct name documents intent at the call site, even though the mechanism is identical to CUSTOM. REQ-WATCHDOG-002
RTE_WATCHDOG_ACTION_CUSTOM	Custom callback

Definition at line 53 of file rte_watchdog.h.

Function Documentation

rte_watchdog_create()

rte_status_t rte_watchdog_create	(	rte_watchdog_t *	handle_out,
		const rte_watchdog_config_t *	config )

Create a watchdog.

Initializes a watchdog with specified configuration. Watchdog starts in disabled state; call rte_watchdog_start() to enable.

Parameters

handle_out	Receives watchdog handle
config	Watchdog configuration

Returns

RTE_STATUS_OK on success RTE_STATUS_ERROR on failure

Precondition

config != NULL

Postcondition

watchdog is created but not started

Safety:

No dynamic memory allocation; all storage pre-allocated.

Example (system watchdog):

rte_watchdog_config_t config = {
    .type = RTE_WATCHDOG_SYSTEM,
    .name = "rbc_main_wd",
    .timeout_ms = 1000,
    .action = RTE_WATCHDOG_ACTION_SAFESTATE
};
rte_watchdog_t wd;
rte_watchdog_create(&wd, &config);

Definition at line 191 of file rte_watchdog.c.

rte_watchdog_start()

rte_status_t rte_watchdog_start

(

rte_watchdog_t

watchdog

)

Start watchdog timer.

Enables watchdog monitoring. Countdown begins from timeout_ms. Must be called after rte_watchdog_create().

Parameters

watchdog

Watchdog handle

Returns

RTE_STATUS_OK on success (watchdog armed from the current time) RTE_STATUS_INVALID_PARAM if watchdog is not a live handle the status of rte_timer_now() unchanged when the clock could not be read, e.g. RTE_STATUS_NOT_INITIALIZED (no timer OSAdapter), RTE_STATUS_NOT_SUPPORTED (adapter without now()) or RTE_STATUS_INTERNAL_ERROR (the adapter's own clock read failed, as the POSIX adapter reports a failed clock_gettime()); the watchdog is then left exactly as it was (not armed). REQ-WATCHDOG-003

Precondition

watchdog != NULL

Postcondition

on RTE_STATUS_OK the watchdog is counting down and will fire if not kicked; on any other status it is unchanged

Safety:

Deterministic: no blocking, no dynamic allocation. A failed clock read is never papered over with a deadline computed from a made-up time (ISS-036).

Definition at line 240 of file rte_watchdog.c.

rte_watchdog_stop()

rte_status_t rte_watchdog_stop

(

rte_watchdog_t

watchdog

)

Stop watchdog timer.

Disables watchdog monitoring. No timeout will occur until restarted. Used during shutdown or maintenance.

Parameters

watchdog

Watchdog handle

Returns

RTE_STATUS_OK on success

Precondition

watchdog != NULL

Postcondition

watchdog is stopped; no timeout possible until start() called

Safety:

Deterministic; disarms watchdog

Definition at line 268 of file rte_watchdog.c.

rte_watchdog_kick()

rte_status_t rte_watchdog_kick

(

rte_watchdog_t

watchdog

)

Kick (pet) watchdog - prove liveness.

Resets timeout countdown. Must be called periodically (before timeout expires) to prevent watchdog from firing.

Typical usage: called in main event loop, task loop, or after checkpoint.

Parameters

watchdog

Watchdog handle

Returns

RTE_STATUS_OK on success (countdown reset) RTE_STATUS_INVALID_PARAM if watchdog is not a live handle RTE_STATUS_INTERNAL_ERROR if the watchdog is not started, or already fired (recovery in progress) and not yet restarted the status of rte_timer_now() unchanged when the clock could not be read, e.g. RTE_STATUS_NOT_INITIALIZED (no timer OSAdapter), RTE_STATUS_NOT_SUPPORTED (adapter without now()) or RTE_STATUS_INTERNAL_ERROR (the adapter's own clock read failed, as the POSIX adapter reports a failed clock_gettime()); the deadline and the kick count are then left unchanged. REQ-WATCHDOG-003

Precondition

watchdog != NULL

Postcondition

on RTE_STATUS_OK the timeout countdown is reset to timeout_ms; on any other status the watchdog is unchanged

Safety:

Deterministic: no blocking, O(1) time

Example:

while (running) {
    process_events();
    rte_watchdog_kick(wd);  // Prove we're alive
}

Definition at line 280 of file rte_watchdog.c.

rte_watchdog_get_status()

rte_status_t rte_watchdog_get_status	(	rte_watchdog_t	watchdog,
		rte_watchdog_status_t *	status_out )

Get watchdog status.

Non-blocking query of watchdog state (kicks, fires, time remaining). Useful for health monitoring and diagnostics.

Parameters

watchdog	Watchdog handle
status_out	Receives watchdog status

Returns

RTE_STATUS_OK on success RTE_STATUS_INVALID_PARAM if watchdog is not a live handle or status_out is NULL the status of rte_timer_now() unchanged when the clock could not be read, e.g. RTE_STATUS_NOT_INITIALIZED (no timer OSAdapter), RTE_STATUS_NOT_SUPPORTED (adapter without now()) or RTE_STATUS_INTERNAL_ERROR (the adapter's own clock read failed, as the POSIX adapter reports a failed clock_gettime()); *status_out is then not written at all (no partial read-out). REQ-WATCHDOG-003

Precondition

watchdog != NULL, status_out != NULL

Postcondition

on RTE_STATUS_OK status_out is populated with the current watchdog state; on any other status it is untouched

Safety:

Non-blocking, read-only, no side effects

Example:

rte_watchdog_status_t status;
rte_watchdog_get_status(wd, &status);
printf("Watchdog: %u kicks, %u fires, %u ms until timeout\n",
       status.kicks, status.fires, status.time_until_fire);

Definition at line 315 of file rte_watchdog.c.

rte_watchdog_destroy()

rte_status_t rte_watchdog_destroy

(

rte_watchdog_t

watchdog

)

Destroy watchdog.

Stops and deallocates watchdog. After destruction, handle is invalid.

Parameters

watchdog

Watchdog handle

Returns

RTE_STATUS_OK on success

Precondition

watchdog != NULL, watchdog was created via rte_watchdog_create()

Postcondition

watchdog is stopped and deallocated

Safety:

Deterministic; safe to call on stopped or running watchdog

Definition at line 346 of file rte_watchdog.c.

rte_watchdog_manager_initialize()

rte_status_t rte_watchdog_manager_initialize

(

void

)

Initialize watchdog manager (call once at startup).

Sets up the central watchdog manager that coordinates all watchdog timers. Must be called before creating any watchdogs.

Returns

RTE_STATUS_OK on success

Postcondition

watchdog manager is ready to create watchdogs

Safety:

Called once at startup

Definition at line 163 of file rte_watchdog.c.

rte_watchdog_manager_shutdown()

rte_status_t rte_watchdog_manager_shutdown

(

void

)

Shutdown watchdog manager (call once at shutdown).

Stops all running watchdogs and shuts down manager. After shutdown, cannot create new watchdogs until re-initialized.

Returns

RTE_STATUS_OK on success

Postcondition

all watchdogs stopped; manager deallocated

Safety:

Safe to call multiple times; idempotent

Definition at line 174 of file rte_watchdog.c.

rte_watchdog_timeout_handler()

void rte_watchdog_timeout_handler

(

uint32_t

watchdog_id

)

Watchdog timeout handler (INTERNAL - called by framework).

Invoked by watchdog timer when timeout occurs. Applies recovery action (log, safe-state, reboot, failover, or custom callback).

This is a framework-internal function; applications don't call it directly.

Parameters

watchdog_id

ID of watchdog that fired

Note

Called from timer interrupt context (may be ISR)

Recovery actions are async/deferred (logged, queued)

Definition at line 363 of file rte_watchdog.c.

rte_watchdog_timer_tick()

void rte_watchdog_timer_tick

(

void

)

Poll all active watchdogs for expiry (call periodically).

This implementation has no OS-specific interrupt/thread of its own (consistent with ADR-005: OS-specific timing belongs in an integrator OSAdapter, not in this module). Instead, the integrating application is responsible for calling this function regularly - e.g. from a rte_timer periodic callback, or once per iteration of a rte_appmanager execute() cycle - so that any watchdog whose deadline has passed is detected and its configured recovery action (rte_watchdog_timeout_handler()) is dispatched.

The clock itself is supervised too (REQ-WATCHDOG-004, ISS-036): when rte_timer_now() fails while at least one watchdog is started and not yet fired, no deadline can be checked and such a watchdog could never expire - so this function logs one ERROR ("watchdog clock failed - entering SAFE state", under the first such watchdog's name) and enters RTE_SAFESTATE_LEVEL_SAFE with reason RTE_SAFESTATE_REASON_CLOCK_FAILED, which does not return (REQ-COMMON-SAFESTATE-002). With no started, unfired watchdog a failed clock read leaves this call a silent no-op.

Note

Non-blocking; O(N) over the fixed watchdog pool per call.

No-op if the watchdog manager has not been initialized.

Does not return when the clock has failed and a watchdog is supervising (RTE_SAFESTATE_LEVEL_SAFE). REQ-WATCHDOG-003 is the matching rule for rte_watchdog_start() / rte_watchdog_kick() / rte_watchdog_get_status(): forward the clock status, change nothing.

Definition at line 446 of file rte_watchdog.c.