Persistence and Crash Recovery

This document describes the single-node persistence contract of Spedo v0.55.0.

Persistence uses standard files on container volumes such as Docker volumes,

Kubernetes persistent volume claims, and cloud block storage. It requires no

host-exclusive shared memory or custom kernel module.

Documented Durable Scope

The qualified checkpoint and recovery contract covers canonical KV and JSON

values. Compressed values and factorized MODEL values are materialized into

canonical bytes before checkpointing.

Specialized in-memory structures must not be assumed to be restart-durable just

because a command has a WAL frame. A durable product contract requires the full

lifecycle to be covered: checkpoint encoding, WAL replay, WAL compaction,

expiration, deletion, corruption handling, and crash tests. Until a structure

is explicitly listed as qualified, treat lists, streams, document collections,

vectors, matrices, priority queues, graphs, workflows, scripts, CDC streams,

tags, Bloom filters, and search indexes as rebuildable in-memory state.

Configuration

export SPEDO_PERSIST_PATH=/data/spedo.snapshot
export SPEDO_SNAPSHOT_INTERVAL_SECS=30
export SPEDO_WAL_FSYNC=always             # disabled | always | everysec | none
export SPEDO_RECOVERY_POLICY=fail          # fail | empty_on_invalid

Omitting or setting SPEDO_PERSIST_PATH to an empty value disables checkpoints.

Any WAL-enabled mode (always, everysec, or none) without a valid

persistence path is a startup configuration error. A valid persistence path

with SPEDO_WAL_FSYNC=disabled is a supported checkpoint-only configuration.

WAL mode	Work completed before mutation ACK	Recovery point after a process crash
`disabled` (default)	In-memory mutation; no WAL is opened	Latest completed snapshot, if one exists
`always`	WAL append and `fsync`, then successful mutation processing	No acknowledged qualified mutation is expected beyond the durable WAL boundary, subject to the storage device honoring `fsync`
`everysec`	WAL append; background `fsync` runs approximately once per second	Up to the latest completed periodic WAL `fsync`
`none`	WAL append without a Spedo `fsync`	Unbounded by Spedo; dependent on the OS page cache and storage behavior

`disabled` is different from `none` and `off`

disabled or an empty value opens no WAL. In v0.51.0, the mutation guard

checks an immutable enabled flag and returns without acquiring the WAL mutex.

none opens the WAL and appends every covered mutation, but Spedo never calls

fsync for that mode. It still pays WAL serialization, append, ordering, and

mutex costs.

off and no are accepted aliases for none, not aliases for

disabled. Use the word disabled when the intent is to open no WAL.

This distinction matters both for crash behavior and for benchmark provenance.

A result labeled only “persistence off” is ambiguous and should not be

published; record the exact SPEDO_PERSIST_PATH and SPEDO_WAL_FSYNC values.

Under always, the default recovery policy is fail to prevent silent startup

with corrupted state. Rebuildable cache profiles may explicitly configure

empty_on_invalid; the recovery outcome remains visible in PERSIST.STATUS.

Operator Commands

SAVE
SAVE WAIT 10000
PERSIST.STATUS

SAVE: Initiates an asynchronous snapshot and returns immediately.
SAVE WAIT <timeout_ms>: Returns OK only after complete write, fsync, atomic rename, and directory fsync complete (bounded from 1 to 600,000 ms).
PERSIST.STATUS: Returns an array of diagnostic pairs: current and .previous paths, active status, last success timestamp, duration, file size, record count, last error, recovery source, snapshot sequence watermark, WAL sequences, and last WAL fsync timestamp.

The Python SDK exposes client.save_wait(timeout_ms) and client.persistence_status().

Snapshot Format and Atomic Disk Commit

Checkpoints use the SPDO2 format: a versioned header containing a timestamp,

WAL watermark, and record count; bounded KV records with absolute Unix

expiration timestamps; and a footer with a repeated record count and global

CRC-32C checksum. Recovery parses and validates the full checkpoint into staged

state before swapping it into the live keyspace. Truncation, checksum failures,

invalid lengths, and duplicate keys reject that candidate instead of exposing a

partially restored keyspace. SPDO1 remains readable for legacy migration.

Disk commits create a unique temporary file on the target filesystem, flush and

fsync the file, retain the previous generation as .previous, atomically

rename the new current file, and fsync the parent directory. On startup,

Spedo validates current, can fall back to .previous, and replays valid WAL

records after the selected snapshot watermark.

Crash Testing and Operational Telemetry

Deterministic failpoints are reserved for testing environments:

SPEDO_PERSIST_FAILPOINT=persist.after_temp_create
SPEDO_PERSIST_FAILPOINT=persist.after_write
SPEDO_PERSIST_FAILPOINT=persist.after_file_fsync
SPEDO_PERSIST_FAILPOINT=persist.after_rename
SPEDO_PERSIST_FAILPOINT=persist.after_dir_fsync
SPEDO_PERSIST_FAILPOINT=restore.before_swap

In production, inspect PERSIST.STATUS for the last error, time since the last

successful snapshot, recovery source, and the gap between wal_last_seq and

wal_last_fsynced_seq. Also monitor free space and I/O errors on the backing

volume. A successful local recovery test does not establish high availability,

replication, automatic failover, remote backup, or protection from complete

volume loss.