# Snapshot Reliability & Persistence Plan — Spedo

> **Status:** Implemented for P0–P3 scope in Spedo v0.42.0; verified by automated unit tests, benchmark suites, and `SIGKILL` crash recovery validation.  
> **Initial Durable Scope:** Key-Value and JSON stored within the primary KV keyspace.  
> **Explicitly Scoped for Qualified & Rebuildable Workloads:** Lists, Streams, collections, vectors, matrices, queues, graphs, workflows, scripts, and search indexes.

---

## Purpose and Core Principles

Spedo strictly distinguishes between four distinct operational properties:

1. **File Replacement Atomicity**: A new checkpoint never overwrites an existing healthy snapshot with a partial/corrupted file.
2. **Checkpoint Integrity**: Corruption, truncation, bit-flips, or version incompatibilities are detected prior to memory restoration.
3. **ACKed Write Durability**: Success acknowledgments reflect only the explicitly selected durability configuration.
4. **High Availability**: Scoped as single-node; a local WAL does not constitute multi-node failover.

Deployment targets standard Docker/Kubernetes container volumes (PVCs) using ordinary filesystems and standard TCP/IP networking, with no shared memory or non-portable kernel modules required.

---

## Baseline State

The legacy `SPDO1` snapshot format was an unchecksummed asynchronous KV export. The modern `SPDO2` format and Universal WAL provide structured, verifiable crash recovery with zero data loss under `SPEDO_WAL_FSYNC=always`.

---

## P0 — Contract, Status, and Observable Errors

### Objectives
Eliminate silent failures and provide transparent observability into storage state.

### Deliverables
- `SAVE` initiates asynchronous snapshotting.
- `SAVE WAIT <timeout_ms>` blocks until disk commit is confirmed.
- `PERSIST.STATUS` reports mode, path, active snapshot status, last success timestamp, file size, duration, record count, previous errors, WAL sequence IDs, and fsync mode.
- Success counters update strictly after write, `fsync`, and atomic rename succeed; I/O errors are preserved and exposed.
- No background snapshot threads execute when `SPEDO_PERSIST_PATH` is disabled.
- Documentation accurately classifies the mechanism as single-node KV/JSON checkpointing with optional synchronous WAL durability.

### Gate P0
Any `ENOSPC`, `EIO`, permission error, partial write, `fsync`, or rename failure raises an observable error, records no false successes, and preserves the last healthy checkpoint.

---

## P1 — Resilient & Verifiable SPDO2 Snapshots

### Target Format
`SPDO2` is a versioned, bounded binary format validated before memory mutation:

```text
magic/version | created_at_unix_ms | record_count
Canonical KV records: type | key_len | value_len | expires_at_unix_ms | key | value
Footer: marker | record_count | global CRC32 checksum
```

- Expirations are absolute unix timestamps: system downtime decrements TTL.
- Compressed and modeled values are materialized in canonical form.
- Tiered ColdArchive values are resolved during snapshotting; failures abort the checkpoint rather than writing empty stubs.

### Disk Commit Pipeline
1. Create a unique temporary file on the target filesystem.
2. Stream records and execute physical `fsync()` on the file descriptor.
3. Atomically rename the temporary file to the target checkpoint path.
4. Execute `fsync()` on the parent directory.
5. Retain a recoverable `.previous` snapshot copy before replacement.

### Recovery Mechanics
- Parse and validate the entire file, record counter, and CRC32 checksum before loading into active memory.
- Atomic state swap: zero partial state restoration.
- Automatic fallback to `.previous` if the primary checkpoint fails validation.

---

## P2 — Crash Testing & Operational Telemetry

### Failpoints & Deterministic Test Harness
Automated CI tests simulate hardware and OS failures:
- `persist.after_temp_create`
- `persist.after_write`
- `persist.after_file_fsync`
- `persist.after_rename`
- `persist.after_dir_fsync`
- `restore.before_swap`

### Test Suite Scope
- File truncation across all byte offsets, bit flips, corrupted record lengths, and invalid checksums.
- Out-of-disk-space (`ENOSPC`), permission denials, and rename failures.
- Instant `SIGKILL` during active snapshotting and immediate restart validation.
- Multi-shard writes during background snapshotting.
- Repeated write → checkpoint → kill → restart stress cycles.

---

## P3 — Single-Node Universal WAL / AOF

### Design
- Append-only WAL with monotonic sequence IDs, record types, length headers, and CRC checksums.
- Canonical state recording (`SET`, `DEL`, `FLUSH`) with absolute expiration timestamps for deterministic replay.
- Configurable `SPEDO_WAL_FSYNC` modes:
  - `always`: Write ACK is transmitted only after WAL append and physical `fsync()` complete.
  - `everysec`: Background fsync every second (bounded 1s RPO).
  - `none`: Maximum throughput without synchronous durability guarantees.
- Checkpoints embed the active WAL sequence; recovery restores the validated SPDO2 snapshot and replays subsequent WAL mutations.

---

## Validation Summary

- Verified by automated Rust integration tests and Python crash recovery harnesses.
- Tested against unexpected `SIGKILL` termination: confirmed zero lost writes under `SPEDO_WAL_FSYNC=always`.
- All benchmark suites (`make test`, `make matrix`, `make wow`, `make bench-pipeline`, `make bench-spedo-capabilities`) execute and pass cleanly.