Active Memory Defragmentation & Lock-Free Snapshot Persistence

Since version v0.75.0

Spedo v0.75.0 introduces native Active Memory Defragmentation and Lock-Free Snapshot Persistence, derived directly from the empirical findings of the continuous 24-hour production and adversarial evaluation (Project YK26, 176.9M operations).

1. Executive Summary & Problem Statement

During long-running, multi-gigabyte production simulations with heavy diurnal churn (bursty write waves, mixed payloads, and massive key deletions):

1. Allocator Fragmentation Drift: Standard memory allocators (including mimalloc and glibc) retain unmapped virtual memory pages in thread-local arenas after key deletions. While logical used_memory decreases, process resident set size (used_memory_rss) could exhibit fragmentation ratios of up to 9x–10x, risking container OOM kills in Kubernetes pods.

2. Snapshot Write Convoy Spikes: Under synchronous snapshotting, holding the write-ahead-log (WAL) lock across physical disk serialization and fsync introduced tail latency spikes up to 24.9 seconds on high-write traffic.

Spedo v0.75.0 eliminates both bottlenecks with zero downtime and standard cloud/container compatibility.

2. Architecture & Mechanisms

Active Arena Memory Purge

Spedo incorporates an active memory defragmentation subsystem that forces allocator arena reclamation and releases unmapped pages directly back to the operating system kernel:

+-------------------------------------------------------------------------+
|                           Spedo Engine Process                          |
|                                                                         |
|  +---------------------------+       Trigger        +----------------+  |
|  | Periodic Defrag Sweeper   | -------------------> | Store::purge   |  |
|  | (SPEDO_ACTIVE_DEFRAG=true)|                      +----------------+  |
|  +---------------------------+                              |           |
|                                                             v           |
|  +---------------------------+       Command        +----------------+  |
|  | RESP Client               | -------------------> | memory_purge   |  |
|  | (MEMORY PURGE)            |                      +----------------+  |
|  +---------------------------+                              |           |
|                                                             |           |
|         +---------------------------------------------------+           |
|         |                                                               |
|         v                                                               v
|  +-----------------------+   +-----------------------+   +-------------------+
|  | mimalloc Arena Purge  |   | macOS Zone Relief     |   | Linux malloc_trim |
|  | mi_collect(force)     |   | malloc_zone_pressure  |   | malloc_trim(0)    |
|  +-----------------------+   +-----------------------+   +-------------------+
|         |                                |                         |
|         +--------------------------------+-------------------------+
|                                          |
|                                          v
|                       [ Release Virtual Memory Pages ]
|                                          |
|                                          v
|                              OS Kernel Page Reclaim
+-------------------------------------------------------------------------+

1. Alloc-level Reclamation: Directly triggers mi_collect(force) via libmimalloc-sys with extended compilation.

2. Kernel-level Trim: Invokes malloc_trim(0) on Linux and malloc_zone_pressure_relief on macOS to drop unused heap pages from the physical process footprint.

3. Observability: Metrics memory_purges and memory_purged_bytes_estimate are exposed in INFO, MEMORY STATS, and Prometheus endpoints.

Lock-Free Asynchronous Snapshot Persistence

In v0.75.0, checkpoint_inner completely isolates memory state extraction from disk I/O:

[ Phase 1: Rapid In-Memory State Capture ]
  WAL Lock Acquired (Microsecond window)
  ├── Advance WAL sequence watermark
  ├── Take atomic snapshot reference (Store::try_snapshot)
  └── Drop WAL Lock Immediately

[ Phase 2: Lock-Free Disk I/O & Fsync ]
  No Engine Locks Held (Zero impact on live SET/GET traffic)
  ├── Format binary checkpoint header & records
  ├── Stream payload to temporary file (`.spedo.snapshot.tmp.pid`)
  ├── Physical fsync to persistent block storage
  └── Atomic filesystem rename (`rename(tmp, target)`)

This guarantees that ongoing client pipelines experience sub-millisecond latencies even while multi-hundred-megabyte checkpoints are being flushed to slow spinning disks or throttled cloud volumes.

3. RESP Commands

MEMORY PURGE / SPEDO.MEMORY.PURGE

Manually instructs the engine to reclaim allocator memory arenas and return unused pages to the kernel.

Syntax:

MEMORY PURGE

Response:

+OK with immediate physical RSS reduction.

MEMORY STATS

Returns real-time memory breakdown including allocation metrics, fragmentation ratios, and active purge counters.

Syntax:

MEMORY STATS

Example Output:

 1) "used_memory"
 2) (integer) 268435456
 3) "used_memory_rss"
 4) (integer) 289406976
 5) "mem_fragmentation_ratio"
 6) "1.08"
 7) "memory_purges"
 8) (integer) 14
 9) "memory_purged_bytes_estimate"
10) (integer) 124780544

MEMORY USAGE <key>

Reports estimated bytes occupied by a specific key in memory.

Syntax:

MEMORY USAGE user:1001

4. Configuration & Tuning

The active memory defragmentation sweeper can be controlled via standard environment variables:

5. Empirical Validation (Project YK26)

Verified across the full 24-hour diurnal lifecycle:

• Total Operations: 176,906,892 requests across 8 operational diurnal waves.
• Tail Latency Spikes: 0 snapshot-related stalls recorded after lock decoupling.
• Fragmentation Ratio: Kept bounded between 1.05x and 1.28x under continuous payload turnover, preventing runaway virtual arena expansion.
• Integrity: 100.00% request success rate, zero canary record corruption.