PsyKoTest: Adversarial Stress & Chaos Engineering
Uncompromising multi-vector robustness testing for Spedo as a single-node engine operating under distributed multi-tenant pressure. Protocol fuzzing, concurrent lock contention, memory churn, and semantic oracles.
Dual-AI Red-Team vs Blue-Team Methodology
Standard database benchmarks measure throughput by throwing requests at a cache and discarding responses. In contrast, PsyKoTest treats Spedo as a production target under adversarial conditions:
- Red-Team Chaos Vectors: Split valid binary frames across individual TCP bytes, issue abrupt TCP resets (
RST), trigger simultaneous cache miss storms across 32 barrier-synchronized microservices, thrash memory under full LRU saturation, and inject signed 64-bit integer overflows. - Blue-Team Diagnostic Oracles: Verify exact SHA-256 payload digests across all acknowledged writes, assert single-winner mutexes on
SET NX, measure OS-level Resident Set Size (RSS) drift vs internal accounting, and prove zero false-negative shortcut misses.
Architectural Breakthroughs in v0.71.0
1. LRU Eviction Lock Convoy Remediation
Under heavy concurrent load with saturated memory, Spedo previously scanned all 256 shards sequentially while holding the exclusive global admission lock. This caused severe lock convoys with P99 latency reaching 43.02 ms.
The Solution: In v0.71.0, Spedo introduces Sampled Candidate Selection.
Instead of traversing every shard, the eviction loop samples 16 rotating shards and inspects up to 16 entries per shard (evaluating up to 256 candidate keys).
The candidate with the lowest last_access is evicted in sub-microsecond time:
- P99 Eviction Latency: Slashed from 43.02 ms down to 3.60 ms (a 12x latency reduction).
- Eviction Thrashing Throughput: Boosted by 3.3x (from 2,348 QPS to 7,798 QPS).
2. Zero False-Miss Deterministic Negative Caching
Previous client SDKs used a probabilistic Bloom filter to shortcut negative reads.
Because Bloom filters have a false-positive rate, valid server keys (e.g. user:3053) occasionally collided with negative bits, causing the client proxy to return None without querying Spedo.
The Solution: Replaced with NegativeCache, an exact bounded LRU map with TTL:
- 0% False Positives: Only explicitly confirmed
Noneresponses from Spedo are tracked. Valid keys are never falsely masked. - Granular Invalidation: Invalidation events selectively remove individual keys rather than wiping the entire negative filter.
3. Atomic SET NX / SET XX Condition Evaluation
Previously, SET NX checked key presence through a separate exists() check before mutating storage, allowing multiple concurrent clients to simultaneously win mutex locks.
The Solution: The condition is now evaluated directly under the shard write lock during set_inner. Mutex contention trials pass with strictly 1 winner per round across all threads.
4. Active TTL Expiration & Watch Outage Resilience
Authoritative TTL expiry is now actively purged via a 50ms background task and on-access checks, publishing real-time SPEDO.WATCH invalidations. The SDK invalidates local caches upon reconnection to eliminate stale state across network gaps.
Chaos Scenario Matrix & Verified Oracles
| Scenario | Attack Vector | Correctness Oracle | Verdict |
|---|---|---|---|
| P0_Inventory_Integrity | 8 concurrent threads, 0-byte to 64KB binary payloads, nulls, unicode | 0 missing keys, 0 SHA-256 mismatches, 0 length discrepancies | PASS |
| P0_Atomic_Contention | 32 barrier-synchronized clients racing INCR and SET NX |
Strictly unique sequence, sum matches ops, 1 winner per round (0 splits) | PASS |
| P0_Rejected_Overwrite_OOM | Oversized mutation into full 1MB noeviction cache |
Explicit OOM, old payload preserved intact, TTL preserved | PASS |
| P0_Negative_Cache_Bug | 60,000 negative keys queried, then real server keys read via SDK | 0 false misses, real keys return authoritative data | PASS |
| P0_RESP_Fuzzer | Byte-by-byte frame splitting, CRLF splits, 2GB length header, RST drop | No panics, no memory runaway, immediate probe recovery | PASS |
| P1_Thundering_Herd_Miss_Storm | 32 clients simultaneously miss on expired hot key, race for mutex | 1 thread recomputes, 0 lock splits, fresh value safely published | PASS |
| P1_Memory_Churn_RSS_Drift | 5 cycles of writing & deleting 2,000 keys, measuring OS RSS | Plateau at +2.4MB allocator warmup; 0 linear memory drift | PASS |
| P1_Eviction_Lock_Convoy | 12 concurrent workers writing to saturated 4MB LRU cache | P50 < 2ms, P99 < 4ms, zero deadlocks or thread stalls | PASS |
| P2_Integer_Boundary_Overflow | INCR on i64::MAX (9223372036854775807), non-numeric mutations |
Checked arithmetic prevents wrap-around, value preserved | PASS |
Adversarial Full Profile Campaign (psyko.py --profile full --repeat 2)
52 / 52 PASS · 0 FAILURES
To provide rigorous independent verification, an autonomous adversarial audit was executed using psyko.py.
The full profile subjects Spedo to 48 concurrent worker clients running 2,000 operations per worker across 2 consecutive complete repetitions (~200,000 operations) with zero degradation:
| Adversarial Scenario | Repeat 0 | Repeat 1 | Details & Verified Oracles |
|---|---|---|---|
| acknowledged_integrity | PASS | PASS | 48 clients × 2,000 ops = 96,000 ops/run; 0 unexpected misses, 0 SHA-256 mismatches |
| atomic_contention | PASS | PASS | 96,000 atomic INCRs (exact sum), 12 SET NX rounds with strictly 1 winner (0 lock split) |
| ttl_aba_binary | PASS | PASS | Strict expiration under rapid mutation; zero key resurrection |
| fragmented_pipeline | PASS | PASS | 200 pipelined requests fragmented byte-by-byte over TCP |
| large_reply_pipeline | PASS | PASS | 42 MB reply payload streaming without memory bloat |
| malformed_slow_clients | PASS | PASS | Malformed frames and partial clients; server PING latency maintained < 0.6 ms |
| slow_readers | PASS | PASS | 16 slow clients queueing up to 268 MB of buffer without server starvation |
| ambiguous_disconnect | PASS | PASS | Abrupt TCP drops without ACK; zero data corruption or torn writes |
| memory_churn | PASS | PASS | 35 alloc/delete cycles; logical memory returns to exact byte baseline |
| tiny_key_churn | PASS | PASS | 12,000 keys × 3 cycles; complete memory reclamation |
| sdk.local_proxy.negative_collision | PASS | PASS | 0 false misses verified on spedo and spedo_client (deterministic NegativeCache) |
| sdk.local_proxy.ttl & mutations | PASS | PASS | Convergence under 2.0s SLO (< 0.27s actual) and instant per-key invalidation |
| sdk.local_proxy.watch_gap | PASS | PASS | Watch stream cut and reconnection with bounded safe local cache flush |
| sdk.bind_var.final_convergence | PASS | PASS | Reactive live variables synchronized across 32 concurrent writers |
| volatile_restart | PASS | PASS | Clean volatile restart with zero lingering unpersisted keys |
| noeviction_pressure | PASS | PASS | 127 keys admitted before capacity; atomic OOM without corrupting existing records |
| eviction_pressure | PASS | PASS | Sampled candidate eviction: zero lock convoys, smooth sub-millisecond throughput |
| sigkill_recovery | PASS | PASS | SIGKILL process termination under WAL always: 60/60 keys recovered, expired purged |
Executing PsyKoTest Locally
The test suite is completely self-contained within the repository and requires zero external Python dependencies:
# Navigate to the PsyKoTest framework directory
cd benchmarks/PsyKoTest
# Run all 9 chaos scenarios against an isolated disposable instance
./run_psyko.sh --all
# Or run individual targeted scenarios
python3 psyko_runner.py --scenario p1_eviction_convoy
python3 psyko_runner.py --scenario p0_negative_cache
Results are automatically exported to machine-readable psyko_results.json and formatted into PSYKO_REPORT.md.