REPRODUCIBLE EVALUATION

Benchmark a release, not a promise.

A Spedo result is meaningful only for the release, command path, data shape, persistence mode, client topology, host, and concurrency that produced it. Artifact v3 records that evidence and prevents an exploratory run from silently becoming a product claim.

Minimum checks

Start with make test and exercise the exact command semantics your application needs.
Use make matrix and make bench-pipeline for direct-server scenarios; keep local-client measurements separate.
When testing optional features, use their dedicated suite and state that they are preview or experimental.
For recovery-sensitive KV/JSON evaluation, record SPEDO_PERSIST_PATH, the exact SPEDO_WAL_FSYNC value, recovery policy, and volume type.
Capture raw trials and resource quality. A missing measurement is unavailable, not zero.

Artifact v3: the evidence envelope

A v3 JSON artifact is valid only when its summaries can be recomputed from its raw trials and its execution context is explicit. The contract supports core_resp_direct, sdk_capability, and feature_scenario runs.

Section	Required evidence
`run`	Unique ID, start and finish timestamps, completion status, publication eligibility, warnings, and errors.
`provenance`	Spedo version, full Git commit, clean/dirty state, producer and configuration hashes, immutable image identities, sanitized argv, and secret-redaction confirmation.
`environment`	OS, architecture, CPU and memory facts, container runtime, topology, and explicit resource limits.
`workload`	Seed, keyspace, payload, warm-up, sample duration, repetitions, workers, pipeline depth, operation, and key distribution.
`trials`	Every raw repetition, execution order, elapsed nanoseconds, completed operations, ACK semantics, comparability, and resource sample.
`summaries`	Median, sample coefficient of variation, exact trial IDs, and only recalculable comparisons.
`resources`	Collector source, sample interval, quality, and scope notes; unavailable values use `null`.

A dirty-tree run is never publishable

Local experiments from a modified working tree are useful for diagnosis. They must set provenance.git.dirty=true, include a sanitized diff SHA-256, and set run.publication_eligible=false. The publication validator rejects them even when their numbers look favorable.

Do not copy a dirty-tree result into the dashboard, release notes, website, sales material, or a Redis/Spedo ratio. Re-run the final commit from a clean tree and retain the new raw artifact.

A clean artifact records dirty=false and diff_sha256=null. It must also identify the benchmark producer and configuration by SHA-256 and the executed images by immutable identity.

Validate before publication

Run the strict validator against the final artifact:

python3 scripts/validate_benchmark_artifact.py \
  --publication \
  --write-sha256 \
  path/to/benchmark-artifact.json

The command verifies structural and semantic rules, recomputes rates and summaries, checks comparison fairness, detects reused raw measurements when multiple artifacts are supplied, and writes a detached .sha256 file. The checksum covers the exact JSON bytes; it is not embedded inside the JSON it hashes.

Structural validity is necessary but not sufficient: the producer still has to measure the workload it claims to measure.

Keep unlike operations separate

ACK semantics	What the measured operation completed	Server ratio allowed?
`confirmed_resp`	The server returned the RESP result.	Yes, only with strict paired Redis/Spedo trials.
`confirmed_durable`	The documented durable acknowledgement boundary was reached.	No direct ratio against a non-durable operation.
`local_enqueue`	A client-side queue accepted the operation; remote flush may still be pending.	No.
`local_l1`	An in-process cache returned the value without a server round trip.	No.
`offline_compute`	A local algorithm completed outside the server request path.	No.

A Redis/Spedo ratio is accepted only when both targets use confirmed_resp, strict comparability, the same comparison group, and paired workload conditions. SDK enqueue speed and local-cache reads remain useful capability measurements, but they are not server throughput comparisons.

Publish the context, not just the winning number

Report all scenario medians and variability, including losses and regressions.
State server versus local-client scope, payload and key distribution, warm-up, repetitions, concurrency, and pipeline depth.
State exact persistence settings. In particular, disabled opens no WAL, while none and its off alias still append to a WAL without calling fsync.
Report CPU, RSS, network, and I/O collector quality alongside throughput and latency, or mark those measurements unavailable.
Preserve warnings, errors, raw trials, the JSON artifact, and its detached checksum.

No benchmark alone establishes high availability, replication, automatic failover, full Redis compatibility, universal latency, strict CPU pinning, zero loss, or production readiness. Those properties require their own implementation, fault tests, and published operating contract.

← Back to Documentation Read Scope & Limits 📊 Open Dashboard