Benchmark report (PRD milestone 7)

Copy Markdown View Source

This is a preliminary, single-container smoke-scale run, not the PRD section 9.6 dedicated-hardware benchmark. It exists to prove the harness (bench/run.exs) produces sane, directionally-correct numbers, and to give a rough sense of where feoxdb_ex sits relative to CubDB, Cachex, :dets, :mnesia, and raw :ets. It is not a substitute for running the harness on real, controlled hardware before publishing performance claims.

Environment

HostShared CI/sandbox container (not a dedicated benchmark host)
CPUIntel(R) Xeon(R) Processor @ 2.80GHz, 4 cores
CPU frequency scalingNot controlled/disabled — a PRD 9.6 requirement this run does not meet
Kernel6.18.44-fc-v33
OSUbuntu 24.04.4 LTS
Filesystemfeox_persistent/CubDB/:dets/:mnesia data files stored on tmpfs (/dev/shm), not a dedicated NVMe/io_uring-capable disk — see caveats below
Elixir / OTP1.20.4 / 29
feoxdb crate0.6.0, jemalloc disabled (see native/feoxdb_nif/Cargo.toml)

Configuration

Dataset size20,000 keys
Value size64 bytes
Concurrency levels1, 4 (Benchee parallel:)
Measurement / warmup time2.0s / 0.5s per job

Run with:

BENCH_DATASET_SIZE=20000 BENCH_VALUE_SIZE=64 BENCH_CONCURRENCY=1,4 \
  BENCH_TIME_S=2 BENCH_WARMUP_S=0.5 MIX_ENV=bench mix run bench/run.exs

Fairness settings applied

  • CubDB: auto_compact: false, auto_file_sync: true (explicit, not left at whatever the library's default happens to be).
  • :dets: opened with type: :set, flush/1 calls :dets.sync/1.
  • :mnesia: table created with disc_copies: [node()] (so it's writing to disk, not ram_copies) and driven via :mnesia.dirty_* operations — the cheapest, least-safe API mnesia offers, included as a best-case-for-mnesia comparison point, not a transactional one.
  • feox_persistent: preload is followed by an explicit FeoxDB.flush/1 call, so background write time is counted.
  • Same generated keys/values fed to every system.
  • Benchee's warmup phase serves as the "discard first run" rule.

Results

Full console + HTML output is regenerated by rerunning bench/run.exs (bench/output/, gitignored). Medians below (average/99th percentile are in the raw output, reproduced further down):

Each workload is split into memory-backed systems (:ets, feox_memory, cachex — no durability) and disk-backed systems (feox_persistent, :dets, :mnesia with disc_copies, CubDB), since comparing across that line is comparing different guarantees, not just different implementations.

Random read (workload 2)

Memory-backed

Systemparallel: 1 (median)parallel: 4 (median)
:ets (control)79.8 μs81.2 μs
cachex83.0 μs81.2 μs
feox_memory87.3 μs120.8 μs

Disk-backed

Systemparallel: 1 (median)parallel: 4 (median)
mnesia79.8 μs90.7 μs
feox_persistent95.9 μs115.6 μs
dets113.7 μs220.5 μs
cubdb185.3 μs638.1 μs

feoxdb_ex (memory mode) tracks raw ETS closely at parallel: 1 but falls behind at parallel: 4 in this environment. In the disk-backed group, mnesia's dirty reads are the fastest — unsurprising since dirty reads don't go through a transaction — and feoxdb_ex (persistent mode) stays well ahead of :dets and CubDB at both levels. Note: at this value size (64 bytes, well within the BEAM's small-integer/immediate binary range) and dataset size, absolute latencies here (tens of μs) are dominated by scheduling/measurement overhead in this container, not by the stores themselves — the relative ordering is the meaningful part of this result, not the absolute numbers.

Write only (workload 4)

Memory-backed

Systemparallel: 1 (median)parallel: 4 (median)
:ets (control)1.27 μs2.17 μs
cachex1.85 μs2.86 μs
feox_memory5.48 μs23.9 μs

Disk-backed

Systemparallel: 1 (median)parallel: 4 (median)
dets2.28 μs10.58 μs
mnesia3.60 μs7.94 μs
feox_persistent24.8 μs28.3 μs
cubdb233.4 μs1054.0 μs

dets's median write is competitive (its own write-buffer/log mechanism is also bursty — see its 99th percentile in the raw output below, which spikes into the milliseconds under parallel: 4). feox_persistent's number is noisy (see caveat below on write-buffer backpressure) but its median stays in the tens-of-μs range once warmed up — the average is pulled up by the tail, not the typical case.

Mixed 80/20 (workload 6)

Memory-backed

Systemparallel: 1 (median)parallel: 4 (median)
:ets (control)77.9 μs79.6 μs
cachex79.3 μs81.3 μs
feox_memory86.0 μs117.5 μs

Disk-backed

Systemparallel: 1 (median)parallel: 4 (median)
mnesia88.2 μs88.9 μs
feox_persistent86.7 μs164.6 μs
dets118.8 μs262.9 μs
cubdb202.7 μs681.2 μs

Delete (workload 8)

Memory-backed

Systemparallel: 1 (median)parallel: 4 (median)
:ets (control)0.23 μs0.43 μs
cachex0.69 μs0.82 μs
feox_memory2.87 μs13.3 μs

Disk-backed

Systemparallel: 1 (median)parallel: 4 (median)
dets1.98 μs8.71 μs
mnesia2.04 μs3.54 μs
feox_persistent3.05 μs9.66 μs
cubdb183.3 μs262.6 μs

An observed finding: persistent-mode write buffer backpressure

Running this harness produced feox: write entry for a N byte key has been retried N times warnings on stderr from the feoxdb crate during the write-heavy scenarios (write-only, delete) in feox_persistent mode. This is FeOxDB's own write-buffer logging that it's retrying a buffered write, not a crash or data-loss signal, and it explains why feox_persistent's average latency and 99th percentile are much worse than its median in the write-only scenario above: under sustained, uninterrupted write pressure at these iteration rates, the background write-behind workers (PRD section 6, "100ms or 16MB per shard" flush window) fall behind and the write path backs up.

This is worth flagging rather than hiding: a production workload that sustains a similar write rate against persistent-mode feoxdb_ex should expect the same backpressure behavior, and this harness's numbers likely undersell it. This finding should be validated against the PRD's own Section 9.5 scheduler-health methodology (not yet implemented — see bench/README.md) before being treated as conclusive.

Raw console output

Full Benchee console output (click to expand) ``` *** random_read (parallel: 1) *** Name ips average deviation median 99th % ets 12.34 K 81.04 μs ±30.20% 79.78 μs 137.93 μs mnesia 12.18 K 82.13 μs ±30.76% 79.81 μs 153.75 μs cachex 11.82 K 84.58 μs ±27.76% 82.98 μs 152.29 μs feox_memory 11.12 K 89.97 μs ±34.58% 87.33 μs 173.20 μs feox_persistent 10.19 K 98.14 μs ±33.01% 95.92 μs 173.98 μs dets 8.50 K 117.62 μs ±31.27% 113.73 μs 229.33 μs cubdb 5.13 K 194.75 μs ±38.63% 185.25 μs 381.47 μs *** write_only (parallel: 1) *** Name ips average deviation median 99th % ets 657.05 K 1.52 μs ±485.77% 1.27 μs 5.08 μs cachex 429.08 K 2.33 μs ±1236.17% 1.85 μs 6.05 μs feox_memory 167.62 K 5.97 μs ±185.79% 5.48 μs 17.41 μs mnesia 155.89 K 6.41 μs ±549.31% 3.60 μs 36.50 μs feox_persistent 36.72 K 27.23 μs ±196.23% 24.78 μs 127.70 μs dets 16.62 K 60.17 μs ±1336.53% 2.28 μs 64.49 μs cubdb 3.99 K 250.65 μs ±85.33% 233.42 μs 404.09 μs *** mixed_80_20 (parallel: 1) *** Name ips average deviation median 99th % ets 12.55 K 79.69 μs ±29.50% 77.87 μs 141.90 μs cachex 12.24 K 81.68 μs ±30.50% 79.27 μs 152.18 μs feox_memory 11.37 K 87.97 μs ±31.82% 86.02 μs 154.88 μs mnesia 11.00 K 90.87 μs ±32.06% 88.24 μs 169.71 μs feox_persistent 10.74 K 93.11 μs ±118.92% 86.69 μs 178.22 μs dets 7.85 K 127.33 μs ±36.72% 118.75 μs 264.55 μs cubdb 4.41 K 226.72 μs ±39.90% 202.74 μs 504.15 μs *** delete (parallel: 1) *** Name ips average deviation median 99th % ets 3738.37 K 0.27 μs ±5506.11% 0.23 μs 0.45 μs cachex 1275.05 K 0.78 μs ±223.71% 0.69 μs 1.55 μs mnesia 325.26 K 3.07 μs ±663.83% 2.04 μs 23.89 μs feox_memory 312.12 K 3.20 μs ±73.92% 2.87 μs 10.51 μs feox_persistent 273.69 K 3.65 μs ±103.04% 3.05 μs 20.11 μs dets 127.61 K 7.84 μs ±614.37% 1.98 μs 391.79 μs cubdb 5.50 K 181.97 μs ±18.75% 183.30 μs 272.41 μs *** random_read (parallel: 4) *** Name ips average deviation median 99th % ets 12.04 K 83.08 μs ±66.93% 81.20 μs 142.41 μs cachex 11.59 K 86.26 μs ±139.16% 81.24 μs 160.85 μs mnesia 9.08 K 110.14 μs ±56.27% 90.72 μs 307.71 μs feox_persistent 8.16 K 122.52 μs ±58.35% 115.58 μs 262.52 μs feox_memory 7.60 K 131.62 μs ±72.93% 120.75 μs 296.83 μs dets 3.98 K 251.54 μs ±81.64% 220.48 μs 649.32 μs cubdb 1.47 K 681.04 μs ±41.53% 638.09 μs 1316.96 μs *** write_only (parallel: 4) *** Name ips average deviation median 99th % ets 334.88 K 2.99 μs ±695.11% 2.17 μs 8.31 μs cachex 276.94 K 3.61 μs ±530.33% 2.86 μs 9.62 μs mnesia 59.87 K 16.70 μs ±438.83% 7.94 μs 143.39 μs feox_memory 29.55 K 33.84 μs ±115.64% 23.89 μs 118.62 μs feox_persistent 15.10 K 66.20 μs ±997.97% 28.27 μs 794.28 μs dets 3.21 K 311.26 μs ±686.99% 10.58 μs 8190.16 μs cubdb 0.79 K 1272.06 μs ±63.11% 1054.00 μs 5205.89 μs *** mixed_80_20 (parallel: 4) *** Name ips average deviation median 99th % ets 11.82 K 84.63 μs ±163.88% 79.60 μs 148.14 μs cachex 11.25 K 88.87 μs ±180.58% 81.25 μs 164.85 μs mnesia 9.29 K 107.68 μs ±172.23% 88.91 μs 428.43 μs feox_memory 7.59 K 131.80 μs ±125.35% 117.54 μs 304.62 μs feox_persistent 5.00 K 199.97 μs ±156.60% 164.57 μs 672.27 μs dets 3.30 K 303.10 μs ±103.31% 262.91 μs 865.34 μs cubdb 0.97 K 1030.96 μs ±96.62% 681.24 μs 4900.36 μs *** delete (parallel: 4) *** Name ips average deviation median 99th % ets 1750.73 K 0.57 μs ±3317.33% 0.43 μs 0.93 μs cachex 952.19 K 1.05 μs ±1319.98% 0.82 μs 1.90 μs mnesia 98.58 K 10.14 μs ±716.83% 3.54 μs 80.08 μs feox_persistent 56.30 K 17.76 μs ±298.57% 9.66 μs 74.62 μs feox_memory 40.18 K 24.89 μs ±313.01% 13.30 μs 86.98 μs dets 33.32 K 30.01 μs ±576.52% 8.71 μs 607.42 μs cubdb 3.68 K 271.63 μs ±23.34% 262.56 μs 440.07 μs ```

Memory and reduction-count tables are in the full run output; broadly, feoxdb_ex (both modes) uses less BEAM heap per operation than :ets's own reported figure (because the value lives on the Rust side, not the BEAM heap) and far less than CubDB, whose GenServer-mediated access adds substantial per-call reduction and allocation overhead. :mnesia's dirty operations and :dets's port-based access both allocate more per call than :ets as well, though less than CubDB.

Caveats / what this run does not establish

  • Single container, not the PRD's two dedicated hosts (x86_64 + Apple Silicon).
  • feox_persistent, CubDB, :dets, and :mnesia data files were placed on tmpfs (/dev/shm) specifically to avoid the write-buffer backpressure noted above swamping stderr; a real disk (with or without io_uring) will behave differently, likely worse for persistent-mode latency across all of them.
  • :mnesia is exercised only via dirty_read/dirty_write/ dirty_delete with a single disc_copies replica — not a transactional or multi-node comparison, which would show materially different (worse) numbers.
  • No disabled CPU frequency scaling, unverified io_uring availability.
  • Only 4 of the PRD's 8 workloads are implemented (see bench/README.md).
  • Only one value size (64 bytes) was run; the PRD also wants 1KB and 64KB.
  • No scheduler-health run (PRD section 9.5) is implemented yet.

Re-run bench/run.exs with a larger BENCH_DATASET_SIZE, the full BENCH_CONCURRENCY list, and each of the three value sizes on appropriate hardware before treating any of these numbers as publishable.