This is a preliminary, single-container smoke-scale run, not the PRD
section 9.6 dedicated-hardware benchmark. It exists to prove the
harness (bench/run.exs) produces sane, directionally-correct numbers,
and to give a rough sense of where feoxdb_ex sits relative to CubDB,
Cachex, :dets, :mnesia, and raw :ets. It is not a substitute for
running the harness on real, controlled hardware before publishing
performance claims.
Environment
| Host | Shared CI/sandbox container (not a dedicated benchmark host) |
| CPU | Intel(R) Xeon(R) Processor @ 2.80GHz, 4 cores |
| CPU frequency scaling | Not controlled/disabled — a PRD 9.6 requirement this run does not meet |
| Kernel | 6.18.44-fc-v33 |
| OS | Ubuntu 24.04.4 LTS |
| Filesystem | feox_persistent/CubDB/:dets/:mnesia data files stored on tmpfs (/dev/shm), not a dedicated NVMe/io_uring-capable disk — see caveats below |
| Elixir / OTP | 1.20.4 / 29 |
| feoxdb crate | 0.6.0, jemalloc disabled (see native/feoxdb_nif/Cargo.toml) |
Configuration
| Dataset size | 20,000 keys |
| Value size | 64 bytes |
| Concurrency levels | 1, 4 (Benchee parallel:) |
| Measurement / warmup time | 2.0s / 0.5s per job |
Run with:
BENCH_DATASET_SIZE=20000 BENCH_VALUE_SIZE=64 BENCH_CONCURRENCY=1,4 \
BENCH_TIME_S=2 BENCH_WARMUP_S=0.5 MIX_ENV=bench mix run bench/run.exsFairness settings applied
- CubDB:
auto_compact: false, auto_file_sync: true(explicit, not left at whatever the library's default happens to be). :dets: opened withtype: :set,flush/1calls:dets.sync/1.:mnesia: table created withdisc_copies: [node()](so it's writing to disk, notram_copies) and driven via:mnesia.dirty_*operations — the cheapest, least-safe API mnesia offers, included as a best-case-for-mnesia comparison point, not a transactional one.feox_persistent: preload is followed by an explicitFeoxDB.flush/1call, so background write time is counted.- Same generated keys/values fed to every system.
- Benchee's warmup phase serves as the "discard first run" rule.
Results
Full console + HTML output is regenerated by rerunning bench/run.exs
(bench/output/, gitignored). Medians below (average/99th percentile
are in the raw output, reproduced further down):
Each workload is split into memory-backed systems (:ets, feox_memory,
cachex — no durability) and disk-backed systems (feox_persistent,
:dets, :mnesia with disc_copies, CubDB), since comparing across
that line is comparing different guarantees, not just different
implementations.
Random read (workload 2)
Memory-backed
| System | parallel: 1 (median) | parallel: 4 (median) |
|---|---|---|
:ets (control) | 79.8 μs | 81.2 μs |
| cachex | 83.0 μs | 81.2 μs |
| feox_memory | 87.3 μs | 120.8 μs |
Disk-backed
| System | parallel: 1 (median) | parallel: 4 (median) |
|---|---|---|
| mnesia | 79.8 μs | 90.7 μs |
| feox_persistent | 95.9 μs | 115.6 μs |
| dets | 113.7 μs | 220.5 μs |
| cubdb | 185.3 μs | 638.1 μs |
feoxdb_ex (memory mode) tracks raw ETS closely at parallel: 1 but
falls behind at parallel: 4 in this environment. In the disk-backed
group, mnesia's dirty reads are the fastest — unsurprising since
dirty reads don't go through a transaction — and feoxdb_ex
(persistent mode) stays well ahead of :dets and CubDB at both
levels. Note: at this value size (64 bytes, well within the BEAM's
small-integer/immediate binary range) and dataset size, absolute
latencies here (tens of μs) are dominated by scheduling/measurement
overhead in this container, not by the stores themselves — the
relative ordering is the meaningful part of this result, not the
absolute numbers.
Write only (workload 4)
Memory-backed
| System | parallel: 1 (median) | parallel: 4 (median) |
|---|---|---|
:ets (control) | 1.27 μs | 2.17 μs |
| cachex | 1.85 μs | 2.86 μs |
| feox_memory | 5.48 μs | 23.9 μs |
Disk-backed
| System | parallel: 1 (median) | parallel: 4 (median) |
|---|---|---|
| dets | 2.28 μs | 10.58 μs |
| mnesia | 3.60 μs | 7.94 μs |
| feox_persistent | 24.8 μs | 28.3 μs |
| cubdb | 233.4 μs | 1054.0 μs |
dets's median write is competitive (its own write-buffer/log
mechanism is also bursty — see its 99th percentile in the raw output
below, which spikes into the milliseconds under parallel: 4).
feox_persistent's number is noisy (see caveat below on write-buffer
backpressure) but its median stays in the tens-of-μs range once warmed
up — the average is pulled up by the tail, not the typical case.
Mixed 80/20 (workload 6)
Memory-backed
| System | parallel: 1 (median) | parallel: 4 (median) |
|---|---|---|
:ets (control) | 77.9 μs | 79.6 μs |
| cachex | 79.3 μs | 81.3 μs |
| feox_memory | 86.0 μs | 117.5 μs |
Disk-backed
| System | parallel: 1 (median) | parallel: 4 (median) |
|---|---|---|
| mnesia | 88.2 μs | 88.9 μs |
| feox_persistent | 86.7 μs | 164.6 μs |
| dets | 118.8 μs | 262.9 μs |
| cubdb | 202.7 μs | 681.2 μs |
Delete (workload 8)
Memory-backed
| System | parallel: 1 (median) | parallel: 4 (median) |
|---|---|---|
:ets (control) | 0.23 μs | 0.43 μs |
| cachex | 0.69 μs | 0.82 μs |
| feox_memory | 2.87 μs | 13.3 μs |
Disk-backed
| System | parallel: 1 (median) | parallel: 4 (median) |
|---|---|---|
| dets | 1.98 μs | 8.71 μs |
| mnesia | 2.04 μs | 3.54 μs |
| feox_persistent | 3.05 μs | 9.66 μs |
| cubdb | 183.3 μs | 262.6 μs |
An observed finding: persistent-mode write buffer backpressure
Running this harness produced feox: write entry for a N byte key has been retried N times warnings on stderr from the feoxdb crate
during the write-heavy scenarios (write-only, delete) in feox_persistent
mode. This is FeOxDB's own write-buffer logging that it's retrying a
buffered write, not a crash or data-loss signal, and it explains why
feox_persistent's average latency and 99th percentile are much
worse than its median in the write-only scenario above: under
sustained, uninterrupted write pressure at these iteration rates, the
background write-behind workers (PRD section 6, "100ms or 16MB per
shard" flush window) fall behind and the write path backs up.
This is worth flagging rather than hiding: a production workload that
sustains a similar write rate against persistent-mode feoxdb_ex should
expect the same backpressure behavior, and this harness's numbers likely
undersell it. This finding should be validated against the PRD's own
Section 9.5 scheduler-health methodology (not yet implemented — see
bench/README.md) before being treated as conclusive.
Raw console output
Full Benchee console output (click to expand)
``` *** random_read (parallel: 1) *** Name ips average deviation median 99th % ets 12.34 K 81.04 μs ±30.20% 79.78 μs 137.93 μs mnesia 12.18 K 82.13 μs ±30.76% 79.81 μs 153.75 μs cachex 11.82 K 84.58 μs ±27.76% 82.98 μs 152.29 μs feox_memory 11.12 K 89.97 μs ±34.58% 87.33 μs 173.20 μs feox_persistent 10.19 K 98.14 μs ±33.01% 95.92 μs 173.98 μs dets 8.50 K 117.62 μs ±31.27% 113.73 μs 229.33 μs cubdb 5.13 K 194.75 μs ±38.63% 185.25 μs 381.47 μs *** write_only (parallel: 1) *** Name ips average deviation median 99th % ets 657.05 K 1.52 μs ±485.77% 1.27 μs 5.08 μs cachex 429.08 K 2.33 μs ±1236.17% 1.85 μs 6.05 μs feox_memory 167.62 K 5.97 μs ±185.79% 5.48 μs 17.41 μs mnesia 155.89 K 6.41 μs ±549.31% 3.60 μs 36.50 μs feox_persistent 36.72 K 27.23 μs ±196.23% 24.78 μs 127.70 μs dets 16.62 K 60.17 μs ±1336.53% 2.28 μs 64.49 μs cubdb 3.99 K 250.65 μs ±85.33% 233.42 μs 404.09 μs *** mixed_80_20 (parallel: 1) *** Name ips average deviation median 99th % ets 12.55 K 79.69 μs ±29.50% 77.87 μs 141.90 μs cachex 12.24 K 81.68 μs ±30.50% 79.27 μs 152.18 μs feox_memory 11.37 K 87.97 μs ±31.82% 86.02 μs 154.88 μs mnesia 11.00 K 90.87 μs ±32.06% 88.24 μs 169.71 μs feox_persistent 10.74 K 93.11 μs ±118.92% 86.69 μs 178.22 μs dets 7.85 K 127.33 μs ±36.72% 118.75 μs 264.55 μs cubdb 4.41 K 226.72 μs ±39.90% 202.74 μs 504.15 μs *** delete (parallel: 1) *** Name ips average deviation median 99th % ets 3738.37 K 0.27 μs ±5506.11% 0.23 μs 0.45 μs cachex 1275.05 K 0.78 μs ±223.71% 0.69 μs 1.55 μs mnesia 325.26 K 3.07 μs ±663.83% 2.04 μs 23.89 μs feox_memory 312.12 K 3.20 μs ±73.92% 2.87 μs 10.51 μs feox_persistent 273.69 K 3.65 μs ±103.04% 3.05 μs 20.11 μs dets 127.61 K 7.84 μs ±614.37% 1.98 μs 391.79 μs cubdb 5.50 K 181.97 μs ±18.75% 183.30 μs 272.41 μs *** random_read (parallel: 4) *** Name ips average deviation median 99th % ets 12.04 K 83.08 μs ±66.93% 81.20 μs 142.41 μs cachex 11.59 K 86.26 μs ±139.16% 81.24 μs 160.85 μs mnesia 9.08 K 110.14 μs ±56.27% 90.72 μs 307.71 μs feox_persistent 8.16 K 122.52 μs ±58.35% 115.58 μs 262.52 μs feox_memory 7.60 K 131.62 μs ±72.93% 120.75 μs 296.83 μs dets 3.98 K 251.54 μs ±81.64% 220.48 μs 649.32 μs cubdb 1.47 K 681.04 μs ±41.53% 638.09 μs 1316.96 μs *** write_only (parallel: 4) *** Name ips average deviation median 99th % ets 334.88 K 2.99 μs ±695.11% 2.17 μs 8.31 μs cachex 276.94 K 3.61 μs ±530.33% 2.86 μs 9.62 μs mnesia 59.87 K 16.70 μs ±438.83% 7.94 μs 143.39 μs feox_memory 29.55 K 33.84 μs ±115.64% 23.89 μs 118.62 μs feox_persistent 15.10 K 66.20 μs ±997.97% 28.27 μs 794.28 μs dets 3.21 K 311.26 μs ±686.99% 10.58 μs 8190.16 μs cubdb 0.79 K 1272.06 μs ±63.11% 1054.00 μs 5205.89 μs *** mixed_80_20 (parallel: 4) *** Name ips average deviation median 99th % ets 11.82 K 84.63 μs ±163.88% 79.60 μs 148.14 μs cachex 11.25 K 88.87 μs ±180.58% 81.25 μs 164.85 μs mnesia 9.29 K 107.68 μs ±172.23% 88.91 μs 428.43 μs feox_memory 7.59 K 131.80 μs ±125.35% 117.54 μs 304.62 μs feox_persistent 5.00 K 199.97 μs ±156.60% 164.57 μs 672.27 μs dets 3.30 K 303.10 μs ±103.31% 262.91 μs 865.34 μs cubdb 0.97 K 1030.96 μs ±96.62% 681.24 μs 4900.36 μs *** delete (parallel: 4) *** Name ips average deviation median 99th % ets 1750.73 K 0.57 μs ±3317.33% 0.43 μs 0.93 μs cachex 952.19 K 1.05 μs ±1319.98% 0.82 μs 1.90 μs mnesia 98.58 K 10.14 μs ±716.83% 3.54 μs 80.08 μs feox_persistent 56.30 K 17.76 μs ±298.57% 9.66 μs 74.62 μs feox_memory 40.18 K 24.89 μs ±313.01% 13.30 μs 86.98 μs dets 33.32 K 30.01 μs ±576.52% 8.71 μs 607.42 μs cubdb 3.68 K 271.63 μs ±23.34% 262.56 μs 440.07 μs ```Memory and reduction-count tables are in the full run output; broadly,
feoxdb_ex (both modes) uses less BEAM heap per operation than :ets's
own reported figure (because the value lives on the Rust side, not the
BEAM heap) and far less than CubDB, whose GenServer-mediated access adds
substantial per-call reduction and allocation overhead. :mnesia's
dirty operations and :dets's port-based access both allocate more per
call than :ets as well, though less than CubDB.
Caveats / what this run does not establish
- Single container, not the PRD's two dedicated hosts (
x86_64+ Apple Silicon). feox_persistent, CubDB,:dets, and:mnesiadata files were placed on tmpfs (/dev/shm) specifically to avoid the write-buffer backpressure noted above swamping stderr; a real disk (with or withoutio_uring) will behave differently, likely worse for persistent-mode latency across all of them.:mnesiais exercised only viadirty_read/dirty_write/dirty_deletewith a singledisc_copiesreplica — not a transactional or multi-node comparison, which would show materially different (worse) numbers.- No disabled CPU frequency scaling, unverified
io_uringavailability. - Only 4 of the PRD's 8 workloads are implemented (see
bench/README.md). - Only one value size (64 bytes) was run; the PRD also wants 1KB and 64KB.
- No scheduler-health run (PRD section 9.5) is implemented yet.
Re-run bench/run.exs with a larger BENCH_DATASET_SIZE, the full
BENCH_CONCURRENCY list, and each of the three value sizes on
appropriate hardware before treating any of these numbers as
publishable.