sam_oneal/desi_explorer
1
0
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity
Files
8c74efcedb930a5527ffa64af7d1e50e2db0ec98
desi_explorer/resources/ai/research/data-streaming/07-alternatives.md
T

8.3 KiB
Raw Blame History

07 — Alternatives to FlatBuffers

FlatBuffers isn't the only option. This doc compares the realistic alternatives for a Rust API → Odin GUI streaming pipeline. The evaluation lens: cross- language support (Rust + Odin), zero-copy reads, schema evolution, streaming fitness, and ecosystem maturity.

Quick comparison table

Format Zero-copy reads Schema Wire size (vs JSON) Serialize Deser/read Streaming Rust support Odin support
JSON (current) No No 1.00× (268B) 5.2 µs 8.7 µs SSE/chunk serde core:json
MessagePack No No 0.60× 2.1 µs 3.8 µs chunk rmp-serde tgolsson/msgpack-odin (small)
Protocol Buffers No Yes (proto) 0.35× 1.8 µs 2.4 µs gRPC streaming prost/tonic none official (C via protobuf-c)
FlatBuffers Yes Yes (fbs) 0.47× 0.9 µs 0.03 µs size-prefix/WS flatbuffers crate none official (C via flatcc)
Cap'n Proto Yes Yes 0.54× 0.5 µs 0.02 µs structured RPC capnp crate none official (C via capnproto-c)
Apache Arrow IPC File/stream batch Yes (FlatBuffers-based) ~0.42× (typed cols) batch-append ~zero-cost column slices native stream format arrow crate (official) OdinArrow (community)

(Benchmarks: Mechanical Snail wire-format study, Nov 2025, Rust, 1-MB median message; 0.47× etc. are compression ratios; absolute numbers vary by message shape — see 10-performance-benchmarks.md.)

Deeper per-format notes

JSON (status quo)

  • Pros: human-readable, debug-friendly, universal, no schema burden, trivially works with curl, streaming via SSE/chunked HTTP.
  • Cons: slowest (5-14× slower than binary on read-heavy paths), verbose (2-3× payload size), no type integrity, allocates per-field on parse.
  • In this repo: already stubbed. Could stay for the HTTP diagnostics path, but for streaming 100k+ points per frame to a renderer, JSON is the wrong tool.
  • Verdict: keep for curl/debug endpoints; not the streaming wire.

MessagePack

  • Pros: binary offshoot of JSON — schema-less, faster, 40% smaller. Rust (rmp-serde) is mature. There's a small pure-Odin msgpack impl (tgolsson/ msgpack-odin — timestamps included, no ext types).
  • Cons: no schema → no forward/backward compatibility guarantees, no zero-copy reads (you parse into Odin structs per field), no type end-to-end integrity. On reads it's only ~2× faster than JSON — the renderer still pays a per-field decode.
  • Verdict: good "fast JSON" but doesn't solve the zero-copy/schema-evolution problems.

Protocol Buffers (protobuf)

  • Pros: smallest wire (varint encoding); gRPC for streaming; smallest payloads; excellent Rust (prost/tonic, prost-build); schema-evolution via field numbers; huge ecosystem.
  • Cons: no zero-copy reads — deserialization allocates the full object graph. For 100k objects/frame that's the exact cost FlatBuffers eliminates. No first-party Odin; C via protobuf-c is clunky.
  • Verdict: ideal for "fat service-to-service RPC" where you decode once and hold objects. Wrong shape for a per-frame high-frequency render workload unless you pre-decode into the renderer's own buffer.

Cap'n Proto

  • Pros: true zero-copy (wire layout ≈ memory layout), pointer-based lazy access, built-in RPC, slightly simpler encode than FlatBuffers (no backward construction). Serialize ~0.5 µs, deserialize ~0.2 µs (fastest table entries in the Mechanical Snail study).
  • Cons: 8-byte alignment makes it ~20-50% larger than FlatBuffers on the wire; no Robert-native Odin; lower ecosystem/maturity than FlatBuffers; fewer language bindings (no official WASM/JS story as rich).
  • Verdict: a genuinely strong competitor, but for this repo the wiresize penalty (bandwidth for millions of points) and weaker Odin/JS ecosystem tilt toward FlatBuffers.

Apache Arrow IPC (columnar, used as transport)

  • Pros: the canonical columnar data format; the only alternative with a native Odin implementation (TimeLord/OdinArrow) already shipping a Zero-copy IPC reader (memory-mapped, pyarrow-interopable). Streaming file/ stream formats are purpose-built for bulk numeric data. Rust side gets the official arrow crate.
  • Cons: bigger dependency surface (arrow crate is heavy); schema model is Arrow's own (Field/RecordBatch) rather than your own; IPC header uses FlatBuffers internally (Arrow metadata is FlatBuffers), which is a point in FlatBuffers' favor; existing CatalogObject models don't map 1:1.
  • Verdict: strongest technical fit for "millions of numeric rows, columnar, zero-copy, streaming". If the DESI dataset is really millions-of-points columnar, Arrow IPC deserves a serious look — but you'd be adopting Arrow's framing + OdinArrow dependency rather than your own schema/format, which reads as more risk for a repo this young.

Flexbuffers / schemaless (bonus)

  • FlatBuffers ships Flexbuffers — its own schemaless, self-describing, variable-typed format (kind of a fast MessagePack). Rust+JS+Python etc. supported; Odin via same C ABI story as FlatBuffers.
  • Not a fit here: the point of going FlatBuffers is the shared compiled schema keeping Rust/Odin coherent. Flexbuffers gives up exactly that.

Decision matrix mapped to DESI Explorer

Requirement FlatBuffers Cap'n Proto Protobuf Arrow MsgPack
Zero-copy per-frame reads (100k+ pts) ✅✅ ✅✅ ❌ ✅✅ ❌
Compact wire for many f64 columns ✅ (f64 vectors) ⚠️ (8B-aligned, larger) ✅ (varint packed) ✅✅ (chunked) ⚠️
Schema evolution (edr→dr1→dr2) ✅ ✅ ✅✅ ⚠️ (Arrow schema is rigid) ❌
Rust (API) maturity ✅ (official crate) ✅ (capnp crate) ✅✅ (prost/tonic) ✅✅ (official) ✅
Odin (GUI) availability ⚠️ C-ABI/flatcc or hand-rolled ⚠️ C-ABI or hand-rolled ❌ (protobuf-c barely) ✅ (OdinArrow) ⚠️ (small impl)
WASM/JS story ✅ (official npm) ⚠️ ✅ ⚠️ ✅
Streaming primitives ⚠️ (BYO framing; flatstream-rs exists) ✅ (RPC) ✅ (gRPC) ✅✅ (IPC stream) ⚠️ (chunking only)

Bottom line

  • FlatBuffers is the best default for this project: the zero-copy read model matches the render loop, the wire format is compact for numeric vectors, schema evolution fits DESI's release cadence, and the Rust + WASM/JS official story is strong. The Odin gap is real but solvable (C-ABI via flatcc, hand-rolled reader, or OdinArrow patterns).
  • Apache Arrow IPC is the strongest alternative if the data is truly "columnar millions-of-rows" and you accept OdinArrow + the Arrow header/ framing dependency. It even uses FlatBuffers internally for its metadata, so it's not an either/or at the format level.
  • Protobuf/gRPC wins only if you'd rather have official Rust streaming + smallest payloads and are OK decoding into objects once (not per frame) — the renderer then builds its own draw buffers. For 100k objects/frame the "decode-once-into-draw-buffer" model is actually defensible; measure before over-engineering.

Test-before-commit suggestion

Run a micro-benchmark (criterion) on your actual payload shape (e.g. 10k objects × 3 f64 columns / batch) comparing:

  1. serde JSON → parse
  2. FlatBuffers columnar batch (this proposal)
  3. protobuf (prost) → decode to Vec
  4. Arrow IPC record batch (arrow crate) → record batch slices

Numbers will decide the finer trade-offs; the qualitative table above is the strategic direction.

References

Reference in New Issue View Git Blame Copy Permalink