desi_explorer/resources/ai/research/data-streaming/07-alternatives.md

07 — Alternatives to FlatBuffers

FlatBuffers isn't the only option. This doc compares the realistic alternatives for a Rust API → Odin GUI streaming pipeline. The evaluation lens: cross- language support (Rust + Odin), zero-copy reads, schema evolution, streaming fitness, and ecosystem maturity.

Quick comparison table

Format	Zero-copy reads	Schema	Wire size (vs JSON)	Serialize	Deser/read	Streaming	Rust support	Odin support
JSON (current)	No	No	1.00× (268B)	5.2 µs	8.7 µs	SSE/chunk	serde	core:json
MessagePack	No	No	0.60×	2.1 µs	3.8 µs	chunk	rmp-serde	tgolsson/msgpack-odin (small)
Protocol Buffers	No	Yes (proto)	0.35×	1.8 µs	2.4 µs	gRPC streaming	prost/tonic	none official (C via protobuf-c)
FlatBuffers	Yes	Yes (fbs)	0.47×	0.9 µs	0.03 µs	size-prefix/WS	flatbuffers crate	none official (C via flatcc)
Cap'n Proto	Yes	Yes	0.54×	0.5 µs	0.02 µs	structured RPC	capnp crate	none official (C via capnproto-c)
Apache Arrow IPC	File/stream batch	Yes (FlatBuffers-based)	~0.42× (typed cols)	batch-append	~zero-cost column slices	native stream format	arrow crate (official)	OdinArrow (community)

(Benchmarks: Mechanical Snail wire-format study, Nov 2025, Rust, 1-MB median message; 0.47× etc. are compression ratios; absolute numbers vary by message shape — see 10-performance-benchmarks.md.)

Deeper per-format notes

JSON (status quo)

• Pros: human-readable, debug-friendly, universal, no schema burden, trivially works with curl, streaming via SSE/chunked HTTP.
• Cons: slowest (5-14× slower than binary on read-heavy paths), verbose (2-3× payload size), no type integrity, allocates per-field on parse.
• In this repo: already stubbed. Could stay for the HTTP diagnostics path, but for streaming 100k+ points per frame to a renderer, JSON is the wrong tool.
• Verdict: keep for curl/debug endpoints; not the streaming wire.

MessagePack

• Pros: binary offshoot of JSON — schema-less, faster, 40% smaller. Rust (rmp-serde) is mature. There's a small pure-Odin msgpack impl (tgolsson/ msgpack-odin — timestamps included, no ext types).
• Cons: no schema → no forward/backward compatibility guarantees, no zero-copy reads (you parse into Odin structs per field), no type end-to-end integrity. On reads it's only ~2× faster than JSON — the renderer still pays a per-field decode.
• Verdict: good "fast JSON" but doesn't solve the zero-copy/schema-evolution problems.

Protocol Buffers (protobuf)

• Pros: smallest wire (varint encoding); gRPC for streaming; smallest payloads; excellent Rust (prost/tonic, prost-build); schema-evolution via field numbers; huge ecosystem.
• Cons: no zero-copy reads — deserialization allocates the full object graph. For 100k objects/frame that's the exact cost FlatBuffers eliminates. No first-party Odin; C via protobuf-c is clunky.
• Verdict: ideal for "fat service-to-service RPC" where you decode once and hold objects. Wrong shape for a per-frame high-frequency render workload unless you pre-decode into the renderer's own buffer.

Cap'n Proto

• Pros: true zero-copy (wire layout ≈ memory layout), pointer-based lazy access, built-in RPC, slightly simpler encode than FlatBuffers (no backward construction). Serialize ~0.5 µs, deserialize ~0.2 µs (fastest table entries in the Mechanical Snail study).
• Cons: 8-byte alignment makes it ~20-50% larger than FlatBuffers on the wire; no Robert-native Odin; lower ecosystem/maturity than FlatBuffers; fewer language bindings (no official WASM/JS story as rich).
• Verdict: a genuinely strong competitor, but for this repo the wiresize penalty (bandwidth for millions of points) and weaker Odin/JS ecosystem tilt toward FlatBuffers.

Apache Arrow IPC (columnar, used as transport)

• Pros: the canonical columnar data format; the only alternative with a native Odin implementation (TimeLord/OdinArrow) already shipping a Zero-copy IPC reader (memory-mapped, pyarrow-interopable). Streaming file/ stream formats are purpose-built for bulk numeric data. Rust side gets the official arrow crate.
• Cons: bigger dependency surface (arrow crate is heavy); schema model is Arrow's own (Field/RecordBatch) rather than your own; IPC header uses FlatBuffers internally (Arrow metadata is FlatBuffers), which is a point in FlatBuffers' favor; existing CatalogObject models don't map 1:1.
• Verdict: strongest technical fit for "millions of numeric rows, columnar, zero-copy, streaming". If the DESI dataset is really millions-of-points columnar, Arrow IPC deserves a serious look — but you'd be adopting Arrow's framing + OdinArrow dependency rather than your own schema/format, which reads as more risk for a repo this young.

Flexbuffers / schemaless (bonus)

• FlatBuffers ships Flexbuffers — its own schemaless, self-describing, variable-typed format (kind of a fast MessagePack). Rust+JS+Python etc. supported; Odin via same C ABI story as FlatBuffers.
• Not a fit here: the point of going FlatBuffers is the shared compiled schema keeping Rust/Odin coherent. Flexbuffers gives up exactly that.

Decision matrix mapped to DESI Explorer

Requirement	FlatBuffers	Cap'n Proto	Protobuf	Arrow	MsgPack
Zero-copy per-frame reads (100k+ pts)	✅✅	✅✅	❌	✅✅	❌
Compact wire for many f64 columns	✅ (f64 vectors)	⚠️ (8B-aligned, larger)	✅ (varint packed)	✅✅ (chunked)	⚠️
Schema evolution (edr→dr1→dr2)	✅	✅	✅✅	⚠️ (Arrow schema is rigid)	❌
Rust (API) maturity	✅ (official crate)	✅ (capnp crate)	✅✅ (prost/tonic)	✅✅ (official)	✅
Odin (GUI) availability	⚠️ C-ABI/flatcc or hand-rolled	⚠️ C-ABI or hand-rolled	❌ (protobuf-c barely)	✅ (OdinArrow)	⚠️ (small impl)
WASM/JS story	✅ (official npm)	⚠️	✅	⚠️	✅
Streaming primitives	⚠️ (BYO framing; flatstream-rs exists)	✅ (RPC)	✅ (gRPC)	✅✅ (IPC stream)	⚠️ (chunking only)

Bottom line

• FlatBuffers is the best default for this project: the zero-copy read model matches the render loop, the wire format is compact for numeric vectors, schema evolution fits DESI's release cadence, and the Rust + WASM/JS official story is strong. The Odin gap is real but solvable (C-ABI via flatcc, hand-rolled reader, or OdinArrow patterns).
• Apache Arrow IPC is the strongest alternative if the data is truly "columnar millions-of-rows" and you accept OdinArrow + the Arrow header/ framing dependency. It even uses FlatBuffers internally for its metadata, so it's not an either/or at the format level.
• Protobuf/gRPC wins only if you'd rather have official Rust streaming + smallest payloads and are OK decoding into objects once (not per frame) — the renderer then builds its own draw buffers. For 100k objects/frame the "decode-once-into-draw-buffer" model is actually defensible; measure before over-engineering.

Test-before-commit suggestion

Run a micro-benchmark (criterion) on your actual payload shape (e.g. 10k objects × 3 f64 columns / batch) comparing:

1. serde JSON → parse
2. FlatBuffers columnar batch (this proposal)
3. protobuf (prost) → decode to Vec
4. Arrow IPC record batch (arrow crate) → record batch slices

Numbers will decide the finer trade-offs; the qualitative table above is the strategic direction.

References

• Mechanical Snail, "Wire Formats for High-Volume Service Communication" (2025-11): https://mechanicalsnail.com/posts/wire-formats-rust/
• Youngju, "Binary Serialization Complete Guide" (2025-2026): https://www.youngju.dev/blog/...-binary-serialization-...
• Maltsev & others, "Impact of Serialization Format on Inter-Service Latency" (2024): https://science.lpnu.ua/sites/default/files/journal-paper/2024/dec/...
• Opentraffic/DEEPWIKI Flatbuffers deep dive: https://deepwiki.com/opentraffic/flatbuffers
• Arrow IPC / OdinArrow: https://arrow.apache.org/docs/format/Columnar.html + https://github.com/TimeLord/OdinArrow
• msgpack-odin: https://github.com/tgolsson/msgpack-odin