updated procedures in data.odin to pass url; added research notes for data-streaming
This commit is contained in:
@@ -0,0 +1,150 @@
|
||||
# 07 — Alternatives to FlatBuffers
|
||||
|
||||
FlatBuffers isn't the only option. This doc compares the realistic alternatives
|
||||
for a Rust API → Odin GUI streaming pipeline. The evaluation lens: **cross-
|
||||
language support (Rust + Odin), zero-copy reads, schema evolution, streaming
|
||||
fitness, and ecosystem maturity.**
|
||||
|
||||
## Quick comparison table
|
||||
|
||||
| Format | Zero-copy reads | Schema | Wire size (vs JSON) | Serialize | Deser/read | Streaming | Rust support | Odin support |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| JSON (current) | No | No | 1.00× (268B) | 5.2 µs | 8.7 µs | SSE/chunk | serde | core:json |
|
||||
| MessagePack | No | No | 0.60× | 2.1 µs | 3.8 µs | chunk | rmp-serde | tgolsson/msgpack-odin (small) |
|
||||
| Protocol Buffers | No | **Yes** (proto) | **0.35×** | 1.8 µs | 2.4 µs | gRPC streaming | **prost/tonic** | none official (C via protobuf-c) |
|
||||
| **FlatBuffers** | **Yes** | **Yes** (fbs) | 0.47× | 0.9 µs | **0.03 µs** | size-prefix/WS | **flatbuffers crate** | none official (C via flatcc) |
|
||||
| Cap'n Proto | **Yes** | Yes | 0.54× | **0.5 µs** | **0.02 µs** | structured RPC | capnp crate | none official (C via capnproto-c) |
|
||||
| Apache Arrow IPC | File/stream batch | Yes (FlatBuffers-based) | ~0.42× (typed cols) | batch-append | ~zero-cost column slices | **native stream format** | **arrow crate (official)** | **OdinArrow (community)** |
|
||||
|
||||
(Benchmarks: Mechanical Snail wire-format study, Nov 2025, Rust, 1-MB
|
||||
median message; 0.47× etc. are compression ratios; absolute numbers vary by
|
||||
message shape — see `10-performance-benchmarks.md`.)
|
||||
|
||||
## Deeper per-format notes
|
||||
|
||||
### JSON (status quo)
|
||||
|
||||
- **Pros**: human-readable, debug-friendly, universal, no schema burden,
|
||||
trivially works with curl, streaming via SSE/chunked HTTP.
|
||||
- **Cons**: slowest (5-14× slower than binary on read-heavy paths), verbose
|
||||
(2-3× payload size), no type integrity, allocates per-field on parse.
|
||||
- **In this repo**: already stubbed. Could stay for the *HTTP diagnostics* path,
|
||||
but for streaming 100k+ points per frame to a renderer, JSON is the wrong tool.
|
||||
- Verdict: keep for curl/debug endpoints; not the streaming wire.
|
||||
|
||||
### MessagePack
|
||||
|
||||
- **Pros**: binary offshoot of JSON — schema-less, faster, 40% smaller. Rust
|
||||
(rmp-serde) is mature. There's a small pure-Odin msgpack impl (tgolsson/
|
||||
msgpack-odin — timestamps included, no ext types).
|
||||
- **Cons**: **no schema** → no forward/backward compatibility guarantees, no
|
||||
zero-copy reads (you parse into Odin structs per field), no type end-to-end
|
||||
integrity. On reads it's only ~2× faster than JSON — the renderer still pays
|
||||
a per-field decode.
|
||||
- Verdict: good "fast JSON" but doesn't solve the zero-copy/schema-evolution
|
||||
problems.
|
||||
|
||||
### Protocol Buffers (protobuf)
|
||||
|
||||
- **Pros**: smallest wire (varint encoding); gRPC for streaming; **smallest
|
||||
payloads**; excellent Rust (prost/tonic, prost-build); schema-evolution via
|
||||
field numbers; huge ecosystem.
|
||||
- **Cons**: **no zero-copy reads** — deserialization allocates the full object
|
||||
graph. For 100k objects/frame that's the exact cost FlatBuffers eliminates.
|
||||
No first-party Odin; C via protobuf-c is clunky.
|
||||
- Verdict: ideal for "fat service-to-service RPC" where you decode once and
|
||||
hold objects. Wrong shape for a per-frame high-frequency render workload
|
||||
unless you pre-decode into the renderer's own buffer.
|
||||
|
||||
### Cap'n Proto
|
||||
|
||||
- **Pros**: true zero-copy (wire layout ≈ memory layout), pointer-based lazy
|
||||
access, built-in RPC, slightly simpler encode than FlatBuffers (no backward
|
||||
construction). Serialize ~0.5 µs, deserialize ~0.2 µs (fastest table entries
|
||||
in the Mechanical Snail study).
|
||||
- **Cons**: **8-byte alignment makes it ~20-50% larger than FlatBuffers** on
|
||||
the wire; **no Robert-native Odin**; lower ecosystem/maturity than
|
||||
FlatBuffers; fewer language bindings (no official WASM/JS story as rich).
|
||||
- Verdict: a genuinely strong competitor, but for this repo the wiresize penalty
|
||||
(bandwidth for millions of points) and weaker Odin/JS ecosystem tilt toward
|
||||
FlatBuffers.
|
||||
|
||||
### Apache Arrow IPC (columnar, used as transport)
|
||||
|
||||
- **Pros**: **the canonical columnar data format**; the *only* alternative with a
|
||||
**native Odin implementation** (TimeLord/OdinArrow) already shipping a
|
||||
Zero-copy IPC reader (memory-mapped, pyarrow-interopable). Streaming file/
|
||||
stream formats are purpose-built for bulk numeric data. Rust side gets the
|
||||
official `arrow` crate.
|
||||
- **Cons**: bigger dependency surface (arrow crate is heavy); schema model is
|
||||
Arrow's own (Field/RecordBatch) rather than your own; IPC header uses
|
||||
FlatBuffers internally (Arrow metadata *is* FlatBuffers), which is a point in
|
||||
FlatBuffers' favor; existing `CatalogObject` models don't map 1:1.
|
||||
- Verdict: **strongest technical fit for "millions of numeric rows, columnar,
|
||||
zero-copy, streaming"**. If the DESI dataset is *really* millions-of-points
|
||||
columnar, Arrow IPC deserves a serious look — but you'd be adopting Arrow's
|
||||
framing + OdinArrow dependency rather than your own schema/format, which reads
|
||||
as more risk for a repo this young.
|
||||
|
||||
### Flexbuffers / schemaless (bonus)
|
||||
|
||||
- FlatBuffers ships **Flexbuffers** — its own schemaless, self-describing,
|
||||
variable-typed format (kind of a fast MessagePack). Rust+JS+Python etc.
|
||||
supported; Odin via same C ABI story as FlatBuffers.
|
||||
- Not a fit here: the point of going FlatBuffers is the *shared compiled
|
||||
schema* keeping Rust/Odin coherent. Flexbuffers gives up exactly that.
|
||||
|
||||
## Decision matrix mapped to DESI Explorer
|
||||
|
||||
| Requirement | FlatBuffers | Cap'n Proto | Protobuf | Arrow | MsgPack |
|
||||
|---|---|---|---|---|---|
|
||||
| Zero-copy per-frame reads (100k+ pts) | ✅✅ | ✅✅ | ❌ | ✅✅ | ❌ |
|
||||
| Compact wire for many f64 columns | ✅ (f64 vectors) | ⚠️ (8B-aligned, larger) | ✅ (varint packed) | ✅✅ (chunked) | ⚠️ |
|
||||
| Schema evolution (edr→dr1→dr2) | ✅ | ✅ | ✅✅ | ⚠️ (Arrow schema is rigid) | ❌ |
|
||||
| Rust (API) maturity | ✅ (official crate) | ✅ (capnp crate) | ✅✅ (prost/tonic) | ✅✅ (official) | ✅ |
|
||||
| Odin (GUI) availability | ⚠️ C-ABI/flatcc or hand-rolled | ⚠️ C-ABI or hand-rolled | ❌ (protobuf-c barely) | ✅ (OdinArrow) | ⚠️ (small impl) |
|
||||
| WASM/JS story | ✅ (official npm) | ⚠️ | ✅ | ⚠️ | ✅ |
|
||||
| Streaming primitives | ⚠️ (BYO framing; flatstream-rs exists) | ✅ (RPC) | ✅ (gRPC) | ✅✅ (IPC stream) | ⚠️ (chunking only) |
|
||||
|
||||
## Bottom line
|
||||
|
||||
- **FlatBuffers is the best default** for this project: the zero-copy read model
|
||||
matches the render loop, the wire format is compact for numeric vectors, schema
|
||||
evolution fits DESI's release cadence, and the Rust + WASM/JS official story
|
||||
is strong. The Odin gap is real but solvable (C-ABI via flatcc, hand-rolled
|
||||
reader, or OdinArrow patterns).
|
||||
- **Apache Arrow IPC is the strongest alternative** if the data is truly
|
||||
"columnar millions-of-rows" and you accept OdinArrow + the Arrow header/
|
||||
framing dependency. It even uses FlatBuffers internally for its metadata, so
|
||||
it's not an either/or at the format level.
|
||||
- **Protobuf/gRPC** wins only if you'd rather have official Rust streaming +
|
||||
smallest payloads and are OK decoding into objects once (not per frame) — the
|
||||
renderer then builds its own draw buffers. For 100k objects/frame the
|
||||
"decode-once-into-draw-buffer" model is actually defensible; measure before
|
||||
over-engineering.
|
||||
|
||||
## Test-before-commit suggestion
|
||||
|
||||
Run a micro-benchmark (criterion) on your **actual** payload shape (e.g. 10k
|
||||
objects × 3 f64 columns / batch) comparing:
|
||||
1. serde JSON → parse
|
||||
2. FlatBuffers columnar batch (this proposal)
|
||||
3. protobuf (prost) → decode to Vec<f64>
|
||||
4. Arrow IPC record batch (arrow crate) → record batch slices
|
||||
|
||||
Numbers will decide the finer trade-offs; the qualitative table above is the
|
||||
strategic direction.
|
||||
|
||||
## References
|
||||
|
||||
- Mechanical Snail, "Wire Formats for High-Volume Service Communication" (2025-11):
|
||||
https://mechanicalsnail.com/posts/wire-formats-rust/
|
||||
- Youngju, "Binary Serialization Complete Guide" (2025-2026):
|
||||
https://www.youngju.dev/blog/...-binary-serialization-...
|
||||
- Maltsev & others, "Impact of Serialization Format on Inter-Service Latency"
|
||||
(2024): https://science.lpnu.ua/sites/default/files/journal-paper/2024/dec/...
|
||||
- Opentraffic/DEEPWIKI Flatbuffers deep dive:
|
||||
https://deepwiki.com/opentraffic/flatbuffers
|
||||
- Arrow IPC / OdinArrow:
|
||||
https://arrow.apache.org/docs/format/Columnar.html + https://github.com/TimeLord/OdinArrow
|
||||
- msgpack-odin: https://github.com/tgolsson/msgpack-odin
|
||||
Reference in New Issue
Block a user