desi_explorer/resources/ai/research/data-streaming/09-pain-points.md

09 — Cons, Gotchas, & Pain Points

A candid list of everything that will hurt if you adopt FlatBuffers for this project — and how to mitigate each.

1. No first-party Odin binding (the biggest risk)

• FlatBuffers officially supports many languages; Odin is not one. The Odin side requires either (a) C ABI FFI to the third-party FlatCC runtime, (b) a hand-rolled pure-Odin reader (100-300 lines, plus maintenance forever), or (c) reuse of OdinArrow's hand-rolled FlatBuffers decoder.
• Impact: schema changes ripple into your Odin code, not generated code. Update discipline + the conformance suite are mandatory, not optional.
• Mitigation: commit static .bin fixtures and hard-code the conformance tests; keep the schema churn low; consider a tiny code generator if the schema grows fast.

2. Serialization is harder than JSON/protobuf (back-to-front building)

• Objects are built children-before-parents: create all strings/vectors/ nested tables first, then the parent that references them. FlatCC's builder is oriented differently (start/end + stack) and also non-obvious.
• Error-prone ordering bugs are caught only by round-trip tests.
• Impact: server-side code is more verbose; onboarding cost.
• Mitigation: wrap builder logic in small helper functions per message type; comprehensive round-trip tests; the columnar layout keeps touching-code minimal (just create_vector × columns).

3. Larger on the wire than protobuf

• vtables add 4-16 bytes per table; scalar vectors are NOT varint-packed (every f64 is 8 bytes regardless). FlatBuffers is ~20-50% larger than protobuf on comparable payloads; in the Go benchmark, proto 82B vs FB 192B on a small object.
• Impact: for millions of objects over a constrained link this is real bandwidth. For a LAN/k3s homelab renderer, negligible.
• Mitigation: columnar vectors eliminate per-row vtable overhead; omit default-valued fields (free savings); compress with gzip/zstd at the transport (WebSocket payloads are compressible — but you lose zero-copy if you decompress into another buffer; keep compressed frames bounded).

4. Not self-describing / schema coupling

• A FlatBuffer is meaningless without its .fbs. Clients (and your future self debugging with curl) cannot inspect a buffer without the schema.
• Schema must be shared and version-disciplined between api/ and gui/ — exactly the drift problem that data.odin's JSON mirrors already suffer.
• Impact: debugging is harder; schema changes are coordinated releases.
• Mitigation: file identifiers (file_identifier "DESI") for self-checking; generate JSON from buffers (flatc --json --schema foo.fbs -- foo.bin) for debugging/inspection; keep a schema/ in-repo single source of truth; flatc --conform in CI.

5. Immutable buffers / no in-place mutation

• Once built, you cannot edit a FlatBuffer in place (the old C++ JSON path had reflective mutation; Rust's reflection/resizing is experimental). "Fix a field" → rebuild the whole buffer.
• For the renderer this is fine (it's read-only). For the server it means any per-client customization (filtering, projections) happens before build, or you build per-client variants (expensive).
• Mitigation: build canonical buffers; push client-side filtering to the client (it's zero-copy cheap for the client to skip rows).

6. 2 GiB buffer ceiling

• Single FlatBuffer cannot exceed 2 GiB (u32 offsets). A full DESI survey is millions of rows × ~24-40 bytes = gigabytes. You must batch.
• Impact: streaming design becomes mandatory (fine — it's the plan).
• Mitigation: batches of ≤ 1-4 MB; batch_seq to reassemble/order; client holds N buffers, not one gigantic one.

7. Verification is required for untrusted input

• Without verification, field reads on a malformed buffer → OOB (Rust: panic / UB with _unchecked; Odin: guaranteed memory unsafety). Your data comes from a network — verify.
• Rust's verifier is good but "experimental" historically: configurable VerifierOptions (max_depth/max_tables) help against DoS via deep buffers.
• Impact: an extra O(n) pass per frame. For a 4 MB batch this is sub-ms — acceptable; still, do it once per batch, not per accessor.
• Mitigation: verify on ingest for untrusted streams; for trusted replay (local fixtures) you may use _unchecked for the benchmark path; add max_* caps.

8. Cross-build/toolchain friction

• flatc/flatcc development video: version pinning matters — generated code must match the runtime crate version. Mismatched flatc ↔ flatbuffers crate → subtle incompatibilities.
• The repo pins Odin version and Rust version; now you pin flatbuffers too (schema → generator → runtime all consistent), plus (Path A) compile flatcc under Emscripten for WASM.
• Impact: build-system surface area grows.
• Mitigation: commit generated code (skip codegen in CI); pin exact crate/ tool versions in api/Cargo.toml, gui Makefile, and CI YAML; centralize flatc install (scripts/ like install_odin.sh pattern).

9. WASM/Emscripten path is the fuzziest area

• Zero-copy across the JS↔WASM boundary doesn't exist; JS decodes FlatBuffers (official lib) then hands typed arrays into WASM heap (one memcpy) — you've given up the headline feature for the web build.
• Or the Odin WASM code reads frames delivered by JS; but then JS→WASM handoff doubles as a copy anyway (unless you use WebAssembly.Memory shared views and careful ownership).
• Impact: web build behaves worse than native (memcpy + JS decode), which is expected but worth stating. Native desktop (make run) is unaffected.
• Mitigation: accept one memcpy for web; keep the native zero-copy path clean; test both.

10. Odin-type ergonomics: optional scalars, enums, slices

• Odin has no Option<T>/Result sugar. Optional scalars (default: null) surface as "present? + value" pairs you hand-roll: an is_set bool flag (or NaN sentinel for f64) alongside value: f64.
• Enums are fine (Odin enum with explicit values) but vector-of-enum reads as []byte with manual cast.
• Reading column vectors → [^]f64/[]f64 requires the buffer outlive the view: zero-copy slicing borrows the frame buffer. Freeing the frame before the render pass = dangling []f64. Ownership discipline (defer-free) needed.
• Mitigation: document the "frame buffer must outlive render pass" rule; write copy-free helper procs that take the buffer explicitly; test with over-release patterns.

11. Debugging binary data

• No curl | jq. Binary payloads are opaque without tooling: flatc --json (official) and flatcc's JSON printer (per-schema C) are your tools.
• IDE display of buffers is limited; hex dumps are the fallback.
• Impact: slower onboarding for new contributors; harder support/debug sessions.
• Mitigation: keep the JSON endpoints (health/catalogs) for humans; add a tiny dev CLI (or make target) that dumps a .bin to JSON via flatc; log length/hash per frame instead of contents.

12. Ecosystem / library maturity wrinkles

• Rust flatbuffers crate is labeled "experimental" in docs although widely used; API churn across minor versions historically (see docs.rs). Pin precisely.
• flatbuffers-build uses a symlink-in-src hack to surface generated code to build.rs — a smell that can break with Cargo changes. Committing generated code avoids it (recommended here).
• No official Odin, no official C# for web-unity-esque paths, no protobuf-style RFC coverage. Small but real gaps.

13. Opinionated schema discipline

• Teams that skip evolution rules (reordering, deleting, default changes) get silent data corruption — worse than an explicit break because reads don't error, they return wrong values.
• Rule: new fields only at table end; deprecate (don't delete); don't change defaults; enums only grow; structs are frozen forever. Enforce with flatc --conform + review discipline.

14. Odin ecosystem: borrowing from OdinArrow is a dependency decision

• OdinArrow is community-maintained (TimeLord). If you extract only its FlatBuffers decoder, you own a fork; if you depend on it wholesale, you hold a third-party Odin dependency in gui/lib/ with its own update cadence.
• License + maintenance verification required before adoption (the repo's add-dep flow exists for exactly this: submodule into gui/lib/local/).

15. Performance expectation management

• Benchmarks are message-shape-dependent. FlatBuffers' deserialization reads are near-zero, but serialization is not fastest (Mechanical Snail: FB serialize 0.9µs vs Cap'n 0.5µs). The distributed-latency study even found protobuf/avro/thrift beat FlatBuffers on end-to-end latency in some microservice topologies because payload size dominates network time.
• For LAN renderer workloads, FlatBuffers' win is real (read-side) but don't over-promise on shrink: it's bigger than protobuf.
• Measure with your own criterion bench before/after (see 07-alternatives.md suggestion).

Pain-point severity ranking

#	Pain	Severity	Mitigation exists?
1	No Odin binding (FFI/hand-roll)	High	Yes (flatcc / OdinArrow / conformance)
2	Back-to-front serialization complexity	Medium	Yes (helpers + tests)
3	Bigger wire than proto	Low-Medium	Yes (columnar + omit defaults)
4	Not self-describing	Medium	Yes (fixture files + file id)
5	Immutability	Low	Yes (rebuild per batch)
6	2GiB ceiling → batching	Low	Yes (batch streaming)
7	Verifier needed for untrusted input	Medium	Yes (verify once per batch)
8	Toolchain/version pinning	Medium	Yes (commit generated, pin versions)
9	WASM zero-copy loss	Low (web is secondary)	Yes (accept 1 memcpy)
10	Odin ergonomics (Option/slices/borrow)	Medium	Yes (patterns + tests)
11	Debugging binary	Low	Yes (flatc --json + keep JSON endpoints)
12	Library maturity wrinkles	Low	Yes (pin, commit generated)
13	Schema discipline	Medium	Yes (flatc --conform in CI)
14	OdinArrow dependency	Low	Yes (license/maintenance review)
15	Performance expectation	Low	Yes (own benchmarks)

Overall: all pain points have identified mitigations; the two that require active, ongoing investment are (1) the Odin reader and (13) schema discipline. Budget for them in the implementation plan.