updated procedures in data.odin to pass url; added research notes for data-streaming
This commit is contained in:
@@ -0,0 +1,199 @@
|
||||
# 09 — Cons, Gotchas, & Pain Points
|
||||
|
||||
A candid list of everything that will hurt if you adopt FlatBuffers for this
|
||||
project — and how to mitigate each.
|
||||
|
||||
## 1. No first-party Odin binding (the biggest risk)
|
||||
|
||||
- FlatBuffers officially supports many languages; **Odin is not one**. The Odin
|
||||
side requires either (a) C ABI FFI to the third-party **FlatCC** runtime, (b)
|
||||
a **hand-rolled pure-Odin reader** (100-300 lines, plus maintenance forever),
|
||||
or (c) reuse of **OdinArrow's** hand-rolled FlatBuffers decoder.
|
||||
- **Impact**: schema changes ripple into *your* Odin code, not generated code.
|
||||
Update discipline + the conformance suite are mandatory, not optional.
|
||||
- **Mitigation**: commit static `.bin` fixtures and hard-code the conformance
|
||||
tests; keep the schema churn low; consider a tiny code generator if the
|
||||
schema grows fast.
|
||||
|
||||
## 2. Serialization is harder than JSON/protobuf (back-to-front building)
|
||||
|
||||
- Objects are built **children-before-parents**: create all strings/vectors/
|
||||
nested tables first, then the parent that references them. FlatCC's builder
|
||||
is oriented differently (start/end + stack) and also non-obvious.
|
||||
- Error-prone ordering bugs are caught only by round-trip tests.
|
||||
- **Impact**: server-side code is more verbose; onboarding cost.
|
||||
- **Mitigation**: wrap builder logic in small helper functions per message type;
|
||||
comprehensive round-trip tests; the columnar layout keeps touching-code
|
||||
minimal (just `create_vector` × columns).
|
||||
|
||||
## 3. Larger on the wire than protobuf
|
||||
|
||||
- vtables add 4-16 bytes per table; scalar vectors are NOT varint-packed
|
||||
(every f64 is 8 bytes regardless). FlatBuffers is ~20-50% larger than
|
||||
protobuf on comparable payloads; in the Go benchmark, proto 82B vs FB 192B on
|
||||
a small object.
|
||||
- **Impact**: for millions of objects over a constrained link this is real
|
||||
bandwidth. For a LAN/k3s homelab renderer, negligible.
|
||||
- **Mitigation**: columnar vectors eliminate per-row vtable overhead; omit
|
||||
default-valued fields (free savings); compress with gzip/zstd at the
|
||||
transport (WebSocket payloads are compressible — but you lose zero-copy if
|
||||
you decompress into another buffer; keep compressed frames bounded).
|
||||
|
||||
## 4. Not self-describing / schema coupling
|
||||
|
||||
- A FlatBuffer is meaningless without its `.fbs`. Clients (and your future
|
||||
self debugging with `curl`) cannot inspect a buffer without the schema.
|
||||
- Schema must be **shared and version-disciplined** between api/ and gui/ —
|
||||
exactly the drift problem that `data.odin`'s JSON mirrors already suffer.
|
||||
- **Impact**: debugging is harder; schema changes are coordinated releases.
|
||||
- **Mitigation**: file identifiers (`file_identifier "DESI"`) for self-checking;
|
||||
generate JSON from buffers (`flatc --json --schema foo.fbs -- foo.bin`) for
|
||||
debugging/inspection; keep a `schema/` in-repo single source of truth;
|
||||
`flatc --conform` in CI.
|
||||
|
||||
## 5. Immutable buffers / no in-place mutation
|
||||
|
||||
- Once built, you cannot edit a FlatBuffer in place (the old C++ JSON path had
|
||||
reflective mutation; Rust's reflection/resizing is **experimental**). "Fix a
|
||||
field" → rebuild the whole buffer.
|
||||
- For the renderer this is fine (it's read-only). For the **server** it means
|
||||
any per-client customization (filtering, projections) happens *before* build,
|
||||
or you build per-client variants (expensive).
|
||||
- **Mitigation**: build canonical buffers; push client-side filtering to the
|
||||
client (it's zero-copy cheap for the client to skip rows).
|
||||
|
||||
## 6. 2 GiB buffer ceiling
|
||||
|
||||
- Single FlatBuffer cannot exceed 2 GiB (`u32` offsets). A full DESI survey is
|
||||
millions of rows × ~24-40 bytes = gigabytes. You **must batch**.
|
||||
- **Impact**: streaming design becomes mandatory (fine — it's the plan).
|
||||
- **Mitigation**: batches of ≤ 1-4 MB; `batch_seq` to reassemble/order; client
|
||||
holds N buffers, not one gigantic one.
|
||||
|
||||
## 7. Verification is required for untrusted input
|
||||
|
||||
- Without verification, field reads on a malformed buffer → OOB (Rust: panic /
|
||||
UB with `_unchecked`; Odin: guaranteed memory unsafety). Your data comes from
|
||||
**a network** — verify.
|
||||
- Rust's verifier is good but "experimental" historically: configurable
|
||||
`VerifierOptions` (max_depth/max_tables) help against DoS via deep buffers.
|
||||
- **Impact**: an extra O(n) pass per frame. For a 4 MB batch this is sub-ms —
|
||||
acceptable; still, do it **once per batch**, not per accessor.
|
||||
- **Mitigation**: verify on ingest for untrusted streams; for *trusted* replay
|
||||
(local fixtures) you may use `_unchecked` for the benchmark path; add
|
||||
`max_*` caps.
|
||||
|
||||
## 8. Cross-build/toolchain friction
|
||||
|
||||
- flatc/flatcc development video: version pinning matters — generated code must
|
||||
match the runtime crate version. Mismatched `flatc` ↔ `flatbuffers` crate →
|
||||
subtle incompatibilities.
|
||||
- The repo pins Odin version and Rust version; now you pin **flatbuffers too**
|
||||
(schema → generator → runtime all consistent), plus (Path A) compile flatcc
|
||||
under Emscripten for WASM.
|
||||
- **Impact**: build-system surface area grows.
|
||||
- **Mitigation**: commit generated code (skip codegen in CI); pin exact crate/
|
||||
tool versions in `api/Cargo.toml`, gui Makefile, and CI YAML; centralize flatc
|
||||
install (scripts/ like `install_odin.sh` pattern).
|
||||
|
||||
## 9. WASM/Emscripten path is the fuzziest area
|
||||
|
||||
- Zero-copy across the JS↔WASM boundary doesn't exist; JS decodes FlatBuffers
|
||||
(official lib) then hands typed arrays into WASM heap (one memcpy) — you've
|
||||
given up the headline feature for the web build.
|
||||
- Or the Odin WASM code reads frames delivered by JS; but then JS→WASM handoff
|
||||
doubles as a copy anyway (unless you use `WebAssembly.Memory` shared views and
|
||||
careful ownership).
|
||||
- **Impact**: web build behaves *worse* than native (memcpy + JS decode), which
|
||||
is expected but worth stating. Native desktop (`make run`) is unaffected.
|
||||
- **Mitigation**: accept one memcpy for web; keep the native zero-copy path
|
||||
clean; test both.
|
||||
|
||||
## 10. Odin-type ergonomics: optional scalars, enums, slices
|
||||
|
||||
- Odin has no `Option<T>`/`Result` sugar. Optional scalars (`default: null`)
|
||||
surface as "present? + value" pairs you hand-roll: an `is_set bool` flag (or
|
||||
NaN sentinel for f64) alongside `value: f64`.
|
||||
- Enums are fine (Odin `enum` with explicit values) but vector-of-enum reads as
|
||||
`[]byte` with manual cast.
|
||||
- Reading column vectors → `[^]f64`/`[]f64` requires the **buffer outlive the
|
||||
view**: zero-copy slicing borrows the frame buffer. Freeing the frame before
|
||||
the render pass = dangling `[]f64`. Ownership discipline (defer-free) needed.
|
||||
- **Mitigation**: document the "frame buffer must outlive render pass" rule;
|
||||
write copy-free helper procs that take the buffer explicitly; test with
|
||||
over-release patterns.
|
||||
|
||||
## 11. Debugging binary data
|
||||
|
||||
- No `curl | jq`. Binary payloads are opaque without tooling: `flatc --json`
|
||||
(official) and flatcc's JSON printer (per-schema C) are your tools.
|
||||
- IDE display of buffers is limited; hex dumps are the fallback.
|
||||
- **Impact**: slower onboarding for new contributors; harder support/debug
|
||||
sessions.
|
||||
- **Mitigation**: keep the JSON endpoints (health/catalogs) for humans; add a
|
||||
tiny dev CLI (or make target) that dumps a `.bin` to JSON via flatc; log
|
||||
length/hash per frame instead of contents.
|
||||
|
||||
## 12. Ecosystem / library maturity wrinkles
|
||||
|
||||
- Rust flatbuffers crate is labeled **"experimental"** in docs although widely
|
||||
used; API churn across minor versions historically (see docs.rs). Pin
|
||||
precisely.
|
||||
- `flatbuffers-build` uses a **symlink-in-src** hack to surface generated code
|
||||
to build.rs — a smell that can break with Cargo changes. Committing generated
|
||||
code avoids it (recommended here).
|
||||
- No official Odin, no official C# for web-unity-esque paths, no protobuf-style
|
||||
RFC coverage. Small but real gaps.
|
||||
|
||||
## 13. Opinionated schema discipline
|
||||
|
||||
- Teams that skip evolution rules (reordering, deleting, default changes) get
|
||||
silent data corruption — *worse* than an explicit break because reads don't
|
||||
error, they return wrong values.
|
||||
- Rule: new fields only at table end; deprecate (don't delete); don't change
|
||||
defaults; enums only grow; structs are frozen forever. Enforce with
|
||||
`flatc --conform` + review discipline.
|
||||
|
||||
## 14. Odin ecosystem: borrowing from OdinArrow is a dependency decision
|
||||
|
||||
- OdinArrow is community-maintained (TimeLord). If you extract only its
|
||||
FlatBuffers decoder, you own a fork; if you depend on it wholesale, you hold
|
||||
a third-party Odin dependency in `gui/lib/` with its own update cadence.
|
||||
- License + maintenance verification required before adoption (the repo's
|
||||
`add-dep` flow exists for exactly this: submodule into `gui/lib/local/`).
|
||||
|
||||
## 15. Performance expectation management
|
||||
|
||||
- Benchmarks are message-shape-dependent. FlatBuffers' deserialization reads
|
||||
are near-zero, but **serialization is not fastest** (Mechanical Snail: FB
|
||||
serialize 0.9µs vs Cap'n 0.5µs). The distributed-latency study even found
|
||||
protobuf/avro/thrift beat FlatBuffers on end-to-end latency in some
|
||||
microservice topologies because **payload size dominates network time**.
|
||||
- For LAN renderer workloads, FlatBuffers' win is real (read-side) but don't
|
||||
over-promise on shrink: it's bigger than protobuf.
|
||||
- Measure with your own criterion bench before/after (see `07-alternatives.md`
|
||||
suggestion).
|
||||
|
||||
## Pain-point severity ranking
|
||||
|
||||
| # | Pain | Severity | Mitigation exists? |
|
||||
|---|---|---|---|
|
||||
| 1 | No Odin binding (FFI/hand-roll) | High | Yes (flatcc / OdinArrow / conformance) |
|
||||
| 2 | Back-to-front serialization complexity | Medium | Yes (helpers + tests) |
|
||||
| 3 | Bigger wire than proto | Low-Medium | Yes (columnar + omit defaults) |
|
||||
| 4 | Not self-describing | Medium | Yes (fixture files + file id) |
|
||||
| 5 | Immutability | Low | Yes (rebuild per batch) |
|
||||
| 6 | 2GiB ceiling → batching | Low | Yes (batch streaming) |
|
||||
| 7 | Verifier needed for untrusted input | Medium | Yes (verify once per batch) |
|
||||
| 8 | Toolchain/version pinning | Medium | Yes (commit generated, pin versions) |
|
||||
| 9 | WASM zero-copy loss | Low (web is secondary) | Yes (accept 1 memcpy) |
|
||||
| 10 | Odin ergonomics (Option/slices/borrow) | Medium | Yes (patterns + tests) |
|
||||
| 11 | Debugging binary | Low | Yes (flatc --json + keep JSON endpoints) |
|
||||
| 12 | Library maturity wrinkles | Low | Yes (pin, commit generated) |
|
||||
| 13 | Schema discipline | Medium | Yes (flatc --conform in CI) |
|
||||
| 14 | OdinArrow dependency | Low | Yes (license/maintenance review) |
|
||||
| 15 | Performance expectation | Low | Yes (own benchmarks) |
|
||||
|
||||
**Overall**: all pain points have identified mitigations; the two that require
|
||||
active, ongoing investment are **(1) the Odin reader** and **(13) schema
|
||||
discipline**. Budget for them in the implementation plan.
|
||||
Reference in New Issue
Block a user