updated procedures in data.odin to pass url; added research notes for data-streaming
This commit is contained in:
@@ -0,0 +1,85 @@
|
||||
# Data Streaming Research: FlatBuffers for DESI Explorer
|
||||
|
||||
This directory aggregates research on using **FlatBuffers** as the wire protocol
|
||||
between the Rust/axum API (`api/`) and the Odin + raylib GUI (`gui/`). The goal
|
||||
is to provide implementation-ready knowledge — not to build anything yet.
|
||||
|
||||
## Context
|
||||
|
||||
DESI Explorer currently serves placeholder JSON from a Rust/axum API and renders
|
||||
procedurally generated points in an Odin + raylib client. The two halves are
|
||||
**disconnected**: the API has stubbed `/api/v1/catalogs` and `/api/v1/objects`
|
||||
endpoints returning JSON; the GUI has hand-written mirror structs in
|
||||
`gui/src/data.odin` with stub `get_catalogs()` / `get_catalog_objects()`
|
||||
procedures. Real DESI catalog data (galaxies, quasars, stars with `ra`, `dec`,
|
||||
`redshift`) will eventually stream server → client.
|
||||
|
||||
FlatBuffers is attractive here because:
|
||||
- **Zero-copy reads**: the GUI can access fields directly from the network buffer
|
||||
with no parsing or allocation — critical when rendering tens of thousands of
|
||||
points per frame.
|
||||
- **Schema-versioned**: forward/backward compatibility as the DESI catalog
|
||||
schema evolves across releases (edr → dr1 → dr2 → ...).
|
||||
- **Cross-language**: Rust and Odin can both generate code from the same `.fbs`
|
||||
schema.
|
||||
- **Compact binary**: smaller payloads than JSON over the wire.
|
||||
|
||||
## Research Documents
|
||||
|
||||
| # | Document | Purpose |
|
||||
|---|----------|---------|
|
||||
| 01 | [architecture.md](01-architecture.md) | Current system architecture and where FlatBuffers fits in |
|
||||
| 02 | [flatbuffers-overview.md](02-flatbuffers-overview.md) | What FlatBuffers is, how the format works internally |
|
||||
| 03 | [rust-integration.md](03-rust-integration.md) | Rust/axum API integration: crates, build tooling, codegen pipelines |
|
||||
| 04 | [odin-client-integration.md](04-odin-client-integration.md) | Odin GUI integration: FFI, flatcc, OdinArrow, HTTP/WebSocket clients |
|
||||
| 05 | [streaming-protocols.md](05-streaming-protocols.md) | WebSocket + HTTP + framing options for streaming FlatBuffers |
|
||||
| 06 | [schema-design.md](06-schema-design.md) | DESI-specific schema proposal and evolution rules |
|
||||
| 07 | [alternatives.md](07-alternatives.md) | Cap'n Proto, Protocol Buffers, MessagePack, Apache Arrow comparison |
|
||||
| 08 | [testing-strategies.md](08-testing-strategies.md) | Unit, integration, cross-language, fuzz, and conformance testing |
|
||||
| 09 | [pain-points.md](09-pain-points.md) | Cons, gotchas, and pain points to watch out for |
|
||||
| 10 | [performance-benchmarks.md](10-performance-benchmarks.md) | Published benchmarks (serialize/deserialize/size) and analysis |
|
||||
|
||||
## TL;DR Recommendation
|
||||
|
||||
**FlatBuffers is a strong fit** for the DESI Explorer streaming use case,
|
||||
specifically for the *server → client* bulk data path (catalog objects). The
|
||||
zero-copy read model matches the render loop perfectly: the GUI ingests a binary
|
||||
blob over WebSocket or HTTP, verifies it once, and reads `ra`/`dec`/`redshift`
|
||||
directly from the buffer each frame without allocations.
|
||||
|
||||
**Key caveats to weigh before committing:**
|
||||
|
||||
1. **Odin has no first-party FlatBuffers binding.** You must go through the
|
||||
C ABI (via `flatcc` headers + Odin `foreign` blocks) or hand-roll a minimal
|
||||
reader. [OdinArrow](https://github.com/TimeLord/OdinArrow) already hand-rolls
|
||||
a FlatBuffers encoder/decoder for the Arrow IPC header — proof the pattern is
|
||||
viable in pure Odin.
|
||||
2. **Serialization is more complex than JSON/protobuf.** The builder API builds
|
||||
buffers back-to-front (children before parents). This is a server-side cost
|
||||
you pay once per batch, not per client — acceptable.
|
||||
3. **Not self-describing.** Binary buffers are opaque without the schema. The
|
||||
API and GUI must share the same `.fbs` file and version discipline. Add file
|
||||
identifiers and keep both sides in lockstep via a shared schema checkout.
|
||||
4. **FlatBuffers is larger on the wire than protobuf** (estimated 20-50% larger
|
||||
on small messages due to vtable overhead), but ~30-100x faster on reads. For
|
||||
a read-heavy renderer this trade is worth it.
|
||||
5. **WASM considerations.** The Web GUI build (Emscripten/WASM) uses raylib via
|
||||
Odin. If FlatBuffers keep-alive buffers share memory between the Odin side and
|
||||
the JS/WebSocket glue, you need to manage Emscripten memory carefully. This
|
||||
is the least-researched area of this report.
|
||||
|
||||
## Immediate Next Steps (when you're ready to implement)
|
||||
|
||||
1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`,
|
||||
and a `ServerMessage` union (handshake / catalog list / chunk of objects / end).
|
||||
2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see
|
||||
`03-rust-integration.md`) and serve over WebSocket using axum's `ws` module
|
||||
(tower-http CORS + `axum::extract::ws`).
|
||||
3. **Prototype the Odin reader.** Pick one of three paths (FFI to `flatcc`,
|
||||
pure-Odin reader port, or OdinArrow reuse) and read a FlatBuffer produced by
|
||||
the Rust side to prove interop. A single `CatalogObject` with `ra`, `dec`,
|
||||
`redshift` is enough to validate the entire pipeline.
|
||||
4. **Static fixture files.** Generate `.fbs` → FlatBuffer binaries once with
|
||||
`flatc --binary`, check them into the repo, and write a cross-language test
|
||||
that both Rust and Odin read the same fixture identically. This is the
|
||||
backbone of your integration test story (see `08-testing-strategies.md`).
|
||||
Reference in New Issue
Block a user