desi_explorer/resources/ai/research/data-streaming/README.md

Data Streaming Research: FlatBuffers for DESI Explorer

This directory aggregates research on using FlatBuffers as the wire protocol between the Rust/axum API (api/) and the Odin + raylib GUI (gui/). The goal is to provide implementation-ready knowledge — not to build anything yet.

Context

DESI Explorer currently serves placeholder JSON from a Rust/axum API and renders procedurally generated points in an Odin + raylib client. The two halves are disconnected: the API has stubbed /api/v1/catalogs and /api/v1/objects endpoints returning JSON; the GUI has hand-written mirror structs in gui/src/data.odin with stub get_catalogs() / get_catalog_objects() procedures. Real DESI catalog data (galaxies, quasars, stars with ra, dec, redshift) will eventually stream server → client.

FlatBuffers is attractive here because:

• Zero-copy reads: the GUI can access fields directly from the network buffer with no parsing or allocation — critical when rendering tens of thousands of points per frame.
• Schema-versioned: forward/backward compatibility as the DESI catalog schema evolves across releases (edr → dr1 → dr2 → ...).
• Cross-language: Rust and Odin can both generate code from the same .fbs schema.
• Compact binary: smaller payloads than JSON over the wire.

Research Documents

#	Document	Purpose
01	architecture.md	Current system architecture and where FlatBuffers fits in
02	flatbuffers-overview.md	What FlatBuffers is, how the format works internally
03	rust-integration.md	Rust/axum API integration: crates, build tooling, codegen pipelines
04	odin-client-integration.md	Odin GUI integration: FFI, flatcc, OdinArrow, HTTP/WebSocket clients
05	streaming-protocols.md	WebSocket + HTTP + framing options for streaming FlatBuffers
06	schema-design.md	DESI-specific schema proposal and evolution rules
07	alternatives.md	Cap'n Proto, Protocol Buffers, MessagePack, Apache Arrow comparison
08	testing-strategies.md	Unit, integration, cross-language, fuzz, and conformance testing
09	pain-points.md	Cons, gotchas, and pain points to watch out for
10	performance-benchmarks.md	Published benchmarks (serialize/deserialize/size) and analysis

TL;DR Recommendation

FlatBuffers is a strong fit for the DESI Explorer streaming use case, specifically for the server → client bulk data path (catalog objects). The zero-copy read model matches the render loop perfectly: the GUI ingests a binary blob over WebSocket or HTTP, verifies it once, and reads ra/dec/redshift directly from the buffer each frame without allocations.

Key caveats to weigh before committing:

1. Odin has no first-party FlatBuffers binding. You must go through the C ABI (via flatcc headers + Odin foreign blocks) or hand-roll a minimal reader. OdinArrow already hand-rolls a FlatBuffers encoder/decoder for the Arrow IPC header — proof the pattern is viable in pure Odin.
2. Serialization is more complex than JSON/protobuf. The builder API builds buffers back-to-front (children before parents). This is a server-side cost you pay once per batch, not per client — acceptable.
3. Not self-describing. Binary buffers are opaque without the schema. The API and GUI must share the same .fbs file and version discipline. Add file identifiers and keep both sides in lockstep via a shared schema checkout.
4. FlatBuffers is larger on the wire than protobuf (estimated 20-50% larger on small messages due to vtable overhead), but ~30-100x faster on reads. For a read-heavy renderer this trade is worth it.
5. WASM considerations. The Web GUI build (Emscripten/WASM) uses raylib via Odin. If FlatBuffers keep-alive buffers share memory between the Odin side and the JS/WebSocket glue, you need to manage Emscripten memory carefully. This is the least-researched area of this report.

Immediate Next Steps (when you're ready to implement)

1. Prototype schema first. Write catalog.fbs covering Catalog, CatalogObject, and a ServerMessage union (handshake / catalog list / chunk of objects / end).
2. Generate Rust code via flatc --rust in an api/build.rs (see 03-rust-integration.md) and serve over WebSocket using axum's ws module (tower-http CORS + axum::extract::ws).
3. Prototype the Odin reader. Pick one of three paths (FFI to flatcc, pure-Odin reader port, or OdinArrow reuse) and read a FlatBuffer produced by the Rust side to prove interop. A single CatalogObject with ra, dec, redshift is enough to validate the entire pipeline.
4. Static fixture files. Generate .fbs → FlatBuffer binaries once with flatc --binary, check them into the repo, and write a cross-language test that both Rust and Odin read the same fixture identically. This is the backbone of your integration test story (see 08-testing-strategies.md).