# Data Streaming Research: FlatBuffers for DESI Explorer

This directory aggregates research on using **FlatBuffers** as the wire protocol
between the Rust/axum API (`api/`) and the Odin + raylib GUI (`gui/`). The goal
is to provide implementation-ready knowledge — not to build anything yet.

## Context

DESI Explorer currently serves placeholder JSON from a Rust/axum API and renders
procedurally generated points in an Odin + raylib client. The two halves are
**disconnected**: the API has stubbed `/api/v1/catalogs` and `/api/v1/objects`
endpoints returning JSON; the GUI has hand-written mirror structs in
`gui/src/data.odin` with stub `get_catalogs()` / `get_catalog_objects()`
procedures. Real DESI catalog data (galaxies, quasars, stars with `ra`, `dec`,
`redshift`) will eventually stream server → client.

FlatBuffers is attractive here because:
- **Zero-copy reads**: the GUI can access fields directly from the network buffer
  with no parsing or allocation — critical when rendering tens of thousands of
  points per frame.
- **Schema-versioned**: forward/backward compatibility as the DESI catalog
  schema evolves across releases (edr → dr1 → dr2 → ...).
- **Cross-language**: Rust and Odin can both generate code from the same `.fbs`
  schema.
- **Compact binary**: smaller payloads than JSON over the wire.

## Research Documents

| # | Document | Purpose |
|---|----------|---------|
| 01 | [architecture.md](01-architecture.md) | Current system architecture and where FlatBuffers fits in |
| 02 | [flatbuffers-overview.md](02-flatbuffers-overview.md) | What FlatBuffers is, how the format works internally |
| 03 | [rust-integration.md](03-rust-integration.md) | Rust/axum API integration: crates, build tooling, codegen pipelines |
| 04 | [odin-client-integration.md](04-odin-client-integration.md) | Odin GUI integration: FFI, flatcc, OdinArrow, HTTP/WebSocket clients |
| 05 | [streaming-protocols.md](05-streaming-protocols.md) | WebSocket + HTTP + framing options for streaming FlatBuffers |
| 06 | [schema-design.md](06-schema-design.md) | DESI-specific schema proposal and evolution rules |
| 07 | [alternatives.md](07-alternatives.md) | Cap'n Proto, Protocol Buffers, MessagePack, Apache Arrow comparison |
| 08 | [testing-strategies.md](08-testing-strategies.md) | Unit, integration, cross-language, fuzz, and conformance testing |
| 09 | [pain-points.md](09-pain-points.md) | Cons, gotchas, and pain points to watch out for |
| 10 | [performance-benchmarks.md](10-performance-benchmarks.md) | Published benchmarks (serialize/deserialize/size) and analysis |

## TL;DR Recommendation

**FlatBuffers is a strong fit** for the DESI Explorer streaming use case,
specifically for the *server → client* bulk data path (catalog objects). The
zero-copy read model matches the render loop perfectly: the GUI ingests a binary
blob over WebSocket or HTTP, verifies it once, and reads `ra`/`dec`/`redshift`
directly from the buffer each frame without allocations.

**Key caveats to weigh before committing:**

1. **Odin has no first-party FlatBuffers binding.** You must go through the
   C ABI (via `flatcc` headers + Odin `foreign` blocks) or hand-roll a minimal
   reader. [OdinArrow](https://github.com/TimeLord/OdinArrow) already hand-rolls
   a FlatBuffers encoder/decoder for the Arrow IPC header — proof the pattern is
   viable in pure Odin.
2. **Serialization is more complex than JSON/protobuf.** The builder API builds
   buffers back-to-front (children before parents). This is a server-side cost
   you pay once per batch, not per client — acceptable.
3. **Not self-describing.** Binary buffers are opaque without the schema. The
   API and GUI must share the same `.fbs` file and version discipline. Add file
   identifiers and keep both sides in lockstep via a shared schema checkout.
4. **FlatBuffers is larger on the wire than protobuf** (estimated 20-50% larger
   on small messages due to vtable overhead), but ~30-100x faster on reads. For
   a read-heavy renderer this trade is worth it.
5. **WASM considerations.** The Web GUI build (Emscripten/WASM) uses raylib via
   Odin. If FlatBuffers keep-alive buffers share memory between the Odin side and
   the JS/WebSocket glue, you need to manage Emscripten memory carefully. This
   is the least-researched area of this report.

## Immediate Next Steps (when you're ready to implement)

```mermaid
flowchart TD
    A["Prototype schema<br/>catalog.fbs: Catalog · CatalogObject · ServerMessage union"]
    B["Generate Rust code<br/>flatc --rust → api/build.rs · serve WS via axum"]
    C["Prototype the Odin reader<br/>flatcc FFI · pure-Odin · OdinArrow"]
    D["Static fixture files<br/>flatc --binary → committed .bin"]
    E["Cross-language tests<br/>Rust + Odin read the same fixtures identically"]
    A --> B --> C --> D --> E
```

1. **Prototype schema first.** Write `catalog.fbs` covering `Catalog`, `CatalogObject`,
   and a `ServerMessage` union (handshake / catalog list / chunk of objects / end).
2. **Generate Rust code** via `flatc --rust` in an `api/build.rs` (see
   `03-rust-integration.md`) and serve over WebSocket using axum's `ws` module
   (tower-http CORS + `axum::extract::ws`).
3. **Prototype the Odin reader.** Pick one of three paths (FFI to `flatcc`,
   pure-Odin reader port, or OdinArrow reuse) and read a FlatBuffer produced by
   the Rust side to prove interop. A single `CatalogObject` with `ra`, `dec`,
   `redshift` is enough to validate the entire pipeline.
4. **Static fixture files.** Generate `.fbs` → FlatBuffer binaries once with
   `flatc --binary`, check them into the repo, and write a cross-language test
   that both Rust and Odin read the same fixture identically. This is the
   backbone of your integration test story (see `08-testing-strategies.md`).