updated procedures in data.odin to pass url; added research notes for data-streaming
This commit is contained in:
@@ -0,0 +1,158 @@
|
||||
# 01 — Current Architecture and Where FlatBuffers Fits
|
||||
|
||||
## What exists today
|
||||
|
||||
### API (Rust + axum)
|
||||
|
||||
- **Entrypoint**: `api/src/main.rs` — binds `API_BIND_ADDR` (default `0.0.0.0:8080`),
|
||||
wires `tracing`, serves the axum app with graceful shutdown.
|
||||
- **Routes** (`api/src/routes/`):
|
||||
- `GET /health` → static `{"status":"ok"}`
|
||||
- `GET /api/v1/catalogs` → hardcoded list of 2 catalogs (`edr`, `dr1`)
|
||||
- `GET /api/v1/objects?catalog=<name>&limit=<n>` → **stubbed**, returns `[]`
|
||||
- **Models** (`api/src/models.rs`):
|
||||
|
||||
```rust
|
||||
pub struct Catalog {
|
||||
pub name: String,
|
||||
pub release: String,
|
||||
pub description: &'static str,
|
||||
pub object_count: Option<u64>,
|
||||
}
|
||||
|
||||
pub struct CatalogObject {
|
||||
pub id: String,
|
||||
pub catalog: String,
|
||||
pub object_type: String, // galaxy / quasar / star
|
||||
pub ra: f64, // degrees
|
||||
pub dec: f64, // degrees
|
||||
pub redshift: f64, // dimensionless
|
||||
}
|
||||
```
|
||||
|
||||
- **Serialization**: serde `Serialize` derives + `axum::Json`. `serde_json` is a
|
||||
dev-dependency only (used by integration tests).
|
||||
- **CORS**: `tower-http` is compiled with `cors` + `trace` features, but no CORS
|
||||
layer is currently added to `routes::app()`. This matters for the WASM GUI.
|
||||
|
||||
### GUI (Odin + raylib)
|
||||
|
||||
- **Entrypoint**: `gui/src/main.odin` — `update()` → `draw()` loop, orbital
|
||||
`Camera3D`, 4,000 procedurally generated points via `make_universe()`.
|
||||
- **Placeholder data**: `Galaxy { position: rl.Vector3, color: rl.Color }`
|
||||
generated in a 500-unit-radius sphere. Colors are a distance-based redshift
|
||||
stand-in.
|
||||
- **Data layer**: `gui/src/data.odin` defines hand-written Odin structs that
|
||||
mirror the Rust API JSON contract (including pointer types `^string`, `^u64`
|
||||
to mirror Rust `Option`), plus stubbed procedures:
|
||||
|
||||
```odin
|
||||
get_catalogs :: proc(url: string) -> ([dynamic]Catalog, ^APIError) // nil, nil
|
||||
get_catalog :: proc(url: string, name: string) -> (^Catalog, ^APIError) // nil, nil
|
||||
get_catalog_objects :: proc(url: string, catalog_name: string) // nil, nil
|
||||
```
|
||||
|
||||
- **No HTTP client exists** on the Odin side — no `core:net`, no curl bindings,
|
||||
nothing wired into the render loop. The structs are a *planned* interface.
|
||||
- **Web build**: `gui/www/` — WASM shell (Emscripten) loading `index.wasm` +
|
||||
`odin.js`, no API URL wiring yet.
|
||||
|
||||
## Data flow gap
|
||||
|
||||
```
|
||||
[DESI catalog store] --(future)--> [Rust/axum API] --(nothing today)--> [Odin + raylib GUI]
|
||||
^ ^
|
||||
| serde JSON models | hand-mirrored structs
|
||||
| | (stubs, never used)
|
||||
```
|
||||
|
||||
There is **no live data flow**. The API currently returns JSON placeholders; the
|
||||
GUI renders procedural points. The intended shape (per code comments):
|
||||
- The /api/v1/objects endpoint "will page through the centralized DESI catalog store."
|
||||
- Real survey data replaces the procedural cloud "once ingestion lands."
|
||||
|
||||
## Where FlatBuffers fits
|
||||
|
||||
There are three natural insertion points:
|
||||
|
||||
### 1. Wire format for the object-stream endpoint (primary fit)
|
||||
|
||||
Replace the JSON array response of `GET /api/v1/objects` (or add an
|
||||
`application/flatbuffer` variant / new `/api/v1/stream` endpoint) with a framed
|
||||
stream of FlatBuffer messages:
|
||||
|
||||
- **Rust side**: build FlatBuffers with `FlatBufferBuilder` per batch (e.g.
|
||||
10k-50k objects per batch), send as `application/octet-stream`.
|
||||
- **Odin side**: receive the byte blob, verify once, and access `ra`/`dec`/
|
||||
`redshift` **directly from the network buffer** — no per-field copy. This is
|
||||
the killer use case for the render loop, where every point is projected and
|
||||
colored per frame.
|
||||
|
||||
### 2. Catalog metadata (secondary fit)
|
||||
|
||||
`GET /api/v1/catalogs` returns tiny JSON today. The FlatBuffers win here is
|
||||
marginal (5-10 records), but using the *same* serialization everywhere keeps the
|
||||
codebase uniform — one schema, one reader path. Recommend serving both a JSON
|
||||
response (for curl/debugging) and a FlatBuffers variant (for the client).
|
||||
|
||||
### 3. Bidirectional client → server channel (future fit)
|
||||
|
||||
If the GUI ever sends selection/filter requests (pan region, redshift range,
|
||||
object type filters), a WebSocket carrying FlatBuffers both ways gives
|
||||
consistent framing. The `get_catalog_objects()` stub signature suggests the GUI
|
||||
is expected to query **on demand** as the camera moves — a WebSocket request/
|
||||
response protocol with a cheap binary payload would fit this well.
|
||||
|
||||
## Integration points (concrete anchors)
|
||||
|
||||
| Layer | Today | FlatBuffers insertion point |
|
||||
|---|---|---|
|
||||
| Rust models | `Catalog`, `CatalogObject` (serde) | Generated `catalog_generated.rs` alongside or replacing |
|
||||
| Rust routes | `routes/catalogs.rs` | New route (e.g. `/ws/catalogs`) or response-format negotiation |
|
||||
| Rust Cargo | `serde`, `axum::Json` | `flatbuffers` crate + `flatbuffers-build` for `.fbs → .rs` |
|
||||
| Odin structs | `data.odin` mirrors JSON | Generated/FFI FlatBuffers reader; or hand-rolled reader |
|
||||
| Odin HTTP | none | `core:net` HTTP, WebSocket, or curl FFI |
|
||||
| Build | `make build`, `make test` per project | Schema shared & compiled by both sides |
|
||||
|
||||
## High-level target architecture
|
||||
|
||||
```
|
||||
catalog.fbs (single source of truth, checked into repo)
|
||||
|
|
||||
+--------+---------+
|
||||
| |
|
||||
flatc --rust flatcc --c (or hand-rolled Odin reader)
|
||||
| |
|
||||
api/ (Rust) gui/ (Odin + raylib)
|
||||
| ^
|
||||
| HTTP / WebSocket (framed FlatBuffer binary stream)
|
||||
+------------------+
|
||||
```
|
||||
|
||||
- One schema file. Two generators. Byte-for-byte identical wire format.
|
||||
- A shared conformance test suite (see `08-testing-strategies.md`) ensures both
|
||||
sides stay in sync.
|
||||
|
||||
## Key architectural decisions to make later
|
||||
|
||||
1. **Transport**: HTTP chunked/range requests vs. WebSocket vs. both
|
||||
(see `05-streaming-protocols.md`).
|
||||
2. **Framing**: size-prefixed FlatBuffers (`size_prefixed_root_*`) vs. a custom
|
||||
length prefix (e.g. flatstream's 4-byte LE length). Needed because a TCP/
|
||||
WebSocket stream is byte-ambiguous — you must know where one message ends and
|
||||
the next begins.
|
||||
3. **Batching**: per-object tables vs. packed tables of arrays (columnar-style).
|
||||
For millions of DESI objects, a `table { ra: [double]; dec: [double]; ... }`
|
||||
layout is dramatically smaller and faster than 1 table per object.
|
||||
4. **Schema ownership**: options include a top-level `schema/` directory shared
|
||||
by both `api/` and `gui/`, checked-in generated code, or generated at build
|
||||
time. Checked-in generated code is the simplest CI-safe option for both
|
||||
`cargo build` and `odin build` isolation.
|
||||
|
||||
## What to read next
|
||||
|
||||
- `02-flatbuffers-overview.md` — the format's internals and guarantees
|
||||
- `03-rust-integration.md` — how the Rust side consumes `.fbs`
|
||||
- `04-odin-client-integration.md` — how the Odin side consumes the same format
|
||||
- `05-streaming-protocols.md` — transport and framing choices
|
||||
- `06-schema-design.md` — proposed DESI schema
|
||||
Reference in New Issue
Block a user