updated procedures in data.odin to pass url; added research notes for data-streaming

2026-09-06 13:52:39 -06:00
parent ebaf956cdf
commit 52a0d5c002
12 changed files with 2165 additions and 3 deletions
@@ -0,0 +1,267 @@
+# 03 — Rust / API Integration
+
+## Crate landscape
+
+### `flatbuffers` (runtime, official)
+
+- **crate**: `flatbuffers` — https://crates.io/crates/flatbuffers
+- **Version**: 25.12.19 (Dec 2025). Note: project moved to CalVer in 2025
+  (24.12, 25.1, 25.2, 25.9, 25.12.19). The old `2.1.x` line is legacy.
+- **Support level**: labeled "experimental" historically, but is the official
+  Google runtime; used in production at Google and elsewhere. API stable enough
+  for real usage.
+- **Features**: `serde` (optional), `std` (default), also supports `no_std`.
+- **Dependencies**: `bitflags` only.
+- **Downloads**: ~17M/month, 174 dependents — well maintained.
+
+### `flatbuffers-build` (build-time codegen, community)
+
+- **crate**: `flatbuffers-build` — wraps `flatc` so `.fbs → .rs` happens in
+  `build.rs` instead of manually. Requires `flatc` on PATH (or `FLATC_PATH`).
+- **Caveat**: it writes generated code into a **symlinked directory under
+  `src/`** (`set_symlink_directory("src/generated")`) because Cargo can't include
+  `OUT_DIR` output as a module. This is a known hack; works but slightly ugly.
+- **Alternative**: the `flatc` crate (`https://crates.io/crates/flatc`) vendors
+  a prebuilt `flatc` binary as a dependency.
+
+### `flatc` (vendored binary, community)
+
+- Crate `flatc 0.2.2+23.5.26` bundles the `flatc` executable. Let's you invoke
+  `flatc` from `build.rs` without requiring it system-wide.
+
+### `flatstream` (framing layer, community — very relevant)
+
+- **crate**: `flatstream` — https://github.com/dallasmarlow/flatstream-rs
+- Adds a **framing layer** for streams of FlatBuffers: each frame is
+  `[4-byte LE length][optional checksum][payload]`.
+- Zero-copy read via `StreamReader::process_all(&[u8])`.
+- Originated as a high-frequency telemetry capture format. Composability:
+  bounded payload caps, checksum via xxhash64/crc32, observers, validators.
+- **Sync IO only** (std `Read`/`Write`). Not async — you'd pair it with
+  `tokio::task::spawn_blocking` for network use, or just write your own framing
+  (it's ~20 lines).
+- Note: the crate is young (2025) and small. Treat as a reference pattern, not
+  a hard dependency. A size-prefixed FlatBuffer
+  (`builder.finish_size_prefixed(root, Some("ID"))`) already gives you framing
+  for free if you only need one message type per stream.
+
+### Choice matrix
+
+| Need | Crate | Notes |
+|---|---|---|
+| Runtime read/write | `flatbuffers` | Required regardless |
+| `.fbs → .rs` manually | `flatc` binary | Simple, deterministic, scriptable |
+| `.fbs → .rs` in build.rs | `flatbuffers-build` | Needs flatc on PATH; symlink hack |
+| Vendored flatc binary | `flatc` crate | Avoids system install |
+| Stream framing | `flatstream` (or hand-rolled) | 4-byte LE length prefix pattern |
+| Testing support | `flatbuffers` verification + `cargo test`, `proptest`, `quickcheck` | See testing doc |
+
+## Codegen options for the API
+
+### Option A — Commit generated code (recommended starting point)
+
+```bash
+flatc --rust -o api/src/generated catalog.fbs
+```
+
+Then `mod catalog_generated;` in `lib.rs`. Pros: dead simple, zero build
+scripts, works with `cargo test` out of the box, diffable in code review. Cons:
+regenerate manually on schema change, risk of stale generated code.
+
+### Option B — build.rs codegen
+
+```rust
+// api/build.rs
+fn main() {
+    let schemas = ["../schema/catalog.fbs"];
+    flatbuffers_build::BuilderOptions::new_with_files(&schemas)
+        .set_symlink_directory("src/generated")
+        .compile()
+        .expect("flatbuffer compilation failed");
+}
+```
+
+Pros: automatic regeneration. Cons: symlink hack, `flatc` must be installed
+(CI needs it too), `cargo::rerun-if-changed` wiring is manual-ish. For a repo
+that already pins toolchains in CI, adding `flatc` to `.gitea/workflows/ci.yml`
+is mechanical.
+
+### Option C — shared schema + make target
+
+The project's build is `make`-based and mono-repo style. A root target like
+
+```make
+schema-gen:
+	flatc --rust -o api/src/generated ../schema/catalog.fbs
+	flatcc --c -o gui/src/generated ../schema/catalog.fbs
+```
+
+run before `make build`/`make test` keeps both sides in sync from **one
+schema file**. This fits the existing "root Makefile delegates" pattern well.
+
+## Recipe: serving FlatBuffers from axum
+
+### Dependencies
+
+```toml
+[dependencies]
+axum = { version = "0.8", features = ["ws"] }   # ws feature for WebSocket
+flatbuffers = "25.12.19"                        # runtime
+# for build.rs codegen (if you choose Option B):
+[build-dependencies]
+flatbuffers-build = "0.2"   # or use `flatc` binary option
+```
+
+### Endpoint that returns a FlatBuffer body (content-negotiated)
+
+```rust
+use axum::{body::Bytes, extract::Query, http::StatusCode, response::{IntoResponse, Response}};
+use flatbuffers::{FlatBufferBuilder, WIPOffset};
+
+mod catalog_generated;
+use catalog_generated::desi::{CatalogObject, CatalogObjectArgs, ObjectBatch, ObjectBatchArgs,
+                               finish_object_batch_buffer, root_as_object_batch};
+
+async fn objects(query: Query<ObjectQuery>) -> Response {
+    let rows = fetch_desi_objects(&query.catalog, query.limit.clamp(1, 100_000)).await;
+
+    let mut fbb = FlatBufferBuilder::with_capacity(1 << 20);
+    let ra = fbb.create_vector(&rows.ra);
+    let dec = fbb.create_vector(&rows.dec);
+    let z = fbb.create_vector(&rows.z);
+    let ids = fbb.create_vector(&rows.id_offsets);
+
+    let batch = ObjectBatch::create(&mut fbb, &ObjectBatchArgs {
+        catalog: Some(fbb.create_string(&rows.catalog)),
+        n: rows.len as u64,
+        ra: Some(ra),
+        dec: Some(dec),
+        redshift: Some(z),
+        ids: Some(ids),
+        ..Default::default()
+    });
+    finish_object_batch_buffer(&mut fbb, batch);
+
+    ([(header::CONTENT_TYPE, "application/octet-stream")], Bytes::copy_from_slice(fbb.finished_data()))
+        .into_response()
+}
+```
+
+### WebSocket endpoint
+
+```rust
+use axum::{
+    extract::ws::{Message, WebSocket, WebSocketUpgrade},
+    response::Response,
+    routing::any,
+};
+use futures_util::{stream::StreamExt, sink::SinkExt};
+
+async fn ws_handler(ws: WebSocketUpgrade) -> Response {
+    ws.on_upgrade(handle_socket)
+}
+
+async fn handle_socket(mut socket: WebSocket) {
+    // On connect, stream catalog batches as Binary(Message::Binary(...)) frames:
+    // each frame's payload is a size-prefixed or length-prefixed FlatBuffer.
+    while let Some(Ok(msg)) = socket.recv().await {
+        match msg {
+            Message::Text(query) => {
+                // Client asked for a catalog region; stream batch FlatBuffers back.
+                let rows = fetch_region(&query).await;
+                for chunk in rows.chunks(50_000) {
+                    let buf = build_object_batch_flatbuffer(chunk);
+                    socket.send(Message::Binary(buf.into())).await.unwrap();
+                }
+            }
+            Message::Close(_) => break,
+            _ => {}
+        }
+    }
+}
+```
+
+Notes:
+- WebSocket **binary messages** (`Message::Binary`) are exactly-typed `Vec<u8>`/
+  `Bytes` — ideal for FlatBuffer payloads with no extra encoding.
+- Use `WebSocketUpgrade.max_message_size(...)` to cap client message size
+  (default 64 MB).
+- For a single server pushing to many clients, use `tokio::sync::broadcast` and
+  split the socket (`socket.split()` → `SinkExt` for sending, `StreamExt` for
+  receiving) — see `05-streaming-protocols.md`.
+- If the Odin client is WASM (browser), CORS matters: add `tower-http`'s
+  `CorsLayer` since the WASM origin differs from the API origin. WebSockets
+  aren't CORS-restricted the same way HTTP is, but the HTTP upgrade handshake
+  still goes through the middleware stack.
+
+## Building buffers efficiently (Rust specifics)
+
+- `FlatBufferBuilder::with_capacity(n)` pre-allocates; `reset()` reuses the
+  buffer across messages. In a loop streaming batches, create one builder, reuse
+  it — avoid repeated reallocation.
+- Builder methods have `try_*` counterparts (`try_create_string`,
+  `try_push`, `try_finish`) that return `Result` instead of panicking on
+  out-of-capacity — useful on the server under memory pressure.
+- `with_internal_capacity(size, field_locs, vtables, strings)` preallocates all
+  internal data structures — removes allocation-related latency spikes in a
+  latency-sensitive server path. Combined with `reset()`, a long-lived stream
+  handler does **zero allocations** after startup.
+- For bulk numeric data, prefer **vectors of scalars** (columnar layout) over
+  `create_vector` of tables. `create_vector` on `&[f64]` is a fast path; on
+  little-endian it back-fills with scalar offsets, enabling `safe_slice`
+  reads. Result: one contiguous `&[f64]` slice per column on the client.
+- Building is **back-to-front**: children (strings, vectors, nested tables)
+  must be created before the parent table referencing them. The `*Args` struct
+  pattern (`ObjectBatchArgs { .. }`) makes this manageable, but be deliberate
+  about creation order.
+- `builder.finished_data()` returns `&[u8]` borrowing the builder — copy to an
+  owned `Bytes` for the response, then `reset()`.
+
+## Serialization on the server (axes to optimize)
+
+- **Batch size tradeoff**: each batch carries one vtable + one table header;
+  amortize over more rows. Choose batch size so the buffer is well under the
+  2 GiB limit and matches your streaming cadence (e.g. 50k objects ≈ ~3 MB).
+- **CPU cost**: FlatBuffers serialization is fast but not free (it's the
+  read-side where FlatBuffers shines). Measure with criterion; if serialization
+  becomes the bottleneck, move builder work to a dedicated worker thread /
+  `spawn_blocking` and hand finished batches to the WebSocket sender via a
+  channel.
+- **Vtables**: tables with identical field layouts share vtables automatically —
+  batching objects with the same schema shape is free (they all share).
+
+## Integration with existing serde models
+
+You don't have to delete serde. Pragmatic path:
+
+- Keep `CatalogObject` (serde) for JSON endpoints / debugging / curl-ability.
+- Add a **separate** FlatBuffers representation for the binary path, generated
+  from the shared `.fbs`.
+- Add a `From<CatalogObject> for ObjectBatchArgs`-style conversion, or better:
+  build FlatBuffers **directly from the raw DESI rows** (skip the serde struct
+  entirely, avoiding a serialize-then-rebuild pass). The DESI store should hand
+  you vectors of `f64`; feed those straight into `create_vector`.
+
+This preserves today's test shape (`api/tests/health.rs` parses JSON) and adds
+the binary path without churn.
+
+## CI / build concerns for this repo
+
+- The Gitea workflow (`ci.yml`) runs `cargo test` + clippy. If you switch to
+  build.rs codegen, CI needs `flatc` installed (or use the vendored `flatc`
+  crate). If you commit generated code, CI needs nothing extra — one more reason
+  Option A (commit generated code) is the lowest-friction start.
+- Pin the **same schema file** for both API and GUI. If the schema lives in a
+  shared `schema/` directory and both sides' Makefiles depend on it, `make test`
+  stays the single verification step: Rust tests + Odin tests + Go tests each
+  validate their own pipeline against the shared contract.
+- Mind the repo convention: `make -C api check` is the passthrough for
+  `cargo test` + clippy. Keep the FlatBuffers stack incrementally addable here.
+
+## References
+
+- https://flatbuffers.dev/languages/rust/ — official Rust usage doc
+- https://docs.rs/flatbuffers — runtime API reference
+- https://github.com/frol/flatc-rust — flatc-as-build-API (alternative)
+- https://github.com/dallasmarlow/flatstream-rs — framing reference
+- https://crates.io/crates/flatbuffers-build — codegen crate