updated procedures in data.odin to pass url; added research notes for data-streaming
This commit is contained in:
@@ -0,0 +1,174 @@
|
||||
# 02 — FlatBuffers Overview
|
||||
|
||||
## What it is
|
||||
|
||||
FlatBuffers is a **cross-platform serialization library** created by Google,
|
||||
originally for game development and other performance-critical applications. Its
|
||||
defining feature is **zero-copy data access**: you can read fields directly out
|
||||
of the serialized byte buffer without parsing, unpacking, or allocating objects.
|
||||
|
||||
- **Homepage**: https://flatbuffers.dev
|
||||
- **GitHub**: https://github.com/google/flatbuffers
|
||||
- **License**: Apache 2.0
|
||||
- **Schema language**: `.fbs` files (similar to `.proto` in spirit, but different)
|
||||
- **Current version**: 25.12.19 (Dec 2025 on crates.io — ships with `flatc`)
|
||||
|
||||
## Core guarantees
|
||||
|
||||
| Property | Description |
|
||||
|---|---|
|
||||
| Zero-copy reads | Fields are accessed in-place via offsets — no deserialization step |
|
||||
| Zero-alloc reads | Reading a buffer allocates nothing |
|
||||
| Cross-language | Same schema compiles to C++, C, Rust, Go, Java, C#, JS/TS, Python, Swift, Dart, PHP, Lobster, Lua |
|
||||
| Forward compatible | Code compiled against schema V1 reads data written with schema V2 (ignores unknown fields) |
|
||||
| Backward compatible | Code compiled against schema V2 reads data written with schema V1 (returns defaults for missing fields) |
|
||||
| Little-endian, fixed-width | Deterministic cross-platform serialization (offsets are u32 LE) |
|
||||
| mmap-friendly | Works with memory-mapped files; only touched bytes are paged in |
|
||||
| Thread-safe reads | Reading requires no synchronization; tables implement `Send + Sync` |
|
||||
|
||||
## How the format works internally
|
||||
|
||||
A FlatBuffer is a **contiguous binary blob** of nested objects arranged so the
|
||||
data can be traversed in place like a pointer-based structure. The key concepts:
|
||||
|
||||
### Tables + vtables
|
||||
|
||||
- A **table** is an object whose fields are optional. It carries an inline
|
||||
`soffset_t` (signed 32-bit) pointing at a **vtable**.
|
||||
- The **vtable** maps field slots to (offset, presence) pairs. If a field is
|
||||
absent (written by older code), the accessor returns the schema's default.
|
||||
- Multiple tables with the same layout **share the same vtable** — this is where
|
||||
the zero-copy "no per-object metadata" win comes from.
|
||||
- Tables are the *only* construct that supports schema evolution.
|
||||
|
||||
### Structs
|
||||
|
||||
- A **struct** is a "naked" fixed-layout record with **no vtable** and **no
|
||||
per-field optionality** (every field required).
|
||||
- Layout is deterministic across platforms: scalars aligned to their own size;
|
||||
struct aligned to its largest member.
|
||||
- Smaller and faster than tables, but **cannot evolve** — fields can't be added
|
||||
or removed.
|
||||
- Perfect for things like `Vec3`, `Color`, or a coordinate record that will
|
||||
never change. Example: DESI's `ra`/`dec`/`redshift` triplet could be a struct,
|
||||
or (for bulk data) arrays of raw doubles.
|
||||
|
||||
### Vectors
|
||||
|
||||
- Native vector type (unlike protobuf's `repeated`). Length-prefixed, contiguous,
|
||||
and for scalars/structs can be accessed via `safe_slice()` returning a
|
||||
`&[T]` view (on little-endian machines) — **zero-copy slice access**.
|
||||
- This is the single most important feature for DESI render workloads:
|
||||
`table { ra: [double]; dec: [double]; z: [double] }` gives you three
|
||||
`&[f64]` slices straight from the wire buffer.
|
||||
|
||||
### Unions
|
||||
|
||||
- A `union` is a type-discriminated tagged field (a `Type` enum + value).
|
||||
- Used for things like `ServerMessage` that can carry different payloads
|
||||
(catalog list, object chunk, error).
|
||||
|
||||
### File identifiers + size prefixes
|
||||
|
||||
- **File identifier**: up to 4 ASCII chars stored at offset 4, enabled in the
|
||||
schema with `file_identifier "DESI"`. Lets a reader check "is this buffer a
|
||||
DESI buffer" before touching it.
|
||||
- **Size prefix**: a `u32` LE length written at the start of the buffer,
|
||||
enabling stream framing (`size_prefixed_root_*`). This is the mechanism you
|
||||
use to frame a stream of FlatBuffer messages.
|
||||
|
||||
## Schema evolution rules (summary)
|
||||
|
||||
These are the "thou shalt" rules for keeping buffers compatible:
|
||||
|
||||
1. Add new fields **only at the end** of a `table`. (Or use explicit `id`
|
||||
attributes on every field to allow reordering.)
|
||||
2. Never remove a field — mark it `(deprecated)` instead.
|
||||
3. Never change a field's type unless the width is identical *and* there are no
|
||||
negative-value edge cases.
|
||||
4. Never change a field's `default` after shipping (the default is baked into
|
||||
generated code, not the buffer).
|
||||
5. Enums: only ever add values, never remove. Handle unknown values defensively.
|
||||
6. Structs: **cannot** gain or lose fields. Ever.
|
||||
7. `required` on a non-scalar field: the verifier fails if it's absent. Adding
|
||||
or removing `required` is a compatibility break.
|
||||
8. Renaming fields/tables is safe for binary compat (buffers address by id, not
|
||||
name) but breaks code that uses the old names.
|
||||
9. Run `flatc --conform schema_v1.fbs schema_v2.fbs` to mechanically check that
|
||||
`schema_v2` properly evolves from `schema_v1`.
|
||||
|
||||
> A subtle trap from the official docs: **scalar fields equal to their default
|
||||
> are NOT written to the buffer**. Non-presence is indistinguishable from "wrote
|
||||
> the default". If presence matters (e.g. `object_count: Option<u64>` where
|
||||
> `Some(0)` is meaningful) use **optional scalars** (`default: null`) or wrap the
|
||||
> scalar in a struct.
|
||||
|
||||
## Reading a buffer (conceptual)
|
||||
|
||||
```
|
||||
bytes: &[u8]
|
||||
|
||||
┌─────────────────────────────┐
|
||||
│ uoffset (root table offset) │
|
||||
│ file_identifier (optional) │
|
||||
│ ... tables, vtables, data ...│
|
||||
└─────────────────────────────┘
|
||||
|
||||
root = follow(bytes) // jump to root table via uoffset
|
||||
vtable = root - root.vtable_off // locate vtable for this table
|
||||
field_ra = vtable.slot_ra != 0 // present?
|
||||
if present: ra = read_f64(bytes, root + slot_ra)
|
||||
```
|
||||
|
||||
There is **no parsing loop**. Each accessor is a few offset dereferences and a
|
||||
read. The Rust runtime exposes this via a `Follow` trait that compiles to near-zero
|
||||
code after optimization.
|
||||
|
||||
## Verification (security)
|
||||
|
||||
- The **verifier** walks a buffer and checks offsets, alignment, required fields,
|
||||
and structural sanity *before* you access it. Protects against malicious or
|
||||
corrupted buffers and out-of-bounds reads.
|
||||
- Rust: `flatbuffers::root::<T>(&opts, bytes)` runs the verifier (default
|
||||
options); `root_unchecked::<T>` skips it for trusted data.
|
||||
- Available since the 25.x era in Rust; the C port (flatcc) has long had full
|
||||
verifier support; vitally, Rust verification is now mature, so the "Rust has
|
||||
no verifier" caveat from old docs is outdated.
|
||||
- For untrusted network data (which is exactly your WebSocket case), **always
|
||||
verify** before zero-copy access.
|
||||
- Verifier limits (`VerifierOptions`) can cap `max_tables`, `max_depth`, etc. to
|
||||
defend against resource-exhaustion attacks.
|
||||
|
||||
## Buffer size limits
|
||||
|
||||
- FlatBuffers uses `u32` offsets internally; a single buffer **cannot exceed
|
||||
2 GiB**. `FlatBufferBuilder` enforces `FLATBUFFERS_MAX_BUFFER_SIZE`.
|
||||
- For DESI catalogs with millions of objects, this means **batch** (chunk) the
|
||||
data — e.g. 100k objects per buffer — rather than one giant buffer. That
|
||||
aligns naturally with streaming/pagination anyway.
|
||||
|
||||
## Why it fits DESI Explorer
|
||||
|
||||
- **Render-loop reads**: 100k Galaxy points → drop the network bytes into a
|
||||
buffer, call `verify()`, then per frame iterate `ra[i]`, `dec[i], z[i]` as raw
|
||||
`f64` slices. Zero allocations, zero GC pressure, cache-friendly contiguous
|
||||
arrays.
|
||||
- **Schema versioning across DESI releases**: `edr` data vs `dr1` vs `dr2`
|
||||
differ in coverage/targets. Add `object_id` in release 2 at the end of the
|
||||
table; release-1 clients still read old buffers fine.
|
||||
- **Cross-language integrity**: the same `.fbs` compiles to Rust (API) and C/Odin
|
||||
(GUI) with byte-identical wire format — no hand-maintained JSON contract drift,
|
||||
which is exactly the `data.odin` mirror-struct problem today.
|
||||
- **Single obvious streaming primitives**: size-prefixed buffers make framing
|
||||
trivial; vectors make bulk data compact.
|
||||
|
||||
## Official resources
|
||||
|
||||
- Tutorial: https://flatbuffers.dev/tutorial/
|
||||
- Schema guide: https://flatbuffers.dev/schema/
|
||||
- Evolution rules: https://flatbuffers.dev/evolution/
|
||||
- Rust usage: https://flatbuffers.dev/languages/rust/
|
||||
- C (flatcc) usage: https://flatbuffers.dev/languages/c/
|
||||
- White paper: https://flatbuffers.dev/white_paper/
|
||||
- API docs (Rust): https://docs.rs/flatbuffers
|
||||
- crates.io: https://crates.io/crates/flatbuffers
|
||||
Reference in New Issue
Block a user