desi_explorer/resources/ai/research/data-streaming/02-flatbuffers-overview.md

02 — FlatBuffers Overview

What it is

FlatBuffers is a cross-platform serialization library created by Google, originally for game development and other performance-critical applications. Its defining feature is zero-copy data access: you can read fields directly out of the serialized byte buffer without parsing, unpacking, or allocating objects.

• Homepage: https://flatbuffers.dev
• GitHub: https://github.com/google/flatbuffers
• License: Apache 2.0
• Schema language: .fbs files (similar to .proto in spirit, but different)
• Current version: 25.12.19 (Dec 2025 on crates.io — ships with flatc)

Core guarantees

Property	Description
Zero-copy reads	Fields are accessed in-place via offsets — no deserialization step
Zero-alloc reads	Reading a buffer allocates nothing
Cross-language	Same schema compiles to C++, C, Rust, Go, Java, C#, JS/TS, Python, Swift, Dart, PHP, Lobster, Lua
Forward compatible	Code compiled against schema V1 reads data written with schema V2 (ignores unknown fields)
Backward compatible	Code compiled against schema V2 reads data written with schema V1 (returns defaults for missing fields)
Little-endian, fixed-width	Deterministic cross-platform serialization (offsets are u32 LE)
mmap-friendly	Works with memory-mapped files; only touched bytes are paged in
Thread-safe reads	Reading requires no synchronization; tables implement Send + Sync

How the format works internally

A FlatBuffer is a contiguous binary blob of nested objects arranged so the data can be traversed in place like a pointer-based structure. The key concepts:

Tables + vtables

• A table is an object whose fields are optional. It carries an inline soffset_t (signed 32-bit) pointing at a vtable.
• The vtable maps field slots to (offset, presence) pairs. If a field is absent (written by older code), the accessor returns the schema's default.
• Multiple tables with the same layout share the same vtable — this is where the zero-copy "no per-object metadata" win comes from.
• Tables are the only construct that supports schema evolution.

Structs

• A struct is a "naked" fixed-layout record with no vtable and no per-field optionality (every field required).
• Layout is deterministic across platforms: scalars aligned to their own size; struct aligned to its largest member.
• Smaller and faster than tables, but cannot evolve — fields can't be added or removed.
• Perfect for things like Vec3, Color, or a coordinate record that will never change. Example: DESI's ra/dec/redshift triplet could be a struct, or (for bulk data) arrays of raw doubles.

Vectors

• Native vector type (unlike protobuf's repeated). Length-prefixed, contiguous, and for scalars/structs can be accessed via safe_slice() returning a &[T] view (on little-endian machines) — zero-copy slice access.
• This is the single most important feature for DESI render workloads: table { ra: [double]; dec: [double]; z: [double] } gives you three &[f64] slices straight from the wire buffer.

Unions

• A union is a type-discriminated tagged field (a Type enum + value).
• Used for things like ServerMessage that can carry different payloads (catalog list, object chunk, error).

File identifiers + size prefixes

• File identifier: up to 4 ASCII chars stored at offset 4, enabled in the schema with file_identifier "DESI". Lets a reader check "is this buffer a DESI buffer" before touching it.
• Size prefix: a u32 LE length written at the start of the buffer, enabling stream framing (size_prefixed_root_*). This is the mechanism you use to frame a stream of FlatBuffer messages.

Schema evolution rules (summary)

These are the "thou shalt" rules for keeping buffers compatible:

1. Add new fields only at the end of a table. (Or use explicit id attributes on every field to allow reordering.)
2. Never remove a field — mark it (deprecated) instead.
3. Never change a field's type unless the width is identical and there are no negative-value edge cases.
4. Never change a field's default after shipping (the default is baked into generated code, not the buffer).
5. Enums: only ever add values, never remove. Handle unknown values defensively.
6. Structs: cannot gain or lose fields. Ever.
7. required on a non-scalar field: the verifier fails if it's absent. Adding or removing required is a compatibility break.
8. Renaming fields/tables is safe for binary compat (buffers address by id, not name) but breaks code that uses the old names.
9. Run flatc --conform schema_v1.fbs schema_v2.fbs to mechanically check that schema_v2 properly evolves from schema_v1.

A subtle trap from the official docs: scalar fields equal to their default are NOT written to the buffer. Non-presence is indistinguishable from "wrote the default". If presence matters (e.g. object_count: Option<u64> where Some(0) is meaningful) use optional scalars (default: null) or wrap the scalar in a struct.

Reading a buffer (conceptual)

bytes: &[u8]

  ┌─────────────────────────────┐
  │ uoffset (root table offset)  │
  │ file_identifier (optional)   │
  │ ... tables, vtables, data ...│
  └─────────────────────────────┘

root = follow(bytes)              // jump to root table via uoffset
vtable = root - root.vtable_off  // locate vtable for this table
field_ra  = vtable.slot_ra != 0  // present?
if present: ra = read_f64(bytes, root + slot_ra)

There is no parsing loop. Each accessor is a few offset dereferences and a read. The Rust runtime exposes this via a Follow trait that compiles to near-zero code after optimization.

Verification (security)

• The verifier walks a buffer and checks offsets, alignment, required fields, and structural sanity before you access it. Protects against malicious or corrupted buffers and out-of-bounds reads.
• Rust: flatbuffers::root::<T>(&opts, bytes) runs the verifier (default options); root_unchecked::<T> skips it for trusted data.
• Available since the 25.x era in Rust; the C port (flatcc) has long had full verifier support; vitally, Rust verification is now mature, so the "Rust has no verifier" caveat from old docs is outdated.
• For untrusted network data (which is exactly your WebSocket case), always verify before zero-copy access.
• Verifier limits (VerifierOptions) can cap max_tables, max_depth, etc. to defend against resource-exhaustion attacks.

Buffer size limits

• FlatBuffers uses u32 offsets internally; a single buffer cannot exceed 2 GiB. FlatBufferBuilder enforces FLATBUFFERS_MAX_BUFFER_SIZE.
• For DESI catalogs with millions of objects, this means batch (chunk) the data — e.g. 100k objects per buffer — rather than one giant buffer. That aligns naturally with streaming/pagination anyway.

Why it fits DESI Explorer

• Render-loop reads: 100k Galaxy points → drop the network bytes into a buffer, call verify(), then per frame iterate ra[i], dec[i], z[i] as raw f64 slices. Zero allocations, zero GC pressure, cache-friendly contiguous arrays.
• Schema versioning across DESI releases: edr data vs dr1 vs dr2 differ in coverage/targets. Add object_id in release 2 at the end of the table; release-1 clients still read old buffers fine.
• Cross-language integrity: the same .fbs compiles to Rust (API) and C/Odin (GUI) with byte-identical wire format — no hand-maintained JSON contract drift, which is exactly the data.odin mirror-struct problem today.
• Single obvious streaming primitives: size-prefixed buffers make framing trivial; vectors make bulk data compact.

Official resources

• Tutorial: https://flatbuffers.dev/tutorial/
• Schema guide: https://flatbuffers.dev/schema/
• Evolution rules: https://flatbuffers.dev/evolution/
• Rust usage: https://flatbuffers.dev/languages/rust/
• C (flatcc) usage: https://flatbuffers.dev/languages/c/
• White paper: https://flatbuffers.dev/white_paper/
• API docs (Rust): https://docs.rs/flatbuffers
• crates.io: https://crates.io/crates/flatbuffers