desi_explorer/resources/ai/research/data-streaming/08-testing-strategies.md

08 — Unit & Integration Testing Strategies

FlatBuffers introduces a cross-language wire contract. The single most important testing principle: both sides must prove they talk the identical bytes, not just that each side round-trips with itself.

Testing layers

0. Schema hygiene (fastest feedback)

• Schema lint / conformance: run flatc --conform base.fbs candidate.fbs in CI to catch accidental schema-evolution violations (fields added in the middle, removals, type changes). Early warning before any of the below.
• Commit-and-diff: check generated code (.rs, .h) into the repo; CI re-runs codegen and git diff --exit-codes — catches "you forgot to regenerate" drift.

1. Rust-side unit tests (API crate)

The flatbuffers crate's own tests live in tests/rust_usage_test/tests/integration_test.rs — copy the patterns:

Round-trip test (write → read own buffer):

#[cfg(test)]
mod object_batch_tests {
    use crate::generated::desi::{ObjectBatch, ObjectBatchArgs, finish_object_batch_buffer, root_as_object_batch};
    use flatbuffers::FlatBufferBuilder;

    fn build_sample_batch() -> Vec<u8> {
        let mut fbb = FlatBufferBuilder::with_capacity(4096);
        let ra = fbb.create_vector(&[0.1, 25.3, 99.9]);
        let dec = fbb.create_vector(&[-5.0, 44.0, -77.2]);
        let z = fbb.create_vector(&[0.5, 1.2, 2.8]);
        let ids = fbb.create_vector(&[1u64, 2, 3]);
        let batch = ObjectBatch::create(
            &mut fbb, &ObjectBatchArgs {
                catalog: Some(fbb.create_string("dr1")),
                batch_seq: 0, n: 3,
                ids: Some(ids), ra: Some(ra), dec: Some(dec), redshift: Some(z),
                ..Default::default()
            });
        finish_object_batch_buffer(&mut fbb, batch);
        fbb.finished_data().to_vec()
    }

    #[test]
    fn roundtrip_columnar_batch() {
        let bytes = build_sample_batch();
        let batch = root_as_object_batch(&bytes).unwrap();
        assert_eq!(batch.catalog(), Some("dr1"));
        assert_eq!(batch.n(), 3);
        assert_eq!(batch.ra().unwrap()[1], 25.3);
        assert_eq!(batch.dec().unwrap()[2], -77.2);
        assert_eq!(batch.redshift().unwrap()[0], 0.5);
    }

    #[test]
    fn verifier_rejects_corruption() {
        let bytes = build_sample_batch();
        for idx in [0usize, 1, bytes.len() - 1] {
            let mut bad = bytes.clone();
            bad[idx] ^= 0xFF;
            assert!(root_as_object_batch(&bad).is_err(), "byte mutate at {idx} should fail verification");
        }
    }
}

Key Rust unit-test patterns:

• Values equal defaults are not serialized → test that explicitly: assert_eq!(batch.missing_field_u64(), 0) when field absent, and that an explicitly-set 0 is indistinguishable (unless optional scalar). Cf. the optional_scalars_test.rs macro in the flatbuffers repo.
• Alignment tests: assert struct/table accessor pointers are properly aligned relative to buffer start (the flatbuffers repo has generated_code_alignment_and_padding tests — replicate for Point/ columnar slices).
• Fuzz (proptest/quickcheck): roundtrip random vectors of f64/u64 through the builder and back (fb's roundtrip_vectors uses quickcheck). Add property tests: for any &[f64] built with create_vector, ra().unwrap() equals the input exactly.

2. Rust-side integration tests (HTTP/WS layer)

The repo already has api/tests/health.rs (spins the app against real routes via axum). Add an api/tests/catalog_fb.rs:

// Uses the same axum::Router::app() as health.rs, hits the new WS or
// application/flatbuffer endpoint with tower::ServiceExt::oneshot(),
// collects the body Bytes, and verifies + reads it with the generated types.
#[tokio::test]
async fn objects_endpoint_returns_valid_flatbuffers() {
    let app = desi_explorer_api::app(); // or routes::app()
    let resp = app.oneshot(Request::builder().uri("/api/v1/objects?catalog=dr1&limit=100").body(Body::empty()).unwrap()).await.unwrap();
    assert_eq!(resp.status(), StatusCode::OK);
    let body = axum::body::to_bytes(resp.into_body(), usize::MAX).await.unwrap();
    let batch = crate::generated::desi::root_as_object_batch(&body).unwrap();
    assert_eq!(batch.n(), 100);
    assert_eq!(batch.ra().unwrap().len(), 100);
}

For WebSocket integration, spawn the server on 127.0.0.1:0 in a test (tokio) and use tungstenite/tokio-tungstenite as a test client, or call the endpoint through axum's mock. At minimum: open a WS, receive N binary frames, verify each frame parses + verifies + row counts match expectations.

3. Odin-side tests (GUI crate)

odin test exists in this repo (make test → odin test in gui/). Structure:

• Reader unit tests in the Odin package: given a hard-coded byte blob (from a fixture file), verify and read expected field values.
• Round-trip against a builder written in Odin (if you hand-roll a builder for Path B) — mirror the Rust patterns.

// gui/src_tests/catalog_fb_test.odin (whatever path `odin test` picks up)
package catalog_fb

import "core:testing"

@(test)
test_read_static_batch :: proc(t: ^testing.T) {
    data := #load("../testdata/catalog_sample.bin", []u8)
    // Read root, iterate ra/dec/redshift via the reader, assert values.
    batch := root_as_object_batch(data)
    testing.expect(t, batch.n == 3)
    testing.expect(t, batch.ra[1] == 25.3)
    testing.expect(t, batch.dec[2] == -77.2)
    testing.expect(t, batch.redshift[0] == 0.5)
}

The critical test here is the one reading a fixture produced by the Rust side / flatc — see the "cross-language fixture" section below.

4. Cross-language conformance suite (THE key integration test)

This is the test that actually catches incompatibility. Design:

1. Static fixtures, committed to the repo (testdata/*.bin):
   • Built once by flatc --binary <schema>.fbs <data>.json (deterministic, reproducible in CI with the pinned flatc), or by the Rust builder in a small crate that writes testdata.
   • Cover every field type: catalogs, list, empty batch, batch with defaults omitted, optional scalars present/absent, enum variants, nested strings.
2. Rust reads fixtures + asserts expected values.
3. Odin reads the same fixtures + asserts identical expected values.
4. Both sides parse each other's output: a Rust test writes a buffer and a companion Odin test reads it (and vice versa). Practically: Rust generates a batch during cargo test, dumps to OUT_DIR, Odin test loads it via #load — but static committed fixtures are simpler and just as strong for the schema-versioned contract.
5. Byte-for-byte equality on builders: for deterministic input, the Rust builder output should byte-identical to flatc's. Assert on fixtures.

Generate fixture data from a JSON descriptor in the repo (testdata/catalog_sample.json) via:

flatc --binary --schema schema/catalog.fbs -- testdata/catalog_sample.json

Committed .bin files mean: no toolchain needed to run the test suite; CI can regenerate and diff to catch drift.

A pure-Odin reader on Path B lives or dies by this suite — it's the only guarantee your hand-rolled offset/jtable/vtable arithmetic matches Google's.

5. Transport-level integration (end-to-end)

Two-app test: run the real axum server with the real DESI fixture data, drive it from an Odin test binary (or a Rust test client for parity):

// api/tests/ws_stream.rs — spawn server on ephemeral port, connect WS
// as the client would, assert: schema message first, then N batches,
// then EOS; each batch verifies and cumulative n == expected.

This closes the loop on the exact runtime path (frame boundaries, file identifier, union tags) that unit tests miss.

6. Property / fuzz testing

• Rust: proptest generating arbitrary &[f64]/Vec<u64> row sets → build batch → read → compare; plus fuzz the verifier with mutated buffers (the flatbuffers repo does a one-byte-corruption sweep on its own tests — replicate: for many positions, flip byte, assert verifier either accepts correctly (data still consistent) or rejects; never panics / OOB).
• Odin: property tests are harder (no built-in fuzzer); rely on the Rust side for mutation fuzzing and keep Odin tests as fixed-fixture + round-trip sanity checks.
• Corrupt-length attack: a stream with length = 0xFFFFFFFF must be rejected at framing, not OOM. Test the length-prefix path explicitly (size prefix + WebSocket frame-size caps).

7. CI wiring in this repo

.gitea/workflows/ci.yml currently: odin test + build, cargo test + clippy, go test + vet.

Additions when FlatBuffers lands:

• A shared schema lane: flatc --conform check + git diff --exit-code after regen (catches schema drift).
• Rust lane: compile generated code included from api/src/generated, run new unit/integration tests (no extra CI deps if generated code is committed).
• Odin lane: run the conformance tests reading committed .bin fixtures (no extra CI deps).
• Optional: the end-to-end WS test (server + client) in the cargo lane.

Keep make test as the single gate: make test = odin test + cargo test + go test, and each now includes the FlatBuffers conformance pieces.

Testing checklist (implementation-ready)

• flatc --conform ran against schema history in CI
• Committed generated code matches a fresh regen (CI diff check)
• Rust: round-trip builder↔reader unit tests (all field types / enum / optional scalar / default-omission cases)
• Rust: verifier-safety & corruption fuzz
• Rust: HTTP + WS integration tests against live router
• Odin: reader unit tests on static fixtures
• Cross-language: Rust reads fixtures ✓; Odin reads same fixtures ✓; byte-for-byte parity on builder output
• End-to-end: real server + real client (WS) with DESI-shaped rows
• Length/frame-stealing attack tests (huge length, truncated frames)
• No regression to existing JSON health/catalogs tests during migration

References

• flatbuffers Rust test suite (patterns to copy): https://github.com/google/flatbuffers/blob/master/tests/rust_usage_test/tests/integration_test.rs
• optional_scalars_test.rs (default-omission semantics): https://github.com/google/flatbuffers/blob/master/tests/rust_usage_test/tests/optional_scalars_test.rs
• flatc --conform docs: https://flatbuffers.dev/evolution/
• Existing repo test conventions: api/tests/health.rs, make test, .gitea/workflows/ci.yml