desi_explorer/README.md

DESI Explorer

An interactive 3D map of the universe, built from data gathered by the Dark Energy Spectroscopic Instrument (DESI).

Renders millions of galaxies / quasars in real time as a navigable 3D point cloud, combining Odin for the renderer with raylib for windowing, input, and 3D drawing.

What is this?

A fun personal project with three interlocking goals:

• Turn DESI's survey into something you can fly through — the Dark Energy Spectroscopic Instrument maps a huge volume of the observable universe (galaxies, quasars, and the Lyman-alpha forest) in three dimensions. This project renders that data as an interactive 3D experience rather than a static plot.
• Serve the data, not the download — the datasets are large, so the experience pulls from a centralized repository / API instead of requiring you to keep the raw catalogs locally (see Goals).
• Ship to native and the web — one Odin codebase compiled to a native desktop executable and to WebGL/WASM, so the map runs anywhere.

Goals

• Scaffold the project: Odin + raylib 3D renderer, Go/Pulumi infra, Gitea CI
• Pull data from a centralized repository to limit storing all the DESI data locally
• Be both a native executable and a WebGL-based interactive experience
• Build utilizing the Odin language
• (Future) Integrate with other space data to be able to incorporate both dark energy information and more

Current State

Early scaffolding. The application currently:

• Opens a resizable 3D raylib window with an orbital camera (zoom + rotate + pan).
• Renders a procedurally generated point cloud standing in for the galaxy catalog (real DESI data ingestion is the next milestone).
• Compiles natively, to WebAssembly, and is deployed to an on-prem Kubernetes cluster as a placeholder web service.

Roadmap

• Ingest real DESI data (EDR/DR1 catalogs) and map survey coordinates (RA / Dec / redshift) into 3D space
• Central data repository / API so large catalogs aren't stored locally
• Efficient rendering of large point clouds (instancing / vertex buffers rather than per-point draws)
• Camera flight / goto-object controls and a minimap
• Color-coding by redshift, survey, or object type
• Object selection & metadata inspection
• Web (WASM/WebGL) parity with the native build
• (Future) Fold in other datasets for dark-energy context

Repository Layout

gui/           Odin renderer — the interactive 3D experience (entrypoint: gui/src/main.odin)
  gui/lib/     Third-party Odin dependencies (submodules / vendored libs)
  gui/www/     Web GUI — WASM shell / static web assets
  gui/scripts/ Build scripts (WASM build helper)
api/           Rust backend API (axum) for serving DESI catalog data
infra/         Go + Pulumi infrastructure-as-code (deploys the experience to k8s)
scripts/       Repo-level build helpers (Odin install)
.gitea/        Gitea Actions CI/CD workflows

The renderer (gui/), the API (api/), and the infrastructure (infra/) are kept in separate directories so their logic stays cleanly separated. Each project carries its own Makefile; the root Makefile delegates the base commands (run, build, test, ...) into them.

Building & Running

Requires Odin with the bundled raylib vendor bindings. The web build additionally requires Emscripten (emcc). The API requires Rust (stable toolchain, pinned via api/rust-toolchain.toml). The infrastructure requires Go and Pulumi.

The root Makefile delegates everything to the per-project Makefiles (make help lists the base targets):

make setup        # pull submodules + tidy Go deps
make run          # run the native app (gui/)
make build        # -> bin/desi_explorer (gui/)
make build-debug  # debug native build (gui/)
make build-web    # -> build/web (wasm + html, gui/)
make test         # odin test + go test + cargo test
make clean        # remove build artifacts from all projects
make fmt          # format all projects

make run / make run-web configure the API and renderer from the example assets under resources/dev/ (see the dotenv lib in gui/lib/local/dotenv). Override any of them on the command line:

make run \
  GUI_ENV_FILE=/path/to/gui.env \
  API_ENV_FILE=/path/to/api.env \
  API_DESI_DATA=/path/to/desi_data.json

The defaults point at resources/dev/gui.env.example (renderer's API_URL), resources/dev/api.env.example (API API_BIND_ADDR), and resources/dev/desi_subset.json (a small JSON catalog subset served by the API's /api/v1/catalogs and /api/v1/objects endpoints).

Project-specific targets live in their own Makefile and are reached with make -C <dir> <target>:

make -C gui help    # Odin renderer (run, build, build-debug, build-web, test, fmt)
make -C api help    # Rust API (run, build, test, check, fmt)
make -C infra help  # Infra (preview, up, down, refresh, static, test)

The root also provides thin passthroughs: make api-<target> and make infra-<target> forward to the corresponding sub-project (e.g. make api-check -> make -C api check).

Dependency Updates (Renovate)

Renovate keeps dependencies up to date by opening PRs. It runs centrally on the homelab Gitea — a dedicated renovate Gitea Actions runner in the homelab k3s cluster runs renovate every hour (renovate workflow in the homelab repo) with autodiscovery filtered to sam_oneal/*. Because this repo carries a renovate.json on main, it is picked up automatically. Config for this repo lives in renovate.json and is validated with make renovate-validate.

It manages four categories:

• Infra dependencies — infra/go.mod (gomod manager); minor/patch updates grouped into a single PR.
• API dependencies — api/Cargo.toml (cargo manager); minor/patch updates grouped into a single PR.
• Submodules — gui/lib/ submodules via .gitmodules (git-submodules manager).
• Workflow actions — action versions in .gitea/workflows/*.yml (github-actions manager, which understands Gitea's .gitea layout).