web/docs/ARCHITECTURE.md

Architecture

How a page load becomes a training node, and how a handful of browsers stay bit-identical while training one model together.

 browser A ──┐                       ┌── browser B
  WebGPU/CPU │   WebRTC data channels │  WebGPU/CPU
  verified   ├──────── gradients ─────┤  verified
  INT8 units │                        │  INT8 units
             └──── server.js (signaling + static + /data) ────┘
                    never sees weights or gradients

The pieces

File	Role
server.js	WebSocket signaling (introduces peers, relays WebRTC offers/ICE), static hosting, and the /data endpoint that streams FineWeb-Edu text. It never sees weights or gradients.
public/app.js	the WebRTC mesh, the training loop, gradient averaging, checkpoints, and the sync guard.
public/transformer.js	the mini transformer LM (attention + MLP blocks, next-token prediction) — forward and backward, every multiply through the units.
public/verified_core.js	the verified INT8 units: block-scaled quantize → exact LUT multiply → int32 accumulate → bit-exact f32 epilogue; plus the JS mirrors, the audit, and the MLP chain reference.
public/webgpu.js	the WGSL kernels (LUT matmul, DP4A int8 dot, fused attention, B2B MLP chain) and the exact init gates that admit them.
public/traincore.js	float reference trainer + deterministic Adam.
public/*.bin	the units as lookup tables (mul_lut is a 65536-entry exact int8 product table).

Peers, groups, and the leader

Peers are grouped by public IP (same network ⇒ same group) or by an explicit ?room=CODE (with host approval per join). Inside a group every pair gets a WebRTC data channel — a full mesh; the signaling server is only used for the handshake.

Whoever presses Start becomes the leader for that run: their settings are broadcast to everyone, they sync mid-run joiners (weights + step), and they are the source of truth for gradient repair (below). Everything else is symmetric — every device computes, sends, and averages the same way.

One training step

1. Each device samples its own batch from its data shard and computes the loss and gradient through the verified units.
2. It broadcasts [step | weight-hash | probe-hash | loss | gradient] to every peer and collects everyone else's.
3. When it has a gradient from every device in the step roster, it averages them in strict roster order — float addition is not associative, so a different order would round differently and fork the weights — and applies one Adam update. Adam's state is identical everywhere, so nothing but the gradients ever crosses the wire.
4. Because every device starts from the same seeded weights and applies the same averaged update with the same rounding, the replicas are bit-identical — verified continuously by the weight hash.

The sync guard

Before applying an update, each device checks every peer's weight hash (FNV-1a of the full weight bytes before the step) against its own. Any mismatch means the replicas have forked — the guard stops the run rather than train past a divergence. The probe hash is different: it re-runs a fixed seeded int8 GEMM through the device's live kernel each step, so a device whose arithmetic has gone wrong is caught even while its weights still match (weights only depend on the gradient bytes everyone receives).

Gradient repair (asymmetric meshes)

WebRTC meshes can be asymmetric: a gradient can reach the leader but not some follower. Instead of stalling, a follower missing a roster gradient asks the leader to re-send that peer's gradient for that step (the leader retains the last 8 steps of everyone's gradients). The repair is bit-exact — the follower averages the identical bytes — so the run continues without a fork. Only the leader may vouch for another peer's gradient.

The wire protocol

All messages are binary on the data channel. Gradient messages start with a non-negative int32 step; control messages use negative sentinels:

Sentinel	Message
-2	checkpoint push (one device restores the whole group)
-3	run config from the starter (c, t, b, steps, lr)
-4	step roster (who is in this step)
-5	fragment — large messages are chunked at 48 KB
-6	mid-run resume (weights + step for a late joiner)
-7	repair request (step, peer)
-8	repair response (leader-only, original header + bytes)

Gradient wire format: [i32 step][u32 whash][u32 phash][f32 loss][f32 grad…].

Compute backends

At init, webgpu.js tries the best available backend and gates each kernel before use (see VERIFICATION.md):

1. DP4A (packed_4x8_integer_dot_product) — hardware int8 dot product.
2. LUT shader — the multiply table as a WebGPU storage buffer; also the oracle twin every other kernel is compared against.
3. CPU (JS) — the same units in plain JavaScript. Not an approximation: the CPU mirror and the GPU kernels produce the same bits, which is what lets GPU and CPU devices train in one group.

There is no plain-float forward path — every multiply in the model goes through the units on every backend.

Training data

server.js exposes /data: it picks a random slice from the FineWeb-Edu 10BT parquet shards and reads it directly off the HF CDN with HTTP range requests (pure-JS hyparquet, SNAPPY decompression, results cached). No datasets-server dependency. Browsers fetch /data per batch; devices that cannot reach it fall back to a small built-in corpus. The dataset is hardcoded — every group trains on FineWeb-Edu.

Checkpoints and the inference kit

The .pt checkpoint stores magic/version, config dims, step, and the flat f32 weights; the loader validates tokenizer vocab and dimensions before accepting, and a load broadcasts to the group (sentinel -2). The inference kit is a generated single-file HTML with the model code and the current weights (base64) baked in — offline generation anywhere.