Segmentation = labelling every pixel of a frame. Sentinel runs it two ways, both on-device in the browser (WebGPU): a tiny custom fine-tuned model for one threat class, and the SAM 3 foundation model to segment & track anything. This page explains both pipelines and when we use each.
A small model (LR-ASPP MobileNetV3, ~12 MB ONNX) trained for one job: fire vs. not-fire, per pixel.
Meta's Segment Anything 3 — segment & track any object by a click, box, or text prompt. Zero-shot, no training.
Both models share the encoder–decoder shape: an encoder compresses the image into semantic features ("what"), a decoder upsamples back to full resolution ("where"), and skip connections carry fine detail across so boundaries stay sharp.
Encoder blocks (left) downsample to a low-res bottleneck (richest semantics); decoder blocks (right) upsample back to a full-resolution per-pixel mask; skip connections carry fine detail straight across so boundaries stay crisp.
Our fire model is exactly this. SAM 3 uses a heavier transformer (Hiera) encoder run once per image, then a fast promptable decoder that turns a click/box/text into a mask — which is why interaction is instant but the first encode takes a beat.
No public fire-segmentation dataset existed, so we built one. This is the exact pipeline behind the Fire alerts:
843 images via Grok/xAI, deliberately over-weighting hard negatives (sunset, brake lights, lava, neon).
Grounded-SAM (Grounding-DINO + SAM) localizes flame; a warm/hot colour gate refines to flame pixels.
LR-ASPP MobileNetV3, weighted loss, domain randomization (JPEG, blur, chyron bars, broadcast-text overlays) + real harvested frames.
Export to a ~12 MB ONNX graph; inputs ImageNet-normalized 512², output 2-class logits.
onnxruntime-web runs it in the browser, ~44 ms/frame — no server, no data leaves the device.
Auto-applied to fire alerts; powers the jump-to-fire timeline (every frame scored).
For everything other than fire, we don't train a model — we prompt a foundation model:
Trained by Meta on the huge SA-Co concept dataset — already knows objects.
transformers.js + WebGPU pulls the ONNX (q4 encoder) into the browser. One-time download.
The alert's keyframe is embedded once (~3–4 s).
Click / box / text → segment that object. ~66 ms per prompt.
Propagate the mask across frames to follow the subject (vehicle, crowd).
| Custom fire model | SAM 3 | |
|---|---|---|
| Task | One class (fire), per-pixel | Any object, promptable + concepts |
| Training | Full pipeline (synth → label → train) | None — zero-shot |
| Size / speed | ~12 MB · ~44 ms/frame (real-time) | Large · ~3–4 s encode, 66 ms/click |
| Best for | The known threat, at video rate | Ad-hoc “segment that” on any alert |
| The moat | Yes — hard-negative discrimination | Generality & speed-to-new-class |
| Runs on-device? | Yes (WebGPU) | Yes (WebGPU) |
The pattern (research §1/§7/§8): a foundation model for breadth + a tiny fine-tuned specialist for the known threat at the edge. SAM 3 lets us stand up a new capability in seconds; the custom pipeline hardens the one threat that has to run fast and never cry wolf.