Jatagan · Engineering Note

Jatagan Filter — second-stage AI verification

An object-detection model on a Hailo-8 accelerator that decides whether a camera's tripwire alarm deserves a human's attention, before an email reaches the monitoring agents.

Status 336 events decided across five days, four cameras Hardware Hailo-8, 26 TOPS · yolox_l_leaky Latency ~8 s from crossing to email

The problem

The camera is not lying. It is answering an easier question than the one we care about.

A Dahua camera with IVS rules reports CrossLineDetection whenever something crosses a tripwire. It reports it for shadows sweeping across pavement, for headlights at night, for rain, for branches moving in wind. Every one of those becomes an email, and every email costs a monitoring agent the seconds it takes to open it, look, and dismiss it. Enough of them and the alerts stop being read.

The obvious fix is to run object detection on the alarm snapshot: if there is no person in the frame, drop it. That removes the shadows and the rain. It does not remove the most common false positive, and that is the whole reason this project has the shape it does.

Why one frame is not enough

Presence and crossing are different questions, and only one of them fits in a photograph.

So the field agent does not send a snapshot. It sends a window: about two seconds before the event and two after, sampled at 4 fps. The office side detects objects in every frame, links them into tracks, and tests whether a track's path actually intersects the rule geometry.

Architecture

Neither end can accept inbound connections, so both dial out.

Why not MQTT, and why not object storage

MQTT is the right tool for telemetry — trailer alive, camera down, temperature — and it will likely carry exactly that later. It is the wrong tool for a megabyte of JPEGs per event: large payloads work against the protocol, and it means operating a broker for something HTTP already does.

Object storage as the primary route (upload to R2, have the office poll) adds a hop and its latency to a path whose whole purpose is getting an operator's attention quickly. The failure it protects against — office unreachable — is covered more cheaply by a disk-backed retry queue on the trailer. Storage remains attractive later as an archive, not as transport.

The event package

What the office needs in order to rule on a crossing.

event.json        rule geometry, timestamps, what the camera itself claimed
frames/0000.jpg   analysis window, ~4 fps, 640 px long side, q80
hd.jpg            full-quality frame, for the operator's email

Analysis frames are downscaled deliberately. The network resizes everything to 640×640 regardless, so shipping 4K over a metered link buys pixels that are thrown away before inference. Only the operator's frame keeps full quality. Rule geometry travels in Dahua's fixed 8192×8192 space, exactly as the camera config reports it, so it stays valid at any stream resolution.

The verdicts

Each answers a different question. Every one of these exists because a real event exposed the need for it.

Crossing is tested at the object's ground point — the bottom-centre of its box, where feet meet the floor. Using the box centre would register a tall person as having crossed while they are still on the near side.

Resolution: why more pixels did not help

The most expensive lesson of the first morning, and the least intuitive.

The filter was missing real people. A camera reported a person crossing, we detected nothing, and the evidence photo looked empty — so the verdict was recorded as the camera hallucinating. It was not. Someone who knew the site looked at the same picture and saw the person immediately.

The obvious response is to send higher-resolution frames. It does nothing. The detector resizes whatever it receives to 640×640 before inference, so a 1080p frame and a 480p frame arrive at the model identically, and a distant person shrinks back to a handful of pixels either way. Measured on the same scene, the substream at 640, the substream at native size and the main stream at native size all returned exactly the same detections.

What works is cropping instead of scaling. The event tells us where the camera saw something; the agent cuts that region out of the main stream at native scale and ships it alongside the frames. The crop is small in bytes precisely because it is small in area, and the person inside it is no longer 45 pixels tall but around 125.

One consequence worth planning for: raw frames at 1080p cost about 6 MB each, and an eight-second buffer per camera became 750 MB. Four agents exhausted the machine's paging file. The buffer now retains 5 fps instead of 15 — every frame is still read so the decoder stays current, but only the ones we would actually ship are kept. Four cameras now run in 1.5 GB and under 6% CPU.

Worked examples

Six real events from the first morning: what the trailer sent, what the office concluded, and whether it was right. Amber is what the camera claimed; green is what the Hailo found; red is the rule.

Audit queue: nine unresolved rejections

Every case where the camera reported a person and we detected nothing there. This is the category that hides false negatives, so none of these should be assumed correct until someone who knows the site has looked.

Of 60 events processed, 31 were rejected. Most of those rejections are straightforward — someone walked near a line without crossing it. These nine are different: the camera insisted a person was at a specific place and our detector found nothing in it. That is exactly the shape of the one confirmed miss of the day, so the whole group is suspect.

Each entry shows a zoom of the exact region the camera pointed at, because at full-frame scale these subjects are a few dozen pixels and impossible to judge — the mistake that started all of this. The full frame is beside it for context. Two cases are already settled and are included first, as calibration for the rest.

The accelerator, and what it cost to get working

Everything below was discovered the hard way and is written down so it is discovered only once.

Measured

The verdict logic is covered by a synthetic test that pans a still image to fabricate motion. It asserts both outcomes that matter: a moving object is confirmed as crossing, and a stationary one beside the line is rejected with the "never crossed" reason. Both pass.

Where the eight seconds go

Most of the delay is a deliberate wait, not slowness. Knowing which part is which is what makes it safe to leave alone.

The three seconds spent building the package are the price of the whole design. After an event arrives, the agent waits two more seconds so the frames following the crossing land in the buffer. Without them there is no trajectory, and without a trajectory we are back to being unable to tell someone crossing a line from someone walking beside it. Cutting that wait would make the system faster and worse.

Two of the remaining stages are recoverable without touching decision quality: the SMTP session is rebuilt on every send rather than held open, and there is a half-second safety margin after the window that could be trimmed. Together they would bring this closer to five or six seconds. That work is deferred on purpose — eight seconds from a crossing to a verified email with a photograph is comfortably inside what monitoring needs, and the camera's own unverified alerts do not arrive faster.

Components

Dependencies were held to numpy and opencv on purpose. After the fight with a missing C compiler, a server that needs nothing installed is one less thing to break at three in the morning.

Five days in the field

336 events, decided and archived. What the numbers say, and what they still cannot tell us.

182 of 336 events were rejected — 54%. If every rejection is right, that is 54% fewer emails reaching the monitoring agents. That figure is the headline, and it should be read carefully: it is what the filter decided, not what has been verified.

The largest single rejection category is the one that justifies the whole design. 98 events — 29% of everything — were people or vehicles genuinely present that never crossed the rule. A filter that only asked "is there a person?" would have forwarded every one of them as a real crossing. That is the trajectory test earning its cost, measured rather than argued.

Three outages, three different causes

None came from the detection logic. All came from the machine, the operating system, or the plumbing around it.

14 August — committed memory exhausted. After six days of uptime the trailer machine reached 97% of its commit limit, with about 24 GB unattributable to any visible process. Agents began failing every few minutes, then stopped decoding video entirely. A reboot brought committed memory to 50% and free disk from 8.9 GB to 28.3 GB. Contributing cause: the system drive was nearly full, preventing Windows from growing its page file — the very first error we saw said exactly that, hours before anyone connected it to the crashes.

17 August — the server was terminated from outside. Its log ends in an interrupt, not a crash, after running three days and processing 88 events without a single failed upload. The cause was never established.

17 August — and a bug of ours that made it worse. When the server went away, one agent tried to queue its package for retry and died outright. The rule is named Rule1 24/7; the filename sanitiser replaced spaces but not the slash, Windows read it as a directory, and the unhandled exception took the process down.

Since the fix, the full cycle has been observed working end to end: upload failed, package queued to disk, server returned, queue drained on its own, and the delayed event was decided normally.

Switching to YOLOX

Driven by licensing, not performance — and the performance turned out better than expected anyway.

From 18 August the server runs yolox_l_leaky instead of yolov8x. The reason is not technical: Ultralytics YOLOv8 is AGPL-3.0, YOLOX is Apache-2.0. For purely internal use the distinction may not bite, but it constrains what the system can later become, and the cost of keeping that door open turned out to be small.

In production YOLOX runs at 30.6 ms per inference against 54.5 ms — a 44% reduction, better than the bench figures suggested, because the NMS the chip now performs was previously costing us CPU on every frame.

The switch required a new decoder. YOLOv8 HEFs expose the raw head; YOLOX emits a single output the accelerator has already resolved into per-class detection lists. yolo_hailo.py now detects which it has been given rather than being told, so changing models is one argument in a startup script.

Validation before going live: the reference image returns the bus and all four people, with boxes matching yolov8x almost exactly, and without the spurious low-confidence detection yolov8 produced in that same frame. Both verdict tests pass.

What this comparison does not yet prove. Contrasting the yolov8x days against the yolox days is not a fair test — different traffic, weather and people. The 6-point mAP gap should surface somewhere, most likely on distant pedestrians, and the only honest way to find out is to replay the same archived packages through both models. We now store 313 raw event packages, so that comparison is available to run.

Storage, after five days

The evidence store holds 989 files, 960 MB, of which 313 are raw event packages. That is roughly 190 MB per day at current traffic — which is why the retention policy exists now rather than being a future concern.

The cleanup process has not yet deleted anything, which is correct: packages expire at five days and the oldest is exactly five days old. The first sweep is due imminently, and it will be the first evidence that retention actually works rather than merely being configured. Under the current limits the store should settle around 4 GB against a 20 GB ceiling.

Open questions

Thresholds are untuned. Confidence sits at 0.30 with yolov8l — more sensitive than it will end up, chosen deliberately after the missed person. Extra detections can be discarded by the trajectory test; something never detected cannot be recovered. The server archives every rejection and now the raw event package too, so thresholds can be raised against recorded evidence instead of intuition.

Verdicts are only spot-checked, and the checks themselves have erred. Of the first events, six were audited against their images. Five verdicts were correct. But two of those five were initially recorded here as failures, and one genuine failure was initially recorded as a success — each corrected only when the site operator, who knew what had actually happened, contradicted the write-up. Reviewing a proper sample is the next real work, and it should be done by someone who knows the location.

Deduplication is not built yet. When it is, it must key on the track or the event — never on position. Position-based dedup was already tried on a related system and it silently discarded legitimate new arrivals.

Night performance is unknown. Every measurement so far is daylight or synthetic. IR illumination, motion blur and rain all change detection confidence, and the thresholds that work at noon may not hold at 3am.