When a raccoon shows up at the glass, you do something automatic: "you look like Notch… but you're not acting like Notch." That tiny friction is the whole project. You just ran two classifiers — one for appearance ("looks like X"), one for behaviour ("but doesn't act like X") — and they disagreed.
The disagreement is the information. It's either a new individual who resembles a known one, or a known one in an unusual state — and which it is, is exactly what you want to know.
So the rule for the whole system: keep appearance and behaviour on separate axes, and surface both. Never collapse them into one confident answer. The job isn't to be an oracle that names every raccoon — it's to give your own instinct a memory and a second opinion. Everything that follows is built to that brief, and it's honest about where it falls short.
A camera left running all day can't run a GPU model on every frame — it'd melt. So the first thing in the loop is the cheapest: a motion gate (MOG2 background subtraction) that does almost nothing until something actually moves. Only then does it wake MegaDetector v6 — Microsoft's camera-trap model, run straight through Ultralytics — to ask "is that an animal?" Each hit becomes a cropped image, a shot-quality score, and one row in a SQLite database.
Step a single real capture through every stage below. Watch the detector stay asleep until the motion blob crosses the threshold — that's why the rig can sit running next to you.
The camera shoots through a sliding glass door, and most of the action is after dark. Reflections and haze soften everything; a raccoon pressed to the pane with a light on it reads clearly, but plenty of frames come out a little flared. There's no fixing the optics — so instead every crop is scored for shot quality: sharpness, with a bump for the bright eyeshine of an animal looking back at the lens.
That score is why the journal can lead each visit with its cutest, sharpest frame rather than merely its most confident one. Scrub from the softest shots to the sharpest and watch the real spread.
Every crop gets a species, zero-shot, from BioCLIP 2 — no training, just a list of candidate species as text. On a real raccoon or crow it's emphatic (0.99, 1.00). But BioCLIP has one blind spot that took a while to understand: it's an organism-only model. Show it an empty deck or a plate of food and it cannot say "not an animal" — it's forced to pick the nearest species anyway.
The fix isn't a better species list; it's a different model asking a different question. A
general CLIP runs first and answers the one thing BioCLIP can't:
"is this even an animal?" Real animals score near zero; genuine non-animals score high.
The cut sits at 0.60, in the wide empty gap between the two.
Drop crops onto the bench and drag the threshold yourself.
Now the hard part: telling individuals apart. Each crop becomes a 1,536-number appearance vector (MegaDescriptor, a wildlife re-ID model). Two vectors that point the same way are two crops that look alike. Plotted below — a real map of raccoon crops — and the first honest finding lands immediately.
Colour by visit and the dots clump into tight little ponds: the same raccoon, same night, scores up to 0.99. Colour by individual and those colours smear across each other — the same raccoon on two different nights barely beats two different raccoons. Background and pose dominate. Single crops can't be matched across sessions.
So average each visit's best crops into one prototype and the noise washes out. The same animal's nights now match at ~0.84; two different raccoons sit near 0.37. Hit "collapse to prototypes" and watch it happen — then note that Elliot, who rarely visits alone, stays stubbornly ambiguous. That's not a bug; it's why a human still confirms.
Before you can name individuals, you have to notice when there are two of them. When the detector returns two boxes, the question is whether they're two raccoons or one raccoon boxed twice. The test is intersection-over-union — how much the boxes overlap. A lot of overlap means the detector double-counted one animal (non-max suppression merges them); only a little means two separate bodies.
Below 0.45, the visit earns a "2+ raccoons" badge, and
its blended appearance is set aside — because one name stamped on two animals poisons the
template. Drag the boxes and watch the verdict flip. (In practice the sparse stills miss most
pairs; the full-rate clips catch them — which is exactly what the next chapter leans on.)
Because appearance is only a hint, the system never names anyone on its own. It suggests, and a human confirms — and every confirmation sharpens the next suggestion. Play the human: here are real visits with the model's actual guess. Call it, then see the truth and how you scored against the machine.
Then the finale — the trick the project is proudest of. A raccoon who only ever shows up with another raccoon never gives you a clean solo photo to learn from. But a behaviour clip tracks each animal separately, so its frames can be split apart. One real two-raccoon visit un-blends into 36 and 29 frames — two distinct animals out of one blurry crowd — which is how a never-solo regular finally earns an identity.
A still says who and when; a clip says how. The detector runs over every frame of a behaviour clip and threads the boxes into one track per animal, then reads its motion — how fast, how straight, how hesitant — and the stride cadence: the rhythm of the body bobbing up and down as it walks, pulled straight off the video with no extra sensor.
Gait is a confound-robust second opinion for identity: a limp reads the same from any angle, where a single soft still does not. Pick a real walk and watch its metronome tick at the cadence the rig actually measured.
Back to the beginning. Appearance is one axis; behaviour is the other. From the visit log, each raccoon gets a profile — when they arrive, how long they stay. Raccoons here run 20:00–02:00; Notch skews even later.
Plot a visit on the two axes at once. When appearance and behaviour agree, it's quiet. When they disagree — looks like Notch, but arrived at 2pm — the alert fires: either a look-alike, or Notch doing something unusual. Grab a visit and drag its arrival time around the clock; watch the behaviour needle swing and the alarm trip. That swing is the whole project, made literal.
None of this is one clever model. It's a lot of small, boring, robust pieces around one idea: most of the value is in capture and accumulation, not the fancy network. One SQLite database is the brain; the live dashboard is hand-written on Python's standard library with no web framework; the GPU sleeps through the quiet hours.
The page you're reading is a frozen, camera-free export of that database — the live rig is localhost-only because it shows a live feed of someone's backyard. This is the opposite: a curated, public slice — person-labelled detections filtered out, a hand-written veto for the ones the labels got wrong, and the survivors checked by eye.