A Multi-Signal Design for Follow-Cam Tracking

A follow-cam needs more than a face detector. It has to establish who to follow, preserve that association through choreography, and frame the person without quietly drifting to someone else in the formation.

This design combines identity, person tracks, appearance, pose, segmentation review, and hybrid framing. Each signal has a different role, so no single detection has to do all the work.

Identity is the starting point

The current pipeline uses InsightFace’s FaceAnalysis model with its detection and recognition modules. It builds one normalized reference embedding by averaging the accepted faces from curated target images.

On each sampled frame, it compares every detected face embedding with that reference. The top match must clear a similarity threshold, keep a sufficient lead over the next-best match, and pass a position-jump check.

Diagram showing accepted target images being encoded, averaged, and normalized into one reference embedding for face matching in sampled video frames. — Identity is established first: curated images become one reference embedding, which each candidate face is measured against.

Accepted matches become identity anchors for the crop. Person boxes, tracks, and body-aware framing extend that anchor; they do not replace the identity check.

A multi-signal design

Each design path receives the same source clip, target reference, renderer, and audio path. Together, they separate three jobs: identifying the performer, carrying that identity across frames, and framing the crop.

The six paths below make those roles visible. Their overlays and metrics show not only how the crop moves, but also the evidence used to make that movement.

Six ways to make the crop follow

Signal map of the six follow-cam paths, all beginning with a face identity anchor and adding tracking, appearance, pose, review, or hybrid evidence. — Every route starts with identity. The added signal, contribution, and limitation match the six design paths used in the comparison.

Tracking uses YOLO11m and BoT-SORT; ReID pairs BoT-SORT with OSNet. Tracking can persist on the wrong person without fresh identity evidence, while ReID is slower and needs distinguishable appearance.

Pose uses YOLO11s estimates to plan the crop around the body. The segmentation adapter draws boxes and guides rather than filled masks; hybrid combines person framing with those review overlays.

When a real model is unavailable, the pipeline uses a face-anchored proxy rather than treating the added signal as identity evidence.

Choosing a design path

The paths solve different problems, so there is no universal winner. On the retained clip, the baseline is best; this map shows when an added signal is the better fit.

Decision map that pairs low cost and strong identity confidence with face anchors, continuity with person tracking, and competing tracks with body re-identification. — The retained clip favours face anchors; tracking and ReID earn their added cost only for their specific continuity or appearance constraints.

BoT-SORT combines motion and optional appearance information for multi-object tracking. Its paper and Ultralytics’ tracker documentation describe the tracker. The OSNet paper describes the appearance model used in the ReID option.

One shared comparison

Each run produces an output video, an overlay, frame-level metrics, and a review panel. The panels form one 2 by 3 grid, making it possible to inspect the same moment across every design path without switching between files.

One retained run uses a 30-second, 4K, 60 FPS source with all six options in real-model mode. Its person-tracking path used YOLO11s; the current implementation uses YOLO11m. The runs take 87 to 460 seconds.

The baseline processes about 19.7 frames per second; the ReID run processes about 3.9. Those figures describe the retained run, not a general performance claim.

The run shows the cost of each design path on one machine and one clip.

Reading the comparison

On this clip, the baseline kept the same selected track throughout. The person-tracking options changed selected tracks several times. The comparison also measures crop smoothness and whether the selected body box stays inside the crop.

A track ID change does not automatically mean a wrong-person crop, and a stable crop does not prove the right person was selected. The overlays and video provide the necessary context for every change.

Every path found the target in every measured frame on this clip. The result compares cost and behaviour, but it does not establish how the added signals perform in the harder situations they are designed for.

Limits and next steps

This solution still depends on curated references and sampled-frame identity anchors. Wrong-person frames and the longest losses still need manual annotations or derived metrics.

Those annotations should focus on a back-facing turn, an occlusion, a tight formation, and a target leaving then returning to frame.

The next step is not a free-form video agent. It is a bounded recovery controller: on low confidence or track conflict, inspect the loss window, re-anchor only with fresh face evidence, or mark the target lost for review.

VideoAgent is a useful reference for temporal, object-centred memory. The VOT long-term methodology and HOTA point to how re-entry, identity association, and localization can be evaluated.

The aim is not more automation. An agent or added signal earns its cost only if it recovers safely, exposes better evidence, and makes wrong-person switches easier to detect.