SAM 3.1 tracks 16 objects at once: real-time video on one H100

On 27 March 2026, Meta released SAM 3.1, a drop-in SAM 3 replacement optimised for video. Its main change is object multiplexing: tracking up to 16 objects in one forward pass instead of one pass per object. Meta reported an increase from 16 to 32 frames per second on one H100 for medium-object-count video.

This vendor result covers a particular workload and does not guarantee 32 FPS at every resolution, duration and instance count. The architectural change still matters: it removes repeated work and memory bottlenecks, while global reasoning considers objects together.

Segmentation, detection and tracking

SAM 3 accepts text concepts, image exemplars and visual prompts such as points, boxes and masks. It detects instances, builds pixel masks and preserves object identity across frames. A concept such as “striped red umbrella” need not be on a fixed class list.

Multiplexing is useful in sport, editing, inspection and monitoring where many objects move at once. Lower GPU demand can move processing closer to the source or increase streams per server. Meta published a checkpoint, code and paper, but teams should reproduce performance on their footage.

Measuring more than FPS

Evaluate mask IoU, identity stability, end-to-end latency and recovery after occlusion. Measure false positives and differences across groups and lighting. Include rare objects, crowds, camera movement, compression and long disappearances.

Production needs model and prompt versioning, backpressure, drift monitoring and human review. “Real time” is a whole-pipeline budget covering decode, preprocessing, inference, postprocessing and delivery.

Privacy and abuse

Segmentation is not identity recognition by itself, yet combined with re-identification it can enable tracking. Video may be personal data; limit purpose, access and retention. High-risk uses require legal and proportionality review, not only technical approval.

SAM 3.1 shows that progress can come from making an existing capability dramatically more efficient. Red-team the model, privacy boundary and API controls before deployment.

Multiplexing and memory planning

Processing objects together shares image features instead of encoding the frame repeatedly. Memory still grows for masks, track history and objects. A limit of 16 requires policy for prioritisation, splitting and dynamic overflow.

The scheduler should avoid starvation: one crowded stream must not consume every GPU. Set per-tenant quotas, maximum resolution and track-retention time. Batching raises throughput but also latency, so tune for the use case.

Occlusion, identity switches and business error

After occlusion, the model may assign identity to another object. In editing that misplaces an effect; in monitoring it attributes an event to a person. Measure identity switches and track fragmentation, not only mask IoU. Sensitive decisions need a second signal.

Text prompts can be ambiguous. “Worker without a hard hat” combines person, object and spatial relation. Segmentation may find a helmet but cannot alone prove safety compliance. Domain logic and uncertainty thresholds belong outside the model.

Video API security

Video is hostile input. Decode in a sandbox with file, frame and time limits and updated codecs. Separate URL fetching from inference, block private addresses and inspect metadata. The model pipeline should not hold camera-system credentials.

Does a public checkpoint guarantee 32 FPS reproduction? It helps, but drivers, framework, preprocessing and data profile must match. Report the whole environment.

Deployment gate

Define minimum mask and tracking quality for each use, maximum p95 latency and overload behaviour. Low-confidence frames should be rejected or reviewed, not forced into an answer. Shadow SAM 3.1 against the current model on recorded traffic and compare failures before making it the only path.

Primary sources: Meta AI — SAM 3.1, Meta SAM 3 codebase.