AI Signal 372
Frame selection is the whole game: notes on making LLMs watch video
If you are building systems that pass video through multimodal LLMs, uniform temporal sampling is likely wasting your token budget on redundant frames while missing the moments that matter. The article details a multi-channel deduplication approach that adapts to content type, catching small-subject motion and subtle UI changes that naive percentage-threshold methods drop.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
A vision LLM can realistically afford about 150 frames per video, so every kept frame must earn its slot through adaptive scene detection and deduplication rather than fixed-interval sampling.
Fixed scene-score thresholds fail on animation and slow camera work; using a rolling-average multiple instead lets high-motion footage raise its own bar and quiet footage lower it without manual tuning.
Deduplication required three separate channels—global RGB signature, action grid for small subjects, and a settled channel for local state changes like captions or UI updates—each added after real footage exposed blind spots in the previous one.
THE CLUSTER