Fast video recognition by learning when, where, and what to compute
Most tokens in a video are redundant. LookWhen trains a tiny selector that looks at a low-res copy of the clip and decides when and where to look. A deep extractor then computes on only the top-K tokens (typically 5–30%) and still approximates features of the whole video.

The selector scores every space-time token from a 2×-downscaled clip. Only the top-scoring tokens are sent at full resolution to the extractor. Everything else is never processed by the deep network.
Selection follows the action through time: the jump and splash, the swimmer, a diver mid-flight, a hand gesture. Tap a frame below the video to scrub.
A shallow selector decides when and where to compute. A deep extractor decides what to represent, from the selected tokens only.

A token is worth computing if nothing else in the clip looks like it: its distance to its nearest neighbor in DINOv3 feature space, across all frames.
Predict InternVideo2's video token and DINOv3's tokens normalized over time, so it learns what changes in the video.
Teachers are only used in pre-training.
LookWhere, the image version of this idea, trained its selector to copy a teacher's attention map. In video that signal breaks down, and selecting by uniqueness is more effective.
We compare against our video teacher InternVideo2-B (IV2) run on all tokens, and against two token-reduction methods, RLT and vid-TLDR, that we upgraded by applying them to IV2.
Linear probe: a linear classifier on frozen features, which measures feature quality. Fine-tune: train the whole model. LookWhen is pre-trained on Kinetics-400 + SSv2; the other four datasets are held out. LookWhen's points vary sparsity from 50% to 95%.

Mean accuracy over the same six datasets (20K-sample training subsets, 90% sparsity). Each bar is a separate pre-training run.
Teacher · target
Targets removed from the full recipe


@inproceedings{salamatian2026lookwhen,
title = {{LookWhen?} Fast Video Recognition by Learning When, Where, and What to Compute},
author = {Salamatian, Ali and Fuller, Anthony and Sarkar, Pritam and Green, James R. and Sigal, Leonid and Shelhamer, Evan},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}