NeurIPS 2026 Efficient video recognition

LookWhen?

Fast video recognition by learning when, where, and what to compute

Ali Salamatian1●, Anthony Fuller2,3●★, Pritam Sarkar1,3, James R. Green2, Leonid Sigal1,3★, Evan Shelhamer1,2,3★
1University of British Columbia  ·  2Carleton University  ·  3Vector Institute
●Co-first author   ★Co-advising author
TL;DR

Most tokens in a video are redundant. LookWhen trains a tiny selector that looks at a low-res copy of the clip and decides when and where to look. A deep extractor then computes on only the top-K tokens (typically 5–30%) and still approximates features of the whole video.

Read the abstract
Transformers dominate video recognition. They split videos into tokens, and processing them has expensive superlinear computational cost. Yet videos are filled with redundancy, so we can question the need for this expense. We introduce LookWhen, a selector–extractor framework that factorizes video recognition into learning when, where, and what to compute. Our shallow selector gets a scaled-down video and quickly scores all tokens across space-time, while our deep extractor gets the top-K selected tokens to approximate full-video representations without actually processing all the tokens. A key challenge is defining effective supervision for selection and extraction. For selection pre-training, we introduce a score on representations that ranks tokens by uniqueness using a simple nearest-neighbor distance. For extraction pre-training, we distill both a video teacher and an image teacher, for which we normalize its frame-wise representations to learn what changes within videos. Through these strategies, our selector-extractor learns general and efficient representations for feature extraction or fine-tuning to a task. Through experiments on Kinetics-400, SSv2, Epic-Kitchens, Diving48, Jester, and Charades, we show that LookWhen achieves a better accuracy-computation trade-off than efficient models and upgraded baselines of similar size. LookWhen Pareto-dominates in accuracy-FLOPs on 9 of 12 cases (6 tasks × 2 settings) and roughly matches on 3. In accuracy-throughput, measuring time in practice, LookWhen is more efficient still at 6.7× faster than InternVideo2-B at equal accuracy.

What LookWhen selects

16 frames of a video with the tokens selected by LookWhen shown bright and skipped tokens dimmed
frame 1 / 16

Skip the background. Keep what changes.

The selector scores every space-time token from a 2×-downscaled clip. Only the top-scoring tokens are sent at full resolution to the extractor. Everything else is never processed by the deep network.

bright = computed dimmed = skipped

Selection follows the action through time: the jump and splash, the swimmer, a diver mid-flight, a hand gesture. Tap a frame below the video to scrub.

6.7×
faster than InternVideo2-B
at equal mean accuracy
9/12
Pareto-dominant in accuracy vs. FLOPs
6 datasets × {linear probe, fine-tune}
10×
fewer FLOPs, higher linear-probe accuracy
than InternVideo2-B on SSv2, Jester & Diving48
+12.8
points over the best efficient baseline
Diving48 linear probe
How it works

Select cheaply, extract sparsely

A shallow selector decides when and where to compute. A deep extractor decides what to represent, from the selected tokens only.

Dense tokensdownscale 2×Token Selector3 layers · low-resSelector map×top-KSparse tokensFeature Extractordeep ViT-B · K tokens onlyregister + frame tokensVideo tokenFrame tokensPatch features Dense tokensdownscale 2×Token Selector3 layers · low-resSelector map×top-KSparse tokensFeature Extractordeep ViT-B · K tokens onlyregister +frame tokensVideo tokenFrame tokensPatch features LookWhen method overview: (a) inference and fine-tuning, (b) computing targets for pre-training

How is it trained? Two new pre-training targets

S Selector: predict top-1 distance

A token is worth computing if nothing else in the clip looks like it: its distance to its nearest neighbor in DINOv3 feature space, across all frames.

U(p) = 1 − maxq ≠ p cos( zp, zq )

E Extractor: distill two teachers

Predict InternVideo2's video token and DINOv3's tokens normalized over time, so it learns what changes in the video.

  Teachers are only used in pre-training.

Key insight

Why select by uniqueness, not attention?

LookWhere, the image version of this idea, trained its selector to copy a teacher's attention map. In video that signal breaks down, and selecting by uniqueness is more effective.

Results

More accuracy per FLOP, on six datasets

We compare against our video teacher InternVideo2-B (IV2) run on all tokens, and against two token-reduction methods, RLT and vid-TLDR, that we upgraded by applying them to IV2.

Linear probe: a linear classifier on frozen features, which measures feature quality. Fine-tune: train the whole model. LookWhen is pre-trained on Kinetics-400 + SSv2; the other four datasets are held out. LookWhen's points vary sparsity from 50% to 95%.

Show all 12 plots from the paper
Accuracy vs. inference FLOPs for linear probing and fine-tuning on six datasets

Against published ViT-B-sized video models

Top-1 accuracy GFLOPs per view (lower is better)
Ablations

Both new targets matter

Mean accuracy over the same six datasets (20K-sample training subsets, 90% sparsity). Each bar is a separate pre-training run.

Selector: what should it learn to pick?

Teacher · target

Uniqueness beats attention. Against a direct video port of LookWhere (InternVideo2 attention), top-1 distance gains +3.9 linear-probe and +7.2 fine-tune points on average. On Diving48 it reaches 83.2% fine-tuned vs. 57.2% for random selection.

Extractor: what should it learn to predict?

Targets removed from the full recipe

Two teachers beat one. Adding the time-normalized DINOv3 video target to InternVideo2's adds +3.0 points of mean linear-probe accuracy. Frame and patch targets add another +3.1, including +10.0 on Jester.
Show the per-dataset ablation tables
Table: ablating when and where to select
Table: ablating what to extract
Cite

BibTeX

@inproceedings{salamatian2026lookwhen,
  title     = {{LookWhen?} Fast Video Recognition by Learning When, Where, and What to Compute},
  author    = {Salamatian, Ali and Fuller, Anthony and Sarkar, Pritam and Green, James R. and Sigal, Leonid and Shelhamer, Evan},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}