Google DeepMind Launches Agentic Video Understanding for Gemini

Instead of decoding a video, sampling frames at a fixed rate and pushing the resulting multimodal tokens into the context window, Gemini can dynamically search frames, audio and transcripts, retrieve relevant temporal segments and increase sampling density when necessary.

Google reports:

• Up to 88% fewer tokens
• Up to 66% lower cost
• Up to 7% higher accuracy
• Sub-second event retrieval
• Improved anomaly detection
• More accurate counting

The feature is available on Gemini 3.7 Flash, Gemini 3.6 Flash and Gemini 3.5 Flash-Lite.

From static sampling to agentic retrieval

Standard Gemini video processing samples frames at roughly 1 FPS.

Google estimates approximately:

~100 tokens/sec at low media resolution~300 tokens/sec at high media resolution

For one hour of video, that's roughly:

360,000 tokens at low resolution1.08 million tokens at high resolution

Agentic processing changes the pipeline from:

Video → Frame Sampling → Tokenization → Context → Inference

To:

Video → Search → Temporal Retrieval → Adaptive Sampling → Inference

Gemini retrieves only the media required for the current query instead of loading a fixed representation of the entire video into context.

Adaptive frame sampling

Fixed 1 FPS sampling creates a temporal-resolution problem. An event lasting 200–300 ms can occur entirely between sampled frames. Increasing the entire video to 10 or 30 FPS solves the resolution problem but dramatically increases multimodal token consumption. Agentic Video Understanding instead uses adaptive temporal sampling.

Gemini can first locate a candidate temporal window and then resample that section at higher FPS:

coarse search → temporal localization → high-FPS inspection → result

This enables sub-second retrieval without processing the entire video at high temporal resolution.

Agentic inference loop

The model can dynamically retrieve:

• frames;
• audio;
• transcripts;
• timestamps;
• specific temporal windows;
• higher-resolution samples.

The resulting inference loop looks more like:

Query → Reasoning → Retrieval → Inspection → Reasoning → Response

If the available evidence is insufficient, Gemini can perform another retrieval step. This effectively turns video understanding into an agentic multimodal retrieval system.

Better anomaly detection and counting

Adaptive sampling is particularly useful for workloads where important information is temporally sparse.

1. For anomaly detection:

coarse scan → anomaly candidate → temporal localization → high-resolution inspection

2. For counting: 

candidate segment → increased FPS → event detection → count

Instead of running expensive high-resolution inference across an entire recording, additional compute is allocated only to relevant temporal regions.

This architecture is useful for industrial monitoring, surveillance, sports analytics, robotics, video QA and automated media analysis.

Why token usage drops

• With static processing, token consumption is largely determined by: video duration × sampling rate × media resolution

• With agentic processing, it becomes closer to: query complexity × retrieved temporal coverage × sampling density

That's the reason Google can report token reductions of up to 88% on long-form video. For large-scale video workloads, this directly affects inference cost, throughput, context utilization and scalability.

Video as external multimodal memory

The architectural change is similar to RAG.

Traditional video processing effectively does: Video → Context Window

Agentic processing does: Video → Retrieval → Relevant Context

The full video remains outside the active context and Gemini retrieves the required evidence during inference. This reduces context-window pressure and avoids spending tokens on irrelevant frames.

Tool-driven media processing

Agentic execution can expose operations such as:

• processing_call and processing_result

A processing_call requests additional information from the video-processing layer. The retrieved media is returned through processing_result, after which Gemini continues inference.

Conceptually:

• Gemini → processing_call → Video Processing Layer → processing_result → Gemini → Response

• This creates an iterative LLM → tool → multimodal data → LLM execution loop rather than a single-pass inference pipeline.

Static vs. agentic processing

Static processing still makes sense for short videos where most of the content is relevant and Time to First Token (TTFT) is important.

Agentic processing introduces additional orchestration and retrieval steps, so it can add latency before generation begins. For long videos and sparse queries, however, the reduction in unnecessary media processing can outweigh that overhead. The two modes can also be configured independently for different videos inside the same multimodal request.

API

Agentic Video Understanding does not require a separate model.

1. With the Interactions API: processing: "agentic"

2. With GenerateContent: media_processing: "AGENTIC"

3. Supported models: Gemini 3.7 FlashGemini 3.6 FlashGemini 3.5 Flash-Lite

There is no separate fee for agentic processing. Standard Gemini API token pricing applies.

What actually changed

The important change isn't simply better video understanding. Google has moved Gemini from fixed multimodal ingestion toward adaptive inference-time retrieval.

Instead of processing every frame equally, the model determines:

1. What data to retrieve.
2. Which temporal regions matter.
3. When higher FPS is required.
4. How much context to consume.

The result is effectively retrieval-augmented, adaptive multimodal inference for video — with up to 88% lower token consumption and 66% lower processing cost.