Most RAG stacks I audit still look like spaghetti: Whisper for audio, a vision model for images, a text embedder for docs, and custom glue to make search feel unified. It works until you need to query across modalities.
Google's answer is Gemini Embedding 2: one natively multimodal embedding model that maps text, images, video, audio, and PDFs into a single embedding space. It went GA in April 2026 on the Gemini API and Vertex AI.
What "one embedding space" actually buys you
An embedding model turns content into a vector. Similar meaning lands near similar vectors. That powers semantic search, clustering, recommendations, and retrieval for RAG.
Legacy multimodal setups chain specialists:
| Old pattern | Pain |
|---|---|
| Transcribe audio → embed text | Loses tone, timing, non-speech cues |
| Caption images → embed captions | Loses layout and fine visual detail |
| Separate indexes per modality | Cross-modal search needs brittle joins |
Gemini Embedding 2 embeds modalities directly and supports interleaved inputs (image + text in one request). You can search with a photo and retrieve a relevant video clip, or ground a chat agent on meeting audio without maintaining parallel pipelines.
Limits you need before you design
These caps shape architecture. Plan around them early.
| Modality | Limit |
|---|---|
| Text | 8,192 input tokens |
| Images | up to 6 per request (PNG, JPEG, WebP, …) |
| Video | up to 120s (80s if audio track extracted); 1 video per request |
| Audio | up to 180s; native embed (no forced transcription) |
| 1 file, up to 6 pages | |
| Output dimensions | 128–3,072 (MRL); Google recommends 768, 1,536, or 3,072 |
Video note from the docs: the API samples up to 32 frames. Short clips get 1 fps; longer clips get uniform sampling. Audio tracks inside video files are not processed unless you use audio_track_extraction.
Cross-modal RAG in practice
The primary use case Google highlights is multimodal RAG: one retrieval backbone for agents that query video libraries, meeting recordings, slide decks, and support docs together.
A pattern I would ship for a client ops team:
- Ingest mixed assets (Loom clips, PDF SOPs, Slack-exported threads, product screenshots) with
gemini-embedding-2. - Store vectors in your existing index (Weaviate, Qdrant, Chroma, Vertex Vector Search, etc.).
- Query with the user's actual input modality (text question, screenshot of an error, voice note from a field tech).
- Generate answers with your chat model, citing retrieved chunks.
Task-specific prompting matters. Google's docs cover retrieval query vs document embedding styles. Treat those instructions as part of your eval suite, not optional flavor text.
from google import genai from google.genai import types client = genai.Client() response = client.models.embed_content( model="gemini-embedding-2", contents=[ "Find tutorials about replacing a hydraulic filter", types.Part.from_bytes(data=image_bytes, mime_type="image/png"), ], config={"output_dimensionality": 768}, ) vector = response.embeddings[0].values
Dimensionality vs cost
Default output is 3,072 dimensions. Matryoshka Representation Learning (MRL) lets you truncate to 768 or 1,536 for storage and latency wins.
| Dimension | Trade-off |
|---|---|
| 3,072 | Highest accuracy, heavier storage |
| 1,536 | Balanced default for many prod indexes |
| 768 | Faster similarity search, smaller bills |
Benchmark on your data. Marketing tables are directionally useful; your ticket corpus, product catalog, or clinic intake forms are the real judge.
Where this fits my consulting work
Gemini Embedding 2 maps cleanly to Applied AI shipping projects:
- Support and ops: search across call recordings, screen recordings, and PDF runbooks without three embedders.
- Commerce: visual search plus spec sheets in one index.
- Compliance-heavy teams: fewer transcription steps means fewer places PHI or PII leaks if you architect retention correctly (still your job to get legal sign-off).
It does not replace judgment on chunking, access control, or eval. Embeddings are the easy part. Governance and refresh pipelines are where projects stall.
Useful links:
What I am testing this month
Before I recommend GE2 on a client RAG rebuild, I run four checks:
- Cross-modal recall: text query → correct image/video chunk on a labeled set of 50–100 items.
- Audio without transcripts: does native audio beat Whisper-then-embed on domain jargon?
- PDF tables: do six-page spec sheets retrieve the right row-level facts?
- Cost at 768 vs 1536 dims: latency and monthly index storage at realistic QPS.
The infrastructure layer is consolidating. One embedding model for every format is the direction Google, and your future self maintaining fewer pipelines, both want.
If you are planning a multimodal RAG rollout and want help scoping ingestion, eval, and production guardrails, book a free discovery call.