Embedding on KnightLi Blog

Gemini Embedding 2: Putting Text, Images, Video, and Audio in One Vector Space

Mon, 04 May 2026 06:01:10 +0800

Google Developers Blog introduced how to build with Gemini Embedding 2. The model is now generally available through the Gemini API and Gemini Enterprise Agent Platform. The important point is not simply that it is a new embedding model, but that it maps text, images, video, audio, and documents into the same semantic space.

This broadens what retrieval systems can handle. Many RAG pipelines previously had to convert images, video, or audio into text or metadata before indexing them separately. Gemini Embedding 2 can process multimodal inputs directly, making it easier for agents, search systems, and classifiers to work with real business materials.

Original article: Building with Gemini Embedding 2: Agentic multimodal RAG and beyond

Model Capabilities

Gemini Embedding 2 supports more than 100 languages. A single request can process:

Up to 8,192 text tokens
Up to 6 images
Up to 120 seconds of video
Up to 180 seconds of audio
Up to 6 pages of PDF

Its key idea is a unified semantic space. Developers can place content from different modalities into one vector representation system, then use the same retrieval, clustering, or reranking logic to process it.

For example, a text description and an image can be included in the same embedding request:

from google import genai
from google.genai import types

client = genai.Client()

with open('dog.png', 'rb') as f:
    image_bytes = f.read()
result = client.models.embed_content(
    model='gemini-embedding-2',
    contents=[
        "An image of a dog",
        types.Part.from_bytes(
            data=image_bytes,
            mime_type='image/png',
        ),
    ]
)

print(result.embeddings)

If you want a separate embedding for each input rather than one aggregated vector, you can use the Batch API. The original article also notes that Agent Platform support for this kind of batch workflow is still in progress.

What It Means for RAG

Multimodal embeddings are useful for agentic RAG. An AI agent may need to inspect a code repository, PDFs, screenshots, charts, audio meeting notes, and product images at the same time. If all of these materials can enter the same semantic space, the retrieval pipeline no longer needs a separate entry point for every format.

Google recommends using task prefixes according to the goal of the task, so the embeddings better match the retrieval objective. For example, question answering, fact checking, code retrieval, and search results can use different prefixes:

# Generate embedding for your task's query:
def prepare_query(query):
    return f"task: question answering | query: {content}"
    # return f"task: fact checking | query: {content}"
    # return f"task: code retrieval | query: {content}"
    # return f"task: search result | query: {content}"
# Generate embedding for document of an asymmetric retrieval task:
def prepare_document(content, title=None):
    if title is None:
        title = "none"
    return f"title: {title} | text: {content}"

This kind of prefix is suitable for asymmetric retrieval: user queries are often short, while documents are often long. Formatting query and document differently for the task can improve matching between short queries and long documents.

The original article gives two real-world examples:

Harvey saw a 3% increase in Recall@20 precision on legal retrieval benchmarks compared with its previous embeddings.
Supermemory saw a 40% increase in Recall@1 search accuracy and uses it across memory, indexing, search, and Q&A pipelines.

These numbers do not mean every scenario will improve by the same amount, but they show that multimodal embeddings are already producing results in real retrieval products, not only demos.

Visual Search

Gemini Embedding 2 is also suitable for image-to-image search, image-text hybrid search, and product identification. The original article mentions Nuuly, URBN’s clothing rental company, using it to match photos of untagged garments in warehouses against its catalog. Match@20 improved from 60% to nearly 87%, and the overall successful identification rate rose from 74% to over 90%.

The point in this type of scenario is not content generation, but understanding which inventory item, document, or product record is closest to a given image. If your business has many images, video clips, or scanned documents, multimodal embeddings can be more natural than text-only indexing.

Search Reranking

Embeddings can also be used for reranking. A common approach is to first retrieve a set of candidate results, then calculate the similarity between each candidate and the user’s query, pushing more relevant content to the top:

# 1. Define a function to calculate the dot product (cosine similarity)
def dot_product(a: np.ndarray, b: np.ndarray):
  return (np.array(a) @ np.array(b).T)
# 2. Retrieve your embeddings
# (Assuming 'summaries' is your list of search results)
search_res = get_embeddings(summaries)
embedded_query = get_embeddings([query])

# 3. Calculate similarity scores
sim_value = dot_product(search_res, embedded_query)

# 4. Select the most relevant result
best_match_index = np.argmax(sim_value)

The original article also mentions another idea: first ask the model to generate a baseline hypothetical answer from its internal knowledge, embed that answer, and compare it with candidate content to find the most semantically relevant result. This is especially useful for Q&A-style RAG.

Clustering, Classification, and Anomaly Detection

Beyond retrieval, embeddings are also useful for clustering, classification, and anomaly detection. Unlike the asymmetric question-answering retrieval above, these are symmetric tasks, where the same task prefix can be used for both query and document:

# Generate embedding for query & document of your task.
def prepare_query_and_document(content):
    # return f'task: clustering | query: {content}'
    # return f'task: sentence similarity | query: {content}'
    # return f'task: classification | query: {content}'

These tasks can be used for sentiment classification, content moderation, similar asset grouping, and anomaly discovery. They can also help agents organize large amounts of context before moving into later reasoning steps.

Storage and Cost

Gemini Embedding 2 outputs 3,072-dimensional vectors by default. It uses Matryoshka Representation Learning, so vectors can be truncated to smaller dimensions with output_dimensionality. Google recommends 1,536 or 768 dimensions when efficiency is the priority:

result = client.models.embed_content(
    model="gemini-embedding-2",
    contents="What is the meaning of life?",
    config={"output_dimensionality": 768}
)

Vectors can be stored in Agent Platform Vector Search, Pinecone, Weaviate, Qdrant, ChromaDB, and similar systems. For cost, the original article notes that the Batch API provides higher throughput and reaches 50% of the default embedding price.

How Developers Can Use It

If you already have text-based RAG, you can start with two incremental upgrades:

Put PDFs, screenshots, image descriptions, and text documents into the same index, then test whether retrieval recall becomes more stable.
Add task prefixes for different tasks, such as question answering, fact checking, code retrieval, and product search. Do not process all content with the same embedding format.

If you are building a new product, consider these directions first:

Enterprise knowledge bases: retrieve documents, charts, presentation screenshots, and meeting materials together.
Visual search: use images, text, or mixed inputs to find products, assets, design drafts, and archives.
Agent toolchains: let coding agents, research agents, or customer support agents retrieve business materials in multiple formats.
Content governance: classify, cluster, and detect anomalies across text, images, and video clips.

The value of Gemini Embedding 2 is that it turns multimodal materials into one searchable asset system. For developers, this reduces the need for an intermediate “convert to text, then retrieve” layer and makes RAG systems closer to the shape of real-world data.

How to Choose Common Embedding Models: OpenAI vs BGE vs E5 vs GTE vs Jina

Thu, 23 Apr 2026 15:23:47 +0800

When people start building RAG systems, semantic search, or knowledge base retrieval, they often get stuck on the same question: there are so many embedding models, so which one should you choose?

Common options can roughly be split into two groups. One group is general-purpose text embeddings that cover Chinese, English, and multilingual tasks. The other group is better suited to Chinese scenarios, especially Chinese retrieval, Chinese QA, and Chinese knowledge bases.

If you want the short version first, this is a practical way to think about it:

If you want the easiest path and prefer using an API directly: text-embedding-3-small or text-embedding-3-large
If you want Chinese retrieval and prefer open-source models you can self-host: bge-base-zh-v1.5, bge-m3, gte-large-zh
If you need multilingual support: multilingual-e5-base, multilingual-e5-large, jina-embeddings-v3
If you want to keep costs down in Chinese scenarios: bge-small-zh-v1.5, gte-base-zh

1. First, Look at Them by Category

1. OpenAI Series

text-embedding-3-small
text-embedding-3-large

The main strengths of these models are simplicity and stability. They are a good fit if you want to call an API directly for retrieval, RAG, classification, and similarity matching. Their advantage is not that they dominate one specific Chinese leaderboard, but that the overall experience is complete: low integration cost, stable quality, and low engineering overhead.

If your team does not want to host models or maintain inference services, OpenAI is usually the most time-saving option.

2. BGE Series

BAAI/bge-small-zh-v1.5
BAAI/bge-base-zh-v1.5
bge-m3

BGE is one of the most common families used in Chinese retrieval. bge-small-zh-v1.5 and bge-base-zh-v1.5 lean more toward Chinese monolingual tasks, making them suitable for Chinese semantic search, knowledge base retrieval, and FAQ matching. bge-m3 is more general-purpose and can cover multilingual, multi-granularity, and more complex retrieval scenarios.

If most of your data is Chinese text, BGE is often one of the easiest families to put on the shortlist.

3. E5 Series

intfloat/multilingual-e5-base
multilingual-e5-large

The strength of the E5 family is more balanced multilingual capability. It works well for mixed Chinese-English data, cross-lingual retrieval, and internationalized content libraries. It is not focused only on Chinese. Instead, it is built around the idea that different languages can live inside one unified retrieval system.

If your corpus is not purely Chinese, but a mix of Chinese, English, Japanese, or even more languages, E5 is usually more reliable than a Chinese-only model.

4. GTE Series

Alibaba-NLP/gte-base-zh
gte-large-zh

GTE is also common in Chinese tasks. Its positioning is similar to BGE: both are practical choices for Chinese retrieval. GTE is usually seen as balanced and easy to use, without much complexity in deployment. It works well for Chinese knowledge bases, site search, and enterprise internal document retrieval.

If you want one more open-source Chinese model family for side-by-side evaluation, GTE is well worth testing.

5. Jina Embeddings

jina-embeddings-v3

Jina is more oriented toward general-purpose and modern engineering scenarios, and often appears in multilingual retrieval, long-text processing, and web content tasks. It is frequently mentioned in discussions around using a single model to cover more task types, so it is a good fit for teams that want one unified embedding layer.

If your content sources are mixed, such as webpages, documents, and multilingual text, Jina is often a strong candidate to test.

2. Which Models Are Most Common in Chinese Scenarios

If we narrow the scope to Chinese use cases, the usual candidates are basically these:

bge-small-zh-v1.5
bge-base-zh-v1.5
bge-m3
gte-base-zh
gte-large-zh
multilingual-e5-base
multilingual-e5-large

Among them, the most useful split is not really “which one is absolutely better,” but these three questions:

Is your data primarily Chinese?
Do you need multilingual support?
Do you care more about quality, cost, or deployment convenience?

3. Put These Models Side by Side

1. If You Only Care About Chinese Performance

For pure Chinese knowledge bases, Chinese QA, and Chinese document retrieval, BGE and GTE are usually the first families to check.

bge-small-zh-v1.5: lighter and better for cost-sensitive scenarios
bge-base-zh-v1.5: usually one of the most balanced options for Chinese use cases
gte-base-zh: similar to lightweight BGE and good for building a baseline first
gte-large-zh: better when retrieval quality matters more
bge-m3: suitable if you want to evaluate Chinese retrieval together with broader capabilities

If your corpus is almost entirely Chinese, E5 can still work, but it often will not be the first priority.

2. If You Need Multilingual Support

The priorities change quite a bit here.

multilingual-e5-base and multilingual-e5-large are better suited to unified multilingual retrieval
jina-embeddings-v3 also fits multilingual and general text tasks
bge-m3 is better than traditional Chinese-only models when you want to expand into multilingual usage
text-embedding-3-small and text-embedding-3-large are good if you want the simplest API-based route

If your corpus contains Chinese, English, product documentation, website copy, and user questions at the same time, multilingual models can save you a lot of future migration work.

3. If You Need to Control Inference and Storage Cost

Lightweight models have the advantage here.

bge-small-zh-v1.5
gte-base-zh
multilingual-e5-base
text-embedding-3-small

These models are usually a better fit when:

You have a large document volume
Data is updated frequently
You need batch vectorization
You are sensitive to latency and cost

If your dataset is large, embedding dimensions, inference speed, and index size will all directly affect total cost. That is why starting with a smaller model as a baseline is often the safer choice.

4. If You Want the Highest Ceiling First

Larger models are usually better suited to complex retrieval or higher-quality recall, for example:

text-embedding-3-large
multilingual-e5-large
gte-large-zh
bge-base-zh-v1.5
bge-m3

But one thing to keep in mind is that a larger model does not automatically lead to a better production experience. In many projects, the real bottleneck is not the model itself, but chunking strategy, recall count, reranking, data cleaning, and evaluation design.

4. What Each Model Is Better At

Model	Better suited for	Quick judgment
`text-embedding-3-small`	General retrieval, RAG, fast integration	Simple API usage and cost-friendly
`text-embedding-3-large`	General retrieval where quality matters more	Quality-first and lowest engineering burden
`bge-small-zh-v1.5`	Lightweight Chinese retrieval	A common entry-level Chinese option
`bge-base-zh-v1.5`	Chinese knowledge bases, FAQ, semantic search	Very balanced in Chinese scenarios
`bge-m3`	Chinese-focused setups that also need more complex retrieval	More extensible
`multilingual-e5-base`	Foundational multilingual retrieval	Common in international products
`multilingual-e5-large`	High-quality multilingual recall	More quality-oriented
`gte-base-zh`	Lightweight Chinese retrieval	Good for building a baseline
`gte-large-zh`	Chinese scenarios that prioritize quality	A good comparison point against BGE
`jina-embeddings-v3`	Multilingual, web, and general text tasks	Worth testing if you want one unified embedding layer

5. A Practical Way to Make the Choice

If you are trying to ship a system rather than write a paper, you can keep the decision process simple.

Scenario 1: Chinese Knowledge Base

Start with these:

bge-base-zh-v1.5
gte-large-zh
bge-small-zh-v1.5

If budget is tight, start from the smaller model first. If retrieval quality matters more, then move upward to larger models.

Scenario 2: Mixed Chinese-English Knowledge Base

Start with:

multilingual-e5-base
multilingual-e5-large
text-embedding-3-small
text-embedding-3-large

If you do not want to self-host, OpenAI is the more direct option. If you want to host the model yourself, E5 is the more common path.

Scenario 3: Mostly Chinese Now, but Possibly Multilingual Later

Start with:

bge-m3
multilingual-e5-base
jina-embeddings-v3

The biggest risk in this kind of setup is optimizing only for Chinese at the beginning and then having to rebuild the whole vector database later.

6. In the End, the Key Is Not “Top of the Leaderboard”

The easiest mistake in embedding model selection is to look only at public benchmark scores and then ship directly to production.

A more reliable process is usually:

Pick 2 to 4 candidate models first
Run embeddings on your own real data
Evaluate one round of retrieval performance
Then make the final decision based on cost, latency, and deployment style

Because in practice, what really determines the result is often not the model name itself, but whether the model matches your corpus, chunking strategy, and query patterns.

Summary

If you only want one practical summary to remember, use this:

Chinese-first: start with bge-base-zh-v1.5 and gte-large-zh
Cost-first: start with bge-small-zh-v1.5, gte-base-zh, and text-embedding-3-small
Multilingual-first: start with multilingual-e5-base, multilingual-e5-large, and jina-embeddings-v3
API-first: start with text-embedding-3-small and text-embedding-3-large
If you want Chinese now and flexibility later: start with bge-m3

There is no single model that fits every project, but for most projects, you can quickly narrow down the first batch of candidates from these few groups.