Implementing Hybrid Search
1. Understanding Hybrid Search
Query
├── Dense embed ──→ semantic match (meaning)
└── Sparse embed ─→ lexical match (keywords)
↓
weighted sum (alpha)
↓
top-K results
| Component | Provides |
|---|---|
| Dense | Semantic similarity |
| Sparse | Term/keyword relevance |
| Alpha | Weight between two signals |
2. Setting Up Hybrid-Enabled Indexes
Example: Create Hybrid Index
pc.create_index(
name="hybrid-docs",
dimension=1536,
metric="dotproduct", # required
spec=ServerlessSpec(cloud="aws", region="us-east-1"),
)3. Generating Dense Embeddings
Example: OpenAI Dense
from openai import OpenAI
client = OpenAI()
resp = client.embeddings.create(
model="text-embedding-3-small",
input="vector databases",
)
dense = resp.data[0].embedding # len=1536| Provider | Model | Dim |
|---|---|---|
| OpenAI | text-embedding-3-small | 1536 |
| Cohere | embed-v3 | 1024 |
| Pinecone | multilingual-e5-large | 1024 |
4. Generating Sparse Embeddings
Example: BM25 Sparse
from pinecone_text.sparse import BM25Encoder
bm25 = BM25Encoder().fit(corpus)
sparse = bm25.encode_queries("vector databases")
# {"indices": [...], "values": [...]}5. Combining Query Vectors
Example: Weighted Combination
def weight(dense, sparse, alpha):
# alpha=1: pure dense; alpha=0: pure sparse
hdense = [v * alpha for v in dense]
hsparse = {
"indices": sparse["indices"],
"values": [v * (1 - alpha) for v in sparse["values"]],
}
return hdense, hsparse
dq, sq = weight(dense, sparse, alpha=0.7)
res = index.query(vector=dq, sparse_vector=sq, top_k=10)6. Tuning Alpha Parameter
| Alpha | Behavior |
|---|---|
| 1.0 | Pure semantic (dense only) |
| 0.7–0.8 | Default for general RAG |
| 0.5 | Balanced |
| 0.2–0.3 | Keyword-heavy (code, IDs, jargon) |
| 0.0 | Pure lexical (sparse only) |
7. Upserting Hybrid Vectors
Example: Hybrid Upsert
index.upsert(vectors=[{
"id": "doc1",
"values": dense,
"sparse_values": sparse_doc,
"metadata": {"text": "..."},
}])8. Understanding Hybrid Scoring Mechanism
| Step | Formula |
|---|---|
| Combined score | score = dot(d_q, d_v) + dot(s_q, s_v) |
| Alpha pre-scaling | Applied client-side to query vectors |
9. Implementing BM25 Integration
Example: Fit + Encode
from pinecone_text.sparse import BM25Encoder
bm25 = BM25Encoder()
bm25.fit(corpus)
bm25.dump("bm25_params.json")
# later: BM25Encoder().load("bm25_params.json")
doc_sparse = bm25.encode_documents(["doc 1 text", "doc 2 text"])
q_sparse = bm25.encode_queries("query text")10. Using SPLADE Models
Example: SPLADE
from pinecone_text.sparse import SpladeEncoder
splade = SpladeEncoder()
sparse = splade.encode_queries("deep learning")| Model | Pro | Con |
|---|---|---|
| BM25 | Fast, no GPU | No semantic expansion |
| SPLADE | Learned term expansion | Slower, needs GPU |
11. Optimizing Hybrid Performance
| Tip | Effect |
|---|---|
| Cache sparse encoder | Avoid repeated init |
| Prune sparse | Keep top-100 terms |
| Co-locate encoder + index | Lower RTT |
| Batch encode | GPU utilization |
12. Benchmarking Hybrid vs Dense-Only
| Workload | Best Approach |
|---|---|
| General Q&A RAG | Dense or hybrid (alpha=0.7) |
| Code/jargon search | Hybrid (alpha=0.3–0.5) |
| Exact identifier lookup | Sparse-heavy (alpha=0.2) |
| Multilingual | Dense (sparse term mismatch) |