Implementing RAG Applications
1. Understanding RAG Architecture
User question
↓
Embed query ──→ Pinecone query (top-K)
↓
Retrieved context chunks
↓
LLM prompt = [system] + context + question
↓
LLM response
| Stage | Component |
|---|---|
| Ingest | Chunk → embed → upsert |
| Retrieve | Embed query → Pinecone |
| Augment | Inject chunks into prompt |
| Generate | LLM produces grounded answer |
2. Chunking Documents
Example: Token-Based Chunker
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
def chunk(text, size=400, overlap=50):
tokens = enc.encode(text)
out = []
for i in range(0, len(tokens), size - overlap):
out.append(enc.decode(tokens[i:i+size]))
return out| Strategy | Use Case |
|---|---|
| Fixed-size tokens | General text |
| Sentence/paragraph | Preserve semantic units |
| Semantic chunking | LLM-based boundary detection |
| Recursive | LangChain RecursiveCharacterTextSplitter |
3. Generating Embeddings
Example: Embed Chunks
chunks = chunk(document_text)
embs = embed_batch(chunks) # OpenAI / Pinecone Inference4. Upserting Document Chunks
Example: Chunks → Pinecone
vectors = [{
"id": f"{doc_id}_chunk_{i}",
"values": emb,
"metadata": {"doc_id": doc_id, "chunk": i, "text": text},
} for i, (text, emb) in enumerate(zip(chunks, embs))]
index.upsert(vectors=vectors, namespace="kb")5. Implementing Semantic Search
| Param | Recommendation |
|---|---|
| top_k | 5–10 for typical RAG |
| include_metadata | True (need chunk text) |
| Filter | Tenant ID, access controls |
6. Retrieving Relevant Context
Example: Retrieve
res = index.query(vector=embed(query), top_k=8,
include_metadata=True, namespace="kb")
context = "\n\n".join(m["metadata"]["text"] for m in res["matches"])7. Constructing LLM Prompts
Example: Prompt Template
prompt = f"""Answer using only the context below. If unknown, say so.
Context:
{context}
Question: {query}
Answer:"""
ans = llm.complete(prompt)8. Implementing Reranking
Example: Pinecone Rerank
candidates = [m["metadata"]["text"] for m in res["matches"]]
r = pc.inference.rerank(
model="pinecone-rerank-v0",
query=query,
documents=candidates,
top_n=5,
)
top = [candidates[d["index"]] for d in r.data]| Reranker | Provider |
|---|---|
| pinecone-rerank-v0 | Pinecone Inference |
| bge-reranker-v2-m3 | HuggingFace |
| Cohere rerank-v3 | Cohere API |
9. Using Metadata Filtering
Example: Filter for ACL
res = index.query(vector=qv, top_k=8,
filter={"$and": [
{"tenant_id": user.tenant_id},
{"visibility": {"$in": ["public", user.role]}},
]})10. Handling Multi-Query Retrieval
Example: Query Expansion
variants = llm.expand(query, n=3) # ask LLM for 3 paraphrases
all_matches = []
for v in [query] + variants:
all_matches += index.query(vector=embed(v), top_k=5)["matches"]
# dedupe by id, sort by best score
unique = {m["id"]: m for m in all_matches}.values()11. Implementing Hybrid RAG
| Step | Detail |
|---|---|
| Index metric | dotproduct |
| Per chunk | Dense + sparse vectors |
| Query | Weighted dense+sparse |
| Win | Better on jargon/IDs vs pure dense |
12. Optimizing Retrieval Quality
| Lever | Effect |
|---|---|
| Smaller chunks (200-400) | More precise retrieval |
| Overlap (10-20%) | Don't miss context at boundary |
| Reranker | +5–15% recall@5 |
| Hybrid search | Better on exact terms |
| Query rewriting | LLM cleans ambiguous queries |