Understanding Distance Metrics
1. Using Cosine Similarity
| Property | Value |
|---|
| Formula | cos(θ) = (A·B) / (|A| × |B|) |
| Range | [-1, 1]; 1=identical, 0=orthogonal, -1=opposite |
| Magnitude | Ignored (length-invariant) |
| Best for | Text embeddings (most LLM models) |
Example: Create Index with Cosine
from pinecone import ServerlessSpec
pc.create_index(
name="docs",
dimension=1536,
metric="cosine",
spec=ServerlessSpec(cloud="aws", region="us-east-1"),
)
2. Using Euclidean Distance
| Property | Value |
|---|
| Formula | d = √Σ(Aᵢ − Bᵢ)² |
| Range | [0, ∞); 0=identical (lower=closer) |
| Magnitude | Sensitive to vector length |
| Best for | Image features, spatial coords, raw numerical data |
3. Using Dot Product
| Property | Value |
|---|
| Formula | A·B = Σ Aᵢ × Bᵢ |
| Range | (-∞, ∞); higher=more similar |
| Magnitude | Sensitive (favors longer vectors) |
| Best for | Hybrid search (required), recommenders, MIPS |
4. Choosing Appropriate Metric
Use case?
├── Text / semantic similarity ─→ cosine
├── Hybrid (dense + sparse) ────→ dotproduct (required)
├── Recommenders (MIPS) ────────→ dotproduct
├── Image / spatial features ───→ euclidean
└── Normalized embeddings ──────→ cosine ≡ dotproduct
| Workload | Recommended Metric |
|---|
| RAG / semantic search | cosine |
| Hybrid search | dotproduct |
| Recommendation | dotproduct |
| Image similarity | cosine or euclidean |
5. Understanding Score Interpretation
| Metric | Higher Score Means | Typical "Good" Threshold |
|---|
cosine | More similar | > 0.75 |
dotproduct | More similar | Domain-specific |
euclidean | API returns sorted best-first | < 0.5 (normalized) |
Warning: Pinecone always returns scores sorted best-first regardless of metric. The raw score field reflects the underlying metric semantics.
6. Normalizing Vectors for Cosine Similarity
Example: L2 Normalization
import numpy as np
def normalize(v):
norm = np.linalg.norm(v)
return (v / norm).tolist() if norm > 0 else v
vec = normalize([0.3, 0.8, 0.5]) # |vec| = 1
| Reason | Benefit |
|---|
| Cosine ≡ dotproduct on unit vectors | Faster compute, same results |
| Stable scoring | Removes magnitude bias |
7. Converting Between Metrics
| From → To | Conversion |
|---|
| cosine ↔ dotproduct | Normalize vectors to unit length |
| euclidean → cosine | cos = 1 − d² / 2 (on unit vectors) |
| dotproduct → cosine | cos = dot / (|A|·|B|) |
| Metric | Compute Cost | Index Build |
|---|
| dotproduct | Lowest (multiply-add) | Fastest |
| cosine | Low (norm cached) | Fast |
| euclidean | Medium (sqrt) | Standard |
9. Handling Negative Dot Product Scores
Note: Dot product can be negative. Filter score > threshold, or normalize vectors to bound scores in [-1, 1].
| Strategy | When |
|---|
| Filter by min score | Quality control on results |
| Normalize vectors | Need bounded, comparable scores |
| Shift scores | UI: (score + 1) / 2 |
10. Optimizing for Specific Metrics
| Optimization | Applies To |
|---|
| Pre-normalize at upsert | cosine, dotproduct |
| Reduce dimensions (PCA) | All metrics — faster query |
| Quantization (int8) | Storage savings, minor accuracy loss |
| Match metric to model | Use embedding provider's recommendation |