Working with Sparse Vectors

1. Understanding Sparse Vector Format

FieldTypeDescription
indicesList[int]Non-zero dimension positions
valuesList[float]Corresponding non-zero values
Length matchlen(indices) == len(values)

2. Creating Sparse Vector Objects

Example: Sparse Vector

sparse = {
    "indices": [10, 45, 102, 2048],
    "values": [0.5, 0.3, 0.8, 0.1],
}

3. Upserting Sparse Vectors

Example: Hybrid Upsert

index.upsert(vectors=[{
    "id": "doc1",
    "values": dense,           # required, even for sparse-heavy
    "sparse_values": sparse,
    "metadata": {"source": "wiki"},
}])
Warning: Sparse vectors require a dotproduct index. Cosine/Euclidean indexes reject sparse values.

4. Querying with Sparse Vectors

Example: Hybrid Query

res = index.query(
    vector=dense_q,
    sparse_vector={"indices": [...], "values": [...]},
    top_k=10,
)

5. Understanding Sparse Vector Use Cases

Use CaseWhy Sparse
Keyword/lexical searchExact term match
BM25 retrievalClassic IR scoring
SPLADELearned sparse embeddings
Domain jargonOut-of-vocab for dense models

6. Validating Sparse Vector Format

Example: Validate

def valid_sparse(s):
    return (
        isinstance(s.get("indices"), list)
        and isinstance(s.get("values"), list)
        and len(s["indices"]) == len(s["values"])
        and all(isinstance(i, int) and i >= 0 for i in s["indices"])
        and len(set(s["indices"])) == len(s["indices"])  # no dupes
    )
RuleDetail
Indices uniqueDuplicates rejected
Non-negative ints0-based positions
Same lengthindices and values

7. Optimizing Sparse Vector Size

TipEffect
Top-K term pruningKeep most-informative terms only
IDF thresholdDrop low-information terms
Avoid stopwordsReduce noise + size

8. Handling Sparse Vector Indexing

RequirementDetail
Metricdotproduct only
Pod typep1, p2, s1 supported
ServerlessSupported with dotproduct

9. Understanding Sparse Performance Considerations

FactorImpact
High term countSlower upsert/query
Hybrid (dense+sparse)~1.5–2× query latency
StorageAdds bytes per vector

10. Debugging Sparse Vector Errors

ErrorCause
Length mismatchindices != values
Duplicate indicesSame position twice
Wrong metricIndex not dotproduct
Empty sparseEmpty indices/values rejected