Upserting Vectors

1. Generating Vector IDs

StrategyPros / Cons
UUID v4Random, no collisions; not human-readable
Content hashDeduplication built-in; expensive to compute
Source IDe.g., doc_42_chunk_3; traceable, requires source schema
Auto-incrementSequential; needs central counter
Note: IDs are strings up to 512 chars. Avoid spaces and special chars for portability.

2. Validating Vector Dimensions

Example: Validate Before Upsert

EXPECTED_DIM = 1536

def validate(vec):
    if len(vec) != EXPECTED_DIM:
        raise ValueError(f"dim {len(vec)} != {EXPECTED_DIM}")
    if not all(isinstance(x, (int, float)) for x in vec):
        raise TypeError("non-numeric value")
ErrorCause
400 Dimension mismatchVector length != index dim
400 Invalid typeNaN, Inf, or non-numeric value

3. Performing Basic Upsert

Example: Single Upsert

index = pc.Index("products")
index.upsert(vectors=[
    {"id": "p1", "values": [0.1, 0.2, ...]},
    {"id": "p2", "values": [0.3, 0.4, ...]},
])

Example: Node.js

const index = pc.index("products");
await index.upsert([
  { id: "p1", values: [0.1, 0.2] },
  { id: "p2", values: [0.3, 0.4] },
]);

4. Upserting with Metadata

Example: Metadata

index.upsert(vectors=[{
    "id": "doc1",
    "values": [...],
    "metadata": {
        "source": "wiki",
        "page": 12,
        "tags": ["ml", "vectors"],
        "published": True,
    },
}])
Metadata TypeSupported
stringYes
number (int/float)Yes
booleanYes
list of stringsYes
nested objectNo (flatten with dot notation)

5. Batch Upserting Vectors

Example: Batched Upsert

def chunks(lst, size):
    for i in range(0, len(lst), size):
        yield lst[i:i+size]

for batch in chunks(all_vectors, 100):
    index.upsert(vectors=batch)
LimitValue
Max batch size2 MB or 1000 vectors
Recommended100 vectors @ 1536d

6. Upserting to Namespace

Example: Namespaced Upsert

index.upsert(vectors=batch, namespace="tenant-a")
ParamBehavior
namespace="name"Creates if missing
namespace=""Default namespace

7. Optimizing Batch Size

DimRecommended Batch
384200–500
768200
1536100
307250
Note: Stay under 2 MB payload. Larger batches reduce overhead but risk timeouts.

8. Handling Upsert Failures

Example: Retry with Backoff

import time

for attempt in range(5):
    try:
        index.upsert(vectors=batch)
        break
    except Exception as e:
        if attempt == 4: raise
        time.sleep(2 ** attempt)
ErrorStrategy
429 Rate limitExponential backoff
413 Payload too largeReduce batch size
500/503Retry with backoff
400 Invalid inputFix data; don't retry

9. Using Async Upsert Operations

Example: Async Python

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=10) as pool:
    futures = [pool.submit(index.upsert, vectors=b) for b in batches]
    for f in futures: f.result()

Example: Node.js Parallel

await Promise.all(batches.map(b => index.upsert(b)));

10. Monitoring Upsert Throughput

MetricSource
Vectors/secClient-side timer
Index growthdescribe_index_stats()
ErrorsApplication log / Pinecone console
Latency p95Pinecone metrics dashboard