Migrating Data
1. Planning Migration Strategy
| Migration | Approach |
|---|---|
| Pod → Serverless | Collection or paginated query+upsert |
| Cross-region | Export → re-upsert in target |
| Other vector DB → Pinecone | Read all vectors → upsert |
2. Exporting Vectors
Example: Paginate by ID Prefix
all_ids = []
for ids in index.list(prefix="", namespace="kb"):
all_ids.extend(ids)
for chunk in batched(all_ids, 1000):
vecs = index.fetch(ids=chunk, namespace="kb")["vectors"]
write_to_file(vecs)3. Importing Vectors
| Method | Detail |
|---|---|
| Bulk Import (serverless) | S3/GCS Parquet → start_import |
| SDK upsert | Batched (100 vectors/batch) |
| CLI | pinecone import |
4. Migrating from Other Vector DBs
| Source | Steps |
|---|---|
| Weaviate | Cursor through objects → upsert |
| Qdrant | Scroll API → upsert |
| Chroma | collection.get() → upsert |
| pgvector | SQL SELECT → upsert |
5. Migrating Between Pinecone Regions
Cross-Region Migration
- Create index in target region.
- Export from source via fetch.
- Upsert to target in batches.
- Dual-write briefly.
- Switch reads; delete source.
6. Migrating from Pod to Serverless
Example: Via Collection
# Note: collections only work for pod indexes.
# Create collection from pod, then upsert into serverless from source data.
pc.create_collection(name="migrate", source="docs-pod")
# Use source data store (S3 parquet) to populate serverless via bulk import.7. Performing Zero-Downtime Migration
Zero-Downtime Steps
- Stand up new index.
- Backfill historical data.
- Enable dual-write (old + new).
- Diff sample queries old vs new.
- Shift reads progressively.
- Stop writes to old; delete.
8. Validating Migration Success
| Check | How |
|---|---|
| Counts match | describe_index_stats both sides |
| Query parity | Sample 100 queries; compare top-K |
| Metadata fidelity | Fetch & deep-compare |
9. Handling Migration Errors
| Error | Mitigation |
|---|---|
| Partial failure | Persist failed IDs; replay |
| Rate limit | Backoff; reduce concurrency |
| Dim mismatch | Re-embed source |
10. Rolling Back Migrations
| State | Action |
|---|---|
| Before cutover | Delete new index; nothing else |
| During dual-write | Switch reads back to old |
| After cutover | Restore old from backup/source |
11. Optimizing Migration Throughput
| Tip | Effect |
|---|---|
| Bulk import (serverless) | 10-100× faster than SDK upsert |
| Async client | Parallel batches |
| Multiple workers | Shard IDs by hash |
12. Documenting Migration Process
| Doc | Detail |
|---|---|
| Runbook | Step-by-step + rollback |
| Validation report | Counts + sample queries diff |
| Owners | Who to page if issues |