Vector Database Migration

Overview

Vector database migration involves moving embedding data between systems or upgrading versions while minimizing downtime and ensuring data integrity.

Migration Scenarios

Platform Migration

• Pinecone → Qdrant
• Milvus → Weaviate
• Self-hosted → Cloud
• Cloud → Self-hosted

Version Upgrades

• Milvus 2.3 → 2.4
• Schema changes
• Index algorithm updates

Migration Strategies

Export/Import

# Export from source
source_data = []
for batch in source_db.scroll(batch_size=1000):
    source_data.extend(batch)

# Import to destination
for batch in chunks(source_data, 1000):
    dest_db.upsert(batch)

Dual-Write Pattern

# Write to both databases
def index_vector(vector, metadata):
    old_db.insert(vector, metadata)
    new_db.insert(vector, metadata)

# Migrate historical data
migrate_historical_data(old_db, new_db)

# Cutover when migration complete
cutover_to_new_db()

Snapshot-Based

1. Take snapshot of source
2. Import snapshot to destination
3. Sync incremental changes
4. Cutover

Zero-Downtime Migration

1. Dual-write: Start writing to both DBs
2. Backfill: Migrate historical data
3. Validation: Compare results
4. Cutover: Switch reads to new DB
5. Monitor: Verify in production
6. Cleanup: Remove old DB

Validation

def validate_migration():
    # Sample random vectors
    samples = old_db.sample(n=1000)
    
    for vector_id in samples:
        # Compare vectors
        old_vec = old_db.get(vector_id)
        new_vec = new_db.get(vector_id)
        
        assert np.allclose(old_vec, new_vec)
        
        # Compare search results
        old_results = old_db.search(old_vec, k=10)
        new_results = new_db.search(new_vec, k=10)
        
        overlap = len(set(old_results) & set(new_results))
        assert overlap >= 8  # 80% overlap threshold

Tools

• Qdrant: Built-in migration from Pinecone, Weaviate, Milvus
• Custom scripts using SDKs
• Data pipeline tools (Airbyte, Fivetran)

Challenges

• Large data volumes (billions of vectors)
• Downtime requirements
• Schema differences
• Index parameter tuning
• Cost of dual-running

Best Practices

1. Test First: Run migration on subset
2. Validate Thoroughly: Compare search results
3. Monitor Closely: Watch metrics during cutover
4. Have Rollback Plan: Quick reversal if issues
5. Document Changes: Index parameters, schema changes

Pricing

Temporary dual-running costs + migration tool costs.