Vector Bible

Standardized benchmark for QPS/latency/recall tests on ANN libraries using datasets like SIFT1M and Deep1B to compare throughput and accuracy. Features metrics for build time, memory usage across HNSW, FAISS, ScaNN. Used for vector DB index selection during development; contrasts with BigANN billion-scale competitions by focusing on million-scale library performance vs full-system custom benchmarks.

Massive Text Embedding Benchmark leaderboard covering 58 datasets across 112 languages and 8 embedding tasks. Industry-standard benchmark for comparing text embedding models.

Comprehensive benchmark dataset comparing 10 vector databases across 19 fields including query latency (p50/p99), throughput, scalability limits, features like hybrid search and ACID compliance, SDK support, and managed pricing. Tested with 1M vectors at 1536 dimensions for RAG and AI search applications. Key highlights include Qdrant for lowest latency, Pinecone for managed scalability, and pgvector for ACID transactions.

Zero-shot benchmark for embedding model evaluation on 18 diverse datasets with NDCG@10 and Recall@100 metrics correlating to vector DB QPS/latency in production. Features heterogeneous tasks like QA, fact-checking, biomedical retrieval for robust comparisons. Use cases include selecting embeddings for RAG pipelines in vector DBs; complements ANN-Benchmarks indexing focus with retrieval task evaluation, differs from VectorDBBench full-DB tests.

BenchmarkQED standardizes QPS/latency/accuracy evaluations for RAG pipelines including vector DB retrieval on diverse datasets. Features comparable methodologies for fair benchmarking of full RAG stacks. Essential for selecting production vector DBs in RAG; emphasizes retrieval fairness unlike ANN-Benchmarks indexing focus or VectorDBBench system-level throughput tests.

Evaluates ANN algorithms on billion-scale datasets with QPS/latency/recall metrics via NeurIPS tracks for out-of-distribution and streaming tests. Features standardized billion-point evaluation for throughput and memory. For production vector DB scalability assessment; contrasts ANN-Benchmarks million-scale libraries with billion-scale algorithm competitions.