BenchmarkQED

Standardized benchmark for QPS/latency/recall tests on ANN libraries using datasets like SIFT1M and Deep1B to compare throughput and accuracy. Features metrics for build time, memory usage across HNSW, FAISS, ScaNN. Used for vector DB index selection during development; contrasts with BigANN billion-scale competitions by focusing on million-scale library performance vs full-system custom benchmarks.

Zero-shot benchmark for embedding model evaluation on 18 diverse datasets with NDCG@10 and Recall@100 metrics correlating to vector DB QPS/latency in production. Features heterogeneous tasks like QA, fact-checking, biomedical retrieval for robust comparisons. Use cases include selecting embeddings for RAG pipelines in vector DBs; complements ANN-Benchmarks indexing focus with retrieval task evaluation, differs from VectorDBBench full-DB tests.

Evaluates ANN algorithms on billion-scale datasets with QPS/latency/recall metrics via NeurIPS tracks for out-of-distribution and streaming tests. Features standardized billion-point evaluation for throughput and memory. For production vector DB scalability assessment; contrasts ANN-Benchmarks million-scale libraries with billion-scale algorithm competitions.

Tests vector DBs on multimodal QPS/latency for heterogeneous embeddings and compound queries including GPU setups. Features Docker-based eval for Milvus etc. on cross-modal retrieval. For selecting multimodal vector DBs; differs from ANN-Benchmarks text-only by adding hybrid workloads vs custom single-DB tests.

Provides QPS/latency/recall benchmarks for ANNS algorithms on billion-point datasets via NeurIPS tools for dataset prep and evaluation. Features scalable testing for extreme throughput and visualization. Key for production vector DBs at scale; extends ANN-Benchmarks with billion-scale tools unlike full-system DB benchmarks.

Evaluates embeddings on 58 datasets/112 languages with retrieval/clustering metrics for vector DB model selection via nDCG/Recall throughput proxies. Features 8 task types for comprehensive perf eval. Standard for RAG embedding choice; text-focused unlike BigVectorBench multimodal, complements ANN-Benchmarks index benchmarks.