Deep1B Dataset

Confident AI evaluates vector DB-integrated LLM apps with 50+ metrics on faithfulness, relevance, tracking QPS/latency in production traces for RAG performance. Key features include DeepEval-powered scoring, observability dashboards, and quality-aware alerting across datasets. Supports prod vector DB RAG selection via real-world eval; broader than ANN-Benchmarks (indexing) or VectorDBBench (DB perf).

GraphRAG-Bench benchmarks vector DB-enhanced GraphRAG vs vanilla RAG on multi-hop queries, measuring perf metrics like QPS/latency for reasoning tasks across domains. Key features include standardized eval for graph vs vector retrieval in 2025 release. Helps select hybrid prod vector DB setups; graph-focused unlike pure ANN-Benchmarks or VectorDBBench.

Massive Text Embedding Benchmark leaderboard covering 58 datasets across 112 languages and 8 embedding tasks. Industry-standard benchmark for comparing text embedding models.

Zero-shot benchmark for embedding model evaluation on 18 diverse datasets with NDCG@10 and Recall@100 metrics correlating to vector DB QPS/latency in production. Features heterogeneous tasks like QA, fact-checking, biomedical retrieval for robust comparisons. Use cases include selecting embeddings for RAG pipelines in vector DBs; complements ANN-Benchmarks indexing focus with retrieval task evaluation, differs from VectorDBBench full-DB tests.

Tests vector DBs on multimodal QPS/latency for heterogeneous embeddings and compound queries including GPU setups. Features Docker-based eval for Milvus etc. on cross-modal retrieval. For selecting multimodal vector DBs; differs from ANN-Benchmarks text-only by adding hybrid workloads vs custom single-DB tests.

Provides QPS/latency/recall benchmarks for ANNS algorithms on billion-point datasets via NeurIPS tools for dataset prep and evaluation. Features scalable testing for extreme throughput and visualization. Key for production vector DBs at scale; extends ANN-Benchmarks with billion-scale tools unlike full-system DB benchmarks.