MyScale's Vector Database Benchmark

Category: Benchmarks & Evaluation
Source: GitHub Repository

Description

MyScale's Vector Database Benchmark is an open-source framework designed to assess the performance of fully-managed vector databases across various search and retrieval tasks. It provides tools and datasets for benchmarking and delivers comparative results for different vector database solutions.

Features

• Benchmarking Framework: Tools to run standardized performance benchmarks on fully-managed vector databases.
• Supported Databases: Includes out-of-the-box support for popular vector databases such as Pinecone, Weaviate, Milvus, Qdrant, and MyScale.
• Workload Types: Measures throughput and cost-performance for both standard vector search and filtered vector search workloads.
• Cost-Performance Analysis: Calculates cost-performance ratio by dividing monthly cost by queries per second (QPS), enabling cost-effectiveness comparisons.
• Dataset Management: Provides scripts and modules for dataset preparation and upload optimization.
• Experiment Management: Organizes and automates experiments for reproducible benchmarking across multiple services.
• Results Visualization: Supports visualization of benchmark results for easier comparison and analysis.
• Open Source: Licensed under Apache-2.0 and extensible for custom needs.
• Hybrid Search Support: Includes support for hybrid search scenarios in MyScale.
• Docker Support: Includes Dockerfiles for containerized environments.

Pricing

Not applicable. This is an open-source tool available for free under the Apache-2.0 license.

Tags

benchmark vector-databases performance retrieval