Milvus Connectors
Milvus Connectors, such as the Spark-Milvus Connector, enable seamless integration of Milvus vector databases with third-party tools like Apache Spark for machine learning and data processing workflows.
About this tool
Milvus Connectors
Milvus Connectors, such as the Spark-Milvus Connector, enable integration between Milvus vector databases and third-party tools like Apache Spark for enhanced machine learning and data processing workflows.
Source: https://github.com/zilliztech/spark-milvus
Features
- Seamless Integration: Connects Apache Spark with Milvus, combining Spark's data processing with Milvus's vector storage and query capabilities.
- Data Formats:
milvus: Write Spark DataFrame data into Milvus collections, with automatic collection creation based on the DataFrame schema.milvusbinlog: Read Milvus's built-in binlog data (parquet-based, not compatible with standard parquet readers).mjson: Generates JSON data in the format required by Milvus's bulk insert feature, improving write performance.
- Milvus Options: Configurable connection and management options for Milvus within Spark jobs.
- MilvusUtils (Scala): Utility functions to simplify code, including:
readMilvusCollection: Loads an entire Milvus collection into a Spark DataFrame by wrapping necessary SDK calls and logic.bulkInsertFromSpark: Imports Spark output files into Milvus using bulk insert operations.
- Multi-language Support: Examples and utilities are available for both Python and Scala.
- Databricks Integration: Demos and usage examples for Databricks notebooks.
- Open Source: Licensed under the Apache License 2.0.
Pricing
- The Spark-Milvus Connector is open source and free to use under the Apache License 2.0.
Category
- Data Integration & Migration
Tags
- milvus, integration, machine-learning, apache-spark
Loading more......
Information
Categories
Similar Products
6 result(s)The Spark-Milvus Connector is an integration that allows Apache Spark jobs to read from and write to Milvus, enabling scalable ETL and analytics workflows for vector data.
MindsDB provides an integration with Milvus, enabling users to connect and manage vector data using SQL-like queries. This integration brings federated AI query capabilities across structured and unstructured data with Milvus as the vector database backend.
Vector Transport Service (VTS) is a tool for transporting vector data efficiently between Milvus clusters or environments, supporting large-scale data migration and synchronization. Vector Transmission Services (VTS) are tools for transferring data between Milvus and various data sources (like Zilliz clusters, Elasticsearch, Postgres/PgVector, or other Milvus instances), facilitating vector data migration and integration.
The Airbyte Milvus connector lets users sync data from various Airbyte-supported sources into Milvus as a destination, enabling low-code vector data ingestion pipelines.
The Kafka Connect Milvus Connector is a plugin for Kafka Connect that streams data into and out of Milvus, supporting real-time vector data ingestion pipelines.
The Milvus destination in Fivetran enables automated ELT pipelines that load data into Milvus as a vector database, supporting AI and similarity search workloads.