Kafka Connect Milvus Connector
The Kafka Connect Milvus Connector is a plugin for Kafka Connect that streams data into and out of Milvus, supporting real-time vector data ingestion pipelines.
About this tool
Kafka Connect Milvus Connector
Category: Data Integration & Migration
Brand: Zilliz
Type: Kafka sink connector (plugin for Kafka Connect)
Overview
Kafka Connect Milvus Connector is a Kafka sink connector that streams vector data from Kafka topics into Milvus-based vector databases, including self-hosted Milvus and Zilliz Cloud. It is designed to build real-time vector data ingestion pipelines where Kafka messages are written directly into Milvus collections.
Features
-
Kafka sink connector for Milvus
- Streams data from Kafka topics into Milvus collections.
-
Vector data ingestion
- Supports ingestion of vector data into Milvus or Zilliz Cloud.
- Targets collections that contain exactly one vector field and optional scalar fields.
-
Schema alignment between Kafka and Milvus
- Requires Kafka message schema to match the Milvus collection schema.
- Enforces one and only one vector field in the schema.
- Supports zero or more scalar fields (e.g., string, integer, float).
- Field names must match exactly between Kafka messages and Milvus collection.
-
Deployment targets
- Works with Confluent Cloud (hosted Kafka).
- Works with open-source Kafka clusters.
- Writes to self-hosted Milvus or Zilliz Cloud.
-
Plugin packaging
- Distributed as a plugin zip (e.g.,
zilliz-kafka-connect-milvus-xxx.zip) for Kafka Connect.
- Distributed as a plugin zip (e.g.,
-
Real-time streaming pipelines
- Designed for real-time or near real-time streaming of vector data from Kafka into Milvus.
Requirements & Usage Notes
- Need a running Kafka environment (Confluent Cloud or open-source Kafka).
- Need a Milvus or Zilliz Cloud instance with collections defined.
- Kafka topic messages must use a schema that matches the target Milvus collection (same fields, same names, one vector field).
Pricing
- Not specified in the provided content. The connector is hosted on GitHub and appears as an open-source project; consult the repository and license file for details.
Resources
- Source code & documentation: https://github.com/zilliztech/kafka-connect-milvus
- Image (pipeline diagram): https://raw.githubusercontent.com/zilliztech/kafka-connect-milvus/main/docs/images/kafka-milvus-pipeline.png
Loading more......
Information
Categories
Tags
Similar Products
6 result(s)The Airbyte Milvus connector lets users sync data from various Airbyte-supported sources into Milvus as a destination, enabling low-code vector data ingestion pipelines.
The Milvus destination in Fivetran enables automated ELT pipelines that load data into Milvus as a vector database, supporting AI and similarity search workloads.
The Spark-Milvus Connector is an integration that allows Apache Spark jobs to read from and write to Milvus, enabling scalable ETL and analytics workflows for vector data.
Vector Transport Service (VTS) is a tool for transporting vector data efficiently between Milvus clusters or environments, supporting large-scale data migration and synchronization. Vector Transmission Services (VTS) are tools for transferring data between Milvus and various data sources (like Zilliz clusters, Elasticsearch, Postgres/PgVector, or other Milvus instances), facilitating vector data migration and integration.
Milvus CDC (Change Data Capture) is a component of the Milvus ecosystem that enables data synchronization between Milvus and other systems. It is useful for maintaining up-to-date vector data pipelines and supporting real-time vector search applications.
Milvus Connectors, such as the Spark-Milvus Connector, enable seamless integration of Milvus vector databases with third-party tools like Apache Spark for machine learning and data processing workflows.