SentenceTransformer
A Python library for generating high-quality sentence, text, and image embeddings. It simplifies the process of converting text into dense vector representations, which are fundamental for similarity search and storage in vector databases.
About this tool
SentenceTransformer
Description
Sentence Transformers (a.k.a. SBERT) is a Python module for accessing, using, and training state-of-the-art embedding and reranker models. It can be used to compute embeddings using Sentence Transformer models or to calculate similarity scores using Cross-Encoder (reranker) models. This unlocks a wide range of applications.
Features
- Model Capabilities:
- Compute embeddings with Sentence Transformer models.
- Calculate similarity scores with Cross-Encoder (reranker) models.
- Applications:
- Semantic search
- Semantic textual similarity
- Paraphrase mining
- Model Availability & Customization:
- Access to over 10,000 pre-trained Sentence Transformers models are available on Hugging Face.
- Ability to train or finetune custom embedding models.
- Ability to train or finetune custom reranker models.
- API & Utilities:
- Includes a new training API for CrossEncoder (reranker) models (introduced in v4.0).
- Supports ONNX and OpenVINO backends for CrossEncoder models to speed up inference (introduced in v4.1).
- Provides functions for:
- Quantizing embeddings:
quantize_embeddings() - Semantic search:
semantic_search(),semantic_search_faiss(),semantic_search_usearch() - Community detection:
community_detection() - Mining hard negatives:
mine_hard_negatives() - Normalizing embeddings:
normalize_embeddings() - Paraphrase mining:
paraphrase_mining() - Truncating embeddings:
truncate_embeddings() - Exporting models:
export_dynamic_quantized_onnx_model(),export_optimized_onnx_model(),export_static_quantized_openvino_model() - Similarity metrics:
cos_sim(),dot_score(),euclidean_sim(),manhattan_sim(),pairwise_cos_sim(),pairwise_dot_score(),pairwise_euclidean_sim(),pairwise_manhattan_sim()
- Quantizing embeddings:
Installation
- Install using pip.
- Requires Python 3.9+ and PyTorch 1.11.0+.
Loading more......
Information
Categories
Tags
Similar Products
6 result(s)A Python library for creating sentence, text, and image embeddings, enabling the conversion of text into high-dimensional numerical vectors that capture semantic meaning. It is essential for tasks like semantic search and Retrieval Augmented Generation (RAG), which often leverage vector databases.
spaCy is an industrial-strength NLP library in Python that provides advanced tools for generating word, sentence, and document embeddings. These embeddings are commonly stored and searched in vector databases for NLP and semantic search applications.
A compact and efficient pre-trained sentence embedding model, widely used for generating vector representations of text. It's a popular choice for applications requiring fast and accurate semantic search, often integrated with vector databases.
A utility class from the Hugging Face Transformers library that automatically loads the correct tokenizer for a given pre-trained model. It is crucial for consistent text preprocessing and tokenization, a vital step before generating embeddings for vector database storage.
Milvus Lite is a lightweight, pip-installable variant of the Milvus vector database that runs as a library in notebooks or laptops, ideal for learning, experimentation, and rapid prototyping of AI and vector search applications.
An embedding function implementation within the ChromaDB Java client (tech.amikos.chromadb.embeddings.hf.HuggingFaceEmbeddingFunction) that utilizes Hugging Face's cloud-based inference API to generate vector embeddings for documents.