tiktoken

GPU-accelerated vector search and clustering library from NVIDIA RAPIDS. Provides 8-12x faster index building and queries with multiple language support (C, C++, Python, Rust). This is an OSS library.

A utility class from the Hugging Face Transformers library that automatically loads the correct tokenizer for a given pre-trained model. It is crucial for consistent text preprocessing and tokenization, a vital step before generating embeddings for vector database storage.

Apache Lucene is a high-performance, full-featured open-source text search engine library written in Java. It provides approximate nearest neighbor (ANN) vector search capabilities using Hierarchical Navigable Small World (HNSW) graphs, enabling semantic search on high-dimensional embedding vectors up to 1024 dimensions by default (extendable via custom codecs).

The Natural Language Toolkit (NLTK) is a leading Python platform for building programs to work with human language data. It provides easy-to-use interfaces to lexical resources like WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning.

Automatic index selection and tuning library for FAISS that selects optimal KNN index configurations to maximize recall given memory and query speed constraints, eliminating manual hyperparameter tuning.

Major update to the Sentence Transformers library introducing a new SentenceTransformerTrainer for easier fine-tuning, multi-GPU support, improved loss logging, and access to 15,000+ pre-trained models on HuggingFace.