Manticore Search adds chunking for improved vector search in long documents

Manticore Search has implemented a chunking mechanism that automatically divides long documents into smaller parts for vector search. This is particularly beneficial for documents exceeding the model's input token limit, allowing for improved retrieval of information that would otherwise be discarded.

With the addition of the chunk_strategy parameter in the CREATE TABLE command, engineers can now easily specify how documents should be chunked and embedded without needing to manually split the text or create additional tables. This reduces the complexity of the ingestion process and minimizes the risk of losing relevant content.

Performance metrics indicate that using this chunking strategy increases recall from 55.1% to 83.3% and improves mean reciprocal rank (MRR) from 0.44 to 0.70, albeit with increased RAM usage and ingest time. The trade-off between improved accuracy and resource consumption must be considered in deployment scenarios.

The flexibility of specifying different chunking strategies (truncate, mean, fixed, recursive, and sentence) allows engineers to tailor the search behavior based on specific document types and query needs. However, it is important to note that queries themselves are not chunked, which ensures they remain efficient.

This feature stops working effectively when documents exceed the chosen chunk size or when the overlap settings are not optimized. Careful tuning of these parameters is essential to maximize search performance while managing resource utilization.