Lance is an open lakehouse format for storing images, videos, and embeddings alongside traditional tabular data. It claims 100x faster random access than Parquet and supports hybrid vector and full-text search directly on object storage. The project includes Python and Rust SDKs compatible with Pandas, DuckDB, and Polars.
It is worth a look if you are building retrieval-augmented generation pipelines and need fast random access for training. The catch is that it is still evolving rapidly, so you must carefully pin stable storage versions to avoid compatibility issues across mixed deployments.
more: https://github.com/lance-format/lance
my other side: ivo.blogspot.com
Leave a Reply