G-Research’s October 19 engineering note asks a precise infrastructure question: what if the file format is optimized not only for scans across columns, but for shuffled training samples, feature lookups, embeddings and images? Its answer is Lance, an independent open-source format written in Rust. G-Research
Parquet compresses data in pages. Looking up one value can therefore require fetching and decoding the whole page, and nested data can require several column reads. G-Research cites the Lance authors’ finding that carefully tuned Parquet improved random access by more than 60 times over its defaults, but only by sacrificing scan performance or memory.
Lance separates structural layout from compression. Its target is at most one I/O operation for a fixed-width column and two for a variable-width column, even when the data is deeply nested. Small values can be packed into mini-blocks; larger items such as embeddings use an offset index that lets a reader fetch one value without its neighbors. Very large images or videos can live as external blobs.
For a research platform, the table semantics may matter as much as the lookup. Appends, deletes and new columns become immutable manifests, old versions remain readable, and vector and scalar indexes can be rebuilt independently. That supports reproducible training and rollback without treating every feature-store read as an analytical scan. The caution is ecosystem breadth: G-Research explicitly says Lance is not a drop-in Parquet replacement.