LinkedIn's retrieval system found a bottleneck in a seemingly convenient place: document-ID lookups on the GPU. Its September paper describes narrowing a billion-plus-document corpus to tens of thousands of candidates, then assembling features for a neural pre-ranker. GPU binary searches took about five milliseconds at batch one and forty-seven at batch eight in the reported tests. Moving those irregular lookups to a parallel CPU hash map brought them to roughly one millisecond across tested batches.

The GPU still gathers dense document features, joins request-specific viewer-to-author edges, and scores candidates. That graph-edge join is separate from resolving document IDs. Their progressive throughput table goes from a tuned single-request baseline to one point seven five times with batching, then ten times with CPU lookups and a larger batch, under a hundred-and-twenty-millisecond ninety-ninth-percentile latency budget. Ten times measures the cumulative redesign, not the isolated lookup change.

They also preallocated cuBLAS workspaces after variable batches fragmented the remaining GPU memory. The document index occupied most of it. Larger batches had made the lookup slower and the spare memory less predictable; the authors changed both the lookup location and workspace allocation.

Source 1