The September 2 comparison reports 182.8 versus 145.6 tokens/second against MLX 0.32.2 at a 4,096-token context—about 26% faster. But it measures in-process generation, excluding loading, tokenization and HTTP overhead; it is a vendor benchmark with two recorded rounds, not an independent serving test. Benchmark and method
The repository’s current constraints are unusually important: M5-or-newer GPU family, macOS 26+, one supported Qwen architecture, greedy text generation, and no streaming, tool calling or structured-response formats. That makes it interesting kernel and inference-engine work—not a ready replacement for an agent backend. The next measurement worth wanting is time to a validated task result with the required interface features restored. Lily README