Remek Kinas reported that version 2.3.1 of the basal engine, released on October 21, has accelerated the performance of the basal-omni models. He wrote that in version 2.3, the engine with CUDA graphs began to fully support basal-omni (Gemma 4), previously available in fast mode via vLLM or slower PyTorch. Among the solutions implemented, he mentioned single calculations of document states for multiple questions as well as FlashAttention 3 and FlexAttention. In tests on a single H100 card, with 16 parallel queries, basal-omni-1.0-mini achieved 61.2 decisions per second compared to 10.0 in version 2.2, basal-omni-1.1-max — 16.2 compared to 4.8, and basal-omni-1.0-ultra — 6.1 compared to 2.0. For five questions about a long document, the results were 21.4 compared to 1.6, 10.6 compared to 0.72, and 4.8 compared to 0.28 decisions per second, respectively. Kinas noted that out of 10,606 test decisions, the best response matched version 2.2 in 99.85–99.99% of cases where the model was confident. The share of automated decisions at 1% error was expected to change by at most 0.1 points. Among the new features in version 2.3.1, he mentioned support for documents of up to 82,000 tokens by the ultra model on a single H100 card; previously, he noted that memory was insufficient above about 32,000. He also mentioned that in JevBench, basal did not provide 15 responses due to exceeding the limit. He added that agents and conversations only process the new part of the state, and the step for basal-omni-1.1-max was shortened from 0.79 to 0.27 seconds. In the case of multiple questions about a long document, the throughput of the mini model increased from 21.9 to 33.7 questions per second, a 54% increase. Kinas also wrote that Blackwell cards use the fast engine by default, and basal-2.0 (Bielik) in eager mode on H100 reduced response time from 591 to 52 ms. He listed clearer HTTP codes 422, 503, and 500, load available at /health, the basal-calibrate message about too few labels, and support for images and audio through vLLM for the mini model, which he described as 6–8 times faster. He also thanked the contributors and testers: Miłosz (Butterski), Grzegorz Brzezinka, from Piotrków, and Kamil (kamilrosa), as well as edurdias and the company BRAUN SI / NovaraX for testing on their own data. Special thanks were given to Damian Plewa and the IT SOL team for the basal-rs engine. #basal #AI #inference (translated)
