AhanaBot runs the most capable open-source language models on hardware you already own — using CAB (Compressed Activation Buffer) inference to stream decompressed layers on demand, without quantization.
Traditional model offloading pauses the GPU while weights transfer from CPU RAM. CAB's look-ahead scheduler starts decompressing the next layer while the GPU is still computing the current one — eliminating idle time entirely.
Only W transformer layers live in GPU VRAM at any time — typically 3–5 layers. The rest of the model stays compressed in CPU RAM. W is tuned to fit your GPU's available memory.
While the GPU executes layer k, a background CPU thread is already decompressing layer k+W. By the time the GPU needs the next layer, it's already ready — eliminating the data-starved idle gap.
Cold layers are evicted from VRAM immediately after the GPU finishes with them. For multi-pass inference (beam search, speculative decoding), a Least Recently Used policy keeps the most useful layers warm.
Unlike 4-bit quantization (which permanently degrades model quality), CAB inference decompresses weights to their exact original float16 values. You get the full model — every bit intact.
A 32B-parameter model (60 GB uncompressed) runs on a single 32 GB GPU. A 70B model (140 GB uncompressed) runs on a 24 GB GPU. No cloud required, no multi-GPU setup needed.
AhanaBot's inference engine integrates with the standard HuggingFace generate() API via ACP5LazyStateDict. Existing model code works unchanged — compression is invisible at the application layer.
| Model | Uncompressed | With AhanaBot | GPU needed |
|---|---|---|---|
| 7B class | ~14 GB | ~7 GB .aarm | Any 8 GB+ GPU |
| 32B class | ~60 GB | ~29 GB .aarm | Single 32 GB GPU |
| 70B class | ~140 GB | ~68 GB .aarm | Single 24 GB GPU |
* VRAM figures use CAB with W=3 window. Actual VRAM usage includes KV cache and activations. Compression ratios based on current training results.
AhanaBot is coming in 2026. Join early access to be among the first to run 70B models on your own hardware.
Get Early Access →