Mainly done through aggressive automatic cuda graph optimizations, but it also has cool custom formats, allows for huggingface pulls from cli, better quantization control, and a lot of other small things that make it easier to use than ollama.
A small developer team and I had worked on this for a while for personal reasons, so I have decided to port it to cpp and release it to the general public. The stats are in the github. I promise if you have a gpu this will improve your speeds by at least 50%, even if you believe your configuration is optimized.
What do you think?