If you adjust for writing speed, this model is already more efficient than humans. It can run at 233 tps on a 5090 which is roughly 10k words per minute compared to human typing speed of 80 wpm. If you duty cycle it down from 575 watts of 5090 to human speeds it’s about 5 watts.
All big LLMs already visibly watermark all their text with easy to detect annoying phrases and turns of speech that everyone is already sick of hearing. Why do AI companies keep making their products worse to appease anti-AI, it’s not like they’ll suddenly start supporting it if you do so. If you’re worried about European customers, just relax your firewalls to let more VPNs through, if the productivity boost is high they will use it anyway if their rules keep crippling their own models
You mean the model itself? Yeah, probably not needed if you self host. If they run a service that they sell and they are a bigger player like Alibaba, I doubt they can just ignore it.
You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“
If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?
An outage is not the same thing. A similar thing would be if Outlook 365 started sending emails on your behalf or changing around words in them, or if SAP / Quickbooks said: "No guarantee that the numbers you type in match the results of the annual statement we produce", or if Tesla was like "You should absolutely not use our cars for driving anywhere".
If you bring language like this into more traditional fields, it feels absolutely bonkers.
If you’re on AMD, use llamacpp with the vulkan backend. For the 35B I get around 115 tps without mtp and 150-170tps with it. Also the pcie slot you use affects it
reply