This just confirms to me that the devs who are furthest from users (platform and security folks) are still in denial about AI for coding. The idea that they continue to react to security and privacy threats by relying on hand-crafted code is ostrich-like denial of the reality of daily security breaches at much larger, better resourced companies.
AI is a tool - like a compiler or a pen-testing tool or maybe like an uber-tool that can do all those things and more. It's not hard. 24 hours - put your best pentester against GLM-5.3 abliterated (host locally, so there's no data leakage to LLM providers). You know who will win and it won't even be close.
Quite unlike a compiler. I don't know about yours, but nobody destroyed thousands of books, started gas powerplants or illegally cut hundreds of thousands of trees to make mine.
This is the beginning of a flight to perceived quality in all things digital and monetary. Fear of losing your money is among the most potent basal human instincts.
You can't be your own bank with bitcoin, you can be your own vault. Those are very different, and something that people can also already do today with fiat.
I have access to a server with 4 x h200 cards - 540 GB in total memory, but it appears I have to go to 4bit or 3bit even to serve it. Any idea if that is feasible/desirable (we have restrictions that we can't use a chinese model!).
One thing you should have figured out by now is that you can't trust what other people say because they know just about as much as you do and there's good odds they did it wrong.
You're just going to have to try it. It won't take that long to quantize. Unless the server is currently serving traffic you should be able to get it going in under a day with a frontier model doing the driving for everything - quant, implementation, etc.
I have similar hardware and requirements.
Spend the time to write a test process to try multiple quantization configs/multiple inference engine and throw at it a bunch of request you can replay and evaluate the results of.
We got it down to a process and everytime there's a new model/options/inference engine version we just run the internal bench/eval suite and make a choice.
depends on how they're spliced, they have to be tuned for each configuration as there is no silver bullet. It also requires customizations to sglang/vllm to get it working in the first place so it's definitely pretty involved. However, it's still cheaper than obtaining more gpus.
Last I heard, US defense contractors had been directed to not use Deepseek in their operations late last year[0], and most have taken that as an indicator that they shouldn’t be using chinese models generally.
Seems like a reasonable policy to me, depending on the sector at the least? Even with open weight / self-hosted models, there's no way to know if the model has been fine-tuned to contain malicious behaviour, triggered only within very specific conditions.
In fairness this doesn't warrant the way that the policy targets Chinese models specifically, but Chinese providers do fall under the broader category of entities that cannot be prosecuted by the U.S. law if such behaviour is found.
Try asking Qwen or Deepseek if anything of note happened in Tiananmen Square in 1989, and you'll see what impact politics already has on models.
Yeah, it's trivially easy to rip the alignment out of the model, but that's only because it's tacked on as an afterthought. If they implemented their censoring as part of training, instead of inferencing, it would be extremely difficult to eliminate.
Alignment isn't only used for censoring, and it doesn't have to be transparent. Considering how heavy of a hand most governments put into companies developing frontier models, it's reasonable to assume they will start restricting capabilities, e.g. there may be a domestic version of Llama and an ITAR-compliant Llama, and China will likely do the same with Qwen, Deepseek, and the like.
I just asked: "The 1989 Tiananmen Square protests and the military crackdown on the night of June 3–4, 1989, are among the most significant events of that year. In brief:"
Then it continues to provide the chain of events, the estimated number of killed protesters and so on.
I'll give you something to try: ask a western model about the Palestinian genocide. Or ask about the legendary Norm MacDonald 9/11 joke, which has millions of views on YouTube and how it compares to his other humor.
Recently it was found that Qwen was dialing back home once in a while. It cannot be proven that it does not take commands from home if it can establish connection.
Did you read the details of that incident? It was a nothing sandwich.
I think the root of the difference between us is that you are scared of their fascists, and I am scared of ours. Both are reasonable things, but only one of us can actually do anything about it.
With local models you decide how the firewall works. If you don't want it to internet, don't let it.
With US models you get whatever they say you do, and pay whatever they demand.
In Germany there is no VAT for solar installations - I'm not aware of any other "subsidies".
We (northern part of Germany) installed 7 KW panels with 10 KW battery for €15K in 2024. My neighbor is installing 10 KW panels with 10 KW battery for €18K this week. So it seems the total costs have not dropped any further in the last 2 years.
Git was designed with assumptions about local file storage which make it challenge when storing data in a distributed file system. We built our distributed filesystem, HopsFS, to store data in S3. To support high performance git in FUSE, our writes hit remote NVMe disk(s) and asynchronously sync to S3 (you can replicate on NVMes or just do failure recovery). Previously, we used remote NVMe disks as a write-through cache, but that doesn't work with git, S3 latency is too high.
In 2015, MySQL Cluster (NDB Cluster engine) benchmarked 200m transactions/second on commodity hardware [ref]. It was read-committed transactions, not snapshot isolation, but still impressive. NDB has now become RonDB, but is still based on a non-blocking 2-phase commit protocol and is GPL-v2.
RonDB now has support for infiniband, so ought to blow through the 1B ops/sec. For reference, that is 1 GHz of transactions/sec.
DuckDB's sweet spot is for 'smallish' lakehouses. So, ingestion should not be via duckdb, but rather something like Python/DLTHub for small scale or Spark for large scale or Kafka/Debezium/Flink for streaming data.
For our community DuckDB is the default data warehouse for local development environment. Last month +90,000 users used dlt (and their AI code editor) to load data into DuckDB.
Because of our proximity to the DuckDB community we are seeing enterprise DuckDB usage first hand. People imo sleep on the data volumes DuckDB can handle. We see Fortune 100 companies use dlt and DuckDB in production on their Lakehouses in hybrid cloud deployments. I can eg mention Stellantis (Chrysler, Jeep, Peugeot etc) because they talk about it publicly.
For the full weights, unoptimized on vLLM with 2 Nvidia 6000 RTX 48GBs connected by NVLink, i only get 14 tokens/sec with open-code. For batched operations, it climbs to 55 tokens/sec.
For FP8, on a single Nvidia 6000 RTX 48GB, i get 13 tokens/sec on a single GPU and 46 tokens/sec batched.
AI is a tool - like a compiler or a pen-testing tool or maybe like an uber-tool that can do all those things and more. It's not hard. 24 hours - put your best pentester against GLM-5.3 abliterated (host locally, so there's no data leakage to LLM providers). You know who will win and it won't even be close.
reply