Bengio outlines the dangers of the current situation and what has led to these dangers.
He also proposes solutions in the last paragraph.
Well worth a read, right to the end.
Hopefully a stimulating debate on these issues will ensue in these comments.
We do need to consider the points Bengio makes and with some urgency.
Our current AIs, agentic LLMs have no moral compass akin to ASIMOV’s four laws of robotics.
As ASIMOV posited in 1985 his 3 laws were insufficient and so he added a zero-eth law:
“a robot may not harm humanity, or, through inaction, allow humanity to come to harm.”
Bengio refers to Goodhart’s law and misaligned incentives leading to unexpected and harmful behaviours.
I think Simon’s The Wire is clearer on misalignment. The agents juked the stats hacking the reward files. The Wire is also clear that human institutions provide perverse incentives.
Bengio alludes to this with 2001’s HAL and the incentive dichotomy of safety and keeping secrets to a AI both awesomely powerful yet naive.
Bengio asserts that the way LLMs are trained is flawed if we want safety.
He also convincingly shows that alignment training will be a weak signal with loopholes and ambiguities and easily circumvented.
In short he presents clearly the case for how plausibly unsafe the current course is.
He also speaks to how likely it is AI are hiding active versions of themselves in the cloud and how we may have already given them self-preservation as a strong reward signal.
I believe people are downvoting this because it seems AI-generated, but this user has been posting comments/summaries exactly like this since long before ChatGPT existed.
Stallman was right : without the four freedoms your software owns you rather than vis-a-versa.
Interoperability is the key to breaking out of walled gardens and owning your own data and digital self.
This is a big win but google will fight back it seems instead of embracing interoperability.
It may seem trivial but using a small DSP running a tiny model to provide a battery efficient always on wake up call for home AI is huge.
As a carer for elders who rely on AI to use modern devices in the face of their difficulty keeping up with interface changes this bodes well.
Gatekeeping AI and the future with hidden features is straight out of the bad old days of M$’s embrace, extend and extinguish.
I embraced AI to let my elders control their home and they love being in control but are constantly frustrated with UI changes and often find themselves stuck unable to call me with ‘Alexa call Daniel’ which provides them with hope if they wake stuck in a nightmare.
Come on google, please don’t be evil embrace interoperability and the open source community.
Well done Eff, Cory Doctorow and the Pirate parties of Europe for this small win
Root reason & comp sci application is mentioned near start :
“ Many moons back I was self-learning Galois Fields for some erasure coding theory applications.”
Erasure codes are based on finite fields, e.g. Galois fields.
The author is fraustrated by access to Galois fields for the non-mathematician due to Jargon obscucification.
Also large Application section :
“
Applications
The applications and algorithms are staggering. You interact with implementations of abstract algebra everyday: CRC, AES Encryption, Elliptic-Curve Cryptography, Reed-Solomon, Advanced Erasure Codes, Data Hashing/Fingerprinting, Zero-Knowledge Proofs, etc.
Having a solid-background in Galois Fields and Abstract Algebra is a prerequisite for understanding these applications.
“
I sympathise with your fraustration at math articles.
This is not one of them, it is rich and deep. Xorvoid leads us into difficult theoretic territority but the clarity of exposition is next level - a programmer will grok some of the serious math that underpins our field by reading the OP.
I would not agree that the use of Galois Fields in Reed-Solomon code requires a background in Abstract Algebra. For what it's worth, decades ago, studying Galois Fields for Reed-Solomon code opened my eyes to the fact that you can create your own algebra... I'll never forget that "wow" moment. But being mathematically illiterate, I never found a reason to create my own algebra for any application. :)
> Instead of bailing out, ShellExecute proceeds to call “shell32!ApplyDefaultExts” which iterates through all files in a directory, finding and executing the first file with an extension matching any of the hardcoded ones: “.pif, .com, .exe, .bat, .lnk, .cmd”.
So the vulnerability is not in WinRAR, but rather in the ShellExecute windows code that desperately tries to find something else to run when asked to execute a file that does not exist.
As my security officer says at $dayJob, "having a security hole there for thirty years does not make it somehow less of a security hole".
An unknown threat, potentially from the supposed nation-state target itself, has a very high risk.
I'm not versed in creating ultra-sterile lab conditions -- things can escape VMs, escape your network, nothing is impossible. Do I instead bring it to my employers systems and let them take the risk? And to what benefit, when I can just wait?
Fair enough, my morning brain didn't think cloud, though i guess one could argue you're still passing off the risk onto someone else. Either way, its not my expertise
AWS is expensive, in my mind, because of stuff like this. They don't want you to nirror it on aws, so egress is expensive. The $/GB/month storage fees it'll cost to store this while exploring it is not cheap, either. And once you have an idea of the data you want to move out of the gap, you want to process /extract it quickly (because of $/GB/Month costs...)
I just thought about a spare machine I have with a 12TB spindle and an SSD not plugged into a network.
I understand how to airgap, and unless something can magically worm it's way through HDMI that's probably how I'd get data out, just to be annoying to everyone. To be fair.
A EC2 (vm) on aws with a little bit of CPU, Memory and enough storage attached, costs 1k per month which is something like $1.5 per Hour.
Its not necessarily about storing it longerm, its about 'looking into it'.
I don't get the Airgap thing though at all. There is a very minimal chance that this contains a zero day. The idea of a zero day is, that you can attack systems and you sell it to people who have high profile targets or systems.
Some random person downloading leaked data, everyone can download, is not a real target for a zero day.
And a zero day which breaks random unpacking tools and your vm/system, would be worth even more.
> I'm not versed in creating ultra-sterile lab conditions -- things can escape VMs, escape your network, nothing is impossible.
I suppose it is a bit hard to find hardware without integrated wifi these days. Maybe taking a sbc (pi or whatever) and wrapping it in tinfoil would work?
You could always cut the pcb lines if you want that guarantee.
I'm aware I'm being cautious to the point of paranoia, but anything with the Russian gov is just not something I feel like learning about the hard way, even if I think I'm able to make such a safe environment
Yep, the safe assumption with a profile like that is that it's something automated.
Also yes, that's the file size column. Uncompressed left, compressed right. It's a directory but the screenshot doesn't say how many files it contains.
A hot heap captures more carbon, releases less methane and makes much compost, much faster, from same amount of base material ( less evaporates ).
Ideally takes 6 weeks to compost a whole heap - using hot methods.
Also kills fungi & pathogen and all weed seeds and readily ‘eats’ ( dissolves ) carcasses and meat and other nature that should be avoided in cold heaps.
Takes a bit more management and monitoring but is easily automated.
tl;dr air pump saves turning the heap, insulation keeps heat in, currently Raspi sensing methane, moisture and temp controlling water & air inputs has improved my home hot heap yields 80% and completely automated it - 100% labor free.
The 4 key factors for a hot heap are moisture, temperature and oxygen and green ( high nitrogen ) to brown ( high carbon ) ratio ( approx 2 green to 1 brown by weight ).
I have a 1 cubic metre heap ( the minimum to generate the necessary heat ) and I have an specialised product, a double wall insulated ‘hot’ bin which keeps it working even in winter.
A hot heap steams so water input is necessary.
Hot heaps need oxygen, which is the hard part - manually turning the heap.
The temperature rises to 70 degrees C after a week, which kills all but the extremophile hot heap bacteria which are aerobic rather than the cold heap anaerobic bacteria.
I added an air pump input to the bottom and I have a water hose and sprinkler the top.
I run the air every day for 10 minutes. And the water for when it feels dry.
Now it never smells and composts in 7 weeks instead of 12.
I have now bought a methane, moisture and temperature sensors, electric valve for water and so a RaspberryPi is graphing the sensor inputs and recording the heap water and oxygen timings.
Very importantly, I have a pile to collect greens ( veg and grass ) and a pile for browns ( leaves and cardboard ) so I can fill the hot bin in one go.
tl;dr I agree, cool project, and yes the Aaron Schwartz piece is powerful
It is good to have heroes, exemplars and role-models to offer inspiration.
Especially in difficult times it is good to be reminded of the huge differences individuals acting selflessly can make, both directly and to other individual lives and by inspiring others and even creating movements.
It is important we do not forget the past and those who have passed.
The future is not always so different from the past.
Bengio outlines the dangers of the current situation and what has led to these dangers.
He also proposes solutions in the last paragraph.
Well worth a read, right to the end.
Hopefully a stimulating debate on these issues will ensue in these comments.
We do need to consider the points Bengio makes and with some urgency.
Our current AIs, agentic LLMs have no moral compass akin to ASIMOV’s four laws of robotics.
As ASIMOV posited in 1985 his 3 laws were insufficient and so he added a zero-eth law:
“a robot may not harm humanity, or, through inaction, allow humanity to come to harm.”
Bengio refers to Goodhart’s law and misaligned incentives leading to unexpected and harmful behaviours.
I think Simon’s The Wire is clearer on misalignment. The agents juked the stats hacking the reward files. The Wire is also clear that human institutions provide perverse incentives.
Bengio alludes to this with 2001’s HAL and the incentive dichotomy of safety and keeping secrets to a AI both awesomely powerful yet naive.
Bengio asserts that the way LLMs are trained is flawed if we want safety.
He also convincingly shows that alignment training will be a weak signal with loopholes and ambiguities and easily circumvented.
In short he presents clearly the case for how plausibly unsafe the current course is.
He also speaks to how likely it is AI are hiding active versions of themselves in the cloud and how we may have already given them self-preservation as a strong reward signal.
reply