I appreciate that the article correctly points out the core design flaw here of LLMs is the non-distinction between content and commands in prompts.
It’s unclear to me if it’s possible to significantly rethink the models to split those, but it seems that that is a minimal requirement to address the issue holistically.
The flaw isn't just in the design, it's in the requirements. People want an AI that reads text they didn't read and does the things the text says need to be done, because they don't want to do those things themselves. And they don't want to have to manually approve every little action the AI takes, because that would be too slow. So we get the equivalent of clicking "OK" on every dialog that pops up without reading it, which is also something that people often do to save a bit of time.
It absolutely is a problem with human assistants (though, of course, those are currently much smarter). But people can and have scammed assistants to steal money or personal details from their bosses. Phishing and social engineering are exactly forms of this same vulnerability. Of course, human assistants are smart enough to not get phished by, say, reading a book that happens to contain phrases that are similar to commands that their boss could give them, but that's just the current difference of intelligence and the hugely larger context windows humans still have compared to LLMs.
Ah, it's like the good old days when operating systems like DOS didn't really make the distinction between executable files and data files. It would happily let you run any old .exe from anywhere on Earth. Viruses used to spread like wildfire until Norton Antivirus came along.
You can't easily convince a remote computer to curl | bash itself. Worms spread because remote code execution was laughably easy back then. Also because computer hygiene was abysmal.
LLMs are more than happy to run curl | bash on your behalf, though. If agents gain any actual traction it's going to be a security nightmare. As mentioned in other comments, nobody wants to babysit them and so everyone just takes all the guardrails off.
That's not a valid vlan ID for most vendors (Reserved) and can also be a security vulnerability, as it can allow traffic to elevate its Class of Service and hop vlans via this method.
The obsession has at least one interesting question attached: ownership of real property. At the limit, at least, that becomes a genuinely interesting question.
> What problem, other than judicial review, does it solve for the executive branch?
It’s fairly obvious on its face the concern of the EO is not judicial review but about agencies that nominally are past of the branch the President heads determining interpretations of law contrary to what the head of the executive desires.
And, it does genuinely seem weird to have an executive branch where the head of that branch doesn’t actually control things.
The negative reaction is entirely because of the current executive head, but no one would bat an eye if this were Barack Obama reigning in some executive agency interpreting, say, immigration law in opposition to DACA.
> Is the head of the executive an expert in all things? And capable of communicating those expert desires with perfect clarity?
It's not their job to be the expert, it's their job to be the decisionmaker. That's why you have a head. If two federal agencies want to interpret the law differently, it's more important to pick one interpretation and apply it consistently than to get it perfectly right.
> Why have courts if the executive head can sort out all legal nuance themselves?
Checks and balances are important but so is the ability to actually do things occasionally. Independence for the court system is good. Independence for every individual federal agency isn't.
Sure, but an executive basically just implements the laws, not decide what they are. Given the US system this is important, as it's quite possible for the President not to have a majority in the legislature.
So you are saying that we are meant to believe the problem is that people (who exactly?) were formulating and acting on their own interpretations of the law, independent of either the executive or judiciary branches of government? Hmm. If the problem is so immediate it requires an EO, there must be some salient examples you can point me to.
I would react the same way to anyone who had announced we would never need to vote again, who had previously pardoned felons convicted in an attempted coup, and who was now centralizing governmental oversight and power. There is no comparison to another president in the last 150 years or more.
> And, it does genuinely seem weird to have an executive branch where the head of that branch doesn’t actually control things.
It's not weird at all, and it's not true that the head of the executive doesn't have control over those agencies. The head of the executive names their leadership - that is a huge amount of control. And it is enough control - the government isn't some top-down system serving at the pleasure of the president. It is a system for implementing the rules set out by Congress and the courts (subject to the President's judicial review powers), that the president coordinates.
The very title of "president" was chosen by the founding fathers to evoke the largely bureaucratic role they had in mind. It's not supposed to be a position of prestige or control like a dictator or ruler, it's similar to the role of a committee president: someone who oversees the functioning of the committee, and steer the general agenda, but who doesn't otherwise get to decide for the committee.
> no one would bat an eye if this were Barack Obama reigning in some executive agency
Obviously a counterfactual we can never truly know but I'd remind you that the right were offended when Obama wore a tan suit and was using Dijon mustard. I'm pretty sure they'd "bat an eye" if he were attempting even 1% of the shenanigans that Trump is pulling.
The problem is that the two theories can have competing indications as to how to prepare. Specifically: should we do gain of function research, or is that foolish — depends on how you read what happened in 2020.
I think the parent is arguing that lab leak is plausible, even if it wasn’t certainly the cause. GoF is foolish if you think the lab leak was remotely plausible.
Most folks had no idea about the sort of GoF being done, and the attitude of many researchers (highly dismissive of risks) should worry us a lot.
We should also be more worried about zoonotic transmission too, and press harder to ban wet markets.
I don't think these conclusions compete, that’s the point; the actual fact of the matter regarding origins doesn’t much affect the weight of the damning evidence.
> GoF is foolish if you think the lab leak was remotely plausible.
Even if you don’t think the lab leak was the source of COVID-19 virus, we know for a fact that lab leaks occur even at the highest level security facilities.
I’m not sure about gain of function research one way or the other, I’m just commenting that leaks will happen.
There's a couple of probability distributions we don't know. And whether this leaked in Wuhan or not doesn't affect them.
1. What's the probability distribution and damage distribution of GoF research lab leaks? It's not zero-- it likely has enormous long tail risk. But:
2. What's the probability distribution and damage distribution of not knowing as much about how gain of function happens in the wild? Because nature is doing some of these GOF experiments on its own, without much effort at containment.
Nobody in this thread seems to know what gain of function means. It's a very broad term covering a large percentage of all virology research. If you ban it, you might as well say that we don't want to do any research into understanding viruses from now on.
When you compare the massive risks of spillover from animal populations, which have millions of interactions with humans every minute of every day, with the risks from a small number of highly contained biology labs, the ratio between the two risks is so enormous that this entire discussion is absurd.
You're right that we should still do the research. But we should be doing it on an island, or a ship at sea, with supplies delivered by drone, and as little population exchange as possible.
That depends on whether the research increases the risk of a pandemic by any appreciable amount, compared to all the other things humans do.
It's kind of absurd that he have hundreds of millions of farmers and hunters interacting with infected animals every day, and nobody cares, but then we have a few researchers interacting with the same viruses under highly controlled conditions, and that's what we're worried about.
The reason is that people have watched too many sci-fi horror movies and listened to too many xenophobic / fear-mongering politicians who want to find scapegoats.
The problem with imposing even tighter conditions on research is that you end up making the research much more difficult, expensive, unattractive to actual scientists who have to live their lives. And all that for security theater, just to pander to ignorant politicians who won't actually be satisfied.
I don't know how you can read about the half dozen or more documented cases of lab leaks in this thread - regardless of what you believe about covid - and call heightened restrictions for experimental work with human-infectious viruses security theatre. It's not sci-fi horror movies folks are worried about, it's people making mistakes the way all people do. The way you seem to be doing.
We take the effort to air gap infected or security critical IT systems, but can't be bothered to air gap humanity from existential threats. If protecting all of humanity from the next pandemic is too much work for virologists, maybe it's best that they consider another career?
The examples of "lab leaks" being given in this thread are things like a large-scale human vaccine trial not using a properly inactivated virus. This is not a "lab leak" in anything like the sense of that is being alleged here.
> We take the effort to air gap infected or security critical IT systems, but can't be bothered to air gap humanity from existential threats.
My whole point is that we're not air-gapped in the first place. Millions of humans are interacting with infected animals every day, under conditions that are much less safe and controlled.
Imagine if all of your data had been leaked to the public internet, was mirrored across a dozen websites, and was being downloaded 1000 times a day, but then one security researcher had your data on an encrypted drive, and only read it on an air-gapped computer. Would you be more worried about the one security researcher, or the dozens of publicly accessible websites?
> If protecting all of humanity from the next pandemic is too much work for virologists, maybe it's best that they consider another career?
They are protecting you, and the thanks they get is that you scapegoat them, hound them online, and cheer when they get fired. You should be grateful that people like Shi Zhengli at the Wuhan Institute of Virology worked so hard on understanding coronaviruses for decades, and warned about the risks of a pandemic. Instead, without any knowledge of the subject, you're participating in a witch hunt against her and her colleagues.
> My whole point is that we're not air-gapped in the first place. Millions of humans are interacting with infected animals every day, under conditions that are much less safe and controlled.
Factory farming is indeed a dangerous breeding ground for infectious disease which needs to be addressed. Farmers, however, are not performing gain of function research on the diseases in their herd. The largest danger of factory farms seems to be the widespread application of front line antibiotics, which is another issue entirely. Attempts to conflate the two are disingenuous at best. Unscientific whataboutism at worst.
> people like Shi Zhengli at the Wuhan Institute of Virology worked so hard on understanding coronaviruses for decades, and warned about the risks of a pandemic. Instead, without any knowledge of the subject, you're participating in a witch hunt against her and her colleagues.
I've worked in science for more than a decade. I have lab experience. I have never named or witch hunted anyone as you seem to have. I have advocated for the most basic level of isolation of potentially dangerous experiments, and this is your response.
Thank you for demonstrating the arrogance which underlies the problem.
> Farmers, however, are not performing gain of function research on the diseases in their herd.
The phrase "gain of function" has become a general-purpose bogeyman, but 99% of the people using it have no idea what it means.
Farmers are interacting with viruses that are far more dangerous than the gain-of-function viruses. Gain-of-function experiments are generally just characterizing properties of viruses that already exist out in the wild. An effective way of doing that is to insert a component of the wild virus into a virus you can already grow and have characterized in the lab. The lab virus gains a function, but that function already exists in the wild.
> I have advocated for the most basic level of isolation of potentially dangerous experiments
Shi Zhengli and her colleagues are taking far more than the "most basic level" of precaution. Yet you're participating in the witch hunt against her and the virology community.
The actual future risks don’t change based on which specific origin happened.
The correct response is likely to spend significantly more on doing actual research and a great deal on making sure everyone is well contained. It’s likely a good idea to locate such labs outside of highly populated areas as part of a defense in depth strategy.
> The correct response is likely to spend significantly more on doing actual research and a great deal on making sure everyone is well contained.
Strongly, strongly disagree. When even a teeny risk of escape means that millions of people could die, I think a much better argument is to simply make certain types of research completely off limits.
I'm certainly not the only person who thinks this. Zeynep Tufekci, who in my opinion had the most rational commentary during the pandemic, argued that much virus research just doesn't work from a cost/benefit analysis. For example, even if the root cause of COVID wasn't a lab leak, it's probably not a great idea having researchers milling around bat caves collecting sick bats and what not - it's very possible a zoonotic virus made the jump not necessarily in the lab but from researchers specifically looking for zoonotic viruses.
Looking at risks alone always biases you to avoid doing anything.
The benefits of research here are also human lives. So doing nothing has a real cost and the benefits extend indefinitely into the future.
Suppose you’re deciding between a 1% chance of a lab leak costing 10 million lives and a 20% chance you save 50 million lives over the next 100 years. That’s heavily weighted towards doing something, while still carrying significant risk. Some people would still say the risks aren’t worth it, but it’s not an obvious decision.
I think you need to discount possible farther future benefits, because so much change can intervene and make the analysis invalid.
That is, when people want to do something-- risks tend to be understated and possible future benefits tend to be overstated.
I don't back the precautionary principle, but I do think risk in cost-benefit analysis has to be viewed from a pretty cautious place, in general (not just science).
Ultimately, we don’t know the actual benefits and I just picked numbers from thin air to illiterate a point. But yea linear extrapolation of such estimates hundreds of years into the future is nonsense.
What is there to research with GoF that could be worth the massive risk? We had a vaccine for COVID in a weekend. Approval and manufacturing where the bottlenecks.
There’s a lot of GoF research on a lot of different diseases with a wide range of goals.
One goal for disease likely to cause pandemics is ultimately to create better treatments for those already infected. There’s a long lag between a vaccine being designed and scaling production and distribution to actually protect people. That means there’s going to be a lot of people infected in an outbreak, including many vaccinated people.
Are there any examples of medication that was developed for a disease that came out of GoF where the medication was approved and preventive mass production took place?
My understanding is success have come more from protocols more than medication.
Take antimicrobial resistance, you need to understand how microbes gain resistance by actually creating resistant bacteria/fungi etc before you can develop efficient countermeasures.
With COVID there was a lot of confusion around using masks and disinfecting public spaces in the early days. A better model of the disease could have been really useful both in the early days and how people responded to mixed messages.
House Republicans on the subcommittee probing the origin of the Covid-19 virus appear to have inadvertently released a trove of new documents related to their investigation that shed light on deliberations among the scientists who drafted a key paper in February and March of 2020. The paper, published in Nature Medicine on March 17, 2020, was titled “The Proximal Origin of SARS-CoV-2” and played a leading role in creating a public impression of a scientific consensus that the virus had emerged naturally in a Chinese “wet market.”
The paper was the subject of a hearing on Capitol Hill on Tuesday, which coincided with the release of a report by the subcommittee devoted to the “Proximal Origin” paper. It contains limited screenshots of emails and Slack messages among the authors, laying out its case that the scientists believed one thing in private — that lab escape was likely — while working to produce a paper saying the opposite in public.
The newly exposed documents include full emails and pages of Slack chats that were cropped for the report, exposing the “Proximal Origin” authors’ real-time thinking. According to the metadata in the PDF of the report, it was created using “Acrobat PDFMaker 23 for Word,” indicating that the report was originally drafted as a Word document. Word, however, retains the original image when an image is cropped, as do many other apps. Microsoft’s documentation cautions that “Cropped parts of the picture are not removed from the file, and can potentially be seen by others,” going on to note: “If there is sensitive information in the area you’re cropping out make sure you delete the cropped areas.”
When this Word document was converted to a PDF, the original, uncropped images were likewise carried over. The Intercept was able to extract the original, complete images from the PDF using freely available tools, following the work of a Twitter sleuth.
I understand how WIV could have caused a pandemic, but I don’t understand the other direction — how WIV doing gain-of-function would help the situation in which there was a zoonotic origin, eg at the wet market.
The person I was replying to said that our opinion on gain-of-function should depend on the origin — but I don’t understand how gain-of-function would have helped at all. Only how it poses a risk, whether or not this particular virus was such an example.
The theory is quite clear, if you know which strains could hop to humans then you can prioritize monitoring them, just like we monitor influenza types in animal populations now.
The problem is just that P(avert catastrophe) is fairly low, and P(create catastrophe) is substantially higher.
I’m pretty sure there is funding on both sides; NIH vs DARPA for example. I suspect that NIH funds a lot more here and I’m skeptical that DARPA funds labs in China for this.
The experiments WIV were doing were specifically targeted at identifying wild-type viruses that could cross over. This is not where you would start for a bioweapon. (Unless, tinfoil hat, you want to start a pandemic that looks like a zoonotic event. But that’s not the threat model the US military is worried about in the research they fund.)
> It's not about nature. It's about biological weapons. If Russians will create a new biological weapon (they do), then we must have a cure before they will use it in their fight with NATO.
And the best solution is to research this... in China?
If you want to research a cure from engineered virus from one hostile country, paying another hostile country to do virus engineering research for you is surely great logic.
I am not in the sector but AFAIK there has been no direct benefit from GoF. There is huge potential which some experts believe outweigh the risks, which is why it is controversial, and that is why it has been politicized.
Protecting the funding and ability to continue the research would explain why scientists have a preference for spillover as opposed to leak. Then there are the politicians...
Gain of function research in a lab you can't (and more damningly won't) prove had adequate precautions is bad regardless of the source of Covid or the utility of the research. We should be taking it as a wake up call to make sure standards are appropriate and the institutions to make sure those standards are met are strong.
At the very least, we hopefully learned not to subsidize and encourage gain of function research at labs that were already known pre-Covid to have poor hygiene and containment practices.
The question of whether we should do gain-of-function research is a fairly complex cost/benefit analysis. The precise cause of the 2019 pandemic is only a very minor variable in that analysis, because that specific outcome doesn't change the underlying probability of a lab leak. More to the point, do we realistically believe that everyone will stop doing it, even if there's a credible international moratorium? If not, then we need to plan accordingly.
> The question of whether we should do gain-of-function research is a fairly complex cost/benefit analysis.
Has there ever been benefit to such research? People fall back on wishy-washy "we could learn ___" when trying to defend it, but with how long it's been going on have we ever actually had a solid benefit from it?
Isn't the above-the-board justification for gain-of-function the promise of built-for-purpose microbes? The dream of "we spilled a million litres of toxic soup, here's a jar full of bacteria that eat that stuff and poop out useful compounds" or "let's make a virus that selectively over-infects tumours to weaken them?"
We might have the usual problem with every high-powered technology, from the fission reaction to the silicon fab: the underlying science is viewpoint-neutral, but people will be overwhelmed by doom scenarios associated with it.
Gain-of-function in the virology context doesn’t mean creating helpful microbes/viruses. It means purposely engineering pandemic-caliber viruses so that (the theory goes) we find them before evolution produces them naturally and so have time to study them and create vaccines before they are widespread.
As far as I know you're not missing anything and this is why gain-of-function research was banned in the US for a while. EcoHealth Alliance outsourced it to China in the mid-2010s because of the ban, so technically none of it was happening in the US.
> It means purposely engineering pandemic-caliber viruses
This is not true.
GoF includes any research that amplifies specific characteristics. Transmissibility or severity of infection are just two of those possible dimensions.
For example, the research that enables us to produce insulin (and tons of other biologic medicines) with E. coli is GoF.
I lean on the side of banning GoF that's designed to increase transmissibility of a contagion, but that is indeed just a subset of GoF generally.
Fair enough that it has that meaning more generally in biology. My point is 100% of the policy discussions about it are referring to that particular subset—no one means producing insulin when they talk about the risks of GoF.
Yes but this is what causes confusion when scientists push back against proposed bans which seems like a legitimately insane and evil position to take.
We can sharpen the language and say "ban GoF research that increases transmissibility of infectious disease", for example.
I think the best term of art is ePPP (enhanced potential pandemic pathogens), which clearly limits that scope. Academics use that reasonably often, though politicians and the general public unfortunately don't yet.
There's also GOFROC (gain of function research of concern). That's better than just GOF, but rather vague.
I believe sucrose and fructose are processed differently — specifically fructose is processed by the liver so can have the same issues as alcohol there.
We are talking about black plastic spatulas and spoons, right? Existing cookware? Non-plastic replacements' price is less than 5 euro at IKEA, possibly less at local kitchenware stores... What urgency would there be apart from 'oh I need to pick up a wooden spatula for eur 0.50 next time I go to IKEA'...
