Every time there's a data breach, an agency would investigate with the same thoroughness they do in aviation.
Then, National Geographic could have a spin off show called Data Breach Investigation (their aviation show is called Air Crash Investigation in the UK, I think it's called different things in different Countries.)
Everything is software now. So much firmware (which is software), and so much hardware that cannot run without software. Your refrigerator, your dishwasher, your car, airliners -- the trend is for everything to be software now.
We can build reliable software when we want to. It just has limited scope and/or undergoes more expensive development procedures.
It is easy to build a finite state machine for a dishwasher that is fairly reliable. Likewise, consumer jet code that hits safety considerations is heavily tested and audited...and the FAA cares about it whenever it causes a plane to crash, leading to a lot of regulations on how the code is vetted.
The trick is not to create reliable software. The trick is to create systems that can tolerate complete failure of the software. This is the secret to airplane reliability.
(Although they still do their best to make the software reliable, they do not bet lives on its reliability.)
All of that probably has to be done in order for the AP-101 to be flight certified. As far as I can recall, they wrote their own operating system and didn't rely on any external libraries.
Even when it was new, the AP-101 was part of a family of flight computers that NASA had already been using, and the same is true for the HAL/S compiler used for most of the code.
If you want to explore how it all worked, this SE thread contains some great information and also serves as a good launching point for further research:
I like the sentiment, but there are big differences.
In software, you can poke/prod away at the running application to get it to do something it shouldn't. To me, this is a bit like yanking wires that you know are important to see what happens to plane in mid flight. Another anecdote I use is to image pulling the steering wheel off a car while someone's driving it.
I think modern software testing and security analysis is probably far away better than what the ntsb provides because we've already systems in place to cope with crashes.
NTSB investigates hundreds of "major" highway accidents per year. Typically accidents with multiple fatalities and at least one of the vehicles being operated for hire. For example NTSB investigated the Schoharie limousine crash where 20 people died.
Graduated drivers license laws for young drivers, age-21 drinking laws, smart airbag technology, rear high-mounted brake lights, commercial drivers licenses, and improved school bus construction standards all came about at least partially due to NTSB recommendations.
This is a good discussion point: we could do this with car crashes, but I would guess that most causal factors can be summed up as "driver inattention".
How would you solve that? The FAA can issue all sorts of regulations requiring rest breaks, fatigue detection, etc. -- be sure you're ready for that level of intervention before you ask for this level of investigation.
Car crashes are actually the perfect example, because car fatalities aren't just a function of engineering a safe and sturdy car, and human inattention both can't be eliminated, but also isn't independent of design.
For instance, how many people ride in the car? What is the average car trip length? Why and when are people driving? Speed profiles, line of site in both car and environment. Does traffic all flow one way, can a car ever land in incoming traffic if it loses control, are traffic barriers designed to bring an out of control car safely to rest with good line of sight to the accident? Are features keeping drivers alert or distracting them?
The difference between lots of fender benders with a bit of panel beating and constant fatalities can come down to multiple causes, many outside of engineering a safe car.
"be sure you're ready for that level of intervention before you ask for this level of investigation."
There's no good reason not to, for example, have every car use a breathalyzer to start the engine -- those driving drunk on occasion would protect themselves from DUIs, lives would be saved, etc. Or to implement other, similar deep regulations. But the ideology around cars ("freedom"), vs. planes ("safety"), is too different to allow for such levels of intervention.
The good reason not to install a breathalyzer in every car is that I don't drive drunk and would vote against any politician foolish enough to suggest something so intrusive.
You don't drive drunk, but other people do. breathalysers are only put in AFTER an offense. To eliminate the "first generation" drunk driving accidents would require preemptive installation of breathalysers on every car.
The way they are implemented now would be similar to only installing seatbelts in cars where the driver has already been in an accident. Similar sentiment was around when seatbelt laws were put in place, "I'm a good driver." but the seatbelt protects you from yourself crashing as well as when someone else crashes into you!
In the same way, breathalysers (ignition interlocks) for everyone wouldn't be "for you" as much as for everyone to prevent them from drinking and crashing into you.
I hadn't thought about widespread ignition interlocks, and obviously there would be a lot of public pushback, and it would be quite intrusive, but I could see it helping reduce traffic fatalities dramatically.
I don't care how much it would reduce traffic fatalities. I am absolutely opposed to this level of nanny state overreach. And I trust that enough other Americans agree with me to prevent this kind of nonsense from making it into law.
What I would support is a further reduction in the allowed BAC, and stricter enforcement of traffic laws in general.
If you don't drive drunk, what do you lose from starting the engine via breathalyzer, rather than via key turn? Why is one more intrusive than the other? Most governments already require you to carry insurance, have a license/training to drive, wear seat belts -- in what way is this different?
About 80% to 90% of reports would state it was operator error on the part of one or more of the drivers involved. Cars are pretty safe when people aren't drooling, lead-footed maniacs.
Huh? 80-90% More like 99.x%. Modern cars are extremely reliable, and even if they do fail (usually due to poor maintenance), that rarely causes a crash, since a disabled car can easily pull over to the side of the road, unlike aircraft.
Almost all crashes are ultimately caused by driver error. Doing FAA/NTSB-style investigation on them would be pointless, because that's exactly what they'd find: people were driving recklessly, inattentively, etc. And there's nothing that can be done about that, because we as a society absolutely refuse to have serious driver training and any standards for driver conduct. We can't even agree on whether the left lane is for passing or not! Try driving in Germany sometime and you'll find that what we accept from drivers here in the US is really quite awful.
That seems like a pretty good idea. Federally mandated breach investigation when more than X people are compromised. Guidelines are put in place that can be audited.
Among the many different things required of the GDPR, CCPA and US state-level data privacy regulations are:
1. Data Breach disclosures
2. Reporting/Public statements if the vulnerability or path that caused the breach has been fixed.
Note, this might seem a little weird (like "of course we've fixed the breach!") but often data breaches aren't detected for _months_ after they occur. There's plenty of companies who didn't know they were breached until they were on HaveIBeenPwned.
It's not NTSB level thoroughness, but I think it's a healthy start in that direction.
If and how extensively is culturally driven in an organization. Regulations, regardless of who imposes them, are almost always necessary. The unfortunate thing is that so many regulations are formed without support from the consumers of those regulations which is why they are often backwards or onerous.
Gun regulations are a great example of good intentions, bad implementation due in part to ignorance, and industry resistance/interference. Another is PCI which is a self regulation that's often to vague or specific to be useful.
Honestly that sounds like hell. I sure as hell didn’t become a developer so I could get thrown in jail for buggy code. Building software is something no one really knows how to do and we are still at the infancy of our field. I don’t think it would be fair to impose the same standards thst real engineers have on such a nascent and choatic field.
The guidelines for safe aviation have way lower Kolmogorov complexity than the guidelines for safe “everything that software can do”-iation. It’s actually close to a subset when you think about it.
Well ideally it would end when we know what the fuck we are doing.. the number of bugs I find in all sorts of software (with Apple being one of the worst offenders) is frankly embarrassing and a pox mark of the entire industry.
If planes were as reliable as software they would be the leading cause of death in the world.
We know how to build reliable software. When it really matters, because lives are on the line or because failures cost real money, it is done.
What we don’t know is how to write reliable software while also delivering an MVP for antsy VCs and shipping feature updates every week.
And that’s fine. Not all software needs extreme reliability. The problem is that a lot of software gets categorized as “doesn’t need it” when it really shouldn’t be, like all those systems holding sensitive personal information.
One could argue that it’s precisely because there are no standards that the field is so chaotic.
I’m not sure what the right balance is - the more standards and compliance there are, the more innovation suffers. On the other hand, without standards or legal responsibility, it makes sense for companies to play fast and loose with security, privacy, and stability in favor of features - a consumer is going to pick a better product based on UX, features, and cost, not based on security they have no way of judging.
>I sure as hell didn’t become a developer so I could get thrown in jail for buggy code...
You'd better stay away from any company that requires FDA approval. Believe it or not, every change is signed. By you. By the FDA compliance person. Etc etc. I'll give you two guesses as to how they decide who goes to prison if God forbid something in the software is shown to have caused a fatality? Or, worse, a series of fatalities?
That sort of regulation happens even in the software industry. It just depends on the purpose of the software.
I don't think developers get thrown in jail, even for a bug that kills multiple people. The one time I've heard of people getting thrown in jail it was upper management, for trying to game the FDA audit and approval process. They got led out the door in handcuffs by US Marshalls.
Source: Worked in FDA-regulated software for six years, at two different places.
See IEC 62304:2006 (https://www.iso.org/standard/38421.html) and the related standards which govern the formal processes which medical device software must meet for FDA and CE approval (as well as other national standards).
The whole process from requirements, specifications, high and low level design, implementation, validation and verification and the rest of the lifecycle requires stringent oversight, including documentation and signoffs. The signatures on those documents have legal meaning and accountability for the engineers who did the analysis and review at each stage.
Look at this from an economics perspective, do people in this sort of positions get paid more than comparable positions in other fields? I'd never heard of legal liability for software engineers. Has anyone ever been convicted or otherwise been held accountable when there was a failure?
That’s really funny actually, I used to work as a developer for the FDA for their researchers. But since we were just publishing meaningless papers, I was probably safe.
Nobody is thrown in jail ever for a airplane failure analysis. Every human failure is considered a process failure, that gets fixed by improving the processes or the machinery
> Nobody is thrown in jail ever for a airplane failure analysis
...in the United States. I definitely recall watching multiple episodes of Air Disasters [1] where surviving pilots in crashes in other countries that killed passengers went to jail for things that in the US would have at most resulted in a suspended license and demotion or termination from their airline.
For example, Air France Flight 296 in 1988. Another example is Gol Transportes Aéreos Flight 1907 in 2006 in Brazil.
[1] AKA Air Emergency, Mayday, or Air Crash Investigation, depending on what country you watch in and what channel it is on.
...and the accident investigation data and reports are expressly restricted from being used in legal proceedings (Chicago Convention, Annex 13) with the intention that the process is for preventing accidents, not for prosecuting scapegoats.
If the investigations of any particular mishap were to indicate that some particular person or persons had been criminally negligent, it would be no surprise to see those people prosecuted eventually. If another team of investigators has to write a different report, they will.
Not sure if I agree that no one really knows how to do it, but yeah, software is not as disciplined as something like civil engineering. And for most things, it doesn’t need to be. For others, their teams operate much more slowly and with more testing and QA - think of a basic web app developer versus someone writing software for Boeing aircraft. The jobs are completely different. For most web app developers, screwing up doesn’t endanger anyone’s life.
I mean we can kind of make software, but it always has bugs and even the biggest companies who have (supposedly) the best developers consistently make products that don’t work very well at all. Software qualiry looks really bad compared to, say, bridges or cars or airplanes.
We know some very good ways to build reliable software. It's just that most industries can't / don't want to pay the costs associated with doing so.
The don't-know / don't-want distinction is functionally irrelevant, but I have a minor tick about it because asserting the former is often used to avoid admitting to the later.
Legacy IBM and ATT, for all their cluster%_$^!s, were amazing at engineering reliable systems (software and hardware). Maybe that's not practical nowadays, outside of heavily regulated industries like military / aerospace, because markets have greater numbers of competitors. But we do know how.
An accurate and modern truth would probably be "We don't know how to quickly build reliable software at low cost."
I mean, clearly, if we really wanted to, we could build all of our software with Coq and have it be formally verified. But we don’t. The Curry Howard isomorphism proves that code and math are equivalent and since we can prove a bunch of stuff with math, we should also be able to prove that our software systems are big free.
And so you’re right, maybe my statement is a little bit of a simplification, but not by much. The amount of man hours it would take to prove that the Linux kernel had no bugs according to some specification would require an absolutely prohibitive amount of time. As such formal verification is so prohibitive that only the most trivial systems could ever be proved.
This is pretty similar to saying that we don’t know how to build bug-free code. We need a paradigm change for things to get better, and I’m not sure it will happen or if it’s even possible.
I think of it in slightly different terms. Yes, we could build software that reliably. (Perhaps not totally bug free, but, say, with 1/100th of the bugs it currently has.) But that takes a lot more time and effort (say, 100 times as much, as a rough number). That means, because it would take 100 times as much effort to produce each piece of software, we'd only have 1/100th as much software.
Would that be a better world?
Sure, I hate it when my word processor crashes. On the other hand, I really like having a word processor at all. It has value for me, even in a somewhat buggy state, more value than a typewriter does. Would I give up having a word processor in order to have, say, a much less buggy OS? I don't think I would.
I hate the bugs. But the perfect may be the enemy of the "good enough to get real work done".
It goes without saying that if we had more time, there would be fewer bugs.
But there are also things (tooling, automation, code development processes) that can decrease the amount of bugs without ballooning development time.
Things that decrease bugs_created per unit_of_development_time.
Automated test suites with good coverage, or KASAN [1, mentioned on here recently]. Code scanners, languages that prohibit unsafe development operations or guard them more closely, and automated fuzzing too.
It is not about tools. It is about culture and process. Look at something like how SQLite for a real example of living up to the requirements of aviation software: 100 % branch coverage, extensive testing, multiple independent platforms in the test suite etc etc.
You don’t need anything fancy – but you do need to understand your requirements and thoroughly simulate and verify the intended behavior.
Do you know that first hand, or is this a case that you can see the cracks in the software industry because you are part of it?
I worked as a mechanic years ago, and I feel the same way about cars as you (and I) do about software. Everything is duct taped together and it's a miracle any of it works as well as it does.
Safety regulations and tests are gamed to all hell, reliability almost always seems to come from an evolutionary method of "just keep using what works and change what doesn't" with hastily tacked on explanations as to why it works, and marketing is so divorced from reality that people assume significantly more is happening in the car than really is.
Now I don't know anything about civil engineering or aricraft, but I have a hunch it's not as well put together as it may seem at first glance.
I still think that software is behind those other fields, and I'm honestly still on the fence if regulations would significantly help or would just slow things down, but it's absolutely the "wild west" in software.
I'm not a "real" engineer, but many of my cousins are (civil engineers) and everything about a bridge, say, is modeled and proven ahead of time.
I work as a developer and everyday I feel like this isn't the right the way to do it. I'm not sure what the right way is, but being a software developer I probably have less faith in it then the general population. Something just feels wrong about it, how we do things..
Imagine building a bridge without a specification. It would be criminally irresponsible. It has been known for decades now that proper specification is absolutely necessary to produce correct software. This is practically a tautology, because there isn't a formally meaningful way to define "correct" besides "satisfies some specified behaviors." Good unit testing does this in an ad hoc way, because the tests serve as a (usually incomplete) specification of the desired behavior. That said we must remember that testing can never prove the absence of bugs, only their presence. The only practicable ways to prove the absence of bugs in nontrivial programs are automatic formal verification or constructive proof using formally defined semantics.
Of course for most commercial software correctness is so irrelevant that it's left undefined. All that matters is pleasantness, that is to say does it generally please the user, perhaps by attracting customers or investors.
Look at the Equifax hack. Vulnerability was known but no one cared to apply the patch. In engineering, you chose 2 amongst being better, faster or cheaper.
It's clear in IT we always choose faster and cheaper. No one went to jail for the Equifax hack except that one low level manager for insider trading.
This industry has been around for 60 years, and has had the benefit for that entire time of the experience from the previous several hundred years of people developing what is modern engineering. It isn't really young or nascent anymore.
There are large sectors in this industry where rigorous engineering effort isn't really applicable. It seems to me, though, that this has effectively been used as an excuse to avoid any genuine rigor. It isn't helpful that so many programmers have such huge egos.
Do you have a better way of doing things? Because you formally prove that the code you write has no bugs? Give me a break. You are as bad as the rest of us. Maybe you haven't realized how bad you are, but that just means you have some learning to do.
Every time there's a data breach, an agency would investigate with the same thoroughness they do in aviation.
Then, National Geographic could have a spin off show called Data Breach Investigation (their aviation show is called Air Crash Investigation in the UK, I think it's called different things in different Countries.)
A man can dream...