drivers came to trust it way too quickly. Drivers who were supposed to be closely monitoring the system instead spent their time looking at their phones, putting on makeup, and other distractions.
Really hope no one dies because of Tesla's autopilot. This is a real risk.
Quoting from below: if the car does it right 99% of the time it's just human nature to stop paying attention. The article makes a great point that slowly improving the autopilot with a person behind the wheel is super dangerous because we get lulled into a false sense of security.
There are some spectacular failures that are very scary because the car does something that a person would never do, unless unconscious, like this one: https://www.youtube.com/watch?v=LfmAG4dk-rU
You don't hear much about it probably because Tesla fanboys are plenty and rabid, so people avoid talking about it online.
The defence is usually stats about human drivers crashing more often and it makes sense until you dive deeper into the numbers because these stats are usually oranges v.s. Apples. It feels like they have some playbook with statistics to slap when someone says something negative. If that doesn't cut, they say that the victim should have followed the manual that says "your attention should always be on the road" then proceed posting a video about how thanks to the latest update they can sleep drive to work and attach a banana to the driving wheel to disable the attention safeguards.
If someone asks how is this an autopilot, there are usually two ways to handle it:
1) Autopilot is just a brand name, the self driving software is in beta, so the victim should pay attention all the times.
2) Autopilot is like on the planes, so only fools think that it is autonomous, therefore it was working as intended but they should have been using it like an airline autopilot. Crash due to user error.
I'm actually a fan of Musk and Tesla but I feel like the community engagement is very unhealthy and lacks scrutiny due to the "online army" of his.
I'm surprised you don't just link to the source that regularly updates itself on all Tesla deaths, and makes a point to cite its sources. Note this is not just for Autopilot, but all deaths occuring from Tesla vehicles (including people not in the vehicles). There are tags for various things like "Autopilot" and "Pedestrian/cyclist"
I'm optimistic for this area of tech and research in general, but agree we need to stop benchmarking against average human crash rates.
Although anyone can be hit at any time, the distribution of human crashes is not purely random. People who drive compromised, for example, are way overrepresented in those events.
So, theoretically, the tech could get to a point with a lower than expected crash rate for humans generally, but still increase your personal crash likelihood.
If you believe you are a better than average driver when manually driving, why couldn't you also be better than average at intervening when autopilot is driving?
Humans don't have forward facing radar nor 360 degree always active vision, so in some cases, cars can see hazards that even a perfect human driver cannot.
It's a natural human tendency to get distracted, especially when we are not engaged with a task (as with autopilot). Driving manually physically engages your body in the task, making it harder to get distracted. On the other hand intervening is subject to distraction and longer response times.
Yes, that's captured by the defences I listed above. You say that this is an autopilot, you make them purchase self driving capability package, you say that the car comes with all the hardware necessary for self driving, you fans post videos online about cars driving themselves and in the small print you say that it's not autopilot but Autopilot and the drivers must pay attention all the time and you put very weak safeguards to enforce that attention.
It's simple plausible deniability for Tesla and the fanboys. Good enough to keep them off the hook, legally.
That's such a hard reach and a bunch of word mincing.
To this day in 2021, many people still don't wear seat belts. Even though that's a solved problem. Some people will do stupid things. It's human nature.
Sure but car makers don't imply or give impression that you don't need seatbelts thanks to the collusion detection system or the airbags. They don't sell seatbelt-free system that only in the small print says that the seatbelt must be worn all the times except for off-road driving.
You're being a bit pedantic here. Every time you engage autopilot, it literally tells you (in bold letters) to "Always Keep Your Hands on the Wheel" and to "Be Prepared to Take Over at Any Time". Keep abusing it and it will actually disable it for the rest of the drive. Did you know that?
There's no "impression" being given. Like I mentioned. There will always be a small fraction of irresponsible humans that will do dumb things. No amount of engineering can fix that.
Case in point is this video: https://youtu.be/VS5zQKXHdpM?t=88 Her mom is even helping this kid film this stupid act just for clout and views.
Tesla owners grilled him and said what he's doing is dangerous and irresponsible. He then deleted them all and turned off commenting. But people like to demonize Tesla owners. Which I find bizarre.
I'm not so sure the second one is something a person would never do. The second car also brakes very late, for instance. Did autopilot break at the last second for the Tesla, or the driver? I'm also bearish on autopilot and am not saying the Tesla performed well, but I bet human highway drivers run into stationary traffic all the time too.
One has to distinguish between "autopilot" and "FSD". The "autopilot" is used for driving on highways and mostly uses radar for obstacle avoidance. It works great with handling traffic, that is cars moving around, but not for static obstactles. The problem is the low spatial resolution of the radar, so nonmoving obstacles are difficult to distinguish from the background reflections.
"FSD" is creating a 3d-model of the environment based on the camera image. But so far it is only active outside of highways. This should be much better at avoiding static obstacles. In any case, they are different systems so experiences with one cannot always be transferred to the other.
Yes - 'autopilot' is auto-steer combined with adaptive cruise control. Now, Tesla's technically do have 'dumb' cruise control that doesn't reduce speed in response to a car getting close, but that configuration is rare since you need to specifically call in to buy one without autopilot and is probably not what the article was referring to.
I think the parent comment was making the same point as you. We, but mostly Tesla, should be educating people on why autopilot doesn't mean it drives itself in all situations. While buying a car[0], the furthest the webpage goes is:
> automatic driving from highway on-ramp to off-ramp including interchanges and overtaking slower cars
And while it does all of this very well right now, there still is that piece of text at the bottom:
> The currently enabled features require active driver supervision and do not make the vehicle autonomous.
Honestly Tesla is doing the bare minimum here to have plausible deniability, but the text at the bottom really should be the same size as the text above it.
People will die. It will also save lives. It’s probably already prevented thousands of accidents, some of which would have been fatal. It just doesn’t make national news when your car moves you out of the way of danger.
I am not so sure about this. Modern cars come with a lot of safety features, like automatic braking and adaptive cruise control. Sure, if you replace all 1995 Honda Civics with Teslas, you will save many lives. I don't think you could save any lives by replacing all modern 2020 cars from other manufacturers with Teslas.
They’re also designed and marketed very differently. The lane keeping in my Subaru isn’t good enough for me to fully trust it, so I monitor it carefully.
I suspect that there’s probably a local minimum for safety, where the autopilot is good enough to lull users into complacency, but not good enough to actually be safe.
> The lane keeping in my Subaru isn’t good enough for me to fully trust it, so I monitor it carefully.
We have a Hyundai Palisade with Adaptive Cruise Control and Lane Keep assist. Amazing for reducing fatigue (the small little corrections). Not good enough to let you get distracted. I always have to be mentally ready to hop in.
I think it's a perfect solution until we have true FSD that can handle absolutely all conditions and situations.
It's even a risk to Waymo because, despite being responsible about it, Tesla made people either scared of autonomous cars or forced out legislation restricting it.
There are six cases of death where autopilot was confirmed to be engaged at the time of the incident, with 15 total that claim AP was engaged but had not been proven (via post-crash analysis or otherwise).
Based on reviewing the articles linked to the data, only one includes a lawsuit that claims the driver fell asleep before the crash. Doing this would either require a weight on the wheel to simulate constant torque (otherwise, on the highway, you have to apply torque every 30 seconds and it'll lock you out from using autosteer if you ignore it for roughly a minute), or falling asleep within 30-45 seconds of an incident, which is entirely possible.
I'm fully aware - there's even a TSLAQ at the bottom of the page which is ironic given the recent stock surge. Nevertheless, it looks like it's the only source that tries to count deaths involving autopilot, so you can take their number then go through the articles to make a more informed conclusion.
> In the 4th quarter, we registered one accident for every 3.45 million miles driven in which drivers had Autopilot engaged. For those driving without Autopilot but with our active safety features, we registered one accident for every 2.05 million miles driven. For those driving without Autopilot and without our active safety features, we registered one accident for every 1.27 million miles driven. By comparison, NHTSA’s most recent data shows that in the United States there is an automobile crash every 484,000 miles.
That doesn't tell the story. Autopilot is essentially forced off for anything but highway, so they are missing the most dangerous but are racking up millions of the least dangerous miles quick.
I can't actually find anything suggesting that the most dangerous roads are not highways. I really tried. Perhaps there are more small accidents on local roads, but they would tend to be less dangerous. The evidence suggests that highways are significantly more fatal[0].
Also, Autopilot is not essential forced off for anything but highways. That is just factually incorrect. Notably, straight-aways are where Autopilot works best and are the most dangerous by crashes.[1]
I found data suggesting that a lot of accidents happen because people rear-end stopped cars at intersections[2] and issues with navigating intersections[3]. That is an area where Autopilot is pretty great today.
IIRC for humans it's like 1 death per million miles driven. I believe Waymo is closing in on that. Not sure if Waymo handles the variety of conditions though. Musk and Tesla seem very flippant in their marketing.
I’m not sure if I feel easy about this “statistic” - deaths per mile?? These are humans, and more importantly to the business - paying customers.
If my product has the possibility of killing my customer, I’d probably not be comfortable with making it market ready (without massively improving its safety features).
Here’s the thing, every car system will fail at some point and potentially cause a death. There is no such thing as perfect safety. It’s really important to quantify things like fatalities per million miles driven. At some point, you push your system into production because, even though at some point the system will fail and kill someone, it will in the mean time save dozens or hundreds of people from dying in preventable accidents. What is critically important is being confident that your system is safer than the alternatives that your customers would use if your product doesn’t ship.
Most products can kill someone. Entire lines of products routinely kill its user. Society deems them worth the risk. A no risk life is sitting alone in a room, where you will die to obesity.
Right, but if the car does it right 99% of the time it's just human nature to stop paying attention. The article makes a great point that slowly improving the autopilot with a person behind the wheel is super dangerous because we get lulled into a false sense of security. I hope Tesla just keeps it to highway lane keeping until they're ready to roll out better sensors.
This principle is glaringly obvious in retrospect, and seeing it put so succinctly actually changed my mind. I previously thought Tesla's approach was an achievable long shot, but I'm now convinced it's fundamentally wrong.
The safety-driver model pursued by Waymo and every other autonomy startup is the only responsible way to train an autonomy network because the drivers are specifically trained to look out for edge cases, versus consumers who will easily be lulled into complacency once the system looks superficially like it's running at level-5 competency.
While I agree that the Waymo, et al., way is the only responsible way, that is not an argument against Tesla’s way being workable, just that it is grossly unethical.
coffee cups say "be careful this is hot", but the regulator allows a product called "Full Self Driving" to be sold - I think ultimately consumer protection will step in here like it has in Germany [0]
Krafcik talks about this very issue in the article on how driver assistance systems give false sense of security to the driver.
> The Waymo team believes its own early experience—when it was the Google self-driving car project—bears that out. In the early 2010s, Google developed a driver-assistance system similar to today's Autopilot and considered selling it to automakers. But when they let Google employees test the software on public roads, they found that drivers came to trust it way too quickly. Drivers who were supposed to be closely monitoring the system instead spent their time looking at their phones, putting on makeup, and other distractions.
> The fundamental challenge here is that the better a driver-assistance system gets, the harder it is to get drivers to pay attention, and the less likely they are to be prepared if the software makes a mistake. The Google team didn't see a good solution to this problem, so they completely changed their strategy. They focused on building a self-driving taxi service that would never have customers in the driver's seat, relying on trained, professional safety drivers to oversee the software during testing.
There's two levels of risk, Tesla owners taking themselves out, and then there is innocent bystanders being taken out.
It's pretty infuriating if you consider that we're all put at risk.
You've always been at risk of other road users, but that is well understood when you get in a car and drive somewhere.
What's new, and on top of that risk, and not well understood is the risk due to drivers mis-using a half-baked feature named "autopilot" as if it's actually an autopilot, because Musk insists that it works when it clearly does not always work.
People can die. Honestly in my opinion it’s worth it. Will it happen to me? No, because I’m extra cautious and won’t slack off but to make progress quickly these things must be done. If you really want to be hyper cautious, you can always make the case that it’s saving lives that would otherwise be lost.
> No, because I’m extra cautious and won’t slack off
A big issue is that some random guy in the Tesla behind you may not be so careful. So you cold still get into an accident because of Tesla's recklessness.
Really hope no one dies because of Tesla's autopilot. This is a real risk.
Quoting from below: if the car does it right 99% of the time it's just human nature to stop paying attention. The article makes a great point that slowly improving the autopilot with a person behind the wheel is super dangerous because we get lulled into a false sense of security.