> Americans drive an aggregate of 3 trillion miles, while Tesla drivers have done 100 million. That's well over an order of magnitude difference.
The data on 100 million is probably enough to compare with the 3 trillion miles. The populations don't need to be equal to compare them, just big enough that they are random and distributed enough.
When trying to get popular opinion for, say, an election -- sure.
However, there was one fire over 100 million miles. The problem isn't the 100 million, it's the 1 fire. This wasn't a controlled experiment, either -- they just stopped the clock as soon as the first fire happened, and multiplied. A week ago, they could have said "You have exactly a 0% chance of your Tesla catching on fire" and have been right by this logic.
To think of it another way -- let's say you get lucky and get a hole-in-one your 10th time golfing. Does that mean you'll have 10 hole-in-ones if you golf 100 times? Doubtful.
EDIT: Also, don't forget that Elon is mixing numbers. There's no fire if someone doesn't run over something. All these numbers show is that the average driver is more likely to run over something. Of course Tesla drivers run over fewer things -- there are no 16 year old kids texting while driving a Tesla... yet.
Forgetting about all of the (completely legitimate) concerns with this comparison aside from statistics, we can get a general idea from simple statistics how convincing having the first crash at 100 million miles should be.
For the purpose of our simple modeling, suppose that there is a constant risk per mile of the car catching fire, making an exponential model reasonable. Under this model, observing the first fire at 100 million miles would give a 95% confidence bound on the rate of fires of about one fire every 33 million miles.
If we're comfortable with the stated rate of about one fire every 20 million miles for other cars, then this would give a 95% confidence upper bound on the Tesla's rate of fires at about 60% of a normal car's rate. This isn't the 20% that Elon's statement would imply, but it does suggest a difference (which could just be due to other problems with the comparison).
It probably penalizes them, assuming the no-fires-for-100m-miles was not a fluke. There's no way to know for sure without knowing the true distribution of fires per mile.
By your "hole in one" argument, Tesla is "doubtful" to keep up their pace of one fire per 100 million miles. In other words, their "true" fire rate is actually less than one per 100 million.
I think a Poisson distribution is what you're looking for here. Roughly, if events happen independently and with a fixed probability per time interval, you get a Poisson distribution. Poisson distributions apply to a lot of things, so it's a very useful distribution to know about.
But since the number of cars is increasing, it's not a Poisson distribution; if the chance per car per time is constant, you'd expect the time to the next fire to be shorter.
Presumably it will go up to some degree over time as the average age of tesla's fleet converges with the industry average, as older cars are more likely to experience malfunctions/breakdowns for obvious reasons.
The median car in the US is ~11.6 years old[1], while Tesla's oldest vehicles were released in 2008, and the vast majority of their fleet was sold in the last couple years.
Obviously, the massive differences between electric powered cars and internal combustions engines means that they may never reach parity, but unless Musk has figured out a way to beat entropy, its pretty safe to assume that older cars will break down/suffer leaks/explode more than newer ones.
Did you consider the possibility that it is in fact you who don't understand?
The Law of Large Numbers states that as more miles are traveled, the fires per mile will approach the expected value. It is entirely possible to have 10 fires in the next week.
We won't know what the expected fires/mile is until a much larger sample is collected. It will take years to prove out.
Why do you believe a larger sample is needed? At what sample size would you believe that the Tesla averages less fires than other cars?
Here's a question (for anyone in this thread arguing statistics) that has an actual numerical answer: given the information in the article, what is the probability that Tesla's indeed experience less fires per mile than other cars? If someone doesn't know how to calculate the answer to that question, he shouldn't be arguing here.
Let's see if I remember any of this. If we assume the null hypothesis that Teslas have 1 fire per 20 million miles same as other cars, then P(0 fires in 100 million miles) = 0.67% and P(1 fire in 100 million miles) = 3.4% from Poisson distribution. So the odds that you'd have no more than 1 fire in 100 million miles is 4%. So I reject the null hypothesis with a p value of 0.04. (edit: fixed values)
This seems a bit dodgy since I'm "designing the experiment" after the fact, but I'm not sure how to correct for that. Any Bayesian experts?
You need an exhauseted state space. You cannot empirically infer a legitimate probabliliy, eg n/100m miles) with only a single failure observation, if there are 100 possible ways to fail. At best you have data on (1) of (N) ways to fail, but surely in the case of car accidents N=large.
A total of 2,650 cars were delivered to retail customers in North America during 2012, 4,900 during the first quarter of 2013, and 5,150 during the second quarter of 2013
Assuming 13000 cars on the road, each car would have logged 9k miles to get 110m road miles, as quoted by Tesla. But we know from past industry experience, that road fires are proportionate also with fleet age.
So, if anything we the probability of a road fire is likely to go up as more failure modes are discovered (including by chance), and as the vehicles cycle through a normal working life.
Assume a Tesla vehicle has a constant risk of catching fire per mile. That is, we have a exponential distribution `P(catch fire after t miles | hazard rate) = P(t|a) = a exp(-at)` where `a` is the hazard rate (average fires per mile). Furthermore we'll assume an exponential prior on `a`: `P(a) = w exp(-wa)`. `w` is a parameter that expresses how much prior knowledge we have of `a`. In the limit `w=0` we know nothing at all, except that it's nonnegative.
Our data is the fact that we went 100 million miles before a fire, after which exactly one fire happened, so we want to find the distribution `P(a|t = 100 million)` which tell us everything we want to know about `a`.
Then use Bayes' theorem: `P(a|t) = P(t|a) P(a) / P(t) = aw exp(-a(t+w)) / P(t)`. The normalization factor `P(t)` involves an integral over `P(t|a) P(a) da` from 0 to ∞, which wolfram alpha tells me evaluates as w / (t+w)^2.
So our posterior probability is `P(a|t) = a (w+t)^2 exp(-a(t+w))`, but we can take the limit `w -> 0` at this point for a fully uninformative prior: `P(a|t) = a t^2 exp(-at)`.
So we can just set `t=100e6 miles`, and now calculate things like the expectation of the distribution: `E[a] = 2/t = 2e-8 per mile`. Or the probability that the hazard rate is less than other cars, which is the integral from 0 to 1/(20 million miles): `P(a < b) = 1 - exp(-bt) (bt + 1) = 0.96`.
If I get the math right (assuming a Poisson distribution) then a week ago, based on 0 fires, they could've said with 90% confidence "The chance of your Tesla catching on fire is less than 2.3 cases per 100 million miles". Not 0; but not very high as well.
The relevant metric here seems to be 'miles between fire events'. On this metric, we have exactly 1 data point for Tesla. I would hardly call that "probably enough".
Of course, I'm exaggerating in the other direction. What we really should be calculating is the odds that Teslas burst into flames less often than the average car, given that the average car does so every 20 million miles and the first such event in a Tesla was at the 100 million-mile mark. We're still failing to account for the fact that the average Tesla is newer and probably better-kept than the average car, but it would at least be a reasonable start.
I don't know enough statistics to perform this calculation, but I would like to see how it is done.
An easy starting point would be to get this kind of data on similarly aged cars - cars sold over the last 2-3 years, then see what stats you can get there. I'd guess that you'd find some models that have never caught fire and some that have done so a lot more than tesla's, but that's pure speculation on my part.
We have enough events, because in this case it is appropriate to model the actual random event as 'million miles driven' with a chance of fire happening or not happening. Gasoline cars have a mean of 0.05 fires per million miles, and given the current Tesla data, the mean is 0.01 fires per million miles. I'm not taking out a calculator, but it would come out to an extremely low (0.0001%) the 'true' fire chance is the 0.05 gas car rate or higher; the 95% confidence interval should be 0.01 +- 0.02 or tighter, so still twice better than gas cars.
For an exaggerated example, if Tesla had driven a billion miles and had 0 fires, you shouldn't say that there's not enough data - you definitely would have enough data to say that the chance of fire is below the gas-car rate of 5 fires per 100 million miles.
Why is million miles driven a better way to model it than billion miles driven, for example? If you happen to choose that, there clearly isn't anywhere near enough data.
I'm honestly curious how one models this type of thing statistically, and I am not convinced enough of its obviousness to just accept numbers that someone throws around.
To put it in perspective, the disparity is equivalent to polling 10K americans and extrapolating to all of america (which, for better or for worse, is what most pollsters do).
It's counterintuitive, but the sample size needed for a good measurement doesn't much depend on the size of the overall population. What matters is getting a properly random sample. This is where pollsters fall down, because their "random" sample tends to be heavily biased toward the sort of person who has a landline telephone and doesn't hang up on pollsters.
Polling 10,000 Americans would be vast overkill, in any case.
You say that as if 10k is too small a sample size to extrapolate accurately with, but actually that's a huge sample size that if done properly would be extremely accurate. You don't need to poll anywhere near 10k people to accurately predict all Americans views.
You can sample less than 2000 people and get 99% accuracy with a 3% margin of error for a population of 325 million. Increasing the sample size to 10k simply reduces the margin of error to 1.29%, hardly worth the extra sampling of 8k people.
Do they really? My only interaction with pollsters has been either having them call me or solicit me on the street, and both approaches have a tremendous inherent bias. As far as I know, this is how the big national agencies do things.
There are all kinds of adjustments done afterwards to correct for various factors. For example, you know the age distributions in USA; and if you find out your phone calls are getting twice as many seniors than the proportion should be[1], then you throw away a random parts of them so that they don't skew final 'data' towards the typical opinions of seniors.
[1] Assuming that you're not measuring average age or measuring 'who is at home', but if you want to see, say, the average political opinion of total USA population, which tends to correlate with age.
It still seems impossible to correct for everything. Sure, you could correct for age as you describe, but I imagine that landline phone ownership correlates with political opinions in all sorts of other ways too.
Furthermore, how do you gather the data needed to correct the polling numbers without being able to accurately poll people in the first place? Seems like a complete chicken-and-egg problem.
"how do you gather the data needed to correct the polling numbers" -> you use the census. You need some info about the total population, you get it periodically and it doesn't change that much; you don't need to repeat it for every survey.
> Of course, the refrain I often hear is that as long as you pick the RIGHT 1,000 Americans, it's as good as polling all 319 million.
If it's a proper random sample, then it's far better than sampling all 319 million because it's 95% accurate with about a 3% margin of error and vastly cheaper and actually practical; you can't poll 319 million people.
The data on 100 million is probably enough to compare with the 3 trillion miles. The populations don't need to be equal to compare them, just big enough that they are random and distributed enough.