You can always slice and dice data to make one side look better.
The actual number of successes vs partial successes vs launch failures vs fatalities are the best data we have. Throwing away any of that data because it makes you look worse isn’t a good idea.
Similarly we needs to understand that there’s a huge difference between risk and what actually happens. People get lucky in Vegas every day, what matters to most of us is the accuracy of the estimate of underlying odds not just the exact outcomes out to seven decimal places.
> actual number of successes vs partial successes vs launch failures vs fatalities are the best data we have
Current vehicles are vastly different from the originals. What we’re trying to do is predict the probability of the next launch failing. Equally weighting far historicals and recents is bogus statistics.
> What we’re trying to do is predict the probability of the next launch failing.
I thought we were comparing methods. Unless the next payload is yours then the odds of the next launch failing is meaningless to most of us, but we can learn something from the methods used.
But sure, if you have a bet in Vegas or something then feel free try and calculate things as closely as possible. Just understand that several of Soyuz failures didn’t kill the crew so there’s other metrics people might care about.
What does this mean? The question most of us care about is which method resulted in a more reliable rocket. And SpaceX’s track record shines uniquely in that respect. The frequency, moreover, makes the results robust. Legacy rockets like Ariane will never reach that confidence because the likelihood of fluke successes won’t have been minimised when the rocket is retired.
As to why their methodology is important this isn’t the Falcon 9 this is a new launch system which is likely going to have multiple failures before it’s own streak can begin.
So sure, we can reasonably assume that Starship will get to a state of reliability similar to current Falcon rocket, eventually. We can’t assume the first few commercial Starship launches are going to even approach that level of reliability. And in fact the best point of comparison may be the early days of Falcon 9.
Speaking of methodology, it's incorrect to relate a development test result to reliability or risk. Source is my personal experience doing reliability calculations for a NASA rocket component and working with the statisticians incorporate my numbers into their risk model.
You have it mixed up. I've worked with the stats at NASA. Mission success and failure counts. Test quantity and quality count, test freedom counts, how they learn from test counts, but the test result does not count. This isn't a mission.
This was a partially successful test nothing more and nothing less. I get people really really think SpaceX had done an excellent job and I don’t disagree but people who are comparing the end result of a long process Aka the current state of falcon 9 with a new system like Starship are going to be disappointed.
Starship is extremely likely to fail repeatedly before achieving anything close to the same streak as the Falcon 9 has. That’s not an issue with SpaceX that’s an inherent aspect of doing something really difficult.
I don't think you realize this, but when you said that we can't exclude the test failure from the risk/reliability assessment, that's exactly what you're saying. I didn't realize until just now that you're actually defending the test failure as being acceptable.
The actual number of successes vs partial successes vs launch failures vs fatalities are the best data we have. Throwing away any of that data because it makes you look worse isn’t a good idea.
Similarly we needs to understand that there’s a huge difference between risk and what actually happens. People get lucky in Vegas every day, what matters to most of us is the accuracy of the estimate of underlying odds not just the exact outcomes out to seven decimal places.