crt.sh is a wonderful tool. I applaud anyone who makes CT log searching more available!
That said, crt.sh can be woefully unreliable. It often returns errors during a query or is just hard down. Large result sets may never return. Queries often take a very long time.
I wanted a more reliable CT log search tool for something I'm working on, so I built a purpose built CT log search tool. I ingest all the data from the logs directly and store in Clickhouse.
Author here. I am seeing a lot of comments about how the graphs are not anchored at 0. The intent with the graphs was not to "lie" or "mislead" but to fit the data in a way that was mostly readable side by side.
The goal was to show the high level change, in a glanceable way, not to get in to individual millisecond comparisons. However, in the future I would pick a different visualization I think :)
The benchmarking has also come under fire. My goal was to just to put the same site/assets on three different continents and retrieve them a bunch of times. No more, no less. I think the results are still interesting, personally. Clean room benchmarks are cool, but so are real world tests, imo.
Finally, there was no agenda with this post to push HTTP/3 over HTTP/2. I was actually skeptical that HTTP/3 made any kind of difference based on my experience with 1.1 to 2. I expected to write a post about "HTTP/3 is not any better than HTTP/2" and was frankly surprised that it was so much faster in my tests.
> However, in the future I would pick a different visualization I think
I think the box plots were a good choice here. I quickly understood what I was looking at, which is a high compliment for any visualization. When it's done right it seems easy and obvious.
But the y-axis really needs to start at 0. It's the only way the reader will perceive the correct relative difference between the various measurements.
As an extreme example, if I have measurements [A: 100, B: 101, C: 105], and then scale the axes to "fit around" the data (maybe from 100 to 106 on thy y axis), it will seem like C is 5x larger than B. In reality, it's only 1.05x larger.
Leave the whitespace at the bottom of the graph if the relative size of the measurements matters (it usually does).
>> It's the only way the reader will perceive the correct relative difference...
Every day, the stock market either goes from the bottom of the graph to the top, or from the top all the way to the bottom. Sometimes it takes a wild excursion covering the whole graph and then retreats a bit toward the middle. Every day. Because the media likes graphs that dramatize even a 0.1 percent change.
No, the media just happens to sometimes share OP’s intend: to show a (small) absolute change. That change may or may not be as dramatic as the graph suggests in both visualizations: measured in Kelvin, your body temperature increasing by 8 K looks like a tiny bump when you anchor it at absolute zero. “You” being the generic “you”, because at 47 deg C body temperature, the other you is dead.
It will be visible if you work in Celsius, a unit that is essentially a cut-off Y axis to better fit the origin within the domains we use it for.
We have an intuitive sense of what 30 degrees is, assuming it is in our preferred system of measurement.
A stock market graph really should be showing the percentage change, not some small absolute change that it’s not immediately understood by the typical layperson.
This notion about cut-off y-axes is the data visualization equivalent of “correlation is not causation”: it’s a valid point that’s easily understood, so everyone latches on to it and then uses it to proof their smartitude, usually with the intonation of revealing grand wisdom.
Meanwhile, there are plenty of practitioners who aren’t obviously to the argument, but rather long past it: they know there are situations where it’s totally legitimate to cut the axis. Other times, they might resort to a logarithmic axis, which is yet another method of making the presentation more sensitive to small changes.
There are plenty of instances where it's appropriate to use a y-axis that isn't "linear starting at zero." That's why I specified that I was only talking about ways to represent relative differences (i.e. relative to the magnitude of the measurements).
In this case, when we're measuring the latency of requests, without any other context, it's safe to say that relative differences are the important metric and the graph should start at zero.
So while it's true that this isn't universally the correct decision, and it's probably true that people regurgitate the "start at zero" criticism regardless of whether it's appropriate, it does apply to this case.
I think these choices are more context specific than is often appreciated. For example
> if I have measurements [A: 100, B: 101, C: 105], and then scale the axes to "fit around" the data (maybe from 100 to 106 on thy y axis), it will seem like C is 5x larger than B. In reality, it's only 1.05x larger.
If you were interested in the absolute difference between the values then starting your axis at 0 is going to make it hard to read.
It is however very rare that absolute differences matter; and even when they do, the scale should (often) be fixed. For example the temperatures:
[A: 27.0, B: 29.0, C: 28.0]
versus:
[A: 27.0, B: 27.2, C: 26.9]
If scale is fit to the min and max values, the charts will look the same.
Still, as a rule of thumb, when Y axis doesn't start at 0, the chart is probably misleading. It is very rare that the absolute size of the measured quantity doesn't matter.
Yeah, you should graph both starting at 0K right? You wouldn't want to mislead people into thinking somthing at 10C is ten times more hot than something at 1C.
Agreed. Next time I'll make the text and other things a little larger too (the real graphs are actually quite large, I had to shrink them to fit the article formatting.) I'd already spent so much time on the article I didn't want to go back and redo the graphs (I didn't really think too many people would read it - it was a big surprise to see it on HN)
You might also want to actually put HTTP 1/2/3 side-by-side in each graph, and separate graphs by use case. Rather than the current visualization, putting use cases side by side, and HTTP 1/2/3 in different graphs.
> I am seeing a lot of comments about how the graphs are not anchored at 0.
Personal preference: for large offsets that makes sense. For small ones (~10% of max here) it seems unnecessary, or, to a suspicious mind, meant to hide something ;)
Yep, I didn't even think about the suspicious angle when I did it. Mostly I was fiddling with how to draw box plots in D3 and that's what came out. Next time I will ensure a 0 axis!
Could it be simply because UDP packets may be treated with higher priority by all the middleman machines? UDP is used by IP phones, video conferences, etc.
Because of the use of UDP, I wonder the variation could end up being wider. There are some large outliers on the chart (though that could be an implementation maturity issue). Also I wonder if the routing environment will continue to be the same in terms of prioritization if http/3 becomes more common. There might be some motivation to slow down http/3 to prioritize traditional udp real-time uses.
Thanks for the article. But the goal of the graphs should be to show level of change and let graph speak for itself, if it's high or low. If they were anchored at 0 it would actually allow to see visual difference and for me personally it would be "way that was mostly readable side by side".
I wouldn't interpret those comments as accusation. It's in all our best interests to critique possibly misleading graphs, even when done so unintentionally.
The charts are ok for the purpose of visualizing the performance of the protocols. At least for me, they are side-by-side with the same min and max values and is easy to compare. The purpose is clear and starting from zero adds nothing of value.
People who think the bottom line is zero don't know how to read a chart.
What maybe is missing is a table with the statistics to compare numbers with maybe a latency number.
Author here. If I were to do it again I would pick a different visualization. The intent was not to "lie" with statistics as other commenters here seem to think, it was to fit the data side by side and have it be reasonably visible.
Off-topic but I'd never heard of CatchJS before. As the founder of TrackJS[1] I can't help but feel they were heavily inspired by our product... almost too inspired considering their logo and marketing copy.
(We've been around since 2014, so we pre-date them by 4 years)
You both have an incredibly generic logo/header/copy template so I'm not really sure what you're trying to imply. Your site copies the format of Stripe and they were around in 2010, pre-dating you by 4 years.
Anyone can go on archive.org and verify that they changed their color scheme and header to ones very similar to CatchJS in Feb 2020, after CatchJS had used that look for 2 years. Thank you for also pointing out the pricing page, where the same is true.
I've now received a cease and desist from their attorney stating that the looks are too similar. It's nice that they've paid for an attorney to make my point for me, but they seem to have mixed up who did what here.
We are building a new SaaS product from scratch and blogging about each step. We are also filming it and putting it on YouTube so people can see how the sausage is made.
It's hosted on Github Pages so hopefully it's available:
That said, crt.sh can be woefully unreliable. It often returns errors during a query or is just hard down. Large result sets may never return. Queries often take a very long time.
I wanted a more reliable CT log search tool for something I'm working on, so I built a purpose built CT log search tool. I ingest all the data from the logs directly and store in Clickhouse.
https://www.certkit.io/tools/ct-logs/