Hacker Newsnew | past | comments | ask | show | jobs | submit | radub's commentslogin

Hi, regarding number 3, I have filed https://github.com/cockroachdb/cockroach/issues/58032. Would love to get your feedback on the proposal; does it address your need? do you have a different idea for how it would work?


There are quite a few case studies / customer stories on the website: https://www.cockroachlabs.com/customers/


[Cockroach Labs engineer here, I worked on statistics]

All table statistics are stored in a system table (system.table_statistics). CREATE STATISTICS scans the table, calculating statistics on a column, and adds a new row to system.table_statistics.

We still have a ways to go in making statistics automatic (right now it requires a manual call to CREATE STATISTICS) and more useful to the optimizer (which currently only uses basic stats). We're working on that right now for the 2.2 release.


What if the tables were partitioned, do you calculate local statistics or you aggregate the stats and expose it to the optimizer?


Are you planning to support multivariate statistics?


We plan to support cardinality estimations for multiple columns. Other than that there are no plans for multivariate stats in the near future.


I don't have a specific timeline but it is something we will be focusing on in the following releases.

Regarding DDL statements, this blog post [1] has details. In a nutshell, online schema changes are possible; the changes become visible to transactions atomically (a concurrent transaction either sees the old schema, or the fully functional new schema).

[1] https://www.cockroachlabs.com/blog/how-online-schema-changes...



Thanks! Yes, I expect we will be moving in that direction before too long. It will work well with log tags which are already key-values. Plus most of our messages make it into traces and OpenTracing is switching to a KV interface as well.


In the majority of cases, the current layer would have added a log tag to the Context, so looking up the log tags structure will be immediate. On the other hand, we also use OpenTracing and we do have to go through the hierarchy to find the current Span, which will be slower with deep hierarchies.

The other thing to consider is that logging a message carries other costs, so we can't go crazy with very frequent log messages anyway. In our case the message usually also makes its way into an OpenTracing span and that involves memory allocation and/or data copying. My guess is that going through a few linked Contexts is not a big deal, comparatively speaking.

I don't have the kind of benchmark that you are looking for (with vs without using Context), but I can tell you that in the workloads we looked at, tracing was taking a few percentage points (1-2%) of CPU usage, and I don't think a significant part of that was in Context code.

Finally, Context is an interface, so if need be, we can make our own Context implementation that always stores the frequently looked-up things (like log tags, tracing Spans) to short-circuit that cost.


It could be great exposure therapy :)


[cockroach labs engineer here] Where are you getting the "ignoring stability" part? Or that we "skipped correctness and stability" (in another comment of yours)?

Correctness and robustness are the main factors behind the design and implementation choices we make. A significant part of the overall engineering effort has been on stability for a long time. We're now making it closer to 90% for a while.

None of this is unexpected in the development of a complex system, especially when there are many factors involved in deciding how much to focus on various aspects. I have worked on a few unrelated systems which turned out to be stable and successful when released and - at a comparable stage in their development - they were much less stable. So personally, I am very optimistic about CockroachDB.


It's implied in the article. It mentions many mounting problems in stability, including nobody was dedicated to working on it. Implies a lack of QA. I based my claims on the article's. If those claims were mistated, then any of mine drawing on them wont apply of course.

Just reads like a lack of QA in general with most correctness effort focused on protocol design itself.


The relevant tests fall somewhere between unit tests and integration tests. Many are intended as unit tests, but we prefer to test them on top of the "real" implementations of the layers underneath (instead of mocks) so they are integration tests as well.

> If it's a feature or integration test, why is it a problem to pull in the other packages?

The compiler does not allow the cyclic dependency (even if it's only caused by testing code).


It's been a while since I touched golang, so I don't recall the nitty-gritty of the cyclic problem.

And to pick a nit, to me, if there are several units of your system pulled in, it's arguably not a unit test. It's an integration test.

I don't love mocks, but I don't hate them either. They have their place as part of an overall testing pyramid.


I write a lot of these tests in my code as well. They are not exactly unit tests, because they test code at multiple levels of abstraction at once, but they are written in exactly the same way as a unit test and are intended for the same purpose: to make sure that some unit of your code functions as it should and does not stop functioning.

To me, mocks are useful for one thing only: there is some expensive or flaky dependency of your code that you may have poor visibility into, and you want to write a test that does not rely on its availability. Mocks for databases, file systems, REST APIs, externally maintained libraries, etc. all make sense to me. But if some library is developed locally (or in the extreme case, by you) and you can easily depend on its existence and stability, then I see no reason not to write code that depends on that library and test against that library. This is a testing strategy for a pragmatist, not a purist.

Go in particular makes it easy to separate a single codebase into separate logical packages, and if you do so, it often makes sense to test them together. That's why this is a pretty Go-specific issue.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: