This article triggers all my “written by an LLM” spidey senses.
Which is ironic considering the subject matter. “Perfect”, but artificially constructed. “Just for me”, but algorithmic slop.
I agree that you can do so much more custom tailoring of bespoke software with the speed an LLM brings. But something inside of me still revolts at calling this anything other than “convenient”.
“Perfect” I will reserve for things I’ve made myself. However “imperfect” they may really be.
Look at it this way. The carver doesn't have to grow the tree. Using an LLM for coding is a lot like being a carver. You can take broad or small strokes and discard what you don't like.
> Using an LLM for coding is a lot like being a carver.
It’s nothing like being a carver. It’s like being a director; “Once more! With feeling!”. “Perfect, brilliant, just one more take, and I want you to consider…”
A sculptor shapes with their hands, and there is pleasure in that. A director shapes with someone else’s hands.
It's like being a master, really. You're getting output from something that can't say no and that was built from the works of millions of unknowing and unconsenting contributors.
We also see actors as artists, and pay them gorillions of dollars. AI is about using distilled human knowledge, while cutting the human out of the loop (and out of the payroll).
Not an implementer of group policy, more of a consumer. There are 2 things that I find extremely problematic about them in practice.
- There does not seem to be a way to determine which machines in the fleet have successfully applied. If you need a policy to be active before doing deployment of something (via a different method), or things break, what do you do?
- I’ve had far too many major incidents that were the result of unexpected interactions between group policy and production deployments.
That's not a problem with group policy. You're just complaining that GPO is not omnipotent. That's out of scope for group policies mate. You win, yeah yeah.... Bye
Auditing is fundamentally different because it has different durability and consistency requirements. I can buffer my logs, but I might need to transact my audit.
For most cases, buffering audit logs on local storage is fine. What matters is that the data is available and durable somewhere in the path, not that it be transactionally durable at the final endpoint.
What are we defining as “audit” here? My experience is with regulatory requirements, and “durable” on local storage isn’t enough.
In practice, the audit isn’t really a log, it’s something more akin to database record. The point is that you can’t filter your log stream for audit requirements.
Take Linux kernel audit logs as an example. So long as they can be persisted to local storage successfully, they are considered durable. That’s been the case since the audit subsystem was first created. In fact, you can configure the kernel to panic as soon as records can no longer be recorded.
Regulators have never dictated where auditable logs must live. Their requirement is that the records in scope are accurate (which implies tamper proof) and that they are accessible. Provided those requirements are met, where the records can be found is irrelevant. It thus follows that if all logs over the union of centralized storage and endpoint storage meet the above requirements then it will satisfy the regulator.
> Regulators have never dictated where auditable logs must live.
That’s true. They specify that logs cannot be lost, available for x years, not modifiable, accessible only in y ways, cannot cross various boundaries/borders (depending on gov in question). Or bad things will happen to you (your company).
In practice, this means that durability of that audit record “a thing happened” cannot be simply “I persisted it to disk on one machine”. You need to know that the record has been made durable (across whatever your durability mechanism is, for example a DB with HA + DR), before progressing to the next step. Depending on the stringency, RPO needs to be zero for audit, which is why I say it is a special case.
I don’t know anything about linux audit, I doubt it has any relevance to regulatory compliance.
> In practice, this means that durability of that audit record “a thing happened” cannot be simply “I persisted it to disk on one machine”
As long as the record can be located when it is sought, it does not matter how many copies there are. The regulator will not ask so long as your system is a reasonable one.
Consider that technologies like RAID did not exist once upon a time, and backup copies were latent and expensive. Yet we still considered the storage (which was often just a hard copy on paper) to be sufficient to meet the applicable regulations. If a fire then happened and burned the place down, and all the records were lost, the business would not be sanctioned so long as they took reasonable precautions.
Here, I’m not suggesting that
“the record is on a single disk, that ought to be enough.” I am assuming that in the ordinary course of business, there is a working path to getting additional redundant copies made, but those additional copies are temporarily delayed due to overload. No reasonable regulator is going to tell you this is unacceptable.
> Depending on the stringency, RPO needs to be zero for audit
And it is! The record is either in local storage or in central storage.
> And it is! The record is either in local storage or in central storage.
But it isn’t! Because there are many hardware failure modes that mean that you aren’t getting your log back.
For the same reason that you need acks=all in Kafka for zero data loss, or synchronous_commit = remote_flush in PostgreSQL, you need to commit your audit log to more than the local disk!
If your hardware and software can’t guarantee that writes are committed when they say they are, all bets are off. I am assuming a scenario in which your hardware and/or cloud provider doesn’t lie to you.
In the world you describe, you don’t have any durability when the network is impaired. As a purchaser I would not accept such an outcome.
> In the world you describe, you don’t have any durability when the network is impaired.
Yes, the real world. If you want durability, a single physical machine is never enough.
This is standard distributed computing, and we’ve had all (most) of the literature and understanding of this since the 70’s. It’s complicated, and painful to get right, which is why people normally default to a DB (or cloud managed service).
The reason this matters for this logging scenario is that I normally don’t care if I lose a bit of logging in a catastrophic failure case. It’s not ideal, but I’m trading RPO for performance. However, when regs say “thou shalt not lose thy data”, I move the other way. Which is why the streams are separate. It does impose an architectural design constraint because audit can’t be treated as a subset of logs.
> If you want durability, a single physical machine is never enough.
It absolutely can be. Perhaps you are unfamiliar with modern cloud block storage, or RAID backed by NVRAM? Both have durability far above and beyond a single physical disk. On AWS, for example, ec2 Block Express offers 99.999% durability. Alternatively, you can, of course, build your own RAID 1 volumes atop ordinary gp3 volumes if you like to design for similar loss probabilities.
Again, auditors do not care -- a fact you admitted yourself! They care about whether you took reasonable steps to ensure correctness and availability when needed. That is all.
> when regs say “thou shalt not lose thy data”, I move the other way. Which is why the streams are separate. It does impose an architectural design constraint because audit can’t be treated as a subset of logs.
There's no conflict between treating audit logs as logs -- which they are -- with having separate delivery streams and treatment for different retention and durability policies. Regardless of how you manage them, it doesn't change their fundamental nature. Don't confuse the nature of logs with the level of durability you want to achieve with them. They're orthogonal matters.
> It absolutely can be. Perhaps you are unfamiliar with modern cloud block storage, or RAID backed by NVRAM? Both have durability far above and beyond a single physical disk. On AWS, for example, ec2 Block Express offers 99.999% durability. Alternatively, you can, of course, build your own RAID 1 volumes atop ordinary gp3 volumes if you like to design for similar loss probabilities.
Certainly you can solve for zero data loss (RPO=0) at the infrastructure level. It involves synchronously replicating that data to a separate physical location. If your threat model includes “fire in the dc”, reliable storage isn’t enough. To survive a site catastrophe with no data loss you must maintain a second, live copy (synchronous replication before ack) in another fault domain.
In practice, to my experience, this is done at the application level rather than trying to do so with infrastructure.
> There's no conflict between treating audit logs as logs -- which they are -- with having separate delivery streams and treatment for different retention and durability policies
It matters to me, because I don’t want to be dependent on a sync ack between two fault domains for 99.999% of my logs. I only care about this when the regulator says I must.
> Again, auditors do not care -- a fact you admitted yourself! They care about whether you took reasonable steps to ensure correctness and availability when needed. That is all.
I care about matching the solution to the regulation; which varies considerably by country and use-case. However there are multiple cases I have been involved with where the stipulation was “you must prove you cannot lose this data, even in the case of a site-wide catastrophe”. That’s what RPO zero means. It’s DR, i.e., after a disaster. For nearly everything 15 minutes is good, if not great. Not always.
> It matters to me, because I don’t want to be dependent on a sync ack between two fault domains for 99.999% of my logs. I only care about this when the regulator says I must.
If you want synchronous replication across fault domains for a specific subset of logs, that’s your choice. My point is that treating them this way doesn’t make them not logs. They’re still logs.
I feel like we’re largely in violent agreement, other than whether you actually need to do this. I suspect you’re overengineering to meet an overly stringent interpretation of a requirement. Which regimes, specifically, dictated that you must have synchronous replication across fault domains, and for which set of data? As an attorney as well as a reliability engineer, I would love to see the details. As far as I know, no one - no one - has ever been held to account by a regulator for losing covered data due to a catastrophe outside their control, as long as they took reasonable measures to maintain compliance. RPO=0, in my experience, has never been a requirement with strict liability regardless of disaster scenario.
> I suspect you’re overengineering to meet an overly stringent interpretation of a requirement. Which regimes, specifically, dictated that you must have synchronous replication across fault domains, and for which set of data? As an attorney as well as a reliability engineer, I would love to see the details.
I can’t go into details about current cases with my current employer, unfortunately. Ultimately, the requirements go through legal and are subject to back and forth with representatives of the government(s) in question. As I said, the problem isn’t passing an audit, it’s getting the initial approval to implement the solution by demonstrating how the requirement will be satisfied. Also, cloud companies are in the same boat, and aren’t certified for use as a result.
This is the extreme end of when you need to be able to say “x definitely happened” or “y definitely didn’t happen” It’s still a “log” from the applications perspective, but really more of a transactional record that has legal weight. And because you can’t lose it, you can’t send it out the “logging” pipe (which for performance is going to sit in a memory buffer for a bit, a local disk buffer for longer, and then get replicated somewhere central), you send it out a transactional pipe and wait for the ack.
Having a gov tell us “this audit log must survive a dc fire” is a bit unusual, but dealing with the general requirement “we need this data to survive a dc fire”, is just another Tuesday. An audit log is nothing special if you are thinking of it as “data”.
You’re a reliability engineer, have you never been asked to ensure data cannot be lost in the event of a catastrophe? Do you agree that this requires synchronous external replication?
> have you never been asked to ensure data cannot be lost in the event of a catastrophe? Do you agree that this requires synchronous external replication?
I have been asked this, yes. But when I tell them what the cost would be to implement synchronous replication in terms of resources, performance, and availability, they usually change their minds and decide not to go that route.
Some kind of ”Error” is of course one of the sane message types. ”Warning” and ”info” might be as well.
”Verbose”, ”debug”, ”trace” and ”silly” are definitely not, as those describe a different thing altogether, and would probably be better instrumented through something like the npm ”debug” package.
> This is a puzzle! Why would the market fail to reward innovative firms, or, conversely, why does it continue rewarding less innovative firms? Unfortunately, here we don’t have clear answers.
Puzzle no more, the answers are obvious! There are two interlinked mechanisms leading to this phenomenon. The rise of inequality (centralisation of power and wealth) and the rise in private debt. Both require coordinated governmental intervention to address, which won’t happen until the next economic crisis and dramatic drop in standards of living. Wish it was different, but economic theory (mainstream anyway) doesn’t account for our present situation and the control system is cycling into instability.
The upside is that we might learn the lessons this time around.
You don't explain the connection at all. You're just injecting your political world-view into the cause so you can hawk your preferred political solution.
It’s nothing to do with politics, it’s just maths. And if you think I’m a leftist, you would also be wrong!
Unfortunately, educating yourself on this topic is not easy and involves differential equations. The economic models that fail to predict our current situation are simplifications. I’d link you, but I don’t think I’ll be getting a very receptive audience!
> Unfortunately, educating yourself on this topic is not easy and involves differential equations.
Which is it? Obvious but... only if you're "educated"?
> The economic models that fail to predict our current situation are simplifications.
Are there economic models that are not simplifications?
If being simplifications makes the models trivial to dismiss, but also all models are simplifications, then how do you successfully "predict our current situation"? I guess not with models. Just from first principles or something, but like, which? And then you need to provide the full chain of reasoning, and don't let that become a model. Or maybe it's simulations, but those are also invariably simplifications.
It's hard to take this seriously. Some links would be appreciated.
> And if you think I’m a leftist, you would also be wrong!
I didn't refer to specific politics, just your politics whatever they happen to be. Now you tell me I'm an ignoramus while you're educated and that's why this stuff is obvious to you but not to me -- and also not to [some? many? most??] economists. Plus:
> I’d link you, but I don’t think I’ll be getting a very receptive audience!
Certainly no link -> non-receptive audience. Links might or might not improve the situation, but we can't tell till you furnish some.
> The rise of inequality (centralisation of power and wealth) and the rise in private debt. [...] present situation and the control system is cycling into instability. [...] this time around.
Your explanation assumes the article is trying to explain a recent phenomenon.
The article actually discusses a puzzling pattern spanning a huge time interval.
You probably point at the right problem (inequality, centralisation of power and wealth), but this article actually indicates this problem has been going since before any of us were even born.
You still misread the article: they observe the nearly constant 2% growth across the long timespan, regardless of the relative increases in researchers, regardless of all the policy changes, regardless of ...
The article is NOT about some recent change. Please cite the article if you believe it is trying to solve a puzzle concerning a recent change.
The whole point is that this 2% seems to be robust, regardless of investing or getting more ideas, invalidating the idea that the growth is a simple result of the production of ideas (say making blueprints for a new kind of factory, which can then be copied without having to make more blueprints).
I am not asking you for a random citation of the article, but one that demonstrates the article discusses a puzzle concerning a recent trend, as opposed to a longtime ongoing one.
Your citation of the article:
> This is a puzzle! Why would the market fail to reward innovative firms, or, conversely, why does it continue rewarding less innovative firms? Unfortunately, here we don’t have clear answers.
does not refer to any recent change, indeed, it uses the word "continue" invalidating your claim that the puzzle is about some recent change.
There are a lot of public facing graphql servers that use it without issue other than frustrating users of non adversarial but complex requirements. The problem is that it is generally on a per request basis.
An adversary is going to utilize more than a single query. It mostly protects against well intentioned folks.
Other forms of protection such as rate limiting are needed for threat models that involve an adversary.
The same problems exist with REST but there it is easier as you can know query complexity ahead of time at end points. GraphQL has to have something to account for the unknown query complexity, thus the additional heuristics.
> When I look at myself, I don't believe I have character. I want to be liked too much, and in my emotional core, I'm frightened.
First of all, thank you for the honesty. It shows good character!
I think you are right that good character is the core of being a good manager. It’s the core of being a good person. Virtue and duty. Unfashionable words, but the secret to “happiness” (the good life). The ancient greeks understood this, and it’s been the heart of western philosophy.
I can believe you not hearing about Sharepoint. But not hearing about Onedrive is basically impossible if you have used a Windows machine in the last decade.
Yeah, one may not use it but it's hard to ignore when Office apps suggest you save the document to the cloud as a default. I do avoid it and don't really need any collaboration but I understand that I'm minority. On my home workstation (which is mainly used for video editing) I have only local account so I don't get sucked into more MS services. But at this point you have to actively try to get around the default setup with online account and cloud apps, so it's indeed hard to ignore.
I have never used windows machines for anything but gaming, but I get your point readily. If I had used the windows finder I would have encountered onedrive.
> Microsoft has a problem that they hire the middle block of talent in the market. They do not chase the top 20% most expensive nor the bottom 20% least expensive.
This is such a BS attitude. A person’s ability doesn’t map directly with their salary.
The very best developers I’ve ever worked with? Small companies outside of major cities (for life style reasons). Not everyone is trying to max their income; some want a good job doing something they love.
Small companies can offer more meaningful impact, if you’re not the type who likes being a tiny cog or commanding armies.
You are coping. Overwhelmingly the quality of developers is reflected in their pay packages. Overwhelmingly most people are motivated by money/benefits/QOL/GPUs.
This is an amusing response. I personally have chased the TC (to an extent), climbed the ladder, and sit at the top (below the c-suite). This isn’t cope, it’s a genuine reflection based on many years of experience having worked for multinationals in various countries.
Perhaps compensation should be a reflection of ability, but the real world is far more nuanced.
Yes, but...does it? How do you know it isn't missing key points raised in the meeting? Not collecting actionable items?
Summarising seems like the absolute lamest thing an LLM could do for me. Like I want a Reader's Digest of my life, written but a word guessing machine.
This is arguably the only use case where a chat interface (or just text) actually works, because it's an information compression task. But building a $50 billion business strategy just on the fact that AI can recap a meeting is risky. If this is the only feature that works well, then the business ROI is questionable
Which is ironic considering the subject matter. “Perfect”, but artificially constructed. “Just for me”, but algorithmic slop.
I agree that you can do so much more custom tailoring of bespoke software with the speed an LLM brings. But something inside of me still revolts at calling this anything other than “convenient”.
“Perfect” I will reserve for things I’ve made myself. However “imperfect” they may really be.
reply