I'm guessing you don't manage any production web servers?
robots.txt isn't even respected by all of the American companies. Chinese ones (which often also use what are essentially botnets in Latin American and the rest of the world to evade detection) certainly don't care about anything short of dropping their packets.
I have been managing production commercial web servers for 28 years.
Yes, there are various bots, and some of the large US companies such as Perplexity do indeed seem to be ignoring robots.txt.
Is that a problem? It's certainly not a problem with cpu or network bandwidth (it's very minimal). Yes, it may be an issue if you are concerned with scraping (which I'm not).
Cloudflare's "solution" is a much bigger problem that affects me multiple times daily (as a user of sites that use it), and those sites don't seem to need protection against scraping.
It is rather disingenuous to backpedal from "you can easily block them" to "is that a problem? who even cares" when someone points out that you cannot in fact easily block them.
I was referring to legitimate ones, which you can easily block. Obviously there are scammy ones as well, and yes it is an issue, but for most sites I would say the cloudflare cure is worse than the problem it's trying to cure.
But is there any actual evidence that any major AI bots are bypassing robots.txt? It looked as if Perplexity was doing this, but after looking into it further it seems that likely isn't the case. Quite often people believe single source news stories without doing any due diligence or fact checking.
I haven't been in the weeds in a few months, but last time I was there we did have a lot of traffic from bots that didn't care about robots. Bytedance is one that comes to mind.
What you're describing is more like someone who doesn't know computer science principles hacking on code, manually. Part of the definition of "vibe coding" is that AI agents (of questionable quality) did the actual work.
I feel like Hacker News commenters love to make analogies more than average people in your average space, though. You can't come across a biology/health topic on here without someone chiming in with "it's like if X was code and it had this bug" or "it's like this body part is the Y of the computer."
Analogies can be useful sometimes, but people also shouldn't feel like they need to see everything through the lens of their primary domain, because it usually results in losing nuances.
That's a great point and an easy way to visualize it as an outsider, but it's not necessarily that simple.
For one thing, the iPad (market-leading tablet) and the iPhone (market-leading pocket touchscreen device) were not their first attempt at doing that. That would be the Newton, which was an actual launched product and a commercial failure.
For another thing, even Apple can't just become the market leader by doing nothing. They need to enter late with a good product, and having a good product takes R&D, which takes time. With MP3 players, smartphones, and tablets, they didn't exactly wait until the industry was burnt through before they came in with their offering; they were just later (with the successful product) than some other people who did it worse. They were still doing R&D during those years when they were "waiting."
Apple could still "show up late" to AI in a few more years or a decade, using their current toe-dipping to inform something better, and it would still fit into the picture you have of how they "should've done it." Not to mention, Apple's also lost its way before with things like convoluted product lines (a dozen models of everything) and experimental products (the Newton then, Apple Vision now); messing up for a while also isn't exactly bucking history.
I see your point, but I see nothing to indicate they’re doing the “polish and wait”. No reason to believe they’re cooking behind the scenes or that this product was a learning exercise for them.
Most of their current products seem to be decaying in the dead march towards the next yearly release. ux and ui are becoming more and more poorly thought (see their last design language). They half pursue ideas and don’t manage to deliver (vr, apple car, etc).
I see cargo culting and fad chasing like any average leadership, only with a fatter stack of cash supporting the endeavour.
I guess I'm not necessarily saying they're secretly working on it now, but I'm responding to your "I don't get why they went for the rush" with "it doesn't seem like they really went for the rush" (any more than the Newton was evidence that they "went for the rush" of smartphones or tablets).
If I understand correctly, you're just basing that statement on climate change or war destroying us before we can do any better than Voyager, right? Because if we don't assume the destruction of humanity or the complete removal of our ability to make things leave Earth, then just based on "finite past vs. infinite future," it seems incredibly unlikely that we'd never be able to beat an extremely old project operating far beyond its designed scope.
I'm pretty bearish on human interstellar travel or even long-term settlement within our solar system but I wouldn't be so pessimistic on unmanned probes. The technical hurdles seem likely to be surmountable given decades or centuries. Economic growth is likely to continue so relative cost will continue to drop.
Absent a general decline in the capacity of our civilization the main hurdle I see is that the cost is paid by people who will not live to see the results of it but I don't think that rules it out, I'd certainly contribute to something like that.
What are some of the other factors you are thinking of?
This is reflexive pessimism with no substance. You're not articulating a set of particular challenges that need to be navigated/overcome, which could provide a roadmap for a productive discussion; it's just doomposting/demoralization that contributes nothing.
I don't want to introduce 50 tangential branches to argue about with no end in sight.
It's not pessimism, it's reality. Think about how unlikely it is. Humanity had one stretch where we reached for the stars and that stretch ended and by sheer luck some crazy guy made it cheap. What happens when he's gone? Will it happen again? Most likely: no. In your lifetime? Even Less likely.
How you acknowledge serious side effects some people have, sweep them aside with "but some people don't," and act like getting children on lifelong medication won't result in them having adult-relevant side effects once they're adults is mind-blowing.
Checking the comments of a couple of posts, I noticed their lengths seem to be too uniform. E.g. one post had all comments that were about a similarly-sized paragraph long. Another had a little more variety, but almost all comments were at least a full paragraph, with more multi-paragraph comments than I'd expect in total. Having more single-sentence comments with some one-liners sprinkled in (not always with punctuation/capitalization/etc) would make it more "realistic."
> Depression is caused by laziness and anxiety by hopelessness.
I wish this were true, but its not even close. I wonder how your kids will react when they move away, and you're not around to police their emotional expression. If they're like me, they will promptly collapse into paralysis and self-destruction.
I strongly suggest that you frequently give your kids long stretches of time (months) to practice regulating themselves, without your interference.
And if they have anxiety or depression, please let them see a professional. If my parents had noticed the signs earlier, they would have saved me decades of pain.
At this age I am teaching emotional regulation on a daily basis.
As for when they grow up, adults who cannot manage their emotions get fired or are sent to jail. It is critical to be slow to anger, quick to forgive, and work at building strong friendships.
You imply we aren't teaching emotional regulation on a daily basis? We have always placed a great deal of emphasis on talking about feelings and developing "tools for our toolboxes" to deal with them. Unfortunately those tools are largely inadequate when there isn't a rational cause for a debilitating emotional state.
People who have never experienced a particular challenge are quick to assume credit for its absence and assign moral failings to others who experience it. It's insufferable but common.
Imagine a millionaire who had millionaire parents lecturing his children on how they're not allowed to be poor. Lo! They're not! The lectures worked.
Anger seems to me to be one of the least understood emotions. In my opinion anger should be practised at every possible opportunity in order to cultivate as good as relationship to it as other emotions. Otherwise it will seem like it's out of your control due to not knowing its limits and how much of it you need due to it being suppressed. It is a necessary emotion for our wellbeing.
They're welcome to share their emotions but poor behavior isn't acceptable. Anger is fine. Yelling isn't. We'll talk it though. Tonight i got yelled at 3 times for not helping. I asked why I got yelled at. "Because I was angry". The anger wasn't justified but real so we discussed. I got a hug shortly after. No discipline. Just patience.
We take the same approach! While I don't agree that depression is laziness, we do teach that feeling sorry for yourself is akin to laziness. When we're disappointed about things not going the way we wanted we're allowed to grump about it for 5 minutes, but then it's time to brush ourselves off and move on.
> Depression is caused by laziness and anxiety by hopelessness. My kids know that they aren't permitted to be lazy or say they are bored. They don't have anxiety because they have hope despite circumstances.
This sounds horrible. If I weren't depressed or anxious, being told that I wasn't ever permitted to be lazy or say that I was bored would make me so; and, if I were, then being told that I was lazy and hopeless would make it worse.
I can't tell if you're being sarcastic or not but that is kind of the truth. A breakup is horrible, in its worst case it is similar to an untimely death and can come with grief and maybe even worse because the person is still alive and in your life so you're reminded by their absence even deeper and could experience jealousy, and the drip feed of interaction can elongate the time it takes you to move on with your life and find the person who is more right for you.
But while the sadness is real and shouldn't be ignored or suppressed, wallowing is unhelpful and you ought to move on with your life - just like if someone close to you dies, the deceased most likely would want you to be happy and continue living a good life and move on.
Objectively you're right, but I hope you think twice before saying to someone who has experienced a tragedy, "You need to move on." Similarly if you're in an argument with a friend or loved one and they are upset I hope you think twice before saying "You need to calm down" or, "You're overreacting." Objectively those may all be correct, but they are unhelpful, and are likely to have the opposite effect. They minimize the weight of the emotions. Most of us cannot just turn emotions on and off, much as we try and wish we could.
> Depression is caused by laziness and anxiety by hopelessness.
Others have responded to the depression/laziness part -- I was wondering if you could clarify what you mean by anxiety being caused by hopelessness? To my way of thinking hopelessness and depression have a lot of overlap, but anxiety not so much... It's more like a feeling of dread.
Fear of the dentist is common but understanding that the inconvenience is required for the desired outcome, the reason for the suffering, there is a hope to look to allowing the suffering to have meaning.
> Depression is caused by laziness [...]. My kids know that they aren't permitted to be lazy.
Do you really believe this, or do you believe your children aren't depressed? Your comment is not in accordance with science. Depression is a complex topic. I'm having trouble imagining a way to be more wrong. Is this satire?
You can use me as an example! When I got diagnosed I was very physically active and also quite clean.
One of the hardest things for me with depression is the incredible guilt I have. What right do I have to be depressed when my life is objectively fine? Why should I get therapy when that might mean one less space for someone who is dealing with trauma or poverty or something else that gives them a "right" to be depressed? This causes a feedback loop of guilt leading to more depression and vice-versa.
Like I tell my kids, it's normal to sometimes feel depressed or hopeless. If you're dealing with a difficult circumstance then it's reasonable to have those feelings, and the only way to address those feelings is to deal with the circumstance. What's not normal is feeling depressed and hopeless for no logical reason at all.
I appreciate your sharing your experience. I think it's very valuable that human beings describe to each other their decision making, actions, and outcomes. Often, people attempt to dissuade the sharing of information, and I think that leads to us, as humans, being less able to form an accurate model of the world. I appreciate your pushing through that form of opposition.
It’s pretty tough to exercise or clean your house when getting out of bed feels like an insurmountable task.
Depression isn’t like an infection or cancer—it’s a diagnosis based on established criteria, as are most mental disorders. Experts may disagree on diagnosis or treatment, but that doesn’t make it useless.
By that logic, you might as well say autism is caused by avoiding eye contact—since there’s no blood test for it either.
Ahhh this explains a lot. You should learn about internalized ablism, and probably stop teaching vulnerable humans the same, under the guise of "emotional regulation".
Did somebody else tell you that your emotions need to be regulated this strictly? Did you learn that if you express your emotions you won't get any help? That is not a normal situation to be in.
I understand, but I can assure you that I don’t expect an autistic person to force eye contact with me—especially when it drains their energy. Insisting on that would be wrong.
I’m not opposed to people pushing themselves a little, but we can’t expect anyone to become “fixed” by trying ever harder. That approach wastes energy, is unrealistic, is non‑inclusive, and is simply cruel.
If you think people with depression, ADHD, or anxiety should just push through their symptoms to meet arbitrary social expectations—expectations that are fundamentally ableist—you’re not doing anyone a favor. In a world that’s becoming less ignorant and more inclusive of neurological differences, that attitude will only alienate you, as this thread clearly shows.
> In a world that’s becoming less ignorant and more inclusive of neurological differences, that attitude will only alienate you, as this thread clearly shows.
What an off-topic threat to try and get someone to pretend like they agree with you.
I hope you can take your personal anecdotes and add them to a larger body of research and other people's experience to refine your understanding. If you're right that everyone who has Depression is actually just lazy, you'll see lots of support for that. If, instead, you find a lot of different experiences you might conclude that Depression is a pretty nuanced and complicated topic, which might both expand your understanding and help you bring more empathy to the suffering people around you.
You are very, very extremely inexperienced in this area, as anyone who deals with or has someone in their close family that deals with serious, life-threatening depression will tell you.
And that's okay!
I would encourage you to look at the number of comments from different people who have a history of dealing with this kind of thing for a long time.
I agree 100% and it is an uphill battle requiring significant effort. That is where discipline is required. Exercising that discipline will bring positive results. There is nothing other than self stopping someone from cleaning instead of crying under a blanket.
WTF? You cannot comment like this on HN, no matter what you're replying to or how wrong you think they are. We've had to ask you before to avoid abusive comments. We have to ban accounts to do this repeatedly, and it's only because the parent comment was pretty bad too that I'm not banning you this time. Please take a moment to remind yourself of the guidelines and avoid ever commenting like this again if you want to keep participating here. https://news.ycombinator.com/newsguidelines.html
You seem to have your mind thoroughly made up, which should raise a flag that you might be suffering from a bad case of Dunning-Kruger, and need to re-ask questions you think you already know the answers to.
> Depression is caused by laziness and anxiety by hopelessness.
While pizzafeelsright gets heavily downvoted, I think their may be some (dark) truth behind it: if you weren't lazy, you would sommit suicide instead of living a depressed life. :-(
I also have a feeling that there might be another (dark) truth behind the link between anxiety and hopelessness.
> Depression is caused by laziness and anxiety by hopelessness.
No offense, but teaching your kids this kind of nonsense is borderline abusive. It's so far away from so many people's experiences, and is just plain wrong. I hope you can recognize how damaging this could be for your kids down the road.
With both my kids in 2nd grade and my wife also a public 2nd grade teacher, I consider myself pretty aware of what kids are being taught these days. They certainly are being gradually introduced to some of the problems of the world, but I think childhood development experts would all agree that's healthy. As for them being told they're the source or victim that's hardly the case. I'm sure there are a few isolated incidents that right-wing media love to bang on about, but not the experience for most.
> but I think childhood development experts would all agree that's healthy.
Could it be that we think it's healthy because we can just give meds to the kids that it affects?
How would someone even have the ability to say "it's healthy" - I'm struggling to think how it comes about. I think it's healthy for my kids to cry about a worm dying in the garden, but anything less than "anxiety about a dying planet"...
Put it another way: climate change messaging IS totally about anxiety and putting human as the cause, so we can (as adults) change our habits and save the planet. Could it be too much for kids though?
I grew up in the 80s and 90s. In school environmentalism was a big deal. We did conservation trips to these ecology parks, we were taught to recycle, and often watched films about animals that might die off due to climate change.
I only felt motivated to annoy my parents to recycle more. Since the effects were not directly in front of me a strong emotional reaction didn't occur.
Perhaps there are children who already have anxiety and latch onto climate change and other problems.
Not sure about you but I didn’t learn my native language phrase by phrase only. You learn the individual parts and concepts and construct sentences from that.
He learned it from being on Earth? And noticing that some people who used to be on Earth aren’t anymore? And it dawning on him that he doesn’t have to be either?
You seem to have a pretty narrow view of the world, especially about “privilege education.” FWIW I’ve never once heard anything about “white people bad” from my kiddo, about 10.
It’s also impossible to to talk about idk Columbus, slavery, Great Britain, or the founding of America (and like ya know, the treatment of native Americans) without ascribing some blame to the people responsible…who were by and large “white”. We also talk about how any judgement based on skin, appearance, gender, ethnicity, or religion, is flat out wrong.
> Also you should try your SSRI prescription. They really aren't very strong drugs. You might get mild relief or if you're like me and the majority of people you will see no effect whatsoever. It's worth a try anyway.
> It's too early to say. Obviously the idea is to get her off it if possible.
You understand that the people who sold you that drug have a vested interest in making sure it's not possible and/or that you & she think it's not possible, right?
I'm big on medications for brain stuff but uh yes, in the US, doctors get lots of kickbacks for prescribing drugs.
Usually this takes the form of "I'm prescribing you with <Brand> instead of generic" or "I'm prescribing you this specific drug from this class of drug"
> doctors get lots of kickbacks for prescribing drugs.
From your own source: "In 2024: $172 or more in general payments have been received by half of physicians."
Even if all of those payments count as kickbacks, a median of $172 in a year (significantly less than 0.1% of the median physician's annual pay) is not "a lot of kickbacks".
robots.txt isn't even respected by all of the American companies. Chinese ones (which often also use what are essentially botnets in Latin American and the rest of the world to evade detection) certainly don't care about anything short of dropping their packets.