Anubis[1] is a superb fix for sites not behind Cloudflare/Fastly/Bunny etc. We had millions of bot requests, on a site serving all countries so we couldn't block by country, with fake user-agents so we couldn't block using that. It uses 'proof of work' to detect real browser software.
How do you figure it hurts real users? The amount of compute/energy used on the proof of work is pretty minimal. You're using more when you watch a YouTube video or browse a JS-heavy web app.
Of course a sophisticated scraper can "figure out" how to bypass. It isn't trying to be foolproof, it's adding an extra cost to deter massive amounts of bot traffic.
I put it in front of my hobby project because I can't afford to serve hundreds of thousands of bot requests from residential proxies all across the world, and I didn't want to route all of my traffic through a third party company like Cloudflare.
1. It makes the web more annoying and time wasting.
2. I'm a real user and I have to enable JavaScript and run arbitrary programs from untrusted websites to access them. One browser that I use doesn't even support JavaScript (w3m). Fortunately it is unaffected because it doesn't have the string "Mozilla" in its UA and Anubis lets every user without that pass to not hurt legitimate bots like search engine crawlers. As said Anubis is trivially bypassed. The only reason it isn't bypassed is because it's simply too irrelevant to the AI companies.
I empathize, I hate the bloated JS-heavy web and avoid it when I can. I just don't see a better solution. The options for me are: sacrifice some legit users by using Anubis, or sacrifice all users by shutting down my service because I can't afford to serve the bot traffic.
There are false positives. Every time I've used a tool like this, I get reports from my users about it blocking their access. It's especially bad for users who use privacy oriented browsers like Brave or use VPNs.
For a personal site where you don't care about 0.1% of your users getting blocked, it's probably fine. For a business, blocking some of your paying customers isn't acceptable.
> For a business, blocking some of your paying customers isn't acceptable.
That depends on your business and the costs you're incurring from the bots. It might be justifiable to lose 0.1% of your customers if it means saving server costs from serving useless bot traffic.
I suppose it would depend, yeah. I find having paying customers being blocked to be completely unacceptable. Not only is a bad customer experience, but they tend to be very vocal about it. Posting online, leaving bad reviews, etc.
You are quite incorrect. I make a living off a website and 99.9% of my traffic is from bots. Literally hundreds of millions of requests from bots. The cost of bots is worth the cost of not harming legit users.
Over a slow mobile connection Anubis doesn't load at all, even if the actual website would load just fine in seconds. Hence the user is locked out from the website.
Most of the friction is just JS overhead for the computations, a compiled solver is like 1000x faster. If Anubis ever gets popular enough that scrapers care, it would be trivial to defeat. And last I checked you could bypass it by just modifying the user agent
I wish they'd ease up on the whole "don't change the logo without paying us" thing. The furry anime character is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
It's actually a genius idea. If you are someone who the professional presentation of not having an anime girl on the loading page is required, then you can afford to fork over the cash to fund development.
I guess the disconnect here is a bunch of HN'ers believing professional companies and websites want to attach their branding to a sexualized anime character and that they are willing to pay to remove it.
Which one then wonders why they would install it in the first place.
In any case, if you run a professional website, this immediately comes off as juvenile and/or amateurish. And am y people just assume it's part of your website.
Perhaps you should check if anybody else sees what you see. After browsing through every image I could find in the history of this project, I, at least, am convinced: any "sexualization" _you_ see is entirely in your head.
> ... in judeo christian societies
I really hope you're not trying to whitewash your own bias by alluding to those religions. From everything I've read of their texts, and interacted with practioners of those religions, I'm unaware of any bias that classifies _cartoon depictions of children_ as "sexual".
Perhaps you should try whitelabeling instead. Don't like the packaging? Package it yourself!
I'm not talking about furries I'm talking about the mascot of the Anubis software program, which is not in any way sexualized.
Do you believe any anthropomorphic cartoon animal must be implicitly sexual because "furries exist and have sex?" I hate to break it to you but just about everyone has sex.
Also, Looney Tunes must horrify you. Wile E. Coyote isn't even wearing any clothes at all, and they show that to children!
> is a turnoff for anyone with a brand or personal image that doesn't mesh with those subcultures.
Most people either don’t know or don’t care about “those subcultures”. I bet a lot of older people think it’s a cartoon figure of Betty Boop (nurse) and miss the furry bit since it appears and disappears quickly. Most people also don’t have a brand.
So it seems like you’re describing a concern that only affects a tiny fraction of people:
- Not interested in paying for custom branding, so obviously not a corporation or influencer
- Dislikes cartoons
- Aware of, and hostile towards, “furry” subculture
That has to be an exceedingly small fraction of potential users of Anubis, and given how much businesses and branders will pay to custom-brand something, I’d counsel them to stay the course. Sure, a few never-payers will never pay, but they wouldn’t have anyways, so they can cope with Nurse Betty or look elsewhere for a competing free product.
If you think about this in physical market square terms — in other words, a bazaar — it seems horrendously rude to complain about a shop logo sticker on a free product handed out to anyone that walks up and asks for it. If you want it white-labeled so you can write your own name/logo on it, you pay for the privilege of displacing their name with yours. But you don’t stand there and loudly complain that their shop mascot has dog ears while holding a freebie bag of product, without losing the respect of everyone who hears you doing so.
I'd argue its rather functioning exactly as intended with regards to obtaining paid users.
Also this view seems a bit elderly. For most towards the end of the millennial curve and younger, anime is no longer subculture, its just general culture at this point. Although I would agree it isnt necessarily what you'd want for every platform and web presence.
How hard is it to maintain a fork that changes nothing except the logo?
If you can't be bothered to maintain a trivial fork, then why should the author of anubis be bothered to serve your branding needs? It's not like you have a service contract or anything do you?
It's MIT licensed, you are free to do whatever you want to it, including removing the logo. They're basically just saying "we'd prefer you didn't do this, but we're not preventing you from doing it."
You wish they'd give your preferable imagery for free and not make you feel bad for using their free software for personal gain. Your subculture is irrelevant and not special.
It seems to come up every time this complaint arises. "I want to use an open source project without being associated with furries" isn't that unreasonable of a request.
There is a difference between some maintainer somewhere being involved in a subculture and graphics from it being the first thing you see when you visit an unrelated website.
My problem is more that I don't want open source infrastructure I don't control to be infected by this. For my own website I can just make it efficient enough not to need hacks like this.
Nah, I don't especially like the logo myself when I come across those sites, but I do like that "brand" people who want to make money off it and not pay dislike it even more.
It's really smart of them to manufacture a negative sentiment about the project because the sexual fetish icon gets injected into all kinds of places where it is absolutely inappropriate?
Someone willing to take the 2 minutes to ask Claude to change the logo can probably also ask it to fix any bugs they find, or add any new features they might want.
This is like getting angry at cookie banners instead of all the companies tracking and selling your data.
You're complaining about the symptom (needing to have these checks) not the cause (if they don't, 99% of their traffic will be bots, the site will slow to a crawl and be unusable anyway).
In any case, I saw the dumb anime girl for about 2s then the site loaded. Not a big deal.
The GCC bug tracker uses the meta-refresh challenge, which does not require JavaScript. Due to the fact that the server makes sure the client has waited at least 75% as long as it should, the HTML has to add one second to the meta-refresh wait. Patches welcome. Meta refresh granularity is in single digit seconds.
I'm assuming a bot running a headless browser instance can still get past it?
It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.
People tried this in 2013 or so, there's no point to it. Doing proof of work in javascript in a browser is so crushingly, pointlessly slow that there's no value at all. Some browsers also intentionally detect attempts to do proof of work and attempt to block it entirely.
To help you combat these unwelcome scripts, we’ve partnered with Disconnect and compiled lists of domains that serve cryptomining scripts so you can block them directly through Firefox. When you download Firefox, tracking protection is enabled by default, blocking known cryptominers.
Technically true, but the list is mostly nonfunctional - cryptominers are basically extinct in the wild, and most of the domains on the list have expired or no longer serve mining scripts.
Which is trivially bypassed by an actual implementation of the proof of work in non-javascript, rendering it absolutely useless. The website is approximately 3800x times slower than native code, and hundreds of thousands of times slower than the CUDA kernel claude wrote. The "proof of work" is just non existent at that point, they're solved in milliseconds for what would take the browser version 10 minutes or more, it's security by obscurity being dressed up as something more.
pow_server http://127.0.0.1:8080 backend avx512-x16
──────────────────────────────────────────────────────────
uptime 00:03:12
solver ● BUSY difficulty 9, 0.3s
queue [####################............] 5/8 peak 12
──────────────────────────────────────────────────────────
accepted 1240 solved 1180
503 shed 48 504 timeout 2 4xx/5xx 10
──────────────────────────────────────────────────────────
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
hashes 3.90GH total avg 65.3MH/s
Ctrl-C to stop
Claude even made a nice little API server for it after implementing midstate compression, AVX multi way hashing, and a CUDA kernel. This doesn't stop the literal LLM it's trying to block from solving the challenges, it's really annoying that everybody is using it and claiming that it's something that's usable in the real world as a result of it using proof of work. It's obscure, and obscure is fine so long as nobody is pretending that it is secure.
It is not absolutely useless, empirically, which you'd discover if you had a website getting hammered by bots and experimented with anubis as a countermeasure.
While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.
The main author is aware that the algorithm is far from ideal for this purpose. See https://news.ycombinator.com/item?id=48869064 . If more bots start to answer the primitive challenge anubis uses now, that'll hasten implementation of a different algorithm.
Don't let the perfect be the enemy of the good enough. For now, the algorithm or challenge scheme almost doesn't matter. Since it's much smaller-scale than cloudflare's challenges, that's probably why very few scrapers and botnets bother to solve anubis's trivial sha2 pow.
Targeted attacks may not be repelled at all. That's not the point.
It does not even require the PoW thing Anubis does. I've setup a simple logic that just:
Checks for existence of a specific static cookie, if it does not exist, output a small page that sets the cookie via JS and reloads. Sadly this kills Noscript, but it would be possible to add a form in <noscript> that when submitted sets the cookie serverside.
Is this trivial to bypass? Yes. It still keeps out 95% of unwanted bots.
Reality is most do not target you specifically they just want to mass-scrape with low effort. Running headless browsers is way more expensive for their op
I've extended this with a FCRDNS checked exclusion for Googlebot.
Another quite effective measure I figured out was checking the existence of Sec-Fetch-Dest header if the User-Agent claims to be a modern browser. If you don't want to close down too much.
Also, I only apply these rules to routes that are not cheap and cached.
That's not far from what anubis does for clients that are determined to have light souls. It doesn't always send a PoW challenge.
For a webapp that sets a long-lived cookie, that cookie could be used to bypass anubis completely, or lower the weight in anubis so that it doesn't send its pow challenge unless there are major red flags. If bots start to abuse that exception, it can be removed.
I once discover you can put escaped XML or json in css content. The purpose was to have static data sets that work cross domain. No headers to configure no letting strangers run all you can eat malicious js on your site.
Define "scale". For any reasonable wait that you're willing to impose on your users, any PoW scheme heavily favors attackers. They have unlimited time and can be scraping even while they're asleep. Your visitors on the other hand don't have that luxury. You might argue that's not the point and it's only to stop dumb scrapers that are effectively ddosing your site, but if it's just dumb scrapers, you could've stopped them less onerous measures like tls or javascript fingerprinting.
It's more or less designed for it, it's SHA256 with a break in the middle for midstate compression to be effective, and the difficulty system is based on a misunderstanding of how bitcoin PoW works ("number of zeros" is never, ever a consideration in bitcoin, it's a match to a floating point target).
sha256(challenge + ascii(nonce)) means that the first compression round of the function can be cached and the second compression round is just the nonce plus the cache. This is the same trick used in Bitcoin mining and would have been avoidable by putting the nonce first, so immediately any non-naive code has to do half the proof of work as the vanilla solver.
write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to be met is in this repository. https://github.com/TecharoHQ/anubis
Then
let’s develop this more. turn this solver into a local HTTP server that can be given work in the request, and it returns solved work. make an end to end tester that sends test work to the solver and waits for a valid response. add support for solving with a GPU using cuda.
Then it was done more or less, it happily made a local server that supports solving the challenges given to it in bulk with priority based queue and can tolerate potentially tens of thousands of requests a second with no issue. The CPU time spent solving the challenges is less than the SSL setup for the connections. The GPU version does in excess of 20GH/s (but with high latency) though I didn't really test it, I'm not using this for anything but proving a point that the LLM itself can write the bypass tools and run them happily.
I'm almost ready to ship the wasm feature in the next version of Anubis after the one that's about to come out.
The big blocker is that testing against dozens of googles chrome to ensure functionality on abandoned smart TV oses takes a nontrivial amount of time. As an example of the level of debugging and the like required: https://github.com/TecharoHQ/anubis/pull/1684/changes/67621f...
My office gets very warm when chromesweep runs.
This is something that is complicated enough that even though LLM tools can help, it's not a magic bullet. It's just complicated in general.
I did the same, but sync takes it to another level - the conflict resolution is far better, especially when you have it on both desktop and mobile, and for multiple users on a team you can work on the same doc at the same time without worrying or having to check if Dropbox is working. Add the version history and it's a no-brainer.
People say "this is the one best system for every human", but if you want to learn how to design a system that's right for you (regardless of which app), read Francis Wade's Perfect Time Based Productivity[1]. (I have no affiliation, it's just one of the best productivity books I ever read)
"died by suicide" is just a modern replacement for "committed suicide", because that phrase dates back to when it was a crime, so it's regarded as making the victim look bad.
I say this as someone whose father killed himself when I was in 5th grade:
The "victims" who suffer after a suicide are the living, not the dead. These kinds of "modernizations" are transparent PC nonsense made up by well-intentioned do-gooders who have no idea how to represent the interests of other people who have a lived experience that they don't understand.
The person is dead either way. There's literally no way to sugarcoat this fact. We'd rather you just speak in plain, honest language than trying to make it sound less bad somehow.
I don't have a big issue with that particular phrase itself. Although the passive voice is designed to conceal or obscure the actor, which doesn't accomplish anything here. Attributing a suicide to anyone other than the actor starts to appear oxymoronic very quickly. Yes life is complex and whatnot -- that's a given, we don't need a reminder every time anything happens.
But really it's the transparent and ham-handed attempts by some others to smooth over the sharp edges of reality merely by re-phrasing how things are written.
People generally don't want pity, but these re-phrasings accomplish nothing other than to make clear that one person feels sorry for another.
> Attributing a suicide to anyone other than the actor starts to appear oxymoronic very quickly.
No one is an island. We’re all deeply intertwined/interconnected. We’re the sum total of our lived experiences and without a doubt some have lived far more challenging lives than others and are influenced by factors that would lead just about anyone down a dark path.
The grief felt by those left behind is the result of that aforementioned interconnectedness.
Getting back to the quoted bit, isn’t this a bit like saying “attributing grief to anyone other than the person experiencing it is oxymoronic”?
My point is not to diminish the impact on those left behind in any way. Clearly this is a traumatic event that causes excruciating grief.
But I think we also need to be honest about the environmental factors that lead to suicide. Hopelessness is one of the large causes. If there are systemic reasons causing people to feel hopeless, and if those systemic problems could theoretically be changed/improved, and such improvement lowered the suicide rate, there’s a strong case to be made that the systemic factors share the responsibility.
> Yes life is complex and whatnot -- that's a given, we don't need a reminder every time anything happens.
I don’t think it’s a given. Clearly some lives are far more complicated than others. There exists a subset of people for whom that complication will become an insurmountable problem. Often those people have been traumatized, or have never learned the tools necessary to work through their feelings.
Some people are bullied into killing themselves. Should that be attributed wholly to the person who was bullied?
Yes I already said that life is complicate because I KNEW that someone would write this very comment. But reminding people that life isn't simple isn't the PSA that you believe it to be.
Yes, everything causes everything, there is no one single thing to blame. Life is hard and complicated. Every rule has exceptions. Every truth has contradictions. Every one is a hypocrite. The world is big and complex.
We all know this already. We don't need this disclaimer to every statement that anyone makes. At a certain point, it just becomes noise.
> Although the passive voice is designed to conceal or obscure the actor, which doesn't accomplish anything here.
No, passive voice is not in general designed to conceal or obscure the actor. Especially not in the sentence here.
There were valid similar complains about crime reporting. But the language there was different. The sentence "The innocent McKay family was inadvertently affected by this enforcement operation" is trying to hide culpability. We can discuss that. These two are incomparable:
- A deputy-involved shooting occurred. (Ok, we are avoiding the actor. We do not know who was shooting.)
- A person died by Suicide. (Clear to anyone who done what.)
The latter implies that suicide just happened to the person, like they got hit by a bus.
The former correctly attributes the action to the person who killed themselves. Certainly the motivations and causes that drive people to suicide are complex, but ultimately it is a choice the person makes.
"Committed" is perhaps not the best word, since it's associated with crimes (and suicide is not a crime in many places anymore), but it's at least more active.
Think about it this way: I have relative who is vegan, so she has been trying to convince me to kill myself for many years now.
I can still choose whether I do it though, and obviously I chose not to so far, although during COVID I didn’t have much other social interaction, so I nearly went through with it.
I had agency throughout though. I’m not dead because I chose not to go through with it.
Many vegans think everyone else is evil/demonic for eating meat. “Meat is murder” etc etc. So the natural conclusion to that is, according to several vegans I know, that everyone who eats meat should be forced to either stop being a mass murderer or kill themselves.
Keep in mind there was a point where I was vegan, I know several vegans, so I know what I’m talking about.
They’re not shy about it either—look up That Vegan Teacher on YouTube for relatively middle-of-the-road vegan behavior in action.
Comparing nagging from a relative to wrongful prosecution is asinine. You might as well say that you had heartburn and it didn't kill you, so what's with all these people dying from heart attacks?
Agency is the ability to act. If someone dies against their own will, they don’t have agency, which is why we don’t use language like “they committed their own death” to refer to such instances.
Because there often is an external attribute, especially if you consider illness to be “external” as is conventional for most deaths caused by illness.
Healthy, sane people in good situations don't kill themselves.
It follows from that fact that if someone kills themselves, at least one of those things was not true. And those things can and often are thrust on people, or at least occur against the will of the person.
In this case, a bad situation was thrust on a whole bunch of people, and it ended up killing some of them.
> Healthy, sane people in good situations don't kill themselves.
Correct. This has no bearing.
> it ended up killing some of them.
No, and it's irresponsible and unhelpful to act like agency and choice is not part of the equation. As if to say that basically everyone chooses the same way (euthanasia) in the face of terminal illness, or depression.
Tautologically, if you want to convey that help is out there and that a better life is possible, then you're saying people have a choice to make.
There's a lot of agency in heart attacks too, but we still say that the heart attack killed them, not that they killed themselves with a heart attack.
There is agency, but it's equally irresponsible and unhelpful to act like outside factors are not part of the equation, and that someone who drives a person to suicide is blameless.
Let's say someone jumps out of a burning building and they're killed by the fall. Did they have agency? Responsibility? Should we describe that as "committed suicide"?
Heart attacks can and do happen regardless of optimal diet and lifestyle, but notwithstanding, indeed CVd can be likened to a slow-motion suicide, like cigarettes, given modern knowledge most people are aware of.
No one is saying outside factors aren't part of it. But you cannot negate or mitigate the fact that people make a choice with suicide that is not inevitable.
Because its obvious. The margin case of escaping certain and imminent painful death in favour of another kind is not working in your favor. Yes they are responsible for their actions, yes they have agency. It is still suicide but that is a moot point when you are choosing between deaths. Here too, jumping is not something that invariably happens. That doesnt make it a bad thing. That is a value judgment that depends on the circumstance of the alternative being factually and irreparably worse.
Years ago, this used to happen a lot because registrars could grab a domain for a short period without any cost - so some of them would see you searching for a domain, and hold it, then try to make you buy it at a higher cost. I don't deal with domains any more, but I thought ICANN had put a stop to that practice.
I heard this rumor and to this day don't use purchase pages to search for domains; I query the whois database directly. If the TLD owner is in cahoots with scalpers, they'd see the search query come in no matter what I use, so that seems like the best option and it's super easy from a unix commandline as well (a whois tool is default installed or a click away in most distributions)
I remember downloading this on a 9.6kbaud modem from a local BBS... and being absolutely blown away by it! My little brother played it to death, so much so that it inspired my first attempt at a game - "kill jazz", where you simply kill Jazz many times over.
Of course there are different environments. If you work in the public sector you won't be fired unless you break the law. If you work somewhere with a lot of investor money coming in, then your employment is not dependent on your productivity. As long as the money keeps coming in, you're safe. Once it stops, everybody is out, even the hardest workers.
And there's even good companies, where they will give a bad employee a chance to become better.
But in more everyday workplaces you first don't get hired unless you're productive, and you secondly get fired if you're not productive. When/if the boss comes around to threaten about working harder, it's almost always a scam, because if there really was any issue, you'd been fired already. This becomes less and less of an issue the better paid a job is, because at the higher levels people know well if they're good or not.
[1] https://anubis.techaro.lol/