A public-facing page at $DAYJOB has 44 boolean filters for a bunch of different hardware components (think screws and fasteners rather than boards and drives), meaning 2^44 different combinations ~= 17 trillion different pages to crawl (from the disrespectful AI crawler's perspective).
That said, it wasn't really an issue except we had a misconfiguration in the in-memory caching database we were using so it didn't delete old stuff from memory and started writing to disk, which meant that we ran out of our (millions of) file inodes on the production server. Just saying to point out that keeping the state in the URL wasn't the problem in this case, and it is generally not a problem; it can be a problem but, when it is, it will be obvious why.
A public-facing page at $DAYJOB has 44 boolean filters for a bunch of different hardware components (think screws and fasteners rather than boards and drives), meaning 2^44 different combinations ~= 17 trillion different pages to crawl (from the disrespectful AI crawler's perspective).
That said, it wasn't really an issue except we had a misconfiguration in the in-memory caching database we were using so it didn't delete old stuff from memory and started writing to disk, which meant that we ran out of our (millions of) file inodes on the production server. Just saying to point out that keeping the state in the URL wasn't the problem in this case, and it is generally not a problem; it can be a problem but, when it is, it will be obvious why.