Core routers don't inspect that field, NAT/ISP boxes can. I believe that with two suitable dedicated linux servers it is very possible to send and receive single custom IP packet between them even using 253 or 254 (= Use for experimentation and testing [RFC3692]) as the protocol number
This is an interesting list; it makes you appreciate just how many obscure protocols have died out in practice. Evolution in networks seems to mimic evolution in nature quite well.
We're about half-way to exhausted, but a huge chunk of the ones assigned are long deprecated and/or proprietary technologies and could conceivably be reassigned. Assignment now is obviously a lot more conservative than it was in the 1980s.
There is sometimes drama with it, though. Awhile back, the OpenBSD guys created CARP as a fully open source router failover protocol, but couldn't get an official IP number and ended up using the same one as VRRP. There's also a lot of historical animosity that some companies got numbers for proprietary protocols (eg Cisco got one for its then-proprietary EIGRP).
Probably use of some type of options. Up to 320 bits, so I think there is reasonable amount of space there for good while. Ofc, this makes really messy processing, but with current hardware not impossible.
It uses them a little differently -- in IPv4, there is one protocol per packet, while in IPv6, "protocols" can be chained in a mechanism called extension headers -- but this actually makes the problem of number exhaustion more acute.
What if extension headers made it better? We could come up with a protocol consisting solely of a larger Next Header field and chain this pseudo header with the actual payload whenever the protocol number is > 255. The same idea could also be used in IPv4.
I didn't mean to imply otherwise. But, as you say, this is equally applicable to IPv4 and IPv6. There were a lot of issues solved by IPv6, but "have even more room for non-TCP/UDP transports" wasn't one of them (and didn't need to be, tbqh).
Core routers don't inspect that field, NAT/ISP boxes can. I believe that with two suitable dedicated linux servers it is very possible to send and receive single custom IP packet between them even using 253 or 254 (= Use for experimentation and testing [RFC3692]) as the protocol number