By social norms, not running matrix multiplication in their head ad nauseam

I don't think that argument holds water, because the scope of what can be reasonably approximated by matrix multiplication is extremely broad (especially when the matrix multiplication is also paired with a source of randomness).

Dude we aren't performing some nonsensical calculation in our brains.

Mathematics is an invention of humans.

Not sure about "nonsensical calculation", but our brains perform computation (read: they transform information). In fact, that is what the brains are for (see: biology 101). As for mathematics, I am with Galileo: it is the language of the universe. Of course, we "invented" its silly glyphs and the like, but those are secondary. My 2 cents.