In English there's a difference between a raw, rough person and raw, rough concrete, but there's a clear etymological link in using brute/brutal to describe them both.
It's similar to the way 'crude' can be used to refer to a person or oil that hasn't been through a refining process.
Regardless, it's a postrationalisation (or retranslation, perhaps?) by British architects to force some sort of etymological ancestry into the name of the 'movement' they decided to champion.
There is a difference between "Une brute" (a brute) and, as mentioned, "béton brut" (raw/rough concrete).