I could also see a matrix representation as lists of lists, and when you do anything you're always consing, and replacing elements of lists, and walking lists to get to elements. Compare that to a C-style 2D array of float or double, and Lisp looks horrible - plausibly 100x as bad, and worse as the size of the matrix increases.
Of course, the problem with that is the Lisp data representation used, not Lisp itself.
Except Lisps that are intended for numerical work, like MACLISP on PDP-10s at the time, did have arrays. And as noted, MACLISP was around then faster than DEC's FORTRAN (an issue DEC fixed not too much later).
Lispers aren't stupid, which should be distinguished from how easy it is make a simple Lisp. Making a performant one takes effort on the scale of making any similar language implication good and fast.
I never said that Lisp didn't have arrays. I said that representing a matrix as a list of lists would not perform well.
What I meant to be saying is that Wolfram plausibly may have looked at a bad data representation for matrices on Lisp, and concluded that Lisp was inherently 100x as slow.
Of course, the problem with that is the Lisp data representation used, not Lisp itself.