You might want to go look at the sun-bleached bones of all your predecessors
who confidently proclaimed that all the dynamic scripting languages were slow
for no good reason and a JIT could totally just JIT all the slowness away.
I ask this with the best of intentions: where can I learn more about this?
(FWIW, I'm curious specifically about the ways in which Common Lisp (or Lisps in general) does or does not avoid the performance pitfalls of - say - Ruby or Python. But I'm definitely interested in the general topic as well...)
Study PyPy, particularly w.r.t. to the definition of RPython and why it had to have the restrictions it had.
There's a lot of issues that Python/Ruby/Perl/JS have at runtime that Lisp takes care of at compile time, where everything is technically a massive set of method resolutions. Lisp may allow macros to do the same thing, but careful construction will do more of the resolution at compile time.
If you are coming from the perspective of a Ruby programmer, there are lots of papers on optimising Ruby in the Bibliography http://rubybib.org.
But what you want to do is take a look at a few of the VM papers there, and then look at their references and see what people have tried previously. The related work sections often talk about previous limitations.
Lisps in general are much more static than a language like Ruby, so it wouldn't be very instructive to compare against them. The trickiest Ruby features are things like Proc#binding and Kernel#set_trace_func, as these require you to modify running code and data structures which may have already been very aggressively modified.
I don't think (but am not an expert) that most Lisps have features like those. Lisps have code as data of course, which sounds like a nightmare to compile, but it's really not - the code trees are immutable (I believe) when generated, so just compile them whenever you see a new tree root and cache.
With just the vtables, Kernel#set_trace_func is reasonably easy - you can do it just by introducing thunks that proxy to the real method after calling the trace function. Wasteful, though, but easy to patch in/out.
The moment you inline, I think it's still possible to do reasonably ok using ptrace as long as sufficient debug info is kept to identify call sites. In both cases I think that for my project, if I ever get to looking at set_trace_func, I'll probably require a compiler option to enable debug info, as it's not exactly frequently used. Alternatively use the approach we've discussed and fall back to a "slow path".
Binding is basically a closure that captures all local variables and arguments of its context, so it's not that tricky in most cases as long as you can identify it and alias / copy variables into a separate environment. This is already how I handle block references to surrounding variables - by hoisting the creation of the environment for the closure up to the start of the relevant scope and aliasing all mentioned variables.
eval() of course complicates this massively, but then eval() complicates everything massively - and doubly so for ahead-of-time compilation.
Features like generic dispatch in Common Lisp result in similar amounts of dynamism; methods and classes can be defined and redefined at any time so a lot of work goes into making that fast; see for example http://metamodular.com/generic-dispatch.pdf
> Lisps in general are much more static than a language like Ruby
What?
You mean running code directly off of data structures like in Lisp interpreters, using late binding, dynamic CLOS dispatch with method combinations, a full blown meta-object protocol, etc. is more static than Ruby?
I specifically addressed this. You may need to re-compile if you see a new code data structure instance, but when you've compiled the fact it came from a data structure isn't interesting. All JIT compilers have code as data - they have an IR.
> late binding
This is no more troublesome than it is in Ruby.
> dynamic CLOS dispatch with method combination
But these are optional features that you sometimes use. In Ruby all operations, method calls and most control structures use the strongest form of dynamic dispatch available in the language.
> full blown MOP
So does Ruby.
So I can't say Lisp is any more dynamic for the compiler than Ruby. A sibling of yours mentioned a trace extension but said people don't expect it to be fast so nobody is bothering. In the implementation of Ruby I work on, we've even made set_trace_func as fast as we can.
Well I'm not an expert in compiling Lisp and I guess you're not an expert in compiling Ruby, so we're probably not going to come to an agreement on this.
> A sibling of yours mentioned a trace extension but said people don't expect it to be fast so nobody is bothering.
That's a bit of a misrepresentation of what I said. I don't know about TRACE in particular (which is not an extension but a core part of the language), but the sbcl developers in particular have been very attentive to the performance of development tools like that. (And, probably, the commercial vendors, as well.)
He means, I think, that method dispatch (i.e. function resolution) in Lisps is less dynamic than in Ruby/JS, where you resolve the call target by searching for the method name in a map -- at least as an abstraction -- belonging to the target object at runtime (or Ruby's method_missing). I am not familiar with all Lisps, but their method polymorphism is usually much more structured.
Actually CLOS (Common Lisp Object System) is more dynamic than Ruby.
Every CLOS method is part of a generic function. Each generic function is a collection of methods. At runtime the applicable methods are collected, based on the runtime class graph of the arguments to dispatch over, and an effective method is assembled by a per generic function method combination.
Method selection is also depending on more than one argument (the target object), but on multiple arguments.
Various parts of that are implemented by a meta-object protocol in CLOS implementations, which make it possible to change the behavior of the object system at runtime in various ways.
(FWIW, I'm curious specifically about the ways in which Common Lisp (or Lisps in general) does or does not avoid the performance pitfalls of - say - Ruby or Python. But I'm definitely interested in the general topic as well...)