> future models can be trained to be even more aware of external knowledge

Then you need longer contexts, which is proving to a much more stubborn problem than general knowledge compression.

I suppose the right approach would be to run queries like the ones you want your model to be able to do, sort weights/experts by usage frequency and then reareange the weight so that all but the most frequently used can stay on disk. Tricky though, it could be that a e.g. database question uses 90% of the network at some point or another

Edit: I guess at the moment this is just having an LRU cache of experts

Note that MoE is a sparsification mechanism and doesn’t actually have to do with expertise or what would commonly be considered areas of expertise.