Beyond learning inference optimization, I’d also spend time understanding context management, memory systems, caching, and agent orchestration. Those seem to be becoming important building blocks for production AI systems.
“Relevant to anyone building failure-attribution systems for agent pipelines — black-box distillation techniques here could feed into causal attribution models without needing white-box access to the underlying model.”
That is easy when you can control the teacher model yourself and you want to transfer its capabilities to a smaller model.
If the teacher model is run by an external entity, e.g. Anthropic or OpenAI, then the number of queries to the blackbox model that is required is so great that it should be easy for the owner of the teacher LLM to detect and stop any such attempts.
