It seems they are doing fine-tuning and soon pre-training with their new cluster technology. They claim to be able to train Llama 3 70GB in 1 day (instead of 1 month). Impressive if true.