This is the MaxText configuration for reproducing OLMo 3 7B pre-training on Google Cloud TPUs. It covers stage 1 and stage 2, matching AI2's reference run on 5.93 trillion tokens. The setup uses JAX/XLA and a specific data pipeline to replicate the PyTorch training curve.
It is valuable if you want to verify that a PyTorch recipe translates faithfully to TPU hardware. The catch is that it requires access to Google Cloud TPUs and the specific OLMo 3 data mix. It is not a generic GPU tutorial, but a concrete validation of MaxText's numerics against a known baseline.
Leave a comment