Today’s major machine learning models-as generative language models or vision-language models, are so great that the process of educating them is typically divided across thousands or even tens of thousands of GPUs. Even with all that parallelism, training still often takes months.
With such a massive implementation of resources, hardware and software errors are common that often occurs several times a day. To reduce wasted work when resources fail, the uninvited procedure for training is great training Control pointingOr regularly copying the model states for storage servers on the network. In this way, if a resource fails, its latest checkpoint can be retrieved and EITH can be reloaded or copied to a new machine and training can continue.
Because the models are so wide, remote storage control can take a while – maybe 30 or 40 minutes. So it’s made sparse, usually sparrow evening three hours. If a resource fails and the training has been back to the last checkpoint, it could mean the loss of several hours of work. And on top of that, it can take 10 to 20 minutes just to pick up checkpoints from storage. If errors occur several times a day, they can seriously slow down training.
In a paper that my colleagues and I present at this year’s Symposium on Operating System Principles (SOSP), we describe a checkpoining process that Instratead to rely on remote storage saves control points in the CPU memory of the machines involved in model training. This makes both checkpoining and retrieval much more effective, to the point that we can control after each training step, so that you put training back so far. In our experience, this reduces approach the training time lost to hardware or software failure by approx. 92%.
In our paper, we explain how we address two major challenges for Ove approach: Optimal control point location on machines and optimal traffic planning to accumulate both checkpointing and training.
GPU training
A typical GPU machine includes CPUs for general treatment tasks-Inclusive Allo-Tros for GPUS and Eight or SO GPUs that have a special-formless architecture that is optimized for massively parallel tasks, such as model training. Each GPU has its own memory, but the CPU memory is much larger.
Large Machine Learning Training (ML) Model – or unit Model – clusters of thousands of such GPU machines. Communication between machines in a cluster is much higher bandwidth than communication with remote storage servers, which is one of the reasons why CPU control points are so effective.
Optimal checkpoint location
In our approval, which we call Gemini, each machine control points for a “Ram Drive” on board – that is, a dedicated part of its own CPU memory. This is sufficient for recovery from software errors that typically do not compromise on the contents of the ram drive. To restore from hardware failure, each machine also checks to the CPU memory of at least one other machine in the cluster.
The person who trains the model can specify how many copies of each checkpoint need to be stored on the network. Typically this number will be two or three but let’s call it M. Gemini shares the training cluster in groups of M Each machinery and each machine checkpoints for the CPU memories of the other machines in its group.
In our paper we come that if the number of machines is evenly divisible by MThis checkpoint location is optimal. If the number of machines is not evenly divisible by MWe create so many M-Machine groups as possible without creating a one-machine group (which may result in a group with M + 1 machines).
Gemini saves control points for recovery of failure in CPU memory while storing control points for other goals, such as transfer learning and model error buging, in remote storage. This procedure is layered, so if the checkpoint is not in local CPU memory, Gemini tries to retrieve it from the CPU memory of adjacent machines; If not available, Gemini looks after it in remote storage.
Intertwined communication
During training in large model, GPUs will share model weights for calculation. Checkpoining for CPU memory uses the same communication network that training traffic does. We need to make sure the two users you get in each other’s way.
Our approaches include a system profiles that learn the lengths of the idle time between training traffic and schedules for checkpoint traffic for these time spans.
However, this approval is so difficult. A GPU that receives part of a checkpoint transmission must save it locally before copying it to CPU memory but the GPU memory is limited. We assign a small ament of each GPU’s memory to the checkpoint and send checkpoints in small enough chunks that they do not flood these assignments.
However, this means that before the GPU can receive the next checkpoint transmission, it must free up its memory allocation by copying the content of the CPU memory. If we wait for the copying to end before we sent transmission of checkpoint we were valuable time.
So we further divide each GPU memory allocation into two halves and pipeline transmission of data to CPU memory, which constantly refills one half of the allocation while the other is emptied. This optimizes our use of the precious idle time between bursts of training traffic for checkpoint traffic.
To evaluate Gemini, we used it for control points during the training of three popular large language models, and as basic lines we trained the same models using two prior checkpoining process. In our evaluation, the Gemini Checkpoint Model States for any iteration could, as a consequence, reduced the lost training time due to hardware or software failure by more than 92% compared to the best priesting baseline.
Recognitions: Zhen zhang, xinwei fu, yida wang