Very cool! Can you say a little bit about the size of the DPO training examples and how long training took?
For DPO I only had around 700 preference examples, so not much data at all. That took about 12 minutes to train on a single GPU.
Pretraining was obviously a a lot slower, the 125M model took roughly half a day.
For DPO I only had around 700 preference examples, so not much data at all. That took about 12 minutes to train on a single GPU.
Pretraining was obviously a a lot slower, the 125M model took roughly half a day.