I'll reply in good faith in case anyone else reads and wonders: if you had a working day, you would eat at your employer's. You could also well be the person doing the shopping for them and yourself for the day. For most of the period when the servants were common, people did not or rarely had fridges. There were different contraptions for keeping the food cool.

If you weren't that wealthy, you might have part-time servants. If you were wealthy, your servants would eat in their own little dining room in your manor, and shop for food for you and the servants in the same shopping trip