That all sounds very weird. We have pot luck dinners over here too, but still the deserts get put out after. If you're late, you might miss out on first course. That's an extra incentive not to be tardy.
My family have a lot of those dinners. We're experts. We all take twice as much as anyone could eat and then take half home, or swap stuff and eat someone else's left overs the next night for dinner.