No, you have to look at the margin. The other 99 people are already there, and you have to choose whether or not to drive; you slow down everyone after you, but you don't slow yourself down.
Everyone before you chose to inconvenience all comers before and after. In a situation like this, especially because it's iterated every day, these people are clearly choosing to slow, and be slowed by, everyone coming after them. Being on the road first doesn't give you moral superiority.
In some situations the margin of newcomers is a fair way to look at it. But with drivers on the road, these people are all getting up in the morning and choosing to use this road at the same time. It is a continuous reaffirmation, day by day, that this is the path they want, no matter who they slow down, and they accept being slowed in return.
