I think it has to do with time lost from misreading the clue. The typical penalty seems to be 30 min. My guess is that if it would be more than 30 minutes, they give them that penalty. Also, they only let the teams go back when it's walking to the pit-stop (minus Heave). I think when the nerds made a mistake, it was in the middle of the leg.