They haven't necessarily given a bad impression..just that they are being misjudged..lets say Sandy and Debra get "voted out" based on the fact that on paper they could be seen as the weakest. They didn't so much make a bad impression as they were just judged prematurely as weak. This twist helps give them a chance to prove themselves. Switching one person to the other tribe before they really get to know their original tribe is just lame and unnecessary.