Lex is like a cult leader. When he's on the island, he's in charge. Colby and Richard don't have the widespread ties to the Survivor community that he does. They didn't have a chance against him in All-Stars.