Basically Rafe think Brian C talk too much. He also resent how Brian C saved Lydia. When Rafe had a choice to give a hidden immunity, it was given to Gary simply because he want Brian gone.