Joseph tells Brittany he is actively trying to avoid the other girls in the house