They claim Jimmy is playing both sides of the house and even told Adiran he’d vote to keep him