Cory says a lot of their mistrust came from the fact that everyone assumed Cory would tell America everything