Kelley and Jimmy agree if either of them are next to Rylie, they’re gone