Mysterious Gamma Rays May Not Be Emanating From The Fermi Bubbles After All