Rogue black holes could be wandering at the edges of the Milky Way