The universe doesn’t care what you think is easier.

It moves, you moving from one point in the universe to another needs to take into account where all the moving parts are going.

Only instant teleportation, where nothing has had time to move, would work. But that would be akin to traveling back in time.