Would 方角 also work here?
方向に飛ぶ refers to a more precise direction than 方角に飛ぶ.
The image of 方角 is like a compass, which usually categorizes directions into eight (north, northwest, north east, south…, etc.) So, you can say
メッカの方角に向かってお祈りをした。(I prayed to the direction to Mecca.)
渡り鳥は北の方角に飛んで行った。(Migratory birds have flown to the north.)
However, we don’t say 方角に飛ぶ in the context of golf game where a few inches/centimeters matter a lot to one’s score.