Hi Chongzhou thanks again for the suggestion.
While making a change to the UI itself will be much trickier, we've added a new feature for an upcoming version that should help.
Starting with Fences v5.87 and onward, when you click the up arrow, it will remember the position the up arrow was in last so long as your mouse doesn't move a significant distance. This means that so long as the mouse does not move too much, even if the up arrow moves, clicking again in the same position will navigate upward.
We're aware this is unintuitive, but, hopefully it makes it easier for power users to get the job done in the meantime without an overhaul of the UI.