I will submit this feature request directly to the developer.
I can see your issue with it, I would think the expected behavior would be to delete items directly in the Fence (easier to get to)
Another alternative for the time being is that F5 (with focus) will also refresh the Fence, saving a little bit of time over right click- refresh
Thank you for bringing this to our attention, it is greatly appreciated!
Associate Technical Support Analyst