If I can recall, the first purpose of the AG building was to reduce champ recruiting cost. In that time, heroes were not so expensive, so it was of little use. For that reason, Devs introduced the XP gain, as a way to give some balance to that building.
Later, recruiting costs were increased, so the building now might have some purpose. The XP bonus has remained. Normal, it is so low, that better leave it, than change code 
Part of the issue comes too from Path of the Governor, wich is intended to work the opposite as you will normally use heroes. But well, supposing that you have a hero crippled and/or want to give him PATG. You will move him through different cities to use the unrest/growing skills, so the XP attached to a single city becomes useless anyway, unless you build it in the city you want to grow and set quiet the hero. But that may change along a game, and it probably will. I have tried, it is SO HARD to keep a hero quiet, even if is POTG, because you want level him to get more traits...
For that reason, I think that if the XP bonus was applied in all cities, it would made at least some sense, as you can have a Governor doing something with some sense, moving him to different cities, and still winning a little bit of XP. Little, because is a very small bonus, but at least, would give a reason to build it.