Town is my default choice. You can use all the towns you can get, because of the grocer line of buildings - build enough of them, and bid your food problems good bye.
Unless I play with home rules that restrict my heroes, I do not build fortresses. Even when I use troops, I first scout my enemies, and shop for a second hand fortress. I my last few games, Resoln has been providing them.
If I decide that I need to build my own fortress, I look for the most essence I can find. Low materials can be worked around with cash, but the bonuses to built troops can come from essence, and are awesome. I will usually have production boosters while I grow the fortress, and switch all spells to Auras when I start producing troops.
If I am not needing a fortress at the moment, cities with 3 or more essence are automatically conclaves.
If a city has 2 essence, I consider the rest of my Empire - am I going to run of food soon? Am I going to be wasting growth? If not, I make it a conclave.