Like a lot of people said here - fun factor is a major problem. Most of us who find ourselfs forcing us to play are real, true old school veterans of the 4X and RPG-genres. And there just is something, something that the game is lacking or has wrong.
From my point of view the bugs aren't a big problem. It's a beta and I can live with bugs. Sure, it might crash or behave weirdly, but I kind of trust Stardock to get these fixed over time.
Starting position problem is big fun-wise. Map generator needs to have "2nd phase" after starting positions have been set. If it has that already, improve it. A fun game needs to start "easy", so that you can quickly and somewhat fast settle a couple cities, get attached to them and think your doing fine. It's important to feel you are actually doing something. Only after getting attached to your little empire challenges should start.
Cities and troops being simply irrelevant is possibly one of the biggest problems. I know I don't need those cities. I know I don't need to built those troops. What I don't know is if 5B will fix this. You will just use those now cheap troops to defend your cities, but still use your sov and your sov alone to to kill and conquer everything.
Minimized UI design is annoying and actually obstrusive. It works on some games, say space RTS-games like Homeworld or The Jupiter Incident. In fantasy setting hiding and making the UI to "tell" as little as possible is just annoying. In this kind of a game seeing UI adds to the immersion, not take away from it.
Btw, font improvement was massive.
Maybe the biggest problem is that the game cannot decide what it is. There's a lot of stuff but it's not really connected well, you don't get attached really to any parts of the game, your empire, your heroes or your sov. It's a little like Microsoft OneNote, you kind of want to like it, you kind of want to use it, it kind of has lots of nice features, but you don't really need it for anything. Take a deep breath and talk together why game lacks fun. What was fun in the games it's recreating or trying to improve upon. What kind of a play session is fun in the first place.