For me, the problem is not with long turns. It's things like laying down a planetary improvement taking 1-2 seconds to display. And if you do like 6 at once, as you might with a new colony, the thing maybe takes 10+ seconds to finish displaying and release control back to you.
Okay, edit: I'm entertaining the notion that this may be the mods I am running now. When I think of then, and I think of performance, the first thing I think of is "scale". The mods create a very large number of planets, resources, and that sort of thing. The odd thing about this, however, is that CPU is at all times by no means challenged. I have a 6-core computer, and flipping over and looking at utilization at any given time, I see: 1) only every other core (of the 12: 6 are hyperthreads) are used; I assume the game code is only schedule one core per HT pair; and 2) of those 6 active cores, there are only about 35% used. So if indeed there are something about my mods causing the input lag, something is hinky: we have a mystery performance problem.
Edit again; I read through my MapSizeDefs and MapSetupDefs. The maximum number of planets I'm using on modded gigantic, while > gigantic, is well under the maximums on other larger settings. The resource settings in map setup defs might be something on the level of double, not sure (as modders are aware, these settings are hard to predict).