Apart from the realism, I also think it's too fast with regards to gameplay. Pop growth should be dependent on pop size, otherwise, like mentioned above, new colonies go from nothing to full within a very short time, also making and pop growth bonus techs or buildings unnecessary.
It also furthers the rush for planets at the beginning. If the pop on your home planet wouldn't grow back as fast, you'd think twice about whether you really can afford to send yet another batch of colonists away. Might be a problem if the AI can't cope with that, though.