Age of consent is a social value. It even varies from state to state, which denies individuals rights on one state they enjoy in another. If male-female marriage is a made-up social value, which I agree it is, then so are age limitations. They've changed over time as well.
The problem here seems to be that some people believe we can't just "make up" rules. Of course we can. If you think every law against an act is invalid if there is no harm done by the act, then you have a long list of battles, far beyond marriage.
In reality, laws are just mandated social values. "Harm" is relative. To someone in NAMBLA there is no harm in their perversion. We don't give them the benefit of the doubt, we decide based upon our cultural values. To polygamists, there is no harm in polygamy. Again, we differ.
To be even more crass, you can kill a sheep, skin it, eat it, make clothes from it's skin and decorate your house with its skull, but you aren't allowed to have sex with it. More social values. Without them, we'd have nothing.
So, God help us if the courts decide "Well, that shouldn't be illegal; I don't see any harm in it." That would leave us at the mercy of only a few people's values, instead of society as a whole.