but i imagine enemy cavalry would be able to knock a rider off their horse.
Or just kill the damn horse. If a cavalryman were to just wade around amongst enemy infantry during an epic conflict, I have no doubt the horse would fall sooner than later, no matter the training of the horse or the rider.
Mounts should be great at hit and run tactics, but extremely vulnerable to being flanked.