When a caravan is in the same square as a hostile unit, I am unable to attack the hostile unit. The diplomacy window for talking to the caravan owner keeps coming up. The hostile unit is selected when I click on the square.
What do I need to do to get around this?
You could declare war on the faction owning the caravan, which will allow you to destroy it. If the location is inside your borders, you can probably drop a strategic damage spell on the tile (for example, Starfall or Pillar of Fire). You can wait and see if the caravan moves out from under the hostile. You can refuse all Economic Treaties offered to you by AIs, which will reduce the number of foreign caravans on the map but which will cost you some potential income.
You also haven't found the really 'fun' version where the caravan is sitting on a city you're trying to invade and blocking the only tile you can attack.
And, casting a tremor spell on said square will not have the stated effect of immobilizing for three [actually two] turns. The spell seems to get divided between the two units, causing the spell to wear off in one turn. On the good side, the caravan owner does not seem to mind.
This has been reported as a bug before. You could try finding that bug report and mention that it's still current (assuming that the bug report was for a game version prior to the one you're using; can't remember if it was a 1.0 bug report or not).
Also, If a caravan is on a road square where I would like to send my unit, clicking the square does not move my unit. It brings up the diplomacy window. I know I can just click on a square farther down the road to get around this, but I was wondering if there is something I need to press on the keyboard be able to click on the square that I want my unit on?
As far as I know, there is no keyboard key or anything like that which will get you around this.