Given that GC has always been about exploration and doing it before the other civs find all the good stuff, here are some tips to help you get started in a more efficient way

• One thing to do is try to find a better colony target as fast as you can close by. Using the executive action Telescope Takeover let's you select a location and see everything within 10 hex radius (just a bit more than your starting influence area). Pick a pair of stars (1 of them yellow if possible) nearby that might fit within that and see if that area has new colonies to exploit
• It would also help to be faster, especially when you have the whole galaxy to explore. An easy way to do that is to hire a leader with a high diligence (6+ on the purple lightning bolt symbol) and make them the minister of Exploration. You get a +1 move starting the next turn and the higher the diligence, the farther your ships can go.
• If your civ starts with an exploration (survey) ship, it's a good idea to help it get better at surveying. The best way is to practice and gain experience, and if you have anomalies nearby go survey 2 of them. This levels up your ship and you will have an option to increase it's sensor range (+2) which really helps you discover more of the immediate area. Note that yellow and red anomalies have defenses, so best to leave them til later. Keep doing this to increase the survey range as you can stack upgrades
• Sometimes, your explored area will expose just part of a solar system. It's worth knowing that if you can mouse over a star and see the name, you can select the star and get a list of planets. At the beginning, this can be very helpful in planning where to send a colony ship, even if you aren't exactly sure where the destination planet is in the system. If a star has a good planet, send your colony ship there and its sensor will reveal the location as you get close enough.
• Any probe ship you have is just for exploration. Only survey ships can unlock the anomalies, so use the probe to go see what's at the nearby stars if you start with one. Early game, manually directly probes to go straight from star to star is more useful than a random sweep.
• One important thing is to keep your people happy, because production on your planets goes up as the population approval goes up. One thing to do immediately is to lower the tax rate from the default at least 1 step to increase your general approval, and now your starting world will be more productive in all areas (manufacturing, ship building, money and science). You lose some income from lower taxes, so start looking for way to increase your revenues.
• Another possible option is to station a ship on your home world to help protect it. Unprotected core worlds take a 10% approval hit, which you can fix by just stationing a warship there. Hire another leader and assign them as a commander to one of your starting ships and move it to your home world. Some civs are lucky enough to have starting ships of a type that increase approval even further. Some civs don't care about wimpy things like protection, so know your people.
• Colonies add production to the nearest core world, and most civs start with at least 1 colony ship so you can now use any exploring you've done and determine where a nice colony world is to go settle. colony values at 2 in mulitple areas are good, 3 is better and anything 4 or higher is worth jumping at. If your civilization needs food, don't overlook planets with low production but a high food value. You need one food total production per colony, and if your people are starving your morale goes down.
• Your executive orders have a Draft Colonists option and it's worth spending some control points if you have more good candidate worlds nearby. There's no reason not to get the ball rolling by executing that option at the beginning and get those colony ships out there grabbing new useful worlds. New colony ships from this option always spawn at your home world.
• If you have any money and and any available leaders left, getting a smart one (6+ on the blue lightbulb icon) to be your minister of science is a good idea. Hire the leader and assign them as the minister of science. This increases your choices of research areas and gives you a civ wide bonus on your actual research based on their intelligence. You should go pick a research area now that you have more choices. Anything available that doesn't take 6+ turns is fine to start with, you'll need it all eventually.
• That being noted, first pick for tech is important. A reasonable default choice is the one that gives you asteroid mining, if your starting system has asteroids. Anything to help increase the production of our home world.
• You should now select a new ship type to build at your home world shipyard. If you have more planets that are worthy, I'd suggest adding another colony ship to the queue. Keep in mind that this will drain 1 population from your planet, but that should fine at the start. You can queue up multiple ship types, so I'd suggest adding another probe right away.
• Lastly I'd suggest setting your initial policy such that it doesn't have an approval loss to start with. Each civ will have different starting policies so what you do choose depends on what's available. More ship speed or citizen approval would be helpful at this point.

Hopefully that's enough to get you working as efficiently as possible at the start

Edit: Updated for grammar and clarity