To capture an enemy colony, send a fleet to their system and initiate a planetary invasion. You must first defeat any defending fleets in orbit.
Invasion Process
Ground combat pits your invading forces against the colony's garrison and defenses. Success depends on your fleet's troop capacity versus the colony's defensive strength.
After Conquest
Captured colonies become part of your empire but may have reduced output initially. Population may be lower after the assault. Fortify captured systems to prevent counter-attacks.