Baby's skin color mainly depends on you and your partner's skin color, the skin color of both of your ancestors and exposure to sunlight, which can trigger genes that darken baby's skin. You may want to spend some time looking at pictures of your grandparents and their parents, says Stevens.