Actually, no.
As long as your baby is healthy to begin with, going outside on a mild day can be a boon for you both (if your baby is a preemie or has a weak immune system or other health problem, talk to your doctor about possible precautions). The nice weather and sunshine are naturally invigorating, and lots of moms swear that babies sleep better after getting some fresh air -- many squeeze in naptime outside, too. Being outdoors is far better than being in a room with poor ventilation; some European hospitals place newborns outside for short stints daily! There's no way you can filter every microbe that floats by, so don't worry too much about taking your infant out for a walk. Enjoy!