They're heavier, hotter (vents would do the opposite of what you're trying to do with an aero helmet, which is to route air up and over the helmet), and it's more difficult to turn your head (it doesn't completely prohibit turning your head but you lose the benefit of the shape as you turn it off-axis) so you have to depend on peripheral vision more.
Plus, if you wear one on a bike path while towing a kiddie stroller federal law requires all other riders (including small children riding Huffys) to treat you with "derision and scorn".