When downshifting the spring of the derailleur is pulling the chain off of the one gear. In contrast when upshifting the tension of the cable is causing the movement and there generally is much more force (tension on the cable). This is why upshifts typically happen faster.
If the downshift is accurate but slow then friction is usually to blame.
In addition to cleaning and lubricating the cable you can also try:
1. Checking the B-screw to make sure the jockey pulley isn't too far from the cassette cogs.
2. Turn the adjusting barrel clockwise (screw it in) a half turn. This will bias the rear derailleur towards the downshift. If you find the upshifts are getting slow or noisy then turn the adjusting barrel counter-clockwise a quarter turn.