The spoon's conductivity would only heat the coffee if you were holding the spoon with a super-hot hand. Assuming the spoon (and your hand) are both cooler than the coffee, they will both conduct heat away from the liquid, causing the coffee to cool a teensy bit more quickly.
Stirring the liquid should logically only speed the process.
Believe it or not, a fellow nerd on physics.stackexchange.com conducted an experiment on this very issue and published the results, which confirm my hypothesis:
<