No, on the contrary the lead rider also gets a boost.
The reason to be unhappy about someone drafting you is that they're too close to be able to react if there's a problem - if you go down they will run over you.
The way I understand the boost is that a solo rider is effectively dragging around a volume of low pressure air - you push the air out of the way creating a high pressure volume in front of you, but that displaced air doesn't magically close in behind you, it flows out then flows back, creating a "missing air" volume behind you. If something fills in that space you don't get sucked back into it as much. There's a mention on wikipedia chain gang (riding) which links to the exploratorium article but not to a primary source that I can see. That explanation is common and makes sense, so I've never chased it further.
Here is a link to a wind tunnel video which shows that the lead rider receives a small benefit from having a drafter.