Turns out it had to do with the nature of the small holes that I was printing. I had to slow the speed of the initial layer down from 25mm/s to 15mm/s and also set Cura to 'optimize wall printing order' so that it didn't jump back and forth between holes constantly.
I also sped up the travel speed to 50mm/s on the initial layer to minimize oozing (although I'm not sure this actually did anything). Print came out beautifully.
Didn't even need the raft.