Let's insert a subject.
Who is doing the Yobu? I am. Who is the target of my Yobu-ing? My friend. Where am I Yobu-ing him to? A party.
marks the object, marks (in this case) the target location, directionally.
is the place in the invite, statically, as opposed to directionally-- You might say , "We had fun _at_ the party." You wouldn't say, "We had fun _to_ the party" in this case. But since you're calling him _to_ the party, we use .