is not rude unless you make it rude. It has no inherent rudeness within it, but because it carries a fairly strong notion that one doesn't know what's going on, it can be seen as rude if you direct it at someone to express frustration. Like you say, I always liken it to the English phrase "on Earth," as in "What on Earth are you doing here?" This sentence could be read in a friendly way or a rude way, depending on context, intonation, etc. But I don't think it's inherently rude, but rather strong feelings of puzzlement are often used in rude ways.
A more neutral usage in Japanese would be like, for example, you heard a loud banging sound outside your house/door/wherever, and you said which I would translate as "I (am strongly perplexed and) wonder what on Earth happened?"