It likely wasn't anywhere. Homer, or rather the author of the _Odyssey_ , had some conception of the broader Mediterranean, but it was largely unexplored by the Greeks at that time, and so magical islands didn't need to be mapped onto the known world. You know, now that I say it, it's true in today's fiction, too.
That said, that didn't stop the ancients from theorizing. Strabo (9.18) thought Ogygia was in the Atlantic Ocean, while Plutarch ( _Moralia_ 12.26) said it was "five day's sail from Britain".
Interestingly, and there's a distinct lack of good research on this topic, but Ogyges, whence the name Ogygia, was tied to both Boeotia and Thessaly (via flood-myths) in early myths. That's clearly not the Ogygia that Homer had in mind, though.