In a category we say that a morphism is an isomorphism if it has a (two-sided) inverse $in$ $that$ $category$. In your case the category is algebraic varieties over a fixed field, and morphisms are algebraic maps. The real problem is that even if your algebraic map has an inverse in the category of topological spaces (i.e. is a homeomorphism) there is no reason a priori that the inverse map should be algebraic. Intuitively, and in the affine case what you should ask is "does the inverse function involve taking roots?" if so it's not an algebraic map.
What exactly the "extra condition" is much more transparent in the context of schemes instead of varieties, there in addition to the underlying topological space we have the extra data of a structure sheaf. The condition for a morphism of schemes to be an isomorphism is that the underlying map of topological spaces is a homeomorphism and the corresponding map of sheaves is an isomorphism.