Since $G/Z(G)$ is the inner automorphism group, and it is normal in $Aut(G)$, this has to be either trivial or all of $Aut(G)$. Thus $G$ is either abelian or a simple complete group. In the abelian case, you've already found a nice set of examples: $Z_2^n$ for $n\ge3$. For the simple complete groups, the Mathieu group $M_{23}$ works.