By The Fitting subgroup centralizes minimal normal subgroups in finite groups we know this is generally wrong, since $C_G(N) \geq \operatorname{Fit}(N)$.
In particular, every non-simple $p$-group gives a counterexample, since a minimal normal subgroup is contained in the center, and so its centralizer is the whole group. For example, if $N=2\mathbb{Z}/4\mathbb{Z} \unlhd G = \mathbb{Z}/4\mathbb{Z}$, then $N$ is minimal normal but $C_G(N)=G$ is bigger than $N$.