The problem has nothing to do with compound interest calculations, but about how machines work.
Although numbers can become infinitely large and require more and more precision, computer memory is _not_ infinite.
(Floating-point) numbers are not represented perfectly in computers or calculators, hence you get these kinds of oscillations for very large (or very small) numbers without it actually happening if you were to do this purely algebraically using pen and paper.