My guess is that this is a _very_ classic error in computer graphics implementation; you have to make sure that you don't accidentally use the new value of X when computing the new value of y! For instance, this code:
> x = x*cos(theta) - y*sin(theta);
> y = x*sin(theta) + y*cos(theta);
will actually perform the following operation:
> x' =x cos θ \- y sin θ
> y' = x cos θ sin θ \+ y (cos θ \- sin2θ)
and for small values of θ (i.e., incremental rotation) will generally cause the point