Rotate vector using transformation matrix, and read some angle !enter image description here I need to rotate a vector using transformation matrix. For example: I heave vector Z (0, 0, 1). I'm rotating it by 100 deg around Z-Axis. Result will be the same as input. How to compute the angle which vector was rotated around own axis? I think I need more support vectors (like I painted on pic above). Of course I'm not rotatig around 3 main axes. Rotation can be any in 3D.

Check out this Wikipedia article on the rotation of vectors in $\mathbb{R}^3$. It describes the transformation matrix used for the rotation of a vector about an arbitrary unit vector by an arbitrary angle.

Regarding the computation of the rotation angle, we consider the rotation of an arbitrary vector $\vec{A}$ about some unit vector (the rotation axis) by an angle of $\theta$ to get $\vec{B}$. The dot product of the two gives $\vec{A}\cdot\vec{B} = |\vec{A}||\vec{B}|\cos{\theta}$, or more explicitly,

$$\cos{\theta} = \frac{\vec{A}\cdot\vec{B}}{|\vec{A}||\vec{B}|}$$ This allows you to compute the rotation angle ($\theta$) from the vectors before and after rotation.

As for the computation of the rotation axis, we simply take and normalize the cross product of $\vec{A}$ with $\vec{B}$. I hope you can see why.