First let's calculate the central angle of each segment in the circle which is cut off by the square.
!enter image description here
Let $\angle BAC=\theta$. It is given that $AC=4$, and $AB={8 \over \sqrt3}$.
From the figure it is easy to deduce that in $\Delta ABC$, $\cos\theta={\sqrt3 \over 2}$, which means that $\theta=30^\circ$ and hence the central angle $\angle BAG$ is equal to $60^\circ$.
The area of a segment in a circle of radius $R$, of central angle $\alpha$ is $$area={R^2 \over 2}{\left( {\pi\alpha \over 180^\circ}-\sin\alpha \right)}$$
So now all you have to do is calculate the area cut off by the four segments together, and then subtract it from the total area of the circle.