How to denote 4th level of nested parentheses? When dealing with equation that contains nested parentheses, they are usually denoted using the {[()]} system, with 'regular' parentheses being the innermost member, square brackets middle member and curly brackets the outermost member. But what if we have a equation containing more than three levels of nesting? Is there a convention on brackets denotation in this case?

I'm going to go out on a limb and say that your convention of alternating parenthesis is not a convention at all and that for consistency one should always use parenthesis. e.g $((x - 3(z +y))^2 + (57 - q(a+b)))^3$ is the "correct" way to do it. However _clarity_ is often more important than "correctness". e.g $(\\{x - 3(z + y)\\}^2 + \\{57 - q(a+b)\\})^3$ is easier to read and $[\\{x - 3(z + y)\\}^2 + \\{57 - q(a+b)\\}]^3$ is even more so.

At any rate I'm confident in saying there is no convention.

Heck, if you are able to, use colors. That's even clearer.