What is a transformation that can't have shearing called? What is a transformation called when it can have separate scaling for x and y, rotation, and translation, but it cannot have shearing or scaling AFTER rotation? Basically if this transformation is applied to a axis aligned rectangle it will always remain a rectangle. It might be have different dimensions, angle, and location, but it will still have 4 right angles. Is there a name for a transformation with these constraints?

If terminology were consistent, the thing you ask for would be called an orthogonal transformation, and the things which are actually called orthogonal transformations would better be called orthonormal transformations.

But since terminology is as inconsistent as it is, I believe the best you can do is stating that your transformation matrix is made from orthogonal vectors, forms an orthogonal basis, or that your transformation maps the unit vectors to pairwise orthogonal vectors.