Edit: I'm pretty sure the operation you're looking for is changing coordinate systems while maintaining Z-up or Y-up. In this case, try setting all the elements of the second column (or row) of your matrix to their inverse.
This question would be better for the Math StackExchange. First, a really helpful read on rotation matrices.
The first problem is the matter of rotation order. I will be assuming the XYZ rotation order. We know the rotation matrices for each axis is as follows:
![Rotation matrices for each XYZ axis component]()
Given a matrix derived from the same rotation order, the resulting matrix would be as follows, where alpha is the X angle, beta is the Y angle, and gamma is the Z angle:
![Completed rotation matrix for XYZ rotation order]()
You can derive the individual components of each axis angle from this matrix. For example, you can derive the Y angle from -sin(beta)
using some inverse trig. Given beta, you can derive alpha from cos(beta)sin(alpha)
. You can also derive gamma from cos(beta)sin(gamma)
. Note that the same number in the matrix can represent multiple values (e.g. sin(0)=0
and sin(180)=0
).
Now that you know alpha, beta, and gamma, you can reverse beta and remake the rotation matrix.
There's a good chance that there's a better way to do this using quaternions, but you should ask the Math StackExchange these kinds of language-agnostic questions.