I have two `3x3`

rotation matrices $R_1$ and $R_2$. Let's say $R_1$ is computed by some (stochastic gradient descent) optimization algorithm and $R_2$ is the target. I am looking for a measurement of the error between $R_1$ and $R_2$.

Naturally, I could compute the rotation angle $\theta$ it takes to rotate $R_1$ into $R_2$ and minimize this angle. Similar questions have been asked here:

However, my problem is that $R_1$ might not be a valid rotation matrix, especially in the beginning of the optimization. I see two ways to correct for that:

- Compute $\theta$ by converting the rotation matrices into quaternions and then compute the error as shown e.g. here (Quaternion distance). When converting $R_1$ to a quaternion, divide by its norm to ensure it is unit and thus a valid rotation. Problem: the formulas to convert a matrix to a quaternion assume that the matrix
*is*a valid rotation. Making the resulting quaternion unit guarantees that the quaternion is a valid rotation, but I don't know how ''far away'' the resulting rotation is and this could be a problem for the optimization. - First extract the closest rotation matrix $R_1'$ from $R_1$ by using the SVD as explained here. Using this, I know that $R_1'$ is the closest valid rotation matrix measured by the Frobenius norm, but if possible I would like to avoid having an SVD in the computation of my loss.

The last possibility is to just not care about the validity of $R_1$ and hope that the SGD is still able to converge. This also makes sense to me, because when $R_1$ is not a valid rotation, defining an angle-dependent metric on that matrix is an ill-posed problem and thus it is not clear how to handle it. I will try this for sure, but I'm also interested in more technical details about the general problem I described and/or what would be the recommended way for this scenario.