I think they do explain some of this, but the first thing to do is analytically continue the integrand into the upper half-plane, and move the contour to get a better integral; basically the idea is to turn the nasty oscillatory integral into a nice exponentially decaying one, essentially by using a steepest descent contour, which for $e^{iz}$ will be lines parallel to the imaginary axis.

The square root has a branch point at $i$ (or $im$ in the original case, so we can't deform the contour past this: instead, it wraps around the branch point, and we end up with a line just to the left of the imaginary axis, one just to the right, and a small circle joining them. The small circle is unimportant because the integrand is less singular than $1/(z-i)$ at the branch point, so we have to evaluate the integral as the difference
$$ im\int_{1}^{\infty} \frac{\rho}{i\sqrt{\rho^2-1}} e^{-\rho mr} \, d\rho + (im)\int_{\infty}^{1} \frac{\rho}{-i\sqrt{\rho^2-1}} e^{-\rho mr} \, d\rho, $$
where in the first integral $\rho=-ip/m$ is just to the right of the contour, and in the second is just to the left. The square root has a relative minus sign on one side of the contour compared to the other when we rearrange the $\sqrt{-m^2\rho^2+m^2} = \pm im \sqrt{\rho^2-1}$; these signs are chosen automatically by following paths from the real axis, although you can guess what they are by looking at the small-$mr$ expansion, at least the first term: very roughly, $p/\sqrt{p^2+m^2}$ is increasing from $0$ to $1/m$, so if $mr$ is very small, the major contribution near zero is positive, while most of the rest is constant and cancels out. Or even more heuristically, if $m=0$, you get a positive delta function, so the sign should be positive. The moral of the story is that the integral is equal to
$$ 2m \int_1^{\infty} \frac{\rho}{\sqrt{\rho^2-1}} e^{-\rho m r} \, d\rho, $$
and then one may integrate this by parts to produce the canonical $\int_1^{\infty} e^{-zx}(x^2-1)^{\nu-1/2} \, dx $ that gives the modified Bessel function $K_{\nu}( z)$ (http://dlmf.nist.gov/10.32.E8).