This is an interesting observation.
Let us first address the problem of convergence.

### Convergence of the nested radical formula

For the nested radical formula, we have
\begin{align}
x_{n+1}^2 = -c -b \, x_{n}.
\end{align}
This means, around the solution $x^*$, we have
\begin{align}
2 \, x_{n+1} \, \Delta x_{n+1}
\approx
-b \, \Delta x_{n},
\end{align}
where $\Delta x_n \equiv x_n - x^*$.
In other words, after a round of iteration, the error is reduced by a factor of
$$
\left| \frac{b}{2\,x_{n+1}} \right|
\approx
\left| \frac{b}{2\,x^*} \right|.
$$
This means the nested radical formula works only if
$$
|x^*| > \frac{|b|}{2}.
$$

### Convergence of the nested square formula

The nested square formula is the opposite, we can similarly show from
\begin{align}
x_{n+1} = -\frac{c+x_{n}^2}{b},
\end{align}
that around the solution $x^*$,
\begin{align}
\frac{ \Delta x_{n+1} }
{ \Delta x_n }
\approx
-\frac{ 2 \, x_n }{b}
\approx
-\frac{ 2 \, x^* }{b},
\end{align}
which means it only works for
$$
|x^*| < \frac{|b|}{2}.
$$

### Comparison of the convergence of the two formulas

This explains your observation that the nested radical formula often works for the larger root and the nested square formula works for the smaller root.
In fact the nested radical formula works *at least* for one of the roots, for
$$
\left| \frac{2 \, x^*}{b} \right|
=
\left| 1 \pm \sqrt{1 - \frac{4c}{b^2}} \right|.
$$
Now with the plus sign, we always have $|x^*| > |b|/2$. Further if
$$
-c > \frac 3 4 \, b^2,
$$
it works for both roots.
For example if $b=-1, c=-2$, with roots $x_1 = 2$, and $x_2 = -1$, the nested square formula works for both roots:
\begin{align}
2 &= +\sqrt{2 + \sqrt{2 + \sqrt{2 + \cdots}}}, \\
-1 &= -\sqrt{2 - \sqrt{2 - \sqrt{2 - \cdots}}},
\end{align}

Conversely, it means that the nested square formula works for *at most* one root, if
$$
-c < \frac{3}{4} \, b^2,
$$
which is, fortunately, your case, with $b = -3, c = -4$.

The nested square formula is actually a variant of the logistic map or a general quadratic map, which is a model of studying chaos. So maybe not the best formula for convergence.

### Convergence of the continued fraction method

For the continued fraction formula, we have
\begin{align}
x_{n+1} = -b -\frac{c}{x_n},
\end{align}
and, around the solution $x^*$, we have
\begin{align}
\Delta x_{n+1}
\approx
\frac{c}{ x_n^2} \, \Delta x_{n},
\end{align}
with the rate of convergence being
$$
\left| \frac{c}{x_{n}^2} \right|
\approx
\left| 1 + \frac{b \, x^*}{c} \right|^{-1}.
$$
This means the continued fraction formula works only if
$$
-\frac{c}{b \, x^*} < \frac{1}{2}.
$$
or, equivalently, with $1/x^* = (-b\pm\sqrt{b^2-4c})/(2c)$, we have
$$
1 \pm \sqrt{1 - \frac{4 \, c}{b^2} } < 1.
$$
So there is precisely one root (with the minus sign) that satisfies this condition.

### Generalization to higher-order polynomial equations

This method is useful for numerically solving higher-order polynomial equations, although I don't suppose it is new. For example, for
$$
x^7 - 3 \, x + 1 = 0,
$$
the nested radical formula $x = \sqrt[7]{3x-1}$ is definitely a convenient way of solving it. But it usually works for the largest root. For example, for
$$
x^7 - 2 \, x^6 + 1 = 0
$$
the formula $x = \sqrt[7]{2 \, x^6 - 1}$ does not converge to the root $x = 1$, because there is a larger root $x \approx 1.98358$. And, for
$$
2 \, x^7 + 2 \, x^6 - 1 = 0
$$
The formula $x = \sqrt[7]{\frac 1 2 -x^6}$ does not converge at all.

### Improving the convergence

We can improve the convergence. Take the nested radical formula for example. For a suitable value of $d$, we have
$$
x_{n+1} + \epsilon = \sqrt{-c + d^2 - (b - 2 \epsilon) \, x_n},
$$
This formula is convergent if $|x^*| > |b/2 - \epsilon|$. So if we can make $\epsilon$ close to $b/2$, the nest radical formula will almost always be convergent.
Indeed if $\epsilon = b/2$, this becomes the exact formula, and no iteration is needed.

We mention in passing that if rapid convergence is the aim, also please consider series acceleration methods.