tl;dr version: A solution $a^\ell + b^\ell = c^\ell$ would generate a weird (i.e,. not modular) elliptic curve over $\mathbb{Q}$ for prime $\ell \geq 5$. Wiles proved that sufficiently weird elliptic curves don't exist, so that means Fermat's Last Theorem holds for $\ell \geq 5$.

There isn't an easy or concise answer to the question, simply because the number theory involved is so complicated. It had been known for a while before Wiles that Taniyama-Shimura conjecture would imply Fermat's Last Theorem, and Wiles proved it for a large enough class of curves to also prove that theorem. (The conjecture is a much deeper result than Fermat's Last Theorem, and Wiles' proof was also extended to the general case a bit later.)

The idea behind the conjecture is that for an elliptic curve $E/\mathbb{Q}$, you can reduce mod $p$ for each prime $p$ and count the number of points on $E$ over $\mathbb{F}_p$. Explicitly, given a Weierstass form $E:y^2 = x^3 + a x + b$, you can just count number of points $(x, y)\pmod{p}$ with $y^2 \equiv x^3 + ax + b\pmod{p}$. These can be tossed into something like an Euler product or a Dirichlet series to get to a resulting object called an $L$-series. (Sort of, anyway. There are corrections that have to be made for a finite number of primes $p$). If the Taniyama-Shimura conjecture holds, then we can do a bit of manipulation to get a nonzero function $f$ that is extraordinarily symmetric under the the group
\begin{align*}
\Gamma_0(N) = \left\{\begin{pmatrix} a & b \\ c & d\end{pmatrix}\in SL_2(\mathbb{Z}):\, c\equiv 0\!\!\pmod{N}\right\}
\end{align*}
acting on $\{z\in \mathbb{C}:\, \operatorname{Im} z > 0\}$ by $\begin{pmatrix} a & b \\ c & d\end{pmatrix}.z = (az + b)/(cz + d)$, where $N$ is a certain integer attached to $E$. (Specifically, $f$ is a modular form, and so the Taniyama-Shimura conjecture is also called the modularity theorem. Also, what I'm calling $N$ here isn't quite the conductor of $E$, but it divides it.)

This condition on the action $\Gamma_0(N)$ is pretty strict, and there aren't many such $f$. In particular, for $N = 2$, there isn't any nonzero function $f$ at all. But it turns out that given coprime (and nonzero) $a, b, c$ with $a^\ell + b^\ell = c^\ell$ for prime $\ell \geq 5$, the curve $E:y^2 = x(x - a^\ell)(x - c^\ell)$ happens to have $N = 2$. That contradiction implies that no such $a, b, c$ exist; that is, Fermat's Last Theorem holds for $\ell$. It's then clear that Fermat's Last Theorem holds for any exponent $n$ divisible by some prime $\ell \not = 2, 3$. But the cases $n = 3, 4$ are classical, and so the theorem holds for all $n > 2$.

So, what happens with $\ell = 2$? Well, the theorem obviously fails: There are plenty of points with $a^2 + b^2 = c^2$, and we're going to have to get a different value of $N$ for $\ell = 2$ in order for Taniyama-Shimura to still hold. There isn't an easy way to describe this part of the proof, but the basic idea is to compute the conductor of a certain elliptic curve, and there are a couple of cases describing its behavior when reduced mod $2$. If $\ell \geq 5$, we can eliminate all but a few easy cases and get an explicit value for $N$. That's probably not particularly useful, but see the caveat above that there isn't an easy or concise answer to this question. The major requirement of the proof is that $\ell$ be sufficiently large; the fact that $\ell \geq 5$ works is a happy coincidence.

If you're looking for a deeper reason why $\ell = 2$ fails, then the reason is that the Fermat curves $C:X^\ell + Y^\ell - Z^\ell$ are very different for $\ell = 2$ than for (prime) $\ell > 2$. The genus of $C$ is $g = \frac{1}{2}(\ell - 1)(\ell - 2) $, which is greater than $1$ for $\ell > 3$. Falting's Theorem states that such curves with $g > 1$ only have finitely many points where you can take the coordinates to all be rational. (The case $\ell = 3$ is different but easily handled classically.) For $\ell = 2$, this is clearly not the case; there are infinitely many Pythagorean triples. Now, "finitely many" is unfortunately a very long way from "zero," but that's the fundamental reason why $\ell = 2$ is different. The reason why *Wiles' proof* fails is that the Frey curve $E:y^2 = x(x - a^\ell)(x - c^\ell)$ gives you a different value of $N$ when $\ell = 2$, and we need $N = 2$ to get a violation of Taniyama-Shimura. As for why that value is different...well, it's a function of $\ell$, and it just happens to be different for $\ell = 2$ versus $\ell \geq 5$. I'm not sure there's anything really deep going on there.

I'm skipping over and simplifying a lot of the details here, but there's a more thorough but readable treatment in, e.g., the last chapter of Knapp's "Elliptic Curves."