This is the most direct and elementary way I know how to prove the result, although it only works for powers in the range $[0,1] \cup [2,\infty)$ which is exactly the uninteresting set, and Joriki's answer is much better regardless. I had already written this, and perhaps somebody finds it useful.

First we consider $p=1$. Set $x_n$ equal to the greatest power of 2 less than $\frac{1}{n}$. That is, $$(x_n) = (\frac{1}{2}, \frac{1}{4},\frac{1}{4}, \frac{1}{8}, \frac{1}{8},\frac{1}{8},\frac{1}{8},\dots).$$

Note that
$$\sum_{i=1}^{\infty} x_n
= \sum_{i=0}^{\infty}\Big(\sum_{j=2^i}^{2^{i+1}-1}x_j\Big)
= \sum_{i=0}^{\infty}\Big(\sum_{j=2^i}^{2^{i+1}-1} \frac{1}{2^{i+1}}\Big)
= \sum_{i=1}^{\infty} \frac{1}{2},
$$
which diverges, and that $x_n < \frac{1}{n}$, which proves $\sum_{n=1}^{\infty} \frac{1}{n}$ also diverges.

If $p = 2$, let $x_n = \frac{1}{n(n-1)} = \frac{1}{n-1} - \frac{1}{n}$ if $n>1$ and 1 if $n=1$.

Observe that
$$\sum_{n=1}^{\infty} x_n = 1 + \sum_{n=2}^{\infty}\Big(\frac{1}{n-1} - \frac{1}{n}\Big) = 1 + 1 - \lim_{n\rightarrow\infty}(1/n) = 2,$$
and so in particular this series converges.

Now
$x_n > \frac{1}{n \cdot n} = \frac{1}{n^2},$
so the series $$\sum_{n=1}^{\infty} \frac{1}{n^2}$$ converges as well.

Finally if $p>2$ then since $\frac{1}{n^p} < \frac{1}{n^2}$, the series $\sum_{n=1}^{\infty} \frac{1}{n^p}$ converges, and
if $0<p<1$ we have $\frac{1}{n^p} > \frac{1}{n}$, and so $\sum_{n=1}^{\infty} \frac{1}{n^p}$ diverges.