Let $f:X\rightarrow Y$ be an arrow in $\mathsf{Set}$. $f$ induces three functors: $$\exists (f),\forall (f):\mathcal P(X)\rightarrow \mathcal P(Y),\; \mathsf I(f):\mathcal P(Y)\rightarrow \mathcal P(X)$$
where the powerset is a poset category. The action on objects is given by
$$\exists (f)(A)=f[A],\; \mathsf I(f)(B)=f^\leftarrow (B),\; \forall (f)(A)=f[A^c]^c.$$
and one can prove
$$\exists (f)\dashv \mathsf I(f) \dashv \forall (f).$$
In particular, the inverse image functor is *right* adjoint to the direct image functor.

Now let $f:X\rightarrow Y$ be an arrow in $\mathsf{Top}$. Then $f$ induces the well known direct and inverse image functors
$$\mathsf{Sh}(X)\leftarrow \mathsf{Sh}(Y):f^\ast \dashv f_\ast:\mathsf{Sh}(X)\rightarrow \mathsf{Sh}(Y).$$
So in this instance, the inverse image functor is *left* adjoint to the direct image functor.

Why does the direction of adjunction change? What is the intuition behind this difference?