Edit: This post has been edited to take Eric Wofsey's comments into account. The end result is the following theorem:
Suppose $X$ is a closed manifold of positive dimension. Then $X$ does NOT admit an averaging function.
If an averaging $f:X\times X\rightarrow X$ exists, it induces a map $f_\ast: \pi_k(X)\oplus \pi_k(X)\rightarrow \pi_k(X)$.
Observation 1: $f_\ast(g,g) = g$. Proof: Consider the composition $X\rightarrow X\times X\rightarrow X$, where the first map is the diagonal embedding. By assumption, this composition is the identity. Now, apply functoriality.
Observation 2: $f_\ast(g,h) = f_\ast(h,g)$. Proof: Consider the composition $X\times X\rightarrow X\times X\rightarrow X$ where the first map swaps the two factors and apply functoriality.
These two observations together have very strong consequences. For example:
Corollary: Suppose $G$ is a group with a homomorphism $f: G\times G\rightarrow G$ satisfying both observations. Then $G$ is abelian and every element of $G$ is twice another element of $G$.
Proof: First we show that every element of $G$ is twice another, or, in multiplicative notation, that for any $g\in G$, $g = h^2$ for some $h\in G$. In fact, setting $h = f(g,e)$, we see $h^2 = f(g,e)f(g,e) = f(g,e)f(e,g) = f(g,g) = g$.
Now we show that $G$ is abelian. We have \begin{align*} [f(g,1)f(h,1)]^2 &= f(gh, 1)^2 \\ &= f(gh,1)f(1,gh) \\ &=f(gh,gh)\\ &= gh \\ &= f(g,1)^2 f(h,1)^2\end{align*} so $[f(g,1)f(h,1)]^2 = f(g,1)^2 f(h,1)^2$. Writing these out an canceling, this reduces to $f(h,1)f(g,1) = f(g,1)f(h,1)$. Replacing $h$ and $g$ with $h^2$ and $g^2$, this becomes $hg = gh$, so $G$ is abelian. $\square$
Now, it is easy to see that in a cyclic group, every element is twice another iff the group has odd order. It follows that a finitely generated abelian groups for which every element is twice another is a direct sum of cyclic groups of odd order. In particular, it is finite.
Theorem: If $X^n$ is a closed manifold of positive dimension $n > 0$, then $X$ does not admit an averaging function.
Proof: Since $X$ is closed, $\pi_1(X)$ is finitely generated, and hence finite for all $k$. Since $\pi_k(X)$ is finitely generated as a $\pi_1(X)$ module, it follows that $\pi_k(X)$ is is finitely generated for all $k$. Pass to the universal cover $\tilde{X}$, which is closed because $\pi_1(X)$ is finite. By the Hurewicz Theorem mod $\mathcal{C}$), it follows that all the homology groups of $\tilde{X}$ are finite. But, by Poincare duality, $H_n(\tilde{X})\cong \mathbb{Z}$. This is a contradiction, unless $n = \dim X = 0$. $\square$
These ideas can also be used to show that $\mathbb{R}^2$ is the only surface (no boundary) admitting an averaging function.
Sketch: We have already ruled out closed manifolds, so we assume $X$ is a non-compact manifold. Then, it is relatively well known that $X$ deformation retracts onto its $1$-skeleton. If follows that $\pi_1(X;\mathbb{Z})$ is a free abelian group. This contradicts the corollary, unless $\pi_1(X;\mathbb{Z})$ is trivial. This, in turn implies that $X$ deformation retracts onto its $0$-skeleton, that is $X$ is contractible. This implies $X$ is homeomorphic to $\mathbb{R}^2$.