First, a warning. I suspect this response is likely not going to be immediately comprehensible. There is a formal set-up for your question, there are tools available to understand what's going on. They're not particularly light tools, but they exist and they're worthy of being mentioned. Before I write down the main theorem, let me set-up some terminology. The tools belong to a subject called *manifold theory* and *algebraic topology*. The names of the tools I'm going to use are called things like: the isotopy extension theorem, fibre-bundles, fibrations and homotopy-groups.

You have a surface $\Sigma$, it's your shirt or whatever else you're interested in, some surface in 3-dimensional space. Surfaces have automorphism groups, let me call it $\operatorname{Aut}(\Sigma)$. These are, say, all the self-homeomorphisms or diffeomorphisms of the surface. And surfaces can sit in space. A way of putting a surface in space is called an embedding. Let's call all the embeddings of the surface $\operatorname{Emb}(\Sigma, \mathbb R^3)$. $\operatorname{Emb}(\Sigma, \mathbb R^3)$ is a set, but in the subject of topology these sets have a natural topology as well. We think of them as a space where "nearby" embeddings are almost the same, except for maybe a little wiggle here or there. The topology on the set of embeddings is called the compact-open topology (see Wikipedia, for details on most of these definitions).

Okay, so now there's some formal nonsense. Look at the quotient space $\operatorname{Emb}(\Sigma, \mathbb R^3)/\operatorname{Aut}(\Sigma)$. You can think of this as all ways $\Sigma$ can sit in space, but without any labelling -- the surface has no parametrization. So it's the space of all subspaces of $\mathbb R^3$ that just happen to be homeomorphic to your surface.

Richard Palais has a really nice theorem that puts this all into a pleasant context. The preamble is we need to think of everything as living in the world of smooth manifolds -- smooth embeddings, $\operatorname{Aut}(\Sigma)$ is the diffeomorphism group of the surface, etc.

There are two locally-trivial fibre bundles (or something more easy to prove -- Serre fibrations), this is the "global" isotopy-extension theorem:

$$\operatorname{Diff}(\mathbb R^3, \Sigma) \to \operatorname{Diff}(\mathbb R^3) \to \operatorname{Emb}(\Sigma, \mathbb R^3)/\operatorname{Aut}(\Sigma)$$

$$\operatorname{Diff}(\mathbb R^3 \operatorname{fix} \Sigma) \to \operatorname{Diff}(\mathbb R^3, \Sigma) \to \operatorname{Aut}(\Sigma)$$
here $\operatorname{Diff}(\mathbb R^3)$ indicates diffeomorphisms of $\mathbb R^3$ that are the identity outside of a sufficiently large ball, say.

So the Palais theorem, together with the homotopy long exact sequence of a fibration, is giving you a language that allows you to translate between automorphisms of your surface, and motions of the surface in space.

It's a theorem of Jean Cerf's that $\operatorname{Diff}(\mathbb R^3)$ is connected. A little diagram chase says that an automorphism of a surface can be realized by a motion of that surface in 3-space if and only if that automorphism of the surface extends to an automorphism of 3-space. For closed surfaces, the Jordan-Brouwer separation theorem gives you an obstruction to turning your surface inside-out. But for non-closed surfaces you're out of tools.

To figure out if you can realize an automorphism as a motion, you literally have to try to extend it "by hands". This is a very general phenomena -- you have one manifold sitting in another, but rarely does an automorphism of the submanifold extend to the ambient manifold. You see this phenomena happening in various other branches of mathematics as well -- an automorphism of a subgroup does not always extend to the ambient group, etc.

So you try your luck and try to build the extension yourself. In some vague sense that's a formal analogy between the visceral mystery of turning the surface inside-out and a kind of formalized mathematical problem, but of a fundamentally analogous feel.

We're looking for automorphisms that reverse orientation. For an arbitrary surface with boundary in 3-space, it's not clear if you *can* turn the surface inside out. This is because the surface might be knotted. Unknotted surfaces are examples like your t-shirt. Let's try to cook up something that *can't* be turned inside-out.

The automorphism group of a 3-times punctured sphere has 12 path-components (12 elements up to isotopy). There are 6 elements that preserve orientation, and 6 that reverse. In particular the orientation-reversing automorphisms reverse the orientation of all the boundary circles. So if you could come up with a knotted pair-of-pants (3-times punctured surface) so that its boundary circles did not admit a symmetry that reversed the orientations of all three circles simultaneously, you'd be done.

Maybe this doesn't seem like a reduction to you, but it is.

For example, there are things called non-invertible knots:

http://en.wikipedia.org/wiki/Invertible_knot

So how do we cook up a knotted pair-of-pants from that?

Here's the idea. The non-invertible knot in the link above is sometimes called $8_{17}$. Here is another picture of it:

http://katlas.org/wiki/8_17

Here is a variant on that.

Interpret this image as a ribbon of paper that has three boundary circles. One boundary circle is unknotted. One is $8_{17}$. The other is some other knot.

It turns out that other knot isn't trivial, nor is it $8_{17}$.

So why can't this knotted pair of pants be turned inside-out? Well, the three knots are distinct, and $8_{17}$ can't be reversed.

The reason why I know the other knot isn't $8_{17}$? It's a hyperbolic knot and it has a different ($4.40083...$) hyperbolic volume than $8_{17}$ ($10.9859...$).

FYI: in some sense this is one of the simplest surfaces with non-trivial boundary that can't be turned inside-out. All discs can be turned inside-out. Similarly, all annuli (regardless of how they're knotted) can be turned inside-out. So for genus zero surfaces, 3 boundary components is the least you can have if you're looking for a surface that can't be turned inside-out.

edited to correct for Jason's comment.

comment added later: I suggest if you purchase a garment of this form you return it to the manufacturer.