If your images contain the object already segmented, as your examples show, you can create a binary image where you indicate object vs background pixels.
After that, assuming the objects are not generally rotated or twisted, you can use simple features to make the classification. For example for the case above, just count the percentage of scan lines where there are 2 runs of foreground pixels. For a shirt this should be a low value, and for pants it should be high.
Obviously, if the given example images are not representative of the problem you're actually trying to solve, this wouldn't work.
EDIT: Some example matlab code:
function ratio=TwoRunFeature(I)
g=rgb2gray(I);
b=imdilate(g<255,ones(5));
d=abs(imfilter(b,[-1 1]));
runs=sum(d,2);
ratio=sum(runs==2) / sum(runs==1);
end
function TestImage(name)
I=imread(name);
fprintf('%s: %f\n',name,TwoRunFeature(I));
end
TestImage('pants.jpg');
TestImage('shirt.jpg');
Prints:
pants.jpg: 1.947977
shirt.jpg: 0.068627
Pants will give high numbers and shirts low. Just threshold anywhere you want and you're done.