I am interested in learning descriptive statistics. But I am completely baffled, that there seem to be no mathematically rigorous books on this subject, as far as I know at least. The Wikipedia page states, that descriptive statistics is not based on probability theory. If that is the case, then on what is it based on?

It could be that this field is simply not an area of mathematics. However, I believe this idea to be rather absurd, since it is clearly using some level of regular mathematical reasoning. I am not in any way saying that when teaching a subject, mainly appealing to intuition is bad, but rather that developing mathematics "in thin air" is fairly unsatisfactory.

Given all this how do I, a rather mathematically-inclined person, go about learning descriptive statistics?

Some general statements about the nature of said subject with respect to what I talked about may be helpful as well as links to any potentially useful material.