When I study Topology, I met with a problem. On my book, it says 'we cannot admit that there exists a set whose members are all the topological spaces. That will lead to a logical contradiction, that there will be a set who is a member of itself.' But why we cannot have a set who is a member of itself?

3You should check out Russell's paradox. – Rustyn Sep 23 '13 at 09:43

5quick, create a set of all paradoxical sets  now, is it in itself? – penguat Sep 23 '13 at 13:01

When something is uncomfortable, just give it a nice name ( regularization, analytical continuation, renomarlization group, 'a new' axiom, etc… ) and everybody is happy. – Felix Marin Sep 23 '13 at 17:40

11Aaaaaarrggghhhh!! Stop mentioning Russell's paradox! Russell's paradox does not prevent a set from being its own element; indeed, there are (seemingly) consistent models of set theory in which we can have $x\in x$ but the RP set $\{xx\not\in x\}$ cannot be constructed. – John Gowers Oct 02 '13 at 12:56
5 Answers
We cannot admit that there exists a set whose members are all the topological spaces. That will lead to a logical contradiction, that there will be a set who is a member of itself.
This is not quite untrue, but at the very least, its a deceptive statement. Sure, in the usual formulation of set theory (namely, ZFC), no set can be a member of itself, but this is basically because (to oversimplify a little) we declare via axiom that "there are no infinite descending membership chains" (see also, axiom of regularity). This means that, in particular, if there was a set $x$ such that $x \in x$, then we'd have
$$\cdots x \in x \in x$$
which is an infinite decreasing membership chain. So, that would be a contradiction. The point, though, is that there are perfectly good variants of ZFC in which there exist sets $x$ satisfying $x \in x$, see also nonwellfounded set theory.
Indeed, topology can be developed pretty much independently of whether or not wellfounded sets exist. Therefore, that quote is not an example of good mathematical writing.
However, the author is correct that there cannot be a set of all topological spaces. Now you may say:
Wait, I can define the notion of a topological space. Its a pair $(X,\tau)$ such that [whatever]. And if I can define a concept, then the set of all instances of that concept must exist. So, the set of all topological spaces must exist.
Sounds convincing, right? Indeed for quite a long time, mathematicians believed the basic idea that "if you can define a concept, then the set of all instances that concept must exist." It was Russell whom first realized that this principle, which is formalized by the (contradictory) axiom schema of unrestricted comprehension, is untenable. The proof goes something like this. Suppose that for any formula $\phi(x)$ I can write down, there is a set of all $x$ satisfying $\phi(x)$. Then, there is a set of all $x$ satisfying $x \notin x$, call it $R$. Thus $y \in R$ iff $y \notin y$, for all $y$. So $R \in R$ iff $R \notin R$, a contradiction. Basically, this happens because we cannot consistently decide whether or not $R$ should be an element of itself. See also, Russell's Paradox.
So, we need some new setexistence principles, which (hopefully!) don't lead to an outright contradiction like that. The standard collection of setexistence principles is called ZFC. What you've got to understand about ZFC is, its based on the philosophy that "the only reason a set shouldn't exist, is if its too large." So for example, we have an axiom schema (namely, replacement) stating that, if the domain of a (definable) function exists, then its range (aka image) must also exist. This makes sense in light of the philosophy of ZFC, because the range of a function always has cardinality lessthanorequalto the cardinality of its domain. So, using this principle, we can always use a set $X$ to prove the existence of a lot of smaller sets (as well as different sets of equal size).
Returning to the issue of topological spaces, it turns out that ZFC cannot prove the existence of a set of all topological spaces. Indeed, ZFC can prove that there is no set of all topological spaces, because such a set would be "too big." The problem with a set thats too big is that we can use the aforementioned principle to prove the existence of a whole slew of smaller sets, some of which will be paradoxical, like the aforementioned $\{x : x \notin x\}$. So, this is the usual approach to (hopefully) avoiding the paradoxes, and it requires that big collections, like the class of all topological spaces, cannot exist. For a similar reason, there is no "set of all things," nor a "set of all sets," nor anything like that.
Of course, there are other set theories where the "set of all sets" actually does exist. This is true in NFU, for example; and, if I'm not mistaken, it is also the case that the set of all topological spaces exists according to NFU. However, these sorts of set theories tend to have their own issues, which is why most people tend to prefer relatively "tame" ZFC (and its extensions) over the relatively "crazy" set theories like NFU.
In conclusion, its true that there is no set of all topological spaces, at least in "tame" theories like ZFC. However, the fundamental reason for this is issues of size, not issues of selfmembership.
 65,164
 15
 90
 252
The statement:
A set cannot be a member of itself.
is a consequence of the socalled Axiom of Foundation or Axiom of Regularity. Other consequences of this axiom are, for example, that we can't have the following situations:
 Membership loops: $X \in Y \in X$
 Infinite membership chains: $X \ni Y \ni Z \ni \cdots$
However, it is nowadays known that a lot of standard mathematics can be done in the socalled $\sf ZF^$ setting (the ZermeloFraenkel axioms without Foundation).
In the situation of a "set of all topological spaces", we run into a more critical problem than that of a set being an element of itself, namely Russell's Paradox. It shows its face when we want to select from our "set" $\mathscr T$ of topological spaces the collection:
$$\mathscr F = \{T \in \mathscr T \mid T \text{ is not a point of itself}\}$$
Then this $\mathscr F$ induces a subspace of $\mathscr T$ (in the discrete topology, say), and we are brought to Russell's paradox.
This paradox, however, is linked to the observation that "$\mathscr T$ is too large to be a set", as opposed to "$\mathscr T$ contains itself".
The axiomatic set theory $\sf ZF$ overcomes this by restricting the Axiom of Comprehension (also called "Axiom of Specification"), which specifies how we may form new sets, in a suitable way.
 17,225
 9
 47
 121

1So you mean, in order to avoid Russell's paradox, we must assume that no set could be a member of itself? I've tried to find out some contradictions by assuming that some set is a member of itself, but failed. – Phil Wang Sep 23 '13 at 12:03

3@Phil No, in order to avoid Russell's paradox, we need to avoid the formation of (sets like) $\mathscr F$. The crucial property that $\mathscr F$ has, is that $\mathscr F \notin \mathscr F$ is accompanied by the information that $\mathscr F \in \mathscr T$. Therefore, we must conclude that $\mathscr F$ is a point of itself, which leads to the contradiction. Normally, we would simply conclude that $\mathscr F \notin \mathscr T$, and no problem would arise. In conclusion, "$\mathscr T$ is too large because it contains its Russell subset $\mathscr F$ as an element". See also user18921's answer. – Lord_Farin Sep 23 '13 at 12:24
In modern set theory we have the axiom of regularity which implies that $A\notin A$ for every set $A$.
However that is not the contradictory part. We can define a topology on the collection of all topological spaces, and if that would be a set then it will be a topological space. Now we run into either of two problems:
 If your assumptions include the axiom of regularity then you hit a straight out contradiction, as mentioned in the first line.
 However, even if you don't assume that axiom you still have a Russell paradox like contradiction, as the other answers have pointed out.
 370,314
 41
 552
 949
You might want to read into Russell's paradox. Basically, the problem is:
Let $S$ be the set of all sets which do not contain themselves.
This leads to a contradiction since $S$ cannot contain itself, nor can it not contain itself. Thus in general you can't simply say "let $S$ be the set of all sets with $Y$ property", or else you could run into a contradiction of this kind.
ZFC set theory avoids this problem by requiring that no set can be an element of itself, via the axiom of regularity. So the collection of all topological spaces cannot be a set, or else we could form a topological space on it, thus making it an element of itself.
 44,071
 8
 68
 157

2Even without regularity, Russell's paradox is not an issue in ZFC. In fact, the only reason it's a "paradox" is because of unrestricted comprehension. – MathematicsStudent1122 Jul 02 '16 at 12:57
This is essentially because of a thing called "Russell's paradox" in naive set theory (where anything you can define can be thought of as a set).
Suppose we allow sets to contain themselves. Then let $S=\{$sets $X:X\not\in X\}$. The question is, does $S$ contain itself? If it does, by the definition of $S$, $X\not\in X$. If it doesn't, again by the definition of $S$, we have that $S\in S$. Either way we have a contradiction. The only way to avoid this paradox is to be stricter when it comes to what we allow to be sets, and one way in which we do this is to not allow sets to contain themselves.
 12,372
 1
 35
 72

3There exist models of set theory in which sets are allowed to be members of themselves. See, for example, https://en.wikipedia.org/wiki/Nonwellfounded_set_theory , some of which have the same consistency strength as ZF (and, in particular, the Russel's paradox set $S$ cannot be constructed within the theory). I suppose Russell's paradox is in a way responsible for the existence of the Axiom of Regularity, which prohibits $x\in x$, since otherwise nonnaive set theory might not exist, but allowing a set to be a member of itself doesn't immediately open the door to Russell's paradox. – John Gowers Sep 23 '13 at 16:14