I was just wondering whether or not there have been mistakes in mathematics. Not a conjecture that ended up being false, but a theorem which had a proof that was accepted for a nontrivial amount of time before someone found a hole in the argument. Does this happen anymore now that we have computers? I imagine not. But it seems totally possible that this could have happened back in the Enlightenment.
Feel free to interpret this how you wish!
In 1933, Kurt Gödel showed that the class called $\lbrack\exists^*\forall^2\exists^*, {\mathrm{all}}, (0)\rbrack$ was decidable. These are the formulas that begin with $\exists a\exists b\ldots \exists m\forall n\forall p\exists q\ldots\exists z$, with exactly two $\forall$ quantifiers, with no intervening $\exists$s. These formulas may contain arbitrary relations amongst the variables, but no functions or constants, and no equality symbol. Gödel showed that there is a method which takes any formula in this form and decides whether it is satisfiable. (If there are three $\forall$s in a row, or an $\exists$ between the $\forall$s, there is no such method.)
In the final sentence of the same paper, Gödel added:
In conclusion, I would still like to remark that Theorem I can also be proved, by the same method, for formulas that contain the identity sign.
Mathematicians took Gödel's word for it, and proved results derived from this one, until the mid-1960s, when Stål Aanderaa realized that Gödel had been mistaken, and the argument Gödel used would not work. In 1983, Warren Goldfarb showed that not only was Gödel's argument invalid, but his claimed result was actually false, and the larger class was not decidable.
Gödel's original 1933 paper is Zum Entscheidungsproblem des logischen Funktionenkalküls (On the decision problem for the functional calculus of logic) which can be found on pages 306–327 of volume I of his Collected Works. (Oxford University Press, 1986.) There is an introductory note by Goldfarb on pages 226–231, of which pages 229–231 address Gödel's error specifically.
When trying to enumerate mathematical objects, it's notoriously easy to inadvertently assume that some condition must be true and conclude that all the examples have been found, without recognizing the implicit assumption. A classic example of this is in tilings of the plane by pentagons: for the longest time everyone 'knew' that there were five kinds of pentagons that could tile the planes. Then Richard Kershner found three more, and everyone knew that there were eight; Martin Gardner wrote about the 'complete list' in a 1975 Scientific American column, only to be corrected by a reader who had found a ninth - and then after reporting on that discovery, by Marjorie Rice, a housewife who devoted her free time to finding tessellations and found several more in the process. These days, she has a web page devoted to the subject, including a short history, at https://sites.google.com/site/intriguingtessellations/home
EDIT: True to my 'I doubt anyone would be shocked' comment below, apparently another tiling has recently been found by some folks at the University of Washington in Bothell. There's a pretty good article about it at The Guardian.
EDIT 2: The problem has now seemingly been established; there are exactly $15$ pentagonal tesselations. Quanta’s article covers the subject pretty well.
Several examples come to my mind:
Hilbert's "proof" of the continuum hypothesis, in which an error was discovered by Olga Taussky when she was editing his collected works. This was shown to be undecidable by Paul Cohen later.
Cauchy's proof (published as lecture notes in his collected papers) of the fact that the pointwise limit of continuous functions is continuous. At the time, there was a poor understanding of the concept of continuity, until Weierstrass came along.
Lamé's proof of Fermat's last theorem, erroneous in that it was supposing unique factorization in rings of algebraic integers, which spurred the invention of ideals by Kummer.
One of the classic examples surely is the Perko pair of knots. For 75 years people thought that these two knots were distinct, even though they had found no invariants to distinguish between them. Then in 1974 Kenneth Perko (a lawyer!) discovered that they were actually the same knot. Even Conway, apparently, in compiling his table, had missed this.
It is not by any means a significant error, but it is an intriguing one nonetheless.
In 2003 a startling breakthrough was made (Review text only available to MathSciNet subscribers) in the theory of combinatorial differential manifolds. This theory was started by Gel'fand and MacPherson as a new combinatorial approach to topology, and one of the objects of its study is the matroid bundle. Much effort was spent in clarifying the relationship between real vector bundles and matroid bundles. From various previous results, the relationship is expected to be "complicated".
The Annals of Mathematics published in 2003 an article by Daniel Biss whose main theorem essentially showed that the opposite is true: that morally speaking there is no difference between studying real vector bundles and matroid bundles. This came as quite a shock to the field. (For an expert's account of the importance of this result, one should read the above-linked MathSciNet review.)
Unfortunately the article was retracted in 2009 after a flaw was found by (among others) Mnev. The story was popularised by Szpiro in his book of essays.
From Wikipedia one also finds the following account of the incident by someone familiar with the details and has expertise in the field, which contradicts some of the assertions/descriptions in Szpiro's essay. According to the various accounts, "experts" may have known about the error in the proof as early as 2005. But in the "recorded history" the first public announcement was not until 2007, and the erratum only published in 2009. So depending on your point of view, this may or may not count as a theorem accepted for some "nontrivial" amount of time.
A fairly recent example that I know of is a paper by the name of "A counterexample to a 1961 'theorem' in homological algebra" by Amnon Neeman (2002). It was a fairly big deal for some people when they realized the 'theorem' was false. I don't know enough about the specifics to discuss it in depth, since it's not terribly close to what I work on, so here is the abstract of Neeman's paper in lieu of any discussion:
In 1961, Jan-Erik Roos published a “theorem”, which says that in an $[AB4 * ]$ abelian category, $\lim^1$ vanishes on Mittag–Leffler sequences. See Propositions 1 and 5 in [4]. This is a “theorem” that many people since have known and used. In this article, we outline a counterexample. We construct some strange abelian categories, which are perhaps of some independent interest.These abelian categories come up naturally in the study of triangulated categories. A much fuller discussion may be found in [3]. Here we provide a brief, self contained, non–technical account. The idea is to make the counterexample easy to read for all the people who have used the result in their work.In the appendix, Deligne gives another way to look at the counterexample.
The "telescope conjecture" of chromatic homotopy theory is an interesting example.
In 1984, Ravenel published a seminar paper called "Localization with respect to certain periodic homology theories" where he made a series of 7 or 8 important conjectures about the global structure of the ($p$-local) stable homotopy category of finite spaces. Four years later, Devinatz-Hopkins-Smith published "Nilpotence I" (while Hopkins was still a grad student!!), which along with the follow-up paper "Nilpotence II" proved all but one of Ravenel's conjectures, the telescope conjecture. Then in 1990, Ravenel published a disproof of this conjecture, and went so far as to write a paper entitled "Life after the telescope conjecture" in 1992 that detailed a new way forward. But then it turned out that his disproof had a flaw in it too! The telescope conjecture remains open to this day, although I think most experts believe that it is false.
A famous example of this involves Vandiver's 1934 "proof" of one of the two steps in a line of attack on (an important case of) Fermat's Last Theorem. In algebraic number theory, there arise important positive integers called class numbers. In particular, for each prime p, a certain class number $h_p^+$ can be defined that is intimately connected with Fermat's Last Theorem.
Kummer proposed that (an important case of) Fermat's Last Theorem could be proved by
i) Proving that $h_p^+$ is not divisible by p
ii) Proving that $h_p^+$ not being divisible by p implies the "first case" of Fermat's Last Theorem.
In 1934, Vandiver published a proof of ii). In the introduction to "Cyclotomic Fields I and II", Serge Lang stated:
"...many years ago, Feit was unable to understand a step in Vandiver's 'proof' that $p$ not dividing $h_p^+$ implies the first case of Fermat's Last Theorem, and stimulated by this, Iwasawa found a precise gap which is such that there is no proof."
(In fact, Vandiver passed away believing that his proof was correct.)
I would like to know more about this history of this myself, and would gladly edit this post with more reliable information. For instance,
http://gow.epsrc.ac.uk/NGBOViewGrant.aspx?GrantRef=EP/C549074/1
says that Feit's observation occurred "around" 1980, which suggests that it was never published.
Some technical results in the disintegration theory of von Neumann algebras (roughly speaking, results expressing an algebraic object as a "direct integral" of "simpler" algebraic objects) stated by Minoru Tomita in the 1950s turned out to not be OK. There was an entire chapter following Tomita's approach in Naimark's book Normed Rings that vanished from later editions when the errors came to light.
I am not clear on the details of exactly how Tomita's stuff was wrong. (This happened before I was born, and I am not that interested in the history of mathematics, so I only know what I have heard about this from people who were there when it happened.) I have heard one person say that Tomita made use of certain technical results that only held under certain "nice" hypotheses that were not met at the level of generality at which he was working. Another person said that Tomita's arguments simply weren't clear enough to admit close analysis of how he went wrong, but that flaws were evident once people produced counterexamples to statements of the results. I don't personally know which of these stories is closer to the truth.
I am not sure to what extent this work was "accepted for a nontrivial amount of time." The person who told me most of what I know about this conveyed to me that at the time, there was a sense in the air that there was something "fishy" about some of the theorems, and that counterexamples were circulated among people working in the area long before it all worked itself out in print.
This seems to be related enough to deserve to be in an answer:
The April 2013 issue of the Notices of the AMS features a long article Errors and Corrections in Mathematics Literature written by Joseph F. Grcar.
Not a specific mistake, rather an analysis of how mathematics journals and mathematicians deal with mistakes in general, compared to other sciences.
A plentiful source of examples of "theorems" that were "proved" is supplied by the Italian school of algebraic geometry.
The Italians, most prominently Guido Castelnuovo, Federigo Enriques and Francesco Severi, derived some remarkable results on classification algebraic surfaces, relying strongly on geometical insight. The problem was, their reliance on intuition ultimately led them astray, to the point where some of things that were intuitively obvious to Severi were plain wrong. For an extreme example, Severi claimed to show a degree 6 surface in a 3 dimensional projective space has at most 52 nodes, while Mumford exhibited such surface that in fact had 65 nodes. Wikipedia provides a short but informative discussion. There is also a great thread on Mathoverflow.
I don't know how long some of his proofs stood, but Legendre is infamous for his repeated attempts at proving the parallel postulate.
Legendre believed that 6 is not a sum of two rational cubes, then $\left({17\over21}\right)^3+\left({37\over21}\right)^3$ came along.
Also, an amicable pair $(1184, 1210)$ got overlooked by early researchers and came into the light when much larger pairs were known for centuries. Not quite a theorem, but anyway. Quoting from this post:
the smallest pair after the one known from antiquity (1184,1210) was found only in 1866 by a 16-year old student Niccolo Paganini.
(This is not Paganini the famous violinist.)
In 1905, Henri Lebesgue claimed to have proved that if $B$ is a Borel set in $\Bbb R^2$, then its projection onto a line is a Borel subset of $\Bbb R$. This is false, and the mistake led Souslin and Luzin to define analytic sets, twelve years later. An analytic set is exactly a set that is the projection of a Borel set. Though not necessarily Borel, these sets are still measurable.
This Math Overflow post briefly explains the error:
It came down to his claim that if $\{A_n\}$ is a decreasing sequence of subsets in the plane with intersection $A$, the the projected sets in the line intersect to the projection of $A$. Of course this is nonsense. Lebesgue knew projection didn't commute with countable intersections, but apparently thought that by requiring the sets to be decreasing this would work.
Well, there have been plenty of conjectures which everybody thought were correct, which in fact were not. The one that springs to mind is the Over-estimated Primes Conjecture. I can't seem to find a URL, but essentially there was a formula for estimating the number of primes less than $N$. Thing is, the formula always slightly over-estimates how many primes there really are... or so everybody thought. It turns out that if you make $N$ absurdly large, then the formula starts to under-estimate! Nobody expected that one. (The "absurdly large number" was something like $10^{10^{10^{10}}}$ or something silly like that.)
Fermat claimed to have had a proof for his infamous "last theorem". But given that the eventual proof is a triumph of modern mathematics running to over 200 pages and understood by only a handful of mathematicians world wide, this cannot be the proof that Fermat had 300 years ago. Therefore, either 300 years of mathematicians have overlooked something really obvious, or Fermat was mistaken. (Since he never write down his proof, we can't claim that "other people believed it before it was proven false" though.)
Speaking of which, I'm told that Gauss or Cauchy [I forget which] published a proof for a special case of Fermat's last theorem - and then discovered that, no, he was wrong. (I don't recall how long it took or how many people believed it.)
Not sure if the following fits the criterion for constraint, but Hans Rademacher incident comes to mind (page 82, The Riemann Hypothesis: For the aficionado and virtuoso alike):
8.2 Hans Rademacher and False Hopes
In 1945, Time Magazine reported that Hans Rademacher had submitted a flawed proof of the Riemann Hypothesis to the journal Transactions of the American Mathematical Society. The text of the article follows: A sure way for any mathematician to achieve immortal fame would be to prove or disprove the Riemann hypothesis. This baffling theory, which deals with prime numbers, is usually stated in Riemann’s symbolism as follows: “All the nontrivial zeros of the zeta function of s, a complex variable, lie on the line where sigma is 1/2 (sigma being the real part of s).” The theory was propounded in 1859 by Georg Friedrich Bernhard Riemann (who revolutionized geometry and laid the foundations for Einstein’s theory of relativity). No layman has ever been able to understand it and no mathematician has ever proved it.
One day last month electrifying news arrived at the University of Chicago office of Dr. Adrian A. Albert, editor of the Transactions of the American Mathematical Society. A wire from the society’s secretary, University of Pennsylvania Professor John R. Kline, asked Editor Albert to stop the presses: a paper disproving the Riemann hypothesis was on the way. Its author: Professor Hans Adolf Rademacher, a refugee German mathematician now at Penn.
On the heels of the telegram came a letter from Professor Rademacher himself, reporting that his calculations had been checked and confirmed by famed Mathematician Carl Siegel of Princeton’s Institute for Advanced Study. Editor Albert got ready to publish the historic paper in the May issue. U.S. mathematicians, hearing the wildfire rumor, held their breath. Alas for drama, last week the issue went to press without the Rademacher article. At the last moment the professor wired meekly that it was all a mistake; on rechecking. Mathematician Siegel had discovered a flaw (undisclosed) in the Rademacher reasoning. U.S. mathematicians felt much like the morning after a phony armistice celebration. Sighed Editor Albert: “The whole thing certainly raised a lot of false hopes.” [142]
Edit: This link has further (dis)proofs of RH including de Branges saga.
For longtime it was believed that it was not possible to know a digit of the decimal expansion of $\pi$ without knowing its preceding digits. It was recently, 1995, that Plouffe discovered his formula $$\pi=\sum_{k=o}^{\infty}\frac {1}{16^k}(\frac{4}{8k+1}-\frac{2}{8k+4}-\frac{1}{8k+5}-\frac{1}{8k+6})$$ from which,unlike others before it, one can get any individual hexadecimal digit of π without calculating all the preceding digits.This was definitely a breakthrough that brought down a big mistake of centuries.
I'm pretty shaky on the history of anything, but I think it's true that twice Georg Cantor thought he had a proof that the continuum hypothesis is true and once thought he had a proof that it was false. Indeed, he announced a proof of it in 1884. Considerably later, in the 1920s?, David Hilbert thought he had a proof. And David Hilbert began his famous turn of the century speech with exhortation that any well-posed mathematical problem had a solution: "Take any definite unsolved problem... However unapproachable the problem may seem to us... we have the firm conviction that the solution must follow by a finite number of purely logical processes." Notice that this last sentence contains the crux of Godel's proof that Hilbert was wrong: there just aren't enough finite number of purely logical processes to cover all the definite unsolved problems. As if to pour salt on the wound, Paul Cohen came along and showed that the continuum hypothesis was one such problem, which you cannot prove is either true or false (unless of course mathematics is inconsistent, in which case you might be able to prove it is both true and false.)
Poincaré's discovery of homoclinic points grew out of a extremely serious mistake he made in his original submission for a prize essay contest sponsored by Acta Mathematica in 1888. His original 200 page manuscript, on the restricted three-body problem, was evaluated by Weierstrass, Mittag-Leffler, and Phragmén, who had great difficulty following his arguments. Poincaré responded with a dozen further explanations, totaling 100 pages. After many further exchanges, the editors finally decided to accept the manuscript (this was, after all, Poincaré, and he must know what he's doing) and awarded him the prize.
But around the time of publication, Phragmén was still puzzled by some points and Mittag-Leffler wrote to Poincaré. They received back a telegram from Poincaré asking that publication be stopped immediately! Poincaré realized that his belief that the stable and unstable manifolds could not intersect transversally was wrong, and that such intersection points, which he later called homoclinic points, immediately forced very complicated dynamically behavior, invalidating much of his work. He wrote to Mittag-Leffler:
"I have written this morning to Mr. Phragmén to tell him about an error which I have committed and he has undoubtedly informed you of my letter. But the consequences of this error are more serious than I first thought. It is not true that the asymptotic surfaces are closed, at least not in the sense that I meant before. What is true, is that if one considers the two parts of that surface (which I yesterday still believed coincided with each other) they intersect along infinitely many asymptotic trajectories and furthermore their distance is an infinitesimal of higher order than μp however big p is.
I don't conceal from you the trouble this discovery gives me."
Mittag-Leffler immediately halted the presses and recalled all copies of this issue he could get, destroying them all (except for a few, one of which remains in the library of the Mittag-Leffler Institute). They asked Poincare to pay for the suppression of this issue, which he did.
Poincare then wrote a new essay, incorporating many of the added notes from the original, and this was the version that Acta Mathematica published (with no mention of the earlier one). Eventually Poincaré used this as the basis of his three volume classic Les méthodes nouvelles de la mécanique céleste.
A riveting account of this story is contained in Poincaré's discovery of homoclinic points by K. G. Anderson, Archive History of Exact Sciences, 48(2) (1994), 133–147.
In Mathematical Recreations and Essays, 12th edition, by Rouse Ball and Coxeter, it states that a proof of the Four-Color Theorem was published in (about) 1880 and (about) 10 years later, a fatal flaw was found. It was assumed that a planar connected trivalent graph (each vertex lies on exactly 3 edges) cannot have an isthmus. An isthmus is an edge, in a connected graph, which, if removed, disconnects the graph. The book gives a very simple counter-example.
I am the proud coauthor of a 2018 paper in which we point out a wrong proof (and a wrong statement) in someone else's 2003 paper, which in turn was supposed to answer a question from yet someone else's 1993 paper. The funny thing is that one of the authors of the 1993 paper had written a Zentralblatt review for the 2003 paper without realizing that the example claimed in the latter was incorrect, and that the original 1993 question remained open. (The question itself is not something of a great importance, so I will only provide a link to the latest of these three papers below, in case someone might want to take a look: Section 4 deals with that error. I also tend to believe that the referee of the 2003 paper had not done a good job. The error in the proof boiled down to the claim that if a family $\mathcal A$ refines a family $\mathcal B$ (i.e. for every $A\in\mathcal A$ there is $B\in\mathcal B$ with $A\subseteq B$) then $\cap\mathcal A\subseteq\cap\mathcal B.$ My coauthor and I realized something was wrong since one of our papers together with those 2003 and 1993 papers produced a contradiction, so we had to dig to see exactly where it came from.)
ON MONOTONE ORTHOCOMPACTNESS
Strashimir G. Popvassilev, John E. Porter
Serdica Math. J. 44 (2018), 177–186
paper link pdf file
In the area of the second part of the Hilbert 16th problem, Dulac's proof and Petrovski-Landis proof are examples of this situation.
The classic Proofs and Refutations by Imre Lakatos discusses an example. Euler’s polyhedral formula holds that the number of vertices of any polyhedron minus the number of edges plus the number of faces is equal to 2. He gave the first (incorrect) proof in 1750, and there have been more than twenty proofs of it since then. However, many of those proofs have turned out to have counterexamples, such as the one based on unfolding the polyhedron onto the plane, which turned out to break on a polyhedron shaped like a ziggurat. Mathematicians kept revising the conditions of the formula for two centuries after its discovery.
Today, it remains a very important theorem, but we’ve drastically changed how we think of polyhedra (Euler thought of vertices as angles or cones extending from a corner and implicitly assumed all polyhedra were convex) and we define the kinds of polyhedra for which the formula applies as “simple.”
The French mathematician, physicist and philosopher Jean le Rond d'Alembert had several wrong ideas about probabilities, that he published, argued and defended despite objections and controversy, some of which made it into the monumental Encyclopédie he was co-authoring with Denis Diderot at the time.
One such (in)famous example is in the article Croix ou Pile in Encyclopédie vol. IV (1754) p. 512, where d'Alembert argues that the probability of at least one head in two fair coin tosses is $\dfrac{2}{3}$ instead of the correct $\dfrac{3}{4}$.
However is this quite correct? For in order to take here only the case of two tosses, is it not necessary to reduce to one the two combinations which give heads on the first toss? For as soon as heads comes one time, the game is finished, & the second toss counts for nothing. So there are properly only three possible combinations:
- Heads, first toss.
- Tails, heads, first & second toss.
- Tails, tails, first & second toss.
Therefore the odds are 2 against 1.
This problem is discussed in full detail at d'Alembert's Misstep.
More comprehensive coverage of d'Alembert's undertakings in probability, with comments, context and references, can be found in the article L'objet du doute. Les articles de D'Alembert sur l'analyse des hasards dans les quatre premiers tomes de l'Encyclopédie.
I'm not sure if this classifies as a mistake, but rather it is an interesting example of someone who is not in the establishment of mathematics who proved something that was not accepted until someone in the community came along and proved it. Namely, it is the Stark-Heegner theorem which states that there are precisely $9$ imaginary quadratic number fields of class number $1$. Heegner was not a classical mathematician, and came up with a proof of the theorem in $1952$; however, the proof was not accepted by the community until Harold Stark came up with a proof in $1967$ which is kind of a shame as Heegner died in $1965$, so it is as though he was not acknowledge for one of his greatest accomplishments until after his death.
Wedderburn's Theorem http://en.wikipedia.org/wiki/Wedderburn%27s_little_theorem The editor says that the body must be at least 30 characters, so, hmmm, three cheers for Esperanto!
Yes, in the past, people used to do their math in Naive set theory. According to the Citizendium article Set theory, Gregg Cantor created a formal system for Naive set theory and then later, it was a surprize to discover that its formal system is inconsistent. After it was discovered to be inconsistent, mathematicians started working in the formal system of Zermelo-Fraenkel set theory and then again, shockingly, it was proven that there is no formal proof of the axiom of choice in Zermelo-Fraenkel set theory.
Here are two examples, one new and one old, the old one being an addendum to Francesco Sica and Airymouse's answers. I hope this will be both a contribution to the original question and to the twist Ali Taghavi introduced.
The Old Example: This is regarding Hilbert's alleged proof of the continuum hypothesis. I report from G.-C. Rota's Indiscrete Thoughts, p. 201:
Once more let me begin with Hilbert. When the Germans were planning to publish Hilbert's collected papers and to present him with a set on the occasion of one of his later birthdays, they realized that they could not publish the papers in their original versions because they were full of errors, some of them quite serious. Thereupon they hired a young unemployed mathematician, Olga Taussky-Todd, to go over Hilbert's papers and correct all mistakes. Olga labored for three years; it turned out that all mistakes could be corrected without any major changes in the statement of the theorems. There was one exception, a paper Hilbert wrote in his old age, which could not be fixed; it was a purported proof of the continuum hypothesis, you will find it in a volume of the Mathematische Annalen of the early thirties. At last, on Hilbert's birthday; a freshly printed set of Hilbert's collected papers was presented to the Geheimrat. Hilbert leafed through them carefully and did not notice anything.
The New Example: This one comes from hyperbolic dynamics. A now-classical result by Franks says that any Anosov diffeomorphism on a torus is topologically conjugate to a hyperbolic Lie group automorphism of the torus (a diffemorphism $f$ is hyperbolic if the induced map on vector fields $X\mapsto Tf\circ X\circ f^{-1}$ is hyperbolic in the sense of https://math.stackexchange.com/a/4346189/169085 ; in either case the linear map ought to have spectrum disjoint from the unit circle). Generalizing this result from the algebraic point of view, one replaces $\mathbb{R}^d$ with an anonymous (connected, simply connected) nilpotent Lie group $N$, and $\mathbb{Z}^d$ with a torsion-free cocompact lattice (of $N$ or of $N\rtimes F$, where $F\leq \operatorname{Aut}_{\text{Lie}}(N)$ is finite). If no translations are involved (i.e. $F=1$) the end result is a nilmanifold (e.g. Heisenberg with real entries modulo Heisenberg with integer entries); otherwise the end result is an infranilmanifold (e.g. Klein bottle).
There was some confusion regarding what an "infranilmanifold endomorphism/automorphism" ought to mean starting from the works of Franks and company, which confusion stemmed from a false proof of Auslander. This confusion propagated to other works, including Gromov's work on expanding maps (and polynomial growth) and Manning's generalization of the Franks theorem I stated above to infranilmanifolds. Gromov's result is among the results that are patched now; Manning's result is still open (for infranilmanifolds), to the best of my knowledge. K. Dekimpe has multiple articles about it; a good start is "What is... an Infra-nilmanifold Endomorphism?" (see https://www.ams.org/notices/201105/rtx110500688p.pdf ), and a more detailed account is in the paper "What An Infra-Nilmanifold Endomorphism Really Should Be...", again by Dekimpe.
