I can't give you advice what to do against it, but I may help you understand why it is happening.
The point is that the "feeling of knowing", or "being certain", is an emotion, just like feeling sad or happy. It can also be compared to visual perception: instead of perceiving something about the state of the outside world (e.g. a blue mug on your desk), you perceive something about the state of your own cognitive processes: you came up with a piece of knowledge and it feels right.
And just like vision, it is susceptible to illusions which can completely fool your brain. Being convinced that (x+1)^2 = x^2 + 1^2 is right is very similar to being convinced that square A and square B are different shades.
The reason these illusions happen come from the way neuronal based intelligence works. Our brains are specialized at recognized similarity in patterns. If we are exposed to one pattern very frequently, it feels more "right" than other patterns. There are also other details, especially for visual illusions, which are dependent on the particular ways neurons in V1 and other perceptional areas work, but here the analogy between visual-illusion and feeling-of-knowing illusion breaks down. But the point is that feeling certain is not related to factual truth directly; it is related to noticing that the new pattern looks similar to older patterns we have come to believe are true trough repeated observation (or being repeatedly assured that they are true). The reason this works is that if we observe a pattern being true frequently enough, or if most people around us have come to recognize it as being true, it is indeed because it is true. Still, it is a matter of persuasion, not logic. Logic can make us understand something, but not make us believe in it intuitively.
So a person who lives in a world where most visible processes are described by simple linear and proportional relationships will intuitively feel that "linear" or "proportional" explanations for everything are right. This happens on a broad level, where exponential growth is completely counterintuitive and people freshly exposed to it are always surprised by the true magnitude of the calculated results even if they have cognitively understood the underlying principle. I think of myself that I should know better by now, but I still get surprised frequently.
It also happens in some specific ways, like the one you describe with math students. Your pupils have been exposed to linear relationships for years. Their neural networks have learned to react with a "this looks good" signal the way pavlov's dog's neural networks have learned to react with "food comes" signal. When they consider possible solutions, once the linear one comes up, it just feels right. Learning to ignore this inner certainty is possible, but it is a hard and slow process which physically requires rewiring the neurons in their brains. You cannot expect a silver bullet for it. Especially trying to find a way to make it better understandable won't work; they have already understood it in their higher, reasoning processes. It is their affect-level response which has to be overruled, and it responds to repeated training, not to logic.
For a better insight in how the feeling of knowing works, read "On being certain" by R. Burton. It is a great book, and I would recommend it for all pedagogues (and actually for everybody else too, but if you are interested in creating a feeling of knowing in your students, it might be especially helpful).
Edit A way of thinking about how to solve the problem is using mental models. A mental model is an understanding of how a mechanism works. "A wolf eats the sun each day and it gets reborn the next day" is a mental model of how days and nights work. "The earth is a sphere revolving around its axis with the sun to one side" is another mental model for the same mechanism. *
Humans are capable of solving problems when they don't have a clear mental model of the forces working in the background, but they usually do it haltingly, step by step, and cannot monitor the outcome of their steps for veracity of the solution. It is like trying to cross a labyrinth using some algorithm like taking only right turns and retracing to the left when you run into a blind end. It is possible to do it, but at no point do you actually know the way through the labyrinth, even after you have emerged on the other side. On the other hand, if you have memorized a map of the labyrinth, and the labyrinth is of low enough complexity to fit in your spatial reasoning brain areas, you have a good mental model of the labyrinth and you can easily find a way to the other side, and at each step you can monitor your concrete surroundings and relate them to the mental model of the whole, and it will always feel right when you are on the right way and wrong when you are on the wrong way, because your spatial reasoning "subsystems" will create a feeling of certainty for you. Another example which is probably much more "intuitively right" :) for math teachers would be simple geometry problems about triangles. Read the word description, and you probably could solve it step by step, but it would be hard, and you can't keep all the details in your mind at once. Make a drawing, and everything falls into place; you know the solution before you have calculated it.
What you certainly want is that your pupils get a mental model of nonlinear relationships which can be reasoned about on an intuitive level. Getting exposed to nonlinear relationships written as abstract numbers is not good enough, even if the exposure is very frequent. We humans don't have inborn neural circuits for evaluating rational numbers, this is a learned skill. We have inborn neural circuits for evaluating tangible entities, visual input, smells, language, etc. If you want your pupils to create a mental model at all, instead of running around the numbers blindly, you will have to help them relate the numbers to something. I don't know what this something will be, centuries of teaching math have tried to find such solutions and to my knowledge have not gotten beyond cutting one apple in thirds and one in halfs and then showing that one piece of each together don't make a fifth of an apple. But any working solution, if it exists, will have to work along the lines of creating a good, solid mental model. Then pupils will be able to think properly about the problem at hand, to reason about it on a level which creates the feeling of knowing at the right times except of floating in uncertainty at each step.
I don't have a single good book recommendation on mental models the way I had on the feeling of certainty. They are researched within the context of usability, so a textbook on software usability might contain relevant chapters and/or lead you to better, more specialized literature on mental models.
- The days and nights provide another nice example of how conviction works against logic and how mental models fit into it all. Note that we as individuals are only convinced that "earth revolves around its axis and around the sun" is the true one because we have been told that it is true. I learned in sixth grade about Foucault's pendulum, and Earth's horizon curvature, and all the other experiments together which prove it; but I have never seen the pendulum or conducted these experiments. It doesn't matter, because when I was four, my father had bought me a globe and told me how it works, and I believed it, long before I knew what a physics experiment is. Had I been constantly told that the Earth is flat up until I started taking sixth grade physics, my teacher describing those experiments wouldn't have convinced me. It was seeing the rotating globe, and hearing the explanation from a person whom I trusted, which helped me create a mental model leading to true convictions, as opposed to mere logical inferences.