I have a surface which is $z=ax+bx^2+cxy+d$, where $a,b,c$ are coefficients and $d$ is the constant. so an arbitrary point would be $(x, y, ax + bx^2 + cxy+ d)$. there are a set of points that are not in this surface and I want to calculate the shortest distance to them, could anyone tell me what is the best way to do it? Or if you could solve this for a point, let say $(5,4,3)$. I would really appreciate.

You can write in maths mode by surrounding your equations with dollar signs. I have done it for you this time; please check that I did not distort the meaning of your question. – Mårten W Sep 24 '13 at 18:39
1 Answers
Note that your surface $S$ can be represented as the graph of a function, that is, $$ S=\{(x,y,F(x,y))\ \ x,y\in\mathbb R\}, $$ where $$ F:\mathbb R^2\to\mathbb R,\ (x,y)\mapsto ax+bx^2+cxy+d. $$ The distance of a given point $(x_0,y_0,z_0)\in\mathbb R^3$ to S is the length of the normal $N$ fixed in a point $(x,y,z=F(x,y))\in S$ and pointing to $(x_0,y_0,z_0)$.
Now, every normal $N$ of $S$ is given by $$ N=(F_x(x,y),F_y(x,y),1), $$ which is in your case $$ N(x,y)=(a+2bx+cy,xc,1). $$ Then our desired $n$ is of the form $n=\lambda N(x,y)$ for suitable $x,y,\lambda\in\mathbb R$. So, all you have to do is to solve the equation system $$ (x,y,F(x,y))+\lambda N(x,y)=(x_0,y_0,z_0) $$ for $x$, $y$ and $\lambda$ to get $n$. Your desired distance is then $n_2$.
 8,307
 1
 13
 32