I had been interested in neural networks for a bit and thought about using one in python for a light project that compares various minimization techniques in a time domain (which is fastest).
Then I realized I didn't even know if a NN is good for minimization. What do you think?
It sounds to me like this is a problem more suited to genetic algorithms than neural networks. Neural nets tend to need a bounded problem to solve, requiring training against known data, etc. - whereas genetic algorithms work by finding better and better approximate solutions to a problem without requiring training.
Back-propagation works by minimizing the error. However, you can really minimize whatever you want. So, you could use back-prop-like update rules to find the Artificial Neural Network inputs that minimize the output.
This is a big question, sorry for the short answer. I should also add that my suggested approach sounds pretty inefficient compared to more established methods and would only find a local minima.
The training process of a back-propagation neural network works by minimizing the error from the optimal result. But having a trained neural network finding the minimum of an unknown function would be pretty hard.
If you restrict the problem to a specific function class, it could work, and be pretty quick too. Neural networks are good at finding patterns, if there are any.
You can teach a NN to approximate a function. If a function is differentiable or your NN has more than one hidden layers, you can teach it to give derivative of a function.
Example:
You can train a 1 input 1 output NN to give output=sin(input)
You can train it also give output=cos(input) which is derivative of sin()
You get a minima/maxima of sin when you equate cos to zero.
Scan for zero output while giving many values from input. 0=cos() -> minima of sin
When you reach zero output, you know that the input value is the minima of the function.
Training takes less, sweeping for zero takes long.
Although this comes a bit too late for the the author of this question. Maybe somebody wants to test some optimization algorithms, when he reads this...
If you are working with regressions in machine learning (NN, SVM, Multiple Linear Regression, K Nearest Neighbor) and you want to minimize (maximize) your regression-function, actually this is possible but the efficiency of such algorithms depends on smootheness, (step-size... etc.) of the region you are searching in.
In order to construct such "Machine Learning Regressions" you could use scikit- learn. You have to train and validate your MLR Support Vector Regression. ("fit" method)
SVR.fit(Sm_Data_X,Sm_Data_y)
Then you have to define a function which returns a prediction of your regression for an array "x".
def fun(x):
return SVR.predict(x)
You can use scipiy.optimize.minimize for optimization. See the examples following the doc-links.
They're pretty bad for the purpose; one of the big problems of neural networks is that they get stuck in local minima. You might want to look into support vector machines instead.
Actually you could use the NN to find a function minimum, but it would work best combined with genetic algorithms mentioned by Erik.
Basically NN tent to find solutions which correspond to a function local minimum or maximum but in doing so are pretty precise (to comment on Tetha answer stating that NN are classifiers you can use if to say it the data input is minimum or not)
in contrast genetic algorithms tend to find more universal solution from the whole range of the inputs possible but then give you the proximate results.
The solution is to combine the 2 worlds
Neural networks are classifiers. They separate two classes of data elements. They learn this separation (usually) by preclassified data elements. Thus, I say: No, unless you do a major stretch beyond breakage.
