Why backpropagation dominates neural networks instead of interpolation

Hi guys,

I was learning machine learning and I found something a bit confusing.
When I studied physics I saw the method of least squares to find the best parameters for the given data, in this case we assume we know the equation and we just minimize the error. So if it is a straight line model we compute the best slope and constant.
With machine learning we don't know the underlying equation or model so we use a general method to "fit" as best as we can the real data.
But, isn't that what we do with interpolation/extrapolation?
What is it so special about a neural network and the backpropagation method that we can't achieve with interpolation?

jonjacson said:

TL;DR Summary: I don't understand what are the advantages of backpropagation in a neural network versus using classical interpolation/extrapolation methods.

With machine learning we don't know the underlying equation or model

I don't think this is correct. Each artificial neuron implements a definite function with parameters (weights and bias), which need to be determined to minimize a definite target function, which is built by composing the functions of the neurons.

Hill said:

I don't think this is correct. Each artificial neuron implements a definite function with parameters (weights and bias), which need to be determined to minimize a definite target function, which is built by composing the functions of the neurons.

The weights are used to compute how much of an output from one neuron enters the next ones, but that is not a function.

jonjacson said:

The weights are used to compute how much of an output from one neuron enters the next ones, but that is not a function.

Here is one way to define the function (sorry, I'd need to dig to find the source of this paper):


.......


.......

jonjacson said:

With machine learning we don't know the underlying equation or model so we use a general method to "fit" as best as we can the real data.
But, isn't that what we do with interpolation/extrapolation?
What is it so special about a neural network and the backpropagation method that we can't achieve with interpolation?

I am not an expert, but here is my understanding:
The backpropagation of neural networks is to develop weights for intermediate layers that can help to model a sophisticated decision process. Those intermediate layers are not given data like interpolation is. They are free to interpretation. Sometimes they are obscure and other times we can imagine a meaning for them. The only hard training data are the inputs and outputs at the first and last layer.