A neural network is a function with many adjustable numbers, and training means nudging these numbers downhill on a loss. This page shows what each part does.
A neuron multiplies its input by a weight, adds a bias, and passes the result through a non-linear activation function. The weight sets how steep the response is, and the bias sets where it switches.
Without the non-linearity, stacking neurons would only ever give a straight line. With it, a sum of many neurons can bend into any smooth shape.
Try: make w negative to flip the curve, and change b to move the switch point.
This network has one input, a row of tanh neurons, and one output that adds them up. The faint curves are the contributions of the single hidden neurons, and the thick curve is their sum. Training changes the weights so that the sum goes through the data.
The loss is the mean squared error. Its gradient with respect to every weight comes from the chain rule, applied from the output backward, and this is all that backpropagation is. Each step moves every weight a little against its gradient.
In the diagram, the line width shows the size of a weight, and the color shows its sign.
Try: one hidden unit can only make one bend, and with too high a learning rate, the loss jumps around. Add points by clicking, and watch the fit follow.
Training is optimization. Plain gradient descent follows the slope. Momentum keeps a running velocity, so it rolls through small bumps. Adam scales each step by the recent size of the gradient, and it is the default choice for most networks.
Try: start the optimizers in different places on this energy surface. Which one finds the deepest minimum depends on where it starts.
The same kind of network, with a sigmoid at the end, outputs the probability of class 1. The colored background is the answer of the network at every point, and more hidden units let the boundary bend around the data.
Try: add points of the wrong class inside a region, and see how hard the network works to include them. This is overfitting in action.
Neural network potentials use exactly these pieces, with atomic descriptors as the input and the energy as the output. The page on machine learning in MD covers this part.