Learn

Statistics for simulations

Simulation results are estimates from finite and correlated data. A few statistical ideas decide whether a number can be trusted, and how its uncertainty should be reported.

    Principal component analysis

    PCA finds the directions in which the data varies the most. You compute the covariance matrix of the centered data. Its eigenvectors are the principal components, and its eigenvalues are the variances along each of them.

    C = (1/(n−1)) XTX, C vk = λk vk

    On MD trajectories, this is called essential dynamics, and the first few components often capture the large and slow motions of a protein or a polymer.

    Try: lower the correlation toward 0, and PC1 explains only half of the variance. Add a point far away to see how sensitive PCA is to outliers.

    The central limit theorem

    If you average n independent samples, the average is close to normally distributed, whatever the original distribution is, with a spread that shrinks as 1/√n. This is why error bars are usually quoted as standard errors.

    standard error = σ / √n

    Try: start from the skewed or two-peaked source with n = 1, and then increase n.

    Least squares

    Ordinary least squares picks the line that makes the sum of squared vertical distances as small as possible. The squares in the picture are exactly the terms being added up.

    Squaring makes the fit sensitive to outliers. Most of the fitting in simulation work, from force field parameters to diffusion coefficients, is some version of least squares.

    Try: click far from the line, and watch a single point pull it.

    Bootstrap error bars

    When you have one data set and no formula for the error of your quantity, you can resample. Draw n values from your own data with replacement, recompute the quantity, and repeat this a few thousand times. The spread of the results is the uncertainty.

    It works for anything, from a mean to a fitted slope or a peak position in g(r).

    Try: a small n with skewed data gives an asymmetric interval, which the ±1.96 s/√n formula cannot show.

    Block averaging

    Consecutive MD frames are correlated, so treating them as independent makes error bars far too small. Block averaging fixes this. You cut the series into blocks, average each block, and compute the standard error of the block means. When the blocks get longer than the correlation time, they become independent, and the error stops growing.

    error from blocks of size B = sd(block means) / √(number of blocks)

    Try: a longer correlation time makes the naive error even more wrong.

    Overfitting

    A flexible enough model can pass through every data point and still be useless on new data. To catch this, keep some data aside and check the error there. This is the reason for train and test splits, cross-validation, and, for molecules, scaffold splits.

    Try: push the degree up, and watch the training error fall while the test error rises.