Scoring algorithm

Scoring algorithm, also known as Fisher's scoring, is a form of Newton's method used in statistics to solve maximum likelihood equations numerically, named after Ronald Fisher.

Sketch of derivation

Let $Y_{1},\ldots ,Y_{n}$ be random variables, independent and identically distributed with twice differentiable p.d.f. $f(y;\theta )$, and we wish to calculate the maximum likelihood estimator (M.L.E.) $\theta ^{*}$ of $\theta$. First, suppose we have a starting point for our algorithm $\theta _{0}$, and consider a Taylor expansion of the score function, $V(\theta )$, about $\theta _{0}$:

$V(\theta )\approx V(\theta _{0})-{\mathcal {J}}(\theta _{0})(\theta -\theta _{0}),\,$

where

${\mathcal {J}}(\theta _{0})=-\sum _{i=1}^{n}\left.\nabla \nabla ^{\top }\right|_{\theta =\theta _{0}}\log f(Y_{i};\theta )$

is the observed information matrix at $\theta _{0}$. Now, setting $\theta =\theta ^{*}$, using that $V(\theta ^{*})=0$ and rearranging gives us:

$\theta ^{*}\approx \theta _{0}+{\mathcal {J}}^{-1}(\theta _{0})V(\theta _{0}).\,$

We therefore use the algorithm

$\theta _{m+1}=\theta _{m}+{\mathcal {J}}^{-1}(\theta _{m})V(\theta _{m}),\,$

and under certain regularity conditions, it can be shown that $\theta _{m}\rightarrow \theta ^{*}$.

Fisher scoring

In practice, ${\mathcal {J}}(\theta )$ is usually replaced by ${\mathcal {I}}(\theta )=\mathrm {E} [{\mathcal {J}}(\theta )]$, the Fisher information, thus giving us the Fisher Scoring Algorithm:

$\theta _{m+1}=\theta _{m}+{\mathcal {I}}^{-1}(\theta _{m})V(\theta _{m})$..

Under some regularity conditions, if $\theta _{m}$ is a consistent estimator, then $\theta _{m+1}$ (the correction after a single step) is 'optimal' in the sense that its error distribution is asymptotically identical to that of the true max-likelihood estimate.

See also

• Score (statistics)
• Score test
• Fisher information

Further reading

• Jennrich, R. I. & Sampson, P. F. (1976). "Newton-Raphson and Related Algorithms for Maximum Likelihood Variance Component Estimation". Technometrics. 18 (1): 11–17. doi:10.1080/00401706.1976.10489395 (inactive 31 January 2024). JSTOR 1267911.{{cite journal}}: CS1 maint: DOI inactive as of January 2024 (link)