A nice review of properties of Deviance for one parameter exponential families.

(Hastie 1987). This short review provides a compendium of useful results
on the *deviance* defined by \(\text -2 \log \mathcal L +2\log\mathcal L^*\),
where \(\mathcal L^*\) denotes the likelihood of a “saturated” model, as explained
in the paper. From the paper’s abstract:

Prediction error and Kullback-Leibler distance provide a useful link between least squares and maximum likelihood estimation. This article is a summary of some existing results, with special reference to the deviance function popular in the GLIM literature.

Of particular interest:

- Clarifies the definition of a “saturated” model for i.i.d. samples.
- Highlights the parallels between \(L_2\) and Kullback-Leibler loss. In particular, the expectation is shown to be the optimal regression function for the general KL loss.
- Discusses optimism in the training error estimate of the in-sample (fixed predictors) error rate in terms of KL loss, within the context of Generalized Linear models.

Hastie, Trevor. 1987. “A Closer Look at the Deviance.” *The American Statistician* 41 (1): 16–20. https://doi.org/10.1080/00031305.1987.10475434.

If you see mistakes or want to suggest changes, please create an issue on the source repository.

Text and figures are licensed under Creative Commons Attribution CC BY-SA 4.0. Source code is available at https://github.com/vgherard/vgherard.github.io/, unless otherwise noted. The figures that have been reused from other sources don't fall under this license and can be recognized by a note in their caption: "Figure from ...".

For attribution, please cite this work as

Gherardi (2024, March 7). vgherard: "A Closer Look at the Deviance" by T. Hastie. Retrieved from https://vgherard.github.io/posts/2024-03-07-a-closer-look-at-the-deviance-by-t-hastie/

BibTeX citation

@misc{gherardi2024"a, author = {Gherardi, Valerio}, title = {vgherard: "A Closer Look at the Deviance" by T. Hastie}, url = {https://vgherard.github.io/posts/2024-03-07-a-closer-look-at-the-deviance-by-t-hastie/}, year = {2024} }