Residual Theory

Robert M. Cook


The ‘residuals.em.glm’ method returns two types of residuals, the deviance and pearson. The residuals are defined using standard GLM theory where the log-likelihood of an observation is expressed in the form:

\[ l(y_i) = \frac{w_i(y_i\theta_i - b(\theta_i))}{\phi} + c(y_i, \phi) \]

Where \(y_i\) is the observed value, \(w_i\) are the observed weights, \(\phi\) is the dispersion parameter (currently assumed to be 1 for the models implemented), and \(b\) and \(c\) are distribution-dependent functions. The function \(b\) is related to \(y_i\) such that \(\mu_i = E(y_i) = b'(\theta_i)\).

The deviance residuals, \(d_i\), are defined as:

\[d_i = sign(y_i - \mu_i) \sqrt{2 \left( l_{sat.}(y_i) - l_{model}(y_i) \right) }\]

where \(l_{sat.}\) is the ‘saturated model’ such that each value of \(\mu_i = y_i\) and \(l_{model}\) is the likelihood of \(y_i\) for the predicted value of \(\mu_i\). We will not go into the relationship between \(\mu_i\), the covariates \(X\), and the fit parameters \(B\) here.

The deviance residuals are implemented as:

rho <- x %*% B
mu <- family()$linkinv(rho)

# Poisson deviance
abs(y - w * mu) * sqrt(2 * (dpois(y, y, log = TRUE) - dpois(y, w * mu, log = TRUE)))

# Binomial deviance
abs(y - w * mu) * sqrt(2 * (dbinom(y, w, y / w, log = TRUE) - dbinom(y, w, mu, log = TRUE)))

where x is the input data, B is the optimal parameters, w is the weights, y is the target values and family is (currently) either poisson or binom.

The Pearson’s residuals, \(r_i\), when the weights all equal 1 are defined as:

\[ r_i = \frac{y_i - \mu_i}{\sqrt{Var(y_i)}} = \frac{y_i - \mu_i}{\sqrt{V(\mu_i)}}\]

where \(V(\mu)\) is the variance function of the GLM and depends on the underlying distribution.

Where the models are weighted (either if we look at rate of successes for a binomial process or have an exposure for a Poisson process) the Pearson residuals become:

\[ r_i = \frac{y_i / w_i -\mu_i}{\sqrt{Var(y_i)}} = \frac{y_i/w_i - \mu_i}{\sqrt{V(\mu_i)/w_i}} = \frac{y - w_i \mu_i}{\sqrt{w_i V(\mu_i)}}\]

The relationship between \(Var(y_i)\) and \(V(\mu_i)\) arises from taking the differentials of \(l\) wrt. \(\theta\). The first differential gives rise to the score vector \(u\):

\[ \frac{d l}{d \theta} = u = \frac{w_i (y_i - b'(\theta))}{\phi}\]

and second differential:

\[ \frac{d^2 l}{d \theta^2} = -\frac{w_i b''(\theta)}{\phi} \]

Combining these two results with Bartlett’s second identity:

\[ Var_\theta\left[ \frac{d l}{d\theta} \right] = - E_\theta\left[ \frac{d^2 l}{d\theta^2} \right] \]

We can deduce that:

\[ Var_\theta\left[ \frac{w_i (y_i - b'(\theta))}{\phi} \right] = - E_\theta\left[ -\frac{w_i b''(\theta)}{\phi} \right] \]

\[ \frac{w_i^2 Var(y_i) }{\phi^2} = \frac{w_i b''(\theta)}{\phi} \]

\[ Var(y_i) = \frac{\phi b''(\theta)}{w_i} = \frac{\phi V(\theta)}{w_i} \]

In the stats package each distribution family carries its own variance function defined relative to \(\mu\) not \(\theta\), \(V(\mu) = b''(b'^{-1}(\mu))\). Hence we use:

\[ Var(y_i) = \frac{\phi V(\mu)} {w_i} \]