The general Multivariate Linear Model (MLM) can be understood as a simple extension of the univariate linear model, with the main difference being that there are multiple response variables considered together, instead of just one, analysed alone. These outcomes might reflect several different ways or scales for measuring an underlying theoretical construct, or they might represent different aspects of some phenomenon that are better understood when studied jointly.
For example, in the first case, there are numerous psychological scales used to assess depression or anxiety and it may be important to include more than one measure to ensure that the construct has been measured adequately. In the second case, student “aptitude” or “achievement” reflects competency in different various subjects (reading, math, history, science, …) that are better studied together.
In this context, there are multiple techniques that can be applied depending on the structure of the variables at hand. For instance, with one or more continuous predictors and multiple response variables, one could use multivariate regression to obtain estimates useful for prediction. Instead, if the predictors are categorical, multivariate analysis of variance (MANOVA) can be applied to test for differences between groups. Again, this is akin to multiple regression and ANOVA in the univariate context – the same underlying model is utilized, but the tests for terms in the model are multivariate ones for the collection of all response variables, rather than univariate ones for a single response.
TODO Use \Epsilon = \(\boldsymbol{\large\varepsilon}\) here, which is defined as \boldsymbol{\large\varepsilon} for residuals. Could also use a larger version, \boldsymbol{\Large\varepsilon} = \(\boldsymbol{\Large\varepsilon}\) if that makes a difference.
In each of these cases, the underlying MLM is given most compactly using the matrix equation,
\[ \mathord{\mathop{\mathbf{Y}}\limits_{n \times p}} = \mathord{\mathop{\mathbf{X}}\limits_{n \times q}} \, \mathord{\mathop{\mathbf{B}}\limits_{q \times p}} + \mathord{\mathop{\mathbf{U}}\limits_{n \times p}} \:\: , \]
where
- \(\mathbf{Y} = (\mathbf{y}_1 , \mathbf{y}_2, \dots , \mathbf{y}_p )\) is the matrix of \(n\) observations on \(p\) responses;
- \(\mathbf{X}\) is the model matrix with columns for \(q\) regressors, which typically includes an initial column of 1s for the intercept;
- \(\mathbf{B}\) is a matrix of regression coefficients, one column for each response variable; and
- \(\mathbf{U}\) is a matrix of errors in predicting \(\mathbf{Y}\).
The structure of the model matrix \(\mathbf{X}\) is the same as the univariate linear model, and may contain, therefore,
- quantitative predictors, such as
age,income, years ofeducation - transformed predictors like \(\sqrt{\text{age}}\) or \(\log{\text{income}}\)
- polynomial terms: \(\text{age}^2\), \(\text{age}^3, \dots\) (using
poly(age, k)in R) - categorical predictors (“factors”), such as treatment (Control, Drug A, drug B), or sex; internally a factor with
klevels is transformed tok-1dummy (0, 1) variables, representing comparisons with a reference level, typically the first. - interaction terms, involving either quantitative or categorical predictors, e.g.,
age * sex,treatment * sex.
10.0.1 Assumptions
The assumptions of the multivariate linear model entirely concern the behavior of the errors (residuals). Let \(\mathbf{u}_{i}^{\prime}\) represent the \(i\)th row of \(\mathbf{U}\). Then it is assumed that
- The residuals, \(\mathbf{u}_{i}^{\prime}\) are distributed as multivariate normal, \(\mathcal{N}_{p}(\mathbf{0},\boldsymbol{\Sigma})\), where \(\mathbf{\Sigma}\) is a non-singular error-covariance matrix;
- The error-covariance matrix \(\mathbf{\Sigma}\) is constant across all observations and grouping factors;
- \(\mathbf{u}_{i}^{\prime}\) and \(\mathbf{u}_{j}^{\prime}\) are independent for \(i\neq j\); and
- The predictors, \(\mathbf{X}\), are fixed or independent of \(\mathbf{U}\).
These statements are simply the multivariate analogs of the assumptions of normality, constant variance and independence of the errors in univariate models.