Quantitative Methods · Reading 10

Least Squares Regression

CFA Level I · Quantitative Methods · Reading 10: Applications of Simple Linear Regression in Finance · about 1 h 1 min

Quantitative Methods notes as a PDF166 pages · 2.6 MB · free, no sign-up · All PDFs

What you'll learn

LOS 10.a Describe and interpret simple linear regression and estimate its coefficients using the least squares criterion.
LOS 10.b Describe the assumptions of simple linear regression, detect violations from residuals, and evaluate goodness of fit, coefficient tests and ANOVA results.
LOS 10.c Calculate and interpret predicted values, the standard error of estimate and prediction intervals, and describe log-lin, lin-log and log-log functional forms.
LOS 10.d Estimate and interpret CAPM beta and related statistics with a simple linear regression of excess returns on the market risk premium.

Module 10.1

Linear Regression Basics

This reading fits a simple linear regression by least squares, interprets its slope and intercept, and checks the model's assumptions with residual plots. It then measures fit with ANOVA, the standard error of estimate, $R^{2}$ and the F-test, tests the coefficients, builds prediction intervals, chooses among log-transformed functional forms, and estimates a stock's CAPM beta by regression.

LOS 10.a — Simple linear regression and the least squares criterion

Purpose and vocabulary

Simple linear regression explains the variation of one dependent variable with the variation of a single independent variable. Variation is the total squared deviation from the mean, $\sum_{i = 1}^{n} (Y_{i} - Y ˉ)^{2}$ ; it is related to, but not the same as, the variance (the variance divides the variation by $n - 1$ ).

	Dependent variable (Y)	Independent variable (X)
Role	The variable being explained	The variable doing the explaining
Other names	Explained, endogenous, predicted variable	Explanatory, exogenous, predicting variable
Example: using GDP growth to forecast stock returns	Stock returns	GDP growth

A model with one dependent variable and one independent variable is a simple regression; a model with one dependent variable and more than one independent variable is a multiple regression (Level II).

The model and the regression line

Key concept

Y_{i} = b_{0} + b_{1} X_{i} + ε_{i} (population model)

Y^_{i} = b^_{0} + b^_{1} X_{i} (estimated regression line, the line of best fit)

The hat marks an estimate. The residual for observation $i$ , $Y_{i} - Y^_{i}$ , is the gap between the actual value and the value the fitted line predicts for that X.

Exam convention: the residual, the error term and the disturbance term are treated as the same thing, written $ε_{i}$ .
Current practice: the error term $ε_{i}$ is the unobservable deviation in the population model, and the residual is its sample estimate.

Least squares criterion

Of all the lines that could be drawn through the scatter plot, ordinary least squares (OLS) regression picks the one that minimizes the sum of squared errors (SSE), found by squaring each vertical gap between an observed value of the dependent variable and its fitted value, then adding the squares:

SSE = i = 1 \sum n (Y_{i} - Y^_{i})^{2}

The resulting estimates are:

Key concept

b^_{1} = \frac{Cov ( X , Y )}{σ _{X}^{2}}, b^_{0} = Y ˉ - b^_{1} X ˉ

The intercept formula shows that the regression line always passes through the point of means $(X ˉ, Y ˉ)$ . The covariance and the variance share the divisor $n - 1$ , which cancels, so from raw data the slope is $b^_{1} = \sum (X_{i} - X ˉ) (Y_{i} - Y ˉ) / \sum (X_{i} - X ˉ)^{2}$ .

Eight observations (X, Y) in percent: (-4, -2.9), (-2, -0.9), (-1, 0.1), (0, 0.2), (1, 1.8), (2, 1.7), (3, 2.6), (5, 4.7). The fitted regression line is Y-hat = 0.51 + 0.80X and passes through the point of means (0.50, 0.91). For the observation at X = 1, the vertical residual is 1.8 - 1.31 = 0.49. — Scatter plot with the least-squares regression line

Worked example (data in the figure)

Eight monthly observations give $X ˉ = 0.50%$ , $Y ˉ = 0.9125%$ , $Cov (X, Y) = 6.593$ and $σ_{X}^{2} = 8.286$ (both in %²). The underlying sums are $\sum (X_{i} - X ˉ) (Y_{i} - Y ˉ) = 46.15$ and $\sum (X_{i} - X ˉ)^{2} = 58.0$ , and $46.15/58.0$ gives the same slope.

b^_{1} = 6.593/8.286 = 0.7957 \approx 0.80, b^_{0} = 0.9125 - 0.7957 (0.50) = 0.51%

For the observation at $X = 1%$ with $Y = 1.8%$ : $Y^= 0.51 + 0.80 (1) = 1.31%$ , so the residual is $1.8 - 1.31 = 0.49%$ .

Interpreting the coefficients

Slope coefficient $b^_{1}$ : how much the dependent variable is expected to move when the independent variable rises by one unit. In the example, a market excess return 1 percentage point higher goes with a stock excess return 0.80 point higher. When Y is a stock's excess return and X the market's, the slope is the stock's beta.
Intercept $b^_{0}$ : the predicted value of Y when X equals zero (where the line crosses the vertical axis).
A large slope does not by itself mean a strong relationship; significance must be tested (t-test or confidence interval).

Types of data

Key concept

Type	Definition	Example
Time series	Observations of one variable at regular intervals over time	A company's quarterly EPS for five years
Cross-sectional data	Observations of many subjects at one point in time	Today's closing prices of 30 stocks in an index
Panel data	Cross-sectional observations repeated over time	Quarterly EPS of 10 companies over five years

Common exam traps

The slope divides the covariance by the variance of X, the independent variable. Dividing by the variance of Y is a common error.
OLS minimizes the squared differences between actual and predicted values of Y. It does not compare coefficients or X values.
The slope is the change in Y per unit change in X. Reading it as the change in X per unit of Y reverses the roles.

Exam shortcuts

The regression line passes through $(X ˉ, Y ˉ)$ , so once the slope is known the intercept is $Y ˉ - b^_{1} X ˉ$ with no further sums.

Bottom line

Simple linear regression explains the variation of a dependent variable Y, $\sum (Y_{i} - Y ˉ)^{2}$ , with the variation of a single independent variable X.
Ordinary least squares chooses the line that minimizes the sum of squared errors, $SSE = \sum (Y_{i} - Y^_{i})^{2}$ .
The OLS estimates are $b^_{1} = Cov (X, Y) / σ_{X}^{2}$ and $b^_{0} = Y ˉ - b^_{1} X ˉ$ , so the fitted line passes through the point of means.
The slope is the expected change in Y for a one-unit rise in X, and the intercept is the predicted value of Y when X equals zero.
The residual $Y_{i} - Y^_{i}$ is the gap between the actual and the fitted value; exam convention treats the residual, error term and disturbance term as the same thing, while current practice treats the residual as the sample estimate of the unobservable error term.
Time series data follow one variable over time, cross-sectional data cover many subjects at one point in time, and panel data repeat cross-sectional observations over time.

Quick check

Question 1Core

A credit analyst gathers data on 80 bond issuers at a single date to see whether an issuer's interest coverage ratio helps explain the yield spread on its bonds. In her simple linear regression, the dependent variable is:

Show answer and explanation

Correct answer: B

The dependent variable is the one whose variation the model explains, here the yield spread. The interest coverage ratio does the explaining, so it is the independent (explanatory) variable. Observations of many issuers at one date are cross-sectional data, which a simple linear regression can use as readily as time-series data.

Why the other options are wrong

A. An explanatory variable is the independent variable. Making the coverage ratio the dependent variable reverses the roles set by the analyst's question.
C. Data on many subjects at one point in time are cross-sectional data, and a regression can be run on them. Time-series data are not required.

Key takeaway Dependent variable (Y): explained, endogenous, predicted. Independent variable (X): explanatory, exogenous, predicting. The research question decides which is which.

Module 10.2

Analysis of Variance (ANOVA) and Goodness of Fit

LOS 10.b — Assumptions, residual analysis, ANOVA and goodness of fit

Assumptions of simple linear regression

Key concept

Linearity: the relationship between X and Y is linear.
Homoskedasticity: the residuals have the same variance for every observation.
Independence: residuals are uncorrelated with one another (the X–Y pairs are independent).
Normality: the residual term is normally distributed.

Violations show up in plots of the residuals against X (or against time). In a well-specified model the residuals scatter randomly around zero with no pattern.

Violation	What the residual plot shows	Name
Nonlinear relationship	A systematic pattern, e.g., residuals positive, then negative, then positive as X rises	—
Residual variance not constant	Spread of residuals widens (or narrows) with X, or changes over time	Heteroskedasticity
Residuals correlated	Recurring pattern, e.g., large errors every December (seasonality); the estimated variances of the coefficients are then wrong	Autocorrelation
Residuals not normal	Skewed or fat-tailed residuals; hypothesis tests may be unreliable in small samples (with a large sample, the central limit theorem means the estimates may still be valid)	—

A straight line fitted to a curved relationship, and the residual pattern it leaves

Two panels of residuals plotted against the independent variable X (1 to 20), about 60 points each. Panel A: residuals scatter randomly within roughly -3 to +2.5 with the same spread at every X (homoskedastic). Panel B: residuals are tightly clustered near zero at low X and spread out steadily as X increases, reaching about -6 to +10 at the highest X (heteroskedastic). — Residual plots: homoskedastic vs. heteroskedastic

Outliers are observations far from the line. They can pull the estimated coefficients so that the line fits the other points poorly.

Analysis of variance (ANOVA)

Analysis of variance (ANOVA) splits the total variation of the dependent variable into an explained and an unexplained part:

Total sum of squares (SST) $= \sum (Y_{i} - Y ˉ)^{2}$ : total variation.
Sum of squares regression (SSR) $= \sum (Y^_{i} - Y ˉ)^{2}$ : variation explained by the regression.
Sum of squared errors (SSE) $= \sum (Y_{i} - Y^_{i})^{2}$ : unexplained variation (also called the residual sum of squares).

SST = SSR + SSE

Scatter of eight illustrative points with a fitted line (slope 0.71, intercept 1.26) and the mean of Y at 4.45. For the point at X = 8, Y = 7.9, the total deviation from the mean splits into an explained part (fitted value minus mean, summed over all points gives SSR) and an unexplained part (Y minus fitted value, summed gives SSE). — Total variation split into explained and unexplained parts (illustrative data)

The sample variance of Y is its total variation divided by its degrees of freedom, $SST / (n - 1)$ .

Mean square regression (MSR) $= SSR / k$ ; with one independent variable ( $k = 1$ ), MSR = SSR.
Mean squared error (MSE) $= SSE / (n - k - 1) = SSE / (n - 2)$ in a simple regression.

Key concept

Source of variation	Degrees of freedom	Sum of squares	Mean sum of squares
Regression (explained)	$k = 1$	SSR	MSR = SSR/1
Error (unexplained)	$n - 2$	SSE	MSE = SSE/( $n - 2$ )
Total	$n - 1$	SST

Goodness-of-fit measures

SEE = MSE = \frac{SSE}{n - 2}

The standard error of estimate (SEE), also called the root mean squared error, is the standard deviation of the residuals; the lower it is, the better the fit.

Key concept

R^{2} = \frac{SSR}{SST} = 1 - \frac{SSE}{SST}

The coefficient of determination $(R^{2})$ measures what fraction of the dependent variable's total variation the independent variable accounts for. In a simple regression $R^{2} = r^{2}$ , so the correlation coefficient is $r = \pm R^{2}$ , with the sign of the slope. When only $R^{2}$ and SSR are known, SST $=$ SSR $/ R^{2}$ and SSE $=$ SST $-$ SSR, which then gives the MSE and the SEE.

F-test

The F-test compares the variation the regression explains, per degree of freedom, with the variation it leaves unexplained, per degree of freedom. A large ratio means the independent variable explains a lot relative to the noise.

F = \frac{MSR}{MSE} df = 1 (numerator), n - 2 (denominator)

The F-test asks whether the independent variables, as a group, explain the variation of Y; with several independent variables its null is that all slope coefficients are zero. With one independent variable, it tests $H_{0} : b_{1} = 0$ vs. $H_{a} : b_{1} \neq = 0$ , the same hypothesis as the t-test of the slope. Reject $H_{0}$ if $F > F_{c}$ (a one-tailed test using the upper tail of the F-distribution). Rejecting $H_{0}$ means X and Y have a significant linear relationship.

Worked example

$n = 30$ ; SSR = 64 and SSE = 112, so SST = 176. MSE $= 112/28 = 4$ ; SEE $= 4 = 2$ ; $R^{2} = 64/176 = 0.364$ ; $r = 0.364 = 0.60$ (if the slope is positive); $F = 64/4 = 16$ , well above the 5% critical value of about 4.2 for 1 and 28 degrees of freedom, so the slope is significant.

t-test of a regression coefficient

t = \frac{b ^ _{1} - b _{1}}{s _{b^_{1}}} df = n - 2

To test significance, set $H_{0} : b_{1} = 0$ (two-tailed) and reject if $∣ t ∣ > t_{c}$ . The test of the slope is equivalent to testing the correlation, $t = r n - 2 / 1 - r^{2}$ . With one independent variable, the F-statistic equals the square of this t-statistic, so the F-test and the t-test of the slope always reach the same decision (in the worked example above, $F = 16$ corresponds to $t = 4$ ). The intercept can be tested the same way ( $H_{0} : b_{0} = 0$ ).

Indicator variables

An indicator variable (dummy variable) takes the value 1 when a condition holds and 0 otherwise (e.g., 1 in recession quarters). Regressing Y on the indicator tests whether the condition matters: the slope equals the difference in the mean of Y between periods with and without the condition, and testing $b_{1} = 0$ is equivalent to a difference-in-means test. The intercept is the mean of Y in periods without the condition (indicator = 0), and intercept plus slope is the mean in periods with it. Example. Regressing a company's quarterly earnings on a recession indicator tests whether earnings are cyclical; for cyclical earnings the slope is expected to be negative (lower earnings in recession quarters). If quarterly EPS averaged 1.40 in expansion quarters and 0.95 in recession quarters, the fitted line is $EPS = 1.40 - 0.45 D$ , where $D = 1$ in a recession quarter.