Null Hypothesis

This reading moves from a sample to the population: how sample means behave under the central limit theorem, how confidence intervals are built and how samples are drawn. It then sets out the steps of a hypothesis test, the test statistic for each parametric test, the two types of error, and the nonparametric tests used for ranks and categories.

LOS 7.a — The central limit theorem, confidence intervals and sampling

The central limit theorem (CLT)

A population parameter such as $\mu$ or $\sigma$ is a fixed number, usually unknown. A sample statistic such as $\bar{X}$ or $s$ is computed from the sample actually drawn, so it would take a different value in another sample: it is itself a random variable. The distribution of its values over all possible samples of the same size is the statistic's sampling distribution.

Take many simple random samples of size $n$ from a population with mean $\mu$ and a finite variance ${\sigma }^2$. The central limit theorem says that the sampling distribution of the sample mean $\bar{X}$ becomes approximately normal as $n$ grows, whatever the shape of the population itself (skewed, flat, lumpy):

Three consequences follow:

• The mean of all possible sample means equals the population mean: $E(\bar{X})=\mu$.
• The variance of the sample mean is the population variance divided by the sample size, ${\sigma }^2/n$, which shrinks as $n$ rises.
• The CLT describes the distribution of the sample mean. Individual observations keep the population's shape, and the CLT says nothing about the distribution of the sample standard deviation.

Standard error and sampling error

Sampling error is the difference between a sample statistic and the population parameter it estimates (e.g., $\bar{X}-\mu$). Its typical size is the standard error of the sample mean:

When $\sigma$ is unknown, $s$ replaces it, and the result is an estimate of the standard error. The standard deviation describes how widely individual observations scatter around the mean, and a larger sample does not make it smaller. The standard error describes how precisely $\bar{X}$ estimates $\mu$.

Calculator. From raw data, enter the observations in the DATA worksheet, read the sample standard deviation $S_x$ in the STAT worksheet (1-V), and divide it by $\sqrt{n}$.

Desirable properties of an estimator

Confidence intervals

A point estimate is a single value (e.g., $\bar{X}$). A confidence interval is a range built around it that contains the true parameter with a stated confidence level $1-\alpha$, where $\alpha$ is the significance level:

Two-tailed z reliability factors $z_{\alpha /2}$ are 1.65 for 90% (1.645 before rounding), 1.96 for 95% and 2.58 for 99% (2.576 before rounding). Three things set the width of an interval. A higher confidence level (a larger reliability factor) widens it, a more dispersed population widens it, and a larger sample (a smaller standard error) narrows it.

The Student's t-distribution is bell-shaped and symmetric but has fatter tails than the normal; its critical values are larger than z-values (wider intervals) and fall toward the z-value as degrees of freedom rise. For a 95% interval, the t reliability factor is 2.228 with 10 df, 2.086 with 20 df and 2.042 with 30 df, against 1.96 for z. Select a t-value from a table using the area in one tail ($\alpha /2$ for a two-sided interval) and $n-1$ degrees of freedom.

Value at risk (VaR) — a one-tailed interval

Value at risk is the loss that will be equaled or exceeded with a stated (small) probability over a given period. "The daily 1% VaR is $0.47 million" means a 1% chance on any day of losing $0.47 million or more (equivalently, 99% confidence that the loss will be smaller). VaR is the minimum loss at the stated probability; larger losses remain possible, so it is not the worst possible loss.

Under the parametric method (normal returns, mean return assumed to be zero):

with the whole $\alpha$ in one tail: $z_{10\%}=1.28$, $z_{5\%}=1.65$ (1.645 before rounding), $z_{1\%}=2.33$. With $\sigma$ in percent and the portfolio value in dollars, VaR is a dollar loss; dropping the portfolio value gives VaR as a percentage of the portfolio. Example. For a $25 million portfolio with daily $\sigma =0.8\%$, the daily 1% VaR $=2.33\times 0.008\times \$25\text{m}=\$466,000$, or 1.864% of the portfolio.

Drawbacks: returns are often skewed/fat-tailed, parameters change over time, history may not reflect new conditions, and it handles nonlinear payoffs (options) poorly.

Sampling methods

Probability sampling is generally more accurate; nonprobability sampling is cheaper. Stratified and cluster sampling both split the population into groups. Stratified sampling draws from every group, so the sample keeps the population's mix across groups; cluster sampling uses only some groups, so differences between groups can be lost.

The CLT does not rescue a non-representative sample.

LOS 7.b — Hypothesis tests of a population mean

What a hypothesis test does

A hypothesis is a claim about what value a population parameter takes (e.g., "the mean daily return of this strategy is zero"). A sample is used to decide whether the data give enough evidence to reject that statement.

• Parametric tests make assumptions about the population's distribution and test its parameters (mean, variance).
• Nonparametric tests make few distributional assumptions or test things that are not parameters (ranks, category counts); Module 7.4 covers them.

The four steps

1. State the null hypothesis $H_0$ and the alternative hypothesis $H_a$.
2. Identify the test statistic and its distribution.
3. Specify the significance level $\alpha$.
4. Establish the decision rule (the critical value(s)) and compare.

Hypotheses. The null is usually the claim the researcher hopes to reject, and it always contains the equality (=, ≤ or ≥). The alternative is what the researcher wants to find evidence for. The two are mutually exclusive and exhaustive.

Test statistic for a mean

The test statistic measures how many standard errors the sample mean lies from the hypothesized value:

When $s$ is used and the population is normal (or approximately normal), the statistic follows a t-distribution with $n-1$ degrees of freedom; with a large sample ($n\ge 30$) the CLT makes the t-test approximately valid even for a non-normal population, and the z-statistic (standard normal) is an acceptable approximation. A small sample from a clearly non-normal population supports neither test. When the population is normal and its variance is known, the statistic uses $\sigma$ and follows the standard normal distribution at any sample size; z critical values have no degrees of freedom. The sign matters: a negative statistic means the sample mean is below ${\mu }_0$.

Critical z-values (exam values; three-decimal values in parentheses):

Decision rule. Reject $H_0$ if the test statistic falls in the rejection region (beyond the critical value); otherwise fail to reject $H_0$. The null is never "accepted"; failing to reject means only that the data were not strong enough to reject it. The confidence level is $1-\alpha$. For a two-tailed test the rule is equivalent to a confidence interval: $H_0$ is rejected when the sample mean falls outside the interval ${\mu }_0\pm$ (critical value $\times$ standard error) built around the hypothesized value.

The critical value depends on three things: the distribution of the test statistic (z, or t with its degrees of freedom), the significance level, and whether the test is one-tailed or two-tailed. At $\alpha =5\%$, for example, a statistic of 1.75 exceeds the critical value of a right-tailed test (1.65) but not that of a two-tailed test (1.96), so the two tests reach different decisions.

What moves the statistic. For a sample mean above ${\mu }_0$, the statistic (and the chance of rejection) rises when the sample mean rises, the standard deviation falls or the sample size rises (smaller standard error).

Type I and Type II errors, power

• The significance level is the probability of a Type I error, that is, of rejecting a true null by chance.
• The power of a test is the probability of correctly rejecting a false null: power $=1-P(\text{Type II error})$.
• Only one error type can occur in a given test: a Type I error needs a true null, a Type II error needs a false null.
• Trade-off: with the sample size and the test statistic unchanged, lowering $\alpha$ (e.g., from 5% to 1%) raises the probability of a Type II error and lowers power; raising power in that setting means accepting a higher $\alpha$. A lower $\alpha$ also means a larger critical value, so the matching confidence interval is wider.
• Exam convention: for a given $\alpha$, power can be raised only by increasing the sample size, which lowers the probability of a Type II error while the probability of a Type I error stays at $\alpha$. Current practice: at a given $\alpha$, less noisy data (a lower standard deviation) or a more powerful test statistic also raise power.
• The chance of a Type II error depends jointly on $\alpha$ and the sample size, but the relation is not simple and that probability is hard to compute in practice. When several test statistics could be used for the same hypothesis, the most powerful one is normally preferred.

LOS 7.b — Other parametric hypothesis tests

Every test below uses the same four steps as the test of a single mean (hypotheses, test statistic, significance level, decision rule). What changes is the test statistic and the distribution its critical values come from.

Summary of the tests

Difference between means vs. paired comparisons

• Independent samples (e.g., returns of small-cap funds vs. returns of large-cap funds): test $H_0:{\mu }_1-{\mu }_2=0$ with the difference-in-means t-test, df $=n_1+n_2-2$. These df belong to the pooled version, which assumes the two population variances are equal and combines both samples into one variance estimate $s_p^2$. A test that does not assume equal variances uses the unpooled standard error $\sqrt{s_1^2/n_1+s_2^2/n_2}$, with df from a separate (Welch) approximation that are usually smaller than $n_1+n_2-2$. When $n_1=n_2$, the pooled and unpooled standard errors are identical, and when the sample variances are similar the two t-values are close.
  • Exam convention: the difference-in-means statistic may be shown with the unpooled standard error $\sqrt{s_1^2/n_1+s_2^2/n_2}$ and still be compared with a t critical value for $n_1+n_2-2$ df.
  • Current practice: $n_1+n_2-2$ df go with the pooled standard error (equal variances assumed); the unpooled (Welch) test uses fewer df.
• Dependent samples (the same members measured twice, e.g., each fund's return before and after a fee cut): compute the difference for each pair, then run a single-mean test on those differences. This is the paired comparisons test, with df $=n-1$.

Test of a single variance (chi-square)

The chi-square distribution is asymmetric (right-skewed) and bounded below by zero. For a two-tailed test at $\alpha$, find a lower critical value (area $1-\alpha /2$ to its right) and an upper one (area $\alpha /2$ to its right). Example. With 21 observations, $s=5.5\%$ and $H_0:\sigma =4.5\%$, ${\chi }^2=20\times 5.5^2/4.5^2=29.88$. With 20 df and $\alpha =5\%$ the critical values are 9.591 and 34.170, so the null is not rejected.

A hypothesis about a standard deviation is written in terms of the variance. With returns in decimals, a claim that $\sigma$ is below 3% becomes $H_0:{\sigma }^2\ge 0.0009$ versus $H_a:{\sigma }^2<0.0009$, and only the lower critical value is used.

Test of two variances (F-test)

The F-distribution is right-skewed, bounded below by zero and defined by two degrees of freedom (numerator $n_1-1$, denominator $n_2-1$). $F=1$ when the sample variances are equal. Putting the larger variance in the numerator means only the upper critical value matters (with $\alpha /2$ in that tail for a two-tailed test).

• $H_0:{\sigma }_1^2={\sigma }_2^2$ (the variances are not different) vs. $H_a:{\sigma }_1^2\ne {\sigma }_2^2$.
• If the F-statistic exceeds the critical F-value, reject $H_0$ and conclude that the variances are significantly different. Example. With $s_1=6.0$ ($n_1=16$) and $s_2=4.2$ ($n_2=21$), $F=36/17.64=2.04$. The critical value $F_{15,20}$ with 2.5% in the upper tail is 2.573, so the null is not rejected.

Test of a correlation coefficient

The correlation coefficient (the Pearson correlation) gauges how strong a linear relationship is and lies between −1 (perfect negative) and +1 (perfect positive); zero means no linear relationship. $H_0:\rho =0$ vs. $H_a:\rho \ne 0$. The statistic needs only the sample correlation $r$ and the sample size $n$; it follows a t-distribution with $n-2$ degrees of freedom. It grows with $|r|$ and with $n$, so a modest correlation can be significant in a large sample: $r=0.25$ gives $t=2.56$ with $n=100$ but only $t=0.86$ with $n=13$. Example. With $r=0.30$ and $n=50$, $t=0.30\sqrt{48}/\sqrt{0.91}=2.18$. This exceeds $t_{48,0.025}=2.011$, so $\rho =0$ is rejected.
When a table is given, select the row for $n-2$ df and the column for the stated (two-tailed) significance level. The null is rejected at a given level only if $|t|$ exceeds that column's value.

p-values and p-hacking

The p-value is the smallest significance level at which the null can be rejected (the probability, if $H_0$ is true, of a statistic at least as extreme as the one observed). Reject $H_0$ when p-value < $\alpha$. p-hacking is manipulating data or the testing process to manufacture a statistically significant result, e.g.:

• testing only the best periods, or deleting inconvenient observations as "outliers";
• data mining: running many tests until one rejects the null (with $\alpha =5\%$, about 1 in 20 tests of true nulls rejects by chance);
• using smoothed (e.g., appraisal-based) data to understate risk.
  Because it produces rejections of nulls that are actually true, p-hacking raises the frequency of Type I errors (false positives). It is not the same as legitimately choosing a significance level or seeking a more powerful test.

LOS 7.c — Parametric vs. nonparametric tests

When is a test nonparametric?

Use a nonparametric test when (1) the data are ranks or categories, (2) the distributional assumptions of a parametric test do not hold (e.g., a small sample from a clearly non-normal population, so the CLT cannot be relied on), or (3) the question is not about a parameter at all, for example whether two sets of ranks are related or whether a set of observations is consistent with a particular distribution.

Spearman rank correlation test

The Spearman rank correlation measures the strength of a monotonic relation between two sets of ranks (e.g., a fund's performance rank this year vs. next year). With integer ranks and $d_i$ = difference between the two ranks of item $i$:

Test of independence using a contingency table

A contingency table counts observations classified by two categorical characteristics (e.g., issuer sector × credit-rating change). The chi-square test of independence asks whether the two characteristics are related.

• $H_0$: the two characteristics are independent. $H_a$: they are not independent.
• Expected frequency of cell $(i,j)$ if independent. Under independence an observation's row says nothing about its column, so each row total is split across the columns in the same proportions as the column totals:

• Test statistic (sum over all cells of squared deviations, each scaled by the expected count):

• Reject independence if ${\chi }^2$ exceeds the critical chi-square value (right tail only).
• The standardized residual (Pearson residual) of a cell is $(O_{i,j}-E_{i,j})/\sqrt{E_{i,j}}$; its square is that cell's contribution to ${\chi }^2$.
  • Exam convention: each standardized residual is approximately standard normal, and cells with values beyond ±2 are flagged as contributing significantly to the lack of independence.
  • Current practice: because the row and column totals are estimated from the data, the raw residual has a variance below 1, so ±2 works as a screening rule of thumb. The adjusted residual, which divides $O_{i,j}-E_{i,j}$ by $\sqrt{E_{i,j}(1-\text{row share})(1-\text{column share})}$, is the one that is approximately standard normal.

Expected (small, upgraded) $=100\times 60/220=27.27$. Summing $(O-E{)}^2/E$ over the six cells gives ${\chi }^2=20.38$ with $(2-1)(3-1)=2$ df. The 5% critical value is 5.991 (1%: 9.210), so independence is rejected. The standardized residual for (small, upgraded) is $(42-27.27)/\sqrt{27.27}=2.82>2$: by the ±2 rule of thumb, small issuers were upgraded notably more often than independence implies.