Contents
31 — Heritability and response to selection Part 6 — Quantitative genetics

31 — Heritability and response to selection

Before this: Ch 30 · Ch 26 · Time: ~50 min

Statistics needed: S5 Variance and regression · S3 Sampling and estimation

is the most misused quantity in biology, and several of the false statements about it are made by people who can define it correctly. This chapter derives it, derives what it predicts, and then spends an unusually long section on what it does not.

What you'll be able to do

The core idea

Variance decomposition and regression coefficients are covered in S5; this chapter assumes them. The content here is that one particular regression coefficient — the slope of offspring on parent phenotype — is causal in exactly one narrow sense (it predicts what happens if you breed from selected parents) and worthless for every other causal question people ask of it.

is not a measure of how genetic a trait is. It is the fraction of the variance in one population, in one environment, at one time that is statistically predictable from parental phenotype. Change the environmental variance and it changes without a single moving. It is a property of a population, not of a trait, and never of an individual.


1. Two heritabilities

Start from the partition established in Ch 30. A is genotypic value plus environmental deviation, P = G + E, and if the two are uncorrelated:

V_P = V_G + V_E = (V_A + V_D + V_I) + V_E
graph TD VP["<b>V_P</b> phenotypic variance"] VG["V_G genotypic"] VE["V_E environmental"] VA["<b>V_A</b> additive<br/><i>transmitted to offspring</i>"] VD["V_D dominance<br/><i>within-locus interaction</i>"] VI["V_I epistatic<br/><i>between-locus interaction</i>"] VP --> VG VP --> VE VG --> VA VG --> VD VG --> VI
DefinitionAnswers
Broad-sense H²V_G / V_PWhat fraction of variance is attributable to at all?
Narrow-sense h²V_A / V_PWhat fraction of variance is predictable from parents?

Why V_A is the transmitted part. Parents do not transmit genotypes. They transmit one per locus, drawn at random (Ch 09). Dominance is a property of a pair of alleles, destroyed and re-drawn every , and a parent and its offspring can never hold a genotype identical by descent — so dominance contributes nothing whatever to parent–offspring covariance. is not so clean. A combination across loci is also broken up every generation, but not completely: the resemblance between relatives picks up epistatic variance with squared coefficients (Ch 30 §7), which for additive-by-additive variance means ¼V_AA between parent and offspring and between full sibs, and ¹⁄₁₆V_AA between half sibs. Epistatic variance is therefore partly transmitted, and is one of the things that quietly inflates any h² estimated from relatives. The clean statement is the narrow one: V_A is what the regression on parents recovers, and everything else either vanishes (dominance) or leaks in at a reduced rate (epistasis).

Make that exact. One locus, alleles A₁ (frequency p) and A₂ (q = 1 − p), genotypic values scaled about the homozygote midpoint:

        A₂A₂        A₁A₂        A₁A₁
         -a           d          +a

The population mean is M = a(p − q) + 2pqd. Now ask: if I hand an offspring one allele, by how much does that shift its expected value? That is the average effect of a substitution, and working through the genotype frequencies gives

α = a + d(q − p)

An individual's is the sum of the average effects of the alleles it carries — by construction, the part of its genotypic value that its offspring inherit. The variance of breeding values and the residual are

V_A = 2pq α²          V_D = (2pq d)²

Two consequences worth carrying. First, α depends on , so V_A is not a property of the gene — it is a property of the gene in a population. A locus with pure dominance (a = 0, d > 0) still contributes additive variance whenever p ≠ q, because at unequal frequencies substituting an allele does change the expectation on average. Second, at p = q = ½ that same locus contributes α = 0 and hence no additive variance at all. "Additive" is a statistical decomposition, not a statement about biochemistry.

2. Estimating h² from relatives is one covariance calculation

Statistics: the covariance algebra this section runs on — Var(X + Y) = Var(X) + Var(Y) + 2Cov(X, Y), and the least-squares slope Cov(x, y)/Var(x) — is S5 §§2 and 4.

Write an offspring's in terms of its parents':

A_O = ½A_sire + ½A_dam + m

where m is the Mendelian sampling term — which of each pair got transmitted — with mean 0 and independent of the parental breeding values. Then

Cov(P_O, P_parent) = Cov(A_O + D_O + E_O , A_p + D_p + E_p) = ½ V_A  (+ ¼ V_AA + …)

Dominance drops out because a parent and offspring share exactly one allele per locus by descent and can never share a by descent. does not drop out: with a coefficient of relationship r = ½, additive-by-additive variance enters as r²V_AA = ¼V_AA (Ch 30 §7), and it is indistinguishable from ½V_A in this one covariance — so it rides along inside every h² you estimate this way. Environment drops out only if parents and offspring do not share environments — the assumption that does all the damage in humans.

Regression on midparent. Let MP = (P_sire + P_dam)/2. Under random mating the parents are uncorrelated, so Var(MP) = (V_P + V_P)/4 = V_P/2, and

Cov(P_O, MP) = ½[Cov(P_O,P_sire) + Cov(P_O,P_dam)] = ½[½V_A + ½V_A] = ½V_A

b_O·MP = Cov(P_O, MP) / Var(MP) = (½V_A) / (½V_P) = V_A / V_P = h²

The slope is the , with no scaling constant. Regress on a single parent instead and Var(P) is twice as large, so the slope is ½h². This is why the midparent regression is the canonical estimator, and it is exact only if: parents and offspring share no environment, there is no genotype–environment correlation, causal loci are in equilibrium, and the trait is measured on the same scale in both generations.

Random mating is conspicuously absent from that list. Let mates correlate ρ for the . Assortment makes each parent's phenotype informative about the other parent's breeding value — Cov(A_dam, P_sire) = h² Cov(P_dam, P_sire) = ρ V_A — so the numerator inflates alongside the denominator, and by the same factor:

Cov(P_O, MP) = ¼[V_A + ρV_A + ρV_A + V_A] = ½V_A(1 + ρ)
Var(MP)      = ¼[V_P + 2ρV_P + V_P]       = ½V_P(1 + ρ)

b_O·MP = V_A / V_P = h²      — the (1 + ρ) cancels

The offspring–midparent regression is the one design essentially robust to assortative mating. What assortment does change is V_A itself: correlated mates build positive gametic- disequilibrium between like-signed loci, so V_A, and therefore h², is genuinely larger than the base-population value. The slope reports that inflated current h² faithfully. The estimators it biases are the ones whose numerator has nothing to cancel against — the single-parent slope becomes ½h²(1 + ρ), and sib correlations rise because the parents' breeding values now covary.

Sib correlations. Full sibs share ½ of their genome on average and both alleles at a quarter of loci; half sibs share ¼ and never both.

RelationshipExpected covarianceSolve for h²
Parent–offspring½V_A + ¼V_AAh² = 2 × (single-parent slope)
Midparent–offspring½V_A + ¼V_AA (over ½V_P)h² = slope
Full sibs½V_A + ¼V_D + ¼V_AA + V_commonh² ≤ 2 t_FS
Half sibs¼V_A + ¹⁄₁₆V_AA + (maternal effects)h² = 4 t_HS
Monozygotic twinsV_A + V_D + V_I + V_commonH² ≤ r_MZ

The half-sib design is the workhorse in animal breeding precisely because ¼V_A carries no dominance at all, only ¹⁄₁₆ of V_AA, and no shared rearing environment — half sibs by a common sire are raised by different dams. The full-sib estimate is an upper bound, not an estimate.

3. Twins, Falconer's formula, and the assumption underneath

Monozygotic twins share their whole genome; dizygotic twins are ordinary full sibs. If both kinds share family environment to the same degree c²:

r_MZ = h² + d² + c²                 (writing d² = V_D/V_P)
r_DZ = ½h² + ¼d² + c²

r_MZ − r_DZ = ½h² + ¾d²

so

h² = 2(r_MZ − r_DZ) − 1.5 d²   →   ĥ² = 2(r_MZ − r_DZ)   only when V_D = 0

Falconer's formula is exact only in the absence of dominance and , and is biased upward otherwise. Non-additive variance inflates r_MZ far more than r_DZ, and the formula charges the whole difference to additivity. Under the same model, c² = 2r_DZ − r_MZ. A negative c² estimate is the diagnostic: it means the model is wrong, usually because of non-additive variance or because the equal-environments assumption has failed.

Worked numerically: adult height in a European twin sample, r_MZ = 0.85, r_DZ = 0.50.

ĥ² = 2(0.85 − 0.50) = 0.70
ĉ² = 2(0.50) − 0.85 = 0.15
check:  ĥ² + ĉ² = 0.85 = r_MZ  ✓

Now suppose r_DZ had been 0.35. Then ĥ² = 2(0.50) = 1.00 and ĉ² = 0.70 − 0.85 = −0.15. The negative shared-environment term tells you the additive model has failed before you report anything.

A binary trait needs one more step. Everything above correlates measured values, and a disease gives you nothing to correlate but a 0/1 indicator — so twin studies of disease work on the latent scale instead: MZ and DZ concordance rates are turned into tetrachoric correlations (the correlation between two standard normals whose thresholded joint distribution reproduces the observed 2×2 concordance table), Falconer's formula is applied to those, and what comes out is a on the liability scale (Ch 30 §2). That distinction is load-bearing rather than pedantic: a heritability on the observed 0/1 scale depends on both the prevalence and the case fraction the study happened to sample, so two estimates for the same disease are not comparable until both have been converted — Ch 30 §2 gives the transformation.

The equal-environments assumption (EEA). The design rests entirely on: MZ and DZ pairs are equally correlated for trait-relevant environments. The critiques are real, and not symmetric.

CritiqueForce
MZ twins are treated more alike — dressed alike, share friends, spend more time togetherReal, and documented. Damages EEA for social and behavioural traits
MZ twins share a chorion about two-thirds of the time; DZ never doReal, and prenatal, so it applies even to physical traits
Twins are not a random sample of the populationRestricts generalisation, does not by itself bias h² within twins
Misclassified zygosityBiases r_MZ and r_DZ toward each other, so deflates h²
Assortative mating on the trait inflates r_DZ but not r_MZBiases Falconer's h² downward — a counterweight to the EEA bias

The defensible position: twin heritabilities for physical and physiological traits are roughly credible; for behavioural and social traits they carry an upward bias of unknown size. "Twin studies are worthless" and "twin studies settle it" are both wrong.

4. Estimating h² without using relatives at all

The move that broke the deadlock: stop using expected relatedness from a and use realised relatedness measured from , among people who are not relatives. Two strangers share, by chance, a genome fraction fluctuating around zero. The fluctuation is tiny, but measurable — and crucially it is uncorrelated with shared environment, because two people who happen to share 0.5% more genome than average do not therefore share a household.

GREML / GCTA. Build a genomic relationship matrix from M SNPs:

Â_jk = (1/M) Σ_i  (x_ij − 2p_i)(x_ik − 2p_i) / [2 p_i (1 − p_i)]

with x the 0/1/2 count and p_i the — a standardised genotype matrix times its transpose, the correlation-of-individuals dual to the LD matrix of Ch 29. Then fit the mixed model

y = Xb + g + e,     g ~ N(0, Â σ²_g),     e ~ N(0, I σ²_e)

by REML and report h²_SNP = σ²_g / (σ²_g + σ²_e). Pairs with  above ~0.025 are dropped, precisely to kill any relatedness close enough to carry shared environment.

Statistics: the mixed model — a random effect whose covariance is a relatedness matrix, and the variance components σ²_g and σ²_e it estimates — is covered in S7 §8.

What this estimates is not h². It is the variance tagged by the genotyped SNPs — the additive variance of causal variants in LD with the array. Rare variants, and variants in low-LD regions, are invisible to it. h²_SNP ≤ h² by construction.

LD score regression gets the same quantity from summary statistics alone. If per-SNP effects are drawn independently, a SNP tagging more neighbours has a proportionally higher expected test statistic:

E[χ²_j] = (N h²_SNP / M) ℓ_j + N a + 1        where  ℓ_j = Σ_k r²_jk

Statistics: χ²_j is the squared standardised effect estimate at SNP j, and a 1-df chi-square has mean 1 under the null — S2 §4. Everything above 1 in that equation is signal or inflation.

Regress χ² on the LD score ℓ. The slope gives h²_SNP; the intercept gives the inflation from and cryptic relatedness — which is the reason LDSC is used far more often as a confounding diagnostic than as a estimator.

GREML / GCTALD score regression
InputIndividual genotypes summary statistics
CostO(N²) matrix, heavyMinutes
Main assumptionEffect size independent of MAF and LDPer-SNP h² independent of LD score
Fails whenStratification not fully controlledSelection makes h² depend on LD (fixed by stratified LDSC)
PrecisionHigher for a given NLower, but scales to any GWAS

5. Missing heritability

Line up the estimates for adult height — the best-measured complex trait there is.

Estimatorh² for heightWhat it captures
Twin / ~0.8All additive variance, plus any EEA violation and non-additive leakage
Common-SNP GREML~0.45Additive variance tagged by array SNPs
Genome-wide-significant SNPs~0.40 (European ancestry, 12,111 SNPs from 5.4M people)Variants that clear p < 5×10⁻⁸
Whole-genome-sequence GREML~0.68Adds rare and low-LD variants

The "" problem of 2009 was the gap between the first row and what genome-wide-significant SNPs explained at the time — about 5% of the variance, from the ~45 loci then known. The third row shows where that same estimator has got to by 2022, and the fourth is a large part of the resolution rather than the problem. The gap has largely closed, and it closed from both ends:

Rare and low-LD variants. WGS-based estimates recover most of the gap: rare variants, especially protein-altering ones in low-LD regions, carry that arrays cannot tag. That pattern is itself a signature of negative selection — variants with larger effects are held at lower frequency (Ch 27).

Pedigree estimates were too high. Shared environment, assortative mating, and non-additive variance all inflate the twin numerator. The true additive h² for height is probably nearer 0.7 than 0.8.

Most causal variants are below the detection threshold. The gap between h²_SNP and -explained variance is not missing at all — it is a power problem, and it closes monotonically with sample size. Height went from 45 loci to 12,111 by adding people, not by adding a new kind of variant. (Ch 51.)

6. The breeder's equation

Select parents whose mean exceeds the population mean by the selection differential S. What is the offspring mean?

Regress on phenotype. The slope is Cov(A,P)/V_P = V_A/V_P = h², so

E[A | P] = h² (P − μ)

The selected parents have mean phenotype μ + S, hence mean breeding value h²S. Offspring get half of each parent's breeding value, so with both sexes selected equally:

R = ½(h²S) + ½(h²S) = h² S

That is it. The is the offspring-on-midparent regression, applied forward. It is a prediction of a response to an intervention — which is why h², alone among 's uses, is genuinely causal here: you are physically choosing who breeds.

Standardise by expressing S in phenotypic standard deviations. For truncation selection keeping the top fraction p, with x the standard normal quantile and φ the standard normal density, the selection intensity is

i = φ(x) / p        and        S = i σ_P        so        R = i h² σ_P = i h σ_A

The last form is the useful one: response per generation scales with the square root of heritability times the additive standard deviation.

Realised heritability runs the equation backwards. After t generations, h²_realised = (Σ R) / (Σ S) — the slope of cumulative response on cumulative selection differential. Divergence between predicted and realised h² is the signal that something in the model has moved.

The multivariate extension. Traits are correlated, and selecting on one drags others. Lande's equation replaces the scalars with matrices:

Δz̄ = G P⁻¹ s = G β

G and P are the additive-genetic and phenotypic variance–covariance matrices, s the vector of selection differentials, and β = P⁻¹s is precisely a vector of partial regression coefficients of relative on the traits — direct selection on each trait holding the others constant — multiple regression in the sense of S5 §6. The genetics is entirely in G. If G has a near-null direction, selection along it produces almost no response no matter how hard you push: populations evolve most easily along the leading eigenvector of G, and that , not the strength of selection, is often what limits response.

7. Why response stops

The Illinois long-term selection experiment has selected maize kernels for high and low oil and protein since 1896 — one generation a year, past 100 generations. High oil rose from about 4.7% to roughly 20%, high protein to roughly 27–32%. Both low lines hit hard limits: you cannot select oil content below zero. The high lines were still responding after a century, far beyond what the starting additive variance could have supported.

Four things end a response, and they are separable:

CauseMechanismDiagnostic
Exhaustion of V_AFavourable fixed; V_A → 0Response decays smoothly; reverse selection also fails
With small N_e, alleles fix at random before selection can act. Robertson's limit: total advance ≈ 2N_e × R₁Replicate lines plateau at different means
Opposing natural selectionExtreme are sterile or inviable; declines until it cancels artificial selectionResponse stops while V_A remains; relaxing selection causes regression
Physiological limitThe trait cannot go further (oil ≥ 0)Asymmetric — one direction stops, the other doesn't

Against those, new supplies fresh additive variance every generation — of order 10⁻³ V_E per generation — which is why lines can keep creeping upward long after the founding variation is spent. The Illinois high lines are the standing demonstration.

Genetic gain versus natural selection. The equation is the same; the control surface is not.

BreederNature
SChosen. Truncate at any pDetermined by the fitness function, usually weak
TargetFixed by the breeder, held for decadesMoves; often stabilising rather than directional
AccuracyImprovable — progeny testing, or genomic prediction from a training set only
Generation interval LShortenable. Rate of gain is i·r·σ_A / LFixed by biology
Fitness costIgnored until bitesIs the criterion

Genomic selection is the modern lever: predict from markers (Ch 53), select before the animal or plant is ever phenotyped, and collapse L. The gain comes from the denominator.


Common misconceptions

What people thinkWhat's actually true
measures how genetic a trait isIt measures what fraction of variance in one population tracks . Number of fingers is almost perfectly genetically determined and has heritability near zero, because nearly all the variance in fingers is accidents
A trait has a heritabilityA trait has a heritability in a population, in an environment, at a time. Equalise the environment and h² rises; diversify it and h² falls. The genotypes need not change at all
h² = 0.5 means half of my height came from my genesHeritability is not a partition of an individual's trait value. Your height is not 50% genes and 50% food; it is 100% both. Variance decomposes; individuals do not
High heritability means the trait can't be changedIt says nothing about malleability. See below — this is the big one
Heritability tells you the cause of a group differenceIt carries no information whatsoever about between-group differences. See the two-pots argument below
A heritability estimate is a fixed constantChange V_E and h² = V_A/(V_A+V_E) changes with no genetic change. Reported heritabilities differ across countries and decades for exactly this reason
Heritability = 0 means genes are irrelevantIt means genes don't explain variation here. A universally fixed contributes nothing to variance and everything to the
SNP heritability and twin heritability are the same number measured two waysThey are different estimands. h²_SNP is the variance tagged by common SNPs; twin h² is total additive variance plus several biases. They are not supposed to agree

High heritability does not mean unchangeable

The single most consequential error. Two counterexamples, both decisive.

Height. Heritability of adult height in developed populations is around 0.7–0.8. Dutch mean male height rose roughly 20 cm between the mid-nineteenth century and now, driven by nutrition, sanitation and disease burden. Heritability stayed high throughout, because it measures the ranking within a cohort, and the ranking is genetically stable even while the whole distribution translates upward by more than two standard deviations. A high h² constrains nothing about the mean.

Phenylketonuria. PKU is caused by variants in PAH, is essentially fully genetic, and untreated causes severe irreversible intellectual disability. It is also completely preventable by a low-phenylalanine diet started in infancy — which is why every newborn in the developed world is screened for it. Heritability near 1; environmental intervention near total. The intervention did not exist in the environments over which heritability was estimated, and heritability could not have told you it would work.

The general principle: heritability is estimated over the range of environments that happened to be present. It is silent about environments outside that range, and interventions are precisely attempts to move outside that range.

The two pots: heritability says nothing about group differences

Lewontin's thought experiment. Take a genetically variable batch of seed. Split it at random into two pots.

POT A — full nutrient soil          POT B — depleted soil
  ██████████ 30 cm                    ████ 12 cm
  ████████   26 cm                    ███  10 cm
  ███████████ 33 cm                   █████ 14 cm
  ████████   27 cm                    ████ 12 cm
  mean 29 cm                          mean 12 cm

WITHIN each pot: environment is uniform, so essentially
all variance is genetic  →  h² ≈ 1 in both pots.

BETWEEN pots: the 17 cm difference is 100% environmental,
by construction — the seed was randomised.

Heritability within each group is as high as it can be. The difference between the groups is entirely environmental. Both statements are true simultaneously, and this is not a special case or a contrived edge — it is the general situation, because heritability is computed from within-group variance and contains no information about between-group means.

Add the reverse case and the point is airtight: run both pots on identical soil, using seed from two genetically distinct sources, and you get zero within-pot heritability differences explaining a between-pot gap that is entirely genetic. The within-group statistic is compatible with any between-group cause. No amount of within-population heritability licenses any inference about the source of a difference between populations — which are, in the human case, never randomised into their environments. Ch 58 takes the social history of this error seriously.

Worked example

A crop breeder measures seed oil content in a large random-mating population.

Step 1 — and its components.

h²  = b_O·MP = 0.45
V_A = h² V_P = 0.45 × 4.00 = 1.80 %²
V_resid (dominance + epistatic + environmental) = 4.00 − 1.80 = 2.20 %²

Step 2 — selection intensity for the top 10%.

Truncation point: x = Φ⁻¹(0.90) = 1.2816.

φ(1.2816) = (1/√(2π)) e^(−1.2816²/2)
          = 0.39894 × e^(−0.82125)
          = 0.39894 × 0.43989 = 0.17549

i = φ(x)/p = 0.17549 / 0.10 = 1.755

Step 3 — first-generation response.

S = i σ_P = 1.755 × 2.00 = 3.510 %      (selected parents mean 8.00 + 3.51 = 11.51%)
R = h² S  = 0.45 × 3.510 = 1.580 %
new mean  = 8.00 + 1.58 = 9.58 %

Step 4 — why generation 2 will disappoint. Truncation does not just shift the mean, it shrinks the variance. For truncation at x with intensity i, the variance-reduction coefficient is

k = i(i − x) = 1.755 × (1.755 − 1.2816) = 1.755 × 0.4734 = 0.831

so the selected parents' phenotypic variance is V_P(1 − k) = 4.00 × 0.169 = 0.68 %². Selection generates negative between loci with like-signed effects — the Bulmer effect. and recombination restore half of it each generation, so:

V_A(1) = V_A (1 − ½ k h²)
       = 1.80 × (1 − 0.5 × 0.831 × 0.45)
       = 1.80 × (1 − 0.1870) = 1.80 × 0.8130 = 1.463 %²

V_P(1) = 1.463 + 2.20 = 3.663 %²        σ_P(1) = √3.663 = 1.914 %
h²(1)  = 1.463 / 3.663 = 0.399

Step 5 — corrected second-generation response.

R(2) = i h²(1) σ_P(1) = 1.755 × 0.399 × 1.914 = 1.340 %

against 1.580% if you had naively reapplied the original h² — a 15% shortfall in one generation. Five generations of naive extrapolation predicts 8.00 + 5(1.58) = 15.90%; the real trajectory falls well short of it, and would fall further once frequencies begin to move and V_A erodes for real rather than by disequilibrium alone.

Step 6 — audit. After five generations the breeder observes a cumulative response of 6.30% against a cumulative selection differential of 16.20%:

h²_realised = 6.30 / 16.20 = 0.389

Lower than the estimate of 0.45, exactly as the Bulmer calculation predicted. Realised heritability is the number to trust, because it is the only one measured on the intervention you actually performed.

Connections