Reality Bending Lab › News › Lighthouse plots: shining a light on differences
Lighthouse plots: shining a light on differences
Jeffrey Girard, the newest member of the easystats team (and the fastest issue-tackler), has just spent some time improving one of our most underrated features: lighthouse plots. If you've never heard of them, that is not really surprising: they originated in one of our GitHub issues, a long time ago, when I was thinking of how to best convey condition differences in within-participant designs, but that idea never saw the light it deserved.
A puzzle
Here are two studies. In each, 30 participants got two scores: once in condition A, once in condition B. For each condition, the figure shows the distribution of the scores, and the mean with its 95% confidence interval.
Is there a difference between A and B? The intervals overlap a lot: the mean of B even falls inside the interval of A. The tempting reading, and a very common one (researchers do it routinely when reading error bars; Belia et al., 2005), is that the difference is probably not significant, in either study. After all, the two figures look the same.
But they are not the same. In Study 1, the difference between A and B is about as conclusive as statistics gets (t(29) = 13.5, p < .001). In Study 2, there is no evidence for it at all (t(29) = 1.0, p = .33).
The reveal
The difference between the two studies is not in the conditions. It is in the participants. Join each participant's two scores with a line, and the two studies stop looking alike:
In Study 1, every single participant scores higher in B than in A. By a little (about three points), but all 30 of them, all in the same direction. In Study 2, the lines go everywhere: 19 participants go up, 11 go down, some of them by twenty points or more.
Now the twist: the two figures do not just look the same, they are the same. Both studies contain exactly the same 30 scores in A, and exactly the same 30 scores in B. The only thing I changed for Study 2 is who got which score in B: I shuffled them across participants. So the means, the standard deviations, the distributions and the confidence intervals of each condition are identical to the last decimal. So is the average difference between A and B (2.88 points). What differs is how consistent that difference is from one participant to the next.
As I often repeated in my career: the uncertainty around each estimate tells us little about the uncertainty around the difference between them. The interval around the mean of A answers the question where is A? Whether A and B differ is another question, and its answer depends on something a plot of the two conditions cannot show: how the two sets of scores relate to each other. In a repeated-measures design like this one, the overlap of the intervals tells you nothing. Even with two independent groups, where it is at least related to the answer, the rule is not "overlap means non-significant": two 95% intervals can overlap by as much as half an arm and the difference still be significant (Cumming & Finch, 2005).
Joining the dots works for two conditions and a few dozen participants. It stops working as soon as the means come from a model: adjusted for covariates, averaged over other factors, estimated within each level of a moderator. What we need then is a way to draw the uncertainty of the difference itself, right next to the means it is the difference of.
Enter the lighthouse
In April 2019, I opened an issue on see about exactly this problem. Russ Lenth, the author of emmeans, had laid out the existing ways of plotting pairwise comparisons, and none of them was satisfying: diagonal "diffograms" that are hard to label, comparison arrows that cannot always be drawn, and letter displays (means that share a letter do not differ) that reduce everything to a yes or a no.
The idea was to keep the usual plot of the means with their intervals, and to add, for each pair of conditions, a lighthouse. It stands on one mean and casts a beam of light towards the other. When the beam arrives at the other condition, its width is the confidence interval of the difference, measured from the lighthouse's own height. So the beam always lands centred on the other mean, and its spread says how sure we are about where that mean is relative to the lighthouse, rather than where it is in absolute terms.
Reading one comes down to the two edges of the beam. If both edges tilt the same way (both rising, or both falling), the difference is significant: every plausible value of the difference has the same sign. If one edge rises while the other falls, the beam straddles the horizontal, and a difference of zero is plausible. This is no accident of drawing: the tilt of each edge is set by one bound of the confidence interval of the difference.
In Study 1, the beam is a needle: both of its edges rise from A to B, because every participant agreed on the direction of the difference. In Study 2, it is a floodlight: one edge rises from A to B, the other falls. The intervals around the means are the same in both panels, as they should be, since they are the same means. Each interval now does one job, and neither pretends to answer the other's question.
Making one
A lighthouse plot is what see draws when you plot contrasts from modelbased together with the means they compare. Any model that modelbased supports will do. Here, a mixed model with a random intercept for each participant:
library(lme4)
library(modelbased)
library(see)
model <- lmer(Score ~ Condition + (1 | Participant), data = data)
means <- estimate_means(model, by = "Condition")
contrasts <- estimate_contrasts(model, contrast = "Condition")
plot(contrasts, means)Thank you, Jeffrey, for reviving and robustifying lighthouse plots. Now is a good time to try them, and to tell us if they need more fixing.
