R | Reality Bending Lab

How to Assess Task Reliability using Bayesian Mixed Models

Mon, 18 Mar 2024 00:00:00 +0000

Using reliable tasks when assessing inter-individual differences is a key issue for differential psychology and neuropsychology, and many research areas are clouded with mixed evidence stemming out of the suboptimal computation of individual scores (e.g., tasks with not enough trials, scores consisting in computing the difference, aka the contrast between two conditions; see Rouder et al., 2024). As such, measuring and reporting the reliability of the paradigms used could be an important step for increasing results replicability.

Recently, a new approach has emerged, suggesting to assess task sensitivity to inter-individual differences by leveraging mixed models (Rouder et al., 2024). In essence, the idea is to fit a statistical model that tests for the general population level effect of a manipulation in a given task/experiment (e.g., the impact of a variable Difficulty on another variable RT), and incorporates a random effect for each participant. This “full” mixed model essentially models the general population level by taking into account all the inter-individual effects and - as a side effect - estimates the effects of interest for each participant separately.

When fitting these models under a Bayesian framework, one can easily estimate the “variability” (or certainty) of the effect in each participant. This is great, because it allows us to assess a “signal-to-noise” ratio, an index of how much the interindividual variability (how participants vary) is larger than the intraindividual variability (e.g., how much participants vary across trial, or how precisely participants’ effects are estimated).

In this “Signal-To-Noise Ratio as Effect Reliability” framework, an ideal task/manipulation would have a strong inter-individual variability (i.e., participants would on average vary a lot) and a low intra-individual variability (each participant would have very consistent effects), which leads to a reliable measure of interindividual effects.

Let’s see how we can do that in R using the brms package for fitting Bayesian mixed model. First, let’s start to generate 4 datasets with different levels of inter-individual and intra-individual variability.

Show code

library(easystats)
library(tidyverse)
library(brms)
library(patchwork)

# Make function to generate data
generate_data <- function(n_trials=25, effect_sd = 0.4, intercept_sd=0.4, noise=0.8, name="df") {
  df <- data.frame()
  for(participant in 1:20) {
    x <- rnorm(n_trials, 0, 1)
    y <- (1 + rnorm(1, 0, effect_sd)) * x + rnorm(1, 0, intercept_sd)
    y <- y + rnorm(n_trials, 0, noise)
    df <- rbind(df, data.frame(Difficulty=x, RT=y,
                               Participant=paste0("S", participant)))
  }
  df$Name <- name
  df
}

# Generate 4 datasets
df1 <- generate_data(n_trials=20, effect_sd = 0.5, intercept_sd=0.5, name="1. Intercept and Effect")
df2 <- generate_data(n_trials=20, effect_sd = 0.1, intercept_sd=0.5, name="2. Intercept Only")
df3 <- generate_data(n_trials=20, effect_sd = 0.5, intercept_sd=0.1, name="3. Effect Only")
df4 <- generate_data(n_trials=200, effect_sd = 0.5, intercept_sd=0.5, name="4. More trials")

# Plot data
rbind(df1, df2, df3, df4) |>
  ggplot(aes(x=Difficulty, y=RT, color=Participant, fill=Participant)) +
  geom_point2(alpha=0.5) +
  geom_smooth(method="lm", se=TRUE, alpha=0.2) +
  theme_minimal() +
  scale_fill_material_d() +
  scale_color_material_d() +
  facet_wrap(~Name, scales="free")

In each of the dataset, we simulated the data of 20 participants undergoing a task with n trials varying in difficulty, and we recorded their reaction time (RT). Note that while in our example difficulty is a continuous variable, it would work the same if it was categorical variable (e.g., effect of condition B over A, intervention vs. baseline, incongruent vs. congruent, etc.).

When we fit a linear regression of the form RT ~ difficulty, we are estimating two parameters; the intercept (which can be seen as the “baseline” RT, i.e., participants’ baseline processing speed when the difficulty is 0) and the slope (how much participants are impacted by this variable). These two parameters are in principle independent (a participant can be very fast regardless of the difficulty, and another one could be equally fast at baseline - same intercept - but very slow when the task is difficult - strong slope).

We simulated 4 datasets with different participant characteristics:

Dataset 1: Both the RT intercept (the “baseline” RT) and the effect of the manipulation (the effect of difficulty) vary across participants.
Dataset 2: Not much interindividual variability in the effect (only the baseline RT varies).
Dataset 3: Not much interindividual variability in the baseline RT (only the effect of difficulty varies from participant to participant).
Dataset 4: Same as dataset 1, but with more trials (200 instead of 20). As you can see, the “precision” ribbon around the regression line is much narrower, indicating that the effect is more precisely estimated.

We expect that reliability of the paradigm to measure 1) the sensitivity to difficulty and 2) the baseline RT will be higher in dataset 4 (because more trials) than in dataset 1. Moreover, the sensitivity to difficulty will be particularly low in dataset 2 (where only the baseline RT is set to varies), and similarly for baseline RT in dataset 3 mutatis mutandis.

Now, let’s fit a Bayesian linear mixed model to each of these datasets (note that we specify the effect of Difficulty as a random slope in addition to estimating the random intercept).

Show code

model1 <- brms::brm(RT ~ Difficulty + (1+Difficulty|Participant), data=df1, iter=600)
model2 <- brms::brm(RT ~ Difficulty + (1+Difficulty|Participant), data=df2, iter=600)
model3 <- brms::brm(RT ~ Difficulty + (1+Difficulty|Participant), data=df3, iter=600)
model4 <- brms::brm(RT ~ Difficulty + (1+Difficulty|Participant), data=df4, iter=600)

This model basically computes the overall relationship (Intercept + Slope) between difficulty and RT, as well as for each participant. We can then extract the posterior distribution of these individual effects (i.e., the value of the Intercept and the Slope for each participant).

Show code

# Random effects extraction
extract_individual <- function(model, name="df") {
  coefs <- coef(model, summary=FALSE)$Participant
  data <- rbind(
    as.data.frame(coefs[, , "Intercept"]) |>
      pivot_longer(everything(), names_to="Participant", values_to="Value") |>
      mutate(Parameter="Intercept", Name=name),
    as.data.frame(coefs[, , "Difficulty"]) |>
      pivot_longer(everything(), names_to="Participant", values_to="Value") |>
      mutate(Parameter="Difficulty", Name=name)
  )
  data
}

re1 <- extract_individual(model1, "1. Intercept and Effect")
re2 <- extract_individual(model2, "2. Intercept Only")
re3 <- extract_individual(model3, "3. Effect Only")
re4 <- extract_individual(model4, "4. More trials")


# Plot Random effects
rbind(re1, re2, re3, re4) |>
  ggplot(aes(x=Value, y=Participant, fill=Participant)) +
  ggdist::stat_slabinterval(adjust=2, linewidth=0.5, size=0.5) +
  scale_fill_material_d() +
  theme_minimal() +
  facet_grid(Name~Parameter, scales="free")

Each participant’s “score” (for the baseline RT score, i.e., the intercept; and the effect of difficulty, i.e., the slope) is represented by a distribution. This distribution is wider when there is less trials, which can be interpreted as more uncertainty about the exact estimate. Some datasets have a low interindividual variability for some parameters (e.g., dataset 2 has not much interindividual variability in the effect of difficulty).

We can now compute, for each participant, the “mean” of its effects (for the intercept and the slope), as well as its own effect SD (intra-individual variability).

Show code

scores <- rbind(re1, re2, re3, re4) |>
  summarize(
    Mean = mean(Value),
    SD = sd(Value),
    .by = c("Name", "Parameter", "Participant")
  )
head(scores)

Name	Parameter	Participant	Mean	SD
1. Intercept and Effect	Intercept	S1	0.37	0.20
1. Intercept and Effect	Intercept	S10	-1.05	0.20
1. Intercept and Effect	Intercept	S11	0.88	0.19
1. Intercept and Effect	Intercept	S12	-0.30	0.18
1. Intercept and Effect	Intercept	S13	0.16	0.19
1. Intercept and Effect	Intercept	S14	0.57	0.19

Finally, we can compute the Signal-to-Noise Ratio for each parameter for each dataset, which is the ratio of the interindividual variability (the SD of the individual mean scores) over the average intraindividual variability (the average of the individual SDs).

Show code

summarize(scores,
          SNR = sd(Mean) / mean(SD),
          .by=c("Name", "Parameter"))

Name	Parameter	SNR
1. Intercept and Effect	Intercept	2.87
1. Intercept and Effect	Difficulty	3.00
2. Intercept Only	Intercept	2.57
2. Intercept Only	Difficulty	0.55
3. Effect Only	Intercept	0.88
3. Effect Only	Difficulty	2.59
4. More trials	Intercept	8.88
4. More trials	Difficulty	7.97

As predicted, the “reliability” of the paradigm to measure the interindividual effect of difficulty on RT is low in dataset 2 (where only the baseline RT varies), moderate in dataset 1 and 3, and high in dataset 4 where there are more trials.

Podcast 'Learn Bayesian Stats' with Dominique Makowski

Tue, 01 Feb 2022 00:00:00 +0000

The ‘Learning Bayesian Statistics’ Podcast

I had the chance of being invited to talk about R, Python, Reality Bending and much more! It was my first experience of that kind, so thanks a ton to the host of the podcast Alex Andorra. Listen to it here:

https://www.learnbayesstats.com/episode/55-neuropsychology-illusions-bending-reality-dominique-makowski

"Your sample is too small": How to Answer to Reviewers

Fri, 05 Nov 2021 00:00:00 +0000

“Your sample is too small”: How to Answer to Reviewers

Reviewers questioning the statistical power of a study, while often valid, is common and unoriginal. The problem of possible lack of power, and thus possible false positives, can also be combined with the desire for stringent multiple comparisons / tests “corrections”.

In many real-life scenarios, we simply cannot increase the sample size (because of money, time, COVID or any other reason). And sometimes, the many tests / comparisons are all needed as we have many hypotheses or variables. Naturally, the best pre-study weapon against power-related issues are preregistration (and registered-reports) with proper power-analysis.

That being said, imagine that for one reason or another we don’t have that. We cannot recruit more participants and have to deal with the data that what we have. What can we do?

Some useful tips can be grouped and referred to as PoSCA (Power, Stringency, Coherence, Acknowledgment):

Power: Don’t waste power
- Use all the information that you have. In particular, the information within participants (if you have multiple trials per participant, don’t average over them!), by using methods such as mixed-models.
Stringent significance
- Increase the significance threshold, either arbitrarily (e.g., lower the p-value threshold to .005; Ioannidis, 2018) or pseudo-arbitrarily (using multiple-comparisons corrections such as the Bonferroni method). This, under the Bayesian framework, is _(somewhat) equivalent to being more script with whatever indices that we use. For instance, consider only Bayes Factors (BF) > 30 instead of 10, or Probabilities of Direction (pd) of 99.9% instead of 97%, or widening the Region of Practical Equivalence (ROPE).
- Constrain the parameters towards 0. We can also make it harder for models to have parameters (i.e., “effects”) that deviate from 0. This is easy to do within the Bayesian framework by setting narrower priors centred around 0. This will naturally pull (known as “shrinkage”) the results towards 0 and is a natural way of “controlling” for false positives. Under the frequentist framework, there are methods of regularization that can introduce a bias, such as LASSO or Ridge regressions.
Coherence: cross-validate results
- There is a need to look at the bigger picture, including non-significant and “trending” effects. Look at all the data together, and all the variables measuring related concepts: do the results go in the same direction? Are they internally coherent? Do they follow the same trend? In a small sample study, I would trust more a conclusion based on 10 tests with independent indices measuring aspects related to “depression”, all barely significant but going in the same direction, over one super-significant (but possibly cherry-picked) result interpreted as a proof of the effect of “depression” in general. In other words, don’t focus on “significant” effects and assess results in their entirety.
Acknowledgment: be honest and humble
- Quantify the evidence against the null hypothesis. Maybe an effect is not “significant”, but that doesn’t necessarily mean there is a lot of evidence that it has no effect either. Quantifying evidence against the null can be easily done using for instance Bayes Factors.
- Be honest and acknowledge the limited power of the study. Don’t try to hide it.
- Temper the interpretation of the results. Don’t over-interpret them if they are not super robust.

Science must be open, replicable & reproductible, slow, and based on methodological best-practices. But first and foremost, it must be honest.

References

Ioannidis, J. P. (2018). The proposal to lower P value thresholds to .005. Jama, 319(14), 1429-1430.

Thanks for reading! Do not hesitate to share or tweet this post, or leave a comment below 🤗

How to sync two folders in two separate GitHub repositories

Sun, 31 Oct 2021 00:00:00 +0000

How to sync two folders in two separate GitHub repositories

The Problem

I have a personal website, stored in a GitHub repo (and hosted via GitHub pages), as well as a lab website (a “company” website, if you will). Both are fairly similar, as they are built using Wowchemy’s academic theme. Importantly, there is a blog in the company websites with posts, but I have one on my personal website too. What I would like is that every time I post something on my website, it gets automatically copied over to the company website. So that I don’t have to manually maintain the content at two separate places.

The Solution

The first step is to go to the settings of your GitHub account, to developers settings, and to personal access tokens. You have to generate a token, and tick the repo authorizations. Copy-paste the key.
Go to the settings of the personal website repo (the source from which the content will be copied), to “Secrets”, and add a new secret called “API_TOKEN_GITHUB” (with the key you just copied).
Create a new GitHub action workflow such as this one. The things to change are the source_file, destination_repo and destination_folder.

Tada 🎉 Everytime I push to my personal repo, the new content of one of the subfolder gets copied to another repo.

Notes

This is a one-way sync, so updates on the target repo won’t affect the source repo (but might get overridden!).
if you want to preserve the original commit message, set commit_message: ${{ github.event.head_commit.message }} _{(thanks @dobbelina)}

Thanks for reading! Do not hesitate to share or tweet this post, or leave a comment below 🤗

How to share data analysis scripts with publications?

Wed, 10 Feb 2021 00:00:00 +0000

Including data analysis as Supplementary Materials can be a tedious task. How can we simplify the sharing of our work? So that it can be fully appreciated, as well as evaluated, improved, worked on, in a transparent and open way?

Option 1: Dump the code in a word file

Most publication portals don’t directly accept code scripts to be included “as is”. In other words, you cannot upload your manuscript and your .R script just like that. So one option is to copy its content, paste it in a word / pdf document, and Voilà!

But what if you don’t have code because you use a point-and-click software? Well, you can note down all the x,y coordinates of your clicks so that one can reproduce the steps and all the clicks. Just kidding, if you don’t have a script, then may god have mercy on your soul.

Why is that a terrible solution? Because let’s face it, unstructured code dumps are horrific. Nobody wants to read it, it does not make justice to your work, and it’s still tedious to create! You have to re-do it everytime you modify your code. And it’s even worse if you want to include all the outputs, figures, tables that are generated by the code? Data analysis is not just the code, but everything that comes with it and that allowed you to make the conclusions that you made.

Option 2: Use Rmarkdown

Rmarkdown is a “framework” that allows you to have files (.Rmd) that can contain a mix of text and code (and not only R, but also Python for instance!).

It can be used to write comprehensive “reports” that include all your thoughts, motivations and interpretations of the results. And the great thing about it is that these files can be converted into beautiful and readable documents like PDF, Word or HTML. It will automatically embed the code and its generated output (as text, tables or figures) alongside the text.

It is an awesome way to write statistical reports, and can even be used to create many other non-stats related stuff, like APA-formatted manuscripts (great for preprints), books, presentation slides, websites or blogs. It’s a must-have skill for every researcher.

And it gets better!

Option 3: Use our Results Template

In the Reality Bending team, we like to have our different projects and studies organized in a consistent way. We heavily use GitHub to store our projects and collaborate on them, and we also like the possibility of making these projects open and accessible (i.e., easy to discover and explore) when the time comes.

That’s why we came up with a Template folder for storing the materials related to a given study, including well-organized analysis scripts. And what’s great about it is that it is setup in a way that allows you to generate multiple files format (word, pdf, html) with a single click (and even without any click, in a fully automatic way)! And what’s even greater is that if you decide to upload it to GitHub, you’ll have a whole website presenting your data analysis!

We use it to have reproducible analysis that we can easily share with publications. We can upload the .pdf or .docx file generated by the template as Supplementary Materials, but we also link the URL of the online repository of the study in the manuscript, where users can access and experience the content in the format that they prefer. It really improves the appeal of a study when the results are trustworthy.

All this is easily made possible with our template. Check it out here:

👉 https://github.com/RealityBending/TemplateResults 👈

☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️

And let us know what you think! You can open an issue on the repo or even contribute to help us improve it :)

Thanks for reading! Do not hesitate to tweet and share this post, and leave a comment below 🤗

🐦 Don’t forget to join me on X @Dom_Makowski

How to extract individual scores from repeated measures

Mon, 14 Sep 2020 00:00:00 +0000

Introduction

Many psychology fields require to extract individual scores, i.e., point-estimates (i.e., a single value) for a participant/patient, to be used as an index of something and later interpreted or re-used in further statistical analyses. This single index is often derived from several “trials”. For instance, the reaction times in the condition A (let’s say, the baseline) will be averaged together, and the same will be done with the condition B. Finally, the difference between these two means will be used an the individual score for a given participant.

However, we can intuitively feel that we lose a lot of information when averaging these scores. Do we deal appropriately with the variability related to individuals, conditions, or the noise aggravated by potential outliers? This is especially important when working with a limited amount of trials.

With the advent of recent computational advances, new easy-to-implement alternatives emerge. For instance, one can “model” the effects at an individual level (e.g., the simplest case, for the two conditions paradigm described above, would be a linear regression with the condition as a unique predictor), and use the parameters of each model as individual scores (e.g., the “slope” coefficient of the effect of the manipulation), rather than the raw mean. This opens up the possibility of including covariates and take into account other sources of known variability, which could lead to better estimates.

However, individual models are also sensitive to outliers and noise. Thus, another possibility is to model the effects at the population level and, at the same time, at the individual level. This can be achieved by modelling the participants as a random factor in a mixed model. In this case, the individual estimates might benefit from the population estimates. In other words, the effects at the population level will “constrain” or “guide” the estimation at an individual level to potentially limit extreme parameters.

Unfortunately, the above method requires to have all the data at hand, to be able to fit the population model. This is often not the case in on-going acquisition, or in neuropsychological contexts, in which the practitioners simply acquire data for one patient, and have to compute individual scores, without having access to the detailed population data. Thus, an in-between alternative could make use of Bayesian models, in which the population effects (for instance, the mean effect of the condition) could be entered as an informative prior in the individual models to, again, “guide” the estimation at an individual level and hopefully limit the impact of noise or outliers observations.

In this post, the aim is to compare these 4 methods (basic individual model - equivalent to using the raw mean, population model, individual model with informative priors) in recovering the “true” effects using a simulated dataset.

Results

Generate Data

We generate several datasets in which we manipulate the number of participants, in which the score of interest is the effect of a manipulation as compared to a baseline condition. 20 trials per condition will be generated with a known “true” effect (the centre of the distribution from which the data is generated). Gaussian noise of varying standard deviation will be added to create a natural variability (See the functions’ definition below).

library(tidyverse)
library(easystats)

data <- get_data(n_participants=1000, n_trials=20)
results <- get_results(data)

Figure 1: *Example of a dataset containing 20 participants (shown with different colors). As can be seen, we introduced modulations in the inter- and intra- individual variability.*

We will then compare the scores obtained by each method to the “true” score of each participant by substracting them from one another. As such, for each method, we obtain the absolute “distance” from the true score.

Fit model

Contrast analysis will be applied to compare the different methods together.

model <- lm(Diff_Abs ~ Method, data=results)

modelbased::estimate_contrasts(model) %>%
  arrange(Difference) %>%
  mutate(Level1 = stringr::str_remove(Level1, "Diff_"),
         Level2 = stringr::str_remove(Level2, "Diff_")) %>% 
  select(Level1, Level2, Difference, CI_low, CI_high, p)

## Level1                 | Level2                 | Difference |            CI |      p
## -------------------------------------------------------------------------------------
## IndividualModel_Priors | PopulationModel        |  -1.85e-03 | [-0.01, 0.01] | > .999
## IndividualModel_Freq   | PopulationModel        |   1.70e-03 | [-0.01, 0.01] | > .999
## IndividualModel_Freq   | IndividualModel_Priors |   3.55e-03 | [-0.01, 0.01] | > .999

Visualize the results

Figure 2: *Average accuracy of the different methods (the closest to 0 the better).*

Figure 3: *Accuracy depending on the number of total participants in the dataset.*

Conclusion

Though not significantly different, it seems that raw basic estimates (that rely only on the individual data) perform consistently worse than the population model or individual models informed by priors, especially for small datasets (between 10 and 100 participants) - though again, the difference is tiny in our simulated dataset. In the absence of the whole population dataset, it seems that using individual Bayesian model with informative priors (derived from the population model) is a safe alternative.

Functions

library(tidyverse)
library(easystats)
library(rstanarm)
library(ggdist)


# Get data ----------------------------------------------------------------

get_data <- function(n_participants = 10, n_trials = 20, d = 1.5, var = 3, noise = 0.1) {
  scores_baseline <- rnorm(n_participants, 0, 1)
  scores_condition <- rnorm(n_participants, d, 1)
  variances <- rbeta(n_participants, 2, 8)
  variances <- 0.1 + variances * (var / max(variances)) # Rescale
  noise_sd <- abs(rnorm(n_participants, 0, noise))

  data <- data.frame()
  for (i in 1:n_participants) {
    a <- rnorm(n_trials, scores_baseline[i], variances[i])
    b <- rnorm(n_trials, scores_condition[i], variances[i])
    a <- a + rnorm(n_trials, 0, noise_sd[i]) # Add noise
    b <- b + rnorm(n_trials, 0, noise_sd[i]) # Add noise
    data <- rbind(data, data.frame(
      "Participant" = sprintf("S%02d", i),
      "Y" = c(a, b),
      "Score_True" = rep(c(scores_baseline[i], scores_condition[i]), each = n_trials),
      "Condition" = rep(c("Baseline", "Manipulation"), each = n_trials)
    ))
  }
  data
}



# Visualize data -----------------------------------------------------------

p <- get_data(n_participants = 20) %>%
  group_by(Participant, Condition) %>%
  mutate(mean = mean(Y)) %>%
  ggplot(aes(y = Y, x = Condition, fill = Participant, color = Participant, group = Participant)) +
  geom_line(aes(y = mean), position = position_dodge(width = 0.66)) +
  ggdist::stat_eye(point_interval = ggdist::mean_hdi, alpha = 0.66, position = position_dodge(width = 0.66), .width = c(0.95)) +
  ylab("Score") +
  theme_modern() +
  theme(legend.position = "none")
ggsave("individual.png", p, width = 7, height = 7, dpi = 450)


# Get results -------------------------------------------------------------


get_results <- function(data) {

  # Raw method ----

  results <- data %>%
    group_by(Participant, Condition) %>%
    summarise_all(mean) %>%
    rename("Score_Raw" = "Y") %>%
    arrange(Condition, Participant) %>%
    ungroup()


  # Population model ----

  model <- lme4::lmer(Y ~ Condition + (1 + Condition | Participant), data = data)

  fixed <- insight::get_parameters(model, effects = "fixed")$Estimate
  random <- insight::get_parameters(model, effects = "random")$Participant

  # Transform coefs into scores
  pop_baseline <- random[, 1] + fixed[1]
  pop_manipulation <- pop_baseline + random[, 2] + fixed[2]

  results$Score_PopulationModel <- c(pop_baseline, pop_manipulation)


  # Individual model ----

  individual_model_data <- data.frame()
  for (participant in unique(data$Participant)) {
    cat(".") # Print progress

    dat <- data[data$Participant == participant, ]

    # Frequentist
    model1 <- lm(Y ~ Condition, data = dat)
    nopriors <- parameters::parameters(model1)$Coefficient

    # Bayesian without priors
    # model2 <- stan_glm(Y ~ Condition, data = dat, refresh = 0)
    # bayes <- parameters::parameters(model2)$Median

    # Bayesian with Priors
    model3 <- stan_glm(Y ~ Condition,
      data = dat,
      refresh = 0,
      prior = normal(fixed[1]),
      prior_intercept = normal(fixed[2])
    )
    priors <- parameters::parameters(model3)$Median

    individual_model_data <- rbind(
      individual_model_data,
      data.frame(
        "Participant" = c(participant, participant),
        "Condition" = c("Baseline", "Manipulation"),
        "Score_IndividualModel" = c(nopriors[1], nopriors[1] + nopriors[2]),
        # "Score_IndividualModel_Bayes" = c(bayes[1], bayes[1] + bayes[2]),
        "Score_IndividualModel_Priors" = c(priors[1], priors[1] + priors[2])
      )
    )
  }

  results <- merge(results, individual_model_data)


  # Clean output ----

  diff <- results %>%
    mutate(
      # Diff_Raw = Score_True - Score_Raw,
      Diff_PopulationModel = Score_True - Score_PopulationModel,
      Diff_IndividualModel = Score_True - Score_IndividualModel,
      # Diff_IndividualModel_Bayes = Score_True - Score_IndividualModel_Bayes,
      Diff_IndividualModel_Priors = Score_True - Score_IndividualModel_Priors
    ) %>%
    select(Participant, Condition, starts_with("Diff")) %>%
    pivot_longer(starts_with("Diff"), names_to = "Method", values_to = "Diff") %>%
    mutate(Diff_Abs = abs(Diff))

  diff
}



# Analysis ----------------------------------------------------------------
results <- data.frame()
for(n in seq.int(10, 300, length.out = 10)){
  data <- get_data(n_participants = round(n), n_trials = 20)
  rez <- get_results(data) %>%
    select(-Participant) %>%
    group_by(Condition, Method) %>%
    summarise_all(mean) %>%
    mutate(n_Participants = n,
           Method = as.factor(Method),
           Dataset=paste0("Dataset", round(n, 2)))
  results <- rbind(results, rez)

  print(n) # Print progress
}

# model <- mgcv::gam(Diff_Abs ~ Method + s(n_Participants, by = Method), data = results)
model <- lm(Diff_Abs ~ Method * poly(n_Participants, 3), data = results)

parameters::parameters(model)


contrasts <- modelbased::estimate_contrasts(model) %>%
  arrange(Difference) %>%
  mutate(
    Level1 = stringr::str_remove(Level1, "Diff_"),
    Level2 = stringr::str_remove(Level2, "Diff_")
  ) %>%
  select(Level1, Level2, Difference, CI_low, CI_high, p)




# Visualize results ---------------------------------------------------------
p <- modelbased::estimate_means(model) %>%
  arrange(Mean) %>%
  mutate(
    Method = stringr::str_remove(Method, "Diff_"),
    Method = factor(Method, levels = Method)
  )  %>%
  ggplot(aes(x = Method, y = Mean, color = Method)) +
  geom_line(aes(group = 1)) +
  geom_pointrange(aes(ymin = CI_low, ymax = CI_high), size = 1) +
  theme_modern() +
  theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
  scale_color_material()
ggsave("featured.png", p, width = 10, height = 6, dpi = 450)



p <- modelbased::estimate_relation(model) %>%
  mutate(Method = stringr::str_remove(Method, "Diff_"))  %>%
  ggplot(aes(x = n_Participants, y = Predicted)) +
  geom_point(data=mutate(results, Method = stringr::str_remove(Method, "Diff_")),
             aes(y=Diff_Abs, color = Method)) +
  geom_ribbon(aes(ymin=CI_low, ymax=CI_high, fill=Method), alpha=0.1) +
  geom_line(aes(color = Method), size=1) +
  theme_modern() +
  theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
  scale_color_material() +
  scale_fill_material()
ggsave("n_participants.png", p, width = 10, height = 6, dpi = 450)


# Save results ------------------------------------------------------------


d <- list("results" = results, "model" = model, "contrasts" = contrasts)
save(d, file = "data.Rdata")

References

_{You can reference this post as follows:}

_{- Makowski, D. (2020, September 14). How to extract individual scores from repeated measures. Retrieved from https://dominiquemakowski.github.io/post/individual_scores/}

Thanks for reading! Do not hesitate to share this post, and leave a comment below 🤗

🐦 And don’t forget to join me on X @Dom_Makowski

R or Python for Psychologists

Fri, 22 May 2020 00:00:00 +0000

Many psychology students or researchers are faced with the challenge - or the opportunity - of learning a programming language. Which one should you learn?

As an ex- psych student and a daily user and developer of some of them, here’s my take on this hot debate.

What has programming to do with psychology?

If you’re a very young psychology student, or a future one, you might wonder: why the heck would I have to learn programming in psychology? “Psychology is kinda like philosophy, it’s just learning how people’s minds work by reading books and overthinking stuff”. If you still think that, you’re in for one hell of a ride!

Psychology is, since its very beginning, a hard and experimental science. The founding fathers of psychology were dedicated to find ways to objectively measure psychological phenomena and uncovering the mathematical laws that govern Human behaviour (one of the fields of psychology is even called psychophysics). True, this sciency nature has been toned down by the booming popularity of pseudo-scientific approaches like psychoanalysis throughout the 20th century, that contributed to the stereotypical public image of the shrink doodling while listening to a neurotic patient. But that’s a distorted and old-fashioned view, not really representative of the future of psychology.

The fact is that psychology is very closely connected with statistics. Many great statistical advances were made by psychologists, and all true psychological discoveries are backed by statistical findings. And this importance of statistics is - and will be - growing further, partly due to the recent realization of some major issues in the field due to improper statistical procedures (coined the “replicability crisis”). Moreover, psychology is more and more relying on advanced data-acquiring methods (smartphone apps, website behaviour data, online surveys, physiological and brain recording devices like EEG, MRI, etc.). And these new formats often require specialized knowledge (web-scraping, database querying, neuroimaging, signal processing, machine learning, …). And with great data-power comes great data-analysis-responsibilities. Even in the most applied kind of clinical or psychotherapeutic specializations, where you’d think you’d be safe, they are starting to use data intensive methods like neuro-feedback, virtual reality, experience sampling, and other forms of smartphone sensing and surveying.

Long story short, no matter which branch of psychology you specialize in, you will be confronted with some technical aspects that won’t be able to solve with Excel. Moreover, these technical skills are the ones that will make the most difference between students, and that will matter a lot if you want to pursue research or want to go work in the private sector. The golden era where people were recruited in research based on their theoretical expertise is over: technical skills are now the golden ticket to enter - and successfully leave - academia.

So, ready to dive into programming? Fear not! It’s not that complicated. Moreover, it’s one of the most rewarding skill you can develop. I can assure you that you won’t regret the time invested in learning it 😊

But where should you start?

Learn both R and Python

This increasing relationship between psychology and statistics on the one hand, and other more general technical aspects on the other, is the reason why R and Python are so popular in psychology. Both languages are free, open-source, suited for beginners, and have a large base of users with a ton of learning material online. What’s the difference between them?

Put simply, R is for statistics, Python is for the rest.

So why is there a virulent debate going on, and a choice to make? It’s true that I, in theory, would agree with some popular recommandations and suggest learning both, as they are complementary and have their own strengths and weaknesses. I myself use both on a daily basis, so why would preach what I practice?

That said, many opinionated people are also arguing in favour of one or the other (usually the only one they know…) will say that learning both is essentially a waste of time. They will put forth a strong argument: you can do whatever you do in R in Python, and vice-versa. In other words, both languages can be used to achieve your goals, so it’s better to specialize in one than have a limited knowledge of both.

Although I do not agree with that statement, I do acknowledge that people have limited time and resources to learn. Saying “just learn both” is easy, but is arguably an unrealistic expectation for the vast majority of people. So why learning both can be a long-term goal (especially if you want to do research), you have to start somewhere. So, what starter language should you select?

Ash choosing his starter programming language. He has the choice between R, Python and Bulbasaur, i.e, Matlab - the one that no one likes.

What about Matlab?

There was a time when Matlab was the boss. It was used everywhere and had the best functionalities for neuroimaging, signal processing and maths. But that time is over. Matlab is already a zombie language, which burial process will continue in the years to come. Why is it dead? Because it is expensive, closed, ugly, and most importantly because the alternatives (namely R and Python) are now more powerful and featured than Matlab.

Agamemnon reacting to king Priam saying "The city of Matlab will never be conquered".

The truth is, there are only two reasons people still use Matlab: habit (it’s hard to learn a new approach if your old way of doing things still works) and SPM (a toolbox for neuroimaging that is still - for now - the leader in the field).

But seriously, don’t waste time on it if you have limited resources, it’s just not worth it. You will learn an outdated tool that you won’t be able to use in another lab if they don’t agree to pay for an expensive license (unless you’re a pirate ☠️). Whereas with open and free languages like R or Python, you have access to the best tools and can use them freely everywhere. Also, it makes you a supporter of open-science, which is good 😁.

How to decide between R and Python

Time has come to make a decision.

Despite what people say, R and Python are not equivalent. You can argue as much as you want, but doing statistics and data visualization in Python is not as fast, easy and neat as it is in R. And signal processing or neuroimaging is not as powerful in R as compared to Python. Note that both languages are still growing and changing, and they are influencing themselves: for instance, many popular Python modules (e.g., pandas, statsmodels, seaborn, …) have been directly inspired by R. As such, the boundaries between the two languages are fading (and I’m not even mentioning the great advances in interoperability, with tools like reticulate that allow you to use one language directly inside the other).

That being said, Python and R remain very different languages at their core, with a different feel and vibe to it. R was made by statisticians for statistics, and the majority of its users are academics and researchers. On the contrary, Python is a true general-purpose “programming” language, widely used outside of academia, in the private sector.

Here are some things to consider when deciding on what language to learn:

Reasons to choose Python

You have some basic knowledge or familiarity with programming

For instance, you know what a logical loop is. Python being a true programming language, having any prior experience will be useful.

You are good with logic and spatial representation (like imagining shapes in 3D, rotating them, etc.)

In Python, you will have to think with a “programming” mindset. That means perceiving things in terms of logical statements and blocks, understanding data as 2D or 3D tables that you have to slice and recombine.

You are comfortable with maths

In Python, numbers and numbers combinations are used a lot. Paradoxically, you will typically see a lot more maths in Python than in R.

You plan to do signal processing or experimental tasks creation

These are some of the domains where Python is well-established (which doesn’t mean that R doesn’t have some great tools in development).

You are good at googling and don’t mind spending time looking for the right answer

Python has so much material online that it’s sometimes hard to find the appropriate thing. Harder than in R, in my opinion, which has more well-defined “gold-standard” textbooks and tutorials.

Reasons to choose R

You have no experience with programming whatsoever

R is not made to be used as a traditional programming language. It’s more of finding what functions to apply to what, and that makes it easy for beginners.

You are interested in statistical analyses, modelling things, and making inferences from data

R excels at this. You can create powerful models super easily and jump into their understanding and interpretation.

You like making nice figures and plots

R, through the ggplot ecosystem, has hands down the best tools for visualization. Your imagination is the limit, and you can even create art (check-out the artworks by Thomas Lin Pedersen and Danielle Navarro 😍).

You are not so good with stats or maths

You heard it right! To start with R you don’t need to know stats or maths like a boss. R, in fact, will help you to become proficient at it, by slowly opening more and more layers of complexity to you, if you are deemed worthy.

You are interested in joining the academic community

Because most of its users are academics, R has a fantastic community online, for instance on Twitter #rstats. It’s also super inclusive (e.g., the R-Ladies).

Other considerations

What your peers are learning

It’s easier to learn together, so try to discuss it with your class or lab mates if you can.

What your lab is using

It might be easier if you have mentors that can understand what you are doing and guide you. But that should not be a priority, as it can lead to cargo cult-like old habits reproduction (especially if your lab has a tradition of Matlab 🤭). Instead of submitting to the tradition, assess what the goals and objectives are, and pick the best tool to achieve them. And if you have any issue convincing your boss / supervisor about it, ask some help on Twitter, I bet you’ll get a lot of it.

Hands on!

👉 Looking for places to start? Check out this 10-min crash course introduction to Python and this collection of resources for R.

Thanks for reading! 🐦 Don’t forget to join me on X (@Dom_Makowski) and leave a comment below 👇

How to correctly analyze reaction time (RT) data

Mon, 18 May 2020 00:00:00 +0000

This is a very, very important topic given the widespread usage of reaction times in psychology. Most of the time, we analyze it as a regular variable, using traditional models such as linear models, ANOVAs etc. The problem is that these models assume that the RT is normally distributed, which is false.

This leads us to adjustements like outliers removal or log-transformation, distorting the data because of our non-appropriate models.

The good news is, it’s very easy to use better models, that account for the non-normal distribution of RT. And these alternatives are beautifully presented by Jonas K. Lindeløv in the guide below:

👉 Reaction time distributions: an interactive overview 👈

It is a must-read for all psychologist. Do check-it out!!

Thanks for reading! Do not hesitate to tweet and share this post, and leave a comment below 🤗

🐦 And don’t forget to join me on X @Dom_Makowski

Introduction to Bayesian statistics with R

Thu, 14 May 2020 00:00:00 +0000

You are a student or a researcher interested in Bayesian statistics and R? But all the tutorials and courses that you have found are too intimidating?

Fear no more!

With the easystats team, we have created a very gentle and introductory course for beginners.

You can find the link here:

👉 Get Started with Bayesian Statistics using R 👈

Thanks for reading! Do not hesitate to tweet and share this post, and leave a comment below 🤗

🐦 Don’t forget to join me on X @Dom_Makowski

R | Reality Bending Lab

How to Assess Task Reliability using Bayesian Mixed Models

Podcast 'Learn Bayesian Stats' with Dominique Makowski

The ‘Learning Bayesian Statistics’ Podcast

"Your sample is too small": How to Answer to Reviewers

“Your sample is too small”: How to Answer to Reviewers

References

How to sync two folders in two separate GitHub repositories

How to sync two folders in two separate GitHub repositories

The Problem

The Solution

Notes

How to share data analysis scripts with publications?

How to share data analysis scripts with publications?

Option 1: Dump the code in a word file

Option 2: Use Rmarkdown

Option 3: Use our Results Template

How to extract individual scores from repeated measures

Introduction

Results

Generate Data

Fit model

Visualize the results

Conclusion

Functions

References

R or Python for Psychologists

What has programming to do with psychology?

Learn both R and Python

What about Matlab?

How to decide between R and Python

Reasons to choose Python

Reasons to choose R

Other considerations

See Also

Hands on!

How to correctly analyze reaction time (RT) data

Introduction to Bayesian statistics with R