Reproducibility | Reality Bending Lab

Pyllusion has been published

Tue, 30 Nov 2021 00:00:00 +0000

Pyllusion has been published!

Pyllusion is a package for Python that implements a systematic way to manipulate and generate illusions using a set of parameters.

For instance, the famous Müller-Lyer illusion below, which causes the observer to perceive the 2 segments of being different lengths depending on the shape of the arrows, can be generated wit the following lines of code:

mullerlyer = pyllusion.MullerLyer(illusion_strength=30)
mullerlyer.to_image()

To understand more about the parametric approach being implemented in the Pyllusion package, we recommend reading our paper, which includes a hands-on example on how to generate some classic illusions (such as the Delbeouf Illusion), and discusses how Pyllusion contributes to address conceptual and methodological issues in illusion science.

import pyllusion

delboeuf = pyllusion.Delboeuf(illusion_strength=3)
delboeuf.to_image()

Don’t forget to keep a look out for our repo for more exciting open-source projects!

"Your sample is too small": How to Answer to Reviewers

Fri, 05 Nov 2021 00:00:00 +0000

“Your sample is too small”: How to Answer to Reviewers

Reviewers questioning the statistical power of a study, while often valid, is common and unoriginal. The problem of possible lack of power, and thus possible false positives, can also be combined with the desire for stringent multiple comparisons / tests “corrections”.

In many real-life scenarios, we simply cannot increase the sample size (because of money, time, COVID or any other reason). And sometimes, the many tests / comparisons are all needed as we have many hypotheses or variables. Naturally, the best pre-study weapon against power-related issues are preregistration (and registered-reports) with proper power-analysis.

That being said, imagine that for one reason or another we don’t have that. We cannot recruit more participants and have to deal with the data that what we have. What can we do?

Some useful tips can be grouped and referred to as PoSCA (Power, Stringency, Coherence, Acknowledgment):

Power: Don’t waste power
- Use all the information that you have. In particular, the information within participants (if you have multiple trials per participant, don’t average over them!), by using methods such as mixed-models.
Stringent significance
- Increase the significance threshold, either arbitrarily (e.g., lower the p-value threshold to .005; Ioannidis, 2018) or pseudo-arbitrarily (using multiple-comparisons corrections such as the Bonferroni method). This, under the Bayesian framework, is _(somewhat) equivalent to being more script with whatever indices that we use. For instance, consider only Bayes Factors (BF) > 30 instead of 10, or Probabilities of Direction (pd) of 99.9% instead of 97%, or widening the Region of Practical Equivalence (ROPE).
- Constrain the parameters towards 0. We can also make it harder for models to have parameters (i.e., “effects”) that deviate from 0. This is easy to do within the Bayesian framework by setting narrower priors centred around 0. This will naturally pull (known as “shrinkage”) the results towards 0 and is a natural way of “controlling” for false positives. Under the frequentist framework, there are methods of regularization that can introduce a bias, such as LASSO or Ridge regressions.
Coherence: cross-validate results
- There is a need to look at the bigger picture, including non-significant and “trending” effects. Look at all the data together, and all the variables measuring related concepts: do the results go in the same direction? Are they internally coherent? Do they follow the same trend? In a small sample study, I would trust more a conclusion based on 10 tests with independent indices measuring aspects related to “depression”, all barely significant but going in the same direction, over one super-significant (but possibly cherry-picked) result interpreted as a proof of the effect of “depression” in general. In other words, don’t focus on “significant” effects and assess results in their entirety.
Acknowledgment: be honest and humble
- Quantify the evidence against the null hypothesis. Maybe an effect is not “significant”, but that doesn’t necessarily mean there is a lot of evidence that it has no effect either. Quantifying evidence against the null can be easily done using for instance Bayes Factors.
- Be honest and acknowledge the limited power of the study. Don’t try to hide it.
- Temper the interpretation of the results. Don’t over-interpret them if they are not super robust.

Science must be open, replicable & reproductible, slow, and based on methodological best-practices. But first and foremost, it must be honest.

References

Ioannidis, J. P. (2018). The proposal to lower P value thresholds to .005. Jama, 319(14), 1429-1430.

Thanks for reading! Do not hesitate to share or tweet this post, or leave a comment below 🤗

How to sync two folders in two separate GitHub repositories

Sun, 31 Oct 2021 00:00:00 +0000

How to sync two folders in two separate GitHub repositories

The Problem

I have a personal website, stored in a GitHub repo (and hosted via GitHub pages), as well as a lab website (a “company” website, if you will). Both are fairly similar, as they are built using Wowchemy’s academic theme. Importantly, there is a blog in the company websites with posts, but I have one on my personal website too. What I would like is that every time I post something on my website, it gets automatically copied over to the company website. So that I don’t have to manually maintain the content at two separate places.

The Solution

The first step is to go to the settings of your GitHub account, to developers settings, and to personal access tokens. You have to generate a token, and tick the repo authorizations. Copy-paste the key.
Go to the settings of the personal website repo (the source from which the content will be copied), to “Secrets”, and add a new secret called “API_TOKEN_GITHUB” (with the key you just copied).
Create a new GitHub action workflow such as this one. The things to change are the source_file, destination_repo and destination_folder.

Tada 🎉 Everytime I push to my personal repo, the new content of one of the subfolder gets copied to another repo.

Notes

This is a one-way sync, so updates on the target repo won’t affect the source repo (but might get overridden!).
if you want to preserve the original commit message, set commit_message: ${{ github.event.head_commit.message }} _{(thanks @dobbelina)}

Thanks for reading! Do not hesitate to share or tweet this post, or leave a comment below 🤗

How to share data analysis scripts with publications?

Wed, 10 Feb 2021 00:00:00 +0000

Including data analysis as Supplementary Materials can be a tedious task. How can we simplify the sharing of our work? So that it can be fully appreciated, as well as evaluated, improved, worked on, in a transparent and open way?

Option 1: Dump the code in a word file

Most publication portals don’t directly accept code scripts to be included “as is”. In other words, you cannot upload your manuscript and your .R script just like that. So one option is to copy its content, paste it in a word / pdf document, and Voilà!

But what if you don’t have code because you use a point-and-click software? Well, you can note down all the x,y coordinates of your clicks so that one can reproduce the steps and all the clicks. Just kidding, if you don’t have a script, then may god have mercy on your soul.

Why is that a terrible solution? Because let’s face it, unstructured code dumps are horrific. Nobody wants to read it, it does not make justice to your work, and it’s still tedious to create! You have to re-do it everytime you modify your code. And it’s even worse if you want to include all the outputs, figures, tables that are generated by the code? Data analysis is not just the code, but everything that comes with it and that allowed you to make the conclusions that you made.

Option 2: Use Rmarkdown

Rmarkdown is a “framework” that allows you to have files (.Rmd) that can contain a mix of text and code (and not only R, but also Python for instance!).

It can be used to write comprehensive “reports” that include all your thoughts, motivations and interpretations of the results. And the great thing about it is that these files can be converted into beautiful and readable documents like PDF, Word or HTML. It will automatically embed the code and its generated output (as text, tables or figures) alongside the text.

It is an awesome way to write statistical reports, and can even be used to create many other non-stats related stuff, like APA-formatted manuscripts (great for preprints), books, presentation slides, websites or blogs. It’s a must-have skill for every researcher.

And it gets better!

Option 3: Use our Results Template

In the Reality Bending team, we like to have our different projects and studies organized in a consistent way. We heavily use GitHub to store our projects and collaborate on them, and we also like the possibility of making these projects open and accessible (i.e., easy to discover and explore) when the time comes.

That’s why we came up with a Template folder for storing the materials related to a given study, including well-organized analysis scripts. And what’s great about it is that it is setup in a way that allows you to generate multiple files format (word, pdf, html) with a single click (and even without any click, in a fully automatic way)! And what’s even greater is that if you decide to upload it to GitHub, you’ll have a whole website presenting your data analysis!

We use it to have reproducible analysis that we can easily share with publications. We can upload the .pdf or .docx file generated by the template as Supplementary Materials, but we also link the URL of the online repository of the study in the manuscript, where users can access and experience the content in the format that they prefer. It really improves the appeal of a study when the results are trustworthy.

All this is easily made possible with our template. Check it out here:

👉 https://github.com/RealityBending/TemplateResults 👈

☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️☝️

And let us know what you think! You can open an issue on the repo or even contribute to help us improve it :)

Thanks for reading! Do not hesitate to tweet and share this post, and leave a comment below 🤗

🐦 Don’t forget to join me on X @Dom_Makowski