What gets left behind
IAS Festival Reflections by Rosa Muilu
As part of the IAS Festival, students attended sessions and wrote a short reflection connecting the discussions within society to their own perspectives. This piece is connected to the session 'Beyond the Numbers: Rethinking How We Measure What Matters'.

A patient is filling in a mental health questionnaire. Simple questions, tidy 7-point Likert scales, but something makes her hesitate. For the first question, she circles the number 4 lightly. Then, she presses the pencil harder to make a harsher circle around the number 6. Second question, she decides to divide the question into two parts, drawing her own likert scale underneath the first one, and a brief explanation why she needed two scales. Her experience just became a measurement error.
The example is not a hypothetical, and Femke Truijens (Erasmus University) has the form to prove it. She presented it at a workshop for the IAS festival, where researchers across disciplines gathered to ask different versions of the same question: when experiences get measured, what gets left behind?
The questionnaire is a scaffold for talking about mental health, but a scaffold meant to support can harden into a wall that shuts things out. A questionnaire was made by someone, with their own experiences and views on what mental health is. Those viewpoints are something that the questionnaire will carry in every copy that is distributed, including the one that the patient is now rewriting.
Berend van der Kolk (Vrije Universiteit) compares measurement to making Lego bricks. When they are made, there is a mold, molten plastic gets poured in, the plastic cools down and the excess is trimmed off. The result is a clean, stackable brick, and the extra plastic is just scraps that get swept away; the complexity gets stripped for standardisation.
This is not something the researchers in the room can escape. Results need to be replicated, and the numbers need to mean the same thing in this lab as the next one. To be able to build on things that came before, you need something to hold onto. But sometimes, the categories become simply too all-encompassing to be usable. Aline Potiron (Utrecht University) found that in microbiome research, cultural background, ethnicity, genetics, and many other factors were molded into the same term, 'race'. Someone else reading the study might be looking for effects of diet, of genetics, or of where someone grew up, but the characteristics held the same name. It stands for anything, and so it stands for nothing.
Then, during a discussion, someone’s Apple Watch announces they are stressed. That gives words to a fleeting feeling. It does not hesitate like the patient did with her pencil, but declares its answer. Yet the watch does not know what the stress is about. A heart-rate of 120 looks the same whether you are in a heated discussion or receiving a diagnosis. The fix is the same for both: breathe in for four, out for six.
By the end of the workshop, the room has collected more examples than solutions. The form that Femke Truijens brought still exists. The Likert scale the patient drew underneath the printed one might never appear in any dataset. It just does not fit.