For best experience please turn on javascript and use a modern browser!
You are using a browser that is no longer supported by Microsoft. Please upgrade your browser. The site may not present itself correctly if you continue browsing.
Institute for Advanced Study (IAS) Institute for Advanced Study (IAS)

Call for Papers: Politics of Machine Learning Evaluation

4 May 2023

Is the data good enough for training purposes? Does the model perform accurately enough? Is the error rate low enough? Such questions of ‘good enough’ are at the very core of the process of Machine Learning (ML) evaluation and can also be considered a highly political process in the development of ML systems. There is already a growing interest in the political implications of ML in relation to, for example, dataset construction and the political capacities of specific ML models or foundational algorithmic techniques. However, there has been less focus on the politics of evaluation practices and techniques in ML.

To further explore this issue, we invite contributions to a workshop on ‘The Politics of Machine Learning Evaluation’ at the University of Amsterdam in November 2023. The aim of this workshop is to collectively engage with conceptualisations and the methodologies of how to study ML evaluation techniques. We invite papers that engage with conceptual, methodological, and political questions in relation to topics, such as but are not limited to:

  • Dataset construction
  • Data labelling practices
  • Ground truths and benchmarks
  • Biases in evaluation
  • Metrics
  • Errors and error analysis
  • Evaluation techniques

Concretely, we invite papers that engage in conceptualising or historizing ML evaluation as a politically contested practice, provide methodological approaches to the study of evaluation techniques or empirical examples of ML evaluation in practice. It will be an interdisciplinary workshop and we invite scholars from a variety of disciplines.


The workshop:

The workshop is organised by  Dieuwertje Luitse, Anna Schjøtt Hansen and Tobias Blanke, and it will take place on Thursday 16 (afternoon) & Friday 17 November 2023, at the Institute for Advanced Study (IAS) located in the city centre of Amsterdam.

The workshop will feature three keynotes by Florian Jaton (University of Lausanne), Nanna Bonde Thylstrup (Copenhagen Business School) and Claudia Aradau (Kings College London).

Accepted papers will get the opportunity to receive feedback from one of the three keynote speakers as well as peer-discussants. Furthermore, we aim to develop a special issue in a peer-reviewed journal on the basis of the submitted papers, for which all workshop participants will have the opportunity to contribute.

Submission details:

Deadline for submission is June 30th of the 300-500 words abstract. Abstracts should be sent to a.s.hansen@uva.nl and d.luitse@uva.nl, with the subject line ‘Workshop: Politics of Machine Learning Evaluation’. Before the workshop in November, participants will be expected to send draft discussion papers for other participants and keynotes to read.

The workshop is funded by the Amsterdam School for Cultural Analysis (ASCA), the Amsterdam school for Heritage, Memory and Material Culture (AHM) and the Research Priority Area Human(e) AI. It will be hosted by the Institute for Advanced Study (IAS) in Amsterdam.