Max Grossman (University of Cologne): “Integrating Machine Behavior into Human Subject Experiments: A User-friendly Toolkit and Illustrations + uproot: An Experimental Framework with a Focus on Performance, Flexibility, and Ease of Use”

Abstract 1: Large language models (LLMs) have the potential to profoundly change and enrich experimental research. We propose a new software framework, “alter_ego,” that makes it easy to develop experiments between LLMs. Our software also allows including LLMs in oTree-based web experiments with human subjects. We demonstrate the utility of our framework by running simple framed prisoners’ dilemmata both between sets of two machines as well as between sets of one machine and one human. We show that the addition of human subjects reduces cooperation. Mutual distrust is pervasive, and GPT-4 (a well-known LLM) becomes vindictive after human defection. Our toolkit is freely available, with a builder for experiments and video tutorials.

Abstract 2: Existing popular software frameworks for conducting behavioral experiments—both online and in the lab—suffer from limitations including architectural inflexibility and limited feature sets that restrict their utility. They are closed source or do not encourage outside contributions. To overcome these restrictions, we introduce “uproot”: a scalable, fully parallelizable, open-source framework for developing and conducting behavioral experiments. uproot revolutionizes data storage with an append-only log, ensuring data persistence and allowing the use of arbitrary data types. uproot provides many additional important features for modern experiments. For example, Likert scales are provided out of the box, file uploads are possible, and error messages can be customized easily. With capabilities such as page repetition, random page orderings, indefinitely repeated sets of pages, management of individual subjects, dynamically sized sessions, and convenient APIs, uproot overcomes many of the limitations of existing frameworks. Our framework is suited for both small-scale and large-scale studies, including surveys and experiments in which participants interact with each other. Finally, we encourage community-driven development.