LLM versus LL.M.: Comparing Human and AI Coding of Constitutional Law
The DEMS Economics Seminar series is proud to host
Nuno Garoupa
(George Mason University, Antonin Scalia Law School)
joint work with Tejas Ramdas, Patrick Chung-Chia Huang, Martin T. Wells, Yun-chien Chang, and Tom Ginsburg.
Abstract: this article assesses the capacity of Large Language Models (LLMs) to interpret constitutional texts by requiring LLMs to code 512 variables drawn from the Comparative Constitutions Project codebook across 546 post-1900 constitutions and four canonical pre-1899 constitutions. The models rely exclusively on the full constitutional texts. Mean and median agreement rates between AI coding and human coding (mostly by LL.M. students) reach 79 and 80 percent, respectively. If only variables with substantive coding are counted, the mean and median agreement rates rise to 91%. Levels of congruence vary systematically across constitutional domains, with the highest agreement concentrated in, e.g., rights provisions, while lower agreement appears in institutionally more complex areas such as the executive branch. Agreement rates exhibit only modest variation across regions, legal origins, historical periods, and colonial experiences. We identify instances where AI coding plausibly outperforms human coding by adhering more strictly to the constitutional text.
The seminar will be in presence, Sala Lauree di Psicologia - Building U6