Understanding differences in outcomes between social groups—such as wage gaps between men and women—remains a central challenge in social science. While researchers have long studied how observable factors contribute to these differences, traditional methods oversimplify complex variables like employment trajectories. Our work adapts recent advances in artificial intelligence—specifically, foundation models that can process rich, detailed histories—to better explain group differences. We develop mathematical theory and computational methods that allow these AI models to provide more accurate and less biased estimates of how much of group differences can be explained by observable factors. Applied to real-world data, our approach reveals that detailed histories explain more of the gender wage gap than previously understood using conventional methods.
That is from a new paper by Keyon Vafa, Susan Athey, and David M. Blei. Via the excellent Kevin Lewis. This is also real progress on the methodological front.
The post Using AI to explain the gender wage gap appeared first on Marginal REVOLUTION.
