Measuring Political Bias in Large Language Models: What is Said and How It is Said

We propose to measure political bias in LLMs by analyzing both the contentand style of their generated content regarding political issues. Existingbenchmarks and measures focus on gender and racial biases. However, politicalbias exists in LLMs and can lead to polarization and other harms in downstreamapplications. In order to provide transparency to users, we advocate that thereshould be fine-grained and explainable measures of political biases generatedby LLMs. Our proposed measure looks at different political issues such asreproductive rights and climate change, at both the content (the substance ofthe generation) and the style (the lexical polarity) of such bias. We measuredthe political bias in eleven open-sourced LLMs and showed that our proposedframework is easily scalable to other topics and is explainable.