Oehler, A., Neuss, C., Horn, M., 2025, Why You Should Not (Yet) Use ChatGPT for Evaluating Annual Reports: Evidence from Sustainability Evaluations, IEREK Interdisciplinary Series for Sustainable Development/Amer, M. (ed.), Sustainable Economy and Ecotechnology (SEE), Springer, forthcoming.

Abstract
Since the end of November 2023, users can drag and drop files onto ChatGPT and have them analysed directly. We evaluate the effectiveness of ChatGPT’s new feature and request ChatGPT to evaluate firm sustainability based on annual reports regarding firms' sustainability reporting. We test the replicability and reliability of the answers. The results provided by ChatGPT significantly differ when analysing the same report multiple times using identical prompts. These inconsistencies imply randomness and raise questions about the reliability of ChatGPT for conducting textual analysis on complex documents such as annual reports. Our findings show a limitation in the current version of ChatGPT's ability to process complex documents using the uploading feature and contribute to the ongoing discourse on the reliability and applicability of artificial intelligence (AI) tools in academic and practical research. While the ease of use of ChatGPT makes it appealing to use it for research, our study cautions against overreliance on current technologies without thorough validation. Our study provides strong implications for investors, researchers, editors, reviewers, and practitioners: They must document and verify that outputs of ChatGPT used in research papers and for decision making are checked for robustness by repeating the same task several times.

Keywords

ChatGPT; Sustainability Reporting; Textual Analysis; Annual Reports; SDGs; ESG



Auch:

Oehler, A., Neuss, C., Horn, M., 2025, Why You Should Not (Yet) Use ChatGPT for Evaluating Annual Reports: Evidence from Sustainability Evaluations; 3rd Conference on Sustainable Banking & Finance, CSBF 2025, 1-2 July, Naples.