Assessing and Verifying Task Utility in LLM-Powered Applications

The rapid development of Large Language Models (LLMs) has led to a surge inapplications that facilitate collaboration among multiple agents, assistinghumans in their daily tasks. However, a significant gap remains in assessing towhat extent LLM-powered applications genuinely enhance user experience and taskexecution efficiency. This highlights the need to verify utility of LLM-poweredapplications, particularly by ensuring alignment between the application'sfunctionality and end-user needs. We introduce AgentEval, a novel frameworkdesigned to simplify the utility verification process by automaticallyproposing a set of criteria tailored to the unique purpose of any givenapplication. This allows for a comprehensive assessment, quantifying theutility of an application against the suggested criteria. We present acomprehensive analysis of the effectiveness and robustness of AgentEval for twoopen source datasets including Math Problem solving and ALFWorld House-holdrelated tasks. For reproducibility purposes, we make the data, code and all thelogs publicly available at https://bit.ly/3w3yKcS .