Recent advancements in natural language generation have opened the door to large language models (LLMs) such as GPT-3.5-turbo, which have shown great potential in evaluating code generation. In a groundbreaking study titled ‘LARGE LANGUAGE MODELS ARE STATE-OF-THE-ART EVALUATORS OF CODE GENERATION,’ Terry Yue Zhuo and his team at Monash University propose a novel evaluation framework based on LLMs that better captures the complex syntax and semantics of code generation tasks.

The novel LLM-based evaluation framework revolutionizes code generation assessment, bridging the gap between human judgment and functional correctness in a way that was previously unimaginable. Traditional token-matching-based metrics, like BLEU, have struggled to align with human judgment in code generation tasks. Additionally, using human-written test suites to evaluate functional correctness can be challenging in low-resource domains.

The new framework proposed by Dr. Kevin’s team addresses these limitations by achieving superior correlations with both functional correctness and human preferences, without the need for test oracles or references. The team evaluated their framework on four programming languages—Java, Python, C, C++, and JavaScript—and demonstrated its effectiveness in assessing both human-based usefulness and execution-based functional correctness.

By employing techniques such as zero-shot Chain-of-Thought (zero-shot-CoT), the researchers significantly improved the reliability of LLM-based code generation evaluation. An important aspect of this study is the minimal impact of data contamination, which has been a concern in evaluations of recent closed-source LLMs. Dr. Terry’s team carefully analyzed the data release years and concluded that only the CoNaLa and HumanEval (Python) datasets may have been contaminated, while it is unlikely that GPT-3.5 has seen any human annotation or generated code during training.

The question remains as to whether LLMs can be utilized to evaluate downstream tasks related to source code beyond code generation. Potential applications include code translation, commit message generation, and code summarization. Although existing studies have not released annotation data or fully described human evaluation criteria for these tasks, Terry Yue Zhuo believes that the LLM-based evaluation framework holds great promise for such applications.

In conclusion, this study marks a significant step forward in the evaluation of code generation tasks. The proposed LLM-based framework offers a more accurate and effective means of assessing code generation, paving the way for future research and development in this area.