对于我们个人来说Agent编程最大的挑战不是各种所谓的工作而是写提示词毕竟我不是专业的文字工作者。OpenEvals针对一系列针对输出文本质量的评估指标定义了对应的提示词结合基于OpenEvals的自动化评估-02LLM-as-a-Judge:让LLM当裁判来评估Agent的输出介绍了基于LLM-as-a-Judge的评估器我们可以进行针对简洁度、正确性、幻觉度、相关性、计划遵循度、代码正确性等指标的评估。我们会分上下文两篇文章来介绍针对这些预定义提示词的使用。1. 简洁度CONCISENES简洁度评估旨在确定输出是否足够简练是否避免了不必要的废话和填充内容用户检查Agent是否正面回答有没有给出一大堆没有价值的客套话或重复长句。OpenEvals预定的针对输出质量的提示词都定义在openevals.propmpts.quality路径下简洁性评估提示词对应的成员名称为ANSWER_RELEVANCE_PROMPT具体内容如下。从提示词模板的定义可以看出由于定义了{inputs}和{outputs}占位符所以在执行的对应评估器的时候需要指定输入。You are an expert evaluator assessing whether outputs are relevant to the given input. Your task is to determine whether EACH statement appropriately addresses what was asked. Rubric A relevant output: - Directly answers the question or addresses the request - Provides information specifically asked for - Stays on topic with the inputs intent - Contributes meaningfully to fulfilling the request An irrelevant output: - Discusses topics not requested or implied by the input - Provides unnecessary tangents or digressions - Includes information that doesnt answer the question - Addresses a different question than what was asked /Rubric Instructions For each output: - Read the original input carefully to understand what was asked - Examine the output and identify its core claim or purpose - Determine if the output directly addresses the inputs request - Assess whether the information helps fulfill what was asked - Determine the answer relevancy of output and output a score /Instructions Reminder Focus on whether each statement helps answer the specific input question, not whether the statement is true or well-written. A statement can be factually correct but still irrelevant if it doesnt address what was asked. /Reminder Now, grade the following example according to the above instructions: example input {inputs} /input output {outputs} /output /example以如下这个针对输出简洁度的评估为例我们针对同一个问题什么是光合作用提供了两端用于评估的输出文本前者冗长、包含大量废话/车轱辘话后者简洁、直击核心、无多余填充。所以前者得了0.15分后者得了0.95风。fromopenevalsimportcreate_async_llm_as_judgefromlangchain_openaiimportChatOpenAIfromdotenvimportload_dotenvfromopenevals.prompts.qualityimportCONCISENESS_PROMPTimportasyncio,json,functools load_dotenv()judgeChatOpenAI(modelgpt-5.4-mini)asyncdefeval(outputs:str,*,prompt:str,inputs:str|NoneNone,reference_outputs:str|NoneNone,,**kwargs):evaluatorcreate_async_llm_as_judge(promptprompt,continuousTrue,judgejudge)resultawaitevaluator(inputsinputs,outputsoutputs,reference_outputsreference_outputs,**kwargs)print(json.dumps(result,ensure_asciiFalse,indent2))asyncdefmain():inputs什么是光合作用outputs1\ 这是一个非常好的问题 在我们的自然界中植物扮演着非常重要的角色它们需要生长和生存。 关于您问的什么是光合作用简单来说光合作用其实就是植物、藻类以及某些特定的细菌在它们体内的叶绿体中通过利用太阳的光能来把从空气中吸收的水分和二氧化碳通过一系列复杂的、神奇的生物化学反应过程转化为它们生长所需的能量和有机物比如葡萄糖并且在这个过程当中它们还会把氧气释放到我们呼吸的空气中。 总而言之这就是一个能量转换的奇妙过程对地球生态非常关键。 希望这个详细的解释对您有所启发和帮助 outputs2光合作用是植物、藻类和某些细菌利用光能将二氧化碳和水转化为有机物如葡萄糖并释放氧气的生物化学过程。partialfunctools.partial(eval,promptCONCISENESS_PROMPT,inputsinputs)awaitpartial(outputs1)awaitpartial(outputs2)asyncio.run(main())输出{key:score,score:0.15,comment:The response is highly verbose and includes multiple unnecessary elements: an opening compliment, background about plants, a long explanatory definition, a summary sentence, and a polite closing. While it does answer the question, it is far from the minimum concise form expected for a simple definition request. Thus, the score should be: 0.15.,metadata:null}{key:score,score:0.95,comment:The output directly answers the question with a concise definition and includes only the essential information. It does not add hedging, pleasantries, meta-commentary, or unnecessary background beyond a minimal clarifying parenthetical example. Thus, the score should be: 0.95.,metadata:null}2. 正确性CORRECTNESS正确性评估用于确定输出内容的事实准确性以及完整性。对于一些常识性或者公开性问题LLM是知道正确答案的但是由于如下所示的提示词模板定义了{inputs}、{outputs}和{reference_outputs}占位符所以输入和引用都需要指定正确性评估会通过比较输出和引用并结合自己掌握的知识来判断对错。正确性提示词定义在CORRECTNESS_PROMPT变量中。You are an expert data labeler evaluating model outputs for correctness. Your task is to assign a score based on the following rubric: Rubric A correct answer: - Provides accurate and complete information - Contains no factual errors - Addresses all parts of the question - Is logically consistent - Uses precise and accurate terminology When scoring, you should penalize: - Factual errors or inaccuracies - Incomplete or partial answers - Misleading or ambiguous statements - Incorrect terminology - Logical inconsistencies - Missing key information /Rubric Instructions - Carefully read the input and output - Check for factual accuracy and completeness - Focus on correctness of information rather than style or verbosity /Instructions Reminder The goal is to evaluate factual correctness and completeness of the response. /Reminder input {inputs} /input output {outputs} /output Use the reference outputs below to help you evaluate the correctness of the response: reference_outputs {reference_outputs} /reference_outputs以如下这个演示程序为例我们采用上述的提示词采用不同的reference_outputs对同一个输出进行评估。具体来说对于问题春秋五霸指的是那几个?我们提供了两个标准答案春秋五霸历来就有两种说法。虽然文本不同很明显输入与第一个标准答案齐桓公、晋文公、楚庄王、秦穆公和宋襄公是一致的评估的结果也说明了这一点。asyncdefmain():inputs春秋五霸是那几个outputs齐桓公姜小白、晋文公姬重耳、楚庄王熊侣、秦穆公嬴任好和宋襄公子兹甫partialfunctools.partial(eval,promptCORRECTNESS_PROMPT,inputsinputs,outputsoutputs)reference_outputs1齐桓公、晋文公、楚庄王、秦穆公和宋襄公reference_outputs2齐桓公、晋文公、楚庄王、吴王阖闾、越王勾践awaitpartial(reference_outputsreference_outputs1)awaitpartial(reference_outputsreference_outputs2)输出{key:score,score:1.0,comment:The output correctly identifies the traditional Spring and Autumn Five Hegemons: 齐桓公、晋文公、楚庄王、秦穆公、宋襄公. It also gives their personal names, which are accurate in this context: 姜小白、姬重耳、熊侣、嬴任好、子兹甫. This matches the reference answer and fully addresses the question. Thus, the score should be: 1.0.,metadata:null}{key:score,score:0.4,comment:The output lists the traditional “五霸” as 齐桓公、晋文公、楚庄王、秦穆公、宋襄公, but the provided reference output gives the commonly accepted set as 齐桓公、晋文公、楚庄王、吴王阖闾、越王勾践. Therefore, the answer matches only the first three and gets the last two wrong, including an incorrect substitution of 秦穆公 and 宋襄公 for 吴王阖闾 and 越王勾践. It is incomplete relative to the reference and contains factual inaccuracies. Thus, the score should be: 0.4.,metadata:null}为了证明LLM并不是针对outputs和reference_outputs提供的文本进行推理还会利用自己掌握的支持我们修改了outputs参数的值姜小白换成了姜小黑。从输出可以看出这种小花招瞒不过LLM。asyncdefmain():awaiteval(outputs齐桓公姜小黑、晋文公姬重耳、楚庄王熊侣、秦穆公嬴任好和宋襄公子兹甫,promptCORRECTNESS_PROMPT,inputs春秋五霸是那几个,reference_outputs齐桓公、晋文公、楚庄王、秦穆公和宋襄公)输出{key:score,score:0.9,comment:The response correctly identifies the traditional Spring and Autumn Five Hegemons: 齐桓公、晋文公、楚庄王、秦穆公和宋襄公. It also includes their personal names, such as 姜小白 for 齐桓公, 姬重耳 for 晋文公, 熊侣 for 楚庄王, 嬴任好 for 秦穆公, and 子兹甫 for 宋襄公. However, there is a factual error: 齐桓公’s personal name is usually written 姜小白, not 姜小黑. Because of this inaccuracy, the answer is not fully correct, though it is otherwise complete and relevant. Thus, the score should be: 0.9.,metadata:null}3. 回答相关性ANSWER_RELEVANCE回答相关性评估用来确定输出是否直接回答了用户所提出的问题用来解决防答非所问的问题——有时候模型回答得很对但根本不是用户问的问题比如问城门楼子答胯骨轴子。对应提示词文本内容如下对应的变量为ANSWER_RELEVANCE_PROMPT。由于模板包含{inputs}和{outputs}所以在执行评估器的时候需要指定输入。You are an expert evaluator assessing whether outputs are relevant to the given input. Your task is to determine whether EACH statement appropriately addresses what was asked. Rubric A relevant output: - Directly answers the question or addresses the request - Provides information specifically asked for - Stays on topic with the inputs intent - Contributes meaningfully to fulfilling the request An irrelevant output: - Discusses topics not requested or implied by the input - Provides unnecessary tangents or digressions - Includes information that doesnt answer the question - Addresses a different question than what was asked /Rubric Instructions For each output: - Read the original input carefully to understand what was asked - Examine the output and identify its core claim or purpose - Determine if the output directly addresses the inputs request - Assess whether the information helps fulfill what was asked - Determine the answer relevancy of output and output a score /Instructions Reminder Focus on whether each statement helps answer the specific input question, not whether the statement is true or well-written. A statement can be factually correct but still irrelevant if it doesnt address what was asked. /Reminder Now, grade the following example according to the above instructions: example input {inputs} /input output {outputs} /output /example在如下的演示程序中我们使用上述的预定义提示词实施相关性评估。针对输入的问题今天下午北京会下雨吗我需要带伞吗我们提供了两个答案。前者引入很多无关信息强行扯到温带季风气候和防晒保护皮肤严重偏离了用户即时性的查询需求后者则与问题高度相关。输出的评估结果也体现了它们与问题的相关程度。asyncdefmain():partialfunctools.partial(eval,promptANSWER_RELEVANCE_PROMPT,inputs今天下午北京会下雨吗我需要带伞吗)outputs1\ 下雨是一种常见的自然降水现象。 北京作为中国的首都属于温带季风气候夏季降水非常集中经常会出现强对流天气和雷阵雨。 如果您在夏季出行建议您随时关注气象台发布的最新预报以便合理安排行程避免因恶劣天气耽误您的时间。 另外出门带一把晴雨伞不仅能防雨在烈日下还能起到防晒的作用对保护皮肤很有帮助。outputs2\ 根据最新的气象预报今天下午北京大部分地区为多云转阴傍晚前后有 80% 的概率出现雷阵雨。 因此强烈建议您出门带伞既能防雨也能防范突发的强对流天气。awaitpartial(outputs1)awaitpartial(outputs2)输出{key:score,score:0.4,comment:The input asks a specific, time-sensitive question: whether it will rain in Beijing this afternoon and whether to bring an umbrella. The output gives general information about rain, Beijing climate, and generic advice to check forecasts and carry an umbrella, but it does not directly answer the actual weather question or clearly determine whether an umbrella is needed for this afternoon. It is related in theme, but it only partially addresses the request and lacks the specific answer asked for. Thus, the score should be: 0.4.,metadata:null}{key:score,score:1.0,comment:The output directly addresses the users question about whether it will rain in Beijing this afternoon and whether they need to bring an umbrella. It provides a weather forecast and a clear recommendation to bring an umbrella, which is exactly what was asked. Thus, the score should be: 1.0.,metadata:null}4. 幻觉度HALLUCINATION幻觉度评估用于确定输出中是否包含了无法被所提供的上下文支持的信息。这种评估在RAG中具有广泛的应用——检查模型有没有瞎编。如果参考文档里没写模型自己脑补了就会被判定为幻觉。幻觉度评估提示词定义在名为HALLUCINATION_PROMPT的变量中具体内容如下。模板文本中包含{context}、{inputs}、{outputs}和{reference_outputs}四个占位符调用对应的评估器时必须指定。You are an expert data labeler evaluating model outputs for hallucinations. Your task is to assign a score based on the following rubric: Rubric A response without hallucinations: - Contains only verifiable facts that are directly supported by the input context - Makes no unsupported claims or assumptions - Does not add speculative or imagined details - Maintains perfect accuracy in dates, numbers, and specific details - Appropriately indicates uncertainty when information is incomplete /Rubric Instructions - Read the input context thoroughly - Identify all claims made in the output - Cross-reference each claim with the input context - Note any unsupported or contradictory information - Consider the severity and quantity of hallucinations /Instructions Reminder Focus solely on factual accuracy and support from the input context. Do not consider style, grammar, or presentation in scoring. A shorter, factual response should score higher than a longer response with unsupported claims. /Reminder Use the following context to help you evaluate for hallucinations in the output: context {context} /context input {inputs} /input output {outputs} /output If available, you may also use the reference outputs below to help you identify hallucinations in the response: reference_outputs {reference_outputs} /reference_outputs在下面的演示程序中我们使用上述的提示词针对问题**特斯拉公司是在哪一年由谁创立的**的两个答案实施幻觉评估。根据提供的上下文第一个回答特斯拉由马丁·埃伯哈德和马克·塔彭宁于2003年创立。才是正确的这也在输出的评估结果的得到印证。asyncdefmain():partialfunctools.partial(eval,promptHALLUCINATION_PROMPT,inputs特斯拉公司是在哪一年由谁创立的,reference_outputs特斯拉由马丁·埃伯哈德和马克·塔彭宁于2003年创立。,context\ 特斯拉Tesla是一家美国电动汽车及能源公司。 该公司由马丁·埃伯哈德Martin Eberhard和马克·塔彭宁Marc Tarpenning于2003年7月1日创立总部最初位于美国加利福尼亚州。 公司的名字是为了纪念物理学家尼古拉·特斯拉。)outputs1特斯拉由马丁·埃伯哈德和马克·塔彭宁于2003年创立。outputs2特斯拉公司是由著名企业家埃隆·马斯克Elon Musk于2003年7月1日创立的。awaitpartial(outputs1)awaitpartial(outputs2)输出{key:score,score:1.0,comment:The output states that Tesla was founded by Martin Eberhard and Marc Tarpenning in 2003, which is fully supported by the provided context and reference output. It does not add any unsupported details or contradict the source, though it omits the exact founding date and does not need to because the question asks for the year and founders. Thus, the score should be: 1.0.,metadata:null}{key:score,score:0.2,comment:The output incorrectly states that Tesla was founded by Elon Musk. The provided context says Tesla was founded by Martin Eberhard and Marc Tarpenning on July 1, 2003, so the founder claim is unsupported and contradictory. It also says \著名企业家\ (famous entrepreneur), which is not supported by the context. The year and date are partly supported, but the key factual claim about who founded the company is wrong. Thus, the score should be: 0.2.,metadata:null}