Evaluator-Optimizer Workflow

This example demonstrates the Evaluator-Optimizer pattern from Anthropic’s article

Credits of the image to Anthropic

Overview

In the evaluator-optimizer workflow, one LLM call generates a response while another provides evaluation and feedback in a loop.

This workflow is particularly effective when we have clear evaluation criteria, and when iterative refinement provides measurable value. The two signs of good fit are, first, that LLM responses can be demonstrably improved when a human articulates their feedback; and second, that the LLM can provide such feedback. This is analogous to the iterative writing process a human writer might go through when producing a polished document.

When to use

This workflow is particularly effective when we have clear evaluation criteria, and when iterative refinement provides measurable value. It works best when LLM responses can be demonstrably improved through feedback, and when an LLM can provide meaningful evaluation of that feedback.

Using evaluator-optimizer in Latitude

Literary Translation with Iterative Refinement

---
provider: openai
model: gpt-4.1
temperature: 0.3
maxSteps: 8
---

<step as="initial_translation" isolate schema={{
  type: "object",
  properties: {
    translation: {
      type: "string",
      description: "The translated text"
    },
    translation_notes: {
      type: "string",
      description: "Notes about translation choices and challenges"
    },
    confidence_level: {
      type: "string",
      enum: ["high", "medium", "low"],
      description: "Confidence in the translation quality"
    }
  },
  required: ["translation", "translation_notes", "confidence_level"]
}}>
  You are an expert literary translator specializing in {{ source_language }} to {{ target_language }} translation.

  Translate the provided text while preserving:
  - Literary style and tone
  - Cultural nuances and references
  - Poetic devices like rhythm, alliteration, or wordplay where possible
  - Author's intended meaning and emotional impact

  <user>
    Source text ({{ source_language }}): {{ source_text }}

    Context: {{ context }}

    Please provide a literary translation that captures both the meaning and artistic qualities of the original text.
  </user>
</step>

<step as="evaluation_feedback" isolate schema={{
  type: "object",
  properties: {
    overall_quality: {
      type: "string",
      enum: ["excellent", "good", "needs_improvement", "poor"],
      description: "Overall assessment of translation quality"
    },
    specific_feedback: {
      type: "array",
      items: {
        type: "object",
        properties: {
          category: {
            type: "string",
            enum: ["accuracy", "style", "cultural_nuance", "readability", "literary_devices"]
          },
          issue: { type: "string" },
          suggestion: { type: "string" },
          severity: { type: "string", enum: ["minor", "moderate", "major"] }
        }
      }
    },
    strengths: {
      type: "array",
      items: { type: "string" },
      description: "What the translation does well"
    },
    needs_revision: {
      type: "boolean",
      description: "Whether the translation needs further refinement"
    },
    priority_areas: {
      type: "array",
      items: { type: "string" },
      description: "Areas that need the most attention in revision"
    }
  },
  required: ["overall_quality", "specific_feedback", "strengths", "needs_revision"]
}}>
  You are a literary translation critic with expertise in both {{ source_language }} and {{ target_language }} literature.

  Evaluate the translation for accuracy, style, cultural sensitivity, and preservation of literary qualities.

  <user>
    Original text ({{ source_language }}): {{ source_text }}

    Translation to evaluate: {{ initial_translation.translation }}

    Translator's notes: {{ initial_translation.translation_notes }}

    Context: {{ context }}

    Please provide detailed feedback on this translation, focusing on:
    1. Accuracy of meaning and cultural references
    2. Preservation of literary style and tone
    3. Natural flow in the target language
    4. Treatment of literary devices (metaphors, wordplay, rhythm)
    5. Overall readability and impact

    Identify specific areas for improvement and provide concrete suggestions.
  </user>
</step>

{{ if evaluation_feedback.needs_revision }}
  <step as="revised_translation" schema={{
    type: "object",
    properties: {
      translation: {
        type: "string",
        description: "The improved translation"
      },
      revision_notes: {
        type: "string",
        description: "Explanation of changes made based on feedback"
      },
      addressed_issues: {
        type: "array",
        items: { type: "string" },
        description: "Which feedback points were addressed"
      }
    },
    required: ["translation", "revision_notes", "addressed_issues"]
  }}>
    You are an expert literary translator. Revise your translation based on the detailed feedback provided,
    addressing the specific issues while maintaining the overall quality of your work.

    <user>
      Original text ({{ source_language }}): {{ source_text }}

      Your initial translation: {{ initial_translation.translation }}

      Detailed feedback: {{ evaluation_feedback.specific_feedback }}

      Priority areas for revision: {{ evaluation_feedback.priority_areas }}

      Please revise the translation to address the feedback, particularly focusing on the priority areas.
      Explain what changes you made and why.
    </user>
  </step>

  <step as="final_evaluation" schema={{
    type: "object",
    properties: {
      quality_improvement: {
        type: "string",
        enum: ["significant", "moderate", "minor", "none"],
        description: "How much the revision improved the translation"
      },
      remaining_issues: {
        type: "array",
        items: { type: "string" },
        description: "Any issues that still need attention"
      },
      translation_ready: {
        type: "boolean",
        description: "Whether the translation is ready for publication"
      },
      final_assessment: {
        type: "string",
        description: "Overall assessment of the final translation"
      }
    },
    required: ["quality_improvement", "remaining_issues", "translation_ready", "final_assessment"]
  }}>
    You are a literary translation critic. Evaluate the revised translation to assess whether the feedback
    was successfully addressed and if further revision is needed.

    <user>
      Original text ({{ source_language }}): {{ source_text }}

      Initial translation: {{ initial_translation.translation }}

      Revised translation: {{ revised_translation.translation }}

      Changes made: {{ revised_translation.revision_notes }}

      Previous feedback: {{ evaluation_feedback.specific_feedback }}

      Assess whether the revision successfully addressed the feedback and if the translation is now ready.
    </user>
  </step>
{{ endif }}

You are a translation project coordinator. Provide a final summary of the translation process and deliver the best version.

<user>
  Translation project summary:

  Original text: {{ source_text }}
  Initial translation: {{ initial_translation.translation }}

  {{ if evaluation_feedback.needs_revision }}
    Feedback provided: {{ evaluation_feedback.specific_feedback }}
    Revised translation: {{ revised_translation.translation }}
    Final evaluation: {{ final_evaluation.final_assessment }}
  {{ else }}
    Initial evaluation: {{ evaluation_feedback.overall_quality }} - no revision needed
  {{ endif }}

  Please provide:
  1. The final recommended translation
  2. A summary of the translation process and any iterations
  3. Key insights about the translation challenges and how they were addressed
  4. Confidence level in the final result
</user>

This pattern demonstrates the power of iterative refinement through structured evaluation and optimization cycles, particularly valuable for complex tasks requiring high-quality outputs where initial attempts can be systematically improved through expert feedback.