arXiv:2611.07123 Introduces PromptOps Closed-Loop Framework for LLM Judge Maintenance
PromptOps treats prompt optimization for LLM judges as continuous maintenance. Experiments confirm recovery after distribution shifts but show results remain deployment-specific. The framework supplies an abstraction for tracking when, how, and on which records to update judge prompts.
The paper frames judge drift as degradation in policy agreement when user interactions and score distributions evolve. It separates operational choices—trigger timing, optimization procedure, feedback record selection, and prompt update mechanism—then tests five methods on synthetic and shifted evaluation sets. Experiments showed recovery of agreement metrics after distribution changes, yet no single algorithm or update schedule generalized across all tested conditions.
Related work on static LLM judges, including Zheng et al. arXiv:2306.05685 and subsequent rubric-based evaluations, treated prompt design as a one-time development-set task. PromptOps demonstrates that such one-shot approaches leave measurable residual drift when reference distributions move, establishing maintenance as a recurring operational requirement rather than an initial configuration step.
For legal and regulatory adjudication pipelines that rely on LLM judges, the results imply periodic recalibration cycles must be instrumented with explicit drift detection thresholds. Context dependence of method efficacy requires per-deployment validation rather than universal adoption of any listed optimizer. Future maintenance protocols will need logged feedback streams and versioned prompt states to support reproducible recalibration.
Sources (2)
- [1]Primary Source(https://arxiv.org/abs/2611.07123)
- [2]Supporting Source(https://arxiv.org/abs/2306.05685)