OpenAI built Codex-powered system to investigate ChatGPT performance regressions

  • OpenAI
  • ChatGPT
  • Codex
  • performance engineering
  • regression

Its Regression Workflow tracked more than 1,700 investigations in just over three months

Agents propose fixes for engineers to review and check performance after deployment

OpenAI built the Performance Factory, a cloud-based system that uses Codex agents to investigate and help resolve performance regressions affecting ChatGPT. Its Regression Workflow starts when a performance alert fires, then assigns agents to investigate the slowdown, propose a fix for an engineer to review, and check performance after deployment.

OpenAI said the Regression Workflow tracked more than 1,700 investigations and identified root causes in more than 600 cases in just over three months. The system uses internal monitoring data, including metrics, logs, traces, CPU profiles and memory snapshots, to help agents analyze regressions.

A separate Codex session reviews proposed diagnoses and fixes, challenging their assumptions before an engineer decides whether to merge a change. After deployment, an agent checks whether performance has returned to its earlier baseline; if it has not, the workflow can send the issue back for investigation or open a task for a remaining slowdown.