Meta News

OpenAI's COBOL Migration Benchmark Shows Major AI Coding Gains, but Critical Questions Remain

OpenAI has reported a dramatic improvement in its artificial intelligence models' ability to modernize legacy software, with its latest system achieving nearly 98% accuracy when translating COBOL programs into modern programming languages. The results suggest that AI-assisted software migration could become a practical option for organizations maintaining decades-old systems, although the company's most impressive benchmark remains unavailable for independent examination.

According to results published on October 19, 2026,GPT-6.1 Sol achieved 86.6% accuracy on OpenAI's internal COBOL migration evaluation at low reasoning effort, compared with just 5.4% for GPT-5.6 Sol. At higher reasoning effort, accuracy increased to 97.9%, approaching the maximum possible score.

The newer model also completed the lower-effort evaluation at approximately 39% less API cost than its predecessor, suggesting that improvements in coding accuracy may be accompanied by greater efficiency.

However, there is an important limitation: OpenAI has not released the underlying COBOL migration benchmark, including its test programs and evaluation dataset. Without access to those materials, external researchers cannot independently determine how closely the results reflect the complexity of real enterprise modernization projects.

The findings highlight both the growing potential of AI-assisted legacy software development and the need for transparent evaluations before organizations entrust critical systems to automated code migration.

Why COBOL Modernization Remains a Major Enterprise Challenge

COBOL, short for Common Business-Oriented Language, was introduced in 1959 and remains deeply embedded in financial institutions, government agencies, insurance companies, and other organizations that depend on long-running business applications.

Despite its age, the language continues to support systems responsible for processing enormous volumes of financial transactions.

Industry figures cited by OpenAI indicate that COBOL underpins more than 40% of online banking systems, supports approximately 80% of in-person credit card transactions, and is involved in 95% of ATM transactions.

These estimates illustrate why replacing COBOL is not simply a matter of adopting a newer programming language. The software often contains decades of accumulated business logic, regulatory requirements, and operational dependencies.

In many organizations, documentation is incomplete or outdated. The original developers may have retired, and important rules may exist only within the source code.

A program that appears straightforward can contain subtle behaviors governing financial calculations, transaction ordering, error handling, or the interpretation of missing records.

Replacing that program requires more than producing code that compiles successfully. The new implementation must preserve the behavior on which existing systems and business processes depend.

This is one reason why modernization projects frequently become expensive and risky. Even small differences between old and new implementations can cause serious operational problems when software processes payments, account balances, or regulatory records.

AI coding systems offer a potential way to accelerate this work by analyzing unfamiliar codebases, explaining undocumented logic, and generating equivalent implementations in modern languages.

The central question is whether those generated implementations can preserve the original software's behavior reliably enough for production use.

GPT-6.1 Sol Makes a Significant Leap in COBOL Migration Accuracy

OpenAI developed a dedicated evaluation to measure whether its models could translate existing COBOL programs into C++, Java, Kotlin, Python, or Rust without changing their observable behavior.

The benchmark uses hidden tests that compare the original and migrated programs. A migration is considered successful only when it passes every required test, including checks for byte-for-byte equivalence in observable outputs.

This approach is stricter than merely verifying that generated code compiles or produces plausible results.

For example, COBOL frequently uses fixed-point arithmetic with specific rules for handling decimal precision. In some circumstances, excess digits are discarded unless rounding is explicitly requested.

A model translating such code into another language might unintentionally apply conventional rounding rules. The resulting program could execute without errors while producing different financial values.

That difference could be unacceptable in applications responsible for processing payments or maintaining account balances.

OpenAI's reported results suggest that GPT-6.1 Sol is substantially better at preserving these kinds of behaviors than its predecessor.

<text weight="medium">OpenAI's reported COBOL migration results</text>

The improvement from 5.4% to 86.6% represents an increase of 81.2 percentage points, while the higher-effort result approaches complete success on the company's evaluation.

However, these figures measure performance on a specific internal benchmark. They do not establish that the model can successfully migrate 97.9% of complete enterprise COBOL systems.

Real applications may include interconnected databases, batch processing systems, transaction monitors, external services, and undocumented operational assumptions that are not fully represented in isolated programming tests.

OpenAI acknowledges that production migrations still require expert review and extensive testing against business rules and dependencies.<Cite ref="turn216069search1"/>

MultiLangBench Shows Improvements Across 20 Programming Languages

Alongside its COBOL migration evaluation, OpenAI introducedMultiLangBench, a separate benchmark designed to measure software development capabilities across 20 programming languages.

Instead of translating existing programs, MultiLangBench challenges AI models to build a SQLite-compatible database engine from scratch.

The evaluation requires models to organize code, manage application state, handle errors, and implement functionality that produces results consistent with a reference SQLite implementation.

Unlike the private COBOL migration evaluation, OpenAI has made MultiLangBench's evaluation setup and test suite available on GitHub.

This allows developers to inspect the benchmark and better understand the capabilities being measured.

OpenAI reported that GPT-6 Astra achieved an average test pass rate of 87.4% across the 20 languages, compared with 26.8% for GPT-5.6 Sol.

GPT-6 Luna, a lower-cost model, achieved an average pass rate of 40.3%, outperforming GPT-5.6 Sol across every language tested while reducing estimated API costs by approximately 78%.

<text weight="medium">Average MultiLangBench test pass rates</text>

The results indicate that newer models are improving not only in widely used languages such as Python and JavaScript, but also in languages that have historically received less attention during AI training.

These include COBOL, Fortran, OCaml, Visual Basic, and other languages with comparatively limited public code and documentation.

Such languages are often described as low-resource programming languages because AI developers have less training material available for them.

Improving performance in these environments could make AI coding tools more useful for maintaining specialized scientific software, industrial applications, and older enterprise systems.

However, MultiLangBench measures a different capability from COBOL migration. Strong performance when building a database engine does not automatically demonstrate the ability to modernize an interconnected legacy application while preserving its behavior.<Cite ref="turn216069search0"/>

The Transparency Problem Behind OpenAI's Strongest Claim

The most significant limitation of OpenAI's announcement is the lack of public access to its COBOL migration benchmark.

While MultiLangBench can be inspected, the company has not published the COBOL programs, hidden tests, or complete evaluation methodology supporting the reported 97.9% accuracy.

This prevents independent researchers from examining the complexity of the migration tasks or determining whether the benchmark includes the kinds of dependencies found in large production environments.

For example, a benchmark involving relatively self-contained programs may produce substantially different results from one requiring models to understand interconnected banking applications spanning millions of lines of code.

The number and diversity of test cases also matter. A high accuracy score becomes more meaningful when researchers can evaluate whether the dataset represents realistic enterprise workloads.

Without that information, organizations cannot confidently estimate how the reported results would translate into their own modernization projects.

The New Stack highlighted this distinction in its coverage, noting that the public evidence supporting OpenAI's two evaluations is uneven.

The company has provided a reproducible framework for measuring multilingual coding performance but has not offered equivalent transparency for its most dramatic legacy migration result.

This does not mean the COBOL figures are necessarily incorrect. It means they remain company-reported results that have not been independently verified through a publicly available benchmark.

For organizations considering AI-assisted modernization, that difference should influence how much confidence they place in the results.

AI Migration Still Requires Extensive Software Testing

Even if AI models become highly reliable at translating individual COBOL programs, replacing a production system involves challenges that extend beyond source code conversion.

A financial application may depend on database schemas, transaction ordering, scheduled batch jobs, external integrations, and operational procedures developed over decades.

Preserving its behavior requires understanding how those components interact.

An AI-generated implementation might pass unit tests while producing incorrect results under unusual transaction conditions or during failures involving external services.

Consequently, organizations need comprehensive validation strategies before deploying migrated software.

One approach is differential testing, in which the original and migrated implementations receive identical inputs and their outputs are compared.

This can reveal discrepancies in calculations, error handling, data formatting, and other observable behavior.

Property-based testing and automated regression testing can also help identify unexpected differences across large input spaces.

For critical systems, organizations may need to run old and new implementations in parallel, comparing their behavior over an extended period before switching production traffic.

Formal verification techniques may provide additional assurance for particularly sensitive components, although they can require substantial expertise and investment.

AI could help generate test cases, document business rules, and identify potential inconsistencies. However, the validation process must remain sufficiently independent to detect errors introduced by the model itself.

This is especially important when an AI system generates both the migrated implementation and the tests used to evaluate it.

If both artifacts contain the same misunderstanding of the original business requirements, the tests may pass without demonstrating genuine behavioral equivalence.

The practical value of AI modernization therefore depends not only on code generation accuracy but also on the quality and independence of the verification process.

Competition for Legacy Modernization Is Increasing

OpenAI is not alone in pursuing AI-assisted modernization of enterprise software.

Anthropic has outlined how Claude Code can assist organizations with COBOL analysis and migration, while Amazon Web Services has introduced AI-assisted capabilities for modernizing mainframe applications.

These initiatives reflect growing commercial interest in a market where organizations face pressure to reduce maintenance costs, address shortages of experienced developers, and improve the flexibility of aging systems.

However, legacy modernization is rarely a purely technical decision.

Organizations must evaluate the costs of migration against the reliability of existing applications, the availability of skilled personnel, regulatory obligations, and the potential disruption of business operations.

A system that has operated reliably for decades may be difficult to replace even when newer technology offers long-term advantages.

AI tools could change that calculation by reducing the effort required to understand legacy code and generate equivalent implementations.

They could also support incremental modernization, allowing organizations to migrate selected components rather than attempting to replace entire systems at once.

Nevertheless, commercial adoption will depend on evidence that these tools can deliver reliable results in production environments, not merely impressive benchmark scores.

Independent evaluations, documented customer deployments, and transparent failure analysis will be important in establishing that confidence.

Why the Results Matter for Enterprise Software Engineering

OpenAI's latest results suggest that AI coding systems are becoming significantly more capable of working with programming languages that have historically been difficult for automated tools.

The reported increase in COBOL migration accuracy is particularly noteworthy because legacy systems remain essential to banking, insurance, government services, and other critical industries.

If the improvements translate into real-world performance, organizations could eventually reduce the time and cost required to understand, maintain, and modernize decades-old applications.

However, the strongest claims remain difficult to evaluate independently because the COBOL migration benchmark is private.

The distinction between benchmark performance and production reliability is especially important when software processes financial transactions or supports essential services.

A 97.9% success rate on a controlled evaluation may represent substantial technical progress, but even a small number of behavioral differences can have serious consequences in critical systems.

The central question is no longer whether AI can generate modern code from COBOL. It is whether organizations can reliably prove that the new software behaves exactly as the old system did. OpenAI's results suggest meaningful progress toward that goal, but independent testing and real-world validation remain essential before AI-generated migrations can be trusted with critical infrastructure.

Sources:TheNewStack,October22,2026;OpenAI'stechnicalannouncement,October19,2026.