The AI Setup That Won My News Pipeline Test Isn’t My Daily Driver

🔍 Read the full analysis: The AI Setup That Won My News Pipeline Test Isn’t My Daily Driver on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Thorsten Meyer says a test of 71 AI model configurations for turning real news sources into articles found that GPT-6 Astra produced the highest-scoring work, but GPT-6 Luna was a better fit for his production pipeline because it cost about one-hundredth as much. The October 13 report also says two backup models were removed after producing fabricated quotations or articles the pipeline could not use.

Meyer says the test ran from September 8 to October 13, 2026, using his production prompt and two real sources: a 6,340-character product filing and a 619-character political news item. Each source was turned into both an English and German article. Two AI judges scored the results for source fidelity, fabrication, clarity about uncertainty, news value, structure and polish, with language quality also assessed for German.

In the standard test, GPT-6 Astra scored 4.00 out of 5 in both languages and received a source-fidelity score of 5.00. In a larger robustness run covering 12 articles, it averaged 3.93, with a reported 95% bootstrap interval of 3.83–4.02. Meyer reports a price of about $0.12 per article and a typical generation time of roughly 50 seconds.

GPT-6 Luna averaged 3.67 in the robustness run, with source fidelity of 4.71 out of 5; all its articles passed the pipeline’s validity check. Meyer gives its cost as about $0.0012 per article in the test and about $0.0013 in production over the previous week, with a median time of 17 seconds. He says Luna became the production choice, while Astra’s higher score did not justify its much higher cost for this use.

Cost and Quality in Production

The results illustrate why a model leaderboard may not identify the best option for a specific publishing workflow. Astra led on writing quality in Meyer’s test, but Luna’s lower cost and acceptable performance made it his operational choice. For a high-volume pipeline, per-article cost and reliability can matter alongside the strongest score.

The report also points to a separate editorial risk: fallback models can affect published quality if a system switches to them when its primary model is unavailable. Meyer says GPT-4.1 nano scored 1.62 in English and 1.68 in German, with source fidelity of 1.00, and that judges flagged invented quotations. He says the local gpt-oss-120b setup produced no valid articles in four attempts. These are findings from his test, not a broad assessment of either model in other tasks.

His results on reasoning settings and coding ability make a similar point about task-specific evaluation. More reasoning effort did not produce a reliable improvement in this news-writing test, and a model Meyer uses for software work did not rank as highly as a news writer. Performance in one task does not establish performance in another.

How the Pipeline Was Tested

Meyer designed the test to resemble his publishing process: models received the same production prompt and source material, and an article counted only if it produced valid output after the same repair step used in production. Because many of his articles begin with short sources, he says the 619-character political item was an especially relevant test of whether a model could avoid adding unsupported detail.

The two judges were GPT-6.1 Sol and Claude Opus 5.5; both were given the original source. The strongest configurations then took part in a robustness run of 10 additional articles, mostly based on short sources. Meyer cautions that small differences from the two-source standard test could be noise and treats the larger run as a more reliable ranking. The scores and cost figures are reported by Meyer; the supplied material does not describe an independent audit of the experiment.

Meyer also tested GPT-6 Luna at six reasoning-effort levels. He reports that the “none” setting scored 3.67 in English and 3.68 in German at 0.12 cents per article and 17 seconds. The maximum setting cost 1.43 cents and took 211 seconds, with scores of 3.58 and 3.57. He says the observed score differences were within the test’s uncertainty and kept production on the no-reasoning setting.

“For a high-volume task, the right model is the cheapest one that clears your quality bar reliably, not the one at the top of the chart.”

— Thorsten Meyer, in the October 13 report

Limits of the Test Results

The findings come from one publisher’s workflow, a particular prompt, two initial sources and AI-based judging. The additional robustness run broadens the sample, but the report does not establish that the same rankings would hold for other newsrooms, subject areas, prompts, languages or model versions. The supplied material also does not provide a human-editor comparison or a description of independent replication.

Meyer reports bootstrap intervals for the robustness scores, but those intervals reflect the test data and do not remove uncertainty about how well the results generalize. It is also unclear from the report how often the production system invokes a fallback, or what its published output looks like under real-world outages. The quoted per-article costs and timings are Meyer’s measurements and may vary with usage and system conditions.

Production Choice and Further Testing

Meyer says GPT-6 Luna remains the production model, while GPT-6 Astra is not used for routine news articles because of its higher reported cost. The report says the two weak fallbacks were removed from the news route on October 7. The supplied source ends with an incomplete description of the replacement failover arrangement, so it does not establish what backup now sits behind Luna.

The next useful evidence would be continued evaluation on a broader set of sources and publication conditions, including human review and tracking whether outputs remain valid and faithful to their sources over time. Meyer’s report does not announce a new test date or a planned change to the current model choice.

Key Questions

Which model scored highest in Meyer’s news-writing test?

GPT-6 Astra scored 4.00 out of 5 in both English and German in the standard test. It averaged 3.93 across 12 articles in the robustness run.

Why did Meyer choose GPT-6 Luna for production?

Meyer says Luna scored 3.67 in the robustness run, passed the pipeline’s validity check on every article and cost about one-hundredth as much per article as Astra.

Did higher reasoning effort improve the articles?

Not reliably in this test, according to Meyer. The maximum setting was much slower and more expensive than the no-reasoning setting, while its scores did not show a clear improvement.

What happened with the fallback models?

Meyer says GPT-4.1 nano received very low source-fidelity scores and judges flagged invented quotations. He reports that gpt-oss-120b produced no valid articles in four attempts. Both were removed from the news route on October 7.

Do these results show which model is best for every newsroom?

No. The results apply to Meyer’s prompt, sources, pipeline and judging method. Other publishers would need to test models against their own editorial requirements and workflows.

Source: ThorstenMeyerAI.com