📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.

Italy’s Minerva project, a large-scale sovereign language model trained entirely from scratch on 2.5 trillion tokens, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite its extensive Italian content. This performance raises questions about the effectiveness of large-scale native-language training for complex language tasks and challenges assumptions about the relationship between investment size and model proficiency.

Minerva was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, with support from Italy’s national supercomputing consortium CINECA and funding through Italy’s National AI strategy under the PNRR. The project trained models up to 7 billion parameters using 128 GPUs on the Leonardo supercomputer, with roughly half of the training data being Italian, totaling approximately 1.14 trillion tokens.

While Minerva’s architecture and open weights set a standard for transparency and European sovereignty in AI, its performance on academic benchmarks was unexpectedly low. Despite outperforming multilingual models on Italian-specific benchmarks, Minerva-3B’s near-chance score on the INVALSI exam indicates a disconnect between training data scale and real-world language understanding. Researchers noted that overall dataset size and parameter count are more critical for complex language tasks than language-specific data alone.

Implications for European Sovereign LLM Strategies

The low performance of Minerva at a large scale suggests that merely increasing data and parameters may not suffice to achieve country-specific language understanding and knowledge depth. This finding questions the assumption that large-scale native-language training automatically leads to effective academic and practical language skills, impacting future European AI policy and investment strategies.

It highlights the need for a nuanced approach that balances scale with targeted, high-quality data and possibly different architectural strategies. The results also underscore that European projects must consider the diminishing returns of scale without corresponding improvements in complex language comprehension, which is critical for applications like education, governance, and national security.

European Sovereign AI Development and the Minerva Approach

Italy’s Minerva project emerged as a counterpoint to the European debate on sovereign LLM strategies. While Portugal’s AMÁLIA model focused on continuation pre-training of multilingual models with limited native-language data, Italy opted for training from scratch on an enormous dataset with a significant portion of Italian content. This approach was supported by substantial institutional infrastructure, including CINECA’s supercomputing resources and national funding, aiming to create a self-sufficient European AI ecosystem.

Previous efforts in European AI focused on multilingual models or incremental adaptation, often with limited native-language data. Minerva’s large-scale, from-scratch training was intended to demonstrate that a dedicated, native-language model could achieve country-specific expertise, but its low benchmark performance complicates this narrative. The project exemplifies the trade-offs and challenges faced in scaling European sovereign AI initiatives.

“Minerva’s performance underscores that scale alone may not produce the country-knowledge depth needed for high-stakes language tasks.”

— Thorsten Meyer

Unresolved Questions About Scale and Effectiveness

It remains unclear whether different architectural choices, data quality improvements, or larger parameter scales could significantly enhance Minerva’s performance on complex language tasks. The ongoing research and iterative training may yield different results, but current benchmarks suggest that scale alone may be insufficient.

Next Steps for European Sovereign Language Models

Researchers and policymakers will likely reassess the balance between dataset quality, scale, and architecture in future projects. Further experiments with larger models, refined training data, and alternative architectures are expected to clarify whether the current limitations can be overcome. Continued transparency and benchmarking will be essential to guide European AI development and investment strategies.

Key Questions

Why did Minerva perform poorly on Italian academic benchmarks despite large-scale training?

Empirical evidence suggests that overall dataset size and model parameters are more critical than language-specific data volume for complex language tasks. The low score indicates that scale alone may not suffice without targeted, high-quality training data and optimal architecture.

What does Minerva’s performance mean for Europe’s AI sovereignty ambitions?

It highlights the need for a strategic focus on data quality, model scalability, and architectural innovation rather than relying solely on large-scale native-language training to achieve expert-level language understanding.

Could future iterations of Minerva improve its benchmark scores?

Yes, ongoing research and model iterations may address current limitations. Adjustments in architecture, data curation, and training methods could enhance performance, but the current results serve as a cautionary benchmark.

How does Minerva compare to other European sovereign models like Portugal’s AMÁLIA?

While Minerva trained from scratch on a much larger dataset, AMÁLIA focused on continuation training with limited native-language data. Minerva’s performance on benchmarks raises questions about the effectiveness of large-scale native-language training compared to incremental or multilingual approaches.

Source: ThorstenMeyerAI.com