Researchers from Swiss universities, including ETH Zurich and EPFL, have collaborated with the Swiss National Supercomputing Centre (CSCS) to create a large language model (LLM). According to ETH, the model is set to launch in late summer and will be capable of understanding more than 1,000 languages. It will be made fully open-source, allowing broad usage and fostering innovation across science, society and industry.

The model will be released in two versions, with 8 and 70 billion parameters respectively. ETH states that the larger version will rank among the most powerful fully open models globally. The model was trained on the CSCS’s Alps supercomputer in Lugano using 100% climate-neutral electricity. With over 15 billion high-quality training tokens and a dataset spanning more than 1,500 languages, the model is expected to achieve a high level of reliability.

The LLM was developed through fully open and transparent processes. Both the source code and model weights will be publicly available, with training data made transparent and reproducible. This approach aims to provide a trustworthy alternative to commercial systems, most of which have been developed through opaque processes.

“Truly open models enable applications that ensure a high level of trust and are essential for advancing research into both the risks and opportunities of AI,” said Imanol Schlag, researcher at the ETH AI Center and co-lead of the project. “Transparent processes also support compliance with legal frameworks.” The development adhered to Swiss data protection laws, Swiss copyright regulations and the transparency obligations of the EU AI Act.

Joël Mesot, President of ETH Zurich, and Anna Fontcuberta i Morral, President of EPFL, are set to speak at the Open-i innovation conference in November to contextualise the significance of such developments. Mesot emphasised: “In uncertain times like these, it is more important than ever to foster close cooperation between business and academia in key areas such as AI to secure prosperity and innovation.”

Fontcuberta i Morral added: “Switzerland stands as a global innovation hub, ranking second in the 2024 World Competitiveness Ranking. Its strong commitment to excellence in research and education, combined with world-class infrastructure – anchored by institutions such as the ETH Domain, including ETHZ and EPFL – continues to fuel its success and shape the future.” Their insights highlight the crucial role of connecting research, industry and policymaking in securing Switzerland’s long-term capacity for innovation.

(Original article by ce/nta)

ETH and EPFL launch large-scale language model

Want to delve deeper into the topic?