IDEA Research Institute's Ziya2-13B Big Model Launches Open Source in the Magic Building Community

Tech 2023-10-16 14:03:35 Source: Network

On October 16th, it was reported that the IDEA Research Institute (Guangdong Hong Kong Macao Greater Bay Area Digital Economy Research Institute) CCNL Fengshen List team has opened up the Chinese base model Ziya2-13B-Base and its dialogue model Ziya2-13B-Chat, both of which are completely free and commercially available, and have made their debut on the ModelScope of the magic community.Three months ago, Meta's open-source Llama2 series of large models, including multiple versions such as 7B, 13B, and 70B, were all trained based on a dataset of over 2 trillion tokens

On October 16th, it was reported that the IDEA Research Institute (Guangdong Hong Kong Macao Greater Bay Area Digital Economy Research Institute) CCNL Fengshen List team has opened up the Chinese base model Ziya2-13B-Base and its dialogue model Ziya2-13B-Chat, both of which are completely free and commercially available, and have made their debut on the ModelScope of the magic community.

Three months ago, Meta's open-source Llama2 series of large models, including multiple versions such as 7B, 13B, and 70B, were all trained based on a dataset of over 2 trillion tokens. On the basis of Llama2-13B, the Fengshen Bang team conducted further training on the 650Btokens self built high-quality Chinese and English dataset, and ultimately trained the Ziya2-13B series model to compensate for the insufficient Chinese language ability of Llama2.

In May of this year, the Fengshen Bang team launched the Ziya-LLaMA-13B model based on the LLaMA generation and opened it up, quickly becoming the best Chinese base model in the LLaMA ecosystem. Compared to Ziya-LLaMA-13B, the initial training loss value of Ziya2-13B-Base is lower, the training speed is increased by 38%, and it also solves the problem of instability in the later stage of training.

The evaluation results show that the Ziya2-13B-Base model performs significantly better than Llama2-13B and Ziya-LLaMA-13B in downstream comprehension tasks such as Chinese, English, mathematics, and code.

Relying on the powerful basic capabilities of Ziya2-13B-Base, the Fengshenbang team optimized the training strategy for the SFT stage. Based on the Ziya2-13B-Base model pre trained with 300B tokens, a dialogue model named Ziya2-13B-Chat was trained using approximately 400000 instruction samples and 8K context windows. In addition, reinforcement learning training was conducted on reward models trained on tens of thousands of high-quality human preference data for various Q&A, writing, and security tasks, making the output of the Ziya2-13B-Chat model more in line with human preferences and with higher security.

The evaluation results show that the Ziya2-13B-Chat model and the Ziya-LLaMA-13B-v1.1 model achieved a 66.5% success rate in side by side evaluation, and a 58.4% success rate compared to the version before human feedback reinforcement learning.

The Magic Community has developed an experience interface based on the Ziya2-13B-Chat model, which ordinary users can directly experience or use; Magic official account also launched a best practice tutorial to advance the deployment, reasoning and fine-tuning of the model for developers' reference.

As an important partner of the Magic Building community, the Fengshen Bang team's open-source series of large models all use Magic Building as their debut platform, and these models are also widely popular among the developer community.

Alibaba Cloud Magic is the largest and most active AI model community in China, with over 1200 high-quality AI models contributed by more than 30 leading AI institutions. It provides one-stop model experience, download, inference, optimization, customization, and other services. The total model load has exceeded 85 million times.

Disclaimer: The content of this article is sourced from the internet. The copyright of the text, images, and other materials belongs to the original author. The platform reprints the materials for the purpose of conveying more information. The content of the article is for reference and learning only, and should not be used for commercial purposes. If it infringes on your legitimate rights and interests, please contact us promptly and we will handle it as soon as possible! We respect copyright and are committed to protecting it. Thank you for sharing.(Email:[email protected])

Mobile advertising space rental

Tag: IDEA Research Institute Ziya2-13B Big Model Launches Open Source