Upstage introduces Solar Mini, a compact open-source language model

Upstage introduced Solar Mini, a compact pretrained large language model built with a scaling method the company calls depth up-scaling. The model is publicly available under the Apache 2.0 license and can be tried through Hugging Face and Poe, according to the announcement.

Upstage said Solar Mini reached the top of Hugging Face’s Open LLM Leaderboard in December 2023. The company also said the model uses fewer parameters while producing responses comparable to GPT-3.5 at 2.5 times the speed. Upstage reported that Solar Mini surpassed Llama 2, Mistral 7B, Ko-Alpaca and KULLM across a range of benchmarks, though the announcement did not provide the underlying scores in its text.

Architecture and training

Solar Mini is based on a 32-layer Llama 2 architecture and initialized with pretrained weights from Mistral 7B, according to Upstage. The company’s depth up-scaling method consists of depthwise scaling followed by continued pretraining.

Upstage said depth up-scaling enlarges smaller models without requiring the additional modules or dynamic operation associated with mixture-of-experts methods. The company said the method works with commonly used large-language-model frameworks such as Hugging Face and can be applied to transformer architectures.

The initial depth up-scaling step reduces the scaled model’s performance relative to the base model, according to Upstage. The company therefore applies continued pretraining to recover performance after scaling.

Solar Mini then undergoes instruction tuning specifically for Korean. Upstage said this stage trains the model to follow instructions presented in a question-and-answer format. An alignment-tuning stage follows, training the instruction-tuned model to align with preferences from humans or more powerful AI systems.

Retrieval and document workflows

Upstage said Solar Mini works particularly well with retrieval-augmented generation systems. According to the company, the model uses retrieved information to improve the precision and relevance of its output.

The announcement also described how Solar Mini can be used alongside Upstage’s optical character recognition and layout-analysis models. Those models extract tables and figures from PDF, PNG and JPG files. Upstage said the system serializes document elements according to their reading order and converts the result into HTML that can be supplied to the language model.

The company presented the compact model as requiring less computation than larger language models. Upstage said smaller models can improve responsiveness, require less work to customize for particular domains or services, and support deployment on local devices with less dependence on extensive GPU resources.

Source: Upstage’s “Introducing Solar Mini: Compact yet Powerful” announcement, published January 25, 2024.