Small language model
Small language models (SLMs) or compact language models are artificial intelligence language models designed for human natural language processing including language and text generation. Unlike large language models (LLMs), small language models are much smaller in scale and scope.
Typically, an LLM's number of training parameters is in the hundreds of billions, with some models even exceeding a trillion parameters. The size of any LLM is vast because it contains a large amount of information, which allows it to generate better content. However, this requires enormous computational power, making it impossible for an individual to train a large language model using just a single computer and GPU.
Small language models, on the other hand, use far fewer parameters, typically ranging from a few million (M) to a few billion (B). This make them more feasible to train and host in resource-constrained environments such as a single computer or even a mobile device.[1][2][3][4]
Most contemporary (2020s) small language models use the same architecture as a large language model, but with a smaller parameter count and sometimes lower arithmetic precision. Parameter count is reduced by a combination of knowledge distillation and pruning. Precision can be reduced by quantization. Work on LLMs mostly translate to SLMs: pruning and quantization are also widely used to speed up LLMs. Some notable models are:[2]
- Below 1B parameters: Llama-Prompt-Guard-2-22M (detects prompt injection and jailbreaking, based on DeBERTa-xsmall), SmolLM2-135M, SmolLM2-360M
- 1–4B parameters: Llama3.2-1B, Qwen2.5-1.5B, DeepSeeek-R1-1.5B, SmolLM2-1.7B, SmolVLM-2.25B, Phi-3.5-Mini-3.8B, Phi-4-Mini-3.8B, Gemma3-4B; closed-weights ones include Gemini Nano
- 4–14B parameters: Mistral 7B, Gemma 9B, Phi-4 14B.
(Phi-4 14B is marginally "small" at best, but Microsoft does market it as a small model. It has good capabilities for its size.)[5]
See also
[edit]References
[edit]- ^ Rina Diane Caballar (31 October 2024). "What are small language models?". IBM.
- ^ a b John JOhnson (25 February 2025). "Small Language Models (SLM): A Comprehensive Overview". Huggingface.
- ^ Kate Whiting. "What is a small language model and how can businesses leverage this AI tool?". The World Economic Forum.
- ^ "SLM (Small Language Model) with your Data". Microsoft. 11 July 2024.
- ^ "Introducing Phi-4: Microsoft's Newest Small Language Model Specializing in Complex Reasoning | Microsoft Community Hub". TECHCOMMUNITY.MICROSOFT.COM.
 | This statistics-related article is a stub. You can help Wikipedia by expanding it. |
 | This article about natural language processing is a stub. You can help Wikipedia by expanding it. |