Small language model

Small language models (SLM) or compact language models are artificial intelligence language models designed for human natural language processing including language and text generation. Small language models typically have less than forty billion parameters. This make them feasible to train or host entirely on consumer electronics such as personal computers, laptops, or smart devices. In contrast, a large language model typically contains hundreds of billions of training parameters, with some models exceeding a trillion parameters. This enables the model to encode vast amounts of information, improving the generalizability and accuracy of its outputs. However, training and hosting such models demands enormous computational resources, rendering it infeasible for an individual to do so using a single computer and graphics processing unit. Most contemporary small language models use the same architecture as a large language model, but with a smaller parameter count and sometimes lower arithmetic precision. Similar to LLMs, SLMs are optimized via knowledge distillation, pruning, and quantization.