The Scale of Small: Parameter Count, Memory Footprint, and the Frontier of On-Device Language Models

Abstract

This paper surveys the relationship between large language model (LLM) parameter count and physical memory footprint, and examines how this relationship determines where models can run, from flagship smartphones down to resource-constrained microcontrollers. We review two influential open-source projects, llama2.c and llama.cpp, as case studies in minimalist and production-grade local inference respectively, and discuss current and emerging directions in small-model deployment, including knowledge distillation, aggressive quantization, and agentic on-device systems.

1. Introduction

For most of the modern LLM era, model capability was described almost exclusively in terms of parameter count: the number of individually learned weights inside a neural network. Terms like "7B" or "70B" became shorthand for a model's position in the capability hierarchy. However, parameter count alone does not determine where a model can be deployed. What matters for deployment is the model's memory footprint, which depends jointly on parameter count and numerical precision.

This paper traces that relationship explicitly, then applies it to a practical question of increasing relevance: as consumer hardware diversifies (phones, tablets, embedded boards, microcontrollers), how far down the hardware stack can a working language model actually go, and what does "working" mean at each tier?

2. Parameters: Definition and Role

A parameter is a single learned numerical weight within a neural network, adjusted during training (typically via gradient descent) so that the network's collective weights produce useful predictions. Parameters are distributed across a model's structural components:

• Embedding layers, which convert tokens into numerical vector representations
• Attention layers, which determine which parts of the input context are relevant to predicting the next token
• Feed-forward layers, which perform the bulk of per-layer computation

A model's parameter count is the sum of all weights across all such layers, repeated across however many layers the architecture stacks.

2.1 Active vs. Total Parameters

Not all architectures use every parameter for every prediction. Mixture-of-Experts (MoE) models maintain a large total parameter count but activate only a subset of parameters per token. For example, DeepSeek-V4-Pro has approximately 1.6 trillion total parameters but activates only around 49 billion (roughly 3%) per token. This distinction matters practically:

• Total parameters determine storage and memory footprint (the full model must be held in memory even if unused per-token)
• Active parameters determine per-token compute cost, and thus inference speed and energy use

3. From Parameters to Memory Footprint

A model's memory footprint is the product of its parameter count and the number of bytes used to represent each parameter:

Memory footprint = parameter count × bytes per parameter

Bytes per parameter is determined by numerical precision:

Applying this formula across representative model scales (weights only; the KV cache and activations add to this at runtime):

This table illustrates the central practical insight of quantization: the same model, defined by a fixed parameter count, can occupy radically different amounts of memory depending purely on how its weights are stored, with no change to the underlying learned knowledge.

4. Capability Is Not a Pure Function of Parameter Count

While larger models tend to have greater representational capacity, training data quality and methodology substantially modulate capability independent of raw parameter count. Recent small models (e.g., Qwen2.5-0.5B, the SmolLM2 family, Microsoft's Phi series) can match or exceed the performance of older models an order of magnitude larger, largely attributable to improved data curation and training technique rather than scale. This has been termed, in recent literature, evidence that models under 1 billion parameters can outperform older models ten times their size.

This finding does not imply general parity between small and large models. Rather, small models have closed specific, narrow capability gaps (simple instruction-following, formatting, short-form Q&A), while general reasoning depth, broad world knowledge, and long-context understanding remain areas where scale continues to confer substantial advantage.

5. Deployment Tiers: Where Models Actually Run

Mapping memory footprint against available hardware RAM yields a practical tiering of deployment targets:

Two commercial examples anchor the smartphone tier: Apple Intelligence's on-device model (approximately 3B parameters) and Google's Gemini Nano (approximately 2B parameters), both of which run on dedicated neural accelerator hardware built into their respective chips, rather than general-purpose CPU cores.

6. Case Studies in Local Inference

6.1 llama2.c

Created by Andrej Karpathy in July 2023, llama2.c is a minimalist implementation of the Llama 2 architecture: models are trained from scratch in PyTorch and exported to a binary format loaded by a several-hundred-line pure C inference engine with no external dependencies. Its released checkpoints ("tinyllamas") range from 15M to 44M parameters, trained on the TinyStories dataset of simple children's narratives, illustrating the principle that small models can perform strongly within a sufficiently narrow domain. The project is explicitly framed by its author as a minimalism and education exercise rather than a production tool.

Repository: github.com/karpathy/llama2.c

6.2 llama.cpp

Created by Georgi Gerganov beginning in March 2023, building on his earlier whisper.cpp project, llama.cpp is a general-purpose, production-oriented inference engine written in C/C++, co-developed with the GGML tensor library. Unlike llama2.c, it is architecture- and scale-agnostic, supporting over 50 model architectures from tens of millions to hundreds of billions of parameters, and is the originating implementation of the now-widely-adopted GGUF quantized model format. llama.cpp is the inference backbone underlying numerous downstream tools, including Ollama, LM Studio, GPT4All, and Jan. The project surpassed 100,000 GitHub stars faster than any prior open-source AI project, and in February 2026 its core team joined Hugging Face while retaining full technical and community autonomy over the project.

Repository: github.com/ggml-org/llama.cpp

6.3 Comparative Summary

7. Emerging Directions

Several trends are shaping the next phase of small-model deployment:

• Knowledge distillation at scale: using large "teacher" models to train compact "student" models, compressing learned reasoning into far fewer parameters (e.g., reported gains from Llama 3.2's 3B model approaching capability levels previously requiring 10-20× the parameters).
• Agentic small models: a shift from passive question-answering toward models that take actions (navigating interfaces, controlling devices), argued in recent research to be a better architectural fit for small, specialized models than for general-purpose giants.
• Deeper quantization: formats such as llama.cpp's Q1_0 (introduced 2026) pushing some models below 1 GB at sub-INT4 precision.
• On-device safety layers: lightweight guardrail systems designed to run alongside small on-device models, driven partly by regulatory pressure such as the EU AI Act.
• Purpose-built silicon: chips increasingly shipping with accelerator hardware tuned specifically for small-model inference workloads, co-evolving with the software side of the field.

It is worth noting that public discourse around "tiny LLMs" includes a substantial volume of low-rigor marketing content with unverifiable product claims. Conclusions in this area should be weighted toward findings with reproducible benchmarks and identifiable, auditable sources.

8. Conclusion

The deployability of a language model is governed by a simple arithmetic relationship, parameter count multiplied by numerical precision, but the practical and research implications of that relationship are substantial. They explain why quantization research is as consequential as architecture research for real-world deployment, why small models trained well can rival older large models on narrow tasks, and why the hardware spectrum from smartphones down to microcontrollers now hosts a genuine, if capability-limited, tier of functioning language models. Projects such as llama2.c and llama.cpp mark two ends of this spectrum: one a minimalist proof of concept, the other the infrastructural backbone of the current local-AI ecosystem.

References (Selected)

1. Karpathy, A. llama2.c. github.com/karpathy/llama2.c
2. Gerganov, G. llama.cpp. github.com/ggml-org/llama.cpp
3. Eldan, R. & Li, Y. (2023). TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
4. Hugging Face Blog (2026). GGML and llama.cpp join HF to ensure the long-term progress of Local AI.
5. DeepSeek-AI. DeepSeek-V4 Technical Report.