Name the constrained resource
Efficiency research trades among memory capacity, memory bandwidth, compute, latency, throughput, energy, and model quality. Improving one can worsen another. Batching increases throughput but may hurt interactive latency. Longer contexts consume cache memory. Offloading fits a model but can make token generation too slow. A useful claim identifies the target workload and bottleneck.
Mental model: optimization moves a system along a quality–cost frontier. Ask whether the new point is better under the same constraints, not whether one metric improved in isolation.
Quantization changes numerical precision
Quantization stores weights—and sometimes activations or caches—with fewer bits. Weight-only methods reduce model memory, while activation quantization can unlock faster kernels but is harder to calibrate. Reported bit width does not capture metadata, scales, untouched layers, or runtime buffers, so measured peak memory matters more than a label.
Quality loss is task-dependent. Average benchmark scores can hide failures in reasoning, rare languages, structured output, or long context. Compare quantized and full-precision models with identical prompts and decoding settings on the actual workload.
Adapters change fewer parameters
Low-rank adapters and related parameter-efficient methods update a small set of trainable weights while leaving the base model fixed. This reduces training storage and makes it easier to maintain task variants. It does not eliminate base-model inference cost. Adapter rank, target layers, data quality, and merging strategy all affect the result.
Distillation changes the model
Distillation trains a smaller student to match a teacher’s outputs or internal behavior. It may compress useful behavior, but it also inherits teacher errors and depends heavily on the transfer data. A fair comparison distinguishes gains from architecture, synthetic data volume, and teacher strength.
Latency has a shape
Interactive generation has at least two phases: processing the input and producing output tokens. Time to first token and steady generation speed answer different questions. Report prompt length, output length, batch size, concurrency, warm-up, kernel versions, and hardware. Median latency alone can hide pauses caused by memory pressure or scheduling.
Local is a deployment property
A model is local when required inference data stays on controlled hardware, but privacy still depends on telemetry, update channels, logs, embeddings, and external tools. Offline operation should be tested by disabling the network. Package size, startup time, thermal throttling, battery use, and platform support matter for real devices.
Questions to ask when reading a paper
- What is held constant? Model, data, sequence length, batch size, hardware, and quality target?
- Which memory is counted? Weights, activations, cache, runtime workspace, and host memory?
- Are kernels available? Theoretical operations do not guarantee wall-clock speed on target hardware.
- Where does quality fall? Per task, language, context length, and output format?
- What does “local” include? Inference, retrieval, speech, telemetry, and tool calls?
A fair local benchmark
Choose representative prompts and outputs, fix decoding settings, warm the system, and record peak memory, time to first token, generation rate, energy if available, and failures. Run the same harness against a simple baseline. Evaluate answer quality blind to configuration. Repeat at several context lengths because cache growth can change the bottleneck.
Bottom line
Efficient AI is credible when it reports the entire operating point: quality, hardware, memory, latency, throughput, and workload. Local deployment is valuable, but only after the boundaries of “local” and the cost of compression are made explicit.