On-Device vs Cloud LLM: A Practical Benchmark

The Question

On-device LLMs promise privacy and zero cost. Cloud LLMs promise deeper reasoning and broader capabilities. But how do they actually compare on real engineering tasks?

I built a benchmark comparing Apple's on-device Foundation Models (~3B parameters) against three tiers of cloud models across three tasks: commit message generation, code review, and text classification.

Results

Surprising finding: The smallest cloud model (700ms) is faster than the on-device model (1647ms). Network latency is negligible compared to model size. On-device wins on privacy, not speed.

Task 1: Commit Message Generation

Input: A git diff adding retry logic with exponential backoff.

The on-device model produced a correct, structured commit message:

{
  "type": "fix",
  "title": "Added retry logic with exponential backoff",
  "body": "Introduces a retry mechanism with exponential backoff
           to handle transient connection errors.",
  "breaking": false
}

Accurate but generic. Larger cloud models identified the specific failure mode being addressed and suggested the commit title reference the payment processing context.

Verdict: On-device is sufficient for commit messages. The smaller model captures the "what" correctly; it misses some "why" context that larger models infer.

Task 2: Code Review

Input: A Python function with validation gaps and naming issues.

On-device catches the obvious functional bugs. Larger models find progressively more subtle issues.

Task 3: Text Classification

Input: A support ticket about payment processing failure.

All models correctly classified it as billing / high priority / negative sentiment. The differences were in action item quality — larger models produced more specific, actionable recommendations.

The Decision Framework

Don't pick one — use them together based on the task:

Key Takeaways

1. Privacy is a feature, not a limitation. On-device models trade capability for guaranteed data sovereignty. For proprietary code, this is non-negotiable.

2. Speed hierarchy is counterintuitive. The smallest cloud model is faster than the on-device model. Network latency matters less than model size.

3. Quality scales predictably with model size. Each tier catches progressively more subtle issues. Choose the tier that matches your accuracy requirement.

4. Structured output is the equalizer. With constrained schemas (anyOf, range, typed fields), even the smallest model produces valid, usable output. Schema design matters more than model size for extraction tasks.

5. Cost per call varies 60x. On-device is free. Large cloud models cost ~$0.015/call. For high-volume tasks (CI pipelines, auto-triage), this difference compounds.