Running on-device models without cloud latency

Dimlit executes natural language parsing and memory recall directly on the phone’s own AI accelerator. By avoiding cloud roundtrips, requests resolve with immediate tactile responsiveness while zero personal data leaves your device boundary.

Quantization and memory footprint#

Deploying transformer models to mobile hardware requires balancing inference speed, memory bandwidth, and battery consumption. Uncompressed floating-point weights quickly saturate thermal budgets and mobile DRAM.

Through uniform INT4 quantization, weights occupy roughly a quarter of their original memory footprint, allowing the on-device accelerator to maintain high memory throughput during token generation:

interface NeuralModelConfig {
  architecture: 'transformer-decoder';
  quantization: 'int4-symmetric';
  contextWindow: 4096;
  vocabularySize: 32000;
  computeTarget: 'accelerator' | 'gpu' | 'cpu';
}

export const activeRuntimeConfig: NeuralModelConfig = {
  architecture: 'transformer-decoder',
  quantization: 'int4-symmetric',
  contextWindow: 4096,
  vocabularySize: 32000,
  computeTarget: 'accelerator',
};

Layer-wise compression benefits#

1. Reduced DRAM bandwidth: INT4 weights require less memory bus traffic per token generated.
2. Thermal predictability: Inference cycles finish within narrow power envelopes, preventing thermal throttling.
3. Instant activation: Model weights remain memory-mapped in flash storage for fast wake-up.

NOTE

All model weights execute entirely inside the operating system sandbox. No voice audio, transcribed tokens, or parsed metadata are transmitted off the physical hardware.

Execution pipeline#

When an instruction arrives from speech or text input, the execution pipeline coordinates several independent local stages:

• Token streaming: Input phonemes or text tokens pass through the acoustic tokenizer.
• Attention caching: Key-value pairs are stored in unified cache memory.
• Intent resolution: The decoder predicts structured slot values matching your daily schedule.

For more details on how these structured entities populate your schedule, refer to our overview on designing private daily briefs.