LLM Checker: Hardware-Aware Model Selection for Local Inference

Hook

There are over 33,000 LLM model variants available for local inference. If you've spent an hour downloading a 70B parameter model only to have it crash with an out-of-memory error, you've experienced the problem LLM Checker was built to solve in under 30 seconds.

Context

The local LLM ecosystem has exploded over the past two years. What started as a handful of quantized LLaMA models on Hugging Face has ballooned into a fragmented universe spanning Ollama's curated registry, GPT4All's optimized builds, and thousands of community GGUF files. Each model comes in multiple quantization levels (Q4_K_M, Q5_K_S, Q8_0), and each quantization has different memory footprints, speed characteristics, and quality trade-offs.

For practitioners, this abundance creates paralysis. You have an M2 MacBook with 16GB of unified memory—should you run Llama 3 8B at Q8, Mistral 7B at Q6, or Phi-3 Medium at Q4? Will your RTX 3090's 24GB VRAM fit a 34B model at Q5, or should you drop to Q4? The math isn't straightforward: different architectures have different memory overhead, context windows multiply KV cache requirements, and Mixture-of-Experts models break the simple parameters-to-bytes formula. Before LLM Checker, you either spent hours reading model cards and doing manual calculations, or you guessed and wasted bandwidth downloading models that wouldn't fit. LLM Checker automates the entire decision tree—it detects your GPU VRAM, CPU RAM, and acceleration backends (CUDA, ROCm, Metal), queries a prepackaged catalog of 229 models expanded from 33,000 registry artifacts, and outputs ranked recommendations with direct installation commands.

Technical Insight

LLM Checker's architecture is deceptively simple: a Node.js CLI that orchestrates hardware detection, a SQLite catalog, and a scoring engine. The genius is in the calibration. Hardware detection works by shelling out to platform-specific utilities and parsing their output. On macOS, it calls system_profiler SPHardwareDataType and sysctl hw.memsize. On Linux with NVIDIA GPUs, it parses nvidia-smi --query-gpu=memory.total --format=csv. For AMD, it tries clinfo to enumerate OpenCL devices. This approach is pragmatic but fragile—it assumes these utilities exist and maintain stable output formats. Here's the detection flow for VRAM:

// Simplified from actual implementation
async function detectVRAM() {
  if (process.platform === 'darwin') {
    const output = execSync('system_profiler SPDisplaysDataType').toString();
    const match = output.match(/VRAM.*?(\d+)\s*(GB|MB)/);
    return match ? parseMemory(match[1], match[2]) : 0;
  }
  if (await hasCommand('nvidia-smi')) {
    const output = execSync('nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits').toString();
    return parseInt(output.trim());
  }
  // Fallback: assume integrated graphics
  return 0;
}

The model catalog is where things get interesting. LLM Checker ships with a 229-model SQLite database that's been curated from Ollama's API, but it doesn't stop there. The sync command pulls fresh metadata and expands each base model into all its quantization variants by cross-referencing Hugging Face GGUF repositories and GPT4All's catalog. Each model entry includes parameter count, default quantization, estimated size in bytes, and installation metadata (registry source, pull command, model file path).

The memory estimation formula is calibrated against real-world Ollama downloads. For standard dense models, it uses parameters * bytes_per_param * quantization_multiplier + context_window * kv_cache_overhead. The v3.7.0 release included a critical fix for Mixture-of-Experts models—early versions counted total parameters (e.g., Mixtral's 46.7B), leading to false negatives where the tool said "won't fit" when only 12.9B active parameters were actually loaded. The corrected formula now checks the architecture metadata:

function estimateModelSize(model) {
  const baseParams = model.isMoE 
    ? model.activeParameters 
    : model.totalParameters;
  
  const quantMult = {
    'Q4_K_M': 0.55,
    'Q5_K_S': 0.69,
    'Q8_0': 1.0
  }[model.quant] || 0.55;
  
  const modelBytes = baseParams * quantMult * (model.precision / 8);
  const kvCacheBytes = model.contextWindow * model.hiddenSize * 2 * 0.5; // FP16 KV cache
  const overhead = modelBytes * 0.1; // Runtime overhead
  
  return modelBytes + kvCacheBytes + overhead;
}

The scoring engine is deterministic and multidimensional. Each model gets four scores: Quality (derived from popularity metrics like pull counts and community ratings), Speed (tokens/sec estimate based on parameter count and quantization), Fit (how much headroom remains after loading), and Context (context window size normalized to 128k). These are weighted and combined into a final score, with Fit getting higher weight when available memory is tight. This encoding of trade-offs is what makes the tool useful—on a 16GB Mac, a 3B model at Q8 might outscore a 13B model at Q4 because the speed penalty from the smaller model is offset by the quality gain from higher quantization.

The Model Context Protocol integration exposes all of this as structured tools. LLM Checker implements an MCP server that Claude Code (or any MCP client) can call. Instead of typing llm-checker scan --json, you tell Claude "check what models I can run," and it invokes the scan tool, parses the JSON response, and presents recommendations conversationally. The MCP server definition looks like:

const tools = [
  {
    name: 'scan',
    description: 'Detect hardware and recommend models',
    inputSchema: {
      type: 'object',
      properties: {
        minQuality: { type: 'number' },
        maxSize: { type: 'number' },
        runtimeFilter: { type: 'string', enum: ['ollama', 'gpt4all', 'gguf'] }
      }
    }
  },
  {
    name: 'calibrate',
    description: 'Generate routing policy by benchmarking models',
    inputSchema: {
      type: 'object',
      properties: {
        categories: { type: 'array', items: { type: 'string' } },
        promptSuite: { type: 'string' }
      },
      required: ['categories']
    }
  }
];

The calibration subsystem deserves special attention. Running llm-checker calibrate --categories "code,chat,reasoning" downloads a set of recommended models, runs a prompt suite against each ("write a Python function," "explain quantum entanglement," "solve this logic puzzle"), measures tokens/sec and subjective quality scores, and outputs a YAML routing policy mapping categories to optimal models. This turns model selection from guesswork into versioned configuration. You can commit the policy to git, share it with your team, and reproduce the exact model routing across environments.

Gotcha

The biggest limitation is the scoring formula's opacity. You can't override the Quality calculation, and it appears to weight popularity (pull counts) heavily, which conflates hype with performance. A model with aggressive marketing will score higher than a technically superior but less-known alternative. There's no way to inject custom benchmark results or specify "I care more about reasoning than speed." The weights are hardcoded.

Hardware detection is brittle by design. Parsing nvidia-smi output works until NVIDIA changes the format (which happens across driver updates). Worse, the tool assumes that if a GPU appears in the utility output, it's fully functional—but you might have a CUDA-capable GPU with broken drivers or mismatched library versions. The tool doesn't do runtime probing (actually trying to allocate VRAM), so it can't catch these failures. The calibration subsystem only works end-to-end for Ollama models. You can calibrate against vLLM or llama.cpp in theory, but the tool won't actually run inference on those runtimes—it just generates placeholder routing policies. This makes the multi-runtime targeting feel half-baked. Finally, the registry sync has no rate limiting or authentication. Running sync repeatedly hammers Ollama's API, and there's no caching layer to avoid re-downloading unchanged catalog data. If Ollama decides to rate-limit aggressive clients, this tool will break.

Verdict

Use if: You're running local LLMs on consumer hardware (M-series Macs, gaming GPUs, workstation builds) and you're tired of the guess-and-download cycle. The tool excels at answering "what should I run right now" in under 30 seconds. The MCP integration makes it essential if you're using Claude Code and want conversational model management. The calibration feature is a strong differentiator for teams that need reproducible model selection policies. Skip if: You're deploying at scale on cloud instances where model choice is cost-driven rather than memory-constrained, or you need fine-grained control over quantization schemes that Ollama doesn't expose. The opaque scoring formula and lack of extensibility make it unsuitable if you have strong opinions about benchmarking methodology. If you're already comfortable with llama.cpp's model probing or you've built custom tooling around vLLM, LLM Checker won't add much value—it's optimized for practitioners who want automation over control.