Local LLMs (Experimental)

SmartCursorX supports running quantized GGUF models locally using a bundled llama-server binary. This enables fully offline AI inference with no API costs and no data leaving your machine.

Quick Start

1. Open Settings → Local LLMs (Exp)
2. Search for a GGUF model on HuggingFace
3. Click Browse files on a model repo
4. Click Download on a GGUF file
5. Click Run next to the downloaded model
6. Chat using the local provider

Model Search

The built-in HuggingFace search lets you find GGUF models:

• Searches huggingface.co/api/models with ?tags=gguf filter
• Returns up to 15 results sorted by downloads
• Shows model ID, description, and download count
• Click Browse files to see available GGUF variants in a repo

HuggingFace Token (Experimental)

Some models (Llama, Mistral, etc.) are gated and require authentication:

1. Get a token at huggingface.co/settings/tokens
2. Enter it in the HuggingFace Token (Exp) field in the Search section
3. Token is stored securely via secureStore
4. Automatically passed as Authorization: Bearer header on download

Downloading Models

• Downloads are streamed with progress display
• Files are saved to %APPDATA%\smart-cursor-x\models\
• Partial downloads are NOT automatically cleaned up on failure
• Downloaded models show a green Downloaded badge in search results

Server Setup

The bundled llama-server binary powers local inference:

Binary Location

The download script (download-llama-server.ps1) installs to:

%APPDATA%\smart-cursor-x\bin\llama-srv.exe

The server searches in this order:

1. %APPDATA%\smart-cursor-x\bin\llama-srv.exe
2. %APPDATA%\smart-cursor-x\bin\llama-server.exe
3. resources\llama-srv.exe (packaged app)
4. resources\llama-server.exe (packaged app)

Server Parameters

When started, the server runs with:

• Host: 127.0.0.1 (localhost only — no external access)
• Port: 8080
• GPU layers: 0 (CPU only by default)
• Context: 2048 tokens

Starting & Stopping

• Click Run next to a downloaded model — button shows "Starting..." immediately
• Server status is displayed with green indicator
• Click Stop to shut down the server
• Server auto-stops on app quit (via will-quit handler)

Provider

The local provider routes chat requests to http://localhost:8080/v1/chat/completions (OpenAI-compatible format):

• No API key required
• Model list auto-populates from downloaded GGUF files
• Zero cost for cost estimation

Architecture

User Chat → AIService.createLanguageModel('local')
  → createOpenAICompatible({ baseURL: 'http://localhost:8080/v1' })
    → llama-server on 127.0.0.1:8080
      → GGUF model file

Troubleshooting

Antivirus False Positives

Norton and Windows Defender may flag llama-server.exe as IDP.Generic. The binary is renamed to llama-srv.exe to minimize this, but exclusions may still be needed.

Missing Binary

If findLlamaServer() returns null, the server binary may be missing or quarantined. Run npm run fetch:llama or check antivirus quarantine.

Server Won't Start

• Process exits immediately: Missing DLLs or corrupted binary — re-download
• Port in use: Another service on port 8080 — stop the conflicting service
• Access denied: Antivirus blocking — add exclusion

Chat Fails with ECONNREFUSED

The server isn't running or crashed. Check if the green "Server Active" indicator is visible. If not, click Run next to your model.