Aller au contenu

Local LLM (Built-in) Setup for Askimo App

Ce contenu n’est pas encore disponible dans votre langue.

Run local models entirely inside Askimo, no Ollama, LM Studio, or LocalAI install required. Askimo downloads a platform-matching llama-server binary and lets you search, download, and run Hugging Face GGUF models directly from the app.

Using Askimo with a team?

There are two ways your team can access AI through Askimo today:

Available now

Bring your own key

Each team member adds their own API keys in Askimo. Keys are encrypted and stored locally on their machine. Best for individuals and small groups who want full independence.

Available now

Shared AI Gateway

Run an OpenAI-compatible gateway on a shared server. Each user connects Askimo with just a URL, with no personal API keys required. The real keys stay on the server with central spend control. See an example setup using LiteLLM.

Coming soon

Askimo Team

A native team edition with shared workspaces and collaboration features is in development. Learn more.

  • Server URL: Managed automatically by Askimo
    • Default: http://127.0.0.1/v1 (local reverse proxy in front of the running llama-server processes)
  • API Key: Not required
  • Available Models: Any .gguf chat or embedding model you download from Hugging Face inside Askimo
  1. Open Settings > Local LLM
  2. Flip the engine toggle on. Askimo downloads the llama-server binary that matches your OS and architecture.
  3. Go to the Models section and search Hugging Face for a GGUF model, or pick one from the default most-downloaded list
  4. Click download on the model you want. It’s saved to Askimo’s local models folder.
  5. Navigate to the “AI Providers” tab and select “Local LLM (built-in)”
  6. Pick the downloaded model from the model dropdown
  7. Click “Test Connection” to verify
Askimo Settings > Local LLM page with the llama-server engine toggle switched on
  1. Click on the menu bar
  2. Select “Settings”
  3. Navigate to the “AI Providers” tab
  4. Select “Local LLM (built-in)” from the provider list

Keyboard Shortcut: ⌘ + , (macOS) or Ctrl + , (Windows/Linux) then click “AI Providers”

Settings > Local LLM > Models shows Hugging Face’s most-downloaded GGUF repos by default, with a search box for anything more specific. Each result lists the repo, file name, size, downloads, and likes pulled live from the Hugging Face API.

Chat models: any .gguf chat model works through llama-server, so there’s no curated list to pick from, search and download whatever fits your use case.

Embedding models: Askimo ships two vetted options, nomic-embed-text and Qwen3-Embedding, because getting the pooling strategy wrong for an embedding model silently breaks results rather than just degrading quality.

Once downloaded, a model shows up in the “Local LLM (built-in)” provider’s model picker right away, no restart needed.

Askimo Hugging Face GGUF model browser showing search results with download counts and likes

Askimo manages the llama-server process directly, so the full set of llama-server runtime flags is exposed in Settings > Local LLM instead of being hidden behind a wrapper:

  • Context size (--ctx-size): cap or raise how much conversation history and RAG context a model can hold
  • GPU offload (--n-gpu-layers): push more or fewer transformer layers onto the GPU
  • KV-cache quantization (--cache-type-k, --cache-type-v): quantize the key/value cache (e.g. q8_0 instead of f16) to cut memory use on long contexts
  • Batching (--batch-size, --ubatch-size, --parallel, --cont-batching): tune prompt-processing throughput and concurrent requests
  • Memory behavior (--no-mmap, --mlock): control whether the model is memory-mapped or pinned in RAM
  • RoPE scaling (--rope-freq-base, --rope-freq-scale): adjust rotary position embedding parameters for extended-context models

Engine Won’t Start?

  • Check your internet connection; the first toggle downloads the llama-server binary
  • Verify you have enough disk space for the binary and models
  • Try toggling the engine off and back on

Model Not Loading?

  • Confirm the .gguf download finished in the Models section
  • Verify sufficient RAM/VRAM for the model size and quantization
  • Try a smaller or more quantized model first

Cannot Connect?

  • Ensure the engine toggle is on in Settings > Local LLM
  • Confirm a model is selected in the “Local LLM (built-in)” provider settings
  • Test with curl http://127.0.0.1/v1/models

Slow Performance?

  • Increase --n-gpu-layers if a GPU is available
  • Use a more quantized model (e.g. Q4_K_M instead of f16)
  • Lower --ctx-size if you don’t need a large context window
  • Enable KV-cache quantization (--cache-type-k, --cache-type-v)

“Context Length Exceeded” Errors?

  • Raise --ctx-size in Settings > Local LLM for the model
  • Note that larger context windows use significantly more VRAM; increase gradually and watch for crashes