Local LLM Configuration

Connect Mythos.XYZ Chat to your local AI models

Mythos.XYZ Chat can connect to local LLM servers like LM Studio, Ollama, or llama.cpp. The challenge is CORS. Browsers block requests from web apps to local servers. The fix is a reverse proxy that adds CORS headers.

The Problem

Mythos.XYZ Chat runs in your browser. When it tries to call http://localhost:1234 (LM Studio) or http://localhost:11434 (Ollama), the browser blocks it. This is a security feature called same-origin policy.

The Solution: Caddy

Caddy is a web server that handles CORS automatically. It sits between Mythos.XYZ Chat and your local LLM.

Handles CORS

Adds the right headers so browsers allow the connection.

Simple Config

One file. A few lines. Done.

HTTPS Optional

Can add SSL if needed. Automatic with Caddy.

Fast

Minimal overhead. Your LLM does the heavy lifting anyway.

Option 1: LM Studio (Recommended)

LM Studio is a GUI app for running local models. It has OpenAI-compatible API built in. The easiest path for desktop users.

1. Download LM Studio

Get it from lmstudio.ai. Available for macOS, Windows, and Linux.

2. Download a Model

Open LM Studio, search for a model (try Llama 3.2 or Phi-3), and download it.

3. Enable Local Server

In LM Studio: Go to the "Local Server" tab (left sidebar). Click "Start Server". Note the port (default: 1234).

4. Install Caddy

# macOS
brew install caddy

# Ubuntu/Debian
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update && sudo apt install caddy

# Windows (with Scoop)
scoop install caddy

5. Create Caddyfile

Create a file named Caddyfile (no extension) in your project directory:

:8080 {
    respond /health "OK" 200
    
    handle /v1/* {
        reverse_proxy localhost:1234
    }
}

6. Run Caddy

caddy run --config Caddyfile

7. Configure Mythos.XYZ Chat

In Mythos.XYZ Chat settings:

API Base URL: http://localhost:8080/v1
API Key: lm-studio (required but ignored)
Model: The model name from LM Studio (or any name works)

Option 2: Ollama

Ollama is a command-line tool for running local models. Great for servers and headless setups.

1. Install Ollama

# macOS/Linux
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull llama3.2

2. Install Caddy

See installation steps in Option 1 above.

3. Create Caddyfile

:8080 {
    respond /health "OK" 200
    
    handle /v1/* {
        reverse_proxy localhost:11434
    }
}

4. Run Caddy

caddy run --config Caddyfile

5. Configure Mythos.XYZ Chat

API Base URL: http://localhost:8080/v1
API Key: ollama (required but ignored)
Model: llama3.2 (or whatever you pulled)

Option 3: llama.cpp Server

llama.cpp is the raw C++ implementation. Maximum control, minimum hand-holding.

1. Build llama.cpp with server

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make llama-server

2. Run the server

./llama-server -m path/to/model.gguf --host 127.0.0.1 --port 8081

3. Caddyfile for llama.cpp

:8080 {
    respond /health "OK" 200
    
    handle /v1/* {
        reverse_proxy localhost:8081
    }
}

4. Configure Mythos.XYZ Chat

API Base URL: http://localhost:8080/v1
API Key: anything (required but ignored)
Model: any name (required but llama.cpp ignores it)

Troubleshooting

Connection Refused

Make sure your LLM server is running. Check the port matches.

CORS Errors

Caddy should handle this. Make sure you hit :8080, not the LLM port directly.

Slow Responses

Local models need RAM and CPU. Try a smaller model or quantization.

API Key Required

Mythos.XYZ Chat requires an API key. Use any string for local servers.

Why Not Direct Connection?

Browsers enforce same-origin policy. A web app at https://alpha.mythos.xyz cannot make requests to http://localhost:1234. The browser blocks it before it even reaches your LLM. A proxy on localhost:8080 adds CORS headers, which tells the browser "this is okay."

Questions?

Ask in the community

Join Community