Local Multimodal LLM

What It Is

Local Multimodal LLM is a Win64 Unreal Engine runtime plug-in for private, offline character dialogue. It embeds llama.cpp for text generation, exposes isolated multi-character sessions, and provides provider-neutral speech input and output components — all running on the player's own hardware.

See It In Action

A look at the plug-in driving character dialogue inside Unreal Engine.

Local inference. No APIs, no cloud, runs entirely on-device.

Included Capabilities

Everything ships as one plug-in — enable only what a project needs.

📦

Manifest-Driven Model Loading

GGUF model discovery via *.localllm.json manifests. One loaded model serves every character session — no duplicate loads per actor.

llama.cpp

🗣️

Streamed Multi-Character Sessions

Isolated, cancellable sessions per character with per-character KV-state reuse. Switching characters lazily saves/restores llama sequence state.

Session-Isolated

📜

Character & World Sheets

Structured character/world Data Assets, compacted conversation memory, and optional custom context — with configurable reasoning mode and generation limits.

Data Assets

💬

Guarded Sentence Streaming

Sentence-guarded streaming for subtitles and low-latency TTS, with jailbreak and immersion guards that sanitize or retry unsafe output before it reaches the player.

Guards

🔧

Constrained Tool Calls

Typed, allow-listed Unreal tool requests. The model never touches reflected functions or console commands directly — game code validates and executes every call.

Type-Safe

🤝

Relationship Evaluation

Optional trust/affinity-style relationship judging with developer-selected criteria and prompt mappings — previewed or applied at deliberate story boundaries.

Optional

🎤

Microphone & Speech Input

Push-to-talk or optional always-on VAD, noise-floor calibration, contextual vocabulary correction, and one-turn rollback for accidental captures.

STT

🔈

Provider-Neutral Speech Output

Native sherpa-onnx TTS in Shipping builds, with optional NeuTTS-2E and Chatterbox Editor/Development sidecars for reference-voice cloning during iteration.

TTS

Runtime Profiles

Two artifact sets, so distribution size scales with what a project actually ships.

Third-Party Footprint

Core

CPU + Vulkan + sherpa-onnx — ~80 MB. Broad Win64 distribution.

Full

Core + CUDA 12 runtime — ~1.0 GB. NVIDIA-optimized development.

Image / Audio Input

Optional projector support via libmtmd.

Speech-to-Text

Packaged Win64 sherpa-onnx + Parakeet.

Quick Start

The full walkthrough lives in the User Guide — this is the shape of it.

Copy LocalMultimodalLLM into the project's Plugins directory.
Keep a matching llama.cpp core runtime under Binaries/ThirdParty/LlamaCpp/Win64.
Put model weights outside the plug-in, under <Project>/Models.
Add a *.localllm.json manifest, or copy an example from Examples/Models.
Enable the plug-in, regenerate project files if required, and compile.
In Blueprint: Get Available Models → Load Model By Id → wait for ModelLoaded → create character sessions → submit text or speech.

Full Documentation

The User Guide covers plugin setup, character/world sheets, the Blueprint coordinator pattern, tool calls, relationship evaluation, microphone/TTS configuration, and troubleshooting.

📘 Open the User Guide

Tech Stack

Built on open-source inference and speech tooling.

llama.cpp Unreal Engine 5 Blueprint + C++ sherpa-onnx Parakeet STT libmtmd CUDA 12 (optional) Vulkan

Shipping Boundaries

Mock STT/TTS providers and the Python NeuTTS-2E/Chatterbox development sidecars are not registered in Shipping builds — the packaged native CPU speech path is Pocket TTS and sherpa-onnx. Model weights, voice datasets, and demo-map assets are project test material, not plug-in payload. An LLM remains nondeterministic: authoritative gameplay state and all mutating actions stay validated by game code.