https://www.knightli.com/en/tags/vllm/ Recent content in VLLM on KnightLi Blog Hugo -- gohugo.io en Fri, 10 Apr 2026 22:54:17 +0800 https://www.knightli.com/en/2026/04/10/gemma4-local-runtime-options/ Fri, 10 Apr 2026 22:54:17 +0800 https://www.knightli.com/en/2026/04/10/gemma4-local-runtime-options/ <p>If you want to run Gemma 4 locally, you can choose from four practical paths depending on your goal and hardware.</p> <h2 id="1-fastest-start-ollama-recommended">1) Fastest start: Ollama (recommended) </h2><p>This is the lowest-friction option for quick testing, daily chat, and local API usage.</p> <div class="highlight"><div class="chroma"> <table class="lntable"><tr><td class="lntd"> <pre tabindex="0" class="chroma"><code><span class="lnt">1 </span></code></pre></td> <td class="lntd"> <pre tabindex="0" class="chroma"><code class="language-bash" data-lang="bash"><span class="line"><span class="cl">ollama run gemma4 </span></span></code></pre></td></tr></table> </div> </div><p>Highlights:</p> <ul> <li>Works on Windows, macOS, and Linux</li> <li>Handles hardware acceleration automatically</li> <li>Offers OpenAI-style local API compatibility</li> </ul> <h2 id="2-gui-workflow-lm-studio--unsloth-studio">2) GUI workflow: LM Studio / Unsloth Studio </h2><p>If you prefer a desktop UI instead of terminal commands:</p> <ul> <li>LM Studio: browse and run Gemma 4 quantized variants from Hugging Face (for example 4-bit, 8-bit), with resource visibility.</li> <li>Unsloth Studio: supports both inference and low-VRAM fine-tuning, often friendlier on 6GB-8GB GPUs.</li> </ul> <h2 id="3-low-spec-and-maximum-control-llamacpp">3) Low-spec and maximum control: llama.cpp </h2><p>Good for older hardware, CPU-focused setups, or users who want deeper runtime control.</p> <p>With <code>.gguf</code> model files and quantization, Gemma 4 can be made practical on much smaller hardware budgets.</p> <h2 id="4-developer-integration-transformers--vllm">4) Developer integration: Transformers / vLLM </h2><p>If you need Gemma 4 inside your own application:</p> <ul> <li>Transformers: straightforward Python integration</li> <li>vLLM: high-throughput inference for stronger GPU environments</li> </ul> <h2 id="quick-selection">Quick selection </h2><table> <thead> <tr> <th>Need</th> <th>Recommended tools</th> <th>Hardware bar</th> </tr> </thead> <tbody> <tr> <td>I just want it running now</td> <td>Ollama</td> <td>Low</td> </tr> <tr> <td>I want a ChatGPT-like UI</td> <td>LM Studio</td> <td>Medium</td> </tr> <tr> <td>My VRAM is limited (6GB-8GB)</td> <td>Unsloth / llama.cpp</td> <td>Low</td> </tr> <tr> <td>I am building local AI apps</td> <td>Ollama / Transformers / vLLM</td> <td>Medium to high</td> </tr> <tr> <td>I need fine-tuning</td> <td>Unsloth Studio</td> <td>Medium to high</td> </tr> </tbody> </table> <h2 id="model-size-suggestion">Model size suggestion </h2><p>Gemma 4 comes in multiple sizes (for example E2B, E4B, 31B).</p> <ul> <li>Start with quantized E2B/E4B on mainstream laptops</li> <li>Move to larger variants only after your baseline pipeline is stable</li> </ul>