> ## Documentation Index > Fetch the complete content index at: https://smallsharpsoftwaretools.com/llms.txt > Use this file to discover all available pages before exploring further. # Run LLMs Locally from the Command Line with Ollama Online AI tools like ChatGPT and Claude require accounts, subscriptions, and send your data to external servers. If you want to experiment with Large Language Models (LLMs) privately, on your own hardware, and without ongoing costs, [Ollama](https://ollama.com) lets you run open source models directly from the command line. In this tutorial you'll install Ollama, download and run models, have interactive conversations, use models to process files and text from the command line, and manage the models on your machine. ## What You Need To complete this tutorial, you need at least 8GB of RAM for smaller models and 16GB or more for larger models. On macOS, you'll need Homebrew installed, which you can do by following the [Install Homebrew]({{< ref "homebrew" >}}) tutorial. ## Installing Ollama On macOS, install Ollama with Homebrew: ```command brew install ollama ``` Then start the Ollama service: ```command brew services start ollama ``` On Linux, run the official install script: ```command curl -fsSL https://ollama.com/install.sh | sh ``` This installs Ollama and sets it up as a systemd service that starts automatically. On Windows, download, and run the installer from [Ollama's download page](https://ollama.com/download). Ollama runs as a background service automatically after installation. Use Ollama from PowerShell, Command Prompt, or Windows Terminal. The Ollama commands in the rest of this tutorial work the same on all platforms. Verify Ollama is running: ```command ollama --version ``` You'll see output showing the version number: ```output ollama version is 0.18.2 ``` With Ollama running, you can download and run your first model. ## Running Your First Model Ollama hosts a library of open source models you can download and run. To test things out, you'll use Meta's Llama 3.2 model, a capable general-purpose model that runs well on most machines. Run the following command to download and run the model: ```command ollama run llama3.2 ``` You'll see a progress bar as it downloads: ```output pulling manifest pulling dde5aa3fc5ff: 100% ▕████████████████▏ 2.0 GB pulling 966de95ca8a6: 100% ▕████████████████▏ 1.4 KB pulling fcc5a6bec9da: 100% ▕████████████████▏ 7.7 KB pulling a70ff7e570d9: 100% ▕████████████████▏ 6.0 KB pulling 56bb8bd477a5: 100% ▕████████████████▏ 96 B pulling 34bb5ab01051: 100% ▕████████████████▏ 561 B verifying sha256 digest writing manifest success ``` Once it finishes, you'll see an interactive prompt where you can start chatting with the model: ```output >>> Send a message (/? for help) ``` Type a question or prompt: ``` >>> What is a shell pipeline? ``` The model responds directly in your terminal: ```output A shell pipeline is a sequence of commands connected by the pipe operator (|), where the output of one command becomes the input of the next. For example, `ls -la | grep ".txt" | wc -l` lists files, filters for text files, and counts them. ``` Type `/bye` to exit the interactive session. ## Using Models from the Command Line Interactive mode is useful, but the real power of a local LLM is integrating it into your command-line workflow. You can pipe text directly to Ollama using the `ollama run` command with a prompt: ```command echo "Explain what this command does: find . -name '*.log' -mtime +30 -delete" | ollama run llama3.2 ``` The model explains the command without entering interactive mode. You can use this to process files as well. To summarize a README file: ```command cat README.md | ollama run llama3.2 "Summarize this document in three bullet points" ``` Or explain a script: ```command cat deploy.sh | ollama run llama3.2 "Explain what this script does step by step" ``` You can also generate content. For example, to generate a commit message from a diff: ```command git diff --staged | ollama run llama3.2 "Write a concise git commit message for these changes" ``` Since Ollama runs locally, none of this data leaves your machine. ## Choosing the Right Model Here are some models worth trying: | Model | Size | Best For | |-------|------|----------| | `llama3.2` | ~2 GB | General use, good balance of speed and quality | | `llama3.2:1b` | ~1 GB | Fast responses on limited hardware | | `codellama` | ~4 GB | Code generation and explanation | | `mistral` | ~4 GB | General use, strong reasoning | | `gemma3` | ~3 GB | General use, Google's open model | | `deepseek-coder-v2` | ~9 GB | Code-focused tasks | | `llama3.3` | ~43 GB | Best quality, needs 64GB+ RAM | The sizes in the table reflect Ollama's default 4-bit quantized versions (Q4_K_M). Quantization compresses a model's weights from full 16-bit precision down to 4 bits per parameter, which cuts the file size by about 75% while keeping most of the quality. A model needs at least its file size in memory to run, plus 1-2GB of overhead for the context window and computation buffers. That means a 2GB model like `llama3.2` needs around 4GB of available memory, and a 4GB model like `mistral` needs around 6GB. On macOS, the CPU and GPU share the same memory pool, so a Mac with 16GB of unified memory can comfortably run any model up to about 10-12GB. On Linux and Windows with a dedicated GPU, the model runs fastest when it fits entirely in VRAM. Models often come in different size variants. For example, `llama3.2` defaults to its 3-billion parameter version, but you can run the smaller 1-billion parameter version with `llama3.2:1b`. Larger parameter counts produce better output but need more memory. Browse all available variants on a model's page in the [Ollama library](https://ollama.com/library). To download a model without starting a conversation, use `ollama pull`: ```command ollama pull codellama ``` ```output pulling manifest pulling 3a43f93b78ec: 100% ▕███████████████████████████▏ 3.8 GB pulling 8c17c2ebb0ea: 100% ▕███████████████████████████▏ 7.0 KB pulling 590d74a5569b: 100% ▕███████████████████████████▏ 4.8 KB pulling 2e0493f67d0c: 100% ▕███████████████████████████▏ 59 B pulling 7f6a57943a88: 100% ▕███████████████████████████▏ 120 B pulling 316526ac7323: 100% ▕███████████████████████████▏ 529 B verifying sha256 digest writing manifest success ``` Run a specific model by name: ```command ollama run codellama "Write a bash function that checks if a port is in use" ``` Smaller models respond faster but produce less nuanced output. Start with `llama3.2` and experiment from there. ## Managing Models Models take up disk space, so you'll want to manage them. Use `ollama list` to see the models you've downloaded: ```command ollama list ``` ```output NAME ID SIZE MODIFIED codellama:latest 8fdf8f752f6e 3.8 GB 2 minutes ago llama3.2:latest a80c4f17acd5 2.0 GB 5 minutes ago ``` To review detailed information about a model, including its parameters and license: ```command ollama show llama3.2 ``` You'll see output similar to the following: ```output Model architecture llama parameters 3.2B context length 131072 embedding length 3072 quantization Q4_K_M Parameters stop "<|start_header_id|>" stop "<|end_header_id|>" stop "<|eot_id|>" License LLAMA 3.2 COMMUNITY LICENSE AGREEMENT ... ``` Remove a model you no longer need using `ollama rm`: ```command ollama rm codellama ``` To determine how much space your models use, check the models directory. On macOS and Linux, use the `du` command: ```command du -sh ~/.ollama/models ``` On Windows, you'll find models in `%USERPROFILE%\.ollama\models`. Use File Explorer to determine the size. ## Creating Custom Models You can create custom models with specific system prompts and parameters using a Modelfile, much like a Dockerfile. This is useful when you want a model to behave in a specific way every time you use it. Create a file called `Modelfile` that configures a model to act as a command-line assistant. Add the following contents to the file: ``` FROM llama3.2 SYSTEM """You are a command-line expert. When asked questions, provide concise answers with practical CLI examples. Prefer one-liners and standard Unix tools. Always explain what each part of a command does.""" PARAMETER temperature 0.3 ``` You create your custom model by basing it on an existing one. The `temperature` parameter controls how creative the model's responses are. Lower values produce more focused, deterministic output. Now run the following command to build your custom model: ```command ollama create cli-helper -f Modelfile ``` You'll see the model build: ```output transferring model data using existing layer sha256:dde5aa3fc5ff... creating new layer sha256:a4b72c51c528... writing manifest success ``` Now run your custom model with a prompt: ```command ollama run cli-helper "How do I find duplicate files in a directory?" ``` The model responds in the style you defined, focusing on practical command-line solutions. Your custom model appears in `ollama list` alongside the downloaded ones, and you can remove it the same way. You're not limited to using the CLI. There's an API you can use as well. ## Using the API Ollama runs a local REST API server on port `11434`, which means you can interact with it using `curl` or any HTTP client. This is useful for scripting and integrating Ollama with other tools. Send a prompt and get a response: Send a prompt and get a response by using the `/api/generate` endpoint: ```command curl -s http://localhost:11434/api/generate -d '{ "model": "llama3.2", "prompt": "What does chmod 755 do?", "stream": false }' ``` The `-s` flag for `curl` silences the progress output from `curl` and the `stream: false` parameter tells the Ollama API to return the full response at once instead of streaming tokens. The response comes back as JSON with the model's output in the `response` field. The `/api/generate` endpoint handles one-off prompts. Ollama also provides a `/api/chat` endpoint for conversations. Unlike `ollama run`, the API does not remember previous messages on its own. To have a back-and-forth conversation, you send the full message history with each request. Start with a single question: ```command curl -s http://localhost:11434/api/chat -d '{ "model": "llama3.2", "stream": false, "messages": [ {"role": "user", "content": "What is a FIFO in Unix?"} ] }' ``` The response includes the model's answer in a `message` object. To ask a follow-up, you include your original question, the model's answer, and your new question. You set the `role` to `user` for your messages and `assistant` for the model's previous responses: ```command curl -s http://localhost:11434/api/chat -d '{ "model": "llama3.2", "stream": false, "messages": [ {"role": "user", "content": "What is a FIFO in Unix?"}, {"role": "assistant", "content": "A FIFO is a named pipe..."}, {"role": "user", "content": "How do I create one?"} ] }' ``` Because you sent the earlier exchange, the model knows what "one" refers to and can give a relevant answer. Each request you make builds on the previous messages you include. This API means any tool that can make HTTP requests can use your local models — shell scripts, Python programs, or other command-line tools. ## Conclusion Ollama gives you a way to run large language models locally, from the command line, without accounts or API keys. Your prompts and data stay on your machine. You can pipe files and text through models, create custom assistants, and integrate local AI into your existing command-line workflow. Explore the full model library at [Ollama's website](https://ollama.com/library), or run `ollama help` to see all available commands.