Run LLMs Locally from the Command Line with Ollama

Introduction

Online AI tools like ChatGPT and Claude require accounts, subscriptions, and send your data to external servers. If you want to experiment with Large Language Models (LLMs) privately, on your own hardware, and without ongoing costs, Ollama lets you run open source models directly from the command line.

In this tutorial you’ll install Ollama, download and run models, have interactive conversations, use models to process files and text from the command line, and manage the models on your machine.

What You Need

To complete this tutorial, you need at least 8GB of RAM for smaller models and 16GB or more for larger models.

On macOS, you’ll need Homebrew installed, which you can do by following the Install Homebrew tutorial.

Installing Ollama

On macOS, install Ollama with Homebrew:

brew install ollama

Then start the Ollama service:

brew services start ollama

On Linux, run the official install script:

curl -fsSL https://ollama.com/install.sh | sh

This installs Ollama and sets it up as a systemd service that starts automatically.

On Windows, download, and run the installer from Ollama’s download page. Ollama runs as a background service automatically after installation. Use Ollama from PowerShell, Command Prompt, or Windows Terminal. The Ollama commands in the rest of this tutorial work the same on all platforms.

Verify Ollama is running:

ollama --version

You’ll see output showing the version number:

ollama version is 0.18.2

With Ollama running, you can download and run your first model.

Running Your First Model

Ollama hosts a library of open source models you can download and run. To test things out, you’ll use Meta’s Llama 3.2 model, a capable general-purpose model that runs well on most machines.

Run the following command to download and run the model:

ollama run llama3.2

You’ll see a progress bar as it downloads:

pulling manifest
pulling dde5aa3fc5ff: 100% ▕████████████████▏ 2.0 GB
pulling 966de95ca8a6: 100% ▕████████████████▏ 1.4 KB
pulling fcc5a6bec9da: 100% ▕████████████████▏ 7.7 KB
pulling a70ff7e570d9: 100% ▕████████████████▏ 6.0 KB
pulling 56bb8bd477a5: 100% ▕████████████████▏   96 B
pulling 34bb5ab01051: 100% ▕████████████████▏  561 B
verifying sha256 digest
writing manifest
success

Once it finishes, you’ll see an interactive prompt where you can start chatting with the model:

>>> Send a message (/? for help)

Type a question or prompt:

>>> What is a shell pipeline?

The model responds directly in your terminal:

A shell pipeline is a sequence of commands connected by the pipe operator (|),
where the output of one command becomes the input of the next. For example,
`ls -la | grep ".txt" | wc -l` lists files, filters for text files, and
counts them.

Type /bye to exit the interactive session.

Using Models from the Command Line

Interactive mode is useful, but the real power of a local LLM is integrating it into your command-line workflow. You can pipe text directly to Ollama using the ollama run command with a prompt:

echo "Explain what this command does: find . -name '*.log' -mtime +30 -delete" | ollama run llama3.2

The model explains the command without entering interactive mode.

You can use this to process files as well. To summarize a README file:

cat README.md | ollama run llama3.2 "Summarize this document in three bullet points"

Or explain a script:

cat deploy.sh | ollama run llama3.2 "Explain what this script does step by step"

You can also generate content. For example, to generate a commit message from a diff:

git diff --staged | ollama run llama3.2 "Write a concise git commit message for these changes"

Since Ollama runs locally, none of this data leaves your machine.

Choosing the Right Model

Here are some models worth trying:

The sizes in the table reflect Ollama’s default 4-bit quantized versions (Q4_K_M). Quantization compresses a model’s weights from full 16-bit precision down to 4 bits per parameter, which cuts the file size by about 75% while keeping most of the quality.

A model needs at least its file size in memory to run, plus 1-2GB of overhead for the context window and computation buffers. That means a 2GB model like llama3.2 needs around 4GB of available memory, and a 4GB model like mistral needs around 6GB. On macOS, the CPU and GPU share the same memory pool, so a Mac with 16GB of unified memory can comfortably run any model up to about 10-12GB. On Linux and Windows with a dedicated GPU, the model runs fastest when it fits entirely in VRAM.

Models often come in different size variants. For example, llama3.2 defaults to its 3-billion parameter version, but you can run the smaller 1-billion parameter version with llama3.2:1b. Larger parameter counts produce better output but need more memory. Browse all available variants on a model’s page in the Ollama library.

To download a model without starting a conversation, use ollama pull:

ollama pull codellama

pulling manifest
pulling 3a43f93b78ec: 100% ▕███████████████████████████▏ 3.8 GB
pulling 8c17c2ebb0ea: 100% ▕███████████████████████████▏ 7.0 KB
pulling 590d74a5569b: 100% ▕███████████████████████████▏ 4.8 KB
pulling 2e0493f67d0c: 100% ▕███████████████████████████▏   59 B
pulling 7f6a57943a88: 100% ▕███████████████████████████▏  120 B
pulling 316526ac7323: 100% ▕███████████████████████████▏  529 B
verifying sha256 digest
writing manifest
success

Run a specific model by name:

ollama run codellama "Write a bash function that checks if a port is in use"

Smaller models respond faster but produce less nuanced output. Start with llama3.2 and experiment from there.

Managing Models

Models take up disk space, so you’ll want to manage them.

Use ollama list to see the models you’ve downloaded:

ollama list

NAME               ID              SIZE      MODIFIED
codellama:latest   8fdf8f752f6e    3.8 GB    2 minutes ago
llama3.2:latest    a80c4f17acd5    2.0 GB    5 minutes ago

To review detailed information about a model, including its parameters and license:

ollama show llama3.2

You’ll see output similar to the following:

  Model
    architecture        llama
    parameters          3.2B
    context length      131072
    embedding length    3072
    quantization        Q4_K_M

  Parameters
    stop    "<|start_header_id|>"
    stop    "<|end_header_id|>"
    stop    "<|eot_id|>"

  License
    LLAMA 3.2 COMMUNITY LICENSE AGREEMENT
    ...

Remove a model you no longer need using ollama rm:

ollama rm codellama

To determine how much space your models use, check the models directory. On macOS and Linux, use the du command:

du -sh ~/.ollama/models

On Windows, you’ll find models in %USERPROFILE%\.ollama\models. Use File Explorer to determine the size.

Creating Custom Models

You can create custom models with specific system prompts and parameters using a Modelfile, much like a Dockerfile. This is useful when you want a model to behave in a specific way every time you use it.

Create a file called Modelfile that configures a model to act as a command-line assistant. Add the following contents to the file:

FROM llama3.2
SYSTEM """You are a command-line expert. When asked questions, provide concise answers with practical CLI examples. Prefer one-liners and standard Unix tools. Always explain what each part of a command does."""
PARAMETER temperature 0.3

You create your custom model by basing it on an existing one. The temperature parameter controls how creative the model’s responses are. Lower values produce more focused, deterministic output.

Now run the following command to build your custom model:

ollama create cli-helper -f Modelfile

You’ll see the model build:

transferring model data
using existing layer sha256:dde5aa3fc5ff...
creating new layer sha256:a4b72c51c528...
writing manifest
success

Now run your custom model with a prompt:

ollama run cli-helper "How do I find duplicate files in a directory?"

The model responds in the style you defined, focusing on practical command-line solutions. Your custom model appears in ollama list alongside the downloaded ones, and you can remove it the same way.

You’re not limited to using the CLI. There’s an API you can use as well.

Using the API

Ollama runs a local REST API server on port 11434, which means you can interact with it using curl or any HTTP client. This is useful for scripting and integrating Ollama with other tools.

Send a prompt and get a response:

Send a prompt and get a response by using the /api/generate endpoint:

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "What does chmod 755 do?",
  "stream": false
}'

The -s flag for curl silences the progress output from curl and the stream: false parameter tells the Ollama API to return the full response at once instead of streaming tokens. The response comes back as JSON with the model’s output in the response field.

The /api/generate endpoint handles one-off prompts. Ollama also provides a /api/chat endpoint for conversations. Unlike ollama run, the API does not remember previous messages on its own. To have a back-and-forth conversation, you send the full message history with each request.

Start with a single question:

curl -s http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "stream": false,
  "messages": [
    {"role": "user", "content": "What is a FIFO in Unix?"}
  ]
}'

The response includes the model’s answer in a message object. To ask a follow-up, you include your original question, the model’s answer, and your new question. You set the role to user for your messages and assistant for the model’s previous responses:

curl -s http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "stream": false,
  "messages": [
    {"role": "user", "content": "What is a FIFO in Unix?"},
    {"role": "assistant", "content": "A FIFO is a named pipe..."},
    {"role": "user", "content": "How do I create one?"}
  ]
}'

Because you sent the earlier exchange, the model knows what “one” refers to and can give a relevant answer. Each request you make builds on the previous messages you include.

This API means any tool that can make HTTP requests can use your local models — shell scripts, Python programs, or other command-line tools.

Conclusion

Ollama gives you a way to run large language models locally, from the command line, without accounts or API keys. Your prompts and data stay on your machine. You can pipe files and text through models, create custom assistants, and integrate local AI into your existing command-line workflow.

Explore the full model library at Ollama’s website, or run ollama help to see all available commands.