Run Ollama and OpenCode on an AWS EC2 instance

These are my personal notes on how to run Ollama and OpenCode on an EC2 instance and connect remotely with Visual Studio Code for code development.

Motivation

As part of the AWS Community Builders Program, I receive lots of free AWS credits, so this is an experiment for me to test open source models for coding.

Advantages

You can gain access to powerful hardware without cost of ownership.
You only pay for what you use. You stop the instance when you are not using it.
You can easily upgrade and downgrade your instance.
You have full control over the instance, so you can install anything you wish.
You run the agents in an isolated environment and not on your personal machine. The agent cannot access your personal files or your credentials.
It is a throwaway instance, if the agent messes up your installation you just delete it and provision a new one.
Your conversations are secure and you do not have to worry about the Terms and Conditions of AI model providers.
You can test different hardware before you commit to built/buy your own machine

Disadvantages

The resources of your instance will most of the time be underutilised.

Instance provisioning

When setting up my instance, I prefer to get a cheap instance type. Once it is all set up, then I can scale it up to one with more resources. I initially provision an instance with the following settings:

Name: devbox
AMI: Deep Learning Base AMI with Single CUDA (Ubuntu 24.04)
Architecture: 64-bit (x86)
Instance type: g4dn.xlarge
Key pair name: select an existing one or create one
Security group: create or select an existing one
EBS Volume: 160 GB 
EBS Volume Encryption: enabled

Advanced Details:
  IAM Instance profile: select or create a new role

Some comments on the above choices:

We are using a Deep Learning Base AMI that has the NVIDIA drivers preinstalled.
You will use the key to SSH to the instance.
The security group should only allow full access to your own IP address.
Choose your own volume size based on how big are the models that you are going to download and what other files you will need to store on the instance (codebases for example).
The role should have the default permissions and no extra permissions. In the future, if your agent needs to access the AWS API you can add permissions using the least privilege principal.

Connecting to your instance

After launching the instance, wait for it to become available. In the meantime, grab its public IP address and set up your SSH config ~/.ssh/config by adding an entry like this one

Host devbox
  HostName <public IP of EC2 instance>
  User ubuntu
  IdentityFile <full path to your .pem file>

This way we can simply do ssh devbox and connect to the instance. If you stop and start the instance again you will need to update this entry with the new IP address. If you cannot connect, check your security group rules.

Installation

Ollama

Install Ollama with

curl -fsSL https://ollama.com/install.sh | sh

and check that it is running with

systemctl status ollama

Pull a model

ollama pull gemma4:12b

Start a quick conversation with

ollama run gemma4:12b

and then ask your question. You can exit with /bye.

>>> How much VRAM is available on a g4dn.xlarge EC2 instance?
...

>>> /bye

btop

Install btop with GPU support

sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
sudo apt update
sudo apt install g++-14 lowdown -y

Run the script found here with to install btop with GPU support

sudo su -c "bash <(wget -qO- https://gist.githubusercontent.com/talmo/b592fb5372630ea823935ec7a047ab26/raw/78593355307ddf00ca08899d47f20733082b35f0/install_btop_gpu.sh)" root

OpenCode

Install OpenCode with

curl -fsSL https://opencode.ai/install | bash

Download a codebase

Let's download some code on our instance so that we can test our agent.

mkdir ~/Repos/
cd ~/Repos/
git clone https://github.com/anomalyco/opencode.git

Connect with VS Code

You can easily connect to your remote instance with VS code running locally on your machine. For this you will need the extension developed by Microsoft named Remote - SSH

Once the extension is downloaded, open the commands menu (command + shift + P on my maC) and run Remote -SSH: Connect to Host... . The setting from your ~/.ssh/config should be visible and you can simply select devbox (or any other name you used). Once connected, you can open the folder of the opencode project that we cloned earlier, i.e. on the path /home/ubuntu/Repos/opencode/ (or any other project you have cloned).

Finally, from the menu of VS Code click Terminal > New Terminal and congratulations, you have your IDE ready for development.

(Optional) Port forwarding

You can do port forwarding from the terminal, or more easily you can port-forward port 11434 from within VS Code.

This would be useful if you want to connect a harness running locally to models served on the EC2 instance.

If you did do the above, you can navigate to http://localhost:11434 and you should see the following message in your browser

Ollama is running

Option 2 (preferred): Configure context length at model level

mkdir ~/OllamaCustom

Create file ~/OllamaCustom/gemma4_12b with the definition of our custom model so that we can increase the context

cat <<EOF > ~/OllamaCustom/gemma4_12b
FROM gemma4:12b
PARAMETER num_ctx 262144
EOF

and create the custom model

ollama create custom-gemma4-12b -f ~/OllamaCustom/gemma4_12b

Then run ollama ls and confirm the custom model is listed.

Option 2: Configure global Ollama Context Length

Stop the Ollama service and edit the configuration

sudo systemctl stop ollama
sudo systemctl edit ollama

Add the following to the configuration

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=262144"

and start the service again

sudo systemctl start ollama

Configure OpenCode

Create the directory

mkdir -p ~/.config/opencode/

and the file that holds the OpenCode configuration (~/.config/opencode/opencode.json)

cat <<EOF > ~/.config/opencode/opencode.json
{
  "\$schema": "https://opencode.ai/config.json",
  "agent":{
    "title": {
      "disable":true
    }
  },
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (Local)",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": {
        "custom-gemma4-12b": {
          "name": "Custom Gemma 4 12B",
          "limit": {
            "context": 262144,
            "output": 8192
          }
        }
      }
    }
  }
}
EOF

Run OpenCode

Start OpenCode in the terminal of your VS Code that is connected to the EC2 Instance.

opencode

Type/models and then you can select the local model

Custom Gemma 4 12B - Ollama (Local)

Having the OpenCode repo open, you can ask the following and watch the agent do its magic.

Review the variants of the README in each language and fix any inconsistencies in the html code. Do not fix text translated in the specific language, this is expected.

Debugging

You can view the logs of the running Ollama service with

journalctl -u ollama -f

Scale up the instance

Shut down the instance with

sudo shutdown now

From the AWS Console, change the instance to p5.2xlarge and start the instance again.

Important: remember to update the IP address of the instance in your ~/.ssh/config

Use a bigger model

SSH to the instance again and create a new file for the new custom model under ~/OllamaCustom/qwen3-coder-30b

cat <<EOF > ~/OllamaCustom/qwen3-coder-30b
FROM qwen3-coder:30b
PARAMETER num_ctx 262144
EOF

then create the Ollama custom model with

ollama create custom-qwen3-coder-30b -f ~/OllamaCustom/qwen3-coder-30b

and update the ~/.config/opencode/opencode.json by adding the model

        "custom-qwen3-coder-30b": {
          "name": "Custom Qwen3 Coder 30B",
          "limit": {
            "context": 262144,
            "output": 65536
          }
        }

Start opencode again and select the new model with /models

Checking performance

Check with ollama ps while running a model. We can see that for g5.2xlarge Gemma 4 12B can run entirely on the GPU

NAME                        ID              SIZE      PROCESSOR    CONTEXT    UNTIL
custom-gemma4-12b:latest    f361b9dbdeb7    8.3 GB    100% GPU     262144     4 minutes from now

while Qwen 3 Coder 30B runs partly on GPU and partly on CPU

NAME                             ID              SIZE     PROCESSOR          CONTEXT    UNTIL
custom-qwen3-coder-30b:latest    e1b819cdf614    45 GB    52%/48% CPU/GPU    262144     4 minutes from now

Scaling options

If you need more VRAM for larger models or longer context windows, moving to a higher tier instance is easy. For example, while g4dn is a great starting point, you can use g5/g6/g6e instances that offer even more GPU memory.

Instance Type	System RAM	GPU Memory (VRAM)
g4dn.xlarge	16 GB	16 GB
g4dn.2xlarge	32 GB	16 GB
g5.xlarge	16 GB	24 GB
g5.2xlarge	32 GB	24 GB
g6.xlarge	16 GB	24 GB
g6.2xlarge	32 GB	24 GB
g6e.xlarge	32 GB	48 GB
g6e.2xlarge	64 GB	48 GB

Note: While VRAM is consistent across a family, larger instance sizes (e.g., moving from xlarge to 2xlarge) provide more system RAM. This extra memory is crucial for supporting larger context windows (Key-Value caches), providing headroom for the OS, and allowing model offloading if weights exceed available VRAM.

Conclusion

This setup provides a powerful, isolated environment for experimenting with local models and agentic coding tools like OpenCode. By offloading the heavy lifting to an EC2 instance, you can experiment freely without compromising your local machine's resources or data privacy.

Bonus: Install Pi

curl -fsSL https://pi.dev/install.sh | sh

cat <<EOF > ~/.pi/agent/models.json
{
  "providers": {
    "ollama": {
      "baseUrl": "http://127.0.0.1:11434/v1",
      "apiKey": "ollama",
      "api": "openai-completions",
      "models": [
        {
          "id": "custom-gemma4-12b:latest",
          "name": "Gemma 4 12B"
        }
      ]
    }
  }
}
EOF

Bonus 2: Run models with llama.cpp and Pi

Uninstall Pi

npm uninstall -g @earendil-works/pi-coding-agent

Install llama.cpp rather than ollama

curl -LsSf https://llama.app/install.sh | sh

Run model

llama serve --ctx-size 262144 -hf ggml-org/gemma-4-12B-it-GGUF:Q8_0

Install pi-llama package

pi install git:github.com/huggingface/pi-llama

Run Pi with pi and select model with /model

Run Ollama and OpenCode on an AWS EC2 instance

Motivation

Advantages

Disadvantages

Instance provisioning

Connecting to your instance

Installation

Ollama

btop

OpenCode

Download a codebase

Connect with VS Code

(Optional) Port forwarding

Option 2 (preferred): Configure context length at model level

Option 2: Configure global Ollama Context Length

Configure OpenCode

Run OpenCode

Debugging

Scale up the instance

Use a bigger model

Checking performance

Scaling options

Conclusion

Bonus: Install Pi

Bonus 2: Run models with llama.cpp and Pi

Comments

More from this blog

AI Agent with controlled access to AWS services in Strands Agents

Bedrock Knowledge Base with S3 Vector Index in AWS CDK

Bedrock Agent with open-ended Action using SQL

Breaking the Restaurant Reservation Agent

Command Palette

Motivation

Advantages

Disadvantages

Instance provisioning

Connecting to your instance

Installation

Ollama

btop

OpenCode

Download a codebase

Connect with VS Code

(Optional) Port forwarding

Option 2 (preferred): Configure context length at model level

Option 2: Configure global Ollama Context Length

Configure OpenCode

Run OpenCode

Debugging

Scale up the instance

Use a bigger model

Checking performance

Scaling options

Conclusion

Bonus: Install Pi

Bonus 2: Run models with llama.cpp and Pi

Comments

More from this blog