# Running Ollama and TranslateGemma on Windows Server 2022: what I learned

# Running Ollama and TranslateGemma on Windows Server 2022: what I learned

Public-sector organisations often need to process personal data while complying with strict data-protection requirements. This can make cloud-based AI services difficult to use, especially when documents contain confidential or personally identifiable information.

One possible solution is to run the AI model locally, on infrastructure controlled by the organisation. That was exactly the approach I wanted to try.

For my experiments, I chose Ollama because I already use it for personal projects and like its simple workflow. For the translation model, I used Google's TranslateGemma.

> I am not affiliated with **Ollama** or **Google**, and I was **not** paid to write this article. I am simply sharing what I learned while trying to get Ollama and TranslateGemma running on a virtual Windows Server 2022 machine.

And, as I quickly found out, running something on my personal computer and running the same thing on a server are two rather different experiences. 😉

## Why I needed a local translation model

At my organisation, one part of our work involves processing documents written in many different languages. To understand and categorise them, we first need to translate their content into German.

I wanted to see whether I could do this locally instead of sending the document content to an external AI service.

`TranslateGemma` is designed for multilingual translation and supports 55 languages. To make it easy to use, I created a simple HTML page with vanilla JavaScript that sends translation requests directly to the Ollama API.

On my personal computer, this worked surprisingly well. So the next logical step was to move the solution to a Windows Server 2022 virtual machine.

And that was when the interesting part started.

The translation model itself was not really the problem. The challenges were much more about everything around it: how to keep Ollama running, how browsers handle connections to the API, and what happens when the available server memory is limited.

These are the three things I ended up spending most of my time on.

## Issue 1: Keeping Ollama running without an interactive session

On my personal computer, Ollama is normally started after I sign in. That is perfectly fine when I am sitting in front of the computer. It is not quite as useful on a server. A server should keep doing its job after a reboot without requiring someone to log in first and start an application manually.

Initially, this was one of those things that sounds simple until you actually try it.

### Starting Ollama with the Windows Task Scheduler

My solution was to start Ollama in `serve` mode using the *Windows Task Scheduler* .

The idea is quite simple: instead of relying on the normal interactive startup, Windows starts Ollama automatically when the server starts, so that the process is constantly running and available to everyone.

I configured a scheduled task with the following settings:

* Run the task using the Windows account that should host Ollama (in my case, it's my account).
* Enable **Run with highest privileges**.
* Configure the trigger to run when Windows starts.
* Start Ollama with the `serve` argument.

The program and argument were configured like this:

```text
Program:
C:\Program Files\Ollama\ollama.exe

Argument:
serve
```

So, effectively, Task Scheduler runs:

```text
C:\Program Files\Ollama\ollama.exe serve
```

> The exact location of `ollama.exe` can of course be different depending on how Ollama was installed. I recommend checking the actual installation path on the server instead of simply assuming that the path above exists.

Once the scheduled task was in place, Ollama could start with Windows without requiring an interactive desktop session.

That solved the first problem.

### A note about permissions

The next thing I ran into was the location of the models.

I did not want to keep the model files in the default location, so I used a custom directory. Ollama can be told to use a different model directory with the `OLLAMA_MODELS` environment variable. For example:

```text
OLLAMA_MODELS=D:\Ollama\Models
```

In my case, the account used by the scheduled task therefore needed permission to access that directory. This is also an important distinction that was not immediately obvious to me. My JavaScript application does not access the model files directly. It talks to Ollama through HTTP.

Therefore, the IIS application-pool identity does **not** need access to the model directory simply because the IIS application uses Ollama. The account running Ollama is the one that needs access to the model files.

In a typical setup, I would therefore check that the Ollama account has access to:

* The Ollama executable
* The model directory
* Any temporary or configuration directories used by Ollama

As always with server permissions, I would avoid simply giving everyone access to the directory. It may solve a permissions problem quickly, but it also creates an unnecessary security risk.

This was a good reminder that when something runs as a Windows service-like background process, I need to think about **which Windows account is actually running the process**.

## Issue 2: CORS errors in the browser

Once Ollama was starting reliably, I thought I had solved the difficult part.

I hadn't. 😬

The next problem appeared when I tried to use the web application from a browser. The browser reported a Cross-Origin Resource Sharing (CORS) error. The reason was actually quite logical.

My web application was running through IIS under one origin, while the Ollama API was running on another origin, using port `11434`.

Browsers enforce the same-origin policy. If JavaScript running on one origin tries to communicate with another origin, the target API has to allow that request. Otherwise, the browser blocks it. This is one of those things that is easy to overlook when everything is running on the same computer.

### Be careful with `localhost`

This was actually one of the more important details in my setup. Initially, my JavaScript application called Ollama like this:

```javascript
const response = await fetch('http://localhost:11434/api/generate', {
```

This worked when I opened the application directly on the Windows Server. However, the application was intended to be used by colleagues from their own devices. 

And this is where `localhost` becomes a problem. The `fetch()` call is executed by the **browser**. It is not executed by IIS and it is not executed by the Windows Server simply because the HTML page was delivered by that server. Therefore, when a colleague opens the application from their device, this:

```text
http://localhost:11434
```

means:

```text
The colleague's notebook
```

and **not**:

```text
The Windows Server running Ollama
```

The browser therefore tried to find Ollama on the user's own computer. That obviously did not work.

### Using the server hostname

Since the application itself was accessed through the organisation's server address, I changed the JavaScript code to dynamically use the hostname from which the page was loaded.

Instead of:

```javascript
const response = await fetch('http://localhost:11434/api/generate', {
```

I used:

```javascript
const response = await fetch(`http://${window.location.hostname}:11434/api/generate`, {
```

`window.location.hostname` contains the hostname used by the browser to access the web application.

For example, if the user opens:

```text
https://translation.mywork.org
```

then:

```javascript
window.location.hostname
```

returns:

```text
translation.mywork.org
```

The browser therefore calls:

```text
http://translation.mywork.org:11434/api/generate
```

instead of:

```text
http://localhost:11434/api/generate
```

This was the missing piece in my setup. The important thing to remember is that the JavaScript code runs **on the client**. The fact that the JavaScript file or HTML page came from the Windows Server does not change where `localhost` points.

### Making Ollama reachable on the network

Once the browser was trying to connect to the server's hostname, Ollama also had to be reachable through that network address. By default, Ollama listens only on the local loopback interface:

```text
127.0.0.1:11434
```

That means connections from other computers cannot reach it. I therefore configured the `OLLAMA_HOST` environment variable on the Windows Server:

```text
OLLAMA_HOST=0.0.0.0:11434
```

The `0.0.0.0` binding tells Ollama to listen on all available network interfaces rather than only on the local loopback interface. This was necessary in my setup because the browser running the JavaScript code was on a different computer.

> Of course, making Ollama listen on all interfaces also means that the service becomes potentially reachable from the network. This is therefore something that needs to be considered together with Windows Firewall rules and the network architecture.
> 
> I would not expose port `11434` more widely than necessary.

For an internal application, I would normally restrict access to the relevant internal network or use a controlled reverse-proxy configuration rather than simply exposing the Ollama API to an untrusted network.

### One more thing: HTTP, HTTPS and CORS

There is another detail worth mentioning. If the web application is loaded over HTTPS, for example:

```text
https://translation.mywork.org
```

but the JavaScript then tries to call:

```text
http://translation.mywork.org:11434
```

the browser may block the request because the page is using HTTPS while the API is using HTTP. This is commonly referred to as mixed-content blocking.

In my setup, I therefore had to consider not only the hostname, but also how the Ollama API was exposed to the browser. This is also where the CORS configuration comes into play. The browser sees the web application and the Ollama API as different origins because the port is different:

```text
https://translation.mywork.org
             |
             +-- web application

http://translation.mywork.org:11434
             |
             +-- Ollama API
```

Therefore, Ollama needs to allow the origin of the web application. For example:

```text
OLLAMA_ORIGINS=https://translation.example.org
```

This is separate from `OLLAMA_HOST`.

In my setup, I therefore needed both:

```text
OLLAMA_HOST=0.0.0.0:11434
OLLAMA_ORIGINS=https://translation.mywork.org
```

The first setting made Ollama reachable from the network. The second setting allowed the browser application to make the cross-origin request.

These two settings solve different problems, and understanding that distinction made troubleshooting much easier.

## Issue 3: the model remained in memory for too long

The last problem was the one I initially found most surprising.

Memory.

When Ollama receives a request, it loads the required model into memory. Once the translation is finished, the model does not necessarily disappear immediately. Ollama keeps it loaded for a while so that the next request can be processed more quickly. On my personal computer, this behaviour is convenient. On our virtual server, it became a problem.

After a translation had finished, the model was still occupying a significant amount of memory. If another process or request subsequently needed memory, there was less available than I expected.

At that point I started looking into Ollama's keep-alive behaviour.

### Configuring `OLLAMA_KEEP_ALIVE`

Ollama uses `OLLAMA_KEEP_ALIVE` to control how long a model remains loaded after a request. The default value is *five minutes*. For my setup, that was longer than I needed. I therefore reduced it to ten seconds:

```text
OLLAMA_KEEP_ALIVE=10s
```

With this setting, the model is unloaded much sooner after a request has completed, making the memory available again more quickly. This solved my memory problem, but - as always - there is a trade-off.

If another request arrives after the keep-alive period has expired, Ollama needs to load the model again. The next request will therefore take longer. So there is no universally correct value. The *right* setting depends on things such as:

* Available memory
* Model size
* Number of users
* Request frequency
* Acceptable response time

For my environment, I decided that releasing the memory quickly was more important than keeping the model loaded for the next request.

I again configured this as a Windows **system environment variable**:

```text
OLLAMA_KEEP_ALIVE=10s
```

After changing it, I restarted the Ollama scheduled task. Another option would be to control the keep-alive period for individual API requests using the `keep_alive` parameter. For my application, however, the environment variable was the simpler solution because I wanted the behaviour to apply generally.

## My final configuration

After going through all of these issues, my relevant system environment variables looked roughly like this:

```text
OLLAMA_MODELS=D:\Ollama\Models
OLLAMA_HOST=0.0.0.0:11434
OLLAMA_ORIGINS=https://translation.mywork.org
OLLAMA_KEEP_ALIVE=10s
```

For troubleshooting CORS, I could temporarily change the origin configuration to:

```text
OLLAMA_ORIGINS=*
```

Ollama was started automatically by the Windows Task Scheduler using:

```text
C:\Program Files\Ollama\ollama.exe serve
```

The Windows account running the scheduled task had permission to access the custom model directory. The web application used the following client-side JavaScript pattern to address the Ollama API:

```javascript
const response = await fetch(
    `http://${window.location.hostname}:11434/api/generate`,
    {
```

This meant that the browser used the same hostname that was used to access the web application, rather than trying to find Ollama on the client's own `localhost`. Because the application was accessed from other computers, Ollama also had to listen on a network interface rather than only on `127.0.0.1`.

For my setup, this was achieved with:

```text
OLLAMA_HOST=0.0.0.0:11434
```

> This configuration should not be copied blindly into every environment. If Ollama is made available on the network, Windows Firewall rules and the surrounding network architecture need to be configured appropriately.

In many environments, using a reverse proxy or another controlled access path may be preferable to simply exposing the Ollama API directly.

## What I learned

Looking back, none of these problems were particularly difficult individually. What surprised me was how many small differences appeared when moving from a personal computer to a server. 

Running Ollama on a personal computer is relatively straightforward. Running it as part of a Windows Server application requires thinking about the environment around Ollama as well.

The main things I learned were:

1. Ollama must start reliably without depending on an interactive user session.
2. The account running Ollama needs access to the model and configuration directories.
3. Browser-side JavaScript runs on the client computer, not inside the IIS application pool.
4. The JavaScript code runs in the user's browser. `localhost` therefore refers to the user's computer, not the server hosting the web application.
5. When the browser needs to access Ollama on the server, the Ollama API must be reachable through the server's network address.
6. `OLLAMA_HOST` controls the network interface and port on which Ollama listens, while `OLLAMA_ORIGINS` controls which browser origins are allowed to access it.
7. Browser-based applications need an appropriate CORS configuration.
8. Environment variables must be available to the process that actually starts Ollama.
9. Model memory usage becomes especially important when the server has limited resources.
10. Broad permissions and unrestricted CORS settings may be useful during troubleshooting, but should not simply become the production configuration.

Once I had Ollama running through the Task Scheduler, the correct directory permissions in place, the network binding configured, CORS configured and a shorter model keep-alive period, the setup finally worked as I had intended.

I could use `TranslateGemma` locally from my IIS-hosted web application without sending the confidential document content to an external AI service. It took a little more configuration than on my personal computer. But that is also what made the exercise interesting.

The actual AI model was only one part of the solution. Getting the surrounding infrastructure right - processes, permissions, networking, browser security and memory management - was just as important.

And that was probably my biggest takeaway from the whole exercise. Running AI locally is not just about choosing a model and installing it. It is also about understanding how that model fits into the environment in which you want to use it.

### A note about exposing ollama

Although making Ollama directly reachable from the network worked for my setup, I would not consider this the preferred architecture for a production environment. A better approach is to keep Ollama on the server side and put a backend or reverse proxy in front of it, ideally exposing only HTTPS to the clients. This avoids exposing the Ollama API directly and also avoids the CORS and mixed-content issues described above. The downside is that this requires a somewhat larger solution: instead of a single static HTML/JavaScript page communicating directly with Ollama, the application needs a server-side component or reverse-proxy configuration. However, this is a topic for another blog article.

## Further Reading

For more information about Ollama, its configuration options, and the Gemma model family, see the following resources:

* [Ollama FAQ](https://docs.ollama.com/faq)
* [Ollama website](https://ollama.com/)
* [Google Gemma](https://ai.google.dev/gemma)
* [Ollama Windows documentation](https://docs.ollama.com/windows)
* [Ollama documentation on environment variables](https://github.com/ollama/ollama/issues/2941)