Search for a command to run...
What I learned when moving a local Ollama and TranslateGemma setup from my personal computer to a virtual Windows Server 2022 environment - including Task Scheduler, CORS, networking, permissions and memory management.
No comments yet. Be the first to comment.
A lightweight, 100% privacy-compliant tool to extract QR codes from PDF documents and images - completely offline and without the need for installation.
Embeddings: the underrated foundation of AI RAG systems
Streamline your Node.js project with TailwindCSS for a powerful and lightweight styling solution—no extra frameworks required
Introduction Microsoft's Windows lock screen greets daily with captivating images. In this article, you will learn where to find this images and how to get them: either directly navigating to the image directory or utilizing a purpose-built Windows a...
Public-sector organisations often need to process personal data while complying with strict data-protection requirements. This can make cloud-based AI services difficult to use, especially when documents contain confidential or personally identifiable information.
One possible solution is to run the AI model locally, on infrastructure controlled by the organisation. That was exactly the approach I wanted to try.
For my experiments, I chose Ollama because I already use it for personal projects and like its simple workflow. For the translation model, I used Google's TranslateGemma.
I am not affiliated with Ollama or Google, and I was not paid to write this article. I am simply sharing what I learned while trying to get Ollama and TranslateGemma running on a virtual Windows Server 2022 machine.
And, as I quickly found out, running something on my personal computer and running the same thing on a server are two rather different experiences. 😉
At my organisation, one part of our work involves processing documents written in many different languages. To understand and categorise them, we first need to translate their content into German.
I wanted to see whether I could do this locally instead of sending the document content to an external AI service.
TranslateGemma is designed for multilingual translation and supports 55 languages. To make it easy to use, I created a simple HTML page with vanilla JavaScript that sends translation requests directly to the Ollama API.
On my personal computer, this worked surprisingly well. So the next logical step was to move the solution to a Windows Server 2022 virtual machine.
And that was when the interesting part started.
The translation model itself was not really the problem. The challenges were much more about everything around it: how to keep Ollama running, how browsers handle connections to the API, and what happens when the available server memory is limited.
These are the three things I ended up spending most of my time on.
On my personal computer, Ollama is normally started after I sign in. That is perfectly fine when I am sitting in front of the computer. It is not quite as useful on a server. A server should keep doing its job after a reboot without requiring someone to log in first and start an application manually.
Initially, this was one of those things that sounds simple until you actually try it.
My solution was to start Ollama in serve mode using the Windows Task Scheduler .
The idea is quite simple: instead of relying on the normal interactive startup, Windows starts Ollama automatically when the server starts, so that the process is constantly running and available to everyone.
I configured a scheduled task with the following settings:
serve argument.The program and argument were configured like this:
Program:
C:\Program Files\Ollama\ollama.exe
Argument:
serve
So, effectively, Task Scheduler runs:
C:\Program Files\Ollama\ollama.exe serve
The exact location of
ollama.execan of course be different depending on how Ollama was installed. I recommend checking the actual installation path on the server instead of simply assuming that the path above exists.
Once the scheduled task was in place, Ollama could start with Windows without requiring an interactive desktop session.
That solved the first problem.
The next thing I ran into was the location of the models.
I did not want to keep the model files in the default location, so I used a custom directory. Ollama can be told to use a different model directory with the OLLAMA_MODELS environment variable. For example:
OLLAMA_MODELS=D:\Ollama\Models
In my case, the account used by the scheduled task therefore needed permission to access that directory. This is also an important distinction that was not immediately obvious to me. My JavaScript application does not access the model files directly. It talks to Ollama through HTTP.
Therefore, the IIS application-pool identity does not need access to the model directory simply because the IIS application uses Ollama. The account running Ollama is the one that needs access to the model files.
In a typical setup, I would therefore check that the Ollama account has access to:
As always with server permissions, I would avoid simply giving everyone access to the directory. It may solve a permissions problem quickly, but it also creates an unnecessary security risk.
This was a good reminder that when something runs as a Windows service-like background process, I need to think about which Windows account is actually running the process.
Once Ollama was starting reliably, I thought I had solved the difficult part.
I hadn't. 😬
The next problem appeared when I tried to use the web application from a browser. The browser reported a Cross-Origin Resource Sharing (CORS) error. The reason was actually quite logical.
My web application was running through IIS under one origin, while the Ollama API was running on another origin, using port 11434.
Browsers enforce the same-origin policy. If JavaScript running on one origin tries to communicate with another origin, the target API has to allow that request. Otherwise, the browser blocks it. This is one of those things that is easy to overlook when everything is running on the same computer.
localhostThis was actually one of the more important details in my setup. Initially, my JavaScript application called Ollama like this:
const response = await fetch('http://localhost:11434/api/generate', {
This worked when I opened the application directly on the Windows Server. However, the application was intended to be used by colleagues from their own devices.
And this is where localhost becomes a problem. The fetch() call is executed by the browser. It is not executed by IIS and it is not executed by the Windows Server simply because the HTML page was delivered by that server. Therefore, when a colleague opens the application from their device, this:
http://localhost:11434
means:
The colleague's notebook
and not:
The Windows Server running Ollama
The browser therefore tried to find Ollama on the user's own computer. That obviously did not work.
Since the application itself was accessed through the organisation's server address, I changed the JavaScript code to dynamically use the hostname from which the page was loaded.
Instead of:
const response = await fetch('http://localhost:11434/api/generate', {
I used:
const response = await fetch(`http://${window.location.hostname}:11434/api/generate`, {
window.location.hostname contains the hostname used by the browser to access the web application.
For example, if the user opens:
https://translation.mywork.org
then:
window.location.hostname
returns:
translation.mywork.org
The browser therefore calls:
http://translation.mywork.org:11434/api/generate
instead of:
http://localhost:11434/api/generate
This was the missing piece in my setup. The important thing to remember is that the JavaScript code runs on the client. The fact that the JavaScript file or HTML page came from the Windows Server does not change where localhost points.
Once the browser was trying to connect to the server's hostname, Ollama also had to be reachable through that network address. By default, Ollama listens only on the local loopback interface:
127.0.0.1:11434
That means connections from other computers cannot reach it. I therefore configured the OLLAMA_HOST environment variable on the Windows Server:
OLLAMA_HOST=0.0.0.0:11434
The 0.0.0.0 binding tells Ollama to listen on all available network interfaces rather than only on the local loopback interface. This was necessary in my setup because the browser running the JavaScript code was on a different computer.
Of course, making Ollama listen on all interfaces also means that the service becomes potentially reachable from the network. This is therefore something that needs to be considered together with Windows Firewall rules and the network architecture.
I would not expose port
11434more widely than necessary.
For an internal application, I would normally restrict access to the relevant internal network or use a controlled reverse-proxy configuration rather than simply exposing the Ollama API to an untrusted network.
There is another detail worth mentioning. If the web application is loaded over HTTPS, for example:
https://translation.mywork.org
but the JavaScript then tries to call:
http://translation.mywork.org:11434
the browser may block the request because the page is using HTTPS while the API is using HTTP. This is commonly referred to as mixed-content blocking.
In my setup, I therefore had to consider not only the hostname, but also how the Ollama API was exposed to the browser. This is also where the CORS configuration comes into play. The browser sees the web application and the Ollama API as different origins because the port is different:
https://translation.mywork.org
|
+-- web application
http://translation.mywork.org:11434
|
+-- Ollama API
Therefore, Ollama needs to allow the origin of the web application. For example:
OLLAMA_ORIGINS=https://translation.example.org
This is separate from OLLAMA_HOST.
In my setup, I therefore needed both:
OLLAMA_HOST=0.0.0.0:11434
OLLAMA_ORIGINS=https://translation.mywork.org
The first setting made Ollama reachable from the network. The second setting allowed the browser application to make the cross-origin request.
These two settings solve different problems, and understanding that distinction made troubleshooting much easier.
The last problem was the one I initially found most surprising.
Memory.
When Ollama receives a request, it loads the required model into memory. Once the translation is finished, the model does not necessarily disappear immediately. Ollama keeps it loaded for a while so that the next request can be processed more quickly. On my personal computer, this behaviour is convenient. On our virtual server, it became a problem.
After a translation had finished, the model was still occupying a significant amount of memory. If another process or request subsequently needed memory, there was less available than I expected.
At that point I started looking into Ollama's keep-alive behaviour.
OLLAMA_KEEP_ALIVEOllama uses OLLAMA_KEEP_ALIVE to control how long a model remains loaded after a request. The default value is five minutes. For my setup, that was longer than I needed. I therefore reduced it to ten seconds:
OLLAMA_KEEP_ALIVE=10s
With this setting, the model is unloaded much sooner after a request has completed, making the memory available again more quickly. This solved my memory problem, but - as always - there is a trade-off.
If another request arrives after the keep-alive period has expired, Ollama needs to load the model again. The next request will therefore take longer. So there is no universally correct value. The right setting depends on things such as:
For my environment, I decided that releasing the memory quickly was more important than keeping the model loaded for the next request.
I again configured this as a Windows system environment variable:
OLLAMA_KEEP_ALIVE=10s
After changing it, I restarted the Ollama scheduled task. Another option would be to control the keep-alive period for individual API requests using the keep_alive parameter. For my application, however, the environment variable was the simpler solution because I wanted the behaviour to apply generally.
After going through all of these issues, my relevant system environment variables looked roughly like this:
OLLAMA_MODELS=D:\Ollama\Models
OLLAMA_HOST=0.0.0.0:11434
OLLAMA_ORIGINS=https://translation.mywork.org
OLLAMA_KEEP_ALIVE=10s
For troubleshooting CORS, I could temporarily change the origin configuration to:
OLLAMA_ORIGINS=*
Ollama was started automatically by the Windows Task Scheduler using:
C:\Program Files\Ollama\ollama.exe serve
The Windows account running the scheduled task had permission to access the custom model directory. The web application used the following client-side JavaScript pattern to address the Ollama API:
const response = await fetch(
`http://${window.location.hostname}:11434/api/generate`,
{
This meant that the browser used the same hostname that was used to access the web application, rather than trying to find Ollama on the client's own localhost. Because the application was accessed from other computers, Ollama also had to listen on a network interface rather than only on 127.0.0.1.
For my setup, this was achieved with:
OLLAMA_HOST=0.0.0.0:11434
This configuration should not be copied blindly into every environment. If Ollama is made available on the network, Windows Firewall rules and the surrounding network architecture need to be configured appropriately.
In many environments, using a reverse proxy or another controlled access path may be preferable to simply exposing the Ollama API directly.
Looking back, none of these problems were particularly difficult individually. What surprised me was how many small differences appeared when moving from a personal computer to a server.
Running Ollama on a personal computer is relatively straightforward. Running it as part of a Windows Server application requires thinking about the environment around Ollama as well.
The main things I learned were:
localhost therefore refers to the user's computer, not the server hosting the web application.OLLAMA_HOST controls the network interface and port on which Ollama listens, while OLLAMA_ORIGINS controls which browser origins are allowed to access it.Once I had Ollama running through the Task Scheduler, the correct directory permissions in place, the network binding configured, CORS configured and a shorter model keep-alive period, the setup finally worked as I had intended.
I could use TranslateGemma locally from my IIS-hosted web application without sending the confidential document content to an external AI service. It took a little more configuration than on my personal computer. But that is also what made the exercise interesting.
The actual AI model was only one part of the solution. Getting the surrounding infrastructure right - processes, permissions, networking, browser security and memory management - was just as important.
And that was probably my biggest takeaway from the whole exercise. Running AI locally is not just about choosing a model and installing it. It is also about understanding how that model fits into the environment in which you want to use it.
Although making Ollama directly reachable from the network worked for my setup, I would not consider this the preferred architecture for a production environment. A better approach is to keep Ollama on the server side and put a backend or reverse proxy in front of it, ideally exposing only HTTPS to the clients. This avoids exposing the Ollama API directly and also avoids the CORS and mixed-content issues described above. The downside is that this requires a somewhat larger solution: instead of a single static HTML/JavaScript page communicating directly with Ollama, the application needs a server-side component or reverse-proxy configuration. However, this is a topic for another blog article.
For more information about Ollama, its configuration options, and the Gemma model family, see the following resources: