Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français
Lire en français

Advanced Ollama: Python SDK, structured output, embeddings

46 min read
Improve this page

Logo Ollama

You already know how to use Ollama from the command line (installation and first steps). This guide teaches you to call it from Python. You first check the local REST API, then compare the native SDK, LiteLLM and the OpenAI client pointed at Ollama. Next you get typed JSON with structured outputs, you compute embeddings for semantic search, and you assemble everything into a DevOps assistant that runs locally. The native SDK has one advantage: it gives direct access to Ollama's own options.

What you will learn

  • REST API, native SDK, LiteLLM and OpenAI client: when to use each approach
  • Streaming: display responses in real time
  • Structured output: get typed JSON with Pydantic
  • Embeddings: generate vectors for semantic search
  • Multi-turn conversation: keep context between exchanges
  • Running project: build DevOps Buddy, a local DevOps assistant

Prerequisites

Before you start, make sure you have:

  1. Ollama installed (installation guide)
  2. Python 3.10 or newer: script 07 uses the str | None notation, which does not exist before that version
  3. Two models downloaded: llama3.2:3b for chat, and nomic-embed-text, a model dedicated to embeddings, required by script 06:
Terminal window
ollama pull llama3.2:3b
ollama pull nomic-embed-text
  1. A dedicated Python environment, with the SDK in pinned versions so the examples behave the same on your machine. The ~/lab-ollama directory will hold all the scripts of this guide:
Terminal window
mkdir -p ~/lab-ollama && cd ~/lab-ollama
python3 -m venv .venv && source .venv/bin/activate
pip install ollama==0.6.3 pydantic==2.14.3

Check the versions:

Terminal window
ollama --version # 0.40.x (replayed version: 0.40.16)
pip show ollama # 0.6.3, the version pinned above

Check that the REST API responds

When Ollama is running, it exposes a REST API on port 11434 of your machine. Any program can then talk to the model: the Python SDK, LiteLLM and the OpenAI client seen below are only layers on top of this API. Before testing a script, check that the server responds:

Terminal window
curl http://localhost:11434/api/tags

The output is a JSON object whose models key lists the downloaded models, with their name and size. If the command fails with Connection refused, the server is not started: see Ollama: troubleshooting and configuration.

Native SDK vs LiteLLM: what is the difference?

Both libraries talk to the same Ollama server: the question is not which one "works", but which one locks you in. The native SDK exposes Ollama's API as it is, including the parameters that have no equivalent at other providers. LiteLLM normalizes calls to the OpenAI format, which makes the code portable at the price of the lowest common denominator. In the table below, structured outputs exist on both sides: the native SDK requests them with format=, LiteLLM with response_format (a JSON schema or a Pydantic class, according to the LiteLLM documentation for Ollama). The deciding criterion is therefore portability versus direct access to Ollama's own options, such as think=True or options.

Aspectollama SDKLiteLLM
Installationpip install ollamapip install litellm
ScopeOllama only100+ providers (OpenAI, Anthropic...)
Structured outputYes, native with format=Yes, via response_format (JSON schema)
EmbeddingsYes, built-in embed()Yes, embedding()
ThinkingYes, think=TrueNot documented for Ollama
Use caseScripts dedicated to OllamaPortable multi-provider code

In practice, take the native SDK if you only target Ollama and want to use its specific features. Take LiteLLM if you want to be able to switch to other providers. A third case exists: code that already uses the openai library, covered just below.

The OpenAI client: reuse existing code

Ollama also exposes an OpenAI-compatible endpoint under the /v1/ path (chat completions, embeddings, model list). If your code already calls the openai library, there is no need to rewrite it: you only have to change the base URL to point at your local server. The client requires an API key, but Ollama ignores it: any string will do, here ollama.

Terminal window
pip install openai==3.49.0
00_client_openai.py
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1/", api_key="ollama")
response = client.chat.completions.create(
model="llama3.2:3b",
messages=[{"role": "user", "content": "Explique une API REST en 3 phrases simples"}],
)
print(response.choices[0].message.content)

The prompt is in French: it asks the model to explain a REST API in 3 simple sentences, so the model answers in French.

The syntax also differs from the native SDK. The response follows the OpenAI format (response.choices[0].message.content), whereas the native SDK returns response.message.content directly, as in the scripts of this lesson. The other Ollama lessons on the site write response["message"]["content"]: the two forms are equivalent, because the SDK returns an object that can be accessed by attribute as well as by key.

Ollama's own options, such as think=True or options, remain the territory of the native SDK. So choose the OpenAI client to plug Ollama into code already written for an OpenAI service, and the native SDK for every new script dedicated to Ollama.

Running project: DevOps Buddy

Throughout this guide, you build DevOps Buddy, a local DevOps assistant that:

  • Answers DevOps questions in streaming
  • Uses a knowledge base with embeddings (mini-RAG)
  • Returns structured commands with Pydantic

The numbered scripts (01_... to 07_...) all go in the ~/lab-ollama directory created in the prerequisites, with the virtual environment activated. If you closed your terminal in the meantime, reactivate it with cd ~/lab-ollama && source .venv/bin/activate.

Basic chat with the native SDK

The SDK's chat() function is the equivalent of litellm.completion(), but with a slightly different API.

Script 01: First call

This first script is mainly meant to show you the shape of the response object. Look at the metadata part (headed "Métadonnées" in the output) rather than the generated text: prompt_eval_count and eval_count count the tokens (the pieces of words the model reads and produces) in input and output, and total_duration is expressed in nanoseconds, hence the division by 1e9. These three values let you measure the real cost of a local call, since no billing comes to remind you of it.

The scripts of this lesson are kept as they were tested, with their prompts, comments and messages in French: the outputs shown below match them exactly, and the model answers in French. Here, "Réponse du modèle" simply means "the model's answer".

01_chat_basique.py
#!/usr/bin/env python3
"""
TP 01 : Appel basique avec le SDK Ollama natif
Objectif : Comprendre la différence avec litellm
"""
from ollama import chat
# Appel simple avec le SDK natif
response = chat(
model='llama3.2:3b',
messages=[
{'role': 'user', 'content': 'Explique Docker en une phrase pour un débutant.'}
]
)
print("=== Réponse du modèle ===")
print(response.message.content)
print("\n=== Métadonnées ===")
print(f"Modèle utilisé : {response.model}")
print(f"Tokens prompt : {response.prompt_eval_count}")
print(f"Tokens réponse : {response.eval_count}")
print(f"Durée totale : {response.total_duration / 1e9:.2f}s")

Run the script:

Terminal window
python 01_chat_basique.py

Expected output:

=== Réponse du modèle ===
Docker est une technologie qui permet de containeriser les applications, ce qui
signifie que chaque application est placée dans son propre environnement isolé et
sécurisé, avec ses propres dépendances et ressources, ce qui facilite la
déploiement, la gestion et la sécurité des applications.
=== Métadonnées ===
Modèle utilisé : llama3.2:3b
Tokens prompt : 36
Tokens réponse : 65
Durée totale : 4.51s

The text and the duration vary from one run to the next, and this output comes from a processor only (Ollama 0.40.16).

Streaming: real-time responses

Streaming displays the response word by word, as it is generated. On a local model, generating a response takes several seconds: without streaming, the screen stays empty until the last line.

Script 02: Streaming

With stream=True, chat() no longer returns an object but a generator: the for loop consumes the fragments at the pace of generation. Two practical consequences. The closing metadata (eval_count, total_duration) only arrives in the last fragment, not at the start. And if you need the full text afterwards, you have to concatenate it yourself during the loop, as DevOps Buddy does further down.

02_streaming.py
#!/usr/bin/env python3
"""
TP 02 : Streaming avec le SDK Ollama
Objectif : Afficher la réponse en temps réel, mot par mot
"""
from ollama import chat
# Streaming : la réponse arrive progressivement
print("=== Réponse en streaming ===")
stream = chat(
model='llama3.2:3b',
messages=[
{'role': 'user', 'content': 'Donne 5 bonnes pratiques Dockerfile, très bref.'}
],
stream=True # Active le streaming
)
# Afficher chaque chunk au fur et à mesure
for chunk in stream:
print(chunk.message.content, end='', flush=True)
print("\n\n=== Fin du streaming ===")

The flush=True parameter forces each fragment to be displayed immediately, without waiting for the end of the line.

Structured output with Pydantic

This feature saves you from writing a parser for free text. Instead of a paragraph, you get typed JSON that matches a Pydantic schema. Pydantic is a Python library that describes a data structure with a class, and checks that a piece of data conforms to it.

Why does it matter?

When you ask "answer in JSON" without a schema, the model often returns clean JSON, but not always: it sometimes wraps it in a Markdown code block (three backticks), and json.loads then rejects it. A single response of this kind is enough to break a script in production. With format=, Ollama constrains decoding token by token: the output is forced to follow the schema, as long as generation runs to the end (script 04 explains the opposite case). The four benefits below follow from this constraint on the server side, not from an instruction in the prompt.

  • Reliable parsing: no more need for regexes or manual parsing
  • Automatic validation: Pydantic checks that the response conforms
  • Typing: your IDE knows the field types
  • Integration: the data is directly usable in your code

Script 03: Basic structured output

Two calls do all the work, and they must not be confused. DockerCommand.model_json_schema() converts the Pydantic class into a JSON schema sent to the server, which uses it to constrain generation. DockerCommand.model_validate_json() reads the response back and instantiates the Python object. The schema constrains the shape, not the content: the level field will indeed be a string, but nothing prevents the model from writing intermédiaire instead of the three expected values. To lock the values too, declare a Literal["debutant", "intermediaire", "avance"] rather than a str.

03_structured_output.py
#!/usr/bin/env python3
"""
TP 03 : Structured Output avec Pydantic
Objectif : Obtenir des réponses JSON typées et validées automatiquement
Feature : Disponible depuis Ollama 0.5.0+
"""
from ollama import chat
from pydantic import BaseModel
# Définir le schéma de sortie avec Pydantic
class DockerCommand(BaseModel):
"""Une commande Docker expliquée"""
command: str # La commande complète
description: str # Explication en une phrase
example: str # Un exemple concret
level: str # "debutant", "intermediaire", "avance"
# Appeler le modèle avec le schéma
response = chat(
model='llama3.2:3b',
messages=[
{
'role': 'user',
'content': 'Explique la commande docker build'
}
],
format=DockerCommand.model_json_schema() # Forcer le format JSON
)
# Valider et parser la réponse avec Pydantic
result = DockerCommand.model_validate_json(response.message.content)
# Afficher le résultat structuré
print("=== Sortie structurée (Pydantic) ===")
print(f"Commande : {result.command}")
print(f"Description : {result.description}")
print(f"Exemple : {result.example}")
print(f"Niveau : {result.level}")
print("\n=== JSON brut ===")
print(response.message.content)

Expected output:

=== Sortie structurée (Pydantic) ===
Commande : docker build
Description : Construction d'un conteneur Docker à partir d'un fichier Dockerfile
Exemple : docker build -t mon-image:latest .
Niveau : utilise pour construire un conteneur Docker à partir d'un fichier Dockerfile dans le répertoire courant.
=== JSON brut ===
{"command": "docker build", "description": "Construction d'un conteneur Docker à partir d'un fichier Dockerfile", "example": "docker build -t mon-image:latest .", "level": "utilise pour construire un conteneur Docker à partir d'un fichier Dockerfile dans le répertoire courant."}

The generated text changes with every run. Look at the "Niveau" line (the label of the level field): level is a plain str, so nothing forces the model to answer "debutant". Here, it wrote a whole sentence in place of the level (Ollama 0.40.16), and depending on the run Administrateur, avancé or Intermediate. To constrain the values, use an enumeration (the lesson on structured outputs, cited after script 04, covers it in detail).

Script 04: Lists and nested objects

A Pydantic schema can contain other schemas: Tutorial references a list of Step, and Pydantic generates a nested JSON schema that Ollama can follow. Two options deserve attention in the call. temperature: 0 makes the output almost deterministic, which matters when you extract data rather than write prose. num_predict: 500 caps the number of generated tokens: a nested structure cut off mid-way produces invalid JSON and makes validation fail. The lower the cap, the more this risk grows with a nested schema.

04_structured_avance.py
#!/usr/bin/env python3
"""
TP 04 : Structured Output avancé (listes et objets imbriqués)
Objectif : Extraire plusieurs éléments structurés en une seule requête
"""
from ollama import chat
from pydantic import BaseModel
class Step(BaseModel):
"""Une étape d'un processus"""
number: int
action: str
command: str
class Tutorial(BaseModel):
"""Un tutoriel complet"""
title: str
steps: list[Step]
total_time: str
print("Définition du schéma Pydantic...")
print("\nEnvoi de la requête à Ollama (llama3.2:3b)...")
# Demander un tutoriel structuré
response = chat(
model='llama3.2:3b',
messages=[
{
'role': 'user',
'content': 'Donne un tutoriel Docker en 2 étapes avec titre, steps (number, action, command) et total_time'
}
],
format=Tutorial.model_json_schema(),
options={'temperature': 0, 'num_predict': 500}
)
print("Réponse reçue!")
print(f" Durée: {response.total_duration / 1e9:.2f}s")
# Parser la réponse
tutorial = Tutorial.model_validate_json(response.message.content)
# Afficher le tutoriel formaté
print(f"\nTutoriel : {tutorial.title}")
print(f"Durée : {tutorial.total_time}")
for step in tutorial.steps:
print(f" {step.number}. {step.action} → {step.command}")

Expected output:

Définition du schéma Pydantic...
Envoi de la requête à Ollama (llama3.2:3b)...
Réponse reçue!
Durée: 12.68s
Tutoriel : Tutoriel Docker en 2 étapes
Durée : }
1. Créer un nouveau projet Docker → docker run -it ubuntu /bin/bash
2. Créer un fichier Dockerfile et l'exécuter → docker build -t mon-image .

As with script 03, the content varies from one run to the next. With a 3-billion-parameter model, the duration field comes back here with an unexpected value (}, observed on Ollama 0.40.16), while the structure remains valid.

To go further on schemas (hand-written JSON Schema, enumerations, limits of the guarantee), the lesson Structured outputs with Ollama covers the format= mechanism in detail, with recent models tested.

Multi-turn conversation with history

To keep context between several exchanges, you have to manage the message history.

Script 05: Conversation

The Ollama server remembers nothing between two calls: the conversation's memory is your list messages. At each turn, you send back the whole history, questions and answers included. The cost therefore grows with each exchange, which you can see in prompt_eval_count, until it saturates the model's context window (the maximum amount of text it reads at once). For an assistant meant to last, plan a truncation of the history or a summary of old turns; the script below, limited to four exchanges, does not need one.

05_conversation.py
#!/usr/bin/env python3
"""
TP 05 : Conversation multi-turn avec historique
Objectif : Maintenir le contexte entre plusieurs échanges
"""
from ollama import chat
# Historique de la conversation
messages = []
# Contexte système
system_message = {
'role': 'system',
'content': '''Tu es DevOps Buddy, un assistant spécialisé DevOps.
Tu réponds de manière concise et technique.
Tu donnes des exemples de commandes quand c'est pertinent.'''
}
messages.append(system_message)
print("DevOps Buddy initialisé")
print("=" * 50)
def ask(question: str) -> str:
"""Pose une question et met à jour l'historique"""
print(f"\nVous : {question}")
# Ajouter la question à l'historique
messages.append({'role': 'user', 'content': question})
# Appeler le modèle avec tout l'historique
response = chat(
model='llama3.2:3b',
messages=messages,
options={'temperature': 0.3}
)
# Ajouter la réponse à l'historique
assistant_message = response.message.content
messages.append({'role': 'assistant', 'content': assistant_message})
print(f"Buddy : {assistant_message}")
return assistant_message
# Simulation d'une conversation
ask("C'est quoi un Pod Kubernetes en une phrase ?")
ask("Et un Deployment, c'est quoi par rapport à un Pod ?")
ask("Donne-moi la commande pour créer un Deployment avec 3 réplicas")
ask("Comment je scale à 5 réplicas maintenant ?")
print("\n" + "=" * 50)
print(f"Total messages dans l'historique : {len(messages)}")

Real output (Ollama 0.40.16, ollama SDK 0.6.3, llama3.2:3b on processor only; the response varies from one run to the next):

DevOps Buddy initialisé
==================================================
Vous : C'est quoi un Pod Kubernetes en une phrase ?
Buddy : Un Pod Kubernetes est une unité de déployement et de gestion de conteneurs, qui est une combinaison logique de l'ensemble d'un conteneur et de ses dépendances, permettant une gestion plus fine-graine des ressources et des communications entre les conteneurs.
Vous : Et un Deployment, c'est quoi par rapport à un Pod ?
Buddy : Un Deployment Kubernetes est une ressource qui définit la stratégie de déploiement d'un ou plusieurs Pods, en spécifiant les conditions de déploiement, les versions des images, les stratégies d'actualisation et les politiques de gestion des ressources.
Vous : Donne-moi la commande pour créer un Deployment avec 3 réplicas
Buddy : La commande pour créer un Deployment avec 3 réplicas est la suivante :
kubectl apply -f deployment.yaml
Et dans votre fichier deployment.yaml (extrait) :
metadata:
name: mon-deployment
spec:
replicas: 3
Notez que vous devez remplacer mon-deployment, mon-app, mon-image par vos propres valeurs.
Vous : Comment je scale à 5 réplicas maintenant ?
Buddy : Pour scaler un Deployment à 5 réplicas, vous pouvez utiliser la commande suivante :
kubectl scale deployment mon-deployment --replicas=5
Cette commande utilise la commande scale pour ajuster le nombre de réplicas du Deployment mon-deployment à 5.
==================================================
Total messages dans l'historique : 9

The output is abridged: the full YAML manifest of the third answer and the model's code fences are not reproduced.

Look at the last question, "Comment je scale à 5 réplicas maintenant ?" (how do I scale to 5 replicas now?). It names no Deployment, and yet the answer targets mon-deployment. The model made up that name itself in its previous answer: it picks it up because the messages list is sent back to it on every call. This is the proof that the context holds.

A 3-billion-parameter model sometimes writes approximate French, such as "déployement" and "fine-graine" in the first answer. This flaw comes from the size of the model, not from your script.

Embeddings turn text into numerical vectors. Two similar texts will have close vectors. This is the basis of RAG (Retrieval-Augmented Generation): retrieve the relevant documents, then give them to the model before it answers.

Script 06: Embeddings and similarity

The embed() function always returns a list of vectors, even for a single input string, hence the repeated response.embeddings[0] in the script. Cosine similarity measures the angle between two vectors, not their distance: it equals 1 for two vectors pointing the same way. The scores of the script nevertheless all stay between 0.55 and 0.66, even for documents that answer the question poorly. So never compare a score in absolute terms: only the relative ranking between documents is meaningful, and it depends entirely on the model used.

06_embeddings.py
#!/usr/bin/env python3
"""
TP 06 : Embeddings avec Ollama
Objectif : Générer des vecteurs pour la recherche sémantique
Feature : endpoint /api/embed disponible depuis Ollama 0.3.4
"""
from ollama import embed
import math
def cosine_similarity(vec1: list[float], vec2: list[float]) -> float:
"""Calcule la similarité cosinus entre deux vecteurs"""
dot_product = sum(a * b for a, b in zip(vec1, vec2))
norm1 = math.sqrt(sum(a ** 2 for a in vec1))
norm2 = math.sqrt(sum(b ** 2 for b in vec2))
return dot_product / (norm1 * norm2)
# Base de connaissances DevOps
documents = [
"Docker est une plateforme de conteneurisation qui permet de packager des applications",
"Kubernetes orchestre les conteneurs à grande échelle dans un cluster",
"Ansible automatise la configuration des serveurs et le déploiement",
"Terraform gère l'infrastructure as code pour le cloud",
"GitLab CI/CD permet l'intégration et le déploiement continus"
]
print("Génération des embeddings pour 5 documents...")
# Générer les embeddings pour chaque document
doc_embeddings = []
for i, doc in enumerate(documents):
response = embed(model='nomic-embed-text', input=doc)
doc_embeddings.append(response.embeddings[0])
print(f" Doc {i+1} : {len(response.embeddings[0])} dimensions")
print(f"\nDimension des vecteurs : {len(doc_embeddings[0])}")
# Question de test
query = "Comment déployer des conteneurs en production ?"
print(f"\nRequête : {query}")
# Générer l'embedding de la requête
query_response = embed(model='nomic-embed-text', input=query)
query_embedding = query_response.embeddings[0]
# Calculer la similarité avec chaque document
print("\nScores de similarité :")
scores = []
for i, doc_emb in enumerate(doc_embeddings):
score = cosine_similarity(query_embedding, doc_emb)
scores.append((score, documents[i]))
print(f" {score:.4f} → {documents[i][:50]}...")
# Trier par score décroissant
scores.sort(key=lambda s: s[0], reverse=True)
print(f"\nDocument le plus pertinent :")
print(f" Score: {scores[0][0]:.4f}")
print(f" {scores[0][1]}")

Expected output:

Génération des embeddings pour 5 documents...
Doc 1 : 768 dimensions
Doc 2 : 768 dimensions
Doc 3 : 768 dimensions
Doc 4 : 768 dimensions
Doc 5 : 768 dimensions
Dimension des vecteurs : 768
Requête : Comment déployer des conteneurs en production ?
Scores de similarité :
0.6602 → Docker est une plateforme de conteneurisation qui ...
0.6462 → Kubernetes orchestre les conteneurs à grande échel...
0.5821 → Ansible automatise la configuration des serveurs e...
0.5903 → Terraform gère l'infrastructure as code pour le cl...
0.5528 → GitLab CI/CD permet l'intégration et le déploiemen...
Document le plus pertinent :
Score: 0.6602
Docker est une plateforme de conteneurisation qui permet de packager des applications

The ranking is consistent: for the question "Comment déployer des conteneurs en production" (how to deploy containers in production), Docker and Kubernetes come first, ahead of Terraform, Ansible and GitLab CI/CD. Your scores may differ by a few hundredths depending on the model version and the processor, but Docker and Kubernetes must stay on top. This is what a dedicated embedding model brings: its vectors are trained so that texts with a close meaning end up as neighbors.

A chat model such as llama3.2:3b is not made for this. On Ollama 0.40.16, embed(model='llama3.2:3b', ...) fails with HTTP error 501 "This server does not support embeddings. Start it with --embeddings". If you run into it, replace the model with nomic-embed-text.

The lesson Embeddings and similarity (in French) covers these notions (dimension, cosine similarity, model choice) outside any assistant.

DevOps Buddy: the complete assistant

Let us combine all the features into an interactive DevOps assistant.

Script 07: DevOps Buddy

The script is long, read it by blocks rather than line by line. Three functions carry the logic. find_relevant_context() performs the retrieval: it compares the embedding of the question with those of the knowledge base and returns the two closest documents. chat_with_rag() performs the augmentation: it pastes that context in front of the question before sending it to the model. get_structured_command() serves the /cmd mode with a structured output. The rest is terminal dressing.

Three limits set this prototype apart from a real RAG system. The embeddings are recomputed at every startup and live in memory: no persistence, no vector database (storage designed to search for close vectors). The search is linear: it compares the question with every document. On six entries that is instant, but on thousands of documents you need a dedicated index. Finally, the embedding model (the EMBED_MODEL constant) is distinct from the chat model (MODEL), as in script 06: if you have not pulled nomic-embed-text, startup fails.

07_devops_buddy.py
#!/usr/bin/env python3
"""
TP 07 : DevOps Buddy - Assistant local complet
Objectif : Combiner toutes les fonctionnalités Ollama dans un assistant interactif
Features utilisées : chat, streaming, structured output, embeddings
"""
from ollama import chat, embed
from pydantic import BaseModel
import math
# === Configuration ===
MODEL = 'llama3.2:3b'
EMBED_MODEL = 'nomic-embed-text' # modèle dédié aux embeddings (ollama pull nomic-embed-text)
SYSTEM_PROMPT = '''Tu es DevOps Buddy, un assistant expert DevOps/DevSecOps.
Tu réponds en français de manière concise et technique.
Tu donnes des exemples de commandes quand c'est pertinent.
Tu structures tes réponses avec des listes à puces.'''
# === Base de connaissances embarquée ===
KNOWLEDGE_BASE = [
{"topic": "docker", "content": "Docker utilise des conteneurs pour isoler les applications. Commandes: docker build, docker run, docker compose"},
{"topic": "kubernetes", "content": "Kubernetes orchestre les conteneurs. Concepts: Pod, Deployment, Service, Ingress. Commandes: kubectl get, kubectl apply"},
{"topic": "ansible", "content": "Ansible automatise la configuration. Concepts: playbook, inventory, role, task. Commande: ansible-playbook"},
{"topic": "terraform", "content": "Terraform gère l'infrastructure as code. Concepts: provider, resource, module, state. Commandes: terraform init, plan, apply"},
{"topic": "gitlab", "content": "GitLab CI/CD pipeline avec .gitlab-ci.yml. Concepts: stages, jobs, artifacts, cache, runners"},
{"topic": "prometheus", "content": "Prometheus collecte les métriques. Concepts: scraping, PromQL, alertmanager, exporters"},
]
# Pré-calculer les embeddings de la base de connaissances
print("Chargement de la base de connaissances...")
knowledge_embeddings = []
for item in KNOWLEDGE_BASE:
response = embed(model=EMBED_MODEL, input=item["content"])
knowledge_embeddings.append(response.embeddings[0])
print(f"{len(KNOWLEDGE_BASE)} documents indexés\n")
# === Fonctions utilitaires ===
def cosine_similarity(vec1: list[float], vec2: list[float]) -> float:
dot = sum(a * b for a, b in zip(vec1, vec2))
norm1 = math.sqrt(sum(a ** 2 for a in vec1))
norm2 = math.sqrt(sum(b ** 2 for b in vec2))
return dot / (norm1 * norm2) if norm1 and norm2 else 0
def find_relevant_context(query: str, top_k: int = 2) -> str:
"""Recherche les documents pertinents pour enrichir le contexte"""
query_emb = embed(model=EMBED_MODEL, input=query).embeddings[0]
scores = []
for i, doc_emb in enumerate(knowledge_embeddings):
score = cosine_similarity(query_emb, doc_emb)
scores.append((score, KNOWLEDGE_BASE[i]))
scores.sort(key=lambda s: s[0], reverse=True)
top_docs = scores[:top_k]
context = "\n".join([f"- {doc['content']}" for _, doc in top_docs])
return context
# === Schéma pour les commandes structurées ===
class CommandSuggestion(BaseModel):
command: str
description: str
warning: str | None = None
def get_structured_command(query: str) -> CommandSuggestion | None:
"""Obtenir une suggestion de commande structurée"""
try:
response = chat(
model=MODEL,
messages=[
{'role': 'system', 'content': 'Tu suggères des commandes DevOps. Réponds uniquement en JSON.'},
{'role': 'user', 'content': f'Suggère une commande pour: {query}'}
],
format=CommandSuggestion.model_json_schema(),
options={'temperature': 0, 'num_predict': 200}
)
return CommandSuggestion.model_validate_json(response.message.content)
except Exception as e:
print(f"Erreur structured output: {e}")
return None
# === Chat avec RAG ===
def chat_with_rag(question: str, history: list) -> str:
"""Répond à une question avec contexte enrichi"""
# Trouver le contexte pertinent
context = find_relevant_context(question)
# Construire le prompt enrichi
augmented_question = f"""Contexte (base de connaissances):
{context}
Question de l'utilisateur: {question}"""
# Ajouter au historique
history.append({'role': 'user', 'content': augmented_question})
# Appeler le modèle avec streaming
response_text = ""
stream = chat(
model=MODEL,
messages=history,
stream=True,
options={'temperature': 0.3}
)
for chunk in stream:
content = chunk.message.content
print(content, end='', flush=True)
response_text += content
print() # Nouvelle ligne après le streaming
# Ajouter la réponse à l'historique
history.append({'role': 'assistant', 'content': response_text})
return response_text
# === Main ===
def main():
print("=" * 60)
print("DevOps Buddy - Assistant DevOps Local (Ollama)")
print("=" * 60)
print(f"Modèle: {MODEL}")
print("Commandes spéciales:")
print(" /cmd <action> : Obtenir une commande structurée")
print(" /quit : Quitter")
print("-" * 60)
history = [{'role': 'system', 'content': SYSTEM_PROMPT}]
while True:
try:
user_input = input("\nVous: ").strip()
if not user_input:
continue
if user_input.lower() == '/quit':
print("À bientôt!")
break
if user_input.startswith('/cmd '):
# Mode commande structurée
query = user_input[5:]
print("\nRecherche de commande...")
result = get_structured_command(query)
if result:
print(f"\nCommande: {result.command}")
print(f"Description: {result.description}")
if result.warning:
print(f"Attention: {result.warning}")
else:
print("Impossible de générer la commande")
else:
# Mode conversation avec RAG
print("\nDevOps Buddy: ", end='')
chat_with_rag(user_input, history)
except EOFError:
print("\nÀ bientôt!")
break
except KeyboardInterrupt:
print("\nInterruption - À bientôt!")
break
if __name__ == "__main__":
main()

Using DevOps Buddy

Startup takes a few seconds, the time to compute the six embeddings of the base. The assistant then offers two modes that are worth comparing: a free question goes through RAG with streaming, while the /cmd prefix switches to the structured output and displays a command, its description and an optional warning. The difference can be felt in use: the conversation mode displays the first words almost immediately, whereas the /cmd mode stays silent until the full JSON has been generated and validated, since a partial JSON would be unusable.

Terminal window
python 07_devops_buddy.py

Example session:

Chargement de la base de connaissances...
6 documents indexés
============================================================
DevOps Buddy - Assistant DevOps Local (Ollama)
============================================================
Modèle: llama3.2:3b
Commandes spéciales:
/cmd <action> : Obtenir une commande structurée
/quit : Quitter
------------------------------------------------------------
Vous: C'est quoi un Pod ?
DevOps Buddy: **Définition d'un Pod**
Un Pod est la principale unité de déployement et de gestion des conteneurs dans Kubernetes. [...]
Vous: /cmd lister les pods kubernetes
Recherche de commande...
Commande: kubectl
Description: Lister les pods Kubernetes
Attention: Assurez-vous d'être connecté à votre cluster Kubernetes pour exécuter cette commande

The session above is real output from Ollama 0.40.16, abridged at the spot marked [...]. It shows the limit of a 3-billion-parameter model: the returned command is plain kubectl, without get pods, even though the JSON schema is perfectly respected. The schema constrains the shape, never the correctness of the content. It does not protect against a cut-off either: if num_predict (200 tokens here) interrupts generation, the JSON is incomplete and model_validate_json raises an error, which is why get_structured_command() catches it and returns None.

Checklist

Go through this list before moving on: each script isolates one capability of the SDK, and DevOps Buddy brings them together. If 03_structured_output.py fails, there is no point launching DevOps Buddy, which uses the same mechanism for its /cmd mode.

Installation

These four checks rule out installation failures before you touch the code. The first two read the version of the server and that of the SDK, which evolve separately. The third checks Pydantic, the fourth that both models are downloaded.

  • Server reachable (curl http://localhost:11434/api/tags) and version read (ollama --version)
  • ollama SDK installed (pip show ollama)
  • Pydantic installed (pip show pydantic)
  • Models downloaded: llama3.2:3b and nomic-embed-text both appear in ollama list

Tested scripts

Run them in order. If 06_embeddings.py or 07_devops_buddy.py fails with model not found, run ollama pull nomic-embed-text. A 501 error means a chat model was passed to embed(). Close scores (around 0.55 to 0.66) are normal output: the good sign is Docker and Kubernetes at the top of the ranking.

  • 01_chat_basique.py: response + metadata
  • 02_streaming.py: real-time display
  • 03_structured_output.py: JSON validated by Pydantic
  • 04_structured_avance.py: lists of objects
  • 05_conversation.py: multi-turn history
  • 06_embeddings.py: cosine similarity
  • 07_devops_buddy.py: complete assistant

Key takeaways

  1. Native SDK or LiteLLM. Both produce structured JSON (format= on the SDK side, response_format on the LiteLLM side). The ollama SDK adds direct access to Ollama's own options (think=True, options), but it only talks to Ollama.

  2. Structured output. Pass Model.model_json_schema() to format= to get JSON that conforms to the schema. The shape is constrained, not the content, and a response cut off by num_predict remains invalid.

  3. Streaming. With stream=True, the response arrives in fragments, and the metadata in the last one.

  4. Embeddings. embed() returns vectors, to be computed with a dedicated model such as nomic-embed-text, never with a chat model.

  5. History. The server retains nothing: you send back the complete list of messages on every call.

  6. Options. options sets the temperature, the maximum number of tokens, etc.

  7. Local RAG. Embeddings find the close documents, which you paste in front of the question before sending it to the model.

Knowledge check

Check that you have the essentials of this guide. The questions cover only what was just explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions
6 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer