Skip to main content

Optional Ollama integration testing (GPU self-hosted runner)

Mock-based pytest remains the required CI gate on every merge request. This document describes an optional live integration lane for a self-hosted runner with Ollama already running (for example a desktop with a GPU).

Why add this lane​

LayerRuns onValidates
test:pytest (mocks)Any runnerProvider wiring, errors, JSON parsing, M4 pipeline
test:integration:ollama (optional)Tagged GPU runnerReal Ollama /v1, prompt, model output, latency

Use the optional job when you want confidence that --provider ollama works against a live model — not as a substitute for mock tests.

Prerequisites (runner host)​

  1. GitLab runner registered with tag gpu-ollama (shell executor on your desktop is fine).
  2. Ollama running (ollama serve or OS service).
  3. Model pulled, e.g. ollama pull llama3.2.
  4. Python 3.11+ and editable install: pip install -e "./packages/a2c-core" -e "./packages/a2c-cli" -e "./packages/a2c-tui" -e ".[dev]" (or let the CI job do it).
  5. a2c on PATH after install.

Default Ollama address: http://127.0.0.1:11434 (override with OLLAMA_HOST).

Local smoke (no CI)​

# Unix
./scripts/smoke-ollama-decompose.sh

# Windows
.\scripts\smoke-ollama-decompose.ps1

Optional:

OLLAMA_MODEL=mistral ./scripts/smoke-ollama-decompose.sh

The script copies tests/fixtures/repos/decompose_source/, runs a2c plan decompose-epic epic-onboarding --provider ollama, and checks for a proposal summary plus at least one task- id. It does not run apply (no writes to a real repo).

GitLab CI job​

Job name: test:integration:ollama in .gitlab-ci.yml.

PropertyValue
tagsgpu-ollama
allow_failuretrue (optional lane; does not block merge by default)
TriggersPipeline schedule, manual Run pipeline, or RUN_OLLAMA_INTEGRATION=true

The job does not use a Docker image: — it runs on the runner host where Ollama lives.

Enable on a pipeline manually​

GitLab → Run pipeline → add variable:

RUN_OLLAMA_INTEGRATION = true

Register the runner (summary)​

  1. Install GitLab Runner on the desktop.
  2. Register with your project/group and set tag gpu-ollama.
  3. Use shell executor (or equivalent that can reach local Ollama).
  4. Ensure the runner user can run pip install -e . and a2c.

What we assert (and what we avoid)​

Assert (soft):

  • Ollama reachable (/api/tags)
  • a2c plan decompose-epic exits 0
  • Output mentions proposed tasks and a task- id

Do not assert:

  • Exact task titles or count (models are non-deterministic)
  • Golden JSON snapshots of LLM output

Operational notes​

  • Concurrency: run one Ollama integration job at a time if GPU memory is tight (resource_group in GitLab).
  • Availability: pipeline succeeds on shared runners even when the desktop is off; this job stays skipped or allow_failure.
  • Pin the model via OLLAMA_MODEL (default llama3.2) for comparable runs over time.