WIP: Configure deepeval and add first behavioural tests #2

Draft

robinlol wants to merge 19 commits from llm-eval-experiment into main

pull from: llm-eval-experiment

merge into: clan:main

clan:main

clan:benchmark-experiment

clan:collin/initial-setup

Conversation 0 Commits 19 Files Changed 8

+935 -2

robinlol commented 2026-02-12 13:30:54 +00:00

Owner

Copy Link

Copy Source

• Updated dependencies: added deepeval to pyproject.toml, updated flake.nix with required packages, and added .envrc for local env var loading.

• Integrated DeepEval as an LLM evaluation framework for testing the clan deployment planner agent, replacing earlier mock-only tests with real end-to-end evaluation.

• Added a GEval helpfulness metric (deepeval_metrics.py) with a shared AGENT_CONTEXT description that frames evaluation criteria around the agent's actual job as generating clan NixOS service configuration proposals,
  not shell commands or sysadmin instructions.

• Implemented the first real capability test (test_agent_helpfulness.py): runs the full agent pipeline against a live LLM (Claude by default, other providers configurable), mocking only flake-dependent I/O and evaluates the response for helpfulness

• Made the LLM provider configurable in the test suite — supports claude, ollama, and openai via a top-level PROVIDER constant

• Updated Ollama model to qwen3:14b.

• Included proposed NixOS service instances in evaluation output, so the helpfulness metric scores both the agent's message and the structured configuration it proposed.

• Added logging to surface the full DeepEval comparison (input, agent message, proposed config, score, reason) in test output for easier debugging.

• Expanded evaluation metrics: Added new DeepEval-based metrics and significantly extended the helpfulness test suite with more test cases covering a broader range of agent behaviors

• Documented the test harness: Added a section to the README explaining how the LLM evaluation framework works and how to run it

- Updated dependencies: added deepeval to pyproject.toml, updated flake.nix with required packages, and added .envrc for local env var loading. - Integrated DeepEval as an LLM evaluation framework for testing the clan deployment planner agent, replacing earlier mock-only tests with real end-to-end evaluation. - Added a GEval helpfulness metric (deepeval_metrics.py) with a shared AGENT_CONTEXT description that frames evaluation criteria around the agent's actual job as generating clan NixOS service configuration proposals, not shell commands or sysadmin instructions. - Implemented the first real capability test (test_agent_helpfulness.py): runs the full agent pipeline against a live LLM (Claude by default, other providers configurable), mocking only flake-dependent I/O and evaluates the response for helpfulness - Made the LLM provider configurable in the test suite — supports claude, ollama, and openai via a top-level PROVIDER constant - Updated Ollama model to qwen3:14b. - Included proposed NixOS service instances in evaluation output, so the helpfulness metric scores both the agent's message and the structured configuration it proposed. - Added logging to surface the full DeepEval comparison (input, agent message, proposed config, score, reason) in test output for easier debugging. - Expanded evaluation metrics: Added new DeepEval-based metrics and significantly extended the helpfulness test suite with more test cases covering a broader range of agent behaviors - Documented the test harness: Added a section to the README explaining how the LLM evaluation framework works and how to run it

robinlol added 4 commits 2026-02-12 13:30:54 +00:00

Added deepeval to Python dependencies 3bca49f848

Added missing package dependency for deepeval 150c1b5e1b

Added loading env vars from local dotfile 6cd6a8042d

Added some mock deepeval tests to the test suite 961ca0b707

robinlol changed title from [DRAFT] Configure deepeval and add mock tests to test suite to WIP: Configure deepeval and add mock tests to test suite 2026-02-12 13:34:06 +00:00

robinlol added 1 commit 2026-02-12 13:39:18 +00:00

Made teafmt happy 5b4e99426d

robinlol added 2 commits 2026-02-12 13:40:35 +00:00

Made treefmt happy 1a34f4d2e4

Merge remote-tracking branch 'refs/remotes/origin/llm-eval-experiment' into llm-eval-experiment 6e568f6ec4

robinlol added 4 commits 2026-02-20 14:25:36 +00:00

Upgraded deepeval model to sonnet4.6 7bebe182bf

Added logging config to show prompts in deepeval tests 4c6a419985

Removed deepeval mock tests fd894d6853

Added first actual deepeval capability test that checks helpfulness metric 9aa0951108

robinlol added 2 commits 2026-02-20 14:56:30 +00:00

Updated ollama model to qwen3:14b ba9678789f

Made provider configurable in helpfulness test 52b55372a8

robinlol added 1 commit 2026-02-20 15:00:40 +00:00

Added analysing the proposed_instances in the helpfulness metric 5434636254

robinlol added 1 commit 2026-02-20 15:54:29 +00:00

Reverted ollama model to qwen3:4b-instruct c7ac83131c

robinlol added 1 commit 2026-02-20 15:58:06 +00:00

Made treefmt happy b038fa1ff8

robinlol changed title from WIP: Configure deepeval and add mock tests to test suite to WIP: Configure deepeval and add first behavioural tests 2026-02-20 15:58:57 +00:00

robinlol added 2 commits 2026-02-27 11:00:16 +00:00

Added more evaluation metrics and test-cases b30b99efa1

Added documentation of test harness to readme d389cf60ed

robinlol added 1 commit 2026-02-27 11:04:10 +00:00

Renamed test harness file 55d4214ba2

All checks were successful

Hide all checks

buildbot/nix-eval Build done.

Details

buildbot/nix-build Build done.

Required

Details

This pull request is marked as a work in progress.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin llm-eval-experiment:llm-eval-experiment

git checkout llm-eval-experiment

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

Good first issue

Help wanted

OS

Darwin

OS

Linux

Priority

Critical

1

Priority

High

2

Priority

Low

4

Priority

Medium

3

Status

Available

Status

Blocked

1

Status

Duplicate

Status

In Progress

5

Status

Needs More Info

3

Status

Needs Review

4

Status

On Hold

2

Status

Won't Fix

6

Type

Bug

2

Type

Community Contribution

1

contributed by members of the Clan community

Type

Discussion

1

Type

Documentation

Type

Enhancement

6

Type

Epic

4

Type

Feature Request

5

Type

Maintenance

7

Type

Question

Question to be resolved

Type

Question

3

Type

User Experience

5

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

admin brianmcgee clan-bot collinarnett DavHau Enzime (Michael Hoang) hsjobeki kenji (Kenji Berthold) lassulus Mic92 nixbot pinpox Qubasa (Luis Hebendanz) renovate robinlol

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: clan/clan-llm#2