Added deal contract framework and testing documentation

checks/impure/flake-module.nix

@@ -1,6 +1,6 @@
 { ... }: {
   perSystem = { pkgs, lib, ... }: {
-    packages = {
+    packages = rec {
       # a script that executes all other checks
       impure-checks = pkgs.writeShellScriptBin "impure-checks" ''
         #!${pkgs.bash}/bin/bash
@@ -15,6 +15,52 @@
         cd "$ROOT/pkgs/clan-cli"
         nix develop "$ROOT#clan-cli" -c bash -c 'TMPDIR=/tmp python -m pytest -m impure -s ./tests'
       '';
+
+      runMockApi = pkgs.writeShellScriptBin "run-mock-api" ''
+        #!${pkgs.bash}/bin/bash
+        set -euo pipefail
+
+        export PATH="${lib.makeBinPath [
+          pkgs.gitMinimal
+          pkgs.nix
+          pkgs.rsync # needed to have rsync installed on the dummy ssh server
+          pkgs.coreutils
+          pkgs.procps
+        ]}"
+        ROOT=$(git rev-parse --show-toplevel)
+        cd "$ROOT/pkgs/clan-cli"
+        nix develop "$ROOT#clan-cli" -c bash -c 'TMPDIR=/tmp python -m clan_cli webui --no-open --port 2979'
+      '';
+
+
+      runSchemaTests = pkgs.writeShellScriptBin "runSchemaTests" ''
+        #!${pkgs.bash}/bin/bash
+        set -euo pipefail
+
+        ${runMockApi}/bin/run-mock-api &
+        MOCK_API_PID=$!
+        echo "Started mock api with pid $MOCK_API_PID"
+        function cleanup {
+            echo "Stopping server..."
+            pkill -9 -f "python -m clan_cli webui"
+        }
+        trap cleanup EXIT
+
+        export PATH="${lib.makeBinPath [
+          pkgs.gitMinimal
+          pkgs.nix
+          pkgs.rsync # needed to have rsync installed on the dummy ssh server
+          pkgs.procps
+          pkgs.coreutils
+        ]}"
+
+        sleep 3
+
+        ROOT=$(git rev-parse --show-toplevel)
+        cd "$ROOT/pkgs/clan-cli"
+        nix develop "$ROOT#clan-cli" -c bash -c 'TMPDIR=/tmp st auth login RHtr8nLtz77tqRP8yUGyf-Flv_9SLI'
+        nix develop "$ROOT#clan-cli" -c bash -c 'TMPDIR=/tmp st run http://localhost:2979/openapi.json --experimental=openapi-3.1 --report --workers 8 --max-response-time=50 --request-timeout=1000 -M GET'
+      '';
     };
   };
 }

docs/contributing.md

@@ -172,6 +172,14 @@ nix build .#checks.x86_64-linux.clan-pytest --rebuild
 
 This command will run all pure test functions.
 
+### Running schemathesis fuzzer on GET requests
+
+```bash
+nix run .#runSchemaTests
+```
+
+If you want to test more request types edit the file `checks/impure/flake-module.nix`
+
 ### Inspecting the Nix Sandbox
 
 If you need to inspect the Nix sandbox while running tests, follow these steps:

docs/testing.md

@@ -14,20 +14,27 @@ Instead of having to write numerous test cases for various input arguments, prop
 
 Property-based testing tools retain test cases and their results, allowing you to reproduce and replay tests in case of failure. This feature is invaluable for debugging and ensuring the stability of your application over time.
 
-### Frameworks for Property-Based Testing
+## Frameworks for Property-Based Testing
 
 To implement property-based testing in FastAPI, you can use the following framework:
 
-- [Using Hypothesis with FastAPI](https://testdriven.io/blog/fastapi-hypothesis/)
-- [Schemathesis](https://schemathesis.readthedocs.io/en/stable/#id2)
 - [Hypothesis: Property-Based Testing](https://hypothesis.readthedocs.io/en/latest/quickstart.html)
+- [Schemathesis](https://schemathesis.readthedocs.io/en/stable/#id2)
 
+## Example
 
-### Nix code
-https://github.com/kadena-io/signing-api/blob/master/schema-tests.nix
-https://github.com/garbas/cicero/blob/67acaaa6f568d6f5032419bdcf0f4c97fb46e5ec/pkgs/schemathesis.nix#L5
+Running schemathesis fuzzer on GET requests
 
-### Why Schemas Are Not Contracts
+```bash
+nix run .#runSchemaTests
+```
+
+If you want to test more request types edit the file [flake-module.nix](../checks/impure/flake-module.nix)
+
+After a run it will upload the results to `schemathesis.io` and give you a link to the report.
+The credentials to the account are `Username: schemathesis@qube.email` and `Password:6tv4eP96WXsarF`
+
+## Why Schemas Are Not Contracts
 
 A schema is a description of the data structure of your API, whereas a contract defines not only the structure but also the expected behavior and constraints. The following resource explains why schemas are not contracts in more detail:
 
@@ -40,18 +47,85 @@ In a nutshell, schemas may define the data structure but often fail to capture c
 Contract-driven testing combines the benefits of type annotations and property-based testing, providing a robust approach to ensuring the correctness of your APIs.
 
 - Contracts become an integral part of the function signature and can be checked statically, ensuring that the API adheres to the defined contract.
-
 - Contracts, like property-based tests, allow you to specify conditions and constraints, with the testing framework automatically generating test cases and verifying call results.
 
-
-
 ### Frameworks for Contract-Driven Testing
 
 To implement contract-driven testing in FastAPI, consider the following framework and extension:
 
-- [iContract: Contract-Driven Development](https://icontract.readthedocs.io/en/latest/introduction.html)
-- [FastAPI-iContract: Extension for FastAPI](https://github.com/mristin/fastapi-icontract)
+- [Deal: Contract Driven Development](https://deal.readthedocs.io/)
+  By adopting contract-driven testing, you can ensure that your FastAPI application not only has a well-defined structure but also behaves correctly, making it more robust and reliable.
+- [Whitepaper: Python by contract](https://users.ece.utexas.edu/~gligoric/papers/ZhangETAL22PythonByContractDataset.pdf) This paper goes more into detail how it works
 
-By adopting contract-driven testing, you can ensure that your FastAPI application not only has a well-defined structure but also behaves correctly, making it more robust and reliable.
+## Examples
 
-**However: icontract uses a 3 year old version of deal and a 3 year old version of typeguard. And icontract-fastapi is not maintained anymore**
+You can annotate functions with `@deal.raises(ClanError)` to say that they can _only_ raise a ClanError Exception.
+
+```python
+import deal
+
+@deal.raises(ClanError)
+def get_task(uuid: UUID) -> BaseTask:
+    global POOL
+    return POOL[uuid]
+```
+
+To say that it can raise multiple exceptions just add after one another separated with a `,`
+
+```python
+import deal
+
+@deal.raises(ClanError, IndexError, ZeroDivisionError)
+def get_task(uuid: UUID) -> BaseTask:
+    global POOL
+    return POOL[uuid]
+```
+
+### Adding deal annotated functions to pytest
+
+```python
+from clan_cli.task_manager import get_task
+import deal
+
+@deal.cases(get_task) # <--- Add function get_task to testing corpus
+def test_get_task(case: deal.TestCase) -> None:
+    case() # <--- Call testing framework with function
+```
+
+### Combining hypothesis with deal
+
+You can combine hypothesis annotations with deal annotations to add example inputs to the function so that the verifier can reach deeper parts of the function.
+
+```python
+import hypothesis
+import deal
+
+@hypothesis.example(["8c3041e0-4512-4b30-aa8e-7be4a75b8b45", "5c2061e0-4512-4b30-aa8e-7be4a75b8b45"])
+@deal.raises(ClanError)
+def get_task(uuid: UUID) -> BaseTask:
+    global POOL
+    if "206" in str(uuid):
+        raise ValueError("206 should not be in here")
+    return POOL[uuid]
+```
+
+You can also annotate what kind of information a value can have by adding a hypothesis "strategy". Which kind of data generation hypothesis supports you can see here:
+[Hypothesis Value Properties](https://hypothesis.readthedocs.io/en/latest/data.html)
+The example above doesn't really need a hypothesis annotation because deal can infer from the UUID type a strategy.
+But as an example how it might look like:
+
+```python
+from hypothesis.strategies import uuids
+
+@hypothesis.given(uuids)
+@deal.raises(ClanError)
+def get_task(uuid: UUID) -> BaseTask:
+    global POOL
+    if "206" in str(uuid):
+        raise ValueError("206 should not be in here")
+    return POOL[uuid]
+```
+
+For a complex example for an [HTTP API look here](https://hypothesis.readthedocs.io/en/latest/examples.html#fuzzing-an-http-api). Just note we are using schemathesis for this already.
+
+You can also add `pre` and `post` conditions. A `pre` condition must be true before the function is executed. A `post` condition must be true after the function was executed. For more information read the [Writing Contracts Section](https://deal.readthedocs.io/basic/values.html).

flake.lock

@@ -62,11 +62,11 @@
     },
     "luispkgs": {
       "locked": {
-        "lastModified": 1699386673,
-        "narHash": "sha256-T9ZrVSf0ymSBHFpRw7K+DSNzY0sCkwkDe0VIzE033Ac=",
+        "lastModified": 1699388703,
+        "narHash": "sha256-4/KKyXlznkCi/q5LQmS9CFd4xEAJzmB5r0GldPSTSak=",
         "owner": "Luis-Hebendanz",
         "repo": "nixpkgs",
-        "rev": "e918074d416f0918ac6c3895c6a5db2ae5ce3c1e",
+        "rev": "6dcb6de2f11562465b3179d541e47a799356426d",
         "type": "github"
       },
       "original": {

pkgs/clan-cli/clan_cli/task_manager.py

@@ -12,6 +12,8 @@ from pathlib import Path
 from typing import Any, Iterator, Optional, Type, TypeVar
 from uuid import UUID, uuid4
 
+import deal
+
 from .custom_logger import ThreadFormatter, get_caller
 from .errors import ClanError
 
@@ -161,6 +163,8 @@ class TaskPool:
 
     def __getitem__(self, uuid: UUID) -> BaseTask:
         with self.lock:
+            if uuid not in self.pool:
+                raise ClanError(f"Task with uuid {uuid} does not exist")
             return self.pool[uuid]
 
     def __setitem__(self, uuid: UUID, task: BaseTask) -> None:
@@ -175,6 +179,7 @@ class TaskPool:
 POOL: TaskPool = TaskPool()
 
 
+@deal.raises(ClanError)
 def get_task(uuid: UUID) -> BaseTask:
     global POOL
     return POOL[uuid]

pkgs/clan-cli/tests/test_with_deal.py

@@ -0,0 +1,9 @@
+import deal
+
+from clan_cli.task_manager import get_task
+
+
+# type annotations below are optional
+@deal.cases(get_task)
+def test_get_task(case: deal.TestCase) -> None:
+    case()