Skip to content

Commit 1dd8140

Browse files
calvinqicalvinqi
authored
[labs] add linting and typechecking (#168)
Adds linting via ruff and typechecking via pyright, generally helpful to have these in the repo for guardrails and correctness! Full CI sequence still only takes around 5min total. ## Summary Builds on the release tooling merged in [#164](#164) by adding linting, type checks, and isolated-wheel smoke coverage. Adds CI jobs for repo-wide Ruff, `lab_core` Pyright, and wheel import/document extraction smoke tests on Python 3.12 and 3.13. Existing Ruff violations have rule-specific file-level directives in the affected files. Existing type errors have file/rule Pyright directives, plus a single-line ignore for the optional Mistral import; runtime behavior is unchanged. Both tools are pinned in the development dependency group. The wheel tests install only the built package and its declared runtime dependencies, then import every host module and extract known content from small synthetic XLSX/PPTX files. They run outside the checkout with Python isolation enabled and use standard-library `unittest`, so development tools and fixture generators cannot supply missing dependencies. ## Validation - Ruff passes. Pyright basic mode passes over 30 `lab_core` source files with the recorded ignores (zero errors or warnings). - Full offline suite: 12,790 passed, 62 skipped, 28 import subtests passed. - Installed-wheel smoke tests and dependency consistency checks pass on Python 3.12 and 3.13. - A new-file probe fails on missing imports, undefined names, and invalid argument types. - The release metadata retains `openpyxl` and `python-pptx`. The isolated-wheel tests pass with those dependencies and reproduced both extraction failures when they were absent. - The wheel-content check passes: 68 members, 146,463 bytes. ## Limits The baseline ignores also suppress new diagnostics of the same type in the same files until removed. New files and other diagnostic types remain checked. Pyright excludes sandbox skill scripts, whose dependencies live in the container; Ruff includes them. The smoke tests cover imports and XLSX/PPTX extraction, not live provider calls or container execution.
1 parent 87a7ed3 commit 1dd8140

30 files changed

Lines changed: 228 additions & 6 deletions

‎.github/workflows/python-checks.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
name: Python checks
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches: [main]
7+
8+
permissions:
9+
contents: read
10+
11+
jobs:
12+
static-checks:
13+
runs-on: ubuntu-latest
14+
timeout-minutes: 10
15+
steps:
16+
- uses: actions/checkout@v4
17+
- uses: astral-sh/setup-uv@v4
18+
with:
19+
python-version: "3.12"
20+
- run: uv sync --locked
21+
- name: Lint Python
22+
run: uv run --no-sync ruff check .
23+
- name: Check host package types
24+
run: uv run --no-sync pyright
25+
26+
installed-wheel:
27+
runs-on: ubuntu-latest
28+
timeout-minutes: 10
29+
strategy:
30+
fail-fast: false
31+
matrix:
32+
python-version: ["3.12", "3.13"]
33+
steps:
34+
- uses: actions/checkout@v4
35+
- uses: astral-sh/setup-uv@v4
36+
with:
37+
python-version: ${{ matrix.python-version }}
38+
- name: Build wheel
39+
run: uv build --wheel
40+
- name: Install wheel with runtime dependencies only
41+
run: |
42+
uv venv --python "${{ matrix.python-version }}" "$RUNNER_TEMP/wheel-venv"
43+
uv pip install --python "$RUNNER_TEMP/wheel-venv/bin/python" dist/*.whl
44+
uv pip check --python "$RUNNER_TEMP/wheel-venv/bin/python"
45+
- name: Exercise imports and document converters outside the checkout
46+
working-directory: ${{ runner.temp }}
47+
run: env -u LAB_ROOT -u PYTHONPATH "$RUNNER_TEMP/wheel-venv/bin/python" -I "$GITHUB_WORKSPACE/tests/test_package_smoke.py"

‎.github/workflows/validate-task-schema.yml‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,10 @@ jobs:
1919
run: uv sync --locked
2020

2121
- name: Run offline test suite
22-
run: uv run pytest tests/ -v
22+
run: uv run --no-sync pytest tests/ -v
2323

2424
- name: Build wheel
2525
run: uv build --wheel
2626

2727
- name: Check wheel contents
2828
run: uv run python scripts/check_wheel.py dist/lab_core-*.whl
29-

‎CONTRIBUTING.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,43 @@ uv run python -m pytest --live --model claude-sonnet-4-6
176176

177177
Live tests require provider API keys and are skipped unless `--live` is passed.
178178

179+
## Python Checks
180+
181+
Install the locked development tools and run the same static checks as CI:
182+
183+
```bash
184+
uv sync --locked
185+
uv run --no-sync ruff check .
186+
uv run --no-sync pyright
187+
uv run --no-sync pytest tests/test_package_smoke.py -v
188+
```
189+
190+
Ruff checks Python errors and unused names across the repository, including the
191+
sandbox skill scripts. Pyright checks the host package in basic mode using the
192+
project's `.venv`. Sandbox skill scripts use a separate dependency set and are
193+
excluded from Pyright.
194+
195+
Ruff's existing violations have file-level `# ruff: noqa` rule directives.
196+
Pyright's existing errors have file-specific `# pyright:` rule directives;
197+
the optional Mistral SDK has an ignore on its import. Remove each ignore when
198+
its violations are resolved. A suppressed rule can also hide new violations
199+
of that rule in the same file; new files and other rules remain checked.
200+
201+
CI also builds a wheel and installs it with only its runtime dependencies into a
202+
fresh environment on Python 3.12 and 3.13. It imports the host modules and extracts
203+
known text from the synthetic XLSX and PPTX fixtures in `tests/fixtures/documents/`.
204+
These tests exercise pandas' Excel engine and MarkItDown's presentation converter,
205+
whose dependencies load at runtime and cannot be verified by static checks alone.
206+
The smoke tests use the standard library's `unittest`, so test tooling cannot
207+
supply missing converter dependencies. Run them against an installed wheel with:
208+
209+
```bash
210+
uv build --wheel
211+
uv venv /tmp/lab-wheel-smoke
212+
uv pip install --python /tmp/lab-wheel-smoke/bin/python dist/*.whl
213+
/tmp/lab-wheel-smoke/bin/python -I tests/test_package_smoke.py
214+
```
215+
179216
## Documentation Changes
180217

181218
When docs mention task counts, model IDs, tool names, or command names, verify them against the code before committing:

‎lab_core/evaluation/charts.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
embedded in HTML. Uses seaborn for styling.
55
"""
66

7+
# pyright: reportAttributeAccessIssue=false, reportPrivateImportUsage=false
8+
79
import math
810
from pathlib import Path
911

‎lab_core/evaluation/compare.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@
1313
uv run python -m lab_core.evaluation.compare --all --save-images
1414
"""
1515

16+
# pyright: reportReturnType=false
17+
# ruff: noqa: F841
18+
1619
import argparse
1720
import json
1821
from pathlib import Path

‎lab_core/evaluation/judge.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
and parses the structured response. Used by all scoring functions.
55
"""
66

7+
# pyright: reportArgumentType=false, reportAttributeAccessIssue=false
8+
79
import json
810
import re
911
from pathlib import Path

‎lab_core/evaluation/scoring.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@
44
relevant deliverable files included in context.
55
"""
66

7+
# pyright: reportAttributeAccessIssue=false
8+
79
from __future__ import annotations
810

911
import json

‎lab_core/harness/adapters/baseten.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@
77
the Baseten Model APIs gateway, or point it at a deployment's ``/sync/v1`` URL.
88
"""
99

10+
# pyright: reportArgumentType=false, reportCallIssue=false
11+
1012
import os
1113
import random
1214
import time

‎lab_core/harness/adapters/fireworks.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
"""Fireworks adapter — OpenAI-compatible Chat Completions API."""
22

3+
# pyright: reportArgumentType=false, reportCallIssue=false, reportGeneralTypeIssues=false
4+
35
import os
46
import time
57

‎lab_core/harness/adapters/google.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@
88
The SDK chat handles thought signatures automatically.
99
"""
1010

11+
# pyright: reportArgumentType=false, reportAttributeAccessIssue=false, reportOptionalIterable=false
12+
1113
import json
1214
import logging
1315

0 commit comments

Comments
 (0)