Skip to content

Add a reusable benchmark card template - #61

Merged
deanwampler merged 1 commit into
The-AI-Alliance:mainfrom
goktugozkanmd:docs/benchmark-card-template
Jul 12, 2026
Merged

deanwampler merged 1 commit into
The-AI-Alliance:mainfrom
goktugozkanmd:docs/benchmark-card-template

Conversation

@goktugozkanmd

Copy link
Copy Markdown
Contributor

Description of Changes

Adds a benchmark-level documentation template covering identity and lifecycle, intended decisions, task composition, taxonomy coverage, protocol, measures and uncertainty, validation evidence, safety boundaries, and governance. It explicitly separates benchmark-level evidence from Unitxt-style executable task cards and includes an abbreviated MedFailBench example showing how to state a high-stakes clinical boundary.

The evaluations landing page now links to the template and exposes it as a child page.

Related Issues

Closes #24

Testing Performed

  • Verified site builds successfully (make view-local requires Ruby >=3 because the current Nokogiri dependency no longer resolves on the macOS Ruby 2.6 runner used here)
  • Checked external-link target declarations with ./check-external-links.sh (0 findings)
  • Checked the new page with markdownlint-cli2 (0 errors; repository-compatible MD013/MD025/MD060 exemptions)
  • Parsed all Markdown front matter with Ruby YAML and ran git diff --check

Code Changes

  • docs/evaluations/benchmark-card-template.markdown
  • docs/evaluations/evaluations.markdown

Preview

Review the field coverage, especially the distinction between executable task cards and benchmark-level composition, validity, and decision-boundary evidence.

Checklist

  • I have read the contributing guide and used a DCO sign-off
  • I have updated related documentation
  • I have included a worked boundary-language example

Additional Context

This is intentionally a documentation POC. It links to Unitxt task cards and the EvalEval catalog rather than duplicating their executable or standardized records.

Signed-off-by: Göktuğ Özkan, MD <drozkan2@gmail.com>

@deanwampler deanwampler left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great. Thanks for the submission!

@deanwampler
deanwampler merged commit ec0cf73 into The-AI-Alliance:main Jul 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

POC candidate: benchmark cards

2 participants