Tuesday, November 17, 2026

SkillEvaluator

★2,983Must watch
Demo

NVIDIA’s open stack that grades agent skills — static quality, overlap, harness fit, and live with/without runs — so you know a skill actually helps before you ship it.

SkillEvaluator

Why it matters

SkillEvaluator is Apache-2.0 from NVIDIA, part of the Verified Skills pipeline. It targets the Agent Skills folder format. Three independent tiers: Tier 1 validation (well-formed, safe, harness-compatible — can pull SkillSpector and Gitleaks), Tier 2 semantic overlap against a skill catalog, Tier 3 live agent evaluation that compares task results with and without the skill in a sandbox (Harbor by default). CLI and PyPI package, CI init templates, and an agent skill so you can ask “is my skill any good?” from the coding agent itself. Research paper on arXiv.

Everyone is dumping SKILL.md packs. Almost nobody measures whether they change agent outcomes. Founders shipping skills into Claude Code, Codex, Hermes, or CI need gates beyond vibes and a security scan. SkillEvaluator is that gate: quality score, dedupe, harness load checks, and sandboxed live eval.

How it works

pip install skillevaluator or uv tool install. Run skillevaluator validate ./my-skill --tiers 1 with no key for static quality and harness-check across Codex, Claude Code, OpenCode, Hermes, OpenClaw, Copilot CLI. Add LLM provider keys for rubric scores. For full stack: doctor, then validate with tiers 1–3 and Docker sandbox. skillevaluator ci init drops a PR that checks every skill change. Or npx skills add NVIDIA/SkillEvaluator and ask the agent to review a skill path.

SkillSpector scans skills for injection and exfil. SkillOpt and Skill Recorder create or optimize skills. SkillEvaluator answers a different question: is this skill loadable, non-overlapping, and does it move the agent’s real task score — with CI and live A/B-style runs, not only a security badge.

Capabilities

Demo
  • Public demo available

Similar tools

agentsskillsevaluationnvidiaclaude-codecodexmcpci
Source ↗

Via github

X