Skip to content
SkillModel-led423 tokens in use

validate-bug

The validation stage of patching-bugs: a fresh agent reruns the unchanged PoC and decides whether the claim and its invariant hold.

8,319 stars on the repository CC-BY-SA-4.0trailofbits/skills · plugins/patching-bugs/skills/validate-bug
/plugin marketplace add trailofbits/skills
View source

Install command as written in the repository's README

Skill label

measured from its files

What installing validate-bug costs

22tokens idle
423tokens in use

Idle is what sits in context while the skill is installed. In use is its instructions, loaded when it runs. Counts are approximate, made offline with cl100k_base.

Model dependence Model-led

Script-ledMixedModel-led

Mostly instructions, so how well it works depends on the model that follows them.

This describes the skill's structure, how much of it is scripts. It does not say how well it works.

Touches

shelldetected

Declared tags come from the skill's frontmatter; detected ones from its scripts, its code blocks and the commands it names in inline code. This is detected, not audited.

How it scores

Quality

How well the skill is written, counted from its own files: its frontmatter, its steps, examples, stated output and constraints, its size and whether every file it ships is one it names.How we score it →
Good
  • Frontmatter12/15
  • Structure23/25
  • Examples6/15
  • Output15/15
  • Size11/15
  • Resources10/15

Security

Starts at 100 and loses points for each thing in the skill's files that a reader should look at first.How we score it →
Excellent

No findings.

Repository health

Pre-computed score based on development activity, maintenance, community, maturity, and trend momentum.How we score it →
Good
  • Development activity100
  • Maintenance52
  • Community64
  • Maturity32
  • Momentum40

Validate-bug runs after reproduction has supplied a finding and a PoC package. The validating agent reads the finding, the proof type, the commands and the raw receipts, and does not read the reproducer’s transcript.

It reruns the same compiler command or rebuilds and reruns the unchanged runtime PoC, saves its own receipts, and judges whether the diagnostic or assertion shows the exact claim and violates the stated invariant. The decision is confirmed, not confirmed or inconclusive, stored with a short reason.

What You Get

  • A validate.json with the outcome, a short reason and the validator’s agent identity
  • Rerun receipts with stdout, stderr, exit status and timing under validate/receipts
  • A manifest of the validate directory and a record check

Common Use Cases

  • Independently confirming a reproduction before spending effort on a patch
  • Rejecting a PoC that does not demonstrate the stated claim
  • Flagging unreliable observations as inconclusive

What Makes It Unique

The validator is blind to the reproducer’s narrative and must not repair the PoC to make the claim work. If an observation is unreliable it returns inconclusive and never infers presence or absence of the bug.

Under The Hood

The agent reads SKILL.md, a record contract and the reproduction package. It rebuilds the target and reruns the saved PoC or compiler command in the local checkout, writes receipts and a JSON record into a validate directory, and uses a bundled Python helper through uv to write a manifest and check records. The commands it runs are those of the project under test, and it does not change the PoC or the finding.

Join founders building with open source

Opinionated takes, migration guides, cost-saving tips, and insights from the open source ecosystem.

Subscribe on SubstackJoin 750+ subscribers