# Testing and publishing a plugin

Course: Claude Code: Pro Level — https://zero2vibecode.com/learn/claude-code-pro
Canonical URL: https://zero2vibecode.com/learn/claude-code-pro/6-2-test-and-publish-a-plugin
Verified against: Claude Code v2.1.292 (2026-10-15)
Last updated: 2026-10-15

claude plugin validate checks a plugin's files, claude plugin eval checks its behaviour by comparing runs with and without it. Publishing is a marketplace.json in a git repository, and a new version reaches users only when you bump version.

In the plugins level you installed them. Now the other side: releasing your own in a way people can trust. There are two kinds of checks, and publishing turns out simpler than it seems.

## Two checks

`claude plugin validate` reads **files**: manifests and frontmatter. With `--strict`, even a warning fails the check.

```bash
claude plugin validate --strict ./plugins/commit-helper
claude plugin validate --strict .
```

Check both the plugin and the marketplace: validating the marketplace looks at its file and each plugin's `plugin.json`, but does not look inside skill files.

`claude plugin eval` measures **behaviour**. Each case is a prompt plus graders — checks that score the run. A case runs in a fresh, isolated session where only your plugin is loaded: by default three times with it (the WITH arm) and as many times without it (W/OUT). The difference between their scores is **Δ**, the plugin's own contribution.

## The first case

```bash
cd plugins/commit-helper
claude plugin eval init --bare names-the-rename
```

`--bare` runs nothing: it creates `prompt.md` and a `graders/criteria.md` draft — a rubric for a judge model. It is usually replaced with free checks.

`prompt.md` is the prompt, with frontmatter:

```markdown
---
max_turns: 10
allowed_tools: [Skill]
---

Write me a commit message for this change: I renamed getUser to fetchUser and updated the three call sites.
```

A run gets only the tools listed, so `allowed_tools: [Skill]` is what lets Claude run the skill. Two checks in `graders/`:

```markdown
---
type: tool_used
tool: Skill
input_match: 'commit-message'
---
```

```markdown
---
type: regex
pattern: 'fetchUser'
---
```

The first means "the skill ran", the second "the answer contains a fact from the prompt". Claude words things differently each time, so the regex looks for what the prompt supplied, not a specific phrase.

## Which graders are free

| Grader | Cost |
|---|---|
| `regex`, `tool_used`, `tool_order`, `file_exists` | free |
| `llm`, `baseline` | a judge model call in every run |

A good rule: one check on the **result** and one on the **steps**. Keep the judge for what free checks cannot see.

## Running it

A cheap trial run — once, with the plugin only, the report stays with you:

```bash
claude plugin eval . --runs 1 --ablation none --no-publish
```

A full run with a threshold:

```bash
claude plugin eval . --threshold 0.8 --no-publish
```

Without `--threshold`, any case below a perfect score counts as failed. Results land in `evals/results/` — keep those out of git, and commit the cases themselves.

One trap. The check "the skill did NOT fire on an unrelated request" is `tool_used` on `Skill` with `min: 0` and `max: 0`. But in a run with two arms, `tool_used` graders on `Skill` do not count by default: without the plugin there is no skill. For the check to count, add `arm: both`.

## Publishing and releasing

To publish is to put `.claude-plugin/marketplace.json` in a git repository and push. No application forms: whoever can clone the repository can install from it.

```bash
claude plugin marketplace add owner/claude-plugins
claude plugin install commit-helper@owner-plugins
```

The marketplace's name comes from `marketplace.json`, not from the repository's name. Some marketplace names are reserved — for example, anything starting with `claudeai-`.

A new release: bump `version` in `plugin.json` (and do not set a version of your own in the marketplace entry), push, and users update with:

```bash
claude plugin update commit-helper@owner-plugins
```

It ends with the line `Restart to apply changes.` If it says the version is already the latest, the new `version` has not been pushed yet.

Treat the plugin's name as permanent: it is installed and configured as `name@marketplace`, and a renamed plugin is a different plugin for everyone. `displayName` exists for a new label.

**Careful.** 
claude plugin eval runs the plugin's skills, hooks and agents on your machine, in your name, and a green run says nothing about safety. And a hook or MCP server you add in the next version will run for everyone who updates, with their permissions. Before making a marketplace repository public, read every file in it. And remember: --max-cost-usd caps an estimate based on list prices, not your plan's limit.

**Practice in the terminal.** Write the line **/evals/results/ into .gitignore, create the plugins/commit-helper/evals/names-the-rename/graders folder, and overwrite plugin.json with version 1.0.1. In a real project, validate --strict, eval with a threshold and git push come next.