Intent Solutions Demos

An MCP plugin audit you can check line by line.

kobiton/automate is Kobiton's public plugin for Claude Code. It connects AI coding agents to Kobiton's real-device cloud through an MCP server, and also runs in Copilot CLI, Gemini CLI, Cursor CLI, and Codex CLI. Kobiton hired Intent Solutions to review it and write about it. We reviewed it in three passes: the skill and repository, the runtime behavior, and the MCP spec and server responses. Every finding went up as a public GitHub issue. Every fix we could make from outside went up as a pull request.

So far: 25 findings in public, 10 of our pull requests merged by Kobiton, 5 more fixes written by Kobiton's own engineers that cite the findings, and the repository went from 0 to 12 GitHub stars. See the counts and sources

Talk to the team See the issues on GitHub
kobiton/automate · issuesPublic
  1. #19Add 6 missing frontmatter fields to the run-automation-suite skillR1 · closed as completed
  2. #34confirmAppUpload async race and missing parsing diagnosticR2 · closed as completed
  3. #41OAuth: declare resource indicators and send WWW-Authenticate on a bad-token 401R3 · closed as completed
  4. #58getDeviceStatus returns 3 fields; its description documents moreR3 · closed as completed
  5. #59getApp.is_expired contradicts listApps.is_expired for the same appR3 · closed as completed
5 of 25 findings, titles shortened. Each opens the real issue.

The engagement at a glance.

Everything below is public and can be checked on GitHub or kobiton.com.

Customer
Kobiton, a real-device testing cloud for mobile apps. A paid engagement, April to June 2026.
The problem
Kobiton had shipped kobiton/automate, its plugin for Claude Code, and needed an outside review of where AI agents using it would go wrong: a skill that agents did not load, tool descriptions that did not match what the server returned, and gaps against the MCP specification.
How we used Claude
We did the review with Claude Code, Anthropic’s coding agent. We installed the plugin in Claude Code, ran its MCP tools against Kobiton’s real-device cloud, and used Claude Code to prepare every finding and pull request, which we checked before filing.
In production
kobiton/automate is Kobiton’s public plugin, installable today. The fixes listed on this page are merged into its main branch.
Measured results
  • 0 → 12 GitHub stars on kobiton/automate: 0 when our review began on 28 April 2026, 12 on 19 September 2026, counted on GitHub.
  • 10 of 15 of our pull requests merged by Kobiton’s maintainers.
  • 5 further fixes written by Kobiton’s own engineers, each citing the finding it closes.
  • 2 → 8 of 8 skill frontmatter fields filled, and 12 of 12 MCP tools given retry-safety annotations.
Published
By Intent Solutions on this page, and by Kobiton on its own blog in two articles by Jeremy Longshore (listed below).

Every finding is a public GitHub issue.

No private PDF to take on faith. The findings live where the maintainers already work, and anyone evaluating the plugin can read what was found and what changed.

  • One START HERE per passEach pass opens with an issue giving the reading order and what the plugin already does well: R1 #20, R2 #32, R3 #43 and #61.
  • One issue per findingA finding names the tool, the observed behavior, and the fix, so a maintainer can act on it without a meeting.
  • A pull request when we canDescriptions, annotations, and docs we can fix from outside. The maintainers decide what merges.
  • Advisories when the ground movesSpec and platform changes filed as issues too: the MCP 2026-07-28 release candidate, the Gemini CLI sunset, Claude enterprise changes.

What the review produced.

Counted from GitHub on 17 September 2026; stars on 19 September 2026. Review passes ran from 28 April to 18 May 2026. Each figure links to its source.

25
findings filed as public issues across 3 passes: 10 on the skill and repo, 5 on runtime behavior, 10 on the MCP spec and response contract. Open the findings
10 of 15
of our pull requests merged by a Kobiton maintainer. Open our pull requests
5
merged fixes written by Kobiton's own engineers that cite the findings they close. See which
8 of 8
skill frontmatter fields filled after finding #19, up from 2, in a pull request by Kobiton's engineers. Pull request #30
12 of 12
MCP tools given idempotent and open-world annotations. Pull request #74
12 from 0
GitHub stars on kobiton/automate: 0 when our review began on 28 April 2026, 12 on 19 September 2026. See the repository

Three passes: the skill, the runtime, the spec.

Each pass reads a different layer of the plugin, and each one builds on what the last one found.

  1. R1 · 10 findings

    The skill and the repository

    Does an agent load the skill, understand when to use it, and find what it needs? Is the repo set up for outside contributors and safe updates?

    R1 START HERE, issue #20
    • #18

      The skill description had no "Use when" or "Trigger with" phrases, so agents had little to go on when deciding to load it.

    • #21

      No Dependabot, issue chooser, or CodeQL. Kobiton added all three in pull request #26.

  2. R2 · 5 findings

    Runtime behavior on real calls

    What do the tools actually do when an agent calls them against real devices: the races, the error messages, the response shapes an agent has to parse.

    R2 START HERE, issue #32
    • #33

      When a device reservation failed, one conflict message covered four different causes, so an agent could not tell whether to wait, retry, or pick another device.

    • #34

      App upload confirmation raced the server's async parsing. Kobiton added a getAppParsingStatus tool in pull request #93.

  3. R3 · 10 findings

    The MCP spec and the response contract

    Does the server declare what the spec expects, and does every response match what its own tool description promises?

    R3 START HERE, issue #43
    • #39

      The server did not declare resources, prompts, or instructions, and advertised protocol version 2025-03-26 when 2025-06-18 was current.

    • #55

      listSessions silently ignored its limit parameter and returned the server's default page size.

What shipped.

Findings only matter if something changes. These are merged pull requests on the public repository.

Ours, merged by Kobiton

  • #74Idempotent and open-world annotations on all 12 tools
  • #767 tool descriptions aligned with what the server actually returns
  • #73listSessions default page lowered from 20 to 10
  • #64Authentication section and allowed tools for the skill
  • #68Install paths for Cursor, ChatGPT, and Continue
  • #69Claude surface compatibility matrix in the README
  • #66CLAUDE.md architecture guide for contributors
  • #75AGENTS.md, version alignment, and README consistency
  • #91Contributing steps aligned with the validator
  • #63Support email and URL in plugin and marketplace metadata

Written by Kobiton's engineers, citing the findings

  • #30Skill frontmatter from 2 of 8 fields to 8 of 8, plus trigger phrases (findings #18, #19)
  • #31Skill body restructured into 7 named sections with examples (#14 to #17)
  • #38Reference tables moved out of the skill body (#13)
  • #26Dependabot, issue chooser, and CodeQL (#21)
  • #93getAppParsingStatus tool for async app parsing (findings F25 and F50)
tools/apps.yaml · confirmAppUploadMerged
  - name: confirmAppUpload
    annotations:
      readOnlyHint: false
      destructiveHint: false
+     idempotentHint: true+     openWorldHint: false
  • idempotentHint: true tells a client a second confirm after success is a lookup, not a duplicate upload, so a retry is safe.
  • openWorldHint: false tells it the tool talks only to Kobiton's bounded API.
Excerpt from pull request #74, one of 12 tools changed. Added lines shaded.

What we published on kobiton.com.

Two articles by Jeremy Longshore, published by Kobiton on its own blog.

AGENTS.md Mobile Testing Plugin Brief: A Case Study from kobiton/automate

How one brief file keeps a plugin consistent across agent CLIs, and what a 5-device benchmark showed about platform variance and session lifecycle.

Read the AGENTS.md case study on kobiton.com

Shipping a Claude Code plugin or MCP server?

A server can pass a security checklist and still mislead the agent calling it. This is what we check.

  • The skill loadsFrontmatter, trigger phrases, structure, and examples, so agents pick the skill up when they should and not when they shouldn't.
  • Tools say what they doRead-only, destructive, idempotent, and open-world annotations that match real behavior, so clients prompt and retry correctly.
  • Responses keep their promisesEvery field a tool description mentions is actually returned, and the same value means the same thing across tools.
  • Failures are legibleError messages an agent can act on, and async work it can poll instead of racing.
  • The spec is currentDeclared primitives, protocol version, and auth headers checked against the MCP specification, with advisories as it changes.
  • Every host worksInstall paths and behavior across the agent CLIs you claim: Claude Code, Copilot CLI, Gemini CLI, Cursor CLI, Codex CLI.

MCP audit questions.

What does an MCP server audit check?

Ours covers three layers. The skill and repository: frontmatter, structure, examples, and governance. Runtime behavior: what the tools actually do on real calls, including races, error messages, and response shapes. The MCP spec and the server's response contract: declared primitives, protocol version, auth headers, tool annotations, and whether each response matches its own tool description.

Is an MCP audit the same as a security audit?

No. Security is part of it: on kobiton/automate we covered OAuth declarations, the 401 challenge header, and repository governance such as Dependabot and CodeQL. It is not a penetration test. Most findings were about behavior an agent relies on, like a tool description promising fields the server never returns.

What are MCP tool annotations and why do they matter?

Annotations are hints on each tool: readOnlyHint, destructiveHint, idempotentHint, and openWorldHint. Clients use them to decide when to ask the user for confirmation and whether a call is safe to retry. kobiton/automate declared only the first two, so we filed pull request #74 adding idempotentHint and openWorldHint to all 12 tools, and Kobiton merged it.

How do you review a Claude Code plugin that also runs in other agent CLIs?

We check the plugin against each host it claims to support, not only Claude Code. kobiton/automate runs in Claude Code, Copilot CLI, Gemini CLI, Cursor CLI, and Codex CLI, so we filed install paths for Cursor, ChatGPT, and Continue, added a Claude surface compatibility matrix, and wrote about AGENTS.md as the shared brief across tools.

How do you keep an MCP server current as the spec changes?

We file advisories when the ground moves. For kobiton/automate we mapped the MCP 2026-07-28 release candidate to earlier findings, flagged the Gemini CLI sunset, and flagged Claude enterprise changes that affect deployment, each as a public issue.

What do we get at the end of an audit?

Findings filed where your team already works, as GitHub issues with a START HERE issue per pass giving the reading order, plus pull requests for the fixes we can make from outside. Your maintainers decide what merges.

Get your MCP server reviewed in public.

Tell us what your plugin or server does and which agent CLIs it needs to work in. We will talk through what the three passes would look like for it.

Talk to the team See kobiton/automate