kobiton/automate is Kobiton's public plugin for Claude Code. It connects AI coding agents to Kobiton's real-device cloud through an MCP server, and also runs in Copilot CLI, Gemini CLI, Cursor CLI, and Codex CLI. Kobiton hired Intent Solutions to review it and write about it. We reviewed it in three passes: the skill and repository, the runtime behavior, and the MCP spec and server responses. Every finding went up as a public GitHub issue. Every fix we could make from outside went up as a pull request.
So far: 25 findings in public, 10 of our pull requests merged by Kobiton, 5 more fixes written by Kobiton's own engineers that cite the findings, and the repository went from 0 to 12 GitHub stars. See the counts and sources
Everything below is public and can be checked on GitHub or kobiton.com.
No private PDF to take on faith. The findings live where the maintainers already work, and anyone evaluating the plugin can read what was found and what changed.
Counted from GitHub on 17 September 2026; stars on 19 September 2026. Review passes ran from 28 April to 18 May 2026. Each figure links to its source.
Each pass reads a different layer of the plugin, and each one builds on what the last one found.
R1 · 10 findings
Does an agent load the skill, understand when to use it, and find what it needs? Is the repo set up for outside contributors and safe updates?
R1 START HERE, issue #20R2 · 5 findings
What do the tools actually do when an agent calls them against real devices: the races, the error messages, the response shapes an agent has to parse.
R2 START HERE, issue #32R3 · 10 findings
Does the server declare what the spec expects, and does every response match what its own tool description promises?
R3 START HERE, issue #43Findings only matter if something changes. These are merged pull requests on the public repository.
- name: confirmAppUpload
annotations:
readOnlyHint: false
destructiveHint: false
+ idempotentHint: true+ openWorldHint: falseidempotentHint: true tells a client a second confirm after success is a lookup, not a duplicate upload, so a retry is safe.openWorldHint: false tells it the tool talks only to Kobiton's bounded API.Two articles by Jeremy Longshore, published by Kobiton on its own blog.
How one brief file keeps a plugin consistent across agent CLIs, and what a 5-device benchmark showed about platform variance and session lifecycle.
Read the AGENTS.md case study on kobiton.comThe improvements that make agents dependable on real cloud devices, with Kobiton as the worked example.
Read about agent reliability on kobiton.comA server can pass a security checklist and still mislead the agent calling it. This is what we check.
Ours covers three layers. The skill and repository: frontmatter, structure, examples, and governance. Runtime behavior: what the tools actually do on real calls, including races, error messages, and response shapes. The MCP spec and the server's response contract: declared primitives, protocol version, auth headers, tool annotations, and whether each response matches its own tool description.
No. Security is part of it: on kobiton/automate we covered OAuth declarations, the 401 challenge header, and repository governance such as Dependabot and CodeQL. It is not a penetration test. Most findings were about behavior an agent relies on, like a tool description promising fields the server never returns.
Annotations are hints on each tool: readOnlyHint, destructiveHint, idempotentHint, and openWorldHint. Clients use them to decide when to ask the user for confirmation and whether a call is safe to retry. kobiton/automate declared only the first two, so we filed pull request #74 adding idempotentHint and openWorldHint to all 12 tools, and Kobiton merged it.
We check the plugin against each host it claims to support, not only Claude Code. kobiton/automate runs in Claude Code, Copilot CLI, Gemini CLI, Cursor CLI, and Codex CLI, so we filed install paths for Cursor, ChatGPT, and Continue, added a Claude surface compatibility matrix, and wrote about AGENTS.md as the shared brief across tools.
We file advisories when the ground moves. For kobiton/automate we mapped the MCP 2026-07-28 release candidate to earlier findings, flagged the Gemini CLI sunset, and flagged Claude enterprise changes that affect deployment, each as a public issue.
Findings filed where your team already works, as GitHub issues with a START HERE issue per pass giving the reading order, plus pull requests for the fixes we can make from outside. Your maintainers decide what merges.
Tell us what your plugin or server does and which agent CLIs it needs to work in. We will talk through what the three passes would look like for it.