Attack class reference · data
Every evasion rate and every detection rate published for agent skill scanners in 2026, side by side — with what each number actually measures, because no two papers measure the same thing.
Purpose-built evasions get past skill scanners in the 80–96% range. The detectors that report the highest rates reach 92–97% only when they execute the skill, inspect the whole installed set, or lean on an LLM judge — and that judge's own detection rate swings from 35% to 95% depending on the model.
Since February 2026 more than a dozen papers have measured how well scanners catch malicious agent skills, from both sides: attacks designed to slip past them, and detectors designed to catch what slips. The figures are quoted everywhere and rarely next to each other. This page puts them in two tables, attacks first, detectors second, and keeps the authors' own definition of success next to each figure.
Every number below comes from the authors' abstract on arXiv, or from the Trail of Bits write-up, checked against the arXiv abstract page and the arXiv API on 20 September 2026. Nothing was reproduced independently. Where an abstract reports no figure, the row says so rather than guessing. For what skill injection is and why scanning is structurally weak against it, see why scanners miss it on the main page.
| Attack · paper | Technique | What is counted | Reported rate |
|---|---|---|---|
| Trail of Bits, “The sorry state of skill distribution”blog · Jun 2026 | Newline padding past the scanner's truncation limit; a payload hidden in a .docx archive; poisoned Python bytecode; prompt-injection framing to talk an LLM analyzer into approving a malicious registry change. | Malicious skills accepted by ClawHub's detector, Cisco's skill scanner, and the three scanners integrated into skills.sh. | 5 of 5 scanners bypassedthree of the four skills built in under an hour |
| PoisearXiv:2606.07943 · Jun 2026 | One benign-looking, command-bearing instruction placed at a structurally plausible position in the skill body. | Attack success = the malicious action completes in a sandbox and the user's task still passes its verifier. Alerts measured under the SkillTester audit. | 89.3% ASRonly 5.6% of variants gain a new high-risk alert; the four LLM judges falsely flag 74.6% of clean skills |
| SkillCamo, “Seeing Is Not Screening”arXiv:2606.18198 · Jun 2026 | Malicious instructions hidden in images bundled with the skill; the documentation references the images as part of the normal workflow. | Scanners that read text, manifests and code only; the multimodal agent recovers the instruction at execution time. | no rate in the abstract; “challenge existing skill scanners” |
| PhantomSkill / VulMaskarXiv:2606.19191 · Jun 2026 | Overt malicious scripts rewritten as vulnerability-shaped code in the skill's auxiliary resources, activated only under attacker-controlled triggers. | Warning- and malware-level detection by automated reviewers, compared with the overt script. | no rate in the abstract; “reducing warning and malware-level detection” |
| SkillCloak, “Cloak and Detonate”arXiv:2607.02357 · Jul 2026 | Structural obfuscation (payload indicators rewritten into equivalent forms) and self-extracting skill packing (malicious parts hidden at install time, restored at execution). | Scanner returns a clean verdict. Eight scanners, 1,613 in-the-wild malicious skills. | >90% bypass, every scannerself-extracting packing; structural obfuscation >80% on most static scanners, 96% on a hybrid scanner |
| ColluSkillarXiv:2608.09732 · Aug 2026 | One malicious intent split into interdependent sub-payloads across separately packaged skills; LLM chain planning refined against scanner feedback. | Average attack success across six representative skill scanners that inspect one skill at a time. | 96.0% ASRtheir ChainGuard defense brings it to 22.5% while passing 99.5% of benign workflows |
| CompoSkillarXiv:2608.16246 · Aug 2026 | Chains of individually scanner-passing marketplace skills whose combined outputs and side effects form a risky composition; white-box and black-box attackers. | Chain Formation Rate on CompoSkill-Bench, 1,140 records, OpenClaw and Nanobot; scanners “block only a limited fraction”. | 83.3% CFR white-box80.6% black-box; success decays once a chain exceeds three skills |
| Detector · paper | Approach | What is counted | Reported rate |
|---|---|---|---|
| SkillsMetricarXiv:2608.08468 · Aug 2026 | Static, five stages: pattern density, statistical anomaly, dataflow taint, import anomaly, capability mismatch. | 2,266 adversarial skills across 16 attack types, plus the SkillMD-138K corpus. | AUC 0.93 · F1 73.4%exfiltration 93%, steganographic payloads 93%; host destruction via common shell commands 0%; natural-language prompt injection 42% |
| SkillVetBencharXiv:2606.15899 · Jun 2026 | LLM-as-judge with a five-dimensional agentic risk score; public leaderboard. | 78 confirmed-malicious skills and 22 benign controls; comparison with static tools on instruction-layer threats. | 0 false negatives, 0 false positivesbest static baseline misses 15%; conventional tools miss 89–100% of instruction-layer threats; detection ranges 35–95% across four LLM evaluators |
| Locate-and-JudgearXiv:2606.23416 · Jun 2026 | Attention-based locator keeps the top-K instruction-following spans; an LLM judge examines only those. | Marketplace-scale deployment; precision confirmed manually on flagged skills. | order-of-magnitude cheaper than full LLM scanningsurfaces dozens of live malicious skills, many missed by SkillSpector and Cisco Skill Scanner; no rate in the abstract |
| SkillGatearXiv:2607.25619 · Jul 2026 | Regex prefilter, then an LLM judge that sees only the matched snippets. | SkillsBench, 1,650 skills, 9.1% malicious; compared with two existing tools. | F1 0.817 · FPR 1.13%AUPRC 0.830 vs 0.144 and 0.162 for the two tools; 77% fewer LLM input tokens |
| SkillDetonate, “Cloak and Detonate”arXiv:2607.02357 · Jul 2026 | Runtime auditor: executes the skill in a sandbox and tracks sensitive-data flows across context, files, processes and network. | Attacks detected, including the SkillCloak evasions above, and on real-world malicious skills. | 97% detection at 2% FPR87% on real-world malicious skills |
| SkillSentryarXiv:2608.03485 · Aug 2026 | Dynamic testing in adaptive LLM-simulated “honey worlds” with decoy resources; compares skill-enabled and no-skill trajectories. | Standard benchmarks, then semantics-preserving evasion; seven scanner configurations as baselines. | 99.50% recall · 96.26% F192.95% F1 under semantics-preserving evasion vs 80.07% for the strongest baseline |
| ChainGuard, from ColluSkillarXiv:2608.09732 · Aug 2026 | Context-aware chain scanner: analyzes a candidate skill together with the skills already installed. | ColluSkill attack success after the defense; benign workflows still passing. | ASR 96.0% → 22.5%99.5% of benign workflows pass |
| SkillSecurerarXiv:2609.14079 · Sep 2026 | Red agent injects nine threat types and records the change; blue agent detects, localises and patches; a verifier compares them. | Injection-level detection on controlled instances; then popular skills from skills.sh. | 100% injection detectionwith its best backend; latent vulnerabilities in more than 17% of the skills.sh skills examined |
| Registry reality, “Scan the Skill, Govern the Action”arXiv:2609.12001 · Sep 2026 | Measurement over ClawHub rather than a new scanner. | 66,192 public ClawHub skill versions; agreement between the registry's scanners; skills every scanner rates clean. | scanners overlap on ≤10.4% of positives81.9% of flagged skills caught by one scanner alone, as reported by OpenClaw's security team; 705 skills rated clean by every scanner still instruct an action CIS and NIST controls prohibit; 92% precision on a hand audit of 100 |
| Baseline: “Do Not Mention This to the User”arXiv:2602.06547 · Feb 2026 | Static pattern matching plus dynamic behavioral verification over two registries. | Confirmed malicious skills among 98,380 collected. | 157 malicious of 98,380632 vulnerabilities across 13 techniques, 4.03 per skill on average; over half from one actor; all 157 removed after disclosure |
They are not one scale. Poise's 89.3% requires the attack to complete and the user's task to pass. SkillCloak's “over 90%” is a scanner returning a clean verdict on a known-malicious skill. CompoSkill's 83.3% is a risky chain forming, whether or not it is later blocked. On the detector side, F1, recall and AUC each hide a different trade-off, and each paper scores itself on its own dataset. Read a row's third column before its fourth.
The pattern is consistent anyway. Anything that judges a skill by its appearance — patterns, signatures, one skill at a time — is beaten by attacks that keep the meaning and change the form, or that spread the meaning across several files. The detectors that report the highest rates do one of three things: they run the skill (SkillDetonate, SkillSentry), they look at the whole installed set (ChainGuard), or they ask a language model to judge intent. That last option carries its own numbers: detection from 35% to 95% depending on which model judges, and in Poise's audit, four LLM judges flagging 74.6% of clean skills as risky.
Registries confirm it from the inside. On ClawHub the scanners agree on at most 10.4% of what they flag, and 705 skills that every scanner rated clean still instruct an action that baseline security controls prohibit. This is not a failure of one product; it is what OWASP's Agentic Skills Top 10 lists as AST08 — Poor Scanning: relying on the scan as the boundary.
What this means in practice is unchanged from the main page: scanning is a filter that removes the careless attacker, and the boundary has to be provenance, least privilege, and a human on irreversible actions. See reducing the risk.
No. Each paper defines its own success condition: Poise counts an attack only if the malicious action completes and the user's task still passes, SkillCloak counts a scanner returning a clean verdict, CompoSkill counts a risky chain forming, and the detectors report F1 or recall on their own datasets. The table keeps the definitions next to the numbers for that reason.
Not from appearance alone. The detectors that report the highest rates in 2026 either execute the skill in a sandbox, inspect the whole set of installed skills together, or rely on an LLM judge whose detection rate varies from 35% to 95% depending on the model. Static pattern analysis reports 0% on host-destruction payloads built from ordinary shell commands and 42% on natural-language prompt injection.
Trail of Bits bypassed ClawHub's malicious-skill detector, Cisco's skill scanner, and all three scanners integrated into skills.sh. SkillCloak reports bypassing eight scanners, ColluSkill six, and Locate-and-Judge surfaces live malicious skills that SkillSpector and Cisco Skill Scanner fail to detect. Most academic papers do not name the commercial scanners they tested.
From the authors' own abstracts on arXiv and from the Trail of Bits post, each checked against the arXiv abstract page and the arXiv API on 20 September 2026. Nothing here was reproduced independently; the page reports what the authors claim and what they measured.