The 11 LangSmith security checks Black Cat runs
Black Cat SSPM evaluates 11 security policies against your LangSmith configuration on every scan, classifies each finding by risk, and provides remediation steps. Browse them by topic below.
How to connect LangSmith — what access Black Cat needs, and why.
AI governance
4 checks · highest severity: high
Lifecycle & offboarding
3 checks · highest severity: high
AI governance (4)
- LangSmith Agent Unmonitored severity: medium
- LangSmith Agent High Run Volume severity: low
- LangSmith Agent No Evaluation severity: medium
- LangSmith Agent High Risk Unblocked severity: high
Lifecycle & offboarding (3)
- LangSmith Agent Stale severity: low
- LangSmith Agent High Error Rate severity: medium
- LangSmith Agent Orphaned Owner severity: high
Other checks (4)
severity: medium LangSmith Agent High Tool Diversity fix difficulty: medium #
Review and reduce the number of tools used by the agent
- Review the agent code for unnecessary tool bindings
- Remove tools that are not actively used
Satisfies: NIS2 Directive NIS2-21.i.4 DORA (SaaS Security) DORA-9.3
severity: high LangSmith Agent External API Tools fix difficulty: medium #
Review external API tool access
- Identify which external APIs the agent calls
- Ensure API keys have minimum required permissions
- Add rate limiting where possible
Satisfies: NIS2 Directive NIS2-21.i.4 DORA (SaaS Security) DORA-9.3
severity: low LangSmith Agent Dataset Access fix difficulty: medium #
Review dataset contents for sensitive data
- Open LangSmith > Datasets
- Review linked datasets for PII or sensitive content
- Apply data masking if needed
Satisfies: NIS2 Directive NIS2-21.a.3 DORA (SaaS Security) DORA-9.12
severity: medium LangSmith Agent Retriever Tools fix difficulty: medium #
Review vector store and retriever tool access
- Identify which knowledge bases the retriever queries
- Ensure data classification matches agent access level
- Restrict retriever to necessary data scopes
Satisfies: NIS2 Directive NIS2-21.a.3 DORA (SaaS Security) DORA-9.12