LangSmith AI governance security checks
AI assistants, agents and model access inside the tenant — what they can read, who can publish them, and how autonomously they act.
On LangSmith, Black Cat runs 4 checks in this area on every scan. Each one below lists its severity, how to fix it, and the compliance controls it satisfies where a control applies. See what access the LangSmith connector needs.
Checks (4)
severity: medium LangSmith Agent Unmonitored fix difficulty: medium #
Add human feedback and monitoring
- Set up feedback collection for the project
- Configure annotation queues in LangSmith
- Add automated evaluation runs
Satisfies: NIS2 Directive NIS2-21.d DORA (SaaS Security) DORA-28.4
severity: low LangSmith Agent High Run Volume fix difficulty: easy #
Review high-volume autonomous operation
- Assess if the run volume is expected
- Add rate limiting or batch processing
- Monitor for cost and quality implications
Satisfies: NIS2 Directive NIS2-21.d DORA (SaaS Security) DORA-28.4
severity: medium LangSmith Agent No Evaluation fix difficulty: medium #
Add evaluation dataset for quality monitoring
- Create an evaluation dataset in LangSmith
- Link it to the project
- Set up periodic evaluation runs
Satisfies: NIS2 Directive NIS2-21.d DORA (SaaS Security) DORA-28.4
severity: high LangSmith Agent High Risk Unblocked fix difficulty: easy #
Block or restrict high-risk LangSmith agents with scores above 70
- Navigate to AI Agent Governance in SSPM
- Review the agent risk factors
- Block the agent until risks are mitigated
Satisfies: NIS2 Directive NIS2-21.d DORA (SaaS Security) DORA-28.4