create_dataset

THE RISK

Medium Risk

Why create_dataset needs a policy

This tool creates new dataset objects in LangSmith, which modifies the platform's state by adding new data structures. This is a Write operation (reversible creation) rather than Read (no side effects), Execute (arbitrary code), Destructive (irreversible), or Financial (no money involved).

From the tool's definition The tool name 'create_dataset' and description 'create datasets in LangSmith' indicate data creation functionality. LangSmith is an observability platform for language models, and datasets are persistent data structures used for testing and evaluation.

Documented attack patterns abuse exactly the kind of access create_dataset gives an agent:

POLICY

How to control create_dataset

PolicyLayer is an MCP gateway — it sits between your AI agents and LangSmith MCP Server, and nothing reaches the server without passing your rules. This is the rule we recommend for create_dataset:

policy.json

{
  "version": "1",
  "default": "deny",
  "tools": {
    "create_dataset": {
      "limits": [
        {
          "counter": "create_dataset_rate",
          "window": "minute",
          "max": 30,
          "scope": "grant"
        }
      ]
    }
  }
}

create_dataset stays usable, but capped — an agent stuck in a loop can't make hundreds of changes a minute. Everything else on the server is denied unless you say otherwise.

Create a free account and register LangSmith MCP Server — nothing to install.
Add this policy — paste it, or build it visually.
Point your MCP client (Claude, Cursor, anything) at your gateway URL.

LIMIT THIS TOOL →

Free to start. No card required.

EXPLORE

Related tools and policies

More LangSmith MCP Server tools

Execute run_experiment Call this tool when you need to understand how to run experiments and evaluations in LangS Write push_prompt Call this tool when you need to understand how to create and push prompts to LangSmith. Write update_examples Call this tool when you need to understand how to update dataset examples in LangSmith. Read fetch_runs fetch_runs Read get_billing_usage get_billing_usage Read get_prompt_by_name Get a specific prompt by its exact name. Args: prompt_name (str): The Read get_thread_history get_thread_history Read list_datasets list_datasets

All 15 LangSmith MCP Server tools →

Write tools on other servers

M-Team MCP Server download_torrent YouTube MCP Server generate_video_title Android Forensics ADB MCP Server combine_reports 0xarchive web3_signup

Go deeper

The MCP Attack Database → Documented attack patterns against MCP deployments — and the policies that stop them.
Argument validation → Checking tool call arguments against schema and policy before they run.
Scoped token → Per-person credentials that grant a defined subset of servers and tools.
Rate Limiting MCP Tool Calls: A Practical Guide →
MCP Security: Why Prompt Guardrails Aren't Enough →

FAQ

Questions about create_dataset

What does the create_dataset tool do? +

Call this tool when you need to understand how to create datasets in LangSmith. It is categorised as a Write tool in the LangSmith MCP Server MCP Server, which means it can create or modify data. Consider rate limits to prevent runaway writes.

How do I enforce a policy on create_dataset? +

Register the LangSmith MCP Server MCP server in PolicyLayer and add a rule for create_dataset: allow, deny, rate-limit, or require approval. Point your MCP client at the PolicyLayer proxy URL and the rule is enforced on every call, before it reaches LangSmith MCP Server. Nothing to install.

What risk level is create_dataset? +

create_dataset is a Write tool with medium risk. Write tools should be rate-limited to prevent accidental bulk modifications.

Can I rate-limit create_dataset? +

Yes. Add a rate_limit block to the create_dataset rule in your PolicyLayer policy. For example, setting max: 10 and window: 60 limits the tool to 10 calls per minute. Rate limits are tracked per agent session and reset automatically.

How do I block create_dataset completely? +

Set action: deny in the PolicyLayer policy for create_dataset. The AI agent will receive a policy violation error and cannot call the tool. You can also include a reason field to explain why the tool is blocked.

What MCP server provides create_dataset? +

create_dataset is provided by the LangSmith MCP Server MCP server (langchain-ai/langsmith-mcp-server). PolicyLayer sits as a proxy in front of this server to enforce policies before tool calls reach the server.

Enforce policy on every LangSmith MCP Server tool call.

Start from LangSmith MCP Server, add the rest of your stack, and see everything your agents can call. Then put policy on all of it.

CHECK YOUR STACK →