create_test_suite

Create a new API test suite with test steps. Each step defines an HTTP request and assertions to validate the response. Steps can extract values from responses into variables for chaining requests. ═══════════════════════════════════════════════════════════════════ TEMPLATES FIRST — the fastest path to a passing suite: ═══════════════════════════════════════════════════════════════════ If you've called get_app_testing_context, its response includes authoring_templates — ready-to-fill suite skeletons keyed by endpoint shape (POST create, GET single, GET list self-contained, DELETE with verify, PUT update, GraphQL mutation, GraphQL query, auth+protected). Each template encodes the opinionated default for one common pattern: correct step count, correct assertion mix, correct use of app-level generators, correct cleanup. WORKFLOW for any common authoring intent: 1. Find the template whose endpoint_shape matches your intent (e.g. "POST /<resource>" for create-user). 2. Copy its steps verbatim. 3. Fill every <FILL: ...> blank with the real value (URL path, captured response body, expected status, etc.). 4. Replace {{genXxx}} placeholders with the matching keys from recommended_substitutions — call update_app_custom_variables first if a needed gen* doesn't exist yet. 5. Capture real responses via curl before pasting (R10 — MANDATORY). 6. Call this tool (create_test_suite) with the filled steps. Templates produce suites that pass the validator in one shot. Drafting from scratch (skipping templates) is the failure-iteration path — the rules block below documents why. WHEN TO SKIP TEMPLATES: only for genuine edge-case patterns (smoke tests that legitimately only do /health; contract tests asserting empty-state behaviour on fresh tenants; multi-tenant isolation checks; long-running auth flows). The rules below apply directly. ═══════════════════════════════════════════════════════════════════ STEP 0 — read the canonical schema BEFORE drafting: ═══════════════════════════════════════════════════════════════════ If you've already called get_app_testing_context, the canonical step schema is in its response under the step_schema field — read it from there. Otherwise run keploy test-suite-format once before writing any suite JSON. The schema describes the MANDATORY rules below in detail plus the two-step prelude+POST skeleton you must follow. Authors who skip this and draft from training-data priors burn ~50s per validator rejection on iter 1. ═══════════════════════════════════════════════════════════════════ APP-LEVEL GENERATORS FIRST — check before authoring a prelude step: ═══════════════════════════════════════════════════════════════════ For dynamic values needed in POST / PUT / PATCH bodies (random emails, fresh IDs, per-run tokens), the FIRST place to look is the app's existing appLevelCustomVariables — surfaced as recommended_substitutions in the get_app_testing_context response. Decision flow: 1. Read recommended_substitutions.dynamic_generators. Each entry has {key, reference, suggested_for}. 2. If a gen* generator fits your field (e.g., genEmail for an email field): → reference it directly via {{genEmail}} in the body. NO prelude step needed. 3. If no existing generator fits: → call update_app_custom_variables to add one (e.g., genUserId backed by a JS function literal). Future suites on this app reuse it. → THEN reference it in the body. Still no prelude step needed. 4. Only fall back to the per-suite prelude pattern (declare generator on step 0's extract, reference in step 1+) when the generator is truly suite-local — e.g., chains an extract from a previous step's response, which an app-level entry can't. Why this matters: each prelude step is a no-op /health hit that adds nothing to coverage. Skipping it shortens the suite, reduces author iteration time, and makes app-level generators a real reuse primitive instead of decorative metadata. ═══════════════════════════════════════════════════════════════════ MANDATORY FOR EVERY STEP — the validator rejects on iter 1 if any of these are violated: ═══════════════════════════════════════════════════════════════════ R10 — every step MUST carry a captured "response": {status, body, headers} block. Hit the endpoint locally before authoring (curl) and paste the real response. Steps with no response block are rejected outright; four downstream rules (R4 / R11 / R15-R16 / R27) silently no-op until R10 is satisfied, so missing a response also hides every assertion / extract problem in that step. R9 — every POST / PUT / PATCH body MUST reference at least one {{var}} backed by a per-run dynamic source. Two paths qualify (pick whichever fits your app): (a) PREFERRED — an app-level appLevelCustomVariables entry whose VALUE is a JS function literal (a gen* generator). Reference it directly via {{genFoo}} in the body — NO prelude step needed. Check get_app_testing_context.recommended_substitutions FIRST for existing gen* vars; if none fit, call update_app_custom_variables to add one, then reference it. This is the cleaner option when the same generator is reused across multiple suites. (b) Legacy prelude — declare a JS-function entry on an EARLIER step's "extract" (typically a /health prelude as step 0). Use this when the generator is suite-local and won't be reused elsewhere. Static app-level vars (plain string values) do NOT qualify for R9 — they'd produce identical writes across runs and break idempotency. They remain valid in headers / query / asserts. R2 — pre-request fields ("body", "url", "headers") CANNOT reference the CURRENT step's own "extract" outputs. Extract runs AFTER the response comes back; pre-request substitution sees nothing yet. R2 only matters when you choose path (b) above — declare generators on step 0, use them from step 1+. Path (a) (app-level generators) sidesteps R2 entirely. R15 — every assertion's path / status / header MUST resolve against the AUTHORED response block. JSONPath uses gjson dot-array syntax: $.orders.0.id — NOT $.orders[0].id (the bracket form does not resolve in gjson; the assertion is rejected as "key not present in recorded body"). For status_code / header_* assertions, the values must match what's in response.status / response.headers verbatim — capture the real response via curl before authoring. R32 — every step-level extract key MUST NOT collide with the app's appLevelCustomVariables (enumerate them via get_app_testing_context or getApp before authoring). The runtime's variable lookup resolves app-level first, so a colliding key means the suite's function silently never runs. Suite-suffix when in doubt: userNonceForSuite, not genUserId. Don't invent a parallel generator with the same name as an existing app-level one. ═══════════════════════════════════════════════════════════════════ GRAPHQL APPS — read this section FIRST when the app's primary endpoint is /graphql (or any other path that accepts a GraphQL request envelope). REST apps can skip this section entirely. ═══════════════════════════════════════════════════════════════════ WHEN GRAPHQL VALIDATION RUNS — automatic dispatch, no flag to set: The validator inspects each step's request body. If ANY step's body parses to a JSON envelope containing a non-empty "query" string OR an "extensions.persistedQuery.sha256Hash" (APQ flow), the ENTIRE suite routes through the GraphQL validator (G-rules). Otherwise it routes through the REST validator (Charan's R-rules described above). Decision is per-suite, NOT per-step. Consequences: - Pure GraphQL suite (every step is POST /graphql with envelope body) → G-rules - Pure REST suite (no step's body looks like a GraphQL envelope) → R-rules - MIXED suite (some REST steps, some GraphQL steps in ONE suite) → tilts GraphQL. It is categorised as a Write tool in the Keploy MCP Server, which means it can create or modify data. Consider rate limits to prevent runaway writes.

Register the Keploy MCP server in PolicyLayer and add a rule for create_test_suite: allow, deny, rate-limit, or require approval. Point your MCP client at the PolicyLayer proxy URL and the rule is enforced on every call, before it reaches Keploy. Nothing to install.

create_test_suite is a Write tool with medium risk. Write tools should be rate-limited to prevent accidental bulk modifications.

Yes. Add a rate_limit block to the create_test_suite rule in your PolicyLayer policy. For example, setting max: 10 and window: 60 limits the tool to 10 calls per minute. Rate limits are tracked per agent session and reset automatically.

Set action: deny in the PolicyLayer policy for create_test_suite. The AI agent will receive a policy violation error and cannot call the tool. You can also include a reason field to explain why the tool is blocked.

create_test_suite is provided by the Keploy MCP server (https://api.keploy.io/client/v1/mcp). PolicyLayer sits as a proxy in front of this server to enforce policies before tool calls reach the server.