---
title: A/B Test Drafting
published: true
index: 0
---

# A/B-Test Drafting Workflow

## 1. Define one experiment

Translate the user's hypothesis, audit recommendation, or reference image into one falsifiable test. Inspect supplied images before implementing them. Use direct target evidence to explain the change; treat analytics as supporting evidence rather than proof of its cause.

Write a brief containing:

- The observed problem or user-supplied hypothesis, supporting evidence, affected routes, audience/segment, and explicitly unknown facts.
- The control experience and bounded treatment, including what a buyer will see or do differently.
- One primary metric with its definition, denominator, measurement source, and observed baseline/window when available.
- Applicable conversion, revenue, accessibility, performance, and interaction-error guardrails; identify how each will be observed.
- A proposed measurement window, exposure/sample requirements when grounded in real data, stop conditions, and the decision or retest condition. Label proposed dates and splits as unapproved.

Do not invent traffic, baseline rates, uplift, sample size, or significance. When measurement evidence is missing, specify what must be established before launch and continue useful local drafting. Do not install tracking or configure analytics as part of preparing the test.

## 2. Resolve the repository, store, and control

Read the target repository's applicable guidance. Identify its source of truth, output theme directory, build commands, existing checks, and handling of generated assets and merchant settings. Use `jg` for behavior discovery when available and authenticated, verify its source excerpts, and use `rg` for exact matches or as the fallback.

Inspect the working tree, fetch intended remote `main` when accessible and record its commit. Branch `codex/ab-<test-name>` in an isolated checkout; preserve unrelated work without stashing/resetting/moving it. If main/freshness is unknown, continue independent brief/evidence work but resolve the gap before claiming a current-main base.

Resolve the exact target store from the request or unambiguous repository configuration, then verify its relationship to the storefront. A folder name or ambient CLI login is insufficient. Identify the published theme by its role and explicit ID, not creation order. For native Rollouts, the control corresponds to the existing store experience; inspect any existing rollout affecting that experience. Selecting an unrelated unpublished source does not make it the native control.

Compare the deployable main artifact with control: merchant settings/templates, app embeds, media, locales and generated assets. Account for all uploaded differences, preserving merchant content. Reconcile authorized baseline changes only in isolation; never overwrite the primary checkout with remote files or silently commit merchant state. Unreconciled differences block upload, not useful local work.

Record store identity, base/source commit, control theme ID, relevant baseline evidence, and the planned file scope. Inspect theme capacity and rollout conflicts through available target-bound read-only capabilities. Use the [Rollouts reference](../tools/rollouts.md) to determine compatibility; lack of rollout access leaves that verification pending.

## 3. Implement and validate the treatment

For requested preview preparation, use `$preview-shopify-theme` and its local simulation or Shopify development procedure (`references/tools/local-storefront.md` or `references/tools/shopify-cli.md`). Preserve the verified source and destination identity in the handoff. Local simulation supports affected interaction QA, not Shopify checkout, native experiment eligibility or conversion evidence. Preparing a development preview does not replace this workflow's separately authorized unpublished treatment upload.

For JSON-only changes, follow `$edit-shopify-json` (`references/workflows/json-editing.md`) as the implementation procedure. It owns target-schema discovery and JSON validation; consume its changed files and verification instead of duplicating those steps. The experiment workflow retains branch/control identity, the brief, complete artifact review and treatment upload. Changes requiring Liquid/CSS/JavaScript stay in this experiment's separately authorized code scope, never inside the JSON editor.

For landing-page treatments, record audience intent, message/offer match, route/template, primary action and measurement. Theme changes stay within the reviewed hypothesis; Page records, campaign/ad settings and tracking are separate writes. Retain local templates and name owner route-assignment dependencies before readiness claims.

Prefer meaningful purchase-decision, discovery, messaging or interaction changes. A button/color-only test needs target evidence that the visual choice affects the customer barrier.

Implement only the hypothesis using existing components/helpers and source-of-truth/build pipeline. Exclude unrelated upgrades/refactors and changes to actual products/prices, catalogs, discounts, shipping, apps, checkout extensions, customer data or tracking/store configuration, even when testing their presentation.

For JSON changes, verify every section/block type, parent-child edge, setting ID, range, and select value against the target theme. The JSON editing workflow from `$edit-shopify-json` (`references/workflows/json-editing.md`) provides additional schema guidance when needed; it does not limit this skill's separately authorized code scope.

Run the target's existing build and relevant checks, parse changed JSON, run Theme Check against the deployable theme, and inspect diff hygiene. Test meaningful changed behavior using the target's existing tests when appropriate; do not add a behavior harness to Store Skills. Explain existing failures separately from introduced failures. Resolve new errors before upload; assess remaining warnings and record their impact.

Review the complete deployment artifact against the control and the declared treatment. Confirm generated output and merchant settings do not introduce undeclared differences. Prepare the brief, file scope, validated artifact, exact store, and new theme name before requesting any missing upload authorization. Do not repeat a permission question already answered for the same scope.

## 4. Upload one unpublished treatment

Follow the complete [Shopify CLI reference](../tools/shopify-cli.md). Use the verified store and output directory explicitly. Create one new unpublished theme; do not upload to the published/source/control theme or an unrelated existing draft. Do not publish or change traffic.

Capture the returned store, theme ID, name, role, preview URL, and editor URL immediately in the host's task artifacts, together with the uploaded source revision and changed-file manifest. Read back the theme role and store before subsequent writes. Keep credentials and sensitive command output out of artifacts.

After ambiguous create/upload outcomes, inspect the store before retrying. Verify run ownership, name/creation context, returned IDs and artifact evidence; names alone are insufficient. Do not repeat `--unpublished` blindly, choose arbitrary matches or delete for capacity. Report uncertainty and continue independent review until identity is safe.

Before resuming/updating the recorded theme, verify store, ownership, unpublished role and absence from scheduled/active rollouts. Otherwise retain local corrections without remote sync; unpublished themes may still serve future experiments. Preserve partial drafts/links on failure/cancellation, without automatic deletion or rollback claims.

## 5. Verify the exact preview

Before preview testing, map every treatment difference/material guardrail to control/treatment revisions, route/variant/state/viewport, expected behavior and observations. Verify deployed identity and compare matching states. Build/upload success proves neither behavior nor assignment readiness. Preserve failed/contradictory assertions and distinguish implementation, preview and launch status; missing analytics does not erase a useful draft.

Open the returned preview and confirm the treatment theme actually renders. Preserve preview selection across storefront/custom-domain redirects; an HTTP success, CLI upload success, or a screenshot of the live control is insufficient.

Inspect affected routes at desktop and mobile widths and exercise changed controls. Include adjacent cart/navigation behavior when the change can affect it. Use anonymous sessions and fictional guest-checkout details only when needed; keep marketing/account options unchecked and stop when payment options render. Never enter payment, invoke a wallet, create an account, or place an order.

Record exact routes, states, viewports, theme ID/revision, screenshots, and interaction results. Correct supported defects, rerun checks affected by the correction, and use the safe update path. When browser access or preview selection is blocked, retain the upload and links, label visual QA unverified, and give exact remaining checks. Do not call the treatment ready to schedule while a material validation or compatibility gap remains.

## 6. Return review links and the scheduling handoff

Return a compact report with:

- Experiment brief, evidence quality, measurement prerequisites, and the proposed decision condition.
- Control identity, base revision, treatment branch/diff, changed files, and baseline reconciliation.
- Validation and desktop/mobile preview evidence, including limitations.
- The actual returned theme preview/editor links and a PR link only when PR creation was authorized and completed. When no PR exists, point to the local diff/files rather than fabricating a review link.
- A manual Rollouts handoff naming the treatment theme, intended audience allocation and control/treatment split, proposed start/end dates with the verified store timezone, metrics, end behavior, and compatibility/conflict checks still needed.

Prepare the handoff using the [Rollouts reference](../tools/rollouts.md). This drafting workflow ends at review; it does not create or schedule a rollout, publish a theme, allocate traffic, apply changes permanently, select a winner, or arrange a recurring follow-up. A later scheduling action needs an appropriate workflow and explicit authorization for the reviewed store, theme, audience, dates, and end behavior.

Distinguish a locally drafted treatment, an uploaded preview, a visually verified treatment, and a draft verified as compatible with Rollouts. Keep completed artifacts and precise remaining work visible when only part of the workflow succeeds. Structured results must follow the compatible host-supplied schema exactly; do not invent an experiment result schema or pass this write workflow to a read-only review executor.

A request to operate the prepared experiment continues through the [Rollouts operations reference](../tools/rollouts.md), using existing explicit or standing authorization. The default drafting handoff does not schedule traffic. Preserve this brief, artifact identity and evidence for the funnel readout (`$report-shopify-funnel`) and later winner evaluation. Use `$promote-shopify-winner` only when installed; it is currently unpublished in Glue. Otherwise return an evaluation handoff to the owner without selecting a winner or promoting the treatment.

## Preserve learning and the next action

Maintain one linked experiment record in the existing host document/tracker: opportunity and stable IDs, hypothesis, PIE rationale when supplied, affected audience/routes, control/treatment revisions and deployment artifacts, assignment, primary metric/denominator, guardrails, agreed decision/stopping criteria, actual window, result/uncertainty, owner decision, learning and next action. Preserve win, loss and inconclusive outcomes. Reuse existing records and read back authorized updates; do not create a tracker or new result schema. Missing write capability leaves a reviewable record proposal.

Return learning to installed `$plan-shopify-optimization` or the owner's roadmap. Do not revive rejected hypotheses without changed evidence. Hosts own requested schedules/triggers; a week elapsed alone warrants neither launch nor stop.