---
title: Check Evaluation
published: true
index: 10
---

# Check Selection and Evaluation

Use this workflow for the public homepage, collection and product-page Core catalog. Evaluate shared Checks on those representative surfaces and applicable browse states; do not expand into policy, cart, checkout, account or a standalone measurement report. Missing optional product configurations are not missing evidence: use representative existing products and verify applicability.

## 1. Resolve scope

- Use focused scope only when the user explicitly names checks or categories and the host has not supplied a persisted catalog.
- Otherwise select every active Core Check from this skill only (15). Full requests use `$comprehensive-shopify-audit`, not additional checks imported into this catalog.
- When the host supplies a persisted catalog, treat it as authoritative and evaluate every catalog ID even when surrounding context sounds narrower.
- Do not narrow the selected Basic audit by page, viewport, store archetype, available tool, or preferred audit lens. Those facts control execution and applicability, not catalog membership.

Before selection, record the storefront URL, available repo/revision, verified store/property, missing optional sources, theme/market, safe browser states, non-secret capabilities, comparable evidence and read-only boundary. Retain no secrets or customer/order personal data.

## 2. Read leading frontmatter

Discover Markdown files beneath `skills/core-shopify-audit/references/checks/`, excluding the navigation and authoring documents `README.md`, `AGENTS.md`, `authoring.md`, and `check-template.md`. A catalog check must begin with frontmatter containing:

- a unique stable `id`;
- a positive integer `version`;
- `status: active` or `status: deprecated`;
- a canonical `category`; and
- a first-level heading after frontmatter for the title.

Ignore deprecated checks. Treat missing or invalid required metadata and duplicate IDs as catalog errors. Read optional `subcategory`, `secondary_lenses`, `surfaces`, `viewports`, `store_archetypes`, `applies_to`, `execution_gate`, `execution_mode`, `checkout_phase`, `required_capabilities`, `credentialed_surfaces`, `required_context`, and `supporting_context` fields to plan safe execution and evidence collection.

Use standard repository searches such as:

```bash
rg -l '^status: active$' skills/core-shopify-audit/references/checks -g '*.md'
rg -l '^category: accessibility$' skills/core-shopify-audit/references/checks -g '*.md'
rg -l '^surfaces:.*product-page' skills/core-shopify-audit/references/checks -g '*.md'
```

Extract metadata before reading check bodies. Sort the selected catalog lexicographically by stable ID. Preserve ID, title, category, version, file path, selectors, execution constraints, required context, and supporting context.

Execution gates never widen permissions. When an authoritative catalog includes an account-only check and the declared account report-secret capabilities are unavailable, continue public and repository evidence first. Use Blocked only when the required signed-in page state remains necessary to decide the check.

## 3. Create bounded batches

Batch the ID-sorted catalog within host context/output limits, retaining one immutable selected-ID ledger. Batching changes no scope, categories, evidence requirements or completion rules.

Persist selected IDs/versions and source identity before collection: host-pinned revision, or external snapshot/file hashes for changing local source. Resume that identity and reconcile source changes before continuing; never mix versions silently.

Read every check in the current batch completely before evaluation. Do not summarize away applicability, exclusions, evidence requirements, evaluation states, or fixtures.

Plan each batch's assertions against exact routes, products/variants, controls, states and evidence. Reuse an applicable latest same-target browse plan, collecting fresh evidence and deciding independently. Otherwise follow visible primary-category order; add different existing products/offers only when a selected Check needs them. Explain added/replaced/absent states. Reuse verified viewports, or use supported 1280×800 desktop and 320×800 mobile; record actual dimensions/substitutions, market/currency, session/cookie limits and known revision.

For each assertion, record expected behavior/basis, exact input/action, actual state/value/evidence and comparison or verified exclusion. Make broad wording concrete before execution; “clear and usable” alone is no test. Invent no feature requirement, design preference or cutoff. Name undecidable catalog ambiguity instead of inventing Pass or a defect.

Match evidence to the assertion: before/after semantics and feedback; commerce identities, quantities and arithmetic; raw comparable measurement samples; rendered output and relevant source dependency for implementation claims. Images cannot prove computed semantics, valid source cannot prove deployment and metrics cannot prove causes. Ownership remains optional unless required by the Check. Generic procedures cannot expand permissions or integrations.

Test distinct assertions independently; another control, price, page type or state cannot stand in. Record verified absence or exact missing evidence. Use permitted representative samples without whole-site claims. Plans/assertion notes are external evidence, not new Checks/schema fields.

Record each decision during evaluation and persist its batch before loading the next. Scripts may validate/assemble existing records, never manufacture checkpoints afterward. Record:

- check ID and version;
- applicability as applicable, not applicable, or unresolved, with target evidence;
- every required and supporting context with availability and source;
- exact route, state, viewport, or repository location used;
- expected behavior, actual assertion observations, terminal outcome and a check-specific reason;
- canonical Issue reference for Fail or shared Limitation reference for Blocked.

Host-side files or standard commands may store, sort, merge, and validate these records. They must not author model-judgment outcomes or fill an unevaluated remainder.

## 4. Build context ledgers

For each selected check, record every named required and supporting context source. Mark each source as available, partial, blocked, or not applicable, with the target-bound reason.

Use safe required sources and material supporting sources; explain unused supporting evidence. Missing required evidence blocks only an otherwise undecidable Check; a reproduced defect still establishes Fail. Supporting gaps affect confidence or remain visible unless the Check requires otherwise.

Resolve context by the decision it must support:

- `browser` may use a verified desktop or mobile viewport plus DOM, accessibility tree, interaction, console, network, or screenshot evidence as the check requires.
- `repository` and `github` may establish ownership, configured features, routes, responsive behavior, variants, history, checks, and absence of optional implementations.
- `shopify-admin` may establish safely scoped aggregate orders or abandonments plus catalog, market, discount, availability, and configuration truth for the verified store.
- `ga4` may establish aggregate reach, comparable funnel shape, segments, and measurement quality for the verified property; it never proves a UX cause by itself.
- `catalog-taxonomy` may be supplied by rendered collections and products, public storefront data, repository configuration, or safely scoped Shopify catalog evidence.
- If an active check still requires an excluded source and the permitted evidence cannot independently decide it, treat that as a catalog defect to deprecate rather than a target blocker.
- `prior-audit` supports Changes only when target, state, method, environment, and relevant revision are comparable.
- `partial` required context blocks only when the missing portion is necessary for the decision. Representative evidence is sufficient when the check permits it and no material contradiction remains.

## 5. Collect and challenge evidence

Use host-provided browser, workbench, and connected-data capabilities plus the target repository's existing non-mutating commands. Evidence may include rendered states, DOM and accessibility data, console and network output, repository source and history, safely scoped Shopify and GA4 aggregates, existing tests and validation, current platform documentation, prior compatible reports, and comparable public competitor evidence.

Reproduce automated leads and test exclusions or contradictory states. Generic best practices and third-party examples are expectations to verify, not proof of a target defect.

- Use representative live routes and repository ownership for ordinary storefront checks unless a check requires broader coverage.
- Use Not applicable when an optional feature, component, business model, catalog condition, or customer job is verified absent.
- Use Fail when a required implemented path is directly observed to be broken. Block only distinct downstream checks whose required state cannot be reached.
- Treat failed screenshot capture as a presentation-evidence gap when other evidence completes the check.
- Treat verified browser emulation as mobile or desktop evidence unless the check explicitly requires a named device, engine, keyboard, or assistive technology.

## 6. Assign one terminal outcome

Every selected check receives exactly one:

- **Pass**: the complete required procedure found no actionable problem.
- **Fail**: direct evidence established an actionable problem and one canonical Issue.
- **Blocked**: missing required evidence or an unresolved contradiction prevents a responsible conclusion.
- **Not applicable**: the relevant feature, condition, state, or customer job was verified absent.

Apply outcomes in this order:

1. Decide applicability from the check's prerequisites and target-bound browser, repository, and catalog evidence.
2. If verified absent, record Not applicable with the absence evidence.
3. If applicable, complete every required step that can affect the result and challenge contradictions.
4. Record Pass or Fail when the exercised evidence decides the check.
5. Record Blocked only when an exact required source or state remains unavailable or contradictory after safe attempts, and name the smallest unblock action. “Not tested”, “matrix incomplete” and time spent are not capability failures: continue safe work when it can complete the procedure. Never silently omit a selected blocked Check.

Never infer outcomes from check ID prefixes, category, surface, viewport, context names, regular expressions, batch position, or a catch-all Limitation. Do not use cart cleanup, comparison incompatibility, or report-artifact failure as check evidence.

## 7. Reconcile batches

After all batches, compare the immutable selected-ID ledger with persisted outcomes. Reject missing, duplicate or unknown IDs and any outcome not individually recorded during its batch. Never fill a remainder from metadata, generic rationales or a catch-all blocker.

Apply the [reporting contract's final reconciliation](../core/reporting-contract.md#final-reconciliation) for Issue/Check category ownership, reciprocal references, deduplication and comparable Changes. Host-requested candidate grouping must preserve checkpointed decisions and explicit evidence-backed amendments; grouping itself cannot revise an outcome.

Before Pass, reconcile every required assertion and contradiction from the run. Later success cannot erase an earlier defect. Append evidence-backed corrections to checkpointed decisions with reason/superseded record; preserve the original.

Retest a prior same-target failure's exact element/claim/state when safe and applicable. Explain changed outcomes with current defects/corrections, changed applicability/version/route/offer or incomparable environment. Prior outcomes are leads, not automatic failures. Block unavailable required states unless another defect proves Fail; document verified inapplicability. Another control/price cannot pass an untested prior assertion. Missing history does not block fresh audits; Resolved Changes still need compatible comparison.

Do not call the audit complete while any selected Check lacks a terminal outcome. Completion covers the selected Core catalog, not storefront certification or a separate performance/funnel report.