UCP Shop Test: Steve Madden
stevemadden.com · run 2026-11-06 · brief: women's black ankle boots, US size 8, under $150, ZIP 10001 (example brief from the skill; no brief was supplied)
Verdict
An agent got from search to a checkout that needs a contact method and a delivery address, with no human help and no browser. Requires a hosted agent profile that negotiates catalog tools; the agent's own profile (cart and checkout only) cannot search the catalog.
A shopper reached the same checkout step, but the product did not appear in site search, the cart already held an item, the cart added a duplicate line, and the size 8 item showed a back-order notice that UCP did not report.
Neither path produces a final total: shipping and tax are unknown until a full delivery address is entered. Nothing was paid, submitted, or entered as personal data.
Product compared
| Item | Value |
|---|---|
| Product | KIMMENS BLACK, tapered square-toe ankle boot (gid://shopify/Product/7260240085125) |
| Variant, size 8, black | gid://shopify/ProductVariant/41577478553733 (SKU KIMMENS, "BLACK / 8 / 001") |
| Price | $71.97 (list $119.95) on both paths |
| Why this product | Clearly an ankle boot from its description, under $150, with a size 8 variant listed as available |
Step table
| # | Step | Result | Time | Evidence |
|---|---|---|---|---|
| 1 | Discover manifest, apex and www | Pass | 0.18 s / 0.20 s | manifest.json (identical bodies) |
| 2 | tools/list over MCP | Pass | 0.17 s | 14 tools listed |
| 3 | search_catalog with cart-and-checkout profile | Fail | 0.27 s | "Tool not found: search_catalog" (profile lacks catalog capability) |
| 4 | search_catalog with valid-with-capabilities profile | Pass | 0.53 s | 10 black results under $150; query "women black ankle boots" |
| 5 | get_product, size 8 selected | Pass | 0.32 s | 03-get_product.response.json |
| 6 | create_cart, 1 unit | Pass | 0.62 s | Subtotal and total $71.97; fulfillment methods empty; response |
| 7 | create_checkout from cart ID | Handoff | 0.97 s | requires_escalation: contact method, extension interaction, delivery address. Tax $0, fulfillment methods empty. response |
| 8 | Website: search for product | Partly | about 1 min | Product not in results; direct product URL worked |
| 9 | Website: select size 8, add to cart | Partly | 19:09:02 UTC | Added, but cart already had an item: now 2 identical lines, $143.94. notes |
| 10 | Website: ZIP 10001 in delivery estimate | Partly | 19:09:54 UTC | Label changed to 10001, delivery date unchanged, no cost shown |
| 11 | Website: checkout to first personal-data step | Reached | 19:10:33 UTC | Email, name, address, phone, and "Enter shipping address to view available shipping methods" |
What differs between the paths
| Topic | UCP (protocol) | Storefront (website) |
|---|---|---|
| Size 8 availability | Available (no ship-date data) | One size 8 line shows "BACKORDERED ITEM", expected to ship 11/03/2026 (a past date) |
| Price | $71.97 | $71.97 per line |
| Shipping cost | Unknown (no fulfillment methods without an address) | Unknown; "Get it by Tue, Nov 10" shown in cart; no cost until address entered |
| Tax | $0 at checkout with no address (not final) | No tax shown before address (not final) |
| Return terms | Only a refund-policy link; the link says "within 30 days of delivery" | Product page text: holiday window (Nov 1 to Dec 31 purchases, returnable to Jan 31 2027) plus "30 days from delivery" and a $6.95 return fee; the page does not say which applies |
| Search | Exact-intent query found the product first | Search for the product name returned an unrelated carousel |
| Access | Needs a hosted agent profile with catalog capability | Open, no login, no bot challenge |
Problems, ranked by how likely they make an agent abandon or pick a competitor
- Availability and ship-readiness disagree. UCP says size 8 is available; the storefront cart says back-ordered with an expected ship date that has already passed. An agent that buys on the protocol answer will be wrong about delivery. Fix: expose back-order and ship-date status in the UCP availability object, and correct the stale 11/03/2026 date.
- Return window is contradictory and not machine-readable. The product page gives a holiday window; the refund policy says 30 days; UCP has no structured return terms. Fix: publish one policy with a structured return window in the checkout or product response.
- No shipping or tax estimate from a ZIP alone. The ZIP changed a label but no cost appeared, and UCP returns no fulfillment methods until a full address exists. The buyer cannot compare landed cost. Fix: return an estimated rate table for the postal code, marked as an estimate.
- Site search misses an exact product name. Fix: index product names and style names, and make the exact-name query return the product first.
- Catalog calls fail silently for an un-negotiated profile. "Tool not found" gives no hint about the missing capability. Fix: return an error naming the capabilities the agent must declare.
- Cart state carries over and duplicates lines. The browser session started with an item already in the cart, so a second identical line was added. Fix: for test accounts, reset carts between runs; for production, merge duplicate variant lines.
- Optional shipping protection at checkout. A $3.66 add-on appears unchecked before the delivery step. Minor, but it adds a decision the agent must handle. Fix: make it explicit in the protocol response as an optional extension.
Notes on the test itself
- Protocol calls used the Shopify-hosted
valid-with-capabilities.jsonagent profile. Thecart-and-checkout.jsonprofile returns 200 but cannot search the catalog. - Total MCP requests: 8, with about 1.5 s between calls. Plus 2 manifest fetches, 2 schema fetches, and 1 profile fetch.
- The storefront cart now holds two KIMMENS BLACK size 8 lines. The browser operator could not tell which was pre-existing and left both in place. Remove the extra line before any real purchase.
- The protocol cart and checkout were left open and will expire on their own (checkout expiry 2026-12-06).
Appendix: manifest and evidence
Manifest summary: protocol version 2026-08-25; MCP endpoint https://stevemadden.myshopify.com/api/ucp/mcp; capabilities include checkout, fulfillment, discount, cart, buyer consent, order, catalog search and lookup, identity linking (OAuth2, not used), and permalink.
Links below point to publication-safe copies: key= query tokens in cart and checkout URLs are replaced with key=REDACTED. The unmodified originals stay private in the workspace under ucp-test/raw/.