Nanobag · NB09 · 9 to 13 Sep 2026 · paused

New homepage banner + product-page tip text

Numbers as of the pause at about 10:00 Hong Kong time on 13 Sep 2026. Checked again 14 Sep 14:40: still paused, nothing restarted. Everything split by version stops at the pause on its own, because Shopify stops labelling visits the moment the rollout ends. Blue is the current site (A), purple is the new version (B).

A · current siteB · new banner + tip text

Where we are

Paused on day five. B's baskets stayed bigger, but the phone banner showed ounces to every market, so the clean read is US phones plus desktop.

Shopify AnalyticsEvery figure in this card, sales report for money and orders, sessions report for visits and conversion. No Rollouts, GA4 or tag numbers appear here.

Why it stopped, and where it stands. Rune paused the rollout at about 10:00 on 13 Sep, and it is still paused on 14 Sep with no restart. The row of tiles under B's homepage banner on phones reads "0.8 - 1.15 oz" in every market, and only the US uses ounces. Desktop never shows those tiles, so desktop is unaffected. Details and what it does to the readout are in The units problem below.
Basket size, the goal: A $47.35 vs B $49.87, B ahead by $2.52 or 5%. On day three it was 9%; the weekend pulled both arms' baskets down and narrowed it. In the US alone, net sales per order are $47.76 against $52.72, so most of the lead is American. A lean, still not a result.
Total sales are level. $15,729.56 for B against $15,660.07 for A, $69.49 apart, with B taking 17 fewer orders and making it back on size.
Orders: A 329, B 312 from the sales report; 292 against 283 tracked visits that bought. Level. Both arms fill carts at the same rate and reach checkout at the same rate.
The wrong units did not measurably hurt B. On phones outside the US, where B showed ounces, B added to cart 9.49% of the time against A's 9.04%, and converted 1.69% against 1.76%. Level. That is reassuring about the data we have; it is not a reason to skip the fix.
Paid and organic. Paid ads are about three quarters of the traffic and are dead level, 203 orders each. Organic and direct leans A, 89 to 80. The day-three picture of the two pulling in opposite directions has washed out on the paid side.

Decision: none from this run. Fix the units, restart, and read US phones plus all-market desktop as the primary cut.

5/8

of 8 planned days, paused

Started 9 Sep · paused 13 Sep 2026, about 10:00 Hong Kong time

The one number that decides it: basket size

Shopify AnalyticsSales report. Every order Shopify recorded, split by version. Saved report NB09 · Orders and AOV by arm, appendix query 2.

Average order value. The tip text under the buy button was added to make baskets bigger. Everything else on this page supports this number.

A · current site

$47.35

329 orders · $15,660.07

B · new version

$49.87

312 orders · $15,729.56

B ahead by 5% On this many orders a lead of 5% is a lean. It would have needed the full run, and a clean one, to become a result.

The scoreboard

Shopify AnalyticsOrders and sales from the sales report; visits and conversion from the sessions report. Money per visitor is sales divided by visits, so it crosses the two.

Money per visitor

$1.19$1.19

sales ÷ visits. Basket size and conversion in one number.

Level

Orders

329312

from 13,150 and 13,248 visits

A ahead by 5%

Visits that bought

2.22%2.14%

conversion rate

Level

Day by day

Shopify AnalyticsSales report grouped by day, appendix query 3.

Orders and basket size for each day on its own. The last row runs to the pause at 10:00. Read the running totals above, never a single day.

Day	A orders	A basket	B orders	B basket
9 Sep	69	$53.28	62	$53.91
10 Sep	57	$46.84	53	$54.57
11 Sep	54	$54.48	62	$49.49
12 Sep	78	$42.48	68	$45.63
13 Sep to 10:00, then paused	71	$41.90	67	$47.05

What each shopper sees

Shoppers in the US, UK, Canada, Australia, Singapore, Hong Kong and the International market are split 50/50 and stay in their version for every visit. EU is not in the test. The sticky "Choose your Nanobag" pill on phones is now standard and is in both versions, so it is not being tested.

A · current site

Homepage: today's hero video with its SHOP NANOBAG button on desktop.
Product page: the normal buy button.

Hero video · SHOP NANOBAGProduct page: Add to cart, nothing underneath.

B · new version

Homepage: Ashit's new banner, "Extreme practicality", with a Get Nanobag button on desktop.
Product page: a short tip line under the buy button.

New banner · Get NanobagProduct page: Add to cart + tip text underneath.

The units problem, and what it does to the readout

Shopify AnalyticsSessions report by version, country and device, appendix query 6. Basket by country from the sales report, appendix query 7.

The row of three tiles under B's new banner on phones carries plain text: "0.8 - 1.15 oz" and "Ultralight". It is a text setting in the theme with an imperial default and no market switch, and the seven test markets all share English, so a translation could not have fixed it either. On desktop the whole tile row is hidden, so desktop visitors in both versions saw no weights at all. In practice that gives three groups of visitors.

Visitors	A visits	A to cart	A to checkout	A bought	B visits	B to cart	B to checkout	B bought
US phones both versions showed the right units	3,478	10.5%	5.5%	3.39%	3,528	10.0%	4.8%	3.26%
Phones outside the US B showed ounces where it should have shown grams	6,822	9.0%	3.6%	1.76%	6,914	9.5%	3.6%	1.69%
Desktop, all markets no weights shown in either version	2,153	5.9%	3.3%	1.72%	2,121	6.6%	4.0%	1.79%
US desktop for completeness	1,482	3.0%	1.9%	1.48%	1,442	3.3%	1.7%	1.32%

Read it like this. US phones are the one group where both versions showed correct units and the tip text was in play. There B is behind on buying, 3.39% against 3.26%, on 3,478 and 3,528 visits, which is inside chance. Phones outside the US, the confounded group, are also level, so the wrong units did not visibly cost B anything. Desktop, with no weights in either version, has B ahead on carts and ahead on orders. None of the three groups separates the versions on the funnel; the only number that separates them is basket size, below.

Basket size by market

Shipping to	A orders	A net sales	A per order	B orders	B net sales	B per order
United States	159	$7,593.72	$47.76	151	$7,961.03	$52.72
Everywhere else	170	$8,066.35	$47.45	161	$7,768.53	$48.25

The basket lead is mostly American. Per order, B is $4.96 ahead in the US and $0.80 ahead everywhere else. The tip text sits on the product page, not on the tiles, so the units bug is not an obvious reason for that split, and at 151 US orders for B it is a lean rather than a finding. It is the first thing to look at when the test restarts.

Next time. The tile text needs to follow the market, either a second set of tiles in grams gated the same way the banner already is, or the units dropped from the tiles altogether. Rune's rule for the restart is the right one: only US phones and desktop give a clean read of the mobile banner, and the page will be cut that way from the start.

The shopper's journey, step by step

Shopify AnalyticsSessions report, appendix query 1. Shopify counts these on its own servers, so ad blockers and cookie banners cannot hide them.

Shopify RolloutsThe bounce figure in the grey note below, and nothing else in this section.

Out of every 100 visits in each version. The bar shows which version is ahead at each step.

Visited

13,15013,248

Shopify splits them evenly.

Added to cart

8.9%9.1%

1165 vs 1200 visits. Level.

Reached checkout

4.0%4.0%

531 vs 529 visits. Level.

Paid

2.2%2.1%

292 vs 283 visits. The only gap, and checkout is the same page in both.

In plain words Just as many people stay on the site: 55.7% of visits bounce on the current site against 56.0% on the new one in the 11 Sep Rollouts snapshot, and Rune reports it still level at the pause. The new version then gets people into the cart and to checkout exactly as often as the current site. The same number pay. Neither the banner nor the tip text is on the checkout page, so the mechanism, if there is one, is who the new banner sends to checkout rather than the checkout itself. At this size it is still inside chance, and it is not a reason to act.

Funnel performance

Shopify AnalyticsSessions report, appendix query 1. Every step and every band on this page comes from here.

Shopify RolloutsBounce rate only, in the last row of the table. Snapshot taken 11 Sep, midday. Nowhere else on this page uses Rollouts figures.

The same four steps read a different way. The bars compare the two versions at each step. The bands in between are the number that matters: of everyone who got that far, how many carried on. That is where a version really wins or loses people. Each bar is drawn to the share of all visits that got that far, so you can see how quickly the funnel narrows. The two versions sit almost on top of each other at every step, which is why the bands, not the bars, are where the comparison lives.

Visitedeveryone in the test

13,150 · 100.0%

13,248 · 100.0%

Of those, carry on

8.9%9.1%Level

Added to cartput something in the basket

1,165 · 8.9%

1,200 · 9.1%

Of those, carry on

45.6%44.1%A keeps more

Reached checkoutstarted paying

531 · 4.0%

529 · 4.0%

Of those, carry on

55.0%53.5%Level

Paidfinished the order

292 · 2.2%

283 · 2.1%

Read it like this On putting something in the basket the new version is level, 9.1% against 8.9%. Of those baskets, the share that goes on to checkout is behind, 44% against 46%. At the final step, paying, it is level, keeping 53% against 55%. Every band is within a few points of the other version.

Why the last step does not count against the new version The banner sits on the homepage and the tip text on the product page. Neither exists on the checkout page, which is identical in both versions. A gap at that last step therefore has no mechanism behind it. Spread over B's 529 checkouts, that 1.5 point difference is worth about 8 orders. At this number of checkouts a swing of 8 is about one ordinary day's wobble, so it is not evidence of anything yet. Worth watching, not worth acting on.

Step	A	B	A carries on	B carries on
Visited	13,150	13,248	—	—
Added to cart	1,165	1,200	8.9%	9.1%
Reached checkout	531	529	45.6%	44.1%
Paid	292	283	55.0%	53.5%
Visit all the way to order	2.22%	2.14%	Bounce 55.7% and 56.0%, level

Where the visitors came from

Shopify AnalyticsSessions report grouped by the ad tag on the landing link, appendix query 5. Saved report NB09 · Sessions by arm and source.

Every visit in the test, split by how it arrived. Paid = any click from a Meta or Google ad. Organic and direct = typed the address, searched, or followed an untagged link. The two add up to all the traffic. Paid is close to three times the size of organic, so read the grey bar first and the buying rate second.

Paid ads

Meta and Google · by far the bigger group · most land on the homepage

Of those visits, how many bought

2.1%2.1%

203 orders from 9,554 visits · 203 from 9,591

Dead level at the pause, 203 orders each. On day three this leaned B; it did not hold.

Organic and direct

typed the address, searched, untagged links · the smaller group

Of those visits, how many bought

2.5%2.2%

89 orders from 3,596 visits · 80 from 3,657

Leans A. These shoppers often know the brand and go straight to the product page, so the banner reaches fewer of them.

Why this matters Paid visitors are new to the brand and land on the homepage, so they are both the bigger group and the audience the banner was built for. Organic and direct visitors usually know Nanobag already and go straight to the product page, so they see the banner less often. Their buying rate is naturally about twice as high, which is normal for returning shoppers and is not a sign that one version is better. By the pause the paid side had levelled and only the organic lean was left, which on 169 orders is not a finding.

What people did on the page

GA4Exploration NB09 – banner + tip text, tabs 3 and 4, filtered to the test markets.

These come from tracking we added to the theme, so Shopify cannot see them at all. Visits, not raw clicks, so one person tapping twice counts once. Refreshed 14 Sep and now covering the whole run, 9 to 13 Sep, the same days as the rest of the page. Read the percentages and ignore the raw counts: GA4 stopped splitting the two versions evenly at the moment of the pause, which is explained under the cards.

📱

Tapped the pill

phones · in both versions

53013692

39.0% and 38.9% of 13,595 and 9,498 homepage visits. Level, as it should be: same pill in both.

🛒

Added to cart

product page · tip text is in B

1363989

14.0% vs 14.2% of 9,761 and 6,951 product-page visits. Level, and Shopify's own cart numbers agree. The main button under the gallery adds 1.6% and 1.6%, level too.

⏱️

Time on site

average engaged time per visit · GA4

1m 00s1m 00s

Identical to the second, on phones and on desktop. The new banner is not holding people longer or losing them faster.

Why GA4 now shows far more visits on A than on B This pull runs to 13 Sep, and the test was paused partway through that day. The arm label is written by the homepage itself: it stamps B when the banner renders and A when it does not. The moment the rollout stopped, every visitor was served the old homepage again, so every visit from then on was filed under A. GA4 therefore reads 16,950 visits on A against 11,861 on B, a 59/41 split, where Shopify's sessions report reads 13,150 and 13,248 across the same dates. The same exploration cut to 9–11 Sep splits 5,633 to 5,608, dead even. It is the same fault as the 13 Sep order tags in a different piece of code. The percentages in the cards above are unharmed, because each one is measured inside a single version.

One number on that tab is not a comparison The homepage button event fires on 403 visits in A and 63 in B. That gap is not behaviour. In A the event comes from a Google Tag Manager click trigger on the old hero's button; in B the banner's own script pushes it. Two different mechanisms with two different hit rates, so they cannot be read against each other, and nothing here says the banner button is tapped less. Settling it needs tag 96's trigger checked and B's clicks split by nb_cta into desktop and mobile.

Where these come from GA4 exploration "NB09 – banner + tip text", tabs 3 and 4, test markets only. These are visits, not raw taps, so one person tapping twice counts once.

Four systems count this test. Here is how they compare.

Shopify AnalyticsSales report and sessions report.

Shopify order tagsOrder search by tag in admin.

Shopify RolloutsThe Rollouts dashboard screen, snapshot at 11 Sep, midday.

GA4Exploration tabs 1 and 2.

Shopify counts this test three separate ways and GA4 counts it a fourth. They do not match, and they are not supposed to. Each one sees a different amount of what really happened, for reasons that are known and boring. What matters is that they all point the same way.

Where it comes from	A	B	How much it sees	What it actually counts
Shopify Analytics sales report	329	312	100%	Every order, one row each, version read from the rollout cookie at checkout. Counted on Shopify's servers, so nothing can hide from it. This is the count that decides the test.
Shopify order tags via Flow, complete days, 9 to 12 Sep	254	235	97%	A tag our own workflow writes onto each order as it is placed. Completely independent of the reports above, which is exactly why we keep it. Complete days only: on 13 Sep the theme kept stamping the arm it had stored in the browser after the rollout stopped, so that day's tags (103 and 69) mix test orders with post-test ones and are left out. That is the bug the pending banner-click fix removes.
Shopify Analytics sessions report	292	283	90%	Tracked visits that ended in an order, not orders. Two orders in one visit count once, and a visit whose tracking was blocked is missing even though its order exists. Lower by design. Carries every rate on this page.
GA4 purchases, to 11 Sep	109	102	59%	Only the orders the shopper's browser managed to report back. Express checkouts like Shop Pay and Apple Pay, ad blockers and cookie banners each remove some. Always the lowest of the four.

The bar is each system's order count as a share of the sales report over the same days, which is the most complete: the whole run for the first two rows, 9 to 12 Sep for the tags, 9 to 11 Sep for GA4. Rollouts is left out of the table because it reports no order count at all: it gives rates only, 1.96% against 1.91%, from a snapshot at 11 Sep, midday when it read 6,847 and 6,958 visits. That lines up with the sessions report to within a rounding step.

What they agree on

All four put A slightly ahead on order count, and all four make the two versions look close. The gap ranges from 3% to 8% depending on which one you read, which is the normal spread when four systems count the same few hundred orders. Every one of them also splits the traffic evenly between the versions, across the days each of them covers. Nothing here suggests the tracking is broken or that one version is being measured unfairly.

The one thing only the sales report can tell you

Order count leans A. Money leans B. That is not a contradiction, it is the whole point of the test. B takes slightly fewer orders but each one is bigger, $49.87 against $47.35, so B has brought in $69.49 more. Only the sales report can show this. The sessions report, the order tags, the Rollouts screen and GA4 all count orders and none of them carries a basket size, so a reader looking at any of those four alone would conclude the new version is slightly behind. It is not.

Why they will never match

Servers beat browsers. Shopify records an order the moment it is paid for, on its own machines. GA4 and our own tags both need the shopper's browser to send something, and a browser can refuse.
Visits are not orders. The sessions report counts visits that ended in a purchase. One shopper placing two orders in one sitting is one visit and two orders, so the two Shopify reports differ even when both are perfectly correct.
GA4 is still filling in. Today's GA4 figures keep rising for up to 48 hours. Everything on the Shopify side is near real time.
They disagree about devices. GA4 files more visits as tablet than Shopify does, because the two read iPad browsers differently. It affects both versions equally, so it does not tilt the comparison.
GA4 money is unusable here. It adds up amounts in different currencies without converting them. Basket size and sales therefore always come from Shopify, never GA4.

The rule Compare A with B inside one system. Never compare a number from one system with a number from another and call the difference a result. If any two of them ever put a different version ahead over the same full days, stop and fix the tracking before anyone reads a result.

Is the test dragging the store down?

Shopify AnalyticsSessions report for the whole store. Two sentences here; the full investigation is its own page.

No, and the question has since been answered in full. Store conversion fell from 9 to 11 Sep because a Meta placement (WhatsApp Status on the global campaign) flooded the store with visits from markets that do not buy; it stopped at about 22:00 on 11 Sep and the store recovered the next morning. Both versions of this test fell and recovered together, and the EU markets, which carry no test, fell at least as hard. The write-up with every query is on its own page, Two halves of the store.

Markets and devices

Shopify AnalyticsSessions report grouped by country and by device, appendix queries 4 and 1.

By country

Country	A visits	A orders	B visits	B orders
United States	5240	150	5261	143
Australia	2010	60	2032	50
Canada	1000	20	1044	18
United Kingdom	1078	28	1074	37
India	774	0	756	0
Singapore	441	12	422	7
New Zealand	257	4	233	5

By device

Device	A visits	A orders	B visits	B orders
Mobile	10,300	238	10,442	232
Desktop	2153	37	2121	38
Tablet	685	17	669	13

Phones carry the tip text effect and, in B, the tile row with the wrong units outside the US. Desktop carries the banner button and no weights. See The units problem above for the split by country and device.

Orders in these tables are visits that completed checkout, which run a little under the sales-report count.

Which numbers to trust, and where they live

Decides

Shopify Analytics · sales report

Orders, sales and basket size per version. Every order, counted by Shopify. Saved as "NB09 · Orders and AOV by arm".

Built in

The Rollouts dashboard

Shopify's own test screen: visits, conversion, checkout, cart and bounce per version. A separate calculation from the analytics reports, so it is a useful independent check. It carries no money at all, which is why basket size and sales never come from it.

Rates

Shopify Analytics · sessions report

Visits, carts and checkouts per version, country, device and traffic source. Saved as "NB09 · Funnel by arm", "NB09 · Sessions by arm and country" and "NB09 · Sessions by arm and source".

Behaviour

GA4 exploration

"NB09 – banner + tip text", four tabs: orders by arm, by market, homepage clicks, product page. Where the three clicks come from.

Eyes

Clarity

Recordings and heatmaps per version through the nb_arm tag, saved as segments NB09 Control and NB09 Treatment.

Order tags

Shopify order tagsShopify Admin, order search by tag. Written onto each order by our own Flow workflow, so it is a genuinely separate count from the two analytics reports.

Every order is tagged NB09-control or NB09-treatment plus its device. Search orders by tag in admin to see the real orders behind any number here.

The section Four systems count this test above compares this count with the other three.

Where every number on this page comes from

Section	Source	Why that one
Basket size, scoreboard, day by day	Shopify Analytics sales report	Counts every order server side, including ones where the shopper blocked tracking.
Journey, funnel performance, markets, devices, traffic sources	Shopify Analytics sessions report	The only source that ties a visit to what that visit did, so it carries all the rates.
Bounce rate, and nothing else	Shopify Rollouts dashboard	A different system from the analytics reports, with its own calculation. The only place that reports bounce per version.
Pill taps, add to cart, time on site	GA4	These are clicks on the page. No Shopify system sees them.
Order tag counts	Shopify order tags via Flow	A count written by our own workflow, used only to check the other three.
The units problem	Shopify Analytics sessions report by country and device, sales report by shipping country	The only way to separate the visitors who saw correct units from those who did not.

Note that Shopify is three separate systems here, not one. The analytics reports, the Rollouts dashboard and the order tags each count the test independently and will never agree to the last order. Money always comes from the sales report and never from GA4, because GA4 adds up amounts in different currencies without converting them. Rates always come from one system at a time, never mixed.

How to read the test

Basket size first. That is the goal.
Money per visitor second. It catches a bigger basket that costs a few orders, or the reverse.
Orders third, from the sales report only.
Clicks and traffic sources explain, they never decide.
Nothing before day three, a lean by day five, a decision after the full run. This run was paused on day five, so it ends at a lean.

Appendix: queries, IDs and events

Everything on this page comes from Shopify's own Analytics on the live store, run in Analytics → Reports → New report. Paste any query below to reproduce the exact figure. Nothing here is from a third-party tool, and nothing is estimated.

The store-wide queries that used to sit here have moved to the Two halves of the store page with the rest of that investigation.

-- 1. Funnel by version and deviceFROM sessions
SHOW sessions, sessions_with_cart_additions, sessions_that_reached_checkout, sessions_that_completed_checkout
WHERE rollout_treatment_ids CONTAINS 17432712 OR rollout_treatment_ids CONTAINS 17465480
GROUP BY rollout_treatment_id, session_device_type
SINCE 2026-09-09 UNTIL today

-- 2. Orders and basket by version
FROM sales
SHOW orders, net_sales, average_order_value
WHERE rollout_treatment_ids CONTAINS 17432712 OR rollout_treatment_ids CONTAINS 17465480
GROUP BY rollout_treatment_id
SINCE 2026-09-09 UNTIL today

-- 3. The same, by day
FROM sales
SHOW orders, net_sales, average_order_value
WHERE rollout_treatment_ids CONTAINS 17432712 OR rollout_treatment_ids CONTAINS 17465480
GROUP BY day, rollout_treatment_id
SINCE 2026-09-09 UNTIL today
ORDER BY day

-- 4. Visits and orders by version and country
FROM sessions
SHOW sessions, sessions_with_cart_additions, sessions_that_completed_checkout
WHERE rollout_treatment_ids CONTAINS 17432712 OR rollout_treatment_ids CONTAINS 17465480
GROUP BY rollout_treatment_id, session_country
SINCE 2026-09-09 UNTIL today
ORDER BY sessions DESC

-- 5. Visits and orders by version and traffic source
FROM sessions
SHOW sessions, sessions_with_cart_additions, sessions_that_completed_checkout
WHERE rollout_treatment_ids CONTAINS 17432712 OR rollout_treatment_ids CONTAINS 17465480
GROUP BY rollout_treatment_id, utm_medium
SINCE 2026-09-09 UNTIL today
ORDER BY sessions DESC

-- 6. The units problem: US only, by version and device. Drop the country clause for all markets.
FROM sessions
SHOW sessions, sessions_with_cart_additions, sessions_that_reached_checkout, sessions_that_completed_checkout
WHERE (rollout_treatment_ids CONTAINS 17432712 OR rollout_treatment_ids CONTAINS 17465480)
  AND session_country = 'United States'
GROUP BY rollout_treatment_id, session_device_type
SINCE 2026-09-09 UNTIL today

-- 7. Basket by version and shipping country
FROM sales
SHOW orders, net_sales, average_order_value
WHERE rollout_treatment_ids CONTAINS 17432712 OR rollout_treatment_ids CONTAINS 17465480
GROUP BY rollout_treatment_id, shipping_country
SINCE 2026-09-09 UNTIL today
ORDER BY orders DESC

-- 8. Returning customers inside the test, by version
FROM sales
SHOW orders, returning_customers, new_customers
WHERE rollout_treatment_ids CONTAINS 17432712 OR rollout_treatment_ids CONTAINS 17465480
GROUP BY rollout_treatment_id
SINCE 2026-09-09 UNTIL today

Order tags: admin order search tag:NB09-control and tag:NB09-treatment. The search's date filter only honours whole shop days, so the complete-day counts here come from subtracting the 13 Sep count from the total. When the rollout stopped, both versions' visits stopped being labelled within the hour, at about 10:00 Hong Kong time on 13 Sep, which is how the pause time on this page was fixed.

Paid = utm_medium paid_social, cpc, paid and the Facebook_* / Instagram_* placements. Organic and direct = blank utm_medium.

Rollout 8716424 "NB-09 Banner". Control 17432712 · Treatment 17465480. Theme arm values 149369520264-A / -B. GA4 events, all carrying nb_arm: nb_arm_view, nb_home_sticky_view / nb_home_sticky_click (pill), nb_home_cta_click (hero button in A, banner button in B), nb_atc_click with atc_location sticky or main.