Nanobag · NB08 · 2 to 8 Sep 2026 · final readout

Sticky "Choose your Nanobag" button + product-page tip text

One page that explains what the test was, which numbers matter, where they live in GA4 and Shopify, and what they said at the end. Written for anyone on the team, no analytics background needed.

Shopify numbers cover the whole run, 2 Sep 00:00 to 8 Sep 22:00 Hong Kong time, pulled 9 Sep after the test stopped, so they are final. GA4 numbers cover 5 to 7 Sep (see why below). Prepared by Shubham Yadav.

Final · 7 test days · 790 orders

Started 2 Sep, paused 3 Sep 18:50 to 5 Sep 00:05, stopped 8 Sep at about 22:00 HKT, four days early, to make room for NB09 on 9 Sep. Shopify only counts a shopper while the test is running, so the pause and the early stop drop out cleanly.

Final result · 2 to 8 Sep 2026 · 26,613 sessions · 790 orders

The tip text did not lift basket size. The sticky button changed nothing that reaches the till.

Average order value, the main goal, is flat. A $51.94, B $51.53. The tip text under the buy button was meant to make baskets bigger. Over 790 orders it moved the number by 40 cents, in the wrong direction. That is noise.
Orders and conversion are level. A 399 orders vs B 391; conversion 2.62% vs 2.55%. Net sales $20,985 vs $20,240.
Revenue per visit leans A by 5%. $1.59 vs $1.51. A small lead that never went away, but not large enough to call a loss with this much data.
The sticky button does what a button does, no more. B adds to cart 10% more often (1,425 vs 1,297 sessions), but a smaller share of those carts reach checkout (44% vs 51%), so the extra carts never become orders. Same picture every week of the test.
No market or device carried a hidden win. Mobile, where the sticky button lives, leans A slightly (269 vs 257 sessions that ordered). Desktop, where only the tip text differs, is level (69 vs 71). Every market is within five orders of even.

What this means: B did not win on the goal we set and did not lose on anything that matters. Decision taken 8 Sep: the sticky button stays on as the site default in the seven test markets from 9 Sep, because it costs nothing and gives the homepage one tracked route to the product page. The tip text goes back into testing inside NB09, paired with the new hero banner, where basket size is measured again with a fresh control.

What we are testing

Shoppers in seven non-EU markets (the US, UK, Canada, Australia, Singapore, Hong Kong and the rest of the International market) are split 50/50 by Shopify. EU markets are not in the test. Each shopper stays in the same arm for every visit.

Arm A · control

The site as it was.

Homepage: the normal "Choose your Nanobag" button inside the hero video.
Product page: standard buy button.

Arm B · treatment

Two small changes.

Homepage on mobile: the hero button is hidden and a sticky pill sits at the bottom of the screen the whole time you scroll. It links to the flagship product page.
Product page: a short tip line under the buy button.

The main goal is a bigger basket. The tip text is there to lift average order value. The sticky button is the supporting change: get more people to the product page so more of them see it. So the first number to read is average order value, then revenue per visit, then orders.

Headline numbers · Shopify, full run 2 to 8 Sep

Shopify is the source of truth for sessions, orders and money. It counts every order server-side. The pause (3 Sep evening to 5 Sep) and everything after the stop on 8 Sep 22:00 are excluded automatically: Shopify only stamps an arm while the test is running.

Average order value · main goal

$51.94$51.53

Shopify average order value

Even · +1% A

Revenue per session

$1.59$1.51

net sales ÷ sessions

A ahead · +5%

Conversion rate

2.62%2.55%

sessions that completed checkout

Even · +3% A

Orders

399391

$20,985 vs $20,240 net sales

Even · +2% A

Arm AArm B"Even" means the gap is under 5%. With about 13,000 sessions and 400 orders per arm, a gap that small cannot be separated from luck even over a full week.

Average order value by day · Shopify

The goal metric, day by day. Faint dots are each day's own average; the solid lines are the running average since 2 Sep, which is the number that decides. 3 Sep stops at the pause, 4 Sep was paused, 8 Sep runs to the stop at 22:00.

Day	A orders	A AOV that day	A running	B orders	B AOV that day	B running
2 Sep	71	$61.28	$61.28	65	$58.89	$58.89
3 Sep to pause	44	$46.34	$55.56	35	$57.35	$58.35
5 Sep	72	$52.22	$54.27	61	$48.37	$54.57
6 Sep	69	$52.53	$53.80	71	$52.06	$53.80
7 Sep	72	$50.52	$53.08	84	$42.62	$50.83
8 Sep to 22:00	71	$46.64	$51.94	75	$54.49	$51.53

The trend Both arms started high on 2 Sep and drifted down together, from about $60 to about $52. They crossed three times and finished 40 cents apart. B's single days swing more, from $43 to $59, because a few large orders move a 70-order day a lot. At no point did the tip text open up a lead in basket size. That is what "flat" means here, and a week of data is enough to say it.

The funnel, step by step · Shopify

Same full-run window. Each pair of bars is one step; the bar length is relative to the larger arm at that step.

Sessions

13,236

13,377

Even

Added to cartA 9.8% · B 10.7% of sessions

1,297

1,425

B ahead · +10%

Reached checkoutA 51% · B 44% of carts

659

620

A ahead · +6%

Completed checkoutA 53% · B 55% of checkouts

347

341

Even

What this says Arm B put more things in carts and a smaller share of those carts reached checkout. The two cancelled out: same orders, same money. The mid-funnel drop held for all seven test days, so it is real rather than noise. The likely reason is that the pill collects taps from people who were only browsing, who add a bag and then leave; it did not cost sales, and it did not add any.

Homepage and product-page behaviour · GA4

These are the numbers only GA4 can show, because they come from our own click tracking on the theme. Sessions in the test markets, 5 to 7 Sep, the three complete days of run 2. 8 Sep is left out because after the stop at 22:00 every visitor was stamped Arm A, and GA4 cannot cut a day by the hour.

Why GA4 stays on 5 to 7 Sep while Shopify covers 2 to 8 During the pause (3 Sep evening to 5 Sep midnight) every visitor saw the control site and was stamped Arm A in GA4. Set GA4 to 2 to 8 Sep and Arm A swells to 14,724 sessions against B's 10,431: about 4,000 of those A sessions were never in the test. GA4 cannot cut a range by the hour, so the clean GA4 window is run 2 only. Shopify does not have this problem because it only stamps an arm while the test is live.

Homepage: did they get to a product page?

Saw the homepagesessions

4,219

4,133

Even

Reached a product pagesessions, any route

2,124

2,123

Even

Tracked button clicksA: hero button only · B: the pill

1,151

1,894

Not comparable

Why the click counts cannot be compared Arm A's homepage has eight buttons that lead to the product page: the hero button plus "Totes | Slings | Backpacks", "FREE gifts automatically added" and four "Shop Now" buttons further down. Only the hero button is tracked. Arm B hides all eight on mobile and leaves just the pill, so the pill collects every product-bound tap that A spreads across eight buttons. That is why B's tracked clicks look 65% higher while product-page arrivals are identical and Shopify shows B with only 2% more pageviews. Verified 7 Sep by tapping the pill in a fresh arm-B session: the click event, the navigation and the product-page view all fire correctly. The fair homepage measure is the second row: the share of homepage sessions that reach a product page, and it is level.

Product page: did they add to cart?

Saw a product pagesessions

2,124

2,123

Even

Clicked add to cartsticky bar (all mobile) + main button (desktop)

569

635

B ahead · +12%

Add-to-cart rate

27%

30%

B ahead

Orders and basket size by day · Shopify

Whether a lead was steady or one lucky day. 3 Sep stops at the pause; 4 Sep was fully paused; 8 Sep runs to the stop at 22:00. The line under each day is average order value.

2 Sep

AOV $61 vs $59

3 Sep to pause

AOV $46 vs $57

5 Sep

AOV $52 vs $48

6 Sep

AOV $53 vs $52

7 Sep

AOV $51 vs $43

8 Sep to 22:00

AOV $47 vs $54

A won the first three test days, B edged the last three, and the totals finished 399 to 391. Basket size swung by $10 or more from one day to the next in both arms, which is why the running average, not any single day, was the number to read.

By market and by device · Shopify

Did one market or device carry the result? Read the conversion column, not the raw counts, and remember that markets with under 20 orders per arm cannot tell you anything on their own.

Market	A sessions	A orders	A conv.	B sessions	B orders	B conv.
United States	6,203	197	3.18%	6,186	192	3.10%
Australia	1,404	55	3.92%	1,392	54	3.88%
Canada	1,320	33	2.50%	1,289	30	2.33%
United Kingdom	1,240	30	2.42%	1,222	26	2.13%
Singapore	438	9	2.05%	483	12	2.48%
India	509	0	0%	571	0	0%
New Zealand	171	3	1.75%	223	3	1.35%
Hong Kong	144	1	0.69%	168	3	1.79%

Orders here are sessions that completed checkout, so the totals are a little lower than the sales report's order count.

Device	A sessions	A carts	A orders	A conv.	B sessions	B carts	B orders	B conv.
Mobile	10,197	1,020	269	2.64%	10,143	1,141	257	2.53%
Desktop	2,295	210	69	3.01%	2,498	213	71	2.84%
Tablet	719	67	9	1.25%	716	71	13	1.82%

Nothing here changes the headline. The US, which is half the test, finished 197 to 192. Australia, which looked like B's best market mid-test, ended level at 55 to 54. Canada and the UK lean A by three or four orders, Singapore leans B by three; all of those are inside the noise for a market that size. The sticky pill only existed on mobile, and mobile leans A slightly (269 vs 257 sessions that ordered, but B added 12% more carts). Desktop, where only the tip text differed, is level, which is the cleanest read on the tip text on its own.

Paid vs organic · Shopify

Did paid and organic shoppers react differently? Sessions split by the ad tag on the landing link. Blank means no tag: organic search, direct, and most social taps.

Source	A sessions	A carts	A orders	A conv.	B sessions	B carts	B orders	B conv.
Meta ads paid_social	6,246	587	139	2.23%	6,215	653	125	2.01%
Organic and direct no tag	3,870	390	116	3.00%	4,096	427	122	2.98%
Google ads cpc	1,837	215	72	3.92%	1,760	190	57	3.24%
Other paid paid	730	46	10	1.37%	721	71	15	2.08%

Organic and direct shoppers behaved identically in both arms. The whole of A's small lead sits in paid traffic: Meta 139 vs 125 orders and Google 72 vs 57. Paid shoppers land on the product page more often than the homepage, so the sticky pill rarely reaches them and the difference is the tip text plus noise. Google's gap is 15 orders on 1,800 sessions a side, which is one good afternoon. Worth remembering for NB09, not worth acting on here.

Two tools, one picture

Same window in both tools, 5 to 7 Sep, so the columns are comparable. GA4 and Shopify will never show the same absolute numbers. They agree on the thing that matters: which arm is ahead and by roughly how much.

	Shopify A	Shopify B	GA4 A	GA4 B
Sessions	6,485	6,464	5,584	5,384
Sessions that ordered	168	162	150	126
Conversion	2.59%	2.51%	2.69%	2.34%
As of	7 Sep 21:00		7 Sep 21:00 · GA4 sees 80 to 90% of Shopify's numbers

A third check: order tags

Three Shopify order numbers, three meanings 399 / 391 is the sales report: every order, one row per order, arm read from the rollout cookie at checkout. 347 / 341 is the sessions report: how many tracked visits ended in an order, so two orders in one visit count once, and a visit whose analytics pixel was blocked is not counted at all even though its order exists. 397 / 378 is the theme's own tag, missing on orders where the browser blocked storage or the cart skipped the theme. Use the sales report for orders and money, the sessions report for rates, and the tags as a sanity check. All three agree: level.

Every order also carries a tag written by the theme itself (NB08-control or NB08-treatment, plus NB08-mobile or NB08-desktop), independent of Shopify's rollout count. Full run, 2 to 8 Sep:

Order tags	Arm A	Arm B	Read
All tagged orders	397	378	A +5%
Mobile	305	286	A +7%
Desktop	82	83	Even

Tags cover 98% of the rollout's orders (775 of 790); a few orders miss the stamp when the browser blocks storage. The tags lean A a little more than the rollout count does (13 orders of the 19-order gap are mobile), which fits the mobile read above: if anything the pill cost a handful of mobile orders, but not enough to call.

Why they differ

GA4 only sees 75 to 85% of traffic. Cookie banners, ad blockers and express checkouts (Shop Pay, Apple Pay) drop hits. Shopify counts everything server-side.
GA4 lags. Today's numbers keep filling in for up to 48 hours. Shopify is near real time.
GA4 revenue is unusable here. It adds up raw amounts in different currencies. Use Shopify for money.

The rule Compare A with B inside one tool. Never compare a GA4 number with a Shopify number. If GA4 and Shopify ever disagree on which arm is ahead over the same complete days, stop and ask before anyone reads a result.

What each tool is for: Shopify for sessions, orders, revenue, conversion, per market and per device. GA4 for what people did on the page: button clicks and add-to-cart, which Shopify cannot see.

Where the numbers live

Everything below is already built and saved. Nobody needs to write a query or a filter.

GA4 · one exploration, four tabs

GA4 → Explore → NB08 – sticky CTA + tip text. Property NB Shopify - info-fc88. Set the date range to 5 to 7 Sep 2026 for the clean run-2 read; the test stopped 8 Sep.

Tab 1

Orders by arm

Question: Does Arm B sell more?
Read it: Rows are device. Arm A and Arm B are the two column groups. Purchases ÷ sessions is the conversion rate.
5 to 7 Sep: A 5,584 / 150 · B 5,384 / 126

Tab 2

Orders by market

Question: Is one country driving the result?
Read it: Same two numbers, rows are countries. Ignore EU rows, they were never in the test.
5 to 7 Sep: US 87 vs 77 · UK 27 vs 17 · AU 29 vs 31

Tab 3

Homepage clicks by arm

Question: Is the pill being used?
Read it: Three rows: homepage seen, hero button clicked, sticky pill clicked. A's hero row and B's pill row are not comparable (A has seven other untracked buttons). Use it only to confirm the pill is live and clicked.
5 to 7 Sep: Pill clicked in 1,766 B sessions

Tab 4

Product page by arm

Question: Does the tip text push more people to add to cart?
Read it: Rows: product page seen, add-to-cart clicked (split by sticky bar vs main button). The "(not set)" row under page seen is expected.
5 to 7 Sep: A 27% · B 30%

Shopify · four saved reports

Shopify admin → Analytics → Reports → search "NB08". Set the range to 2 to 8 Sep 2026; nothing is stamped after the stop. ID 16842888 is Arm A, 16875656 is Arm B.

Report

NB08 · Funnel by arm

Question: Where does each arm lose people?
Read it: Sessions → added to cart → reached checkout → completed. Matches GA4 tab 1 plus the middle steps GA4 undercounts.

Report

NB08 · Orders and AOV by arm

Question: Which arm makes more money?
Read it: Orders, net sales, average order value. Divide net sales by the funnel report's sessions for revenue per session.

Report

NB08 · Orders per day by arm

Question: Is a lead steady or one lucky day?
Read it: One row per day per arm. 8 Sep runs to the 22:00 stop.

Report

NB08 · Sessions by arm and market

Question: Is one market skewing it?
Read it: Matches GA4 tab 2. Read the conversion column; ignore markets under 20 orders per arm.

How this test was read

Average order value was the goal, revenue per session the judge. A bigger basket only counts if it does not cost orders. Clicks explain the mechanism and never decide anything.
Gaps under 5% are "Even". At this traffic level a week cannot separate a 5% gap from chance. The largest gap on any decision metric was 5% on revenue per session, so nothing crossed the line.
Read whole days. 8 Sep ran to 22:00 and is treated as a full day; both arms lost the same two hours.
Read from 2 Sep. The pause (3 Sep 18:50 to 5 Sep 00:05) drops out of Shopify automatically. 3 Sep afternoon had a store-wide order dip that hit both arms equally and is unrelated to the test. GA4 is read for 5 to 7 Sep only, for the reason given above.
Nothing changed mid-test. No theme pushes and no editor saves in the test markets while it ran. The one GTM publish (4 Sep, hero click tracking) added a tracking event only, during the pause.

What happens next Test closed. The sticky "Choose your Nanobag" button is the default in the seven markets from 9 Sep; the tip text is being tested again inside NB09 (new hero banner + tip text, 9 to 17 Sep), which has its own report on this hub. Two things to carry forward: the pill's extra carts do not become orders, so a homepage button alone will not move revenue; and basket size needs a different lever than a line of text under the buy button.

Appendix · queries, IDs and links

Shopify report queries (ShopifyQL)

Paste into Analytics → Reports → New exploration → code editor. The filter must use the plural rollout_treatment_ids with CONTAINS; the grouping must use the singular rollout_treatment_id.

FROM sessions
SHOW sessions, sessions_with_cart_additions,
     sessions_that_reached_checkout, sessions_that_completed_checkout, conversion_rate
WHERE rollout_treatment_ids CONTAINS 16842888 OR rollout_treatment_ids CONTAINS 16875656
GROUP BY rollout_treatment_id
SINCE 2026-09-02 UNTIL 2026-09-09
ORDER BY sessions DESC
LIMIT 2

FROM sales
SHOW orders, net_sales, average_order_value
WHERE rollout_treatment_ids CONTAINS 16842888 OR rollout_treatment_ids CONTAINS 16875656
GROUP BY rollout_treatment_id
SINCE 2026-09-02 UNTIL 2026-09-09
ORDER BY orders DESC

FROM sales
SHOW orders, net_sales
WHERE rollout_treatment_ids CONTAINS 16842888 OR rollout_treatment_ids CONTAINS 16875656
GROUP BY rollout_treatment_id
TIMESERIES day
SINCE 2026-09-02 UNTIL 2026-09-09
ORDER BY day ASC

FROM sessions
SHOW sessions, sessions_that_completed_checkout, conversion_rate
WHERE rollout_treatment_ids CONTAINS 16842888 OR rollout_treatment_ids CONTAINS 16875656
GROUP BY rollout_treatment_id, session_country
SINCE 2026-09-02 UNTIL 2026-09-09
ORDER BY sessions DESC
LIMIT 14

FROM sessions
SHOW sessions, sessions_with_cart_additions, sessions_that_completed_checkout
WHERE rollout_treatment_ids CONTAINS 16842888 OR rollout_treatment_ids CONTAINS 16875656
GROUP BY rollout_treatment_id, utm_medium
SINCE 2026-09-02 UNTIL 2026-09-09
ORDER BY sessions DESC

GA4 exploration spec

Segments: Arm A and Arm B are session segments on the event parameter nb_arm (theme id + -A / -B). Test Market is a session segment on country.
Tabs 1 and 2: free-form, rows Device category / Country, segment comparison Arm A vs Arm B, metrics Sessions and E-commerce purchases.
Tab 3: segment Test Market, rows Event name, columns nb_arm, filters event name matches ^nb_(arm_view|home_cta_click|home_sticky_click)$, nb_arm contains the theme id, page path matches the homepage.
Tab 4: segment Test Market, rows Event name + atc_location, columns nb_arm, filters event name matches ^nb_(arm_view|atc_click)$, page path contains /products/.
Hero click tracking: GTM container WNJCZ75Z version 5 (published 4 Sep 21:24 HKT). Sticky pill and product-page events come from the theme.

IDs

Shopify Rollout 8421512 · control 16842888 (A) · treatment 16875656 (B). Started 2 Sep 00:00, paused 3 Sep 18:50, resumed 5 Sep 00:05, stopped 8 Sep about 22:00 HKT (stamped sessions fall from about 140 an hour to a trickle at 14:00 UTC). Same IDs throughout.
Orders are also tagged NB08-control / NB08-treatment and NB08-mobile / NB08-desktop by Flow.