-
Notifications
You must be signed in to change notification settings - Fork 45
Created new desktop-messaging tab under messaging/desktop for desktop… #830
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
2423599
2c8a243
0af793a
4fd5c2c
3ec7fa6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,238 @@ | ||||||
| --- | ||||||
| id: desktop-messaging | ||||||
| title: Desktop Messaging | ||||||
| slug: /messaging/desktop/desktop-messaging | ||||||
| --- | ||||||
|
|
||||||
| import Tabs from '@theme/Tabs'; | ||||||
| import TabItem from '@theme/TabItem'; | ||||||
|
|
||||||
| Step-by-step guide for creating and launching messaging system experiments. | ||||||
|
|
||||||
| ## Before You Start | ||||||
|
|
||||||
| Before beginning, familiarize yourself with the key messaging system concepts: | ||||||
|
|
||||||
| - **[Surfaces](/messaging/desktop/desktop-messaging-surfaces)** - Different message types (doorhangers, feature callouts, spotlights, etc.) | ||||||
| - **[Display Logic](/messaging/desktop/display-logic)** - Triggers, targeting, and frequency capping | ||||||
| - **[Configuration & Frequency](/messaging/desktop/message-configuration)** - Detailed configuration options | ||||||
|
|
||||||
| Also review the [Getting Started guide](/getting-started/for-engineers) and [Workflow overview](/workflow/overview) for experimenter fundamentals. | ||||||
|
|
||||||
| ### Helpful Tools & Resources | ||||||
|
|
||||||
| - **[Skylight](https://fxms-skylight.netlify.app/)** - View live messaging experiments, rollouts, and dashboards | ||||||
| - **[ASRouter Devtools](https://firefox-source-docs.mozilla.org/browser/components/asrouter/docs/debugging-docs.html)** - Preview and test message JSON | ||||||
| - **[Nimbus DevTools Guide](/resources/nimbus-devtools-guide)** - Advanced testing and debugging | ||||||
| - **[#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH)** - Slack channel for messaging system questions and reviews | ||||||
|
|
||||||
| ## Getting Started Workflow | ||||||
|
|
||||||
| When an experiment is kicked-off, an [experiment brief](/workflow/overview) is created by a Product Manager, which will detail the experiment's hypothesis, design, content, targeting, and audience sizing. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. replace "kicked-off" |
||||||
|
|
||||||
| As an engineer, your goal will be to create the experiment in Experimenter and the applicable JSON for each treatment branch that matches the brief's design, content and targeting. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
in this house we use the oxford comma :p |
||||||
|
|
||||||
| At a high level, the following is how to configure the experiment: | ||||||
|
|
||||||
| ### 1. Experiment Overview | ||||||
|
|
||||||
| - Navigate to [https://experimenter.services.mozilla.com/nimbus/](https://experimenter.services.mozilla.com/nimbus/) and select **Create New** | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The link is now "Create Experiment" |
||||||
| - Add the public name which is visible to users in `about:studies` (this will also create the unique experiment slug) | ||||||
| - Add the hypothesis as defined in the experiment brief | ||||||
| - Set the application (typically Firefox Desktop) | ||||||
| - Include links to the Experiment Brief and applicable Jira ticket | ||||||
|
|
||||||
| ### 2. Configure Branches | ||||||
|
|
||||||
| The number of treatment branches and their content will be defined in the experiment brief. Ensure that: | ||||||
|
|
||||||
| - The appropriate [feature configuration](/messaging/desktop/message-configuration) has been set | ||||||
| - The JSON defined here matches the [message schema](/messaging/desktop/message-configuration) | ||||||
| - For multi-branch experiments, the control branch includes a `message_id` and is not an empty object to ensure exposure event | ||||||
|
|
||||||
| :::warning | ||||||
|
|
||||||
| An empty object for control will never trigger or go through ASRouter and won't record an exposure event. As a result, the exposure analysis on Experimenter won't work correctly as there's no baseline to compare the treatments to. | ||||||
|
|
||||||
| ::: | ||||||
|
|
||||||
| Ask in [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) for review if you need clarification on multi-branch experiment setup. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Don't link to individual slack rooms from a public doc |
||||||
|
|
||||||
| ### 3. Select a Feature ID | ||||||
|
|
||||||
| #### Holdback Experiments & Rollouts | ||||||
|
|
||||||
| For rollouts and holdback experiments, you should pick up a placeholder Feature ID (`fxms-message`) that is not in use. See the [list of current messaging system feature IDs](https://experimenter.services.mozilla.com/nimbus/?status=Live#). You can verify what is in use by checking the Experimenter UI and filtering for the specific feature ID using the **All Features** dropdown on the left. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| #### Non-Holdback Experiments | ||||||
|
|
||||||
| When picking a feature ID for a non-holdback experiment, you should first prefer to run it on the [feature that matches the surface](/messaging/desktop/desktop-messaging-surfaces) that your experiment is using (`spotlight`, `featureCallout`, `infobar`, `cfr`, etc.). | ||||||
|
|
||||||
| If there are any other experiments using that feature ID that overlap in time with yours, you may still be able to use it, but ask in [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) to verify. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. again don't link to slack. |
||||||
|
|
||||||
| **Audience Overlap Considerations:** | ||||||
|
|
||||||
| In the case of experiment targeting overlaps, for each segment that overlaps, you must ensure that the sum of all audience sizes, including the audience size of your experiment is no more than 100%. For example, if an experiment exists that uses the same targeting segment as yours is enrolling at 30%, your experiment may use that same feature ID at up to 70%, for a combined 100% of the population of that targeting segment. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| If you are unable to use the feature that matches the surface of your message, reach out to [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) or a data scientist to discuss other options. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. again don't link to slack. |
||||||
|
|
||||||
| **Rollout Priority:** | ||||||
|
|
||||||
| Experiments take priority over rollouts, so one option is to share an ID with an existing rollout. If all IDs are taken by an experiment, it may still be usable depending on audience overlap. Check whether the targeting populations intersect. If your experiment targets Germany and the existing one targets Canada, they don't share users and can safely reuse the same ID. | ||||||
|
|
||||||
| The key is that the 'effective audience' (after all targeting criteria) doesn't overlap, not just that the IDs are "free." This also includes things like the channel, locales, and the enrollment %. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| ## Testing the Experiment | ||||||
|
|
||||||
| When creating the JSON for treatment branches, you can verify the changes by updating the experiment to Preview Mode and by selecting **Launch to Preview**. This will provide a link that will force enroll the user into the experiment. Alternatively, you can copy paste the JSON into `about:asrouter` to preview it. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Coming soon: enrolling via Nimbus devtools integrations into Experimenter :) But don't update your doc with this. |
||||||
|
|
||||||
| ### Previewing in `about:asrouter` | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also coming soon, previewing messages with nimbus-devtools directly from experimenter :) |
||||||
|
|
||||||
| `about:asrouter` (the [ASRouter Devtools](https://firefox-source-docs.mozilla.org/browser/components/asrouter/docs/debugging-docs.html)) is the fastest way to iterate on a message's UI without enrolling. To enable it, set `browser.newtabpage.activity-stream.asrouter.devtoolsEnabled` to `true` in `about:config`, then open `about:asrouter`. | ||||||
|
|
||||||
| Paste your message JSON into the editor to render it immediately and check the layout, copy, and buttons. You can also inspect the messages currently available to the client and the provider they came from, and reset a message's impressions and frequency-cap history (or block and unblock it) so you can re-test as if you had never seen it. | ||||||
|
|
||||||
| ### Force Enrollment Testing | ||||||
|
|
||||||
| To test the experiment using forced enrollment: | ||||||
|
|
||||||
| 1. In `about:config`, set `nimbus.debug` to `true` | ||||||
| 2. Go to Experimenter and select the branch of the experiment you want to enroll yourself within | ||||||
| 3. Copy the `about:studies?...` enrollment link for that branch and paste it into the address bar | ||||||
| 4. You should now be opted into the experiment. Verify by checking `about:studies` | ||||||
| 5. Trigger the surface your message uses to confirm the branch is live (e.g. open `about:welcome` for an onboarding/spotlight message, or perform the relevant [trigger](/messaging/desktop/display-logic) for an infobar, CFR, or feature callout) | ||||||
| 6. To unenroll, go to `about:studies` and remove yourself from the study | ||||||
| 7. To enroll into another branch, repeat steps 2-4 and re-trigger the surface | ||||||
|
|
||||||
| :::tip | ||||||
|
|
||||||
| The [Nimbus Developer Tools](/resources/nimbus-devtools-guide) provide a faster way to force enroll — you can paste the experiment's `Recipe JSON` (or a feature configuration) directly instead of using the `about:studies` link. | ||||||
|
|
||||||
| ::: | ||||||
|
|
||||||
| :::info | ||||||
|
|
||||||
| Using `about:asrouter` is helpful for quickly previewing and configuring the UI elements of the message, but it will **not** preview the targeting and triggering. | ||||||
|
|
||||||
| Force enrolling through Experimenter will preview the message-level targeting and triggering but **not** the advanced targeting. To test the full behavior, see the section on [testing natural enrollment](#natural-enrollment-testing). | ||||||
|
|
||||||
| ::: | ||||||
|
|
||||||
| ### Natural Enrollment Testing | ||||||
|
|
||||||
| To test experiment with natural enrollment on first startup: | ||||||
|
|
||||||
| #### 1. Generate a Normandy ID | ||||||
|
|
||||||
| We first need to generate a normandy ID, which will later be passed into the `user.js` file. In `about:config` set `devtools.chrome.enabled` pref to `true`. | ||||||
|
|
||||||
| In the browser console, paste the following (replacing `{YOUR_EXPERIMENT_SLUG}` with your experiment slug): | ||||||
|
|
||||||
| ```javascript | ||||||
| await ChromeUtils.importESModule("resource://nimbus/lib/ExperimentManager.sys.mjs").ExperimentManager.generateTestIds((await (await fetch("https://firefox.settings.services.mozilla.com/v1/buckets/main-preview/collections/nimbuspreview/records/{YOUR_EXPERIMENT_SLUG}")).json()).data) | ||||||
| ``` | ||||||
|
Comment on lines
+131
to
+133
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please update this to reference Nimbus DevTools |
||||||
|
|
||||||
| This will return an object with the respective branch IDs, which can then be populated in your `user.js` file. | ||||||
|
|
||||||
| #### 2. Update the user.js File | ||||||
|
|
||||||
| - Open browser profile (`about:profiles`) | ||||||
| - Create a new profile and create a `user.js` file in browser's profile folder (`../Firefox/Profiles`) | ||||||
| - Fill it with the normandy ID generated above: | ||||||
|
|
||||||
| ```javascript title="user.js" | ||||||
| /* This Source Code Form is subject to the terms of the Mozilla Public | ||||||
| * License, v. 2.0. If a copy of the MPL was not distributed with this | ||||||
| * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think this is copyrightable FWIW |
||||||
|
|
||||||
| // Common preferences file used by both unittest and perf harnesses. | ||||||
| /* globals user_pref */ | ||||||
| user_pref( | ||||||
| "messaging-system.rsexperimentloader.collection_id", | ||||||
| "nimbus-preview" | ||||||
| ); | ||||||
| user_pref("app.normandy.user_id", "${GENERATED_BRANCH_NORMANY_ID_GOES_HERE}"); | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. normandy ID is no longer used for bucketing. profile group IDs are and those are trickier to fake but the pref is "toolkit.telemetry.cachedProfileGroupID" |
||||||
| user_pref("messaging-system.log", "all"); | ||||||
| user_pref("browser.ping-centre.log", true); | ||||||
| ``` | ||||||
|
|
||||||
| :::tip | ||||||
|
|
||||||
| Use "Show Finder" (for root directory) to get quick access to the folder. Also save the path to the profile for the next step. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
Show in Finder is only on mac. On windows its "Open Folder" and I'm not sure about linux. |
||||||
|
|
||||||
| ::: | ||||||
|
|
||||||
| #### 3. Launch Firefox with the Profile | ||||||
|
|
||||||
| Run the appropriate Firefox build that matches the preview targeting (i.e. Nightly, Release, Beta) and add the profile with first-startup syntax: | ||||||
|
|
||||||
| <Tabs> | ||||||
| <TabItem value="mac" label="macOS"> | ||||||
|
|
||||||
| ```bash | ||||||
| /Applications/Firefox/Contents/MacOS/firefox --first-startup --profile "/Users/{username}/Library/Application Support/Firefox/Profiles/{profileName}" | ||||||
| ``` | ||||||
|
|
||||||
| </TabItem> | ||||||
| <TabItem value="windows" label="Windows"> | ||||||
|
|
||||||
| ```bash | ||||||
| "C:\Program Files\Mozilla Firefox\firefox.exe" --first-startup --profile "C:\Users\{username}\AppData\Roaming\Mozilla\Firefox\Profiles\{profileName}" | ||||||
|
Comment on lines
+179
to
+180
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is not bash. |
||||||
| ``` | ||||||
|
|
||||||
| </TabItem> | ||||||
| </Tabs> | ||||||
|
|
||||||
| #### 4. Verify Enrollment | ||||||
|
|
||||||
| Check `about:studies` page and `about:telemetry#events` page to verify natural enrollment. | ||||||
|
|
||||||
| ### Testing Telemetry | ||||||
|
|
||||||
| To test the experiment's telemetry pings, this should be done regularly during development to ensure no breaking changes were introduced and telemetry is working as intended: | ||||||
|
|
||||||
| ```bash | ||||||
| ./mach run --temp-profile --setpref "browser.ping-centre.log=true" --setpref "browser.newtabpage.activity-stream.telemetry=true" --setpref "nimbus.debug=true" | ||||||
| ``` | ||||||
|
|
||||||
| You can verify telemetry when running the browser console (**Tools > Web Developer > Browser Toolbox**). | ||||||
|
|
||||||
| ## Launching & Post-Launch | ||||||
|
|
||||||
| Once the experiment has been created and is in preview mode, ensure that the experiment has been tested locally through forced and/or natural enrollment from the above steps. | ||||||
|
|
||||||
| ### QA Testing | ||||||
|
|
||||||
| Create a QA ticket in the Jira experiment FXE ticket (**Automation > Create QA when in Preview Mode**) and link the experiment to the generated QA ticket. | ||||||
|
|
||||||
| ### Launch Request | ||||||
|
|
||||||
| When the experiment has received green from QA: | ||||||
|
|
||||||
| 1. Select **"Request Launch"** in the experiment | ||||||
| 2. Copy the link into the [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) Slack channel for review | ||||||
| 3. Someone on the team will approve or request changes in the experiment | ||||||
|
|
||||||
| :::info | ||||||
|
|
||||||
| Whenever you are using messaging system, please leverage [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) for an experiment reviewer/launch request rather than [#ask-experimenter](https://mozilla.enterprise.slack.com/archives/CF94YGE03). | ||||||
|
|
||||||
| ::: | ||||||
|
|
||||||
| ### Post-Launch Monitoring | ||||||
|
|
||||||
| After the experiment launches: | ||||||
|
|
||||||
| 1. Monitor the experiment Looker dashboard (found on the left-hand side of the experiment under **Links** > **"Live Monitoring Dashboard"**) | ||||||
| 2. Set reminders to check enrollment trends and anomalies. You can use: `/remind me to check experiment enrolments everyday at 1pm` | ||||||
| 3. If enrollment is trending towards the audience sizing, on the enrollment end date, manually end enrollment by requesting to end enrollment and linking the experiment into the [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) Slack channel for review | ||||||
| 4. If enrollment is not trending towards the sizing, flag this with data science in the experiment Slack channel | ||||||
| 5. The experiment must also be manually ended once the observation period has concluded, following the same procedure as requesting launch and enrollment end | ||||||
|
|
||||||
| ## Additional Resources | ||||||
|
|
||||||
| - [Messaging System Documentation](/messaging/overview) | ||||||
| - [Experimenter Documentation](https://experimenter.info) | ||||||
| - [Feature Monitoring](/feature-monitoring) | ||||||
| - [Testing Guide](/workflow/testing) | ||||||
| - [Nimbus DevTools Guide](/resources/nimbus-devtools-guide) | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,6 +23,23 @@ The message will display when the user navigates to a URL that matches the provi | |
|
|
||
| The message will display when a preference has changed. Note that this does *not* trigger if the preference was already set to the desired value at startup. | ||
|
|
||
| ### `defaultBrowserCheck` | ||
|
|
||
| The message will display when Firefox checks whether it is the default browser. This check runs both at startup and when a new tab is opened, so the trigger supplies a `source` value to tell the two apart: | ||
|
|
||
| * `source == 'startup'` — the check ran during browser startup | ||
| * `source == 'newtab'` — the check ran because the user opened a new tab | ||
|
|
||
| A common pattern is to gate a default-browser message to new tabs only by combining the trigger with `source == 'newtab'` in your targeting. Pairing `defaultBrowserCheck` with `source == 'newtab'` is appropriate when you want the message to appear on the new tab page; use the new-tab feature-callout trigger (below) instead when you are rendering a feature callout anchored to new-tab UI. See the [complete trigger list](https://firefox-source-docs.mozilla.org/toolkit/components/messaging-system/docs/TriggerActionSchemas/index.html#available-trigger-actions) in Firefox source docs for the exact trigger names and the `source` values each one provides. | ||
|
|
||
| :::warning | ||
|
|
||
| You may notice `source == 'newtab'` appears to "fail" in the JEXL debugger. This is because `source` is part of the **trigger context** — it only exists when the trigger actually fires. The standalone JEXL debuggers (`about:asrouter` and Nimbus DevTools) evaluate targeting against your **client targeting context**, which does *not* include trigger-supplied variables like `source`. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: |
||
|
|
||
| As a result, an expression referencing `source` evaluates against an unknown variable and returns an [empty result](/platform-guides/desktop/targeting-debug#targeting) (neither `true` nor `false`) in the debugger — even though it will work correctly at runtime when the trigger fires. This is expected: targeting that depends on trigger context must be verified through [force or natural enrollment](/messaging/desktop/desktop-messaging#testing-the-experiment), not the debugger. | ||
|
|
||
| ::: | ||
|
|
||
|
|
||
| ## Targeting | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -18,7 +18,7 @@ Once an idea has been developed, the next step is designing the experiment messa | |
| ### Running an Experiment | ||
|
|
||
| First, determine if your experiment will require any on-train development work to support the launch as well as any translated strings that need to land prior. Once that's been determined, we can get started with creating the experiment. | ||
| For a more in-depth guide and step by step process, visit [OMC: Experimenter onboarding document](https://mozilla-hub.atlassian.net/wiki/spaces/FIREFOX/pages/233406786/OMC+Experimenter+Onboarding) | ||
| For a more in-depth guide and step by step process, visit the [Desktop Messaging Onboarding](/messaging/desktop/desktop-messaging) guide. | ||
|
|
||
| ### Message in Firefox | ||
|
|
||
|
|
@@ -107,6 +107,14 @@ It is also possible to specifically include or exclude users into experiments ba | |
| Note that when using targeting or groups in this way the message **impressions** are removed once the message has been removed or the experiment has ended. | ||
| ::: | ||
|
|
||
| :::info | ||
|
|
||
| When re-running an experience, frequency caps and [blocks](/messaging/desktop/display-logic#blocking) are keyed on `message_id`. If you re-run an experiment (or convert it to a rollout) with a **new** `message_id`, users who already saw the message have no impression or block history under the new ID and may be shown it again, leading to duplicate messaging and message fatigue. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. wording: experience? "When re-launching a message" ? Not sure |
||
|
|
||
| To avoid this, reuse the same `message_id` so history carries over, and end the original experiment or rollout before launching the re-run (or ensure the audiences don't overlap). Note that because impressions are cleared once the original ends, a gap between the two can still re-expose users even with the same `message_id`. | ||
|
|
||
| ::: | ||
|
|
||
| ### No Action to Block Message | ||
|
|
||
| For the doorhanger template we had a built-in (into the message surface) button to block a message. For all other surfaces there is no way for the user to block a message from ever being shown. Usually campaigns have had a maximum of 1-2 impressions and it was not considered to add a block action. The benefit of blocking would be that we could group similar messages as part of a **"campaign"** (for example Mozilla VPN) and use the block signal as an indicator not to show future messages with similar content. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nimbus-devtools