diff --git a/docs/getting-started/contributing.md b/docs/getting-started/contributing.md index e62e44d0b..59d5953ca 100644 --- a/docs/getting-started/contributing.md +++ b/docs/getting-started/contributing.md @@ -4,7 +4,7 @@ title: For Contributors slug: /getting-started/for-contributors --- -The Experimenter documentation hub is managed in the [mozilla/experimenter-docs](https://github.com/mozilla/experimenter-docs) repository. You will need a GitHub account to contribute, and if you are not already in the [Project Nimbus GitHub team](https://github.com/orgs/mozilla/teams/project-nimbus/members), you may need to request write access in the #nimbus-project Slack channel. +The Experimenter documentation hub is managed in the [mozilla/experimenter-docs](https://github.com/mozilla/experimenter-docs) repository. You will need a GitHub account to contribute, and if you are not already in the [Project Nimbus GitHub team](https://github.com/orgs/mozilla/teams/project-nimbus/members), you may need to request write access in the #nimbus-dev Slack channel. :::info diff --git a/docs/messaging/desktop/desktop-messaging.md b/docs/messaging/desktop/desktop-messaging.md new file mode 100644 index 000000000..cb7e020b1 --- /dev/null +++ b/docs/messaging/desktop/desktop-messaging.md @@ -0,0 +1,238 @@ +--- +id: desktop-messaging +title: Desktop Messaging +slug: /messaging/desktop/desktop-messaging +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Step-by-step guide for creating and launching messaging system experiments. + +## Before You Start + +Before beginning, familiarize yourself with the key messaging system concepts: + +- **[Surfaces](/messaging/desktop/desktop-messaging-surfaces)** - Different message types (doorhangers, feature callouts, spotlights, etc.) +- **[Display Logic](/messaging/desktop/display-logic)** - Triggers, targeting, and frequency capping +- **[Configuration & Frequency](/messaging/desktop/message-configuration)** - Detailed configuration options + +Also review the [Getting Started guide](/getting-started/for-engineers) and [Workflow overview](/workflow/overview) for experimenter fundamentals. + +### Helpful Tools & Resources + +- **[Skylight](https://fxms-skylight.netlify.app/)** - View live messaging experiments, rollouts, and dashboards +- **[ASRouter Devtools](https://firefox-source-docs.mozilla.org/browser/components/asrouter/docs/debugging-docs.html)** - Preview and test message JSON +- **[Nimbus DevTools Guide](/resources/nimbus-devtools-guide)** - Advanced testing and debugging +- **[#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH)** - Slack channel for messaging system questions and reviews + +## Getting Started Workflow + +When an experiment is kicked-off, an [experiment brief](/workflow/overview) is created by a Product Manager, which will detail the experiment's hypothesis, design, content, targeting, and audience sizing. + +As an engineer, your goal will be to create the experiment in Experimenter and the applicable JSON for each treatment branch that matches the brief's design, content and targeting. + +At a high level, the following is how to configure the experiment: + +### 1. Experiment Overview + +- Navigate to [https://experimenter.services.mozilla.com/nimbus/](https://experimenter.services.mozilla.com/nimbus/) and select **Create New** +- Add the public name which is visible to users in `about:studies` (this will also create the unique experiment slug) +- Add the hypothesis as defined in the experiment brief +- Set the application (typically Firefox Desktop) +- Include links to the Experiment Brief and applicable Jira ticket + +### 2. Configure Branches + +The number of treatment branches and their content will be defined in the experiment brief. Ensure that: + +- The appropriate [feature configuration](/messaging/desktop/message-configuration) has been set +- The JSON defined here matches the [message schema](/messaging/desktop/message-configuration) +- For multi-branch experiments, the control branch includes a `message_id` and is not an empty object to ensure exposure event + +:::warning + +An empty object for control will never trigger or go through ASRouter and won't record an exposure event. As a result, the exposure analysis on Experimenter won't work correctly as there's no baseline to compare the treatments to. + +::: + +Ask in [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) for review if you need clarification on multi-branch experiment setup. + +### 3. Select a Feature ID + +#### Holdback Experiments & Rollouts + +For rollouts and holdback experiments, you should pick up a placeholder Feature ID (`fxms-message`) that is not in use. See the [list of current messaging system feature IDs](https://experimenter.services.mozilla.com/nimbus/?status=Live#). You can verify what is in use by checking the Experimenter UI and filtering for the specific feature ID using the **All Features** dropdown on the left. + +#### Non-Holdback Experiments + +When picking a feature ID for a non-holdback experiment, you should first prefer to run it on the [feature that matches the surface](/messaging/desktop/desktop-messaging-surfaces) that your experiment is using (`spotlight`, `featureCallout`, `infobar`, `cfr`, etc.). + +If there are any other experiments using that feature ID that overlap in time with yours, you may still be able to use it, but ask in [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) to verify. + +**Audience Overlap Considerations:** + +In the case of experiment targeting overlaps, for each segment that overlaps, you must ensure that the sum of all audience sizes, including the audience size of your experiment is no more than 100%. For example, if an experiment exists that uses the same targeting segment as yours is enrolling at 30%, your experiment may use that same feature ID at up to 70%, for a combined 100% of the population of that targeting segment. + +If you are unable to use the feature that matches the surface of your message, reach out to [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) or a data scientist to discuss other options. + +**Rollout Priority:** + +Experiments take priority over rollouts, so one option is to share an ID with an existing rollout. If all IDs are taken by an experiment, it may still be usable depending on audience overlap. Check whether the targeting populations intersect. If your experiment targets Germany and the existing one targets Canada, they don't share users and can safely reuse the same ID. + +The key is that the 'effective audience' (after all targeting criteria) doesn't overlap, not just that the IDs are "free." This also includes things like the channel, locales, and the enrollment %. + +## Testing the Experiment + +When creating the JSON for treatment branches, you can verify the changes by updating the experiment to Preview Mode and by selecting **Launch to Preview**. This will provide a link that will force enroll the user into the experiment. Alternatively, you can copy paste the JSON into `about:asrouter` to preview it. + +### Previewing in `about:asrouter` + +`about:asrouter` (the [ASRouter Devtools](https://firefox-source-docs.mozilla.org/browser/components/asrouter/docs/debugging-docs.html)) is the fastest way to iterate on a message's UI without enrolling. To enable it, set `browser.newtabpage.activity-stream.asrouter.devtoolsEnabled` to `true` in `about:config`, then open `about:asrouter`. + +Paste your message JSON into the editor to render it immediately and check the layout, copy, and buttons. You can also inspect the messages currently available to the client and the provider they came from, and reset a message's impressions and frequency-cap history (or block and unblock it) so you can re-test as if you had never seen it. + +### Force Enrollment Testing + +To test the experiment using forced enrollment: + +1. In `about:config`, set `nimbus.debug` to `true` +2. Go to Experimenter and select the branch of the experiment you want to enroll yourself within +3. Copy the `about:studies?...` enrollment link for that branch and paste it into the address bar +4. You should now be opted into the experiment. Verify by checking `about:studies` +5. Trigger the surface your message uses to confirm the branch is live (e.g. open `about:welcome` for an onboarding/spotlight message, or perform the relevant [trigger](/messaging/desktop/display-logic) for an infobar, CFR, or feature callout) +6. To unenroll, go to `about:studies` and remove yourself from the study +7. To enroll into another branch, repeat steps 2-4 and re-trigger the surface + +:::tip + +The [Nimbus Developer Tools](/resources/nimbus-devtools-guide) provide a faster way to force enroll — you can paste the experiment's `Recipe JSON` (or a feature configuration) directly instead of using the `about:studies` link. + +::: + +:::info + +Using `about:asrouter` is helpful for quickly previewing and configuring the UI elements of the message, but it will **not** preview the targeting and triggering. + +Force enrolling through Experimenter will preview the message-level targeting and triggering but **not** the advanced targeting. To test the full behavior, see the section on [testing natural enrollment](#natural-enrollment-testing). + +::: + +### Natural Enrollment Testing + +To test experiment with natural enrollment on first startup: + +#### 1. Generate a Normandy ID + +We first need to generate a normandy ID, which will later be passed into the `user.js` file. In `about:config` set `devtools.chrome.enabled` pref to `true`. + +In the browser console, paste the following (replacing `{YOUR_EXPERIMENT_SLUG}` with your experiment slug): + +```javascript +await ChromeUtils.importESModule("resource://nimbus/lib/ExperimentManager.sys.mjs").ExperimentManager.generateTestIds((await (await fetch("https://firefox.settings.services.mozilla.com/v1/buckets/main-preview/collections/nimbuspreview/records/{YOUR_EXPERIMENT_SLUG}")).json()).data) +``` + +This will return an object with the respective branch IDs, which can then be populated in your `user.js` file. + +#### 2. Update the user.js File + +- Open browser profile (`about:profiles`) +- Create a new profile and create a `user.js` file in browser's profile folder (`../Firefox/Profiles`) +- Fill it with the normandy ID generated above: + +```javascript title="user.js" +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ + +// Common preferences file used by both unittest and perf harnesses. +/* globals user_pref */ +user_pref( + "messaging-system.rsexperimentloader.collection_id", + "nimbus-preview" +); +user_pref("app.normandy.user_id", "${GENERATED_BRANCH_NORMANY_ID_GOES_HERE}"); +user_pref("messaging-system.log", "all"); +user_pref("browser.ping-centre.log", true); +``` + +:::tip + +Use "Show Finder" (for root directory) to get quick access to the folder. Also save the path to the profile for the next step. + +::: + +#### 3. Launch Firefox with the Profile + +Run the appropriate Firefox build that matches the preview targeting (i.e. Nightly, Release, Beta) and add the profile with first-startup syntax: + + + + +```bash +/Applications/Firefox/Contents/MacOS/firefox --first-startup --profile "/Users/{username}/Library/Application Support/Firefox/Profiles/{profileName}" +``` + + + + +```bash +"C:\Program Files\Mozilla Firefox\firefox.exe" --first-startup --profile "C:\Users\{username}\AppData\Roaming\Mozilla\Firefox\Profiles\{profileName}" +``` + + + + +#### 4. Verify Enrollment + +Check `about:studies` page and `about:telemetry#events` page to verify natural enrollment. + +### Testing Telemetry + +To test the experiment's telemetry pings, this should be done regularly during development to ensure no breaking changes were introduced and telemetry is working as intended: + +```bash +./mach run --temp-profile --setpref "browser.ping-centre.log=true" --setpref "browser.newtabpage.activity-stream.telemetry=true" --setpref "nimbus.debug=true" +``` + +You can verify telemetry when running the browser console (**Tools > Web Developer > Browser Toolbox**). + +## Launching & Post-Launch + +Once the experiment has been created and is in preview mode, ensure that the experiment has been tested locally through forced and/or natural enrollment from the above steps. + +### QA Testing + +Create a QA ticket in the Jira experiment FXE ticket (**Automation > Create QA when in Preview Mode**) and link the experiment to the generated QA ticket. + +### Launch Request + +When the experiment has received green from QA: + +1. Select **"Request Launch"** in the experiment +2. Copy the link into the [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) Slack channel for review +3. Someone on the team will approve or request changes in the experiment + +:::info + +Whenever you are using messaging system, please leverage [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) for an experiment reviewer/launch request rather than [#ask-experimenter](https://mozilla.enterprise.slack.com/archives/CF94YGE03). + +::: + +### Post-Launch Monitoring + +After the experiment launches: + +1. Monitor the experiment Looker dashboard (found on the left-hand side of the experiment under **Links** > **"Live Monitoring Dashboard"**) +2. Set reminders to check enrollment trends and anomalies. You can use: `/remind me to check experiment enrolments everyday at 1pm` +3. If enrollment is trending towards the audience sizing, on the enrollment end date, manually end enrollment by requesting to end enrollment and linking the experiment into the [#omc](https://mozilla.enterprise.slack.com/archives/C90HG2UQH) Slack channel for review +4. If enrollment is not trending towards the sizing, flag this with data science in the experiment Slack channel +5. The experiment must also be manually ended once the observation period has concluded, following the same procedure as requesting launch and enrollment end + +## Additional Resources + +- [Messaging System Documentation](/messaging/overview) +- [Experimenter Documentation](https://experimenter.info) +- [Feature Monitoring](/feature-monitoring) +- [Testing Guide](/workflow/testing) +- [Nimbus DevTools Guide](/resources/nimbus-devtools-guide) diff --git a/docs/messaging/desktop/display-logic.md b/docs/messaging/desktop/display-logic.md index 8d885b64d..8bd9aea15 100644 --- a/docs/messaging/desktop/display-logic.md +++ b/docs/messaging/desktop/display-logic.md @@ -23,6 +23,23 @@ The message will display when the user navigates to a URL that matches the provi The message will display when a preference has changed. Note that this does *not* trigger if the preference was already set to the desired value at startup. +### `defaultBrowserCheck` + +The message will display when Firefox checks whether it is the default browser. This check runs both at startup and when a new tab is opened, so the trigger supplies a `source` value to tell the two apart: + +* `source == 'startup'` — the check ran during browser startup +* `source == 'newtab'` — the check ran because the user opened a new tab + +A common pattern is to gate a default-browser message to new tabs only by combining the trigger with `source == 'newtab'` in your targeting. Pairing `defaultBrowserCheck` with `source == 'newtab'` is appropriate when you want the message to appear on the new tab page; use the new-tab feature-callout trigger (below) instead when you are rendering a feature callout anchored to new-tab UI. See the [complete trigger list](https://firefox-source-docs.mozilla.org/toolkit/components/messaging-system/docs/TriggerActionSchemas/index.html#available-trigger-actions) in Firefox source docs for the exact trigger names and the `source` values each one provides. + +:::warning + +You may notice `source == 'newtab'` appears to "fail" in the JEXL debugger. This is because `source` is part of the **trigger context** — it only exists when the trigger actually fires. The standalone JEXL debuggers (`about:asrouter` and Nimbus DevTools) evaluate targeting against your **client targeting context**, which does *not* include trigger-supplied variables like `source`. + +As a result, an expression referencing `source` evaluates against an unknown variable and returns an [empty result](/platform-guides/desktop/targeting-debug#targeting) (neither `true` nor `false`) in the debugger — even though it will work correctly at runtime when the trigger fires. This is expected: targeting that depends on trigger context must be verified through [force or natural enrollment](/messaging/desktop/desktop-messaging#testing-the-experiment), not the debugger. + +::: + ## Targeting diff --git a/docs/messaging/desktop/message-lifecycle.md b/docs/messaging/desktop/message-lifecycle.md index 01271e527..4a690a9bb 100644 --- a/docs/messaging/desktop/message-lifecycle.md +++ b/docs/messaging/desktop/message-lifecycle.md @@ -18,7 +18,7 @@ Once an idea has been developed, the next step is designing the experiment messa ### Running an Experiment First, determine if your experiment will require any on-train development work to support the launch as well as any translated strings that need to land prior. Once that's been determined, we can get started with creating the experiment. -For a more in-depth guide and step by step process, visit [OMC: Experimenter onboarding document](https://mozilla-hub.atlassian.net/wiki/spaces/FIREFOX/pages/233406786/OMC+Experimenter+Onboarding) +For a more in-depth guide and step by step process, visit the [Desktop Messaging Onboarding](/messaging/desktop/desktop-messaging) guide. ### Message in Firefox @@ -107,6 +107,14 @@ It is also possible to specifically include or exclude users into experiments ba Note that when using targeting or groups in this way the message **impressions** are removed once the message has been removed or the experiment has ended. ::: +:::info + +When re-running an experience, frequency caps and [blocks](/messaging/desktop/display-logic#blocking) are keyed on `message_id`. If you re-run an experiment (or convert it to a rollout) with a **new** `message_id`, users who already saw the message have no impression or block history under the new ID and may be shown it again, leading to duplicate messaging and message fatigue. + +To avoid this, reuse the same `message_id` so history carries over, and end the original experiment or rollout before launching the re-run (or ensure the audiences don't overlap). Note that because impressions are cleared once the original ends, a gap between the two can still re-expose users even with the same `message_id`. + +::: + ### No Action to Block Message For the doorhanger template we had a built-in (into the message surface) button to block a message. For all other surfaces there is no way for the user to block a message from ever being shown. Usually campaigns have had a maximum of 1-2 impressions and it was not considered to add a block action. The benefit of blocking would be that we could group similar messages as part of a **"campaign"** (for example Mozilla VPN) and use the block signal as an indicator not to show future messages with similar content. diff --git a/docs/workflow/ending.md b/docs/workflow/ending.md index 95485eb29..8ecf977f1 100644 --- a/docs/workflow/ending.md +++ b/docs/workflow/ending.md @@ -41,3 +41,9 @@ The process to end the experiment is similar to that of ending enrollment: press Some notes: * Automated analyses are based on the actual observation period (period between enrollment and experiment ending), not the planned period. So experiments that remain open past their scheduled end date will have more days worth of data included in their results. + +:::info + +For messaging system experiments, ending an experiment or rollout **clears the message impression history** tied to it. If you plan to re-run the experience (for example, converting an experiment into a rollout), reuse the same `message_id` and avoid a gap between ending the original and launching the re-run — otherwise users who already saw the message may see it again. See [Re-running an experience](/messaging/desktop/message-lifecycle#history-state-of-past-experiments) for details. + +::: diff --git a/sidebars.js b/sidebars.js index 58dfec927..44c5a6f6b 100644 --- a/sidebars.js +++ b/sidebars.js @@ -233,6 +233,7 @@ module.exports = { label: "Messaging System", items: [ "messaging/experiments-and-user-messaging", + "messaging/desktop/desktop-messaging", "messaging/desktop/messaging-surfaces", "messaging/desktop/display-logic", "messaging/desktop/message-configuration",