[Feature] Extend personal pattern discovery across behavioral, treatment, and environmental factors

[lutzkind]

Summary

Extend HealthLog’s personal correlation-discovery system so it can analyze timed and quantified contextual factors—including medication events, lifestyle events, illness, symptoms, workouts, sleep timing, travel, cycle context, and environmental exposure—and surface only evidence-backed, confidence-gated explanations through Coach and Insights.

This request combines the analytical engine and its Coach presentation because the evidence contract, confidence thresholds, causal wording, and source traceability must remain consistent from computation to narration.

Problem

HealthLog already contains many potentially relevant contributors and outcomes, but they are not analyzed through one general contextual-pattern framework.

Examples include:

• Medication intake, missed doses, schedule changes, and dose changes.
• Alcohol, caffeine, nicotine, meals, hydration, stress, and sleep aids.
• Workouts and training load.
• Illness episodes and symptom severity.
• Mood tags and rated factors.
• Sleep timing and naps.
• Menstrual-cycle context when enabled.
• Travel and timezone changes.
• Temperature, daylight, pollution, pollen, pressure, heat, and smoke.

The existing correlation infrastructure is a strong foundation: bounded channel sets, lagging, minimum paired-day requirements, statistical testing, false-discovery correction, effect-size floors, shrinkage, confidence tiers, and non-causal framing.

However, many meaningful questions require more than one daily yes/no channel:

• Does alcohol quantity or time of last drink relate to nocturnal heart rate, HRV, or sleep stages?
• Does late caffeine differ from morning caffeine?
• Are symptoms more common after missed medication doses or after a dose revision?
• Does outdoor exercise during high heat or poor air quality relate to recovery differently from indoor exercise?
• Do late meals, travel, illness, or weekends explain an apparent sleep pattern?
• Are several weak factors repeatedly present together before the same outcome change?

Goals

1. Consume existing and new contextual contributors through a common, source-aware interface.
2. Support timing, duration, quantity, and dose where relevant.
3. Match factor-specific exposure windows to plausible outcome windows.
4. Preserve HealthLog’s current statistical safeguards and strengthen them for a larger hypothesis space.
5. Produce an explicit evidence object that Insights and Coach must follow.
6. Surface proactive explanations only when evidence and user preferences justify them.
7. Never convert association into diagnosis or causal certainty.

Input architecture

The engine should consume authoritative records through adapters rather than copying them into a second analytics ledger.

Potential inputs:

• Medication intake/adherence, missed-dose, pause, schedule-revision, and dose-change records.
• Workout sessions, activity, strain, and recovery data.
• Illness episodes and symptom logs.
• Mood entries, structured tags, and rated factors.
• Sleep sessions, stage composition, naps, bedtime, and wake time.
• Cycle events when enabled.
• Travel and timezone context.
• Environmental context.
• Structured lifestyle events proposed in [Feature] Add a unified contextual-factor timeline and structured lifestyle-event logging #613.
• Long-term contextual modifiers from [Feature] Expand the encrypted health profile with a structured health and lifestyle questionnaire #614 where relevant.
• Expanded environmental channels proposed in [Feature] Expand environmental context with air quality, heat, UV, pollen, and smoke exposure #615.

Each input should preserve:

• Factor key/category.
• Timestamp, duration, and timezone.
• Quantity and unit where available.
• Source and authoritative-record link.
• Confirmed versus inferred status.
• Confidence and data-quality metadata.

Time relationships

A single fixed D → D+1 lag is insufficient for all factors. Support a curated set of factor-specific windows, for example:

Same window or same day

• Workout intensity versus exercise heart rate or symptoms.
• Heat/pollution during an outdoor workout versus workout performance.
• Medication intake versus near-term symptom response where clinically appropriate and user configured.

Same night

• Alcohol, caffeine, nicotine, late meals, stress, exercise timing, bedroom/environmental context versus:
  • Sleep duration.
  • Sleep stages.
  • Awakenings or fragmentation.
  • Nocturnal/resting heart rate.
  • HRV.
  • Respiratory rate.
  • Oxygen saturation or breathing disturbances.

Next morning or next day

• Prior-evening factors versus readiness, recovery, mood, resting pulse, HRV, activity, or symptoms.
• Missed medication or adherence changes versus subsequent symptoms or vital drift.
• Daylight, activity, and stress versus later sleep timing or quality.

Multi-day or cumulative windows

• Consecutive poor-air-quality or high-heat days.
• Training load over several days versus recovery.
• Medication changes versus gradual symptom or vital trends.
• Repeated short sleep versus mood and activity.

Windows should be curated by factor family rather than allowing an unrestricted combinatorial scan over arbitrary lags.

Quantity, timing, and dose-response

Where data permits, analyze more than factor present/absent:

• Alcohol standard drinks or grams of ethanol.
• Caffeine milligrams and hours before sleep.
• Nicotine quantity and timing.
• Meal timing and approximate size.
• Medication dose and dose changes.
• Workout duration/intensity/load.
• Heat hours, pollutant concentration, or pollen level.
• Sleep opportunity and bedtime shift.

Potential methods:

• Bounded bins with minimum support.
• Continuous association when assumptions and sample size are sufficient.
• Monotonic trend or simple regression with robust checks.
• With/without comparison as a fallback when quantity is too sparse.

Do not infer a dose-response from only a few events.

Confounder-aware comparisons

Passive personal data is observational. The engine should attempt to reduce obvious confounding where sufficient data exists.

Potential matching or adjustment variables:

• Weekday versus weekend.
• Travel/timezone change.
• Illness episode.
• Exercise load.
• Sleep opportunity and bedtime.
• Medication changes.
• Temperature when evaluating pollution.
• Alcohol when evaluating late meals.
• Seasonal or long-term trend.
• Data-source/device changes.

The engine should degrade to an unadjusted descriptive association when adjustment is not defensible, and label it accordingly.

Statistical safeguards

Preserve and extend the current quality discipline:

• Minimum paired observations.
• Minimum exposed and unexposed support where applicable.
• Effect-size floor.
• Multiple-comparison/FDR control across the tested family.
• Sample-size shrinkage.
• Confidence tiers.
• Outlier-resistant summaries where appropriate.
• Explicit missing-data and source-coverage checks.
• Avoid same-family tautologies and duplicated derived channels.
• Version the analysis method so old and new results remain explainable.
• Do not surface a pattern solely because p < 0.05.

Because the number of possible factors and outcomes can grow substantially, the tested hypothesis set must remain curated, bounded, and inspectable.

Multi-factor coincidences

A useful later capability is detecting repeated combinations, for example:

Late alcohol plus a late meal and a hotter-than-usual night often preceded higher nocturnal pulse.

This should not be unrestricted association-rule mining. Suggested safeguards:

• Curated combinations of at most two or three factors.
• Strong minimum support.
• Holdout or stability checks where data volume allows.
• Multiple-comparison control.
• Comparison against the strongest single-factor explanation.
• Never describe a combination as causal.
• Do not surface a cluster that is merely a proxy for weekend, travel, or illness.

Evidence contract

Every surfaced pattern should produce a structured evidence object containing at least:

• Factor and outcome.
• Time window/lag.
• Analysis window dates.
• Exposed and comparison sample counts.
• Quantity/timing definition where relevant.
• Outcome difference and effect size.
• Raw and adjusted significance/confidence information where applicable.
• Confidence tier.
• Data sources used.
• Missing-data or source-change warnings.
• Adjustment/matching variables used.
• Whether the result is established, emerging, or insufficient.
• Plain-language non-causal interpretation.
• Links to the underlying days/events where privacy and UX permit.

Insights, notifications, briefings, and Coach should render from this contract rather than independently inventing stronger wording.

Coach and proactive explanations

Proactive behavior

Coach may proactively surface a pattern only when:

• The user has enabled contextual insights/Coach use.
• The pattern passes the configured statistical and effect-size gates.
• There is enough recent or retrospective support.
• It has not already been dismissed or repeatedly shown without new evidence.
• The message is actionable or meaningfully explanatory.
• No higher-priority safety or data-quality warning contradicts it.

Confidence-gated language

Examples:

Established personal association

Across 24 comparable evenings, alcohol-logged nights were associated with a higher nocturnal pulse and lower HRV. The result is observational and does not prove alcohol caused the change.

Emerging signal

There is an early hint that caffeine within six hours of bedtime coincides with shorter sleep. This is based on seven paired nights, so more data is needed.

Treatment context

Symptoms were logged more often on days after a missed dose during this period. Other changes, including illness and sleep, may also contribute. Do not change medication without professional guidance.

Environmental context

Resting pulse was somewhat higher after high-PM2.5 days in the available data. This is modeled ambient exposure, not a measurement of your personal inhaled dose.

Language that should not be allowed

• “Alcohol caused your poor sleep.”
• “Pollution caused your high heart rate.”
• “Your medication is ineffective.”
• “Stop or change this medication.”
• “This pattern proves you have a condition.”

User controls and feedback

• Show why the insight appeared.
• Expand to evidence and source days.
• Mark useful, not useful, incorrect event, or not relevant.
• Dismiss a pattern or mute a factor.
• Correct underlying events.
• Choose whether Coach can use lifestyle, medication, reproductive, or environmental context.
• Avoid repeating the same insight unless evidence materially changes.

Optional guided N-of-1 experiments

A later phase could turn a passive association into a user-chosen experiment, for example:

• No caffeine after 14:00 for two weeks.
• Earlier dinner period.
• Air purifier on versus normal routine.
• Stable bedtime period.
• Alcohol-free comparison period.

Requirements:

• Explicit user opt-in.
• Clear protocol and duration.
• Adherence tracking.
• Predefined outcomes.
• No experiment involving medication changes without clinician direction.
• Honest reporting when results are inconclusive.

This can be implemented later, but the evidence contract should not block it.

Privacy and safety

• Sensitive contextual categories remain optional.
• Existing domain authorization rules apply to every adapter.
• Prompt payloads should contain only the evidence needed for the current explanation, not an unbounded raw timeline.
• No plaintext sensitive events in logs or telemetry.
• Inferred factors must never be silently promoted to confirmed facts.
• Medication findings must carry a stronger safety frame and must not recommend dose changes.
• Reproductive and substance-use context require explicit module/user controls.
• Environmental data must retain modeled/measured provenance.

Acceptance criteria

• The engine can consume existing medication, workout, illness, mood, sleep, cycle, travel, and environmental contributors without duplicating their records.
• Timed and quantified lifestyle events from [Feature] Add a unified contextual-factor timeline and structured lifestyle-event logging #613 can participate in analysis.
• Factor-specific same-day, same-night, next-day, and bounded cumulative windows are supported.
• Quantity/timing analysis degrades safely to present/absent when data is sparse.
• Tested pairs are bounded and pass minimum-support, effect-size, and FDR safeguards.
• Obvious source changes and missing-data problems can suppress or downgrade results.
• Every surfaced result provides a structured evidence object with sample counts, lag, effect, confidence, provenance, and non-causal wording.
• Coach, Insights, and proactive notifications cannot exceed the evidence object’s confidence tier.
• Users can inspect, dismiss, mute, correct, and give feedback on surfaced patterns.
• Medication-related insights cannot recommend starting, stopping, or changing treatment.
• Tests cover timing windows, timezone/travel behavior, dose normalization, source projection, FDR family construction, confidence wording, suppression, redaction, and evidence-to-Coach rendering.

Dependencies and related issues

• Unified contextual factors and lifestyle logging: [Feature] Add a unified contextual-factor timeline and structured lifestyle-event logging #613.
• Structured health/lifestyle profile: [Feature] Expand the encrypted health profile with a structured health and lifestyle questionnaire #614.
• Expanded environmental exposure context: [Feature] Expand environmental context with air quality, heat, UV, pollen, and smoke exposure #615.

The engine can begin with existing contributors, while #613 and #615 expand the available factor set.

[MBombeck]

Thank you for this, @lutzkind, and for the care you put into writing it up. Broadening pattern discovery across behavioral, treatment, and environmental factors is exactly the kind of depth this is meant to grow into.

I want to be straight about timing: the near-term focus is on making the foundation rock solid, reliability, sync, and the quality issues you and others have surfaced, rather than new features. So I am capturing this as a roadmap item rather than starting on it right away. It is noted and valued, and I would rather build it well later than rush it now. Please keep the ideas coming.

[MBombeck]

This is live in v1.33.0, @lutzkind.

A mood tag can now be paired against the next morning's resting heart rate and heart-rate variability, at the same D to D+1 lag the recovery channel already used.

It is worth saying what was actually wrong, because it was not what I expected. The engine had the right idea all along: the tag crosstab has carried a next-day mode since it shipped, and the comment above it names an evening alcohol tag as the example. But that mode had exactly one channel, a wearable recovery score, and plenty of people never record one. Meanwhile the board right next to it, the one for rated factors, has tested next-day resting heart rate and HRV since v1.14.0. A rated factor is a continuous 1 to 5 score, and alcohol is a binary tag, so a binary tag could never reach either vital. Two boards, two maps, and the one you were using was missing the two channels most people actually have.

Same engine, same day floors per side, same Benjamini-Hochberg family, same card. No second query: the fetch filter derives from the map, so the channels came along on their own.

One deliberate change to how it reads. The tag board used to colour a positive difference green and a negative one red. With resting heart rate on the board that is wrong in both directions at once, since a rise is worse on one row and better on the next. The difference is now printed without a colour verdict, matching what the rated-factor board already did. The sign is still there, so direction is not lost, only the judgement the association cannot carry.

What I cannot tell you is whether it will surface something for your account. The mechanism, the floors and the correction family are covered by tests. Whether the numbers clear them on real data is a question only your data answers. If you try it and it stays empty where you expected a finding, tell me and I will look at where it stops.

You are named in the release notes for this one. Thanks for the report, and for the run of them before it.

[MBombeck]

v1.34.0 extends what the correlation engine can check against your own outcomes: an account-defined custom metric can now feed the same engine your built-in metrics run through, and caffeine and nicotine join alcohol as taggable behavioural factors. A correlation or crosstab finding can also be dismissed once you've looked at it, so it stops recomputing itself back into view. The engine itself is still daily-only, so it cannot separate two events that happen a few hours apart, and the environmental-factor expansion this issue also touches on is not in this release. Refs #616.