diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 2e202985cd2..ff58b727dcc 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -525,3 +525,7 @@ /packages/zoom @elastic/security-service-integrations /packages/zscaler_zia @elastic/security-service-integrations /packages/zscaler_zpa @elastic/security-service-integrations +/packages/cisco_meraki_metrics @elastic/obs-infraobs-integrations +/packages/panw_metrics @elastic/obs-infraobs-integrations +/packages/o365_metrics @elastic/obs-infraobs-integrations +/packages/chargeback @elastic/customer-architects diff --git a/packages/chargeback/_dev/build/build.yml b/packages/chargeback/_dev/build/build.yml new file mode 100755 index 00000000000..97fc8aa10cd --- /dev/null +++ b/packages/chargeback/_dev/build/build.yml @@ -0,0 +1,3 @@ +dependencies: + ecs: + reference: git@v8.17.0 diff --git a/packages/chargeback/_dev/build/docs/README.md b/packages/chargeback/_dev/build/docs/README.md new file mode 100755 index 00000000000..a21f691b820 --- /dev/null +++ b/packages/chargeback/_dev/build/docs/README.md @@ -0,0 +1,183 @@ +# Chargeback + +_Technical preview: This integration is being developed by Elastic's Customer Engineering team. Please report any issues to the Elastician who shared this integration with you._ + +The Chargeback integration provides FinOps visibility into Elastic usage across tenants. By integrating data from the [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) and [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integrations, it enables the determination of value provided by each deployment, data stream, and tier across the organisation. This allows Centre of Excellence (CoE) teams to accurately allocate costs back to the appropriate tenant. + +The integration creates several transforms that aggregate billing and usage data into lookup indices optimized for cost analysis and chargeback reporting. + +## What is FinOps? + +FinOps is an operational framework and cultural practice aimed at maximizing the business value of cloud usage. It facilitates timely, data-driven decision-making and promotes financial accountability through collaboration among engineering, finance, and business teams. + +## Purpose + +The Chargeback integration assists organisations in addressing a crucial question: + +> **"How is my organisation consuming the Elastic solution, and to which tenants can I allocate these costs?"** + +The integration provides a breakdown of Elastic Consumption Units (ECUs) per: + +- Deployment +- Data tier +- Data stream +- Day + +Currently, Chargeback calculations consider only Elasticsearch data nodes. Contributions from other assets, like Kibana or ML nodes, are assumed to be shared proportionally among tenants. To incorporate indexing, querying, and storage in a weighted manner, a blended value is created using the following default weights (modifiable): +- Indexing: `20` (applicable only to the hot tier) +- Querying: `20` +- Storage: `40` + +This default weighting means storage contributes most to the blended cost calculation, with indexing considered only on the hot tier. Adjust these weights based on your organisation's needs and best judgment. + +Chargeback costs are presented based on a configured rate and unit. These are used to display cost in your local currency, for instance `EUR`, with a rate of `0.85` per ECU. + +## Configuration + +Configuration values are stored in the `chargeback_conf_lookup` index. The dashboard automatically applies the correct configuration based on the billing date falling within the `conf_start_date` and `conf_end_date` range. + +### Update the default configuration: + +Using `_update/config` updates the document with ID `config`: + +``` +POST chargeback_conf_lookup/_update/config +{ + "doc": { + "conf_ecu_rate": 0.85, + "conf_ecu_rate_unit": "EUR", + "conf_indexing_weight": 20, + "conf_query_weight": 20, + "conf_storage_weight": 40, + "conf_start_date": "2024-01-01T00:00:00.000Z", + "conf_end_date": "2024-12-31T23:tie" + } +} +``` + +### Add a new configuration period (for time-based rate changes): + +Using `_doc` creates a new document with an auto-generated ID: + +``` +POST chargeback_conf_lookup/_doc +{ + "conf_ecu_rate": 0.95, + "conf_ecu_rate_unit": "EUR", + "conf_indexing_weight": 20, + "conf_query_weight": 20, + "conf_storage_weight": 40, + "conf_start_date": "2025-01-01T00:00:00.000Z", + "conf_end_date": "2025-12-31T23:59:59.999Z" +} +``` + +This allows you to have different rates for different time periods (e.g., quarterly or annual rate changes). + +**Configuration Options:** +- `conf_ecu_rate`: The monetary value per ECU (e.g., 0.85) +- `conf_ecu_rate_unit`: The currency code (e.g., "EUR", "USD", "GBP") +- `conf_indexing_weight`: Weight for indexing operations (default: 20, only applies to hot tier) +- `conf_query_weight`: Weight for query operations (default: 20) +- `conf_storage_weight`: Weight for storage (default: 40) +- `conf_start_date`: Start date/time for the configuration period (ISO 8601 format) +- `conf_end_date`: End date/time for the configuration period (ISO 8601 format) + +## Data and Transforms + +The integration creates the following transforms to aggregate cost and usage data: + +1. **billing_cluster_cost** - Aggregates daily ECU usage per deployment from ESS Billing data, with support for deployment groups via `chargeback_group` tags +2. **cluster_deployment_contribution** - Calculates per-deployment usage metrics (indexing time, query time, storage) from Elasticsearch monitoring data +3. **cluster_datastream_contribution** - Aggregates usage per data stream for detailed cost attribution +4. **cluster_tier_contribution** - Aggregates usage per data tier (hot, warm, cold, frozen) +5. **cluster_tier_and_ds_contribution** - Combined view of usage by both tier and data stream + +These transforms produce lookup indices that are queried by the dashboard using ES|QL LOOKUP JOINs to correlate billing costs with actual usage patterns. + +### Transform Auto-Start + +All Chargeback transforms start automatically when the integration is installed. No manual intervention is required to start the transforms. + +**Performance Note:** On clusters with months of historical monitoring data for multiple deployments, the initial transform execution may process a large volume of data. This can cause temporary performance impact during the first run. The transforms will then run incrementally on their configured schedules (15-60 minute intervals), processing only new data with minimal overhead. + +You can verify the transforms are running by navigating to **Stack Management → Transforms** and filtering for `chargeback`. + +### Transform Health Monitoring + +The integration includes a **Transform Health Monitoring** alert rule template that can be installed from the integration page. This rule monitors all Chargeback transforms and alerts when they encounter issues or failures, providing proactive notification of any problems with data processing. + +## Dashboard + +Chargeback data can be viewed in the `[Chargeback] Cost and Consumption breakdown` dashboard, which provides: + +- Cost breakdown by deployment, data tier, and data stream +- Time-series cost trends +- Deployment group filtering for team/project-based analysis +- Blended cost metrics combining indexing, querying, and storage usage +- ECU consumption vs. monetary cost comparison + +![Cost and Consumption breakdown](../img/chargeback.png) + +## Deployment Groups + +The integration supports organizing deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by teams, projects, or any organizational structure. + +To assign a deployment to a chargeback group, add a tag to your deployment in the Elastic Cloud console in the format: +``` +chargeback_group: +``` + +For example: `chargeback_group:team-search` or `chargeback_group:project-analytics` + +The `billing_cluster_cost` transform automatically extracts these tags from the `deployment_tags` field in ESS Billing data using runtime mappings. The dashboard includes a deployment group filter to view costs by specific groups, making it easy to track expenses per team or project. + +**Note:** Each deployment should have only one `chargeback_group` tag. Having multiple tags can cause issues and lead to unpredictable cost allocation. + +## Observability Alerting + +This integration includes 3 pre-configured alert rule templates that can be installed directly from the integration page in Kibana: + +1. **Transform Health Monitoring** - Monitors the health of all Chargeback transforms and alerts when they encounter issues or failures +2. **New Chargeback Group Detected** - Notifies when a new `chargeback_group` tag is added to a deployment +3. **Deployment with Chargeback Group Missing Usage Data** - Detects when a deployment has a chargeback group assigned but is not sending usage/consumption data + +**Important:** For alert rules 2 and 3, ensure that the Chargeback transforms are running before setting them up. These alerting rules query the lookup indices created by the transforms (`billing_cluster_cost_lookup`, `cluster_deployment_contribution_lookup`, etc.). If the transforms are not started, the alerts will not function correctly. + +### Alert actions + +**Configure an action** with the following message template appended to the default content (keep the new lines, as it helps with legibility): + +``` +Details: + +{{`{{#context.hits}}`}} +• {{`{{_source}}`}} + +{{`{{/context.hits}}`}} + +Total: {{`{{context.hits.length}}`}} +``` + +## Requirements + +To use this integration, the following prerequisites must be met: + +**Monitoring Cluster:** +- Must be on Elasticsearch version **9.2.0+** due to the use of smart [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join) (conditional joins) in transforms and dashboard queries +- This is where the Chargeback integration should be installed + +**Required Integrations:** +- [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) integration (v1.4.1+) must be installed and collecting billing data from your Elastic Cloud organization +- [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integration (v1.16.0+) must be installed and collecting [usage data](https://www.elastic.co/docs/reference/integrations/elasticsearch/#indices-and-data-streams-usage-analysis) from all deployments you want to include in chargeback calculations + +**Required Transforms:** +- The transform `logs-elasticsearch.index_pivot-default-{VERSION}` (from the Elasticsearch integration) must be running to aggregate usage metrics per index + +**Data Flow:** +1. ESS Billing data is collected into `metrics-ess_billing.billing-*` +2. Elasticsearch usage data is collected into `metrics-elasticsearch.stack_monitoring.*` (or `monitoring-indices` for Stack Monitoring) +3. Chargeback transforms process and correlate this data +4. Dashboard queries the resulting lookup indices using ES|QL + +**Note:** This integration must be installed on a centralized monitoring cluster that has visibility to both billing and usage data from your deployments. \ No newline at end of file diff --git a/packages/chargeback/changelog.yml b/packages/chargeback/changelog.yml new file mode 100644 index 00000000000..946a8685d98 --- /dev/null +++ b/packages/chargeback/changelog.yml @@ -0,0 +1,96 @@ +# newer versions go on top +- version: 0.2.10 + changes: + - description: "Fix visualizations not loading by adding TO_DOUBLE type conversion in ESQL queries to prevent integer division returning zero. Add automated chargeback_conf_lookup index creation via transform with default configuration values. Bump transform pipeline versions to 0.2.10." + type: bugfix + link: https://github.com/elastic/integrations/pull/16936 +- version: 0.2.9 + changes: + - description: "Updating transforms for CSS support on monitoring data" + type: enhancement + link: https://github.com/elastic/integrations/pull/16326 +- version: 0.2.8 + changes: + - description: "Add Kibana alerting rule templates for transform health monitoring, detecting new chargeback groups, and identifying deployments with missing usage data. Templates can be installed directly from the package. All transforms now auto-start on installation." + type: enhancement + link: https://github.com/elastic/integrations/pull/16229 +- version: 0.2.7 + changes: + - description: "Add observability alerting rule templates and documentation for monitoring new chargeback groups and missing usage data. Update Elasticsearch version requirement to 9.2.0+ for smart lookup join support." + type: enhancement + link: https://github.com/elastic/integrations/pull/16205 +- version: 0.2.6 + changes: + - description: "Fixing bug around sku based cost allocation" + type: bugfix + link: https://github.com/elastic/integrations/pull/16192 +- version: 0.2.5 + changes: + - description: "Add deployment_group field extracted from ESS Billing deployment tags using runtime mappings to enable tag-based cost allocation and filtering. Fix transforms to use correct field type for elasticsearch.cluster.name." + type: enhancement + link: https://github.com/elastic/integrations/pull/16185 +- version: 0.2.4 + changes: + - description: "Adding sku and cost_type to the billing_cluster_cost_lookup for future utilization" + type: enhancement + link: https://github.com/elastic/integrations/pull/16182 +- version: 0.2.3 + changes: + - description: "Adding deployment filter, dataview and moving config portion to bottom of dashboard for better usability." + type: enhancement + link: https://github.com/elastic/integrations/pull/16153 +- version: 0.2.2 + changes: + - description: "Allow setting the Conversion Rate per time window in the configuration lookup index and adding collapsable sections in the dashboard for better usability." + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 +- version: 0.2.1 + changes: + - description: "Fixing the issue of visualisation not displaying values due to integer value division in ESQL. Changed to use `double` values instead." + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 +- version: 0.2.0 + changes: + - description: "Make use of the new elastic-package version, which will create the lookup index automatically when installing the package. " + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 +- version: 0.1.7 + changes: + - description: "Swap the use of deployment_id or deployment name to a concatenation of both, to make it easier to identify the deployment in the dashboard." + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 +- version: 0.1.6 + changes: + - description: "Remove the use of usage alias, and stick to using `monitoring-indices` as usage sorce. ES Integration transform should be run regardless of wether the ES integration has been installed on an agent or not. This fix will increase performance, when relying on Stack Monitoring data. Also, use `metrics-ess_billing.billing-*` to be able to use not only the default namespace." + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 +- version: 0.1.5 + changes: + - description: "Fixing the control error in the dashboard by adding a data view." + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 +- version: 0.1.4 + changes: + - description: "Consistent naming of `datastream`. Add `| LIMIT 5000` to ESQL top query to cater for large organisations." + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 +- version: 0.1.3 + changes: + - description: "Made sure the colour palette is predictable by using the eui_amsterdam_color_blind palate. Add ECU rate to the dashboard." + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 +- version: 0.1.2 + changes: + - description: "Added the necessary fields to the billing_cluster_cost_lookup in the Elasticsearch transform to allow for correlation with the ES integration." + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 +- version: 0.1.1 + changes: + - description: "Fixed the dashboard chargeback timeframe calculation for cost and ECU utilisation" + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 +- version: 0.1.0 + changes: + - description: "Initial release of the chargeback integration." + type: enhancement + link: https://github.com/elastic/integrations/pull/14545 diff --git a/packages/chargeback/docs/README.md b/packages/chargeback/docs/README.md new file mode 100644 index 00000000000..a3ab4167eaa --- /dev/null +++ b/packages/chargeback/docs/README.md @@ -0,0 +1,183 @@ +# Chargeback + +_Technical preview: This integration is being developed by Elastic's Customer Engineering team. Please report any issues to the Elastician who shared this integration with you._ + +The Chargeback integration provides FinOps visibility into Elastic usage across tenants. By integrating data from the [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) and [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integrations, it enables the determination of value provided by each deployment, data stream, and tier across the organisation. This allows Centre of Excellence (CoE) teams to accurately allocate costs back to the appropriate tenant. + +The integration creates several transforms that aggregate billing and usage data into lookup indices optimized for cost analysis and chargeback reporting. + +## What is FinOps? + +FinOps is an operational framework and cultural practice aimed at maximizing the business value of cloud usage. It facilitates timely, data-driven decision-making and promotes financial accountability through collaboration among engineering, finance, and business teams. + +## Purpose + +The Chargeback integration assists organisations in addressing a crucial question: + +> **"How is my organisation consuming the Elastic solution, and to which tenants can I allocate these costs?"** + +The integration provides a breakdown of Elastic Consumption Units (ECUs) per: + +- Deployment +- Data tier +- Data stream +- Day + +Currently, Chargeback calculations consider only Elasticsearch data nodes. Contributions from other assets, like Kibana or ML nodes, are assumed to be shared proportionally among tenants. To incorporate indexing, querying, and storage in a weighted manner, a blended value is created using the following default weights (modifiable): +- Indexing: `20` (applicable only to the hot tier) +- Querying: `20` +- Storage: `40` + +This default weighting means storage contributes most to the blended cost calculation, with indexing considered only on the hot tier. Adjust these weights based on your organisation's needs and best judgment. + +Chargeback costs are presented based on a configured rate and unit. These are used to display cost in your local currency, for instance `EUR`, with a rate of `0.85` per ECU. + +## Configuration + +Configuration values are stored in the `chargeback_conf_lookup` index. The dashboard automatically applies the correct configuration based on the billing date falling within the `conf_start_date` and `conf_end_date` range. + +### Update the default configuration: + +Using `_update/config` updates the document with ID `config`: + +``` +POST chargeback_conf_lookup/_update/config +{ + "doc": { + "conf_ecu_rate": 0.85, + "conf_ecu_rate_unit": "EUR", + "conf_indexing_weight": 20, + "conf_query_weight": 20, + "conf_storage_weight": 40, + "conf_start_date": "2024-01-01T00:00:00.000Z", + "conf_end_date": "2024-12-31T23:tie" + } +} +``` + +### Add a new configuration period (for time-based rate changes): + +Using `_doc` creates a new document with an auto-generated ID: + +``` +POST chargeback_conf_lookup/_doc +{ + "conf_ecu_rate": 0.95, + "conf_ecu_rate_unit": "EUR", + "conf_indexing_weight": 20, + "conf_query_weight": 20, + "conf_storage_weight": 40, + "conf_start_date": "2025-01-01T00:00:00.000Z", + "conf_end_date": "2025-12-31T23:59:59.999Z" +} +``` + +This allows you to have different rates for different time periods (e.g., quarterly or annual rate changes). + +**Configuration Options:** +- `conf_ecu_rate`: The monetary value per ECU (e.g., 0.85) +- `conf_ecu_rate_unit`: The currency code (e.g., "EUR", "USD", "GBP") +- `conf_indexing_weight`: Weight for indexing operations (default: 20, only applies to hot tier) +- `conf_query_weight`: Weight for query operations (default: 20) +- `conf_storage_weight`: Weight for storage (default: 40) +- `conf_start_date`: Start date/time for the configuration period (ISO 8601 format) +- `conf_end_date`: End date/time for the configuration period (ISO 8601 format) + +## Data and Transforms + +The integration creates the following transforms to aggregate cost and usage data: + +1. **billing_cluster_cost** - Aggregates daily ECU usage per deployment from ESS Billing data, with support for deployment groups via `chargeback_group` tags +2. **cluster_deployment_contribution** - Calculates per-deployment usage metrics (indexing time, query time, storage) from Elasticsearch monitoring data +3. **cluster_datastream_contribution** - Aggregates usage per data stream for detailed cost attribution +4. **cluster_tier_contribution** - Aggregates usage per data tier (hot, warm, cold, frozen) +5. **cluster_tier_and_ds_contribution** - Combined view of usage by both tier and data stream + +These transforms produce lookup indices that are queried by the dashboard using ES|QL LOOKUP JOINs to correlate billing costs with actual usage patterns. + +### Transform Auto-Start + +All Chargeback transforms start automatically when the integration is installed. No manual intervention is required to start the transforms. + +**Performance Note:** On clusters with months of historical monitoring data for multiple deployments, the initial transform execution may process a large volume of data. This can cause temporary performance impact during the first run. The transforms will then run incrementally on their configured schedules (15-60 minute intervals), processing only new data with minimal overhead. + +You can verify the transforms are running by navigating to **Stack Management → Transforms** and filtering for `chargeback`. + +### Transform Health Monitoring + +The integration includes a **Transform Health Monitoring** alert rule template that can be installed from the integration page. This rule monitors all Chargeback transforms and alerts when they encounter issues or failures, providing proactive notification of any problems with data processing. + +## Dashboard + +Chargeback data can be viewed in the `[Chargeback] Cost and Consumption breakdown` dashboard, which provides: + +- Cost breakdown by deployment, data tier, and data stream +- Time-series cost trends +- Deployment group filtering for team/project-based analysis +- Blended cost metrics combining indexing, querying, and storage usage +- ECU consumption vs. monetary cost comparison + +![Cost and Consumption breakdown](../img/chargeback.png) + +## Deployment Groups + +The integration supports organizing deployments into logical groups using the `chargeback_group` tag on ESS Billing deployments. This enables cost allocation and filtering by teams, projects, or any organizational structure. + +To assign a deployment to a chargeback group, add a tag to your deployment in the Elastic Cloud console in the format: +``` +chargeback_group: +``` + +For example: `chargeback_group:team-search` or `chargeback_group:project-analytics` + +The `billing_cluster_cost` transform automatically extracts these tags from the `deployment_tags` field in ESS Billing data using runtime mappings. The dashboard includes a deployment group filter to view costs by specific groups, making it easy to track expenses per team or project. + +**Note:** Each deployment should have only one `chargeback_group` tag. Having multiple tags can cause issues and lead to unpredictable cost allocation. + +## Observability Alerting + +This integration includes 3 pre-configured alert rule templates that can be installed directly from the integration page in Kibana: + +1. **Transform Health Monitoring** - Monitors the health of all Chargeback transforms and alerts when they encounter issues or failures +2. **New Chargeback Group Detected** - Notifies when a new `chargeback_group` tag is added to a deployment +3. **Deployment with Chargeback Group Missing Usage Data** - Detects when a deployment has a chargeback group assigned but is not sending usage/consumption data + +**Important:** For alert rules 2 and 3, ensure that the Chargeback transforms are running before setting them up. These alerting rules query the lookup indices created by the transforms (`billing_cluster_cost_lookup`, `cluster_deployment_contribution_lookup`, etc.). If the transforms are not started, the alerts will not function correctly. + +### Alert actions + +**Configure an action** with the following message template appended to the default content (keep the new lines, as it helps with legibility): + +``` +Details: + +{{#context.hits}} +• {{_source}} + +{{/context.hits}} + +Total: {{context.hits.length}} +``` + +## Requirements + +To use this integration, the following prerequisites must be met: + +**Monitoring Cluster:** +- Must be on Elasticsearch version **9.2.0+** due to the use of smart [ES|QL LOOKUP JOIN](https://www.elastic.co/docs/reference/query-languages/esql/esql-lookup-join) (conditional joins) in transforms and dashboard queries +- This is where the Chargeback integration should be installed + +**Required Integrations:** +- [**Elasticsearch Service Billing**](https://www.elastic.co/docs/reference/integrations/ess_billing/) integration (v1.4.1+) must be installed and collecting billing data from your Elastic Cloud organization +- [**Elasticsearch**](https://www.elastic.co/docs/reference/integrations/elasticsearch/) integration (v1.16.0+) must be installed and collecting [usage data](https://www.elastic.co/docs/reference/integrations/elasticsearch/#indices-and-data-streams-usage-analysis) from all deployments you want to include in chargeback calculations + +**Required Transforms:** +- The transform `logs-elasticsearch.index_pivot-default-{VERSION}` (from the Elasticsearch integration) must be running to aggregate usage metrics per index + +**Data Flow:** +1. ESS Billing data is collected into `metrics-ess_billing.billing-*` +2. Elasticsearch usage data is collected into `metrics-elasticsearch.stack_monitoring.*` (or `monitoring-indices` for Stack Monitoring) +3. Chargeback transforms process and correlate this data +4. Dashboard queries the resulting lookup indices using ES|QL + +**Note:** This integration must be installed on a centralized monitoring cluster that has visibility to both billing and usage data from your deployments. \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/ingest_pipeline/billing.yml b/packages/chargeback/elasticsearch/ingest_pipeline/billing.yml new file mode 100644 index 00000000000..ce79f1bb99f --- /dev/null +++ b/packages/chargeback/elasticsearch/ingest_pipeline/billing.yml @@ -0,0 +1,53 @@ +--- +description: "Chargeback: Set composite_key from @timestamp and deployment_id, and extract cost_type from SKU." +processors: + - script: + ignore_failure: true + lang: painless + source: | + if (ctx['@timestamp'] != null && ctx.deployment_id != null) { + ctx.composite_key = ZonedDateTime.parse(ctx['@timestamp']).toLocalDate().toString() + '_' + ctx.deployment_id; + } + tag: ess_billing + - set: + field: cost_type + value: 'data-transfer' + if: ctx.sku =~ /^(aws|gcp|azure).data-transfer.*/ + - set: + field: cost_type + value: 'snapshots' + if: ctx.sku =~ /^(aws|gcp|azure).snapshot-storage&/ + - set: + field: cost_type + value: 'inference' + if: ctx.sku == 'global.inference-chat-input' + - grok: + field: sku + ignore_failure: true + patterns: + - "%{WORD}.es.%{WORD:cost_type}.%{GREEDYDATA}" + - "%{WORD}.%{DATA:cost_type}-vcu_%{GREEDYDATA}" + - "%{WORD}.%{WORD:cost_type}.%{GREEDYDATA}" + - "%{WORD}.%{DATA:cost_type}.%{GREEDYDATA}" + description: retrieve a human readable cost type from the sku field + if: ctx.cost_type == null + - set: + field: cost_type + value: 'datahot/datacontent' + if: ctx.cost_type == 'datahot' + - pipeline: + description: '[Fleet] Global pipeline for all data streams' + ignore_missing_pipeline: true + name: global@custom + - pipeline: + description: '[Fleet] Pipeline for all data streams of type `metrics`' + ignore_missing_pipeline: true + name: metrics@custom + - pipeline: + description: '[Fleet] Pipeline for all data streams of type `metrics` defined by the `chargeback` integration' + ignore_missing_pipeline: true + name: metrics-chargeback.integration@custom + - pipeline: + description: '[Fleet] Pipeline for the `chargeback.billing` dataset' + ignore_missing_pipeline: true + name: metrics-chargeback.billing@custom diff --git a/packages/chargeback/elasticsearch/ingest_pipeline/chargeback_conf_lookup.yml b/packages/chargeback/elasticsearch/ingest_pipeline/chargeback_conf_lookup.yml new file mode 100644 index 00000000000..c82b57add84 --- /dev/null +++ b/packages/chargeback/elasticsearch/ingest_pipeline/chargeback_conf_lookup.yml @@ -0,0 +1,17 @@ +--- +description: Post-process chargeback configuration to extract currency unit from terms aggregation +processors: + - script: + description: Extract currency unit key from terms aggregation bucket + lang: painless + source: | + if (ctx.conf_ecu_rate_unit instanceof Map) { + def keys = new ArrayList(ctx.conf_ecu_rate_unit.keySet()); + if (keys.size() > 0) { + ctx.conf_ecu_rate_unit = keys[0]; + } + } +on_failure: + - set: + field: error.message + value: "Failed to process chargeback config: {{ _ingest.on_failure_message }}" diff --git a/packages/chargeback/elasticsearch/ingest_pipeline/usage.yml b/packages/chargeback/elasticsearch/ingest_pipeline/usage.yml new file mode 100644 index 00000000000..81ad97a5616 --- /dev/null +++ b/packages/chargeback/elasticsearch/ingest_pipeline/usage.yml @@ -0,0 +1,40 @@ +--- +description: 'Chargeback: Set correlation keys and blended calculation weights.' +processors: + - script: + ignore_failure: true + lang: painless + source: | + if (ctx.cluster_name != null) { + ctx.deployment_id = ctx.cluster_name; + } else if (ctx.elasticsearch?.cluster?.name != null) { + ctx.deployment_id = ctx.elasticsearch.cluster.name; + } + if (ctx['@timestamp'] != null && ctx.deployment_id != null) { + def date = ZonedDateTime.parse(ctx['@timestamp']).toLocalDate().toString(); + ctx.composite_key = date + '_' + ctx.deployment_id; + + if (ctx.tier != null) { + ctx.composite_tier_key = ctx.composite_key + '_' + ctx.tier.replace('/', '_'); + } + + if (ctx.datastream != null) { + ctx.composite_datastream_key = ctx.composite_key + '_' + ctx.datastream; + } + } + - pipeline: + description: '[Fleet] Global pipeline for all data streams' + ignore_missing_pipeline: true + name: global@custom + - pipeline: + description: '[Fleet] Pipeline for all data streams of type `metrics`' + ignore_missing_pipeline: true + name: metrics@custom + - pipeline: + description: '[Fleet] Pipeline for all data streams of type `metrics` defined by the `chargeback` integration' + ignore_missing_pipeline: true + name: metrics-chargeback.integration@custom + - pipeline: + description: '[Fleet] Pipeline for the `chargeback.usage` dataset' + ignore_missing_pipeline: true + name: metrics-chargeback.usage@custom diff --git a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/fields/fields.yml b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/fields/fields.yml new file mode 100644 index 00000000000..3f885fdb813 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/fields/fields.yml @@ -0,0 +1,24 @@ +- name: "@timestamp" + type: date + description: The timestamp representing the day of the billing data. +- name: deployment_group + type: keyword + description: User-defined group of the deployment (based on tag:chargeback_group). +- name: deployment_id + type: keyword + description: Unique ID of the deployment. +- name: deployment_name + type: keyword + description: Human-readable name of the deployment. +- name: total_ecu + type: double + description: Total ECU usage aggregated for the deployment per day. +- name: composite_key + type: keyword + description: Composite key used for billing attribution (set by ingest pipeline) consisting of date and deployment ID. +- name: sku + type: keyword + description: Component where the ecu is being spend +- name: cost_type + type: keyword + description: Component where the ecu is being spent, derived from SKU field. \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/manifest.yml b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/manifest.yml new file mode 100644 index 00000000000..ccaa87d98f1 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/manifest.yml @@ -0,0 +1,8 @@ +start: true +destination_index_template: + settings: + index: + mode: "lookup" + codec: best_compression + mappings: + dynamic: false \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml new file mode 100644 index 00000000000..2273c9b3cd1 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/billing_cluster_cost/transform.yml @@ -0,0 +1,62 @@ +description: Aggregates daily total ECU usage per deployment from billing metrics, using ingested timestamps with a 1-hour sync delay and running every 60 minutes. +source: + index: + - metrics-ess_billing.billing-* + query: + range: + ess.billing.total_ecu: + gt: 0 + runtime_mappings: + deployment_group: + type: keyword + script: + source: | + if (params._source?.ess?.billing?.deployment_tags != null) { + for (tag in params._source.ess.billing.deployment_tags) { + if (tag.startsWith('chargeback_group:')) { + emit(tag.substring('chargeback_group:'.length())); + return; + } + } + } + emit(''); +dest: + index: billing_cluster_cost_lookup + pipeline: 0.2.10-billing +frequency: 60m +sync: + time: + field: event.ingested + delay: 1h +pivot: + group_by: + "@timestamp": + date_histogram: + field: "@timestamp" + calendar_interval: 1d + deployment_group: + terms: + field: deployment_group + deployment_id: + terms: + field: ess.billing.deployment_id + deployment_name: + terms: + field: ess.billing.deployment_name + sku: + terms: + field: ess.billing.sku + aggregations: + total_ecu: + sum: + field: ess.billing.total_ecu +settings: + # This is required to prevent the transform from clobbering the Fleet-managed mappings. + deduce_mappings: false + unattended: true +_meta: + managed: true + run_as_kibana_system: false + # Bump this version to delete, reinstall, and restart the transform during package. + # Version bump is needed if there is any code change in transform. + fleet_transform_version: 0.2.5 diff --git a/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml new file mode 100644 index 00000000000..c0c21466c88 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml @@ -0,0 +1,24 @@ +- name: config_join_key + type: keyword + description: Join key for chargeback configuration +- name: conf_ecu_rate + type: float + description: ECU rate for cost calculation +- name: conf_ecu_rate_unit + type: keyword + description: Currency unit for ECU rate (e.g., EUR, USD) +- name: conf_indexing_weight + type: integer + description: Weight factor for indexing costs +- name: conf_query_weight + type: integer + description: Weight factor for query costs +- name: conf_storage_weight + type: integer + description: Weight factor for storage costs +- name: conf_start_date + type: date + description: Configuration validity start date (defaults to Elasticsearch birthdate) +- name: conf_end_date + type: date + description: Configuration validity end date (defaults to 2046-12-31) diff --git a/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/ecs.yml b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/ecs.yml new file mode 100644 index 00000000000..fe51488c706 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/ecs.yml @@ -0,0 +1 @@ +[] diff --git a/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/manifest.yml b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/manifest.yml new file mode 100644 index 00000000000..ccaa87d98f1 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/manifest.yml @@ -0,0 +1,8 @@ +start: true +destination_index_template: + settings: + index: + mode: "lookup" + codec: best_compression + mappings: + dynamic: false \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml new file mode 100644 index 00000000000..c3cd99acf0f --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/transform.yml @@ -0,0 +1,73 @@ +description: Bootstrap chargeback configuration lookup index with default settings using billing data as source trigger. +source: + index: + - metrics-ess_billing.billing-* + runtime_mappings: + config_join_key: + type: keyword + script: + source: "emit('chargeback_config')" + conf_ecu_rate: + type: double + script: + source: "emit(0.85)" + conf_ecu_rate_unit: + type: keyword + script: + source: "emit('EUR')" + conf_indexing_weight: + type: long + script: + source: "emit(20)" + conf_query_weight: + type: long + script: + source: "emit(20)" + conf_storage_weight: + type: long + script: + source: "emit(40)" + conf_start_date: + type: date + script: + source: "emit(1262304000000L)" # 2010-01-01T00:00:00.000Z - Elasticsearch birthdate + conf_end_date: + type: date + script: + source: "emit(2430518399999L)" # 2046-12-31T23:59:59.999Z - 20 years from now +dest: + index: chargeback_conf_lookup +pivot: + group_by: + config_join_key: + terms: + field: config_join_key + conf_ecu_rate_unit: + terms: + field: conf_ecu_rate_unit + aggregations: + conf_ecu_rate: + max: + field: conf_ecu_rate + conf_indexing_weight: + max: + field: conf_indexing_weight + conf_query_weight: + max: + field: conf_query_weight + conf_storage_weight: + max: + field: conf_storage_weight + conf_start_date: + max: + field: conf_start_date + conf_end_date: + max: + field: conf_end_date +settings: + deduce_mappings: false + unattended: true +_meta: + managed: true + run_as_kibana_system: false + fleet_transform_version: 0.1.0 diff --git a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/fields/fields.yml b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/fields/fields.yml new file mode 100644 index 00000000000..f489ab2bbf6 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/fields/fields.yml @@ -0,0 +1,24 @@ +- name: "@timestamp" + type: date + description: Daily timestamp representing when the metric was recorded. +- name: cluster_name + type: keyword + description: Name of the Elasticsearch cluster. +- name: datastream + type: keyword + description: The name of the data stream. +- name: datastream_sum_indexing_time + type: long + description: Total indexing time in milliseconds for the data stream. +- name: datastream_sum_query_time + type: long + description: Total query time in milliseconds for the data stream. +- name: datastream_sum_store_size + type: long + description: Total primary shard store size in bytes for the data stream. +- name: datastream_sum_data_set_store_size + type: long + description: Total dataset size in bytes for the data stream. +- name: composite_key + type: keyword + description: Composite key used for billing attribution (set by ingest pipeline) consisting of date and deployment ID. diff --git a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/manifest.yml b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/manifest.yml new file mode 100644 index 00000000000..ccaa87d98f1 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/manifest.yml @@ -0,0 +1,8 @@ +start: true +destination_index_template: + settings: + index: + mode: "lookup" + codec: best_compression + mappings: + dynamic: false \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml new file mode 100644 index 00000000000..8c0c7e99876 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_datastream_contribution/transform.yml @@ -0,0 +1,64 @@ +description: Aggregates daily total ECU usage per datastream from billing metrics, using ingested timestamps with a 1-hour sync delay and running every 60 minutes. +source: + index: + - "monitoring-indices" + - "*:monitoring-indices" # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. +dest: + index: cluster_datastream_contribution_lookup + pipeline: 0.2.10-usage +frequency: 60m +sync: + time: + field: "@timestamp" + delay: 1h +pivot: + group_by: + "@timestamp": + date_histogram: + field: "@timestamp" + calendar_interval: 1d + cluster_name: + terms: + field: elasticsearch.cluster.name + datastream: + terms: + script: + lang: painless + source: > + if (doc.containsKey('elasticsearch.index.datastream') && !doc['elasticsearch.index.datastream'].empty) { + return doc['elasticsearch.index.datastream'].value; + } + + def name = doc.containsKey('elasticsearch.index.name') && !doc['elasticsearch.index.name'].empty + ? doc['elasticsearch.index.name'].value + : null; + if (name == null) return 'unknown'; + + def matcher = /^(?:partial-)?(?:restored-)?(?:shrink-.{4}-)?(?:\.ds-)?([a-z_0-9\-.]+?)(?:-\d{4}\.\d{2}(?:\.\d{2})?)?(?:-\d+)?$/.matcher(name); + if (matcher.matches()) { + return matcher.group(1); + } + return 'unknown'; + aggregations: + datastream_sum_indexing_time: + sum: + field: elasticsearch.index.total.indexing.index_time_in_millis + datastream_sum_query_time: + sum: + field: elasticsearch.index.total.search.query_time_in_millis + datastream_sum_store_size: + sum: + field: elasticsearch.index.total.store.size_in_bytes + datastream_sum_data_set_store_size: + sum: + field: elasticsearch.index.primaries.store.total_data_set_size_in_bytes +settings: + # This is required to prevent the transform from clobbering the Fleet-managed mappings. + deduce_mappings: false + unattended: true +_meta: + managed: true + run_as_kibana_system: false + # Bump this version to delete, reinstall, and restart the transform during package. + # Version bump is needed if there is any code change in transform. + fleet_transform_version: 0.2.9 diff --git a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/fields/fields.yml b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/fields/fields.yml new file mode 100644 index 00000000000..63866dda2e0 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/fields/fields.yml @@ -0,0 +1,21 @@ +- name: "@timestamp" + type: date + description: Daily timestamp representing when the metric was recorded. +- name: cluster_name + type: keyword + description: Name of the Elasticsearch cluster. +- name: deployment_sum_indexing_time + type: long + description: Total indexing time in milliseconds for the deployment. +- name: deployment_sum_query_time + type: long + description: Total query time in milliseconds for the deployment. +- name: deployment_sum_store_size + type: long + description: Total primary shard store size in bytes for the deployment. +- name: deployment_sum_data_set_store_size + type: long + description: Total dataset size in bytes for the deployment. +- name: composite_key + type: keyword + description: Composite key used for billing attribution (set by ingest pipeline) consisting of date and deployment ID. diff --git a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/manifest.yml b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/manifest.yml new file mode 100644 index 00000000000..ea03fc70dcb --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/manifest.yml @@ -0,0 +1,8 @@ +start: true +destination_index_template: + settings: + index: + mode: "lookup" + codec: best_compression + mappings: + dynamic: false \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml new file mode 100644 index 00000000000..d4621193828 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_deployment_contribution/transform.yml @@ -0,0 +1,45 @@ +description: Aggregates daily total ECU usage per deployment from billing metrics, using ingested timestamps with a 1-hour sync delay and running every 60 minutes. +source: + index: + - "monitoring-indices" + - "*:monitoring-indices" # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. +dest: + index: cluster_deployment_contribution_lookup + pipeline: 0.2.10-usage +frequency: 60m +sync: + time: + field: "@timestamp" + delay: 1h +pivot: + group_by: + "@timestamp": + date_histogram: + field: "@timestamp" + calendar_interval: 1d + cluster_name: + terms: + field: elasticsearch.cluster.name + aggregations: + deployment_sum_indexing_time: + sum: + field: elasticsearch.index.total.indexing.index_time_in_millis + deployment_sum_query_time: + sum: + field: elasticsearch.index.total.search.query_time_in_millis + deployment_sum_store_size: + sum: + field: elasticsearch.index.total.store.size_in_bytes + deployment_sum_data_set_store_size: + sum: + field: elasticsearch.index.primaries.store.total_data_set_size_in_bytes +settings: + # This is required to prevent the transform from clobbering the Fleet-managed mappings. + deduce_mappings: false + unattended: true +_meta: + managed: true + run_as_kibana_system: false + # Bump this version to delete, reinstall, and restart the transform during package. + # Version bump is needed if there is any code change in transform. + fleet_transform_version: 0.2.9 \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/fields/fields.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/fields/fields.yml new file mode 100644 index 00000000000..54cc1c6bb6c --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/fields/fields.yml @@ -0,0 +1,27 @@ +- name: "@timestamp" + type: date + description: Daily timestamp representing when the metric was recorded. +- name: cluster_name + type: keyword + description: Name of the Elasticsearch cluster. +- name: tier + type: keyword + description: The data tier where the index resides (e.g. hot, warm, cold, frozen). +- name: datastream + type: keyword + description: The name of the data stream. +- name: tier_and_datastream_sum_indexing_time + type: long + description: Total indexing time in milliseconds for the tier and data stream. +- name: tier_and_datastream_sum_query_time + type: long + description: Total query time in milliseconds for the tier and data stream. +- name: tier_and_datastream_sum_store_size + type: long + description: Total primary shard store size in bytes for the tier and data stream. +- name: tier_and_datastream_sum_data_set_store_size + type: long + description: Total dataset size in bytes for the tier and data stream. +- name: composite_key + type: keyword + description: Composite key used for billing attribution (set by ingest pipeline) consisting of date and data tier and data stream name. diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/manifest.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/manifest.yml new file mode 100644 index 00000000000..ccaa87d98f1 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/manifest.yml @@ -0,0 +1,8 @@ +start: true +destination_index_template: + settings: + index: + mode: "lookup" + codec: best_compression + mappings: + dynamic: false \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml new file mode 100644 index 00000000000..10f8b12ca50 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_and_ds_contribution/transform.yml @@ -0,0 +1,79 @@ +description: Aggregates daily total ECU usage per tier and datastream from billing metrics, using ingested timestamps with a 1-hour sync delay and running every 60 minutes. +source: + index: + - "monitoring-indices" + - "*:monitoring-indices" # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. +dest: + index: cluster_tier_and_datastream_contribution_lookup + pipeline: 0.2.10-usage +frequency: 60m +sync: + time: + field: "@timestamp" + delay: 1h +pivot: + group_by: + "@timestamp": + date_histogram: + field: "@timestamp" + calendar_interval: 1d + cluster_name: + terms: + field: elasticsearch.cluster.name + tier: + terms: + script: + lang: painless + source: > + if (doc.containsKey('elasticsearch.index.tier') && !doc['elasticsearch.index.tier'].empty) { + return doc['elasticsearch.index.tier'].value; + } + def pref = doc.containsKey('elasticsearch.index.tier_preference') && !doc['elasticsearch.index.tier_preference'].empty ? doc['elasticsearch.index.tier_preference'].value : null; + if (pref == null) return 'unknown'; + if (pref.contains('data_hot') || pref.contains('data_content')) return 'hot/content'; + if (pref.contains('data_warm')) return 'warm'; + if (pref.contains('data_cold')) return 'cold'; + if (pref.contains('data_frozen')) return 'frozen'; + return 'unknown'; + datastream: + terms: + script: + lang: painless + source: > + if (doc.containsKey('elasticsearch.index.datastream') && !doc['elasticsearch.index.datastream'].empty) { + return doc['elasticsearch.index.datastream'].value; + } + + def name = doc.containsKey('elasticsearch.index.name') && !doc['elasticsearch.index.name'].empty + ? doc['elasticsearch.index.name'].value + : null; + if (name == null) return 'unknown'; + + def matcher = /^(?:partial-)?(?:restored-)?(?:shrink-.{4}-)?(?:\.ds-)?([a-z_0-9\-.]+?)(?:-\d{4}\.\d{2}(?:\.\d{2})?)?(?:-\d+)?$/.matcher(name); + if (matcher.matches()) { + return matcher.group(1); + } + return 'unknown'; + aggregations: + tier_and_datastream_sum_indexing_time: + sum: + field: elasticsearch.index.total.indexing.index_time_in_millis + tier_and_datastream_sum_query_time: + sum: + field: elasticsearch.index.total.search.query_time_in_millis + tier_and_datastream_sum_store_size: + sum: + field: elasticsearch.index.total.store.size_in_bytes + tier_and_datastream_sum_data_set_store_size: + sum: + field: elasticsearch.index.primaries.store.total_data_set_size_in_bytes +settings: + # This is required to prevent the transform from clobbering the Fleet-managed mappings. + deduce_mappings: false + unattended: true +_meta: + managed: true + run_as_kibana_system: false + # Bump this version to delete, reinstall, and restart the transform during package. + # Version bump is needed if there is any code change in transform. + fleet_transform_version: 0.2.9 \ No newline at end of file diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/fields/fields.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/fields/fields.yml new file mode 100644 index 00000000000..548f3e8f290 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/fields/fields.yml @@ -0,0 +1,24 @@ +- name: "@timestamp" + type: date + description: Daily timestamp representing when the metric was recorded. +- name: cluster_name + type: keyword + description: Name of the Elasticsearch cluster. +- name: tier + type: keyword + description: The data tier where the index resides (e.g. hot, warm, cold, frozen). +- name: tier_sum_indexing_time + type: long + description: Total indexing time in milliseconds for the tier. +- name: tier_sum_query_time + type: long + description: Total query time in milliseconds for the tier. +- name: tier_sum_store_size + type: long + description: Total primary shard store size in bytes for the tier. +- name: tier_sum_data_set_store_size + type: long + description: Total dataset size in bytes for the tier. +- name: composite_key + type: keyword + description: Composite key used for billing attribution (set by ingest pipeline) consisting of date and data tier. diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/manifest.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/manifest.yml new file mode 100644 index 00000000000..bf148d40424 --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/manifest.yml @@ -0,0 +1,8 @@ +start: true +destination_index_template: + settings: + index: + mode: "lookup" + codec: best_compression + mappings: + dynamic: false diff --git a/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml new file mode 100644 index 00000000000..adb9383f1fc --- /dev/null +++ b/packages/chargeback/elasticsearch/transform/cluster_tier_contribution/transform.yml @@ -0,0 +1,61 @@ +description: Aggregates daily total ECU usage per tier from billing metrics, using ingested timestamps with a 1-hour sync delay and running every 60 minutes. +source: + index: + - "monitoring-indices" + - "*:monitoring-indices" # Dependency: ES Integration added, but not neceserily installed. Need the output of the transform only. +dest: + index: cluster_tier_contribution_lookup + pipeline: 0.2.10-usage +frequency: 60m +sync: + time: + field: "@timestamp" + delay: 1h +pivot: + group_by: + "@timestamp": + date_histogram: + field: "@timestamp" + calendar_interval: 1d + cluster_name: + terms: + field: elasticsearch.cluster.name + tier: + terms: + script: + lang: painless + source: > + if (doc.containsKey('elasticsearch.index.tier') && !doc['elasticsearch.index.tier'].empty) { + return doc['elasticsearch.index.tier'].value; + } + + def pref = doc.containsKey('elasticsearch.index.tier_preference') && !doc['elasticsearch.index.tier_preference'].empty ? doc['elasticsearch.index.tier_preference'].value : null; + if (pref == null) return 'unknown'; + if (pref.contains('data_hot') || pref.contains('data_content')) return 'hot/content'; + if (pref.contains('data_warm')) return 'warm'; + if (pref.contains('data_cold')) return 'cold'; + if (pref.contains('data_frozen')) return 'frozen'; + return 'unknown'; + aggregations: + tier_sum_indexing_time: + sum: + field: elasticsearch.index.total.indexing.index_time_in_millis + tier_sum_query_time: + sum: + field: elasticsearch.index.total.search.query_time_in_millis + tier_sum_store_size: + sum: + field: elasticsearch.index.total.store.size_in_bytes + tier_sum_data_set_store_size: + sum: + field: elasticsearch.index.primaries.store.total_data_set_size_in_bytes +settings: + # This is required to prevent the transform from clobbering the Fleet-managed mappings. + deduce_mappings: false + unattended: true +_meta: + managed: true + run_as_kibana_system: false + # Bump this version to delete, reinstall, and restart the transform during package. + # Version bump is needed if there is any code change in transform. + fleet_transform_version: 0.2.9 \ No newline at end of file diff --git a/packages/chargeback/img/chargeback.png b/packages/chargeback/img/chargeback.png new file mode 100644 index 00000000000..971f878e988 Binary files /dev/null and b/packages/chargeback/img/chargeback.png differ diff --git a/packages/chargeback/img/elastic-logo.png b/packages/chargeback/img/elastic-logo.png new file mode 100644 index 00000000000..d7a56a3ecc0 Binary files /dev/null and b/packages/chargeback/img/elastic-logo.png differ diff --git a/packages/chargeback/img/elastic-logo.svg b/packages/chargeback/img/elastic-logo.svg new file mode 100644 index 00000000000..6268dd88f3b --- /dev/null +++ b/packages/chargeback/img/elastic-logo.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/packages/chargeback/kibana/alerting_rule_template/chargeback-deployment-missing-usage-data.json b/packages/chargeback/kibana/alerting_rule_template/chargeback-deployment-missing-usage-data.json new file mode 100644 index 00000000000..f3ed9a70802 --- /dev/null +++ b/packages/chargeback/kibana/alerting_rule_template/chargeback-deployment-missing-usage-data.json @@ -0,0 +1,25 @@ +{ + "id": "chargeback-deployment-missing-usage-data", + "type": "alerting_rule_template", + "attributes": { + "name": "[Chargeback] Deployment with chargeback group missing usage data", + "tags": ["Chargeback"], + "ruleTypeId": ".es-query", + "schedule": { + "interval": "1h" + }, + "params": { + "searchType": "esqlQuery", + "timeWindowSize": 3, + "timeWindowUnit": "d", + "esqlQuery": { + "esql": "FROM billing_cluster_cost_lookup\n| WHERE deployment_group != \"\"\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| WHERE cluster_name IS NULL\n| INLINE STATS count = COUNT(*) BY deployment_id, deployment_name, deployment_group\n| EVAL result = CONCAT(\"Deployment `\", deployment_name,\"` (`\", deployment_id,\"`) in deployment group `\", deployment_group, \"` did not have usage data since \", left(composite_key,10),\".\")\n| STATS result = VALUES(result)\n| MV_EXPAND result" + }, + "size": 100, + "threshold": [0], + "thresholdComparator": ">", + "timeField": "@timestamp", + "excludeHitsFromPreviousRun": true + } + } +} diff --git a/packages/chargeback/kibana/alerting_rule_template/chargeback-new-group-detected.json b/packages/chargeback/kibana/alerting_rule_template/chargeback-new-group-detected.json new file mode 100644 index 00000000000..cd79993ecfd --- /dev/null +++ b/packages/chargeback/kibana/alerting_rule_template/chargeback-new-group-detected.json @@ -0,0 +1,25 @@ +{ + "id": "chargeback-new-group-detected", + "type": "alerting_rule_template", + "attributes": { + "name": "[Chargeback] New chargeback group detected", + "tags": ["Chargeback"], + "ruleTypeId": ".es-query", + "schedule": { + "interval": "1h" + }, + "params": { + "searchType": "esqlQuery", + "timeWindowSize": 3, + "timeWindowUnit": "d", + "esqlQuery": { + "esql": "FROM billing_cluster_cost_lookup | STATS count = COUNT(*) BY deployment_group | SORT deployment_group | KEEP deployment_group" + }, + "size": 100, + "threshold": [0], + "thresholdComparator": ">", + "timeField": "@timestamp", + "excludeHitsFromPreviousRun": true + } + } +} diff --git a/packages/chargeback/kibana/alerting_rule_template/chargeback-transform-health.json b/packages/chargeback/kibana/alerting_rule_template/chargeback-transform-health.json new file mode 100644 index 00000000000..1e19bba294a --- /dev/null +++ b/packages/chargeback/kibana/alerting_rule_template/chargeback-transform-health.json @@ -0,0 +1,26 @@ +{ + "id": "chargeback-transform-health", + "type": "alerting_rule_template", + "attributes": { + "name": "[Chargeback] Transform health monitoring", + "tags": ["Chargeback", "Transforms"], + "ruleTypeId": "transform_health", + "schedule": { + "interval": "1m" + }, + "params": { + "includeTransforms": [ + "logs-chargeback.billing_cluster_cost-*", + "logs-chargeback.cluster_deployment_contribution-*", + "logs-chargeback.cluster_datastream_contribution-*", + "logs-chargeback.cluster_tier_contribution-*", + "logs-chargeback.cluster_tier_and_ds_contribution-*" + ], + "excludeTransforms": null, + "testsConfig": null + }, + "alertDelay": { + "active": 1 + } + } +} diff --git a/packages/chargeback/kibana/dashboard/chargeback-39a39857-746c-4a29-adca-3c2fcb6bcfb6.json b/packages/chargeback/kibana/dashboard/chargeback-39a39857-746c-4a29-adca-3c2fcb6bcfb6.json new file mode 100644 index 00000000000..7bee717f006 --- /dev/null +++ b/packages/chargeback/kibana/dashboard/chargeback-39a39857-746c-4a29-adca-3c2fcb6bcfb6.json @@ -0,0 +1,4411 @@ +{ + "attributes": { + "controlGroupInput": { + "chainingSystem": "HIERARCHICAL", + "controlStyle": "oneLine", + "ignoreParentSettingsJSON": { + "ignoreFilters": false, + "ignoreQuery": false, + "ignoreTimerange": false, + "ignoreValidations": false + }, + "panelsJSON": { + "2c7df183-8a05-4ee0-b7db-415e874636cd": { + "explicitInput": { + "dataViewId": "chargeback_integration", + "exclude": false, + "existsSelected": false, + "fieldName": "deployment_name", + "searchTechnique": "prefix", + "selectedOptions": [], + "sort": { + "by": "_count", + "direction": "desc" + }, + "title": "Deployment name" + }, + "grow": false, + "order": 1, + "type": "optionsListControl", + "width": "medium" + }, + "b680ef70-4697-41e5-bf5f-e7e1e700ec3e": { + "explicitInput": { + "dataViewId": "chargeback_integration", + "exclude": false, + "existsSelected": false, + "fieldName": "deployment_group", + "searchTechnique": "prefix", + "selectedOptions": [], + "sort": { + "by": "_count", + "direction": "desc" + }, + "title": "Deployment group" + }, + "grow": false, + "order": 0, + "type": "optionsListControl", + "width": "medium" + } + }, + "showApplySelections": false + }, + "description": "Provides visual insights into cluster-level cost and usage metrics, helping track Elastic consumption across tiers, deployments, and data streams. Built from transformed billing and usage data, this dashboard enables transparent chargeback calculations and trend analysis.", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "adHocDataViews": { + "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f": { + "allowHidden": false, + "allowNoIndex": false, + "fieldFormats": {}, + "id": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", + "managed": false, + "name": "billing_cluster_cost_lookup", + "runtimeFieldMap": {}, + "sourceFilters": [], + "timeFieldName": "@timestamp", + "title": "billing_cluster_cost_lookup", + "type": "esql" + } + }, + "datasourceStates": { + "textBased": { + "indexPatternRefs": [ + { + "id": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", + "timeField": "@timestamp", + "title": "billing_cluster_cost_lookup" + } + ], + "layers": { + "aa55ae31-395f-4738-b9fd-3da1d649d640": { + "columns": [ + { + "columnId": "@timestamp", + "customLabel": false, + "fieldName": "@timestamp", + "label": "@timestamp", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "deployment", + "customLabel": false, + "fieldName": "deployment", + "label": "deployment", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "total_value", + "customLabel": false, + "fieldName": "total_value", + "inMetricDimension": true, + "label": "total_value", + "meta": { + "esType": "double", + "type": "number" + } + } + ], + "index": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL total_ecu_value = total_ecu * conf_ecu_rate, deployment = CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\")\n| STATS `total_value`=SUM(total_ecu_value) by @timestamp, deployment\n| KEEP @timestamp, deployment, total_value | SORT @timestamp desc | LIMIT 5000" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL total_ecu_value = total_ecu * conf_ecu_rate, deployment = CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\")\n| STATS `total_value`=SUM(total_ecu_value) by @timestamp, deployment\n| KEEP @timestamp, deployment, total_value | SORT @timestamp desc | LIMIT 5000" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "total_value" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "default", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "aa55ae31-395f-4738-b9fd-3da1d649d640", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "deployment", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": true, + "position": "right" + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "total_value over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL total_ecu_value = total_ecu * conf_ecu_rate, deployment = CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\")\n| STATS `total_value`=SUM(total_ecu_value) by @timestamp, deployment\n| KEEP @timestamp, deployment, total_value | SORT @timestamp desc | LIMIT 5000" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "timeRange": { + "from": "now-6M", + "to": "now" + }, + "title": "Normalised cost over the last 6 months" + }, + "gridData": { + "h": 11, + "i": "d9069c7e-bfaf-4045-aedf-ca1042c21010", + "w": 48, + "x": 0, + "y": 0 + }, + "panelIndex": "d9069c7e-bfaf-4045-aedf-ca1042c21010", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "adHocDataViews": { + "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f": { + "allowHidden": false, + "allowNoIndex": false, + "fieldFormats": {}, + "id": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", + "managed": false, + "name": "billing_cluster_cost_lookup", + "runtimeFieldMap": {}, + "sourceFilters": [], + "timeFieldName": "@timestamp", + "title": "billing_cluster_cost_lookup", + "type": "esql" + } + }, + "datasourceStates": { + "textBased": { + "indexPatternRefs": [ + { + "id": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", + "timeField": "@timestamp", + "title": "billing_cluster_cost_lookup" + } + ], + "layers": { + "28174526-fb0d-4645-a2b7-f5101203fb4e": { + "columns": [ + { + "columnId": "dff61ce4-8b3c-40bf-8972-0970a0a7ecc9", + "customLabel": true, + "fieldName": "deployment_group", + "label": "Deployment group", + "meta": { + "esType": "keyword", + "params": { + "id": "string" + }, + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup", + "sourceField": "deployment_group" + }, + "type": "string" + } + }, + { + "columnId": "04726936-593d-48df-9ebc-9963d006ad1c", + "fieldName": "Normalised cost", + "label": "Normalised cost", + "meta": { + "esType": "double", + "params": { + "id": "number" + }, + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup", + "sourceField": "Normalised cost" + }, + "type": "number" + } + } + ], + "index": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| STATS `Normalised cost` = SUM(total_ecu * conf_ecu_rate) by deployment_group" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| STATS `Normalised cost` = SUM(total_ecu * conf_ecu_rate) by deployment_group" + }, + "visualization": { + "columns": [ + { + "columnId": "dff61ce4-8b3c-40bf-8972-0970a0a7ecc9", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "04726936-593d-48df-9ebc-9963d006ad1c", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "28174526-fb0d-4645-a2b7-f5101203fb4e", + "layerType": "data" + } + }, + "title": "Normalised cost", + "version": 1, + "visualizationType": "lnsDatatable" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| STATS `Normalised cost` = SUM(total_ecu * conf_ecu_rate) by deployment_group" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false + }, + "gridData": { + "h": 14, + "i": "43ab691d-4b1f-4e83-b2e4-026c16f6a5eb", + "sectionId": "2b8b2365-5e05-4f1d-8880-f35892a8e60f", + "w": 9, + "x": 0, + "y": 0 + }, + "panelIndex": "43ab691d-4b1f-4e83-b2e4-026c16f6a5eb", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "adHocDataViews": { + "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f": { + "allowHidden": false, + "allowNoIndex": false, + "fieldFormats": {}, + "id": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", + "managed": false, + "name": "billing_cluster_cost_lookup", + "runtimeFieldMap": {}, + "sourceFilters": [], + "timeFieldName": "@timestamp", + "title": "billing_cluster_cost_lookup", + "type": "esql" + } + }, + "datasourceStates": { + "textBased": { + "indexPatternRefs": [ + { + "id": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", + "timeField": "@timestamp", + "title": "billing_cluster_cost_lookup" + } + ], + "layers": { + "c6398250-abb1-4935-a3a8-41af3c8ea057": { + "columns": [ + { + "columnId": "@timestamp", + "customLabel": false, + "fieldName": "@timestamp", + "label": "@timestamp", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "deployment_group", + "customLabel": false, + "fieldName": "deployment_group", + "label": "deployment_group", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "total_value", + "customLabel": false, + "fieldName": "total_value", + "inMetricDimension": true, + "label": "total_value", + "meta": { + "esType": "double", + "type": "number" + } + } + ], + "index": "2bf6c0d816ef0a2d56d03ede549c16c08c35db2cf02d78c12756a98a33f50e4f", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL total_ecu_value = total_ecu * conf_ecu_rate\n| STATS `total_value`=SUM(total_ecu_value) by @timestamp, deployment_group\n| KEEP @timestamp, deployment_group, total_value | SORT @timestamp desc | LIMIT 5000" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL total_ecu_value = total_ecu * conf_ecu_rate\n| STATS `total_value`=SUM(total_ecu_value) by @timestamp, deployment_group\n| KEEP @timestamp, deployment_group, total_value | SORT @timestamp desc | LIMIT 5000" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "total_value" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "default", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "c6398250-abb1-4935-a3a8-41af3c8ea057", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "deployment_group", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": true, + "position": "right" + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "total_value over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL total_ecu_value = total_ecu * conf_ecu_rate\n| STATS `total_value`=SUM(total_ecu_value) by @timestamp, deployment_group\n| KEEP @timestamp, deployment_group, total_value | SORT @timestamp desc | LIMIT 5000" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "timeRange": { + "from": "now-6M", + "to": "now" + }, + "title": "Normalised cost over the last 6 months per deployment group" + }, + "gridData": { + "h": 14, + "i": "003538c5-ca13-4130-949f-2b70047abb38", + "sectionId": "2b8b2365-5e05-4f1d-8880-f35892a8e60f", + "w": 39, + "x": 9, + "y": 0 + }, + "panelIndex": "003538c5-ca13-4130-949f-2b70047abb38", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "# Deployment statistics", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 3, + "i": "08bdbcda-fa42-4fb2-bb50-b4ed2620ef6b", + "sectionId": "5092dfac-3f29-4eab-8c8f-f97bf1479370", + "w": 48, + "x": 0, + "y": 0 + }, + "panelIndex": "08bdbcda-fa42-4fb2-bb50-b4ed2620ef6b", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "ba345378-8ee0-40ec-9fb0-d26588d09107": { + "columns": [ + { + "columnId": "c33c8ff5-8a22-4f4f-9b92-5e16485f5d9a", + "customLabel": true, + "fieldName": "deployment", + "label": "Deployment", + "meta": { + "esType": "keyword", + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup" + }, + "type": "string" + } + }, + { + "columnId": "2ab259ce-29c3-4c5c-817b-b7f036c62be8", + "customLabel": true, + "fieldName": "agg_blended", + "label": "Normalised cost", + "meta": { + "esType": "double", + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup" + }, + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "af3de6ff-d14d-4cea-bb56-4ee9012054b6", + "customLabel": true, + "fieldName": "tier", + "label": "Data tier", + "meta": { + "esType": "keyword", + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup" + }, + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu), querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu), data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu), store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu), storage = CASE (store == 0, data_set, store), total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight, total_weight_cold = conf_storage_weight + conf_query_weight, blended = CASE (tier == \"hot/content\", ((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight)) / total_weight_hot, ((storage * conf_storage_weight) + (querying * conf_query_weight)) / total_weight_cold) * conf_ecu_rate, deployment = CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\")\n| STATS agg_blended = sum(blended) BY deployment, tier\n| KEEP deployment, tier, agg_blended\n| WHERE agg_blended \u003e 0" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu), querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu), data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu), store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu), storage = CASE (store == 0, data_set, store), total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight, total_weight_cold = conf_storage_weight + conf_query_weight, blended = CASE (tier == \"hot/content\", ((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight)) / total_weight_hot, ((storage * conf_storage_weight) + (querying * conf_query_weight)) / total_weight_cold) * conf_ecu_rate, deployment = CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\")\n| STATS agg_blended = sum(blended) BY deployment, tier\n| KEEP deployment, tier, agg_blended\n| WHERE agg_blended \u003e 0" + }, + "visualization": { + "columns": [ + { + "columnId": "c33c8ff5-8a22-4f4f-9b92-5e16485f5d9a", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "2ab259ce-29c3-4c5c-817b-b7f036c62be8", + "isMetric": true, + "isTransposed": false + }, + { + "columnId": "af3de6ff-d14d-4cea-bb56-4ee9012054b6", + "isMetric": false, + "isTransposed": true + } + ], + "layerId": "ba345378-8ee0-40ec-9fb0-d26588d09107", + "layerType": "data" + } + }, + "title": "Table ", + "version": 1, + "visualizationType": "lnsDatatable" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu), querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu), data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu), store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu), storage = CASE (store == 0, data_set, store), total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight, total_weight_cold = conf_storage_weight + conf_query_weight, blended = CASE (tier == \"hot/content\", ((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight)) / total_weight_hot, ((storage * conf_storage_weight) + (querying * conf_query_weight)) / total_weight_cold) * conf_ecu_rate, deployment = CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\")\n| STATS agg_blended = sum(blended) BY deployment, tier\n| KEEP deployment, tier, agg_blended\n| WHERE agg_blended \u003e 0" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Normalised cost per deployment and tier" + }, + "gridData": { + "h": 15, + "i": "8049f13b-c56d-4a6e-a4e3-5cf4274fef60", + "sectionId": "5092dfac-3f29-4eab-8c8f-f97bf1479370", + "w": 24, + "x": 0, + "y": 3 + }, + "panelIndex": "8049f13b-c56d-4a6e-a4e3-5cf4274fef60", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "91b4be84-6eb5-4b77-ad13-b85d992f3bbf": { + "columns": [ + { + "columnId": "deployment", + "customLabel": true, + "fieldName": "deployment", + "label": "Deployment", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "tier", + "customLabel": true, + "fieldName": "tier", + "label": "Data tier", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "agg_blended", + "customLabel": true, + "fieldName": "agg_blended", + "inMetricDimension": true, + "label": "Normalised cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu), querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu), data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu), store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu), storage = CASE (store == 0, data_set, store), total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight, total_weight_cold = conf_storage_weight + conf_query_weight, blended = CASE (tier == \"hot/content\", ((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight)) / total_weight_hot, ((storage * conf_storage_weight) + (querying * conf_query_weight)) / total_weight_cold) * conf_ecu_rate, deployment = CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\")\n| STATS agg_blended = sum(blended) BY deployment, tier\n| KEEP deployment, tier, agg_blended\n| WHERE agg_blended \u003e 0" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu), querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu), data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu), store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu), storage = CASE (store == 0, data_set, store), total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight, total_weight_cold = conf_storage_weight + conf_query_weight, blended = CASE (tier == \"hot/content\", ((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight)) / total_weight_hot, ((storage * conf_storage_weight) + (querying * conf_query_weight)) / total_weight_cold) * conf_ecu_rate, deployment = CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\")\n| STATS agg_blended = sum(blended) BY deployment, tier\n| KEEP deployment, tier, agg_blended\n| WHERE agg_blended \u003e 0" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "91b4be84-6eb5-4b77-ad13-b85d992f3bbf", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_blended" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "deployment", + "tier" + ] + } + ], + "shape": "treemap" + } + }, + "title": "Pie", + "version": 1, + "visualizationType": "lnsPie" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu), querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu), data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu), store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu), storage = CASE (store == 0, data_set, store), total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight, total_weight_cold = conf_storage_weight + conf_query_weight, blended = CASE (tier == \"hot/content\", ((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight)) / total_weight_hot, ((storage * conf_storage_weight) + (querying * conf_query_weight)) / total_weight_cold) * conf_ecu_rate, deployment = CONCAT(deployment_name, \" (\", LEFT(deployment_id, 6), \")\")\n| STATS agg_blended = sum(blended) BY deployment, tier\n| KEEP deployment, tier, agg_blended\n| WHERE agg_blended \u003e 0" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Normalised cost per deployment and tier" + }, + "gridData": { + "h": 15, + "i": "normalised-cost-per-deployment-and-tier", + "sectionId": "5092dfac-3f29-4eab-8c8f-f97bf1479370", + "w": 24, + "x": 24, + "y": 3 + }, + "panelIndex": "normalised-cost-per-deployment-and-tier", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "# Data tier and data stream overview", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "784ed861-b61f-4c71-a3bc-02cb513a8019", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 48, + "x": 0, + "y": 0 + }, + "panelIndex": "784ed861-b61f-4c71-a3bc-02cb513a8019", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "9d199172-f9fa-4fc4-82af-c7e2672a3f54": { + "columns": [ + { + "columnId": "a76076ce-f372-4a48-857d-75010bbf5970", + "fieldName": "tier", + "label": "tier", + "meta": { + "esType": "keyword", + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup" + }, + "type": "string" + } + }, + { + "columnId": "6e881cac-4719-493c-95f8-e3bfa38c3219", + "customLabel": true, + "fieldName": "agg_indexing", + "label": "Indexing", + "meta": { + "esType": "double", + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup" + }, + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "97a86307-08fc-4b2d-a82a-36a2ecc9c75d", + "customLabel": true, + "fieldName": "agg_querying", + "label": "Querying", + "meta": { + "esType": "double", + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup" + }, + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "2b086331-8a38-4df2-b042-7827d3519d63", + "customLabel": true, + "fieldName": "agg_storage", + "label": "Storage", + "meta": { + "esType": "double", + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup" + }, + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "307273f7-1b4c-450c-80cd-ef0326b38f6e", + "customLabel": true, + "fieldName": "agg_blended", + "label": "Blended", + "meta": { + "esType": "double", + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup" + }, + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n agg_indexing = sum(indexing),\n agg_querying = sum(querying),\n agg_storage = sum(storage),\n agg_blended = sum(blended)\n BY\n tier\n| WHERE agg_blended \u003e 0" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n agg_indexing = sum(indexing),\n agg_querying = sum(querying),\n agg_storage = sum(storage),\n agg_blended = sum(blended)\n BY\n tier\n| WHERE agg_blended \u003e 0" + }, + "visualization": { + "columns": [ + { + "columnId": "a76076ce-f372-4a48-857d-75010bbf5970", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "6e881cac-4719-493c-95f8-e3bfa38c3219", + "isMetric": true, + "isTransposed": false + }, + { + "columnId": "97a86307-08fc-4b2d-a82a-36a2ecc9c75d", + "isMetric": true, + "isTransposed": false + }, + { + "columnId": "2b086331-8a38-4df2-b042-7827d3519d63", + "isMetric": true, + "isTransposed": false + }, + { + "columnId": "307273f7-1b4c-450c-80cd-ef0326b38f6e", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "9d199172-f9fa-4fc4-82af-c7e2672a3f54", + "layerType": "data" + } + }, + "title": "agg_indexing \u0026 agg_querying \u0026 agg_storage \u0026 agg_blended of tier", + "version": 1, + "visualizationType": "lnsDatatable" + }, + "description": "Normalised cost breakdown per tier, based on Indexing, Querying, Storage and a blended ratio of all of these.", + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n agg_indexing = sum(indexing),\n agg_querying = sum(querying),\n agg_storage = sum(storage),\n agg_blended = sum(blended)\n BY\n tier\n| WHERE agg_blended \u003e 0" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Normalised cost per tier" + }, + "gridData": { + "h": 10, + "i": "18643569-92d3-4a30-aafe-6ae11acc4c58", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 48, + "x": 0, + "y": 3 + }, + "panelIndex": "18643569-92d3-4a30-aafe-6ae11acc4c58", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Indexing", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "4d8d5ff0-b63f-4d39-be03-e6f85a8b60ce", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 12, + "x": 0, + "y": 13 + }, + "panelIndex": "4d8d5ff0-b63f-4d39-be03-e6f85a8b60ce", + "type": "visualization" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Querying", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "40a91be7-f01d-47ce-ba7f-683bcee42626", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 12, + "x": 12, + "y": 13 + }, + "panelIndex": "40a91be7-f01d-47ce-ba7f-683bcee42626", + "type": "visualization" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Storage", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "d0c647b4-4b26-49d3-b54a-1ddbdf995143", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 12, + "x": 24, + "y": 13 + }, + "panelIndex": "d0c647b4-4b26-49d3-b54a-1ddbdf995143", + "type": "visualization" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Blended", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "540d6135-d996-43ed-a192-30d51e2e5454", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 12, + "x": 36, + "y": 13 + }, + "panelIndex": "540d6135-d996-43ed-a192-30d51e2e5454", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "0f6d5278-b325-49bc-9eb7-2febd2a5de0f": { + "columns": [ + { + "columnId": "agg_indexing", + "customLabel": true, + "fieldName": "agg_indexing", + "inMetricDimension": true, + "label": "Indexing", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n datastream\n| WHERE agg_indexing \u003e 0\n| SORT agg_indexing DESC\n| Limit 20" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n datastream\n| WHERE agg_indexing \u003e 0\n| SORT agg_indexing DESC\n| Limit 20" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "emptySizeRatio": 0.3, + "layerId": "0f6d5278-b325-49bc-9eb7-2febd2a5de0f", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_indexing" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "datastream" + ] + } + ], + "shape": "donut" + } + }, + "title": "Bar vertical stacked", + "version": 1, + "visualizationType": "lnsPie" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n datastream\n| WHERE agg_indexing \u003e 0\n| SORT agg_indexing DESC\n| Limit 20" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Indexing: Normalised cost totals by data stream" + }, + "gridData": { + "h": 15, + "i": "ed65354e-848c-4d93-bcd6-b9c50793e242", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 12, + "x": 0, + "y": 16 + }, + "panelIndex": "ed65354e-848c-4d93-bcd6-b9c50793e242", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "3b03b6a8-35ae-4554-8bd6-a887d8b36748": { + "columns": [ + { + "columnId": "agg_querying", + "customLabel": true, + "fieldName": "agg_querying", + "inMetricDimension": true, + "label": "Querying", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying)\n BY \n datastream\n| WHERE agg_querying \u003e 0\n| SORT agg_querying DESC\n| Limit 20" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying)\n BY \n datastream\n| WHERE agg_querying \u003e 0\n| SORT agg_querying DESC\n| Limit 20" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "emptySizeRatio": 0.3, + "layerId": "3b03b6a8-35ae-4554-8bd6-a887d8b36748", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_querying" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "datastream" + ] + } + ], + "shape": "donut" + } + }, + "title": "Bar vertical stacked", + "version": 1, + "visualizationType": "lnsPie" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying)\n BY \n datastream\n| WHERE agg_querying \u003e 0\n| SORT agg_querying DESC\n| Limit 20" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Querying: Normalised cost totals by data stream" + }, + "gridData": { + "h": 15, + "i": "4a2ecf27-21c7-4fff-986b-05b25e0ce8f8", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 12, + "x": 12, + "y": 16 + }, + "panelIndex": "4a2ecf27-21c7-4fff-986b-05b25e0ce8f8", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "2813628b-2a20-484c-9a19-11955d86e9ff": { + "columns": [ + { + "columnId": "agg_storage", + "customLabel": true, + "fieldName": "agg_storage", + "inMetricDimension": true, + "label": "Storage", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n datastream\n| WHERE agg_storage \u003e 0\n| SORT agg_storage DESC\n| limit 20" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n datastream\n| WHERE agg_storage \u003e 0\n| SORT agg_storage DESC\n| limit 20" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "emptySizeRatio": 0.3, + "layerId": "2813628b-2a20-484c-9a19-11955d86e9ff", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_storage" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "datastream" + ] + } + ], + "shape": "donut" + } + }, + "title": "Bar vertical stacked", + "version": 1, + "visualizationType": "lnsPie" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n datastream\n| WHERE agg_storage \u003e 0\n| SORT agg_storage DESC\n| limit 20" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Storage: Normalised cost totals by data stream" + }, + "gridData": { + "h": 15, + "i": "3269b898-7190-40b6-ad0e-d29fcf98e762", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 12, + "x": 24, + "y": 16 + }, + "panelIndex": "3269b898-7190-40b6-ad0e-d29fcf98e762", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "ee58e17d-eba7-48a7-bae8-2df3607d9653": { + "columns": [ + { + "columnId": "agg_blended", + "customLabel": false, + "fieldName": "agg_blended", + "inMetricDimension": true, + "label": "agg_blended", + "meta": { + "esType": "double", + "type": "number" + } + }, + { + "columnId": "datastream", + "customLabel": false, + "fieldName": "datastream", + "label": "datastream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu),\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu),\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n agg_blended = sum(blended)\n BY\n datastream\n| WHERE agg_blended \u003e 0\n| SORT agg_blended DESC\n| limit 20" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu),\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu),\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n agg_blended = sum(blended)\n BY\n datastream\n| WHERE agg_blended \u003e 0\n| SORT agg_blended DESC\n| limit 20" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "default", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "ee58e17d-eba7-48a7-bae8-2df3607d9653", + "layerType": "data", + "legendDisplay": "default", + "metrics": [ + "agg_blended" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "datastream" + ] + } + ], + "shape": "pie" + } + }, + "title": "Treemap", + "version": 1, + "visualizationType": "lnsPie" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu),\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu),\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot,\n TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight))) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n agg_blended = sum(blended)\n BY\n datastream\n| WHERE agg_blended \u003e 0\n| SORT agg_blended DESC\n| limit 20" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Blended: Normalised cost totals by data stream" + }, + "gridData": { + "h": 15, + "i": "cb467fdf-5cb1-41b5-8235-8b823cbd9310", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 12, + "x": 36, + "y": 16 + }, + "panelIndex": "cb467fdf-5cb1-41b5-8235-8b823cbd9310", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "23693fb0-43a5-4f95-bd77-d6a54e18e34a": { + "columns": [ + { + "columnId": "Data stream", + "customLabel": false, + "fieldName": "Data stream", + "inMetricDimension": true, + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "Data tier", + "customLabel": false, + "fieldName": "Data tier", + "inMetricDimension": true, + "label": "Data tier", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "Indexing", + "customLabel": false, + "fieldName": "Indexing", + "inMetricDimension": true, + "label": "Indexing", + "meta": { + "esType": "double", + "type": "number" + } + }, + { + "columnId": "Querying", + "customLabel": false, + "fieldName": "Querying", + "inMetricDimension": true, + "label": "Querying", + "meta": { + "esType": "double", + "type": "number" + } + }, + { + "columnId": "Storage", + "customLabel": false, + "fieldName": "Storage", + "inMetricDimension": true, + "label": "Storage", + "meta": { + "esType": "double", + "type": "number" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE(deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE(deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE(deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE(deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE(store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE(\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n Indexing = round(sum(indexing),2),\n Querying = round(sum(querying),2),\n Storage = round(sum(storage),2),\n Blended = round(sum(blended),2)\n BY datastream, tier\n| SORT Blended DESC\n| RENAME tier as `Data tier`, datastream as `Data stream`\n| Keep `Data stream`, `Data tier`, Indexing, Querying, Storage, Blended\n| WHERE `Data stream` is not null" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE(deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE(deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE(deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE(deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE(store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE(\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n Indexing = round(sum(indexing),2),\n Querying = round(sum(querying),2),\n Storage = round(sum(storage),2),\n Blended = round(sum(blended),2)\n BY datastream, tier\n| SORT Blended DESC\n| RENAME tier as `Data tier`, datastream as `Data stream`\n| Keep `Data stream`, `Data tier`, Indexing, Querying, Storage, Blended\n| WHERE `Data stream` is not null" + }, + "visualization": { + "columns": [ + { + "columnId": "Data stream" + }, + { + "columnId": "Data tier" + }, + { + "columnId": "Indexing" + }, + { + "columnId": "Querying" + }, + { + "columnId": "Storage" + } + ], + "layerId": "23693fb0-43a5-4f95-bd77-d6a54e18e34a", + "layerType": "data" + } + }, + "title": "Indexing \u0026 Querying \u0026 Storage of Data tier", + "version": 1, + "visualizationType": "lnsDatatable" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_tier_and_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE(deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_and_datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE(deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_and_datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE(deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_and_datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE(deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_and_datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE(store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE(\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n Indexing = round(sum(indexing),2),\n Querying = round(sum(querying),2),\n Storage = round(sum(storage),2),\n Blended = round(sum(blended),2)\n BY datastream, tier\n| SORT Blended DESC\n| RENAME tier as `Data tier`, datastream as `Data stream`\n| Keep `Data stream`, `Data tier`, Indexing, Querying, Storage, Blended\n| WHERE `Data stream` is not null" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Normalised cost per data stream per tier" + }, + "gridData": { + "h": 15, + "i": "964a716b-bb61-4cc0-ba77-c1e73e2da836", + "sectionId": "96918af7-e003-4af8-8c37-245db141d60c", + "w": 48, + "x": 0, + "y": 31 + }, + "panelIndex": "964a716b-bb61-4cc0-ba77-c1e73e2da836", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "# Data tier and data stream per day", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "b1e12ff9-83d2-4dd7-a16e-ec4597d14dbc", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 48, + "x": 0, + "y": 0 + }, + "panelIndex": "b1e12ff9-83d2-4dd7-a16e-ec4597d14dbc", + "type": "visualization" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Indexing details", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "9cd4730b-d3f5-438e-a816-4d5aebf1eae6", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 48, + "x": 0, + "y": 3 + }, + "panelIndex": "9cd4730b-d3f5-438e-a816-4d5aebf1eae6", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "0ee24895-1496-4b23-96ec-ec09a8f49332": { + "columns": [ + { + "columnId": "agg_indexing", + "customLabel": true, + "fieldName": "agg_indexing", + "inMetricDimension": true, + "label": "Normalised cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp, \n datastream\n| WHERE agg_indexing \u003e 0\n| SORT agg_indexing DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp, \n datastream\n| WHERE agg_indexing \u003e 0\n| SORT agg_indexing DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_indexing" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "0ee24895-1496-4b23-96ec-ec09a8f49332", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "datastream", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_indexing over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp, \n datastream\n| WHERE agg_indexing \u003e 0\n| SORT agg_indexing DESC" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Indexing: Normalised per data stream per day" + }, + "gridData": { + "h": 14, + "i": "dafb7c36-3a1d-4cbb-9944-51beedf535f6", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 0, + "y": 6 + }, + "panelIndex": "dafb7c36-3a1d-4cbb-9944-51beedf535f6", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "d24af063-ec41-4736-bf4e-70f1fe4da4cf": { + "columns": [ + { + "columnId": "agg_indexing", + "customLabel": true, + "fieldName": "agg_indexing", + "inMetricDimension": true, + "label": "Normalised cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "tier", + "customLabel": true, + "fieldName": "tier", + "label": "Data tier", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time \u003e 0,\n TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n tier\n| WHERE agg_indexing \u003e 0" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time \u003e 0,\n TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n tier\n| WHERE agg_indexing \u003e 0" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_indexing" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "d24af063-ec41-4736-bf4e-70f1fe4da4cf", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "tier", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_indexing over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time \u003e 0,\n TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n tier\n| WHERE agg_indexing \u003e 0" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Indexing: Normalised cost per tier per day" + }, + "gridData": { + "h": 14, + "i": "45a29dd3-1990-4717-bf6b-25a5d85d2344", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 16, + "y": 6 + }, + "panelIndex": "45a29dd3-1990-4717-bf6b-25a5d85d2344", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "d5520b0e-567f-4f0a-b424-fc79eb79e2cc": { + "columns": [ + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "agg_indexing", + "customLabel": true, + "fieldName": "agg_indexing", + "inMetricDimension": true, + "label": "% Normalised cost", + "meta": { + "esType": "long", + "type": "number" + }, + "params": { + "format": { + "id": "percent", + "params": { + "decimals": 0 + } + } + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time \u003e 0,\n TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n datastream\n| WHERE agg_indexing \u003e 0\n| SORT agg_indexing DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time \u003e 0,\n TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n datastream\n| WHERE agg_indexing \u003e 0\n| SORT agg_indexing DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_indexing" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "d5520b0e-567f-4f0a-b424-fc79eb79e2cc", + "layerType": "data", + "seriesType": "bar_percentage_stacked", + "splitAccessor": "datastream", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_indexing over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n indexing = CASE (\n deployment_sum_indexing_time \u003e 0,\n TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_indexing = sum(indexing)\n BY \n @timestamp,\n datastream\n| WHERE agg_indexing \u003e 0\n| SORT agg_indexing DESC" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Indexing: % Normalised cost per data stream per day" + }, + "gridData": { + "h": 14, + "i": "9bc8fa5a-d47a-4a79-ab6a-032d726f085f", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 32, + "y": 6 + }, + "panelIndex": "9bc8fa5a-d47a-4a79-ab6a-032d726f085f", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Querying details", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "8de95af1-8e9a-424a-baaf-8cfcb2577f59", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 48, + "x": 0, + "y": 20 + }, + "panelIndex": "8de95af1-8e9a-424a-baaf-8cfcb2577f59", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "1a2f8269-f56f-4c02-8506-76b723acc828": { + "columns": [ + { + "columnId": "agg_querying", + "customLabel": true, + "fieldName": "agg_querying", + "inMetricDimension": true, + "label": "Normalised cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n querying = CASE (\n deployment_sum_query_time \u003e 0,\n TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying) \n BY \n @timestamp, \n datastream\n| WHERE agg_querying \u003e 0\n| SORT agg_querying DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n querying = CASE (\n deployment_sum_query_time \u003e 0,\n TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying) \n BY \n @timestamp, \n datastream\n| WHERE agg_querying \u003e 0\n| SORT agg_querying DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_querying" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "1a2f8269-f56f-4c02-8506-76b723acc828", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "datastream", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_querying over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n querying = CASE (\n deployment_sum_query_time \u003e 0,\n TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying) \n BY \n @timestamp, \n datastream\n| WHERE agg_querying \u003e 0\n| SORT agg_querying DESC" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Querying: Normalised cost per data stream per day" + }, + "gridData": { + "h": 14, + "i": "83bbd574-571b-4401-af7f-d0998cd9af2e", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 0, + "y": 23 + }, + "panelIndex": "83bbd574-571b-4401-af7f-d0998cd9af2e", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "61842c44-afd9-429c-bf2e-18a4a079e5e8": { + "columns": [ + { + "columnId": "agg_querying", + "customLabel": true, + "fieldName": "agg_querying", + "inMetricDimension": true, + "label": "Normalised cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "tier", + "customLabel": false, + "fieldName": "tier", + "label": "tier", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n querying = CASE (\n deployment_sum_query_time \u003e 0,\n TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp,\n tier\n| WHERE agg_querying \u003e 0" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n querying = CASE (\n deployment_sum_query_time \u003e 0,\n TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp,\n tier\n| WHERE agg_querying \u003e 0" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_querying" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "61842c44-afd9-429c-bf2e-18a4a079e5e8", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "tier", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_querying over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| EVAL \n querying = CASE (\n deployment_sum_query_time \u003e 0,\n TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu\n ) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp,\n tier\n| WHERE agg_querying \u003e 0" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Querying: Noralised cost per tier per day" + }, + "gridData": { + "h": 14, + "i": "b0df8c79-87d4-4a7a-8c7b-4752862fb2dd", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 16, + "y": 23 + }, + "panelIndex": "b0df8c79-87d4-4a7a-8c7b-4752862fb2dd", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "91bcc41e-3a2c-434d-8df7-29c01340e712": { + "columns": [ + { + "columnId": "agg_querying", + "customLabel": true, + "fieldName": "agg_querying", + "inMetricDimension": true, + "label": "% Normalised cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp, \n datastream\n| WHERE agg_querying \u003e 0\n| SORT agg_querying DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp, \n datastream\n| WHERE agg_querying \u003e 0\n| SORT agg_querying DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_querying" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "91bcc41e-3a2c-434d-8df7-29c01340e712", + "layerType": "data", + "seriesType": "bar_percentage_stacked", + "splitAccessor": "datastream", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_querying over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate\n| STATS \n agg_querying = sum(querying)\n BY \n @timestamp, \n datastream\n| WHERE agg_querying \u003e 0\n| SORT agg_querying DESC" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Querying: % Normalised cost per data stream per day" + }, + "gridData": { + "h": 14, + "i": "b3e62c24-abb9-431e-a228-cbc5e7b8b02a", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 32, + "y": 23 + }, + "panelIndex": "b3e62c24-abb9-431e-a228-cbc5e7b8b02a", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Storage details", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "4d3f1bfb-ea89-4978-ab6f-9033a507785f", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 48, + "x": 0, + "y": 37 + }, + "panelIndex": "4d3f1bfb-ea89-4978-ab6f-9033a507785f", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee": { + "columns": [ + { + "columnId": "agg_storage", + "customLabel": true, + "fieldName": "agg_storage", + "inMetricDimension": true, + "label": "Normalised cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage \u003e 0\n| SORT agg_storage DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage \u003e 0\n| SORT agg_storage DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_storage" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "datastream", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_storage over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage \u003e 0\n| SORT agg_storage DESC" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Storage: Nomalised cost per data stream per day" + }, + "gridData": { + "h": 15, + "i": "6430d172-9b7f-45a0-a836-c46af0276b19", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 0, + "y": 40 + }, + "panelIndex": "6430d172-9b7f-45a0-a836-c46af0276b19", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "93fd5f12-463f-4e8e-a9b8-0ac77d0522a9": { + "columns": [ + { + "columnId": "agg_storage", + "customLabel": true, + "fieldName": "agg_storage", + "inMetricDimension": true, + "label": "Normalised cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "tier", + "customLabel": true, + "fieldName": "tier", + "label": "Data tier", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp,\n tier\n| WHERE agg_storage \u003e 0" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp,\n tier\n| WHERE agg_storage \u003e 0" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_storage" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "93fd5f12-463f-4e8e-a9b8-0ac77d0522a9", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "tier", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_storage over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp,\n tier\n| WHERE agg_storage \u003e 0" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Storage: Normalised cost per tier per day" + }, + "gridData": { + "h": 15, + "i": "ba7efeab-de6c-4301-8e97-7adb98dae58b", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 16, + "y": 40 + }, + "panelIndex": "ba7efeab-de6c-4301-8e97-7adb98dae58b", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee": { + "columns": [ + { + "columnId": "agg_storage", + "customLabel": true, + "fieldName": "agg_storage", + "inMetricDimension": true, + "label": "% Normalised cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "datastream", + "customLabel": true, + "fieldName": "datastream", + "label": "Data stream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage \u003e 0\n| SORT agg_storage DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage \u003e 0\n| SORT agg_storage DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_storage" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "b72a0ab5-dbf5-4fd2-bdef-c855fe354bee", + "layerType": "data", + "seriesType": "bar_percentage_stacked", + "splitAccessor": "datastream", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_storage over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL \n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate\n| STATS \n agg_storage = sum(storage)\n BY \n @timestamp, \n datastream\n| WHERE agg_storage \u003e 0\n| SORT agg_storage DESC" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Storage: % Normalised cost per data stream per day" + }, + "gridData": { + "h": 15, + "i": "a672f0fa-cdf6-4e27-a2c8-f753eef30a06", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 32, + "y": 40 + }, + "panelIndex": "a672f0fa-cdf6-4e27-a2c8-f753eef30a06", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "params": { + "fontSize": 12, + "markdown": "## Blended details", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + }, + "title": "" + }, + "gridData": { + "h": 3, + "i": "a2cb12f7-9b9e-4bf7-864e-9101107e0c3e", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 48, + "x": 0, + "y": 55 + }, + "panelIndex": "a2cb12f7-9b9e-4bf7-864e-9101107e0c3e", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "52906f25-a114-4d45-b425-f0ff154f4ac1": { + "columns": [ + { + "columnId": "agg_blended", + "customLabel": true, + "fieldName": "agg_blended", + "inMetricDimension": true, + "label": "Normalised cost", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "datastream", + "customLabel": false, + "fieldName": "datastream", + "label": "datastream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu),\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu),\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n blended = TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot * conf_ecu_rate\n| STATS\n agg_blended = sum(blended)\n BY\n @timestamp,\n datastream\n| WHERE agg_blended \u003e 0\n| SORT agg_blended DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu),\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu),\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n blended = TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot * conf_ecu_rate\n| STATS\n agg_blended = sum(blended)\n BY\n @timestamp,\n datastream\n| WHERE agg_blended \u003e 0\n| SORT agg_blended DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "agg_blended" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "52906f25-a114-4d45-b425-f0ff154f4ac1", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "datastream", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_blended over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu),\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu),\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store),\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n blended = TO_DOUBLE(((storage * conf_storage_weight) + (querying * conf_query_weight) + (indexing * conf_indexing_weight))) / total_weight_hot * conf_ecu_rate\n| STATS\n agg_blended = sum(blended)\n BY\n @timestamp,\n datastream\n| WHERE agg_blended \u003e 0\n| SORT agg_blended DESC" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Blended value: Normalised cost per data stream per day" + }, + "gridData": { + "h": 14, + "i": "4c97109a-2048-470f-ae18-8c424ba2c12e", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 0, + "y": 58 + }, + "panelIndex": "4c97109a-2048-470f-ae18-8c424ba2c12e", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "43f9cfe6-d828-4773-ab65-129f92cf7f4d": { + "columns": [ + { + "columnId": "@timestamp", + "customLabel": true, + "fieldName": "@timestamp", + "inMetricDimension": true, + "label": "Date", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "50a91b89-2e68-4a69-90a0-d2ab79115b96", + "customLabel": true, + "fieldName": "tier", + "label": "Data tier", + "meta": { + "esType": "keyword", + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup" + }, + "type": "string" + } + }, + { + "columnId": "1501af72-2509-4ae0-88a0-c1481e438377", + "customLabel": true, + "fieldName": "agg_blended", + "label": "Normalised cost", + "meta": { + "esType": "double", + "sourceParams": { + "indexPattern": "billing_cluster_cost_lookup" + }, + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n agg_blended = sum(blended)\n BY\n @timestamp,\n tier\n| WHERE agg_blended \u003e 0" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n agg_blended = sum(blended)\n BY\n @timestamp,\n tier\n| WHERE agg_blended \u003e 0" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "1501af72-2509-4ae0-88a0-c1481e438377" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "43f9cfe6-d828-4773-ab65-129f92cf7f4d", + "layerType": "data", + "seriesType": "bar_stacked", + "splitAccessor": "50a91b89-2e68-4a69-90a0-d2ab79115b96", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "agg_indexing \u0026 agg_querying \u0026 agg_storage \u0026 agg_blended over @timestamp", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(tier_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(tier_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(tier_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(tier_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n agg_blended = sum(blended)\n BY\n @timestamp,\n tier\n| WHERE agg_blended \u003e 0" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Blended value: Normalised cost per tier per day" + }, + "gridData": { + "h": 14, + "i": "5732f95c-3cc0-46c8-a8ab-14f8c6937f9f", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 16, + "y": 58 + }, + "panelIndex": "5732f95c-3cc0-46c8-a8ab-14f8c6937f9f", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "datasourceStates": { + "textBased": { + "indexPatternRefs": [], + "layers": { + "d720a72f-fe61-4baf-bdee-8a50d4247670": { + "columns": [ + { + "columnId": "% Normalised cost", + "customLabel": false, + "fieldName": "% Normalised cost", + "inMetricDimension": true, + "label": "% Normalised cost", + "meta": { + "esType": "double", + "type": "number" + } + }, + { + "columnId": "@timestamp", + "customLabel": false, + "fieldName": "@timestamp", + "label": "@timestamp", + "meta": { + "esType": "date", + "type": "date" + } + }, + { + "columnId": "datastream", + "customLabel": false, + "fieldName": "datastream", + "label": "datastream", + "meta": { + "esType": "keyword", + "type": "string" + } + } + ], + "index": "chargeback_integration", + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n `% Normalised cost` = sum(blended)\n BY\n @timestamp,\n datastream,\n tier\n| WHERE `% Normalised cost` \u003e 0\n| SORT `% Normalised cost` DESC" + }, + "timeField": "@timestamp" + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n `% Normalised cost` = sum(blended)\n BY\n @timestamp,\n datastream,\n tier\n| WHERE `% Normalised cost` \u003e 0\n| SORT `% Normalised cost` DESC" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "% Normalised cost" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "default", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rules": [ + { + "type": "other" + } + ], + "touched": false + } + ] + }, + "layerId": "d720a72f-fe61-4baf-bdee-8a50d4247670", + "layerType": "data", + "seriesType": "bar_percentage_stacked", + "splitAccessor": "datastream", + "xAccessor": "@timestamp" + } + ], + "legend": { + "isVisible": false, + "position": "right", + "showSingleSeries": false + }, + "preferredSeriesType": "bar", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "Pie", + "version": 1, + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM billing_cluster_cost_lookup\n| LOOKUP JOIN chargeback_conf_lookup ON @timestamp \u003e= conf_start_date AND @timestamp \u003c= conf_end_date\n| LOOKUP JOIN cluster_deployment_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_datastream_contribution_lookup ON composite_key\n| LOOKUP JOIN cluster_tier_contribution_lookup ON composite_key\n| EVAL\n indexing = CASE (deployment_sum_indexing_time \u003e 0, TO_DOUBLE(datastream_sum_indexing_time) / deployment_sum_indexing_time * total_ecu) * conf_ecu_rate,\n querying = CASE (deployment_sum_query_time \u003e 0, TO_DOUBLE(datastream_sum_query_time) / deployment_sum_query_time * total_ecu) * conf_ecu_rate,\n data_set = CASE (deployment_sum_data_set_store_size \u003e 0, TO_DOUBLE(datastream_sum_data_set_store_size) / deployment_sum_data_set_store_size * total_ecu),\n store = CASE (deployment_sum_store_size \u003e 0, TO_DOUBLE(datastream_sum_store_size) / deployment_sum_store_size * total_ecu),\n storage = CASE (store == 0, data_set, store) * conf_ecu_rate,\n total_weight_hot = conf_storage_weight + conf_query_weight + conf_indexing_weight,\n total_weight_cold = conf_storage_weight + conf_query_weight,\n blended = CASE (\n tier == \"hot/content\",\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight) + (indexing / conf_ecu_rate * conf_indexing_weight)) / total_weight_hot,\n ((storage / conf_ecu_rate * conf_storage_weight) + (querying / conf_ecu_rate * conf_query_weight)) / total_weight_cold\n ) * conf_ecu_rate\n| STATS\n `% Normalised cost` = sum(blended)\n BY\n @timestamp,\n datastream,\n tier\n| WHERE `% Normalised cost` \u003e 0\n| SORT `% Normalised cost` DESC" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Blended value: % Normalised cost per data stream per day" + }, + "gridData": { + "h": 14, + "i": "4dbbad0a-7ceb-443b-912b-0c1ae9fb3a6a", + "sectionId": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "w": 16, + "x": 32, + "y": 58 + }, + "panelIndex": "4dbbad0a-7ceb-443b-912b-0c1ae9fb3a6a", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "adHocDataViews": { + "8f632b34627f697e9702333f8ef042ab73e3436b01f9a29710a96cc9dad323b9": { + "allowHidden": false, + "allowNoIndex": false, + "fieldFormats": {}, + "id": "8f632b34627f697e9702333f8ef042ab73e3436b01f9a29710a96cc9dad323b9", + "managed": false, + "name": "chargeback_conf_lookup", + "runtimeFieldMap": {}, + "sourceFilters": [], + "title": "chargeback_conf_lookup", + "type": "esql" + } + }, + "datasourceStates": { + "textBased": { + "indexPatternRefs": [ + { + "id": "8f632b34627f697e9702333f8ef042ab73e3436b01f9a29710a96cc9dad323b9", + "title": "chargeback_conf_lookup" + } + ], + "layers": { + "4dba9fd5-8ec5-498a-b710-1512bd5dc18f": { + "columns": [ + { + "columnId": "conf_ecu_rate_unit", + "customLabel": true, + "fieldName": "conf_ecu_rate_unit", + "label": "Unit", + "meta": { + "esType": "keyword", + "type": "string" + } + }, + { + "columnId": "conf_ecu_rate", + "customLabel": true, + "fieldName": "conf_ecu_rate", + "inMetricDimension": true, + "label": "Rate", + "meta": { + "esType": "double", + "type": "number" + }, + "params": { + "format": { + "id": "number", + "params": { + "decimals": 2 + } + } + } + }, + { + "columnId": "e7785ad9-760e-4215-ad19-40fc21eeb27e", + "customLabel": true, + "fieldName": "conf_start_date", + "label": "Start date", + "meta": { + "esType": "date", + "params": { + "id": "date" + }, + "sourceParams": { + "appliedTimeRange": { + "from": "2025-11-24T15:31:59.427Z", + "to": "2025-11-24T15:46:59.427Z" + }, + "indexPattern": "chargeback_conf_lookup", + "params": {}, + "sourceField": "conf_start_date" + }, + "type": "date" + } + }, + { + "columnId": "ef4ddf0e-7371-482e-98e1-7e131392d91f", + "customLabel": true, + "fieldName": "conf_end_date", + "label": "End date", + "meta": { + "esType": "date", + "params": { + "id": "date" + }, + "sourceParams": { + "appliedTimeRange": { + "from": "2025-11-24T15:32:12.392Z", + "to": "2025-11-24T15:47:12.392Z" + }, + "indexPattern": "chargeback_conf_lookup", + "params": {}, + "sourceField": "conf_end_date" + }, + "type": "date" + } + } + ], + "index": "8f632b34627f697e9702333f8ef042ab73e3436b01f9a29710a96cc9dad323b9", + "query": { + "esql": "FROM chargeback_conf_lookup \n| keep conf_ecu_rate_unit, conf_ecu_rate, conf_start_date, conf_end_date" + } + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM chargeback_conf_lookup \n| keep conf_ecu_rate_unit, conf_ecu_rate, conf_start_date, conf_end_date" + }, + "visualization": { + "columns": [ + { + "columnId": "conf_ecu_rate_unit" + }, + { + "alignment": "left", + "columnId": "conf_ecu_rate" + }, + { + "alignment": "center", + "columnId": "e7785ad9-760e-4215-ad19-40fc21eeb27e", + "isMetric": true, + "isTransposed": false + }, + { + "alignment": "center", + "columnId": "ef4ddf0e-7371-482e-98e1-7e131392d91f", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "4dba9fd5-8ec5-498a-b710-1512bd5dc18f", + "layerType": "data" + } + }, + "title": "New suggestion", + "version": 1, + "visualizationType": "lnsDatatable" + }, + "description": "ECU rates and units configured over time", + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM chargeback_conf_lookup \n| keep conf_ecu_rate_unit, conf_ecu_rate, conf_start_date, conf_end_date" + }, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "ECU Rates" + }, + "gridData": { + "h": 8, + "i": "54ff8784-cfb6-41dc-ad15-50547f49f9e7", + "sectionId": "1ae779e4-e93f-47b4-b0af-cc25ed49ba34", + "w": 22, + "x": 0, + "y": 10 + }, + "panelIndex": "54ff8784-cfb6-41dc-ad15-50547f49f9e7", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [], + "state": { + "adHocDataViews": { + "8f632b34627f697e9702333f8ef042ab73e3436b01f9a29710a96cc9dad323b9": { + "allowHidden": false, + "allowNoIndex": false, + "fieldFormats": {}, + "id": "8f632b34627f697e9702333f8ef042ab73e3436b01f9a29710a96cc9dad323b9", + "name": "chargeback_conf_lookup", + "runtimeFieldMap": {}, + "sourceFilters": [], + "title": "chargeback_conf_lookup", + "type": "esql" + } + }, + "datasourceStates": { + "textBased": { + "indexPatternRefs": [ + { + "id": "8f632b34627f697e9702333f8ef042ab73e3436b01f9a29710a96cc9dad323b9", + "title": "chargeback_conf_lookup" + } + ], + "layers": { + "005d9023-92e0-4783-a349-4a2e05b0bce4": { + "columns": [ + { + "columnId": "conf_indexing_weight", + "customLabel": true, + "fieldName": "conf_indexing_weight", + "inMetricDimension": true, + "label": "Weight", + "meta": { + "esType": "integer", + "type": "number" + } + }, + { + "columnId": "conf_query_weight", + "customLabel": true, + "fieldName": "conf_query_weight", + "inMetricDimension": true, + "label": "Weight", + "meta": { + "esType": "integer", + "type": "number" + } + }, + { + "columnId": "conf_storage_weight", + "customLabel": false, + "fieldName": "conf_storage_weight", + "inMetricDimension": true, + "label": "Weight", + "meta": { + "esType": "integer", + "type": "number" + } + } + ], + "index": "8f632b34627f697e9702333f8ef042ab73e3436b01f9a29710a96cc9dad323b9", + "query": { + "esql": "FROM chargeback_conf_lookup | KEEP *weight" + } + } + } + } + }, + "filters": [], + "needsRefresh": false, + "query": { + "esql": "FROM chargeback_conf_lookup | KEEP *weight" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "Linear", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "conf_indexing_weight", + "conf_query_weight", + "conf_storage_weight" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "005d9023-92e0-4783-a349-4a2e05b0bce4", + "layerType": "data", + "seriesType": "bar_percentage_stacked" + } + ], + "legend": { + "horizontalAlignment": "right", + "isInside": true, + "isVisible": false, + "position": "right", + "showSingleSeries": false, + "verticalAlignment": "bottom" + }, + "preferredSeriesType": "bar_stacked", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "conf_indexing_weight \u0026 conf_query_weight \u0026 conf_storage_weight of (empty)", + "visualizationType": "lnsXY" + }, + "enhancements": { + "dynamicActions": { + "events": [] + } + }, + "filters": [], + "query": { + "esql": "FROM chargeback_conf_lookup | KEEP *weight" + }, + "searchSessionId": "62d2fe44-dd84-4cd0-ac53-38b090b0eff6", + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "title": "Blended rate weighting percentages" + }, + "gridData": { + "h": 8, + "i": "a3a091e7-6bff-4edf-bc65-993e8a3648de", + "sectionId": "1ae779e4-e93f-47b4-b0af-cc25ed49ba34", + "w": 6, + "x": 22, + "y": 10 + }, + "panelIndex": "a3a091e7-6bff-4edf-bc65-993e8a3648de", + "type": "lens" + }, + { + "embeddableConfig": { + "content": "## Blended weight calculation\n\nThe chargeback integration calculates a blended cost value by combining indexing time (hot tier only), query time, and storage size with configurable weights set in `chargeback_conf_lookup` (default: indexing=20, query=20, storage=40). Each weight is a percentage representing how much that activity contributes to the total cost, allowing you to prioritize storage-heavy vs compute-heavy workloads based on your organization's usage patterns. Adjust weights by updating the lookup index to match your cost model (e.g., increase query weight for analytics-heavy teams, increase storage weight for archival use cases).", + "title": "" + }, + "gridData": { + "h": 10, + "i": "e55da6bd-a4be-4610-b4f8-ecb4f97dbc55", + "sectionId": "1ae779e4-e93f-47b4-b0af-cc25ed49ba34", + "w": 14, + "x": 22, + "y": 0 + }, + "panelIndex": "e55da6bd-a4be-4610-b4f8-ecb4f97dbc55", + "type": "DASHBOARD_MARKDOWN" + }, + { + "embeddableConfig": { + "content": "## Deployment group extraction\n\nThe chargeback integration extracts `deployment_group` from ESS Billing integration deployment tags (requires ESS Billing integration v1.7.0+) using runtime mappings that scan for tags matching the pattern `chargeback_group:\u003cgroup_name\u003e`. Set tags on your deployments via the Elasticsearch Service Console (cloud.elastic.co → deployment settings → add tag `chargeback_group:your_team_name`) or the ESS API. Deployments without the tag will have `deployment_group: \"\"` and still work, just without tag-based grouping.", + "title": "" + }, + "gridData": { + "h": 10, + "i": "a9db815d-08c9-49a1-9699-f51be117e502", + "sectionId": "1ae779e4-e93f-47b4-b0af-cc25ed49ba34", + "w": 12, + "x": 36, + "y": 0 + }, + "panelIndex": "a9db815d-08c9-49a1-9699-f51be117e502", + "type": "DASHBOARD_MARKDOWN" + }, + { + "embeddableConfig": { + "content": "## Normalised cost\nNormalised cost is calculated as ECU (Elastic Consumption Units) multiplied by the conversion rate (`conf_ecu_rate`) from the `chargeback_conf_lookup` index, where the rate is selected based on the time window being queried. This allows you to convert raw ECU consumption into your preferred currency (e.g., EUR, USD) and apply different rates for different time periods. You can adjust this by updating the lookup index with period-specific rates." + }, + "gridData": { + "h": 10, + "i": "b50797e9-6976-4df6-9ada-2f64b8b9365e", + "sectionId": "1ae779e4-e93f-47b4-b0af-cc25ed49ba34", + "w": 10, + "x": 0, + "y": 0 + }, + "panelIndex": "b50797e9-6976-4df6-9ada-2f64b8b9365e", + "type": "DASHBOARD_MARKDOWN" + }, + { + "embeddableConfig": { + "content": "## Configuration date span\nThe `chargeback_conf_lookup` index supports time-windowed configuration using `conf_start_date` and `conf_end_date` fields, allowing you to set different ECU rates and weights for specific time periods. Create multiple configuration documents with different date ranges (e.g., Q1 rates vs Q2 rates) and the dashboard will apply the appropriate configuration based on the query time window. This enables historical cost analysis with period-accurate pricing and weight adjustments.", + "title": "" + }, + "gridData": { + "h": 10, + "i": "03e2d9f7-5217-4e71-ba05-667fdca2b1b9", + "sectionId": "1ae779e4-e93f-47b4-b0af-cc25ed49ba34", + "w": 12, + "x": 10, + "y": 0 + }, + "panelIndex": "03e2d9f7-5217-4e71-ba05-667fdca2b1b9", + "type": "DASHBOARD_MARKDOWN" + } + ], + "sections": [ + { + "collapsed": false, + "gridData": { + "i": "2b8b2365-5e05-4f1d-8880-f35892a8e60f", + "y": 11 + }, + "title": "Deployment group statistics" + }, + { + "collapsed": false, + "gridData": { + "i": "5092dfac-3f29-4eab-8c8f-f97bf1479370", + "y": 12 + }, + "title": "Deployment statistics" + }, + { + "collapsed": true, + "gridData": { + "i": "96918af7-e003-4af8-8c37-245db141d60c", + "y": 13 + }, + "title": "Data tier and data stream overview" + }, + { + "collapsed": false, + "gridData": { + "i": "be60066e-c6d8-4e27-b8f4-948f37126ecf", + "y": 14 + }, + "title": "Data tier and data stream per day" + }, + { + "collapsed": false, + "gridData": { + "i": "1ae779e4-e93f-47b4-b0af-cc25ed49ba34", + "y": 15 + }, + "title": "Chargeback Configuration Information" + } + ], + "timeRestore": false, + "title": "[Chargeback] Cost and Consumption breakdown", + "version": 1 + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2025-12-02T09:15:39.552Z", + "id": "chargeback-39a39857-746c-4a29-adca-3c2fcb6bcfb6", + "references": [ + { + "id": "chargeback_integration", + "name": "controlGroup_b680ef70-4697-41e5-bf5f-e7e1e700ec3e:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "chargeback_integration", + "name": "controlGroup_2c7df183-8a05-4ee0-b7db-415e874636cd:optionsListDataView", + "type": "index-pattern" + } + ], + "type": "dashboard", + "typeMigrationVersion": "10.3.0", + "updated_by": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0" +} \ No newline at end of file diff --git a/packages/chargeback/kibana/index_pattern/chargeback_integration.json b/packages/chargeback/kibana/index_pattern/chargeback_integration.json new file mode 100644 index 00000000000..bdc3d12f016 --- /dev/null +++ b/packages/chargeback/kibana/index_pattern/chargeback_integration.json @@ -0,0 +1,22 @@ +{ + "attributes": { + "allowHidden":true, + "fieldAttrs":"{}", + "fieldFormatMap":"{}", + "fields":"[]", + "name":"[Chargeback] Billing Cluster Cost", + "runtimeFieldMap":"{}", + "sourceFilters":"[]", + "timeFieldName":"@timestamp", + "title":"billing_cluster_cost_lookup" + }, + "coreMigrationVersion":"8.8.0", + "created_at":"2025-11-28T09:22:34.415Z", + "id":"chargeback_integration", + "managed":false, + "references":[], + "type":"index-pattern", + "typeMigrationVersion":"8.0.0", + "updated_at":"2025-11-28T09:22:34.415Z", + "version":"WzE4MCwzXQ==" +} \ No newline at end of file diff --git a/packages/chargeback/manifest.yml b/packages/chargeback/manifest.yml new file mode 100644 index 00000000000..994c0f01b85 --- /dev/null +++ b/packages/chargeback/manifest.yml @@ -0,0 +1,27 @@ +format_version: 3.4.0 +name: chargeback +title: "Chargeback" +version: 0.2.10 +description: "This package calculates chargeback based on billing and consumption data" +type: integration +categories: + - cloud + - custom +conditions: + kibana: + version: "^9.2.0" + elastic: + subscription: "basic" +screenshots: + - src: /img/elastic-logo.png + title: Elastic logo + size: 600x600 + type: image/png +icons: + - src: /img/elastic-logo.svg + title: Elastic logo + size: 32x32 + type: image/svg+xml +owner: + github: elastic/customer-architects + type: elastic \ No newline at end of file diff --git a/packages/chargeback/validation.yml b/packages/chargeback/validation.yml new file mode 100644 index 00000000000..bcc8f74ac3a --- /dev/null +++ b/packages/chargeback/validation.yml @@ -0,0 +1,3 @@ +errors: + exclude_checks: + - SVR00002