DRAFT: [Chargeback] new integration#14545
Conversation
sholzhauer-es
added
the
New Integration
andrewkroh
added
dashboard
|
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
|
Still relevant. Dependent on features in the elastic-package repo. |
|
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
…16185) * Add deployment_group extraction from ESS Billing tags - Extract chargeback_group tag value to deployment_group field in billing pipeline - Add deployment_group to billing_cluster_cost transform group_by - Add deployment_group field definition - Fix transforms to use elasticsearch.cluster.name without .keyword - Update changelog for v0.2.4 * Add deployment_group extraction using runtime mappings from ESS Billing tags * Update dashboard with deployment_group filter and definitions * Bump version to 0.2.5 for deployment_group feature after merging SKU/cost_type changes * known [bug](elastic/elasticsearch-chargeback#60) from 0.2.4
* adding sku and cost_type to billing data for node granularity * working on sku with pipeline to do parsing * downplaying version * transform * merge * dashboard * setting version
JohannesMahne
force-pushed
the
johannes-chargeback-wip
branch
from
a43f775 to
d694959
Compare
* Add observability alerts for chargeback integration - Add two ES|QL alerting rules: detect new chargeback groups and detect deployments missing usage data - Add comprehensive documentation for alert setup and configuration - Update Elasticsearch version requirement to 9.2.0+ for smart lookup join support - Add transform startup and monitoring instructions to README * Update changelog with PR #16205 * Remove wrong information * Update chargeback README documentation * Improve observability alert action message formatting * Clarify configuration update vs add new period documentation
* Add alerting rule templates and enable auto-start for all transforms - Add 3 Kibana alerting rule templates: - Transform health monitoring for all Chargeback transforms - New chargeback group detection - Deployment with missing usage data detection - Enable auto-start for all transforms (start: true in manifests) - Update transform pipeline references to version 0.2.8 - Add performance warning about initial transform execution - Update README with alerting documentation - Bump package version to 0.2.8 * Fix: Revert transform frequencies back to 60m * Update PR number in changelog
* WIP: early chargeback code for review * Working config integration - 0.0.2 * Version 0.0.3: working from Stack monitoring data * Fixed query for one visualisation * Update instructions * Working with the correct alias * Changes to transforms * Bug fix: Fix sorting on visualisation. * Update setup instructions * 0.1.0: Adding ECU value (normalised cost). * Bug: Aligned fields returned to field names used in visualisation * Fixing bug: aligning esql returned field names with field names used in lens * move to packages * not starting transforms on integration installation * Update version number * Made sure the colour palette is predictable by using the eui_amsterdam_color_blind palate. Add ECU rate to the dashboard. * Update sequence and comments on pre-setup to promote ES integration * Consistent naming of datastream. Add LIMIT 5000 to ESQL top query to cater for large organisations. * Add correct code owner * Delete wrong test files * Updated the directory structure to remove superfluous directory * Rem reference to sample logs and logos * Switch off dynamic mappings for the results of the transforms - we know exactly what the output be. * Removed agent folders in data stream, as it is not used. * Updated the readme file to refer to integration, rather than module. Also added explanation about the rest of the config. * Re-add image * Formatting * NOT WORKING: settings index.mode: lookup is not supported * Fixing the control error in the dashboard by adding a data view. * Updated to push back usage data transform to ES Integration * Updated readme * Update transfrom version numbers * Swap the use of deployment_id or deployment name to a concatenation of both, to make it easier to identify the deployment in the dashboard. * Make use of the new elastic-package version, which will create the lookup index automatically when installing the package. * Update version number * Updated pre-setup, and version number * Adding casting to double for division to avoid null instead of very small numbers. elastic/elasticsearch-chargeback#50 * Update version * Allowing for setting converion rate per time window * fixing pipeline versions * adding pipeline stuff * correcting version * [Chargeback] Dashboard control and Dataview (#16153) * dashboard control * updating version + DV * SKU based chargeback (#16182) * adding sku and cost_type to billing data for node granularity * working on sku with pipeline to do parsing * downplaying version * transform * Chargeback Integration: Extract deployment group from Billing tags (#16185) * Add deployment_group extraction from ESS Billing tags - Extract chargeback_group tag value to deployment_group field in billing pipeline - Add deployment_group to billing_cluster_cost transform group_by - Add deployment_group field definition - Fix transforms to use elasticsearch.cluster.name without .keyword - Update changelog for v0.2.4 * Add deployment_group extraction using runtime mappings from ESS Billing tags * Update dashboard with deployment_group filter and definitions * Bump version to 0.2.5 for deployment_group feature after merging SKU/cost_type changes * known [bug](elastic/elasticsearch-chargeback#60) from 0.2.4 * wip on css * adding "local" cluster for ones without remote clusters --------- Co-authored-by: Johannes Mahne <johannes.mahne@elastic.co>
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
|
still wip |
…ookup (#16936) * Fix visualizations not loading by adding TO_DOUBLE type conversion - Add TO_DOUBLE() wrapper to all division operations in ESQL queries - Prevents integer division from returning zero - Fixes tier_sum_indexing_time / deployment_sum_indexing_time - Fixes tier_sum_query_time / deployment_sum_query_time - Fixes tier_sum_data_set_store_size / deployment_sum_data_set_store_size - Fixes tier_sum_store_size / deployment_sum_store_size - Bump version to 0.2.10 Fixes: elastic/elasticsearch-chargeback#69 * Add automated chargeback_conf_lookup index creation via transform - Add bootstrap transform that creates chargeback_conf_lookup index with default config - Uses cluster_deployment_contribution_lookup as source - Sets default values: ECU rate 0.85 EUR, weights 20/20/40 - Date range: 2010-01-01 (ES birthdate) to 2046-12-31 - Eliminates need for manual index creation * Bump transform pipeline versions to 0.2.10 - Update pipeline references from 0.2.9 to 0.2.10 - Revert billing_cluster_cost sync field to event.ingested (was temporarily @timestamp) * Removed now redundant pre-setup. * Update PR number in changelog, and recover billing cost sync time field. * Update packages/chargeback/elasticsearch/transform/chargeback_conf_lookup/fields/base-fields.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix chargeback_conf_lookup transform source index dependency Change source from cluster_deployment_contribution_lookup to metrics-ess_billing.billing-* to fix transform loading in fresh setups. The transform uses runtime_mappings to generate all config values, so the source index content doesn't matter - it only needs any document to trigger. Billing data is guaranteed to exist in chargeback deployments. Fixes feedback from @elastic-abhi review. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
💚 Build Succeeded
History
|
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
|
👍 |
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
andrewkroh
removed
dashboard
andrewkroh
added
the
New Integration
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
|
👍 |
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
4 participants
Proposed commit message
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots