Sync Devlake from upstream to konflux#71
Conversation
…pache#8699) * chore(deps): update graphql dependency from merico-dev to merico-ai - Update github.com/merico-dev/graphql to github.com/merico-ai/graphql - Update import paths across all files using the graphql package - Update go.mod and go.sum with new dependency version Co-Authored-By: Claude <noreply@anthropic.com> * fix(github): fix ineffassign lint error in remote_api.go Use named return values consistently with bare returns to avoid variable shadowing of err in listGithubOwnerRepos. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
…eport paths (apache#8725) User now provides basePath and accountId instead of manually typing the full S3 prefix. The collector automatically constructs and scans both by_user_analytic and user_report paths under {basePath}/AWSLogs/{accountId}/KiroLogs/…/{region}/{year}[/{month}]. Includes migration, updated blueprint, multi-prefix collector iteration, user_report model/extractor/dashboard, and frontend Account ID input field. Old scopes without accountId continue to work unchanged.
* feat(github_graphql): add finished_date to GithubDeployment model Introduce FinishedDate field in GithubDeployment to represent success-priority deployment completion timestamp. This field allows distinguishing SUCCESS timestamp from latest status update time while preserving backward compatibility. * feat(github_graphql): compute success-priority deployment finish time Add logic to derive FinishedDate from deployment statuses. Preference order: 1. Latest SUCCESS timestamp 2. Latest terminal timestamp (FAILURE/ERROR/INACTIVE/ACTIVE) This improves deployment lifecycle accuracy without modifying existing LatestStatus semantics. * feat(github_graphql): use finished_date in deployment domain conversion Update ConvertDeployment to use tool-layer FinishedDate when populating domain TaskDatesInfo.FinishedDate. Duration is now calculated using resolved finished time instead of UpdatedDate. * chore(github_graphql): add migration for finished_date column Add migration script to auto-migrate GithubDeployment and create nullable finished_date column. * test(github_graphql): update e2e snapshots for deployment finished date and duration Introduce adjustments to existing e2e tests to validate backward compatibility after adding deployment finished date and duration calculation logic. Add `deployment_finished_date` field to `_tool_github_deployments.csv` and `duration_seconds` field to `cicd_deployments.csv`. Ensure historical data remains compatible with the updated behavior. * test(github_graphql): add tests for deployment finished date and duration calculation logic Add test coverage for deployment finished date and duration calculation based on the SUCCESS state instead of relying on the inactive state. Verify fallback behavior when a SUCCESS state is not present. * fix(github_graphql): decouple add finished_date migration from active model Define a dedicated versioned struct for the finished_date migration instead of referencing the evolving GithubDeployment model to ensure deterministic and reproducible schema changes. * chore(github_graphql): fix duplicate import and lint issues in deployment extractor Remove redundant restTasks alias and use githubTasks constants consistently to resolve lint warnings and improve import clarity.
…apache#8714) * perf(dora): optimize change lead time calculation using batch queries Eliminates N+1 query problem by implementing batch fetching for: - First commits per PR (batchFetchFirstCommits) - First reviews per PR (batchFetchFirstReviews) - Deployments per project (batchFetchDeployments) Performance improvement: - Before: 3 queries per PR (30,001 queries for 10K PRs) - After: 3 batch queries total (99.99% reduction) Also fixes NULL author_id handling in review queries to properly handle PRs with empty author_id field. Tested with E2E tests confirming correctness and performance gains. Signed-off-by: Ankesh <athakur@g2.com> * fix(dora): remove obsolete single-query functions replaced by batch queries Signed-off-by: Ankesh <athakur@g2.com> * ci: add workflow to build and push amd64 image to GHCR Signed-off-by: Ankesh <athakur@g2.com> * revert: remove GHCR build workflow Signed-off-by: Ankesh <athakur@g2.com> --------- Signed-off-by: Ankesh <athakur@g2.com>
* feat(gh-copilot): add backend plugin * feat(config-ui): add gh-copilot connection UI * feat(grafana): add GitHub Copilot dashboards * fix: gh-copilot CI fixes
…en using devcontainers (apache#8645) * fix(devcontainer): Make config-ui dev server accessible outside the container when started by listening on 0.0.0.0 with the `--host` flag. Also ensure it uses port 4000 or fails to start with `--strictPort` to prevent silent port change that will mismatch with the exported dev container port. * fix(devcontainer): make Config UI and Grafana accessible from host when using devcontainers
…#8716) * feat(github): add username filtering helper for bot exclusion Implements shouldSkipByUsername() function to filter bot accounts by username using GITHUB_PR_EXCLUDELIST environment variable. - Case-insensitive matching - Comma-separated list support - Whitespace trimming - Returns false for empty usernames or empty exclusion list Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: AvivGuiser <avivguiser@gmail.com> * feat(github): filter bot PRs in extractor Adds username filtering to PR extractor to skip bot-authored PRs when GITHUB_PR_EXCLUDELIST is set. - Checks author username before extraction - Logs debug message when PR is skipped - Includes unit test for bot filtering - Includes e2e test data for bot filtering Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(github): filter bot reviews in extractor Adds username filtering to review extractor to skip bot reviews when GITHUB_PR_EXCLUDELIST is set. - Checks reviewer username before extraction - Logs debug message when review is skipped - Includes e2e test for bot filtering Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(github): filter bot PR review comments in extractor Adds username filtering to PR review comment extractor to skip bot comments when GITHUB_PR_EXCLUDELIST is set. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat(github): filter bot issue comments in extractor Adds username filtering to issue comment extractor to skip bot comments when GITHUB_PR_EXCLUDELIST is set. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * docs(github): add bot filtering documentation Documents GITHUB_PR_EXCLUDELIST configuration and usage. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * fix(github): fix CI errors in bot filtering PR - Add Apache license header to README_FILTERING.md - Export ResetExcludedUsernamesForTest to fix unused lint error - Call ResetExcludedUsernamesForTest in e2e tests to reset sync.Once cache before setting GITHUB_PR_EXCLUDELIST env var Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: AvivGuiser <avivguiser@gmail.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Add automatic URL enrichment to PutScopes function to fetch missing htmlUrl and cloneUrl fields from GitHub API when only fullName is provided. This ensures gitextractor has the required URLs for cloning repositories. - Fetch repo details from GitHub API when URLs are missing - Populate htmlUrl and cloneUrl from API response - Add graceful error handling for inaccessible repos
…les (apache#8737) * fix(q_dev): prevent data duplication in user_report and user_data tables Replace auto-increment ID with composite primary keys so that CreateOrUpdate can properly deduplicate rows on re-extraction. - user_report PK: (connection_id, scope_id, user_id, date, client_type) - user_data PK: (connection_id, scope_id, user_id, date) - Switch db.Create() to db.CreateOrUpdate() in s3_data_extractor - Migration drops old tables, rebuilds with new PKs, resets s3_file_meta processed flag to trigger re-extraction * fix(q_dev): gofmt archived user_data_v2 model
…pache#8719) * feat(github): extend PR size exclusion for specified file extension to github plugin * fix: register migration script * fix: move PR size to 'Additional settings' and change so the comma doesn't get removed while typing * fix: linting
The Slack invite links in README.md were expired and returning "This link is no longer active." Updated both occurrences (badge and community section) to match the current link on the official DevLake website. Closes apache#8738 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com>
…che#8733) Adds gh-devlake as a third installation method alongside Docker Compose and Helm. gh-devlake is a GitHub CLI extension that automates DevLake deployment, configuration, and monitoring from the terminal. Closes apache#8732
GitLab's makeScopeV200 did not create a repos scope when scopeConfig.Entities was empty or only contained CROSS. This caused project_mapping to have no table='repos' row, breaking downstream DORA metrics, PR-issue linking, and all PR dashboard panels that join on project_mapping. The fix aligns GitLab with the GitHub plugin by: 1. Defaulting empty entities to plugin.DOMAIN_TYPES 2. Adding DOMAIN_TYPE_CROSS to the repo scope condition Closes apache#8742 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com>
…a sources (apache#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes apache#8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com>
* fix: modify cicd_deployments name from varchar to text * fix: update the year
…thods for MySQL 8.x compatibility (apache#8745)
…in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749
…pesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748
…apache#8757) Update CircleCI connection form to indicate Server v4.x+ requirement and provide guidance for server endpoint configuration. Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>
…#8758) Add a new Asana plugin that integrates with Asana's REST API to collect projects, sections, tasks, subtasks, stories (comments), tags, and users, mapping them to DevLake's ticket/board domain model. Backend: - Plugin implementation with all required interfaces (PluginMeta, PluginTask, PluginModel, PluginMigration, PluginSource, PluginApi, DataSourcePluginBlueprintV200) - Collectors, extractors, and converters for projects, sections, tasks, subtasks, stories, tags, and users - Remote API scope picker (Workspaces -> Teams/Portfolios -> Projects) - Scope config with issue-type regex transformation rules - Migration scripts for schema evolution - E2E tests with CSV fixtures for project and task data flows Config UI: - Plugin registration with connection form (PAT auth, endpoint, proxy) - Scope config transformation form for issue-type mapping - Dashboard URL integration for onboarding flow Grafana: - Asana dashboard with task metrics and visualizations Made-with: Cursor
* feat(github): auto-refresh GitHub App installation tokens Add transport-level token refresh for GitHub App (AppKey) connections. GitHub App installation tokens expire after ~1 hour; this adds proactive refresh (before expiry) and reactive refresh (on 401) using the existing TokenProvider/RefreshRoundTripper infrastructure. New files: - app_installation_refresh.go: refresh logic + DB persistence - refresh_api_client.go: minimal ApiClient for token refresh POST - cmd/test_refresh/main.go: manual test script for real GitHub Apps Modified: - connection.go: export GetInstallationAccessToken, parse ExpiresAt - token_provider.go: add refreshFn for pluggable refresh strategies - round_tripper.go: document dual Authorization header interaction - api_client.go: wire AppKey connections into refresh infrastructure - Tests updated for new constructors and AppKey refresh flow * feat(github): add diagnostic logging to GitHub App token refresh Add structured logging at key decision points for token refresh: - Token provider creation (connection ID, installation ID, expiry) - Round tripper installation (connection ID, auth method) - Proactive refresh trigger (near-expiry detection) - Refresh start/success/failure (old/new token prefixes, expiry times) - DB persistence success/failure - Reactive 401 refresh and skip-due-to-concurrent-refresh All logs route through the DevLake logger to pipeline log files. * fix(github): prevent deadlock and fix token persistence in App token refresh Deadlock fix: NewAppInstallationTokenProvider now captures client.Transport (the base transport) before wrapping with RefreshRoundTripper. The refresh function uses newRefreshApiClientWithTransport(baseTransport) to POST for new installation tokens, bypassing the RefreshRoundTripper entirely. Token persistence fix: PersistEncryptedTokenColumns() manually encrypts tokens via plugin.Encrypt() then writes ciphertext via dal.UpdateColumns with conn.TableName() (a string) as the first argument. Passing the table name string makes GORM use Table() instead of Model(), preventing the encdec serializer from corrupting the in-memory token value. The encryption secret is threaded from taskCtx.GetConfig(ENCRYPTION_SECRET) through CreateApiClient to TokenProvider to persist functions. Also persists the initial App token at startup for DB consistency, and adds TestProactiveRefreshNoDeadlock with a real RSA key to verify the deadlock scenario is resolved. * fix(grafana): update dashboard descriptions to list all supported data sources (apache#8741) Several dashboard introduction panels hardcoded "GitHub and Jira" as required data sources, even though the underlying queries use generic domain layer tables that work with any supported Git tool or issue tracker. Updated to list all supported sources following the pattern already used by DORA and WorkLogs dashboards. Closes apache#8740 Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> * fix: modify cicd_deployments name from varchar to text (apache#8724) * fix: modify cicd_deployments name from varchar to text * fix: update the year * fix(q_dev): replace MariaDB-specific IF NOT EXISTS syntax with DAL methods for MySQL 8.x compatibility (apache#8745) * fix(azuredevops): default empty entities and add CROSS to repo scope in makeScopeV200 (apache#8751) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopeV200 produced zero scopes, leaving project_mapping with no rows. Additionally, the repo scope condition did not check for DOMAIN_TYPE_CROSS, so selecting only CROSS would not create a repo scope, breaking DORA metrics. This adds the same fixes applied to GitLab in apache#8743. Closes apache#8749 * fix(bitbucket): default empty entities to all domain types in makeScopesV200 (apache#8750) When scopeConfig.Entities is empty (common when no entities are explicitly selected in the UI), makeScopesV200 produced zero scopes, leaving project_mapping with no repo rows. This adds the same empty-entities default applied to GitLab in apache#8743. Closes apache#8748 * fix(github): remove unused refresh client constructor and update tests --------- Co-authored-by: Spiff Azeta <35563797+spiffaz@users.noreply.github.com> Co-authored-by: Spiff Azeta <spiffazeta@gmail.com> Co-authored-by: Dan Crews <crewsd@gmail.com> Co-authored-by: Tomoya Kawaguchi <68677002+yamoyamoto@users.noreply.github.com>
…pache#8767) * feat(q-dev): add logging data ingestion and enrich Kiro dashboards Add support for ingesting S3 logging data (GenerateAssistantResponse and GenerateCompletions events) into new database tables, and enrich all three Kiro Grafana dashboards with additional metrics. Changes: - New models: QDevChatLog and QDevCompletionLog for logging event data - New extractor: s3_logging_extractor.go parses JSON.gz logging files - Updated S3 collector to also handle .json.gz files - Added logging S3 prefixes (GenerateAssistantResponse, GenerateCompletions) - New dashboard: "Kiro AI Activity Insights" with 10 panels including model usage distribution, active hours, conversation depth, feature adoption (Steering/Spec), file type usage, and prompt/response trends - Enriched "Kiro Code Metrics Dashboard" with DocGeneration, TestGeneration, and Dev (Agentic) metric panels - Fixed "Kiro Usage Dashboard" per-user table to sort by user_id - Migration script for new tables * fix(q-dev): use separate base path for logging S3 prefixes Logging data lives under a different S3 prefix ("logging/") than user report data ("user-report/"). Add LoggingBasePath option (defaults to "logging") so logging prefixes are constructed correctly. * fix(q-dev): auto-scan logging path without extra config Kiro exports to two well-known S3 prefixes in the same bucket: - user-report/AWSLogs/{accountId}/KiroLogs/ (CSV reports) - logging/AWSLogs/{accountId}/KiroLogs/ (interaction logs) When AccountId is set, automatically scan both paths. The "logging" prefix is hardcoded since it's a standard Kiro export convention. No additional configuration needed. * fix(q-dev): update scope tooltip to mention logging data scanning * fix(q-dev): fix scope ID routing and CSV/JSON file separation Three fixes: 1. Use *scopeId (catch-all) route pattern instead of :scopeId so scope IDs containing "/" (e.g. "034362076319/2026") work in URL paths 2. CSV extractor now filters for .csv files only, preventing it from trying to parse .json.gz logging files as CSV 3. Frontend scope API calls now encodeURIComponent(scopeId) for safe URL encoding * fix(q-dev): resolve *scopeId route conflict with dispatcher pattern The catch-all *scopeId route conflicts with *scopeId/latest-sync-state. Follow Jenkins/Bitbucket pattern: use a single *scopeId route with a GetScopeDispatcher that checks for /latest-sync-state suffix and dispatches accordingly. All scope handlers now TrimLeft "/" from scopeId. * fix(q-dev): use URL-safe scope ID format (underscore separator) Scope IDs like "034362076319/2026" break URL routing because "/" is a path separator. Change ID format to "034362076319_2026" (underscore) when AccountId is set. The Prefix field still uses "/" for S3 path matching. Revert to standard :scopeId routes since IDs are now safe. Note: existing scopes need to be recreated after this change. * fix(q-dev): use NoPKModel instead of Model in archived logging models archived.Model only has ID+timestamps, missing RawDataOrigin fields (_raw_data_params etc.) that common.NoPKModel includes. This caused "Unknown column '_raw_data_params'" errors at runtime. * fix(q-dev): fix GROUP BY in per-user table to merge display_name variants Remove display_name from GROUP BY so same user_id with different display_name values gets merged. Use MAX(display_name) in SELECT. * fix(q-dev): normalize logging user IDs to match CSV short UUID format Logging data uses "d-{directoryId}.{UUID}" format while CSV user-report uses plain "{UUID}". Strip the "d-xxx." prefix so the same user maps to one user_id across both data sources. * fix(q-dev): normalize user IDs in CSV extractors and sort table DESC Apply normalizeUserId to both createUserReportData and createUserDataWithDisplayName so user_report CSV data also strips the "d-{directoryId}." prefix. Change per-user table sort to ORDER BY user_id DESC. * style(q-dev): fix gofmt formatting in chat_log models * perf(q-dev): parallelize logging S3 downloads and batch DB writes Optimize logging extractor performance: - 10 goroutine workers for parallel S3 file downloads - Batch 50 files per DB transaction instead of 1-per-file - sync.Map cache for display name resolution (avoid repeated IAM calls) - Parse records in memory during download, write all at once This should improve throughput from ~1.5 files/sec to ~15+ files/sec for typical logging file sizes. * fix(q-dev): check tx.Rollback error return to satisfy errcheck lint * feat(q-dev): add per-user model usage table and models column Add "Per-User Model Usage" table (panel 11) showing each user's request count and avg prompt/response length per model_id. Also add "Models Used" column to the Per-User Activity table. * fix(q-dev): remove per-user model usage table, keep models column only * feat(q-dev): add Kiro Executive Dashboard with cross-source analytics New dashboard "Kiro Executive Dashboard" with 12 panels covering: - KPIs: WAU, credits efficiency, acceptance rate, steering adoption - Trends: weekly active users, new vs returning users - Adoption funnel: Chat→Inline→CodeFix→Review→DocGen→TestGen→Agentic→Steering→Spec - Cost: credits pace vs projected monthly, idle power users - Quality: acceptance rate trends, code review findings, test generation - Efficiency: per-user productivity table with credits/line ratio Correlates data across user_report (credits), user_data (code metrics), and chat_log (interaction patterns) for holistic Kiro usage insights. * fix(q-dev): fix pie charts to show per-row slices instead of single total Set reduceOptions.values=true so Grafana treats each SQL result row as a separate pie slice. Fixes Model Usage Distribution, File Type Usage, Kiro Feature Adoption, and Active File Types pie charts. * fix(q-dev): cast Hour to string for Active Hours bar chart x-axis * fix(q-dev): fix pie chart single-slice and GROUP BY display_name issues 1. qdev_user_report Panel 4 (Subscription Tier Distribution): set reduceOptions.values=true to show per-tier slices 2. qdev_user_data Panel 6 (User Interactions): remove display_name from GROUP BY, use MAX(display_name) to merge same user * fix(q-dev): prevent data inflation in user_report JOIN user_data user_report has multiple rows per (user_id, date) due to client_type (KIRO_IDE, KIRO_CLI), but user_data has only one row per (user_id, date). A direct JOIN causes user_data metrics to be counted multiple times. Fix: pre-aggregate user_report by (user_id, date) in a subquery before joining, so the JOIN is always 1:1. Affects: Credits Efficiency stat and User Productivity table.
* feat(qa): add is_invalid field to qa_test_case_executions Add is_invalid boolean field to the domain layer qa_test_case_executions table to allow QA teams to flag test executions as invalid due to environmental issues, flaky tests, false positives, or false negatives. Changes: - Add IsInvalid field to QaTestCaseExecution domain model - Create migration script (20260313_add_is_invalid_to_qa_test_case_executions) - Register migration in migrationscripts/register.go - Update customize service to set default value for is_invalid - Update E2E test data to include new column Resolves apache#8763 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(qa): handle missing is_invalid column in CSV import Fix PostgreSQL compatibility issue when CSV files don't contain the is_invalid column. The field now defaults to false instead of an empty string. Changes: - Update qaTestCaseExecutionHandler to check for empty string values - Add E2E test for backward compatibility with CSV files lacking is_invalid - Add explicit IsInvalid initialization in Testmo plugin converter Resolves apache#8763 --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(linker): branch names containing issue keys * chore: add testing data
Co-authored-by: Klesh Wong <klesh@qq.com>
…ations (apache#8866) * fix(migrations): add new fields and improve migration scripts for various plugins * feat(docker): add Dockerfile.local for local development and update docker-compose configuration * chore: add license to dockerfile --------- Co-authored-by: Fábio Luciano <fabio.gois@sicoob.com.br>
* feat(rootly): scaffold plugin skeleton
Add the plugin entry point, impl shell with all required plugin
interfaces except DataSourcePluginBlueprintV200 (wired in U2),
connection model with Bearer auth, service + scope-config + placeholder
incident/user/assignment models, and the initial migration script. API
resources and subtasks are intentionally empty and will be populated by
U2-U5.
* feat(rootly): add API handlers and blueprint v200
Wire up connection test, remote scope list/search, scope CRUD, scope
sync state, and blueprint v200 endpoints. Remote scope listing speaks
JSON:API and page-based pagination to match the Rootly API shape.
TestConnection validates the bearer token via GET /users/current.
impl.go now wires api.Init and restores DataSourcePluginBlueprintV200
so the plugin can produce pipeline plans for selected Rootly services.
* feat(rootly): collect and convert services to domain boards
Add services_collector, services_extractor, and service_converter
subtasks. Collector fetches GET /services/{id} for the scoped service,
unwraps the JSON:API envelope, and populates _tool_rootly_services.
Converter emits one ticket.Board per service using the standard
domain-id generator, feeding into the existing TICKET pipeline.
Also harden remote-scope pagination termination to key off the page
we requested rather than meta.current_page so a response missing that
field does not silently truncate the service list.
* feat(rootly): collect and extract incidents with inline users
Finalize the Incident and User models and add role-specific user-id
fields (creator, started_by, mitigated_by, resolved_by, closed_by)
directly on the Incident row. Add the incidents_collector and
incidents_extractor subtasks. The collector is single-phase,
filter[services]-scoped and filter[updated_at][gt]-incremental; the
extractor unwraps the JSON:API envelope and pulls inline nested user
objects from the incident attributes, emitting deduplicated User rows
per incident.
Drop the Assignment table entirely. Rootly's incident data model is
role-per-lifecycle-event, not a list of assignees, so PagerDuty's
n-assignees shape does not fit. The schema and GetTablesInfo are
adjusted accordingly.
* feat(rootly): convert incidents to domain issues
Add incidents_converter. For each tool-layer incident, emit a
ticket.Issue (Type=INCIDENT) with status, priority, lead time, and
resolution date derived from the corresponding Rootly fields; emit one
ticket.IssueAssignee per distinct role user (creator, started_by,
mitigated_by, resolved_by, closed_by); and emit a ticket.BoardIssue
linking the incident to its service board. This feeds DevLake's
existing DORA pipeline so change-failure rate and MTTR compute
correctly for teams running on Rootly.
Unknown incident statuses fall back to IN_PROGRESS with a warning log
rather than panicking (a deliberate divergence from the PagerDuty
plugin, motivated by Rootly's more volatile status enum). Severity
mapping accepts case-insensitive sev0-sev4; unknown severities are
preserved as-is.
Guard computeLeadTime against resolved-before-started timestamps (clock
skew or backfill anomalies) by returning nil rather than the wraparound
garbage a naive uint() cast would produce. Tighten test coverage on the
dedup, key-fallback, and known-status-warning boundaries flagged during
code review.
* test(rootly): add e2e test for extract and convert pipeline
Add an end-to-end test that drives the extract and convert subtasks
over a crafted raw-incident fixture and verifies the tool-layer and
domain-layer output against snapshot CSVs. Fixtures cover every branch
of the status and severity mapping tables, the same-user-across-roles
dedup path, the zero-user case, and the safety-net filter that drops
incidents whose relationships point at a different service than the
one the task was scoped to.
* feat(rootly): register plugin in backend and config-ui
Add rootly to the backend plugin startup test and the table-info test
so CI exercises it alongside the other plugins. Register RootlyConfig
in the config-ui plugin list so operators can create connections,
browse scopes, and run blueprints through the UI.
The cloud endpoint default includes the /v1/ API version prefix,
which is what Rootly's REST API actually expects; without it requests
land on the marketing site and return 404. The DOC_URL entries point
at a Configuration/Rootly docs page that still needs to be written.
Use the Rootly wordmark glyph as the plugin icon, rewritten as a
fill-aware (currentColor) SVG so the config-ui can recolor it for
selected/unselected states the same way it does every other plugin
icon.
* fix(rootly): align archived models with live model schemas
The U1 init migration's archived Connection, Service, and ScopeConfig
models were missing columns contributed by the live models' embedded
helpers, so AutoMigrate produced tables the live models could not
read or write. Each gap surfaced as an "Unknown column" error from
MySQL the first time the table was touched:
- Connection was missing endpoint, proxy, rate_limit_per_hour
(contributed by helper.RestConnection on the live struct).
- Service was missing scope_config_id (contributed by common.Scope).
- ScopeConfig was missing connection_id and name (contributed by
common.ScopeConfig, which the archived base type does not include).
Fold the missing fields into the archived models so a single init
migration produces the correct schema. Since rootly has not been
deployed anywhere, keeping one init migration is cleaner than
chaining follow-up ALTERs; a fresh migrate creates the correct
tables in one pass.
* fix(rootly): align API contract with real Rootly /v1 responses
The original plugin shape was built from docs summaries rather than an
actual response capture and diverged from the Rootly API in ways that
silently produced zero incidents. Reconcile every code path with
ground truth from the OpenAPI spec and a captured GET /v1/incidents
response:
- Connection test hits /users/me (Rootly's real "who am I" path); the
original /users/current returns 404.
- Incident list filter is filter[service_ids]=<uuid>, not
filter[services]=<uuid>. The latter exists but accepts names and
silently matches nothing for a UUID.
- Role-bearing user fields (user, started_by, mitigated_by,
resolved_by, closed_by) and severity are JSON:API response
envelopes nested on attributes: {"data":{"id":...,"attributes":...}}.
The previous flat NestedUser / SeverityAttrs shapes were reading the
wrong paths, so those fields were always empty.
- Service membership lives on the sibling relationships block as
JSON:API id+type pointers, not on attributes. The safety-net
scope-filter check now reads from the right place.
- The incident resource does not have an urgency field. Drop the
corresponding column from the model and archived schema.
Also harden the collector: split the ResponseParser / next-page logic
so pagination state is captured during parse (rather than re-reading
the already-drained response body in GetNextPageCustomData), and add
lightweight request/response diagnostics gated behind Debug logging.
Verified end-to-end against a live Rootly tenant: 3 of 6 scoped
services returned incidents, all 3 extracted and converted into
ticket.Issue rows with creator assignees and board linkage.
* style(rootly): trim narrative comments to match plugin conventions
Match PagerDuty's comment density. Keep the few comments that flag
non-obvious invariants (archived-base field overrides, 1-based
pagination, deliberate divergence from PagerDuty's panic-on-unknown
behavior, clock-skew guard).
* refactor(rootly): address code review feedback
Apply fixes from a multi-lens code review pass:
- API: rename swagger {serviceId} to {scopeId} to match registered
route; remove dead Proxy handler.
- Models: add Sanitize() on RootlyConn; add RoleUserIds() helper on
Incident; index ServiceId; drop unused Url field on User; remove
dead RootlyResponse/ApiUserResponse types.
- Migrations: mirror live schema in archived models (index on
service_id; drop user.url).
- Collector: switch pagination to reqData.Pager.Page (avoids
divide-by-zero), cap at 10000 pages, extract buildIncidentsQuery
as a pure helper, drop unreachable lastPageEmpty branch and unused
TotalCount, remove diagnostic logs; add unit test pinning the
filter[service_ids] param literal as a regression guard.
- Services: preserve ScopeConfigId across re-collections; declare
ProductTables on collector and extractor metas.
- Extractor: skip emitting User rows with neither name nor email so
sibling scope tasks can fill in fuller data; use generic resolve()
for SequentialId; type ServiceRef as a named struct.
- Converter: consolidate mapStatus to return (mapped, known); use
Incident.RoleUserIds() instead of an inline slice.
- impl.go: comment justifying services-before-incidents subtask
ordering.
- e2e: rewrite raw incident fixtures to JSON:API envelope shape;
regenerate snapshots (drop urgency column).
* feat(rootly): add rootly dashboard
* chore: run gofmt
* chore: add isBeta flag to rootly plugin
* fix: harden dbt pipeline inputs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(test): disable auth for local e2e server Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Klesh Wong <kleshwong@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… logic (apache#8879) Co-authored-by: Klesh Wong <kleshwong@gmail.com>
Reject spoofed X-Forwarded-User headers unless a configured shared secret matches the forwarded secret header. Co-authored-by: Klesh Wong <kleshwong@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* chore: upgrade go to 1.26 * ci: update github action versions * refactor: replace deprecated go APIs --------- Co-authored-by: Klesh Wong <klesh@qq.com>
…assigned (apache#8892) * fix: map Rootly 'completed' status to DONE in incident converter The mapStatus function was missing 'completed' as a terminal status, causing incidents with that status to be written as IN_PROGRESS instead of DONE. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: use updated_date as MTTR fallback for Rootly 'completed' incidents Rootly's 'completed' status represents a fully resolved incident that has gone through post-incident review. These incidents may not have a resolved_date populated. Fall back to updated_date so MTTR is computed correctly instead of showing the incident as unresolved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: remove service filter requirement from Rootly incident collection Allow collecting all incidents without filtering by service ID. Previously, all incidents required a service association to be collected; incidents created without service tags (common in Rootly) were silently dropped. Changes: - ValidateTaskOptions: service_id is now optional (no longer required) - GetParams: falls back to scope_id 'all' when service_id is empty - buildIncidentsQuery: only adds filter[service_ids] when service_id is set - extractRootlyIncident: skips the service-membership guard when collecting globally (service_id = '') to avoid false drops Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: Add claude code metrics data integration (cherry picked from commit 78fd47d9e7647ff3ac8b886b2ddb45f1df55e2b4) (cherry picked from commit fa69cf9ec0fa0db6aa5fb0aec2e5ca2796355ffd) (upstream) story: XPL-551: Add license headers (cherry picked from commit a12ece8c1ca65ff27acc14d4630ac3d404d51ce8) * feat: Fix header validation issues, migrations and tests * feat: Update plugin table tests * feat: Update claude_code unit tests --------- Co-authored-by: tamas.albert <tamas.laczkoalbert@concentrix.com> Co-authored-by: Eldrick Wega <eldrick.wega@outlook.com>
…y pipelineId (apache#8885) - Add HTTP status code check in listBitbucketWorkspaces, mirroring the existing check in listBitbucketRepos. A non-2xx Bitbucket response was silently unmarshalled as an empty WorkspaceResponse, causing the UI to show No data to select with no error surfaced to the user. - Guard against calling the pipeline subtasks API with an empty pipelineId in card.tsx and step-4.tsx. The record is initialised with pipelineId="" in step-2 before a pipeline has been triggered, causing the frontend to issue GET /pipelines//subtasks and receive a 400 invalid pipeline ID format error from the backend. Fixes apache#8868
…th was enabled (apache#8893) * feat: require authentication for /proceed-db-migration endpoint if auth was enabled * fix: linting * fix: disable codespell due to it is blocked by asf runner --------- Co-authored-by: Klesh Wong <kleshwong@gmail.com>
… Rate (apache#8890) The Change Failure Rate panels summed `has_incident`, which was the count of distinct incidents per deployment. A deployment that caused multiple incidents was counted multiple times, inflating CFR above the DORA definition (deployments that caused a failure / total deployments). Make `has_incident` a 0/1 flag so each deployment counts once regardless of how many incidents it caused, and relabel the companion stat from "incident count" to "failed deployment count" so the displayed counts match the rate. Validated against a production dataset: a 30-day window with 6 incidents across 4 deployments out of 23 total dropped from 26.1% to the correct 17.4%. Signed-off-by: Spiff Azeta <spiffazeta@yahoo.com>
…ckward compatibility (apache#8895) Co-authored-by: Klesh Wong <kleshwong@gmail.com>
Sync latest upstream changes including rootly plugin, claude code metrics, OIDC auth, AES-GCM encryption, push API, GitHub incremental collection, dark theme, Go 1.26 upgrade, and numerous bug fixes. Preserves all downstream customizations (testregistry, aireview, codecov plugins, Konflux/Tekton integration). Dockerfiles unchanged. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolves conflicts in migrationscripts/register.go (keep both upstream Apache and downstream aireview migrations), config-ui.yml (keep our simplified lint workflow), and run_tests.sh (keep upstream uv-based runner). Ran go mod tidy to sync go.sum. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Align with upstream apache/devlake Go version (1.26) across Dockerfile, Dockerfile.konflux, and lake-builder Dockerfile. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Upstream switched from Poetry to uv for Python plugin builds. The merged build.sh now calls uv, which fails with Poetry installed. Also add --chown=devlake:devlake to COPY python/ to fix permission denied when uv creates .venv. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Code Review by Qodo
1. CORS blocks CSRF header
|
| // CSRF only applies when the caller is authenticating via the session | ||
| // cookie. Bearer tokens and proxy headers aren't replayable cross-site. | ||
| if _, err := c.Cookie(oidchelper.SessionCookieName); err != nil { | ||
| c.Next() | ||
| return | ||
| } |
There was a problem hiding this comment.
1. Csrf blocks non-cookie auth 🐞 Bug ≡ Correctness
auth.Service.CSRFProtect() enforces CSRF whenever a devlake_session cookie is present, even if the request was authenticated via OAuth2Proxy headers (or other non-cookie methods), which can cause valid unsafe requests to fail with 403. This is inconsistent with the middleware’s own comment and the global middleware ordering (OAuth2ProxyAuthentication runs before CSRFProtect).
Agent Prompt
### Issue description
`CSRFProtect` currently decides to enforce CSRF solely by checking whether the session cookie exists on the request. In the global middleware chain, authentication can be established via non-cookie mechanisms (e.g., `OAuth2ProxyAuthentication`) while the request still carries a stale/irrelevant `devlake_session` cookie, leading to unexpected 403 responses.
### Issue Context
- The code comment says CSRF should apply only when authenticating via the session cookie, but the implementation uses cookie presence as a proxy for “auth via cookie”.
- Since `CSRFProtect` is installed globally after `OAuth2ProxyAuthentication`, a request can be fully authenticated without using the cookie.
### Fix Focus Areas
- backend/server/api/auth/middleware.go[54-96]
- backend/server/api/auth/middleware.go[128-168]
- backend/server/api/api.go[120-129]
### Suggested fix
- In `OIDCAuthentication`, when a session cookie is successfully parsed/validated and used to set `common.USER`, also set a context flag (e.g., `c.Set("auth_via_session_cookie", true)`).
- In `CSRFProtect`, enforce CSRF only when that flag is set (or equivalent signal that cookie-based session auth was actually used for this request), instead of checking only for cookie presence.
- Keep current behavior for unauthenticated/public routes and safe HTTP methods.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
Code review by qodo was updated up to the latest commit 1fba255 |
| "github.com/spf13/cobra" | ||
| ) | ||
|
|
||
| var PluginEntry impl.Asana //nolint |
There was a problem hiding this comment.
1. Unjustified //nolint directives 📘 Rule violation ⚙ Maintainability
New //nolint directives were added without an accompanying justification comment. This violates linting compliance requirements and makes it harder to audit why lint rules are being suppressed.
Agent Prompt
## Issue description
New `//nolint` directives were introduced without justification comments.
## Issue Context
Compliance requires every `//nolint` to include a justification comment explaining why suppression is necessary.
## Fix Focus Areas
- backend/plugins/asana/asana.go[26-26]
- backend/plugins/icla/api/swagger.go[28-28]
- backend/plugins/icla/api/swagger.go[44-44]
- backend/plugins/taiga/models/connection.go[79-79]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| "github.com/spf13/cobra" | ||
| ) | ||
|
|
||
| var PluginEntry impl.Asana //nolint |
There was a problem hiding this comment.
2. pluginentry lacks godoc 📘 Rule violation ⚙ Maintainability
The new exported identifier PluginEntry is missing a proper Godoc comment. This violates the requirement to document exported Go identifiers for maintainability and lint compliance.
Agent Prompt
## Issue description
The exported variable `PluginEntry` is missing a Godoc comment.
## Issue Context
Exported identifiers must have comments that start with the identifier name.
## Fix Focus Areas
- backend/plugins/asana/asana.go[26-26]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| } | ||
|
|
||
| var sessionProvider string | ||
| if len(s.cfg.SessionSecret) > 0 { |
There was a problem hiding this comment.
3. sessionsecret checked with len() 📘 Rule violation ⚙ Maintainability
New code uses len(s.cfg.SessionSecret) > 0 / == 0 for empty-string checks instead of direct `!= "" / == ""` comparisons. This violates the required Go string conventions and reduces readability.
Agent Prompt
## Issue description
String emptiness is checked via `len(s) > 0` / `len(s) == 0` in new/modified code.
## Issue Context
Compliance requires direct empty-string checks (e.g., `s != ""`) for clarity and idiomatic Go.
## Fix Focus Areas
- backend/server/api/auth/auth.go[360-360]
- backend/server/api/auth/middleware.go[67-67]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| body, e := json.Marshal(map[string]string{ | ||
| "type": "normal", | ||
| "username": tc.Username, | ||
| "password": tc.Password, | ||
| }) | ||
| if e != nil { | ||
| return "", errors.Default.WrapRaw(e) | ||
| } | ||
|
|
||
| resp, e := http.Post(authURL, "application/json", bytes.NewReader(body)) //nolint:noctx | ||
| if e != nil { | ||
| return "", errors.Default.WrapRaw(e) | ||
| } | ||
| defer resp.Body.Close() | ||
|
|
||
| if resp.StatusCode != http.StatusOK { | ||
| return "", errors.Default.New(fmt.Sprintf("taiga auth failed with status %d", resp.StatusCode)) | ||
| } | ||
|
|
||
| var result map[string]interface{} | ||
| if e := json.NewDecoder(resp.Body).Decode(&result); e != nil { | ||
| return "", errors.Default.WrapRaw(e) | ||
| } | ||
| // Taiga returns auth_token (v5) or token (v6) | ||
| for _, key := range []string{"auth_token", "token"} { | ||
| if t, ok := result[key]; ok { | ||
| if token, ok := t.(string); ok && token != "" { | ||
| return token, nil | ||
| } | ||
| } | ||
| } | ||
| // fallback: read raw body hint | ||
| raw, _ := io.ReadAll(bytes.NewReader(body)) | ||
| return "", errors.Default.New(fmt.Sprintf("taiga auth response missing token field, body: %s", string(raw))) |
There was a problem hiding this comment.
4. Taiga error includes password 📘 Rule violation ⛨ Security
fetchToken() constructs an error message that includes the serialized request body, which contains tc.Password. If this error is logged upstream, it can leak credentials into logs.
Agent Prompt
## Issue description
An error path includes `tc.Password` by embedding the marshaled request body in the returned error message.
## Issue Context
Errors are commonly logged by callers; including credentials in error strings risks secret leakage.
## Fix Focus Areas
- backend/plugins/taiga/models/connection.go[70-74]
- backend/plugins/taiga/models/connection.go[101-103]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| // Auth chain order matters: REST API key first (its own short-circuit), | ||
| // then the push API key gate, then OIDC session, then oauth2-proxy header | ||
| // (only sets USER if not yet set and the forwarded secret matches), then the terminal 401 gate, | ||
| // finally CSRF on unsafe methods. | ||
| router.Use(RestAuthentication(router, basicRes)) | ||
| router.Use(RequirePushAuthentication(basicRes)) | ||
| router.Use(auth.OIDCAuthentication()) | ||
| router.Use(OAuth2ProxyAuthentication(basicRes)) | ||
| router.Use(auth.RequireAuth()) | ||
| router.Use(auth.CSRFProtect()) |
There was a problem hiding this comment.
5. Cors blocks csrf header 🐞 Bug ≡ Correctness
The PR adds CSRF enforcement via the X-CSRF-Token header, but the server’s CORS configuration does not allow that header, causing browser preflight failures for credentialed requests from the UI. As a result, authenticated unsafe requests (POST/PUT/PATCH/DELETE) from config-ui can be blocked by the browser before reaching the API.
Agent Prompt
## Issue description
CSRF protection now requires the `X-CSRF-Token` header on unsafe methods, but `cors.Config.AllowHeaders` does not include `X-CSRF-Token`. Browsers will fail CORS preflight for these requests, breaking the UI’s authenticated mutations.
## Issue Context
The frontend Axios client is configured to send `X-CSRF-Token` automatically (double-submit cookie pattern). The backend must advertise this header in CORS responses.
## Fix Focus Areas
- backend/server/api/api.go[103-112]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
/retest |
1 similar comment
|
/retest |
pr-type/bug-fix,pr-type/feature-development, etc.Summary
What does this PR do?
Does this close any open issues?
Closes xx
Screenshots
Include any relevant screenshots here.
Other Information
Any other information that is important to this PR.