diff --git a/docs/maintenance-guides/README.md b/docs/maintenance-guides/README.md new file mode 100644 index 00000000000..937f292cd0b --- /dev/null +++ b/docs/maintenance-guides/README.md @@ -0,0 +1,170 @@ +# PD Maintenance Guides + +This directory is a maintainership-oriented index for the PD repository. It is +not a user manual. It is meant to help maintainers, reviewers, and contributors +answer four questions quickly: + +1. Which subsystem owns this behavior? +2. Which files are the real entry points? +3. Which invariants are easy to break? +4. Which tests, metrics, and docs should move with the change? + +## Maintainer Contract + +These files are part of the repository's maintenance surface. + +- Developers and reviewers should read the relevant guide here before making or + reviewing non-trivial changes. +- These guides are assistant documents for implementation and review, not + optional afterthoughts. +- If a change modifies ownership boundaries, startup order, data contracts, + invariants, operational signals, or the recommended reading map for a covered + subsystem, update the matching guide in the same change. +- A code change that invalidates these guides but does not update them should be + treated as an incomplete maintenance change. + +## Standard Section Contract + +Each subsystem guide is expected to cover these domains: + +1. Purpose and scope +2. Core concepts +3. Architectural views +4. Process lifecycle and startup sequencing +5. Data model and metadata contracts +6. Observability and operational signals +7. Change management guidance +8. Reading map and companion docs when they add subsystem-specific context +9. Glossary +10. Must-read file order +11. Change-impact matrix or review checklist + +The depth varies by subsystem size. Small or focused guides can keep sections +short. Reading-map material can be covered by `Must-Read File Order` and +[repo-overview](./repo-overview.md), but `Core Concepts` and `Glossary` should +remain present because they are the main retrieval anchors for agents and new +reviewers. + +## How To Use This Set + +- Start with [Repository Overview](./repo-overview.md) to understand layer + boundaries. +- Open the guide for the subsystem you are touching. +- Use the "Core Concepts", "Glossary", "Must-Read File Order", and any + "Change-Impact Matrix" or "Review Checklist" sections first. +- Treat every guide as a map, not a complete specification. The source of truth + is still the code. + +## Agent Retrieval Entry Points + +Agents should treat this directory as a retrieval map: + +1. Read this file to classify the subsystem and find the matching guide. +2. Read [repo-overview](./repo-overview.md) for cross-component ownership, + common terms, and change-impact routing. +3. Read the subsystem guide's `Core Concepts` and `Glossary` before searching + code so terms like leader, primary, fallback, region, keyspace, and rule are + interpreted in the right PD context. +4. Use the subsystem guide's `Must-Read File Order` as the first code-search + seed, then expand with `rg` from the concrete symbols listed there. + +## System Map + +- Process bootstrap, embedded etcd, PD leader election, service wiring: + [server](./server.md) +- Static config, persisted options, dynamic config side effects: + [config](./config.md) +- PD member, leader election, service primary election, etcd utilities: + [member-election](./member-election.md) +- gRPC, HTTP APIs, request forwarding, and client compatibility: + [api-and-client](./api-and-client.md) +- Cluster metadata, store/region cache, heartbeats, and background cluster jobs: + [cluster](./cluster.md) +- Scheduler coordinator, checkers, schedulers, operators, placement rules: + [scheduling](./scheduling.md) +- Region/store statistics, hot cache, store load and hot peer signals: + [statistics](./statistics.md) +- Placement rules, region labels, affinity groups and policy fitting: + [placement-policy](./placement-policy.md) +- Metadata storage backends and endpoint contracts: + [storage](./storage.md) +- Timestamp oracle, TSO allocation, and TSO primary behavior: + [tso](./tso.md) +- Keyspace metadata, keyspace groups, microservice discovery and split services: + [keyspace-and-microservices](./keyspace-and-microservices.md) +- Resource groups, RU token buckets, service limits, resource manager service: + [resource-manager](./resource-manager.md) + +## Guide Index + +### Repository + +- [Repository Overview](./repo-overview.md) + +### Core PD + +- [server](./server.md) +- [config](./config.md) +- [member-election](./member-election.md) +- [api-and-client](./api-and-client.md) +- [cluster](./cluster.md) +- [scheduling](./scheduling.md) +- [statistics](./statistics.md) +- [placement-policy](./placement-policy.md) +- [storage](./storage.md) +- [tso](./tso.md) +- [keyspace-and-microservices](./keyspace-and-microservices.md) +- [resource-manager](./resource-manager.md) + +## Cross-Cutting Review Checklist + +- Check whether the change touches a request hot path: + TSO, region heartbeat, store heartbeat, scheduler operator dispatch, or API + forwarding. +- Verify leader/primary ownership: + PD leader, embedded etcd leader, TSO primary, scheduling primary, resource + manager primary, and microservice fallback are separate concepts. +- Check metadata contracts: + cluster ID, store ID, region epoch, placement rules, keyspace state, keyspace + group assignment, service primary keys, and persisted config. +- Check storage routing: + region metadata can use the dedicated local region storage, while most other + metadata remains etcd-backed. +- Check runtime config behavior: + persisted option updates must have the matching in-memory side effect. +- Check statistics and policy inputs: + placement rules, region labels, affinity groups, store loads, and hot cache + state can change scheduler behavior without touching scheduler code. +- Check failure semantics: + leader change, context cancellation, stream close, follower forwarding, + timeout, stale region, and retry behavior. +- Check observability: + metrics, logs, API responses, gRPC errors, and health/readiness endpoints + should move with behavior changes. +- Move tests with the behavior: + use failpoint-aware make targets and prefer package-level targeted tests for + focused changes. + +## Current Scope + +This first guide set covers the main PD maintenance boundaries: + +- `server/` +- `server/config/` +- `server/api/` +- `server/apiv2/` +- `server/cluster/` +- `pkg/member/` +- `pkg/utils/etcdutil/` +- `pkg/core/` +- `pkg/schedule/` +- `pkg/statistics/` +- `pkg/storage/` +- `pkg/tso/` +- `pkg/keyspace/` +- `pkg/mcs/` +- `pkg/mcs/resourcemanager/` +- `client/` + +The overview mentions additional packages for cross-component reasoning. They +do not all have dedicated subsystem guides yet. diff --git a/docs/maintenance-guides/api-and-client.md b/docs/maintenance-guides/api-and-client.md new file mode 100644 index 00000000000..cb01796e0bb --- /dev/null +++ b/docs/maintenance-guides/api-and-client.md @@ -0,0 +1,155 @@ +# API And Client + +This guide covers PD's external API boundary and client compatibility surface. + +## Purpose And Scope + +Covered paths: + +- gRPC services in `server/grpc_service.go` +- REST APIs in `server/api/` +- v2 APIs and middleware in `server/apiv2/` +- PD client module in `client/` +- request forwarding to leaders or split microservices +- API response compatibility and error semantics + +This guide does not define scheduler or storage behavior, but API changes often +need review with those subsystem guides. + +## Core Concepts + +- The API boundary includes gRPC, HTTP v1, HTTP v2, middleware, forwarding, and + the Go client submodule. +- Request ownership can require PD leader, any PD member, or a specific + microservice primary. +- gRPC `ResponseHeader` and HTTP error shapes are compatibility contracts with + TiKV, TiDB, tools, and external users. +- Forwarding and redirect behavior must be bounded by context, timeout, and + service-discovery freshness. +- Route labels, rate limiter labels, audit labels, metrics, and swagger/client + types should move with route changes. + +## Architectural Views + +### gRPC view + +`GrpcServer` wraps `server.Server` and implements PD gRPC methods such as TSO, +bootstrap, store heartbeat, region heartbeat, split, scatter, and operator +lookup. Several methods validate leadership, redirect or forward requests, and +then call `RaftCluster`, `TSO`, or scheduler-owned logic. + +### HTTP view + +`server/api/` exposes the v1 operational API for members, config, stores, +regions, schedulers, rules, operators, health, diagnostics, maintenance, and +debug behavior. `server/apiv2/` adds v2 handlers and middleware for readiness, +keyspace, safe point, microservice redirects, affinity, and maintenance. + +### Client view + +`client/` is a Go submodule. It owns service discovery, HTTP helpers, TSO and +resource-manager clients, keyspace and meta-storage clients, options, metrics, +and compatibility types shared with external users. + +## Process Lifecycle And Startup Sequencing + +- gRPC services are registered through the embedded etcd config's + `ServiceRegister` hook during server creation. +- HTTP handlers are installed through legacy service builders and the + microservice registry. +- gRPC service labels are initialized after embedded etcd starts and the gRPC + server service info is available. +- Client-side service discovery watches PD or microservice endpoints and + updates request routing after leader or primary changes. + +Maintenance rule: + +- Handler registration is part of process startup. Do not move it without + checking rate limit labels, audit labels, and readiness behavior. + +## Data Model And Metadata Contracts + +API-visible contracts include: + +- `pdpb.ResponseHeader` and region error semantics +- stream close and timeout behavior for TSO and region heartbeat +- JSON response fields in v1 and v2 handlers +- `client/http/types.go` structures mirrored from server responses +- status types exported from `server/cluster` +- service discovery endpoint format +- HTTP status codes and errcode response shape + +Maintenance rule: + +- Changing exported response fields or error behavior is a compatibility change + even when no Go API signature changes. + +## Observability And Operational Signals + +Open these first: + +- `server/metrics.go` +- `server/api/metric.go` +- `server/api/health.go` +- `server/api/status.go` +- `client/metrics/metrics.go` + +Signals to preserve: + +- forwarding failure counters +- TSO proxy stream counters and timeout logs +- heartbeat stream send/receive errors +- API rate limiter labels +- health and readiness behavior + +## Change Management Guidance + +- For gRPC changes, review TiKV and TiDB callers through kvproto expectations. +- For HTTP changes, review `client/http` type compatibility and swagger + annotations if the route is documented. +- For forwarding changes, check leader, primary, retry, and timeout behavior. +- For client changes, run client submodule tests through its make targets. + +## Must-Read File Order + +1. `server/grpc_service.go` +2. `server/forward.go` +3. `server/api/router.go` +4. `server/api/server.go` +5. `server/api/middleware.go` +6. `server/apiv2/router.go` +7. `server/apiv2/middlewares/redirector.go` +8. `server/apiv2/middlewares/microservice_redirector.go` +9. `client/client.go` +10. `client/servicediscovery/service_discovery.go` +11. `client/http/client.go` +12. `client/http/types.go` + +## Glossary + +- gRPC boundary: + protobuf-defined service methods implemented by `GrpcServer`. +- HTTP v1: + legacy operational API under `server/api`. +- HTTP v2: + newer API surface under `server/apiv2`. +- Forwarding: + proxying or redirecting requests to the current leader or service primary. +- `ResponseHeader`: + gRPC response metadata carrying cluster, leader, and error information. +- Service discovery: + client-side or server-side lookup of PD and microservice endpoints. +- Client submodule: + separate Go module under `client/` consumed by external callers. + +## Review Checklist + +- Does the handler require PD leader, any member, or a specific microservice + primary? +- Is request forwarding bounded by context, timeout, and retry limits? +- Are stream send and receive paths closed on both endpoint errors? +- Are response headers and error fields compatible with existing TiKV and TiDB + clients? +- Does the HTTP handler use the existing errcode/error response pattern? +- Do new APIs need swagger annotations or client type updates? +- Are audit, rate limit, and service labels updated with route changes? diff --git a/docs/maintenance-guides/cluster.md b/docs/maintenance-guides/cluster.md new file mode 100644 index 00000000000..5152e907696 --- /dev/null +++ b/docs/maintenance-guides/cluster.md @@ -0,0 +1,164 @@ +# Cluster + +This guide covers cluster metadata ownership in `server/cluster/` and the +in-memory store/region model in `pkg/core/`. + +## Purpose And Scope + +`RaftCluster` owns: + +- cluster bootstrap state +- store and region heartbeat processing +- store and region metadata persistence +- background cluster jobs +- region synchronization to followers +- replication mode and GC integration +- scheduling coordinator wiring +- slow-store and service fallback checks + +`pkg/core` owns the in-memory data structures and query interfaces used by +schedulers, APIs, and statistics. + +## Core Concepts + +- `RaftCluster` is leader-owned cluster runtime state, not a raft + implementation. +- `BasicCluster` is the in-memory store/region snapshot model consumed by + schedulers, APIs, and statistics. +- Heartbeat processing separates validation, cache update, statistics, storage + persistence, and follower sync. +- Region and store objects are snapshot-like. Prefer clone/update helpers over + mutating shared structures. +- Service-independent mode changes whether classic PD or a split scheduling + service owns scheduling behavior. + +## Architectural Views + +### Metadata cache view + +`BasicCluster` combines `StoresInfo` and `RegionsInfo`. `StoreInfo` and +`RegionInfo` are mostly immutable snapshots with clone/update helpers. The +cluster updates these snapshots from storage, heartbeats, and follower sync. + +### Heartbeat view + +Store heartbeats update store stats and state. Region heartbeats update region +metadata, detect stale or overlapping regions, update trees, persist when +needed, trigger statistics collection, and sync follower PD members. + +### Background job view + +When the PD member becomes leader, `RaftCluster.Start` initializes the cluster +and launches service checks, metrics collection, node state checks, region +syncing, replication mode, min resolved TS, store config sync, store stats, +GC-related jobs, progress GC, and storage size collection. + +## Process Lifecycle And Startup Sequencing + +Important anchors: + +1. `cluster.NewRaftCluster` +2. `RaftCluster.InitCluster` +3. `RaftCluster.Start` +4. `RaftCluster.LoadClusterInfo` +5. `RaftCluster.HandleStoreHeartbeat` +6. `RaftCluster.HandleRegionHeartbeat` +7. `RaftCluster.processRegionHeartbeat` +8. `RaftCluster.Stop` + +Maintenance rules: + +- `Start` is leader-owned behavior. Do not make follower-only code depend on + leader-only initialization. +- Background jobs must stop through `RaftCluster.Stop`. +- Region heartbeat processing deliberately separates cache update, async stats, + async persistence, and follower sync. + +## Data Model And Metadata Contracts + +Hot contracts: + +- `metapb.Cluster` +- `metapb.Store` +- `metapb.Region` +- region epoch, key range, peers, leader, buckets, source freshness, and + approximate statistics +- store labels, state, address fields, slow-store status, store limits, and min + resolved TS +- `core.RegionInfo` clone and inheritance rules +- `core.StoreInfo` clone and store stats rules +- region tree and sub-tree reference behavior + +Maintenance rule: + +- A region heartbeat can be fresher for some fields and stale for others. Always + inspect `PreCheckPutRegion`, `Inherit`, `regionGuide`, and overlap handling + before changing region cache behavior. + +## Observability And Operational Signals + +Open these first: + +- `server/cluster/metrics.go` +- `pkg/core/metrics.go` +- `pkg/statistics/metrics.go` +- `pkg/schedule/metrics.go` + +Signals to preserve: + +- region cache update and KV update counters +- cluster start duration +- heartbeat processing logs and errors +- store status and slow-store metrics +- region statistics and hot-region metrics +- min resolved TS and storage size signals + +## Change Management Guidance + +- Region metadata changes usually need scheduler and API review. +- Store metadata changes usually need scheduler filter and placement review. +- Persistence changes need `pkg/storage` review. +- Background job changes need startup/shutdown and leader-transfer tests. +- Follower sync changes need tests for leader/follower region storage behavior. + +## Must-Read File Order + +1. `server/cluster/cluster.go` +2. `server/cluster/cluster_worker.go` +3. `server/cluster/scheduling_controller.go` +4. `pkg/core/basic_cluster.go` +5. `pkg/core/region.go` +6. `pkg/core/region_tree.go` +7. `pkg/core/store.go` +8. `pkg/core/store_stats.go` +9. `pkg/statistics/region.go` +10. `pkg/statistics/store.go` +11. `pkg/storage/region_storage.go` + +## Glossary + +- `RaftCluster`: + PD's leader-owned cluster metadata and background-job coordinator. +- `BasicCluster`: + in-memory store and region collection used by schedulers and APIs. +- Region heartbeat: + TiKV report that updates region metadata, freshness, statistics, and operator + responses. +- Store heartbeat: + TiKV report that updates store stats, state, slow-store signals, and limits. +- Region epoch: + version metadata used to reject stale region state across splits and merges. +- Follower sync: + propagation of region changes from the PD leader to follower PD members. +- Service independent: + state indicating a split microservice owns behavior instead of classic PD. + +## Review Checklist + +- Does the change preserve region epoch and key-range validation? +- Are overlapping regions removed from cache and storage consistently? +- Does async persistence failure remain non-fatal where intended? +- Does follower sync still receive all required region changes? +- Are store and region clones used instead of mutating shared snapshots? +- Does the scheduler see consistent store/region state after the change? +- Are background jobs started and stopped under the right leader state? diff --git a/docs/maintenance-guides/config.md b/docs/maintenance-guides/config.md new file mode 100644 index 00000000000..e98eb4734b7 --- /dev/null +++ b/docs/maintenance-guides/config.md @@ -0,0 +1,169 @@ +# Config + +This guide covers PD configuration in `server/config/` and the runtime +configuration contracts consumed by server, cluster, scheduling, keyspace, TSO, +and resource manager code. + +## Purpose And Scope + +Config owns: + +- static config parsing, defaults, validation, and embedded etcd config + generation +- persisted options that can be loaded from and saved to storage +- runtime-safe accessors for scheduling, replication, PD server, keyspace, + microservice, store, and replication-mode config +- service middleware config and persisted service middleware options +- config API response compatibility +- controller config wiring for resource manager + +Config does not own the side effects of every option. The subsystem that +consumes a config value owns the corresponding runtime behavior. + +## Core Concepts + +- Static config is loaded from flags and TOML before the server starts. +- Persisted options are stored through PD storage and reloaded by the leader. +- Runtime side effects belong to the consuming subsystem, not to config parsing. +- Dynamic config changes must keep storage, in-memory accessors, API responses, + and metrics aligned. +- Some options are temporary or TTL-scoped; review the override lifetime before + changing reload behavior. + +## Architectural Views + +### Static config view + +`Config` is the server-level structure loaded from flags and TOML. It includes +listen URLs, initial cluster settings, leader lease, logging, metrics, +scheduling, replication, PD server options, security, dashboard, replication +mode, keyspace, microservice, resource manager controller, and metering config. + +### Persisted option view + +`PersistOptions` wraps dynamically persisted settings and exposes thread-safe +accessors. It uses `atomic.Value` for most config groups, an atomic pointer for +cluster version, and TTL state for temporary config overrides. + +### Runtime side-effect view + +Updating persisted config is not enough by itself. Consumers must still apply +runtime effects: scheduler limits, placement rule behavior, store config sync, +microservice fallback, TSO settings, keyspace behavior, or resource manager +limits. + +## Process Lifecycle And Startup Sequencing + +Important anchors: + +1. `server/config/config.go` +2. `server/config/persist_options.go` +3. `server/config/service_middleware_config.go` +4. `server/config/service_middleware_persist_options.go` +5. `server/config/util.go` +6. `server/server.go` +7. `server/api/config.go` + +Startup path: + +1. Parse and validate static config. +2. Generate embedded etcd config. +3. Create `PersistOptions` from static config. +4. Start the server and reload persisted config after leader campaign. +5. Subsystems consume config through shared providers and live accessors. + +Maintenance rules: + +- Do not add a config field without a default, validation story, API exposure + decision, and runtime side-effect owner. +- Do not mutate config structs in place when existing code expects clone/store + semantics. +- Persisted config reload must not silently diverge from in-memory state. + +## Data Model And Metadata Contracts + +Hot contracts: + +- TOML tags and JSON tags in `Config` +- exported config structs returned by HTTP APIs +- `PersistOptions` accessor semantics +- scheduler config keys and TTL override keys +- replication config and placement-rule enablement +- PD server config values such as region storage and flow rounding +- microservice fallback and dynamic switching config +- store config reported by TiKV and synced by PD +- controller config used by resource manager + +Maintenance rule: + +- A config rename or default change is a compatibility change. Check command + line flags, TOML files, HTTP APIs, persisted storage, and tests together. + +## Observability And Operational Signals + +Open these first: + +- `server/config/metrics.go` +- `server/api/config.go` +- `server/cluster/cluster.go` +- `pkg/schedule/metrics.go` +- `pkg/mcs/resourcemanager/server/metrics.go` + +Signals to preserve: + +- config update logs +- schedule config metrics +- cluster version and metadata gauges +- resource controller config behavior +- errors returned by config APIs + +## Change Management Guidance + +- Static config changes need validation, default, docs, and startup tests. +- Dynamic config changes need persisted storage, API, and runtime side-effect + tests. +- Scheduling config changes need scheduler/checker/operator review. +- Replication config changes need placement and cluster metadata review. +- Microservice config changes need fallback and service discovery review. +- Resource manager controller config changes need token bucket and metadata + watcher review. + +## Must-Read File Order + +1. `server/config/config.go` +2. `server/config/persist_options.go` +3. `server/config/util.go` +4. `server/config/service_middleware_config.go` +5. `server/config/service_middleware_persist_options.go` +6. `server/api/config.go` +7. `server/server.go` +8. `server/cluster/cluster.go` +9. `pkg/schedule/config/config.go` +10. `pkg/schedule/config/store_config.go` +11. `pkg/mcs/resourcemanager/server/config.go` + +## Glossary + +- Static config: + startup-only configuration loaded from command line flags and TOML. +- Persisted option: + dynamic setting saved in storage and exposed through `PersistOptions`. +- Runtime side effect: + in-memory behavior that must change after a persisted option is updated. +- TTL override: + temporary config value that expires after a configured lifetime. +- Config provider: + shared interface used by subsystems to read current config safely. +- Clone/store semantics: + pattern where config structs are copied before mutation and atomically + replaced for readers. + +## Review Checklist + +- Does the new or changed field have a default and validation? +- Is the field intended to be static, dynamic, or persisted with TTL? +- Are JSON/TOML names stable and compatible? +- Does a persisted update apply the matching runtime side effect? +- Are clone and atomic-store semantics preserved? +- Does the change affect microservice fallback or keyspace group behavior? +- Are API responses and client expectations still compatible? diff --git a/docs/maintenance-guides/keyspace-and-microservices.md b/docs/maintenance-guides/keyspace-and-microservices.md new file mode 100644 index 00000000000..8299653a1e1 --- /dev/null +++ b/docs/maintenance-guides/keyspace-and-microservices.md @@ -0,0 +1,171 @@ +# Keyspace And Microservices + +This guide covers keyspace metadata in `pkg/keyspace/` and PD microservice +infrastructure in `pkg/mcs/`. + +## Purpose And Scope + +Keyspace owns: + +- keyspace creation, lookup, state transitions, and config mutation +- reserved/default keyspace bootstrap +- keyspace region split coordination +- keyspace group assignment, split, merge, and keyspace movement +- meta-service group assignment + +Microservices own: + +- service registration and discovery +- primary election for split services +- TSO, scheduling, resource manager, router, and meta-storage servers +- forwarding and fallback behavior between PD and independent services + +## Core Concepts + +- A keyspace is user-visible control-plane metadata with ID, name, state, and + config. +- Keyspace group metadata drives TSO routing and keyspace movement. +- Meta-service group assignment is stored with keyspace config and can affect + external meta-storage routing. +- Microservices register endpoints in etcd and elect service primaries + independently from PD leadership. +- Classic PD fallback must be explicit because it changes which process serves + TSO, scheduling, or resource-manager behavior. + +## Architectural Views + +### Keyspace metadata view + +`keyspace.Manager` validates requests, serializes mutations with keyed locks, +persists metadata through endpoint storage, updates local lookup caches, and +coordinates region-bound labels and group assignment. + +### Keyspace group view + +`keyspace.GroupManager` manages keyspace group metadata used by TSO routing. It +supports bootstrap, group creation, keyspace movement, split, merge, and service +address lookup. + +### Microservice view + +`pkg/mcs/discovery` and `pkg/mcs/registry` provide service discovery and API +registration. Individual service packages own their server lifecycle, primary +election, metadata watches, and gRPC APIs. + +## Process Lifecycle And Startup Sequencing + +Important anchors: + +1. `keyspace.NewKeyspaceManager` +2. `Manager.Bootstrap` +3. `Manager.CreateKeyspace` +4. `Manager.UpdateKeyspaceConfig` +5. `Manager.UpdateKeyspaceState` +6. `keyspace.NewKeyspaceGroupManager` +7. `GroupManager.Bootstrap` +8. `GroupManager.SplitKeyspaceGroupByID` +9. `GroupManager.MergeKeyspaceGroups` +10. `pkg/mcs/discovery/register.go` +11. `pkg/mcs/*/server/server.go` + +In classic PD, keyspace managers are created during `server.startServer`. In +keyspace group mode, PD also initializes service primary watchers. Independent +microservices register themselves and elect primaries through etcd. + +Maintenance rules: + +- Keyspace group bootstrap happens after other `RaftCluster` startup steps to + avoid stuck goroutines on partial failure. +- Keyspace metadata mutation must hold the appropriate keyed lock. +- Microservice fallback behavior must be explicit and observable. + +## Data Model And Metadata Contracts + +Hot contracts: + +- keyspace ID, name, state, created time, state changed time +- keyspace config keys such as user kind, TSO keyspace group ID, GC management + type, region bound type, and meta-service group fields +- keyspace group membership and split/merge state +- meta-service group assignment +- service registry entries and primary keys +- expected primary flags +- service-independent markers in cluster state + +Maintenance rule: + +- Keyspace metadata is user-visible control-plane state. Compatibility and + rollback behavior matter even when the change is internal. + +## Observability And Operational Signals + +Open these first: + +- `pkg/keyspace/metrics.go` +- `pkg/mcs/*/server/metrics.go` +- `pkg/mcs/discovery` +- `server/apiv2/handlers/keyspace.go` +- `server/apiv2/handlers/microservice.go` + +Signals to preserve: + +- keyspace operation logs and errors +- service discovery and primary election logs +- fallback state transitions +- resource manager service limit and token metrics +- scheduling and TSO service health signals + +## Change Management Guidance + +- Keyspace config changes need storage, API, and TSO routing review. +- Keyspace group split/merge changes need concurrency and retry review. +- Microservice primary changes need failure-path tests and discovery review. +- Resource manager changes should be reviewed with service middleware and + client compatibility. +- Scheduling service changes should be reviewed with classic scheduling + fallback. + +## Must-Read File Order + +1. `pkg/keyspace/keyspace.go` +2. `pkg/keyspace/tso_keyspace_group.go` +3. `pkg/keyspace/meta_service_group.go` +4. `pkg/storage/endpoint/keyspace.go` +5. `pkg/storage/endpoint/tso_keyspace_group.go` +6. `pkg/mcs/discovery/register.go` +7. `pkg/mcs/discovery/discover.go` +8. `pkg/mcs/registry/registry.go` +9. `pkg/mcs/server/server.go` +10. `pkg/mcs/tso/server/server.go` +11. `pkg/mcs/scheduling/server/server.go` +12. `pkg/mcs/resourcemanager/server/server.go` +13. `pkg/mcs/router/server/server.go` + +## Glossary + +- Keyspace: + logical metadata namespace with ID, name, state, and config. +- Default keyspace: + reserved keyspace used for normal non-serverless paths. +- Keyspace group: + group of keyspaces assigned to TSO ownership and routing. +- Meta-service group: + external meta-service assignment used by keyspace metadata. +- Service registry: + etcd-backed record of available split-service endpoints. +- Service primary: + elected owner for one split microservice. +- Fallback: + mode where classic PD serves behavior when the split service is unavailable or + disabled. + +## Review Checklist + +- Is the keyspace mutation serialized on the correct key? +- Are name and ID lookup caches updated consistently with storage? +- Does the change preserve reserved/default keyspace behavior? +- Does keyspace group movement preserve TSO routing correctness? +- Are split and merge operations retry-safe? +- Does microservice discovery distinguish no service, stale service, and + current primary states? +- Does fallback behavior match config and leave clear metrics or logs? diff --git a/docs/maintenance-guides/member-election.md b/docs/maintenance-guides/member-election.md new file mode 100644 index 00000000000..eca849a9a50 --- /dev/null +++ b/docs/maintenance-guides/member-election.md @@ -0,0 +1,172 @@ +# Member And Election + +This guide covers PD membership, leader election, service primary election, and +etcd utilities in `pkg/member/`, `pkg/election/`, and `pkg/utils/etcdutil/`. + +## Purpose And Scope + +Member and election code owns: + +- PD member identity and member metadata +- PD leader key and lease ownership +- service participant identity and primary keys for microservices +- leader/primary campaign, watch, resign, and transfer behavior +- etcd client helpers, leader movement, health checks, and watch loops +- serving-state checks used by APIs, TSO, scheduling, and resource manager + +It does not own the business logic that becomes active after leadership is +obtained. That belongs to server, TSO, cluster, scheduling, or resource manager. + +## Core Concepts + +- Embedded etcd leader, PD leader, and microservice primary are separate + ownership states with different serving guarantees. +- `member.Member` owns PD member identity and PD leader campaign state. +- `member.Participant` generalizes the election model for split-service + primaries. +- Lease-backed serving checks are stronger than cached leader identity. +- Watch loops must handle initial load, live updates, compaction, cancellation, + and retry without splitting ownership. + +## Architectural Views + +### PD member view + +`member.Member` wraps embedded etcd identity, election client, leadership, and +the current PD leader value. `server.leaderLoop` calls `CheckLeader`, aligns +PD leader campaigning with the embedded etcd leader, and calls `Campaign`. + +### Service participant view + +`member.Participant` generalizes the same election model for microservice +primaries. TSO, scheduling, resource manager, router, and meta-storage services +use participant-style primary election. + +### etcd utility view + +`pkg/utils/etcdutil` centralizes etcd client creation, cluster ID checks, +member operations, leader transfer, health checks, proto reads, and loop +watchers. + +## Process Lifecycle And Startup Sequencing + +Important anchors: + +1. `server.startClient` +2. `server.initMember` +3. `server.leaderLoop` +4. `server.campaignLeader` +5. `member.NewMember` +6. `Member.CheckLeader` +7. `Member.Campaign` +8. `Member.Resign` +9. `member.Participant` +10. `etcdutil.NewLoopWatcher` + +Classic PD sequence: + +1. Start embedded etcd. +2. Create separate etcd clients for server metadata and election operations. +3. Initialize PD member info with the embedded etcd server ID. +4. Check the persistent PD leader key. +5. Watch an existing leader or campaign when no valid leader exists. +6. Keep the leadership lease before enabling leader-owned services. +7. Resign and reset serving state when leadership ends. + +Maintenance rules: + +- Do not conflate embedded etcd leader, PD leader, and microservice primary. +- Serving checks must use lease-backed leadership, not just cached leader + identity. +- Watch loops must have context cancellation and load/error behavior that + callers can reason about. + +## Data Model And Metadata Contracts + +Hot contracts: + +- PD member ID, name, member value, client URLs, peer URLs, binary version, git + hash, and deploy path +- election path for PD leader +- primary paths for microservices +- expected primary flags used by transfer APIs +- leader key revision and delete-by-revision behavior +- etcd cluster ID and member list +- leadership lease TTL and campaign frequency guard + +Maintenance rule: + +- Election keys are coordination contracts. Changing their path, value, lease, + or delete behavior can split ownership across processes. + +## Observability And Operational Signals + +Open these first: + +- `pkg/member/metrics.go` +- `pkg/utils/etcdutil/metrics.go` +- `server/server.go` +- `pkg/mcs/*/server/server.go` + +Signals to preserve: + +- member and service role gauges +- leader/primary campaign logs +- leader watch and leader-change logs +- etcd leader transfer logs +- loop watcher load and watch errors +- etcd health signals + +## Change Management Guidance + +- PD leader election changes must be reviewed with TSO, cluster startup, and + server readiness. +- Microservice primary changes must be reviewed with service discovery and + fallback behavior. +- etcd client changes must preserve separate election and metadata client + purposes. +- Loop watcher changes need tests for initial load, live put/delete, compaction, + cancellation, and error retry behavior. + +## Must-Read File Order + +1. `pkg/member/member.go` +2. `pkg/member/participant.go` +3. `pkg/member/election.go` +4. `pkg/election` +5. `pkg/utils/etcdutil/etcdutil.go` +6. `pkg/utils/etcdutil/health_checker.go` +7. `server/server.go` +8. `pkg/mcs/tso/server/server.go` +9. `pkg/mcs/scheduling/server/server.go` +10. `pkg/mcs/resourcemanager/server/server.go` +11. `pkg/mcs/utils/expected_primary.go` +12. `pkg/utils/keypath` + +## Glossary + +- PD member: + one PD process identity backed by embedded etcd member metadata. +- Embedded etcd leader: + raft leader of the embedded etcd cluster. +- PD leader: + elected PD owner for leader-only cluster mutations and services. +- Service primary: + elected owner for a microservice such as TSO or resource manager. +- Lease: + time-bounded ownership proof used to decide whether serving is safe. +- Expected primary: + coordination marker used by transfer APIs to steer primary ownership. +- Loop watcher: + etcd watcher helper that performs an initial load and then consumes updates. + +## Review Checklist + +- Which ownership state is required: etcd leader, PD leader, or service + primary? +- Is the leadership lease kept before serving starts? +- Does the code resign and clear serving state on every failure path? +- Are campaign retries bounded or backed off where needed? +- Does leader transfer respect expected-primary semantics? +- Can loop watchers recover from transient etcd errors? +- Are member values and key paths compatible across rolling upgrade? diff --git a/docs/maintenance-guides/placement-policy.md b/docs/maintenance-guides/placement-policy.md new file mode 100644 index 00000000000..2a37ff6abaa --- /dev/null +++ b/docs/maintenance-guides/placement-policy.md @@ -0,0 +1,175 @@ +# Placement Policy + +This guide covers placement rules, region labels, and affinity policy in +`pkg/schedule/placement/`, `pkg/schedule/labeler/`, and +`pkg/schedule/affinity/`. + +## Purpose And Scope + +Placement policy owns: + +- placement rule lifecycle and validation +- rule groups, rule lists, and fit calculation +- label constraints, location labels, isolation scoring, witness rules, and + orphan peer detection +- region label rules and key-range matching +- affinity groups, key-range to group mapping, and affinity availability +- policy inputs used by checkers, schedulers, scatter, split, and API handlers + +It does not own operator execution. Operators consume placement decisions and +belong to scheduling. + +## Core Concepts + +- Placement rules declare desired replica roles, counts, labels, key ranges, and + constraints. +- `RuleManager` owns rule loading, validation, adjustment, default-rule + migration, and region fitting. +- `RegionLabeler` maps key ranges to labels that can affect scheduling and API + behavior. +- `affinity.Manager` overlays affinity group policy and has stricter lock-order + requirements than normal rule fitting. +- Policy metadata is persisted and user-visible, so validation and deterministic + fit behavior are compatibility concerns. + +## Architectural Views + +### Placement rule view + +`RuleManager` loads rules and groups from storage, creates default rules when +needed, validates rule content, builds rule lists, and fits regions to rules. +`RegionFit` describes which peers satisfy which rules and which peers are +orphans. + +### Region label view + +`RegionLabeler` loads label rules, indexes key ranges through `rangelist`, and +periodically removes expired labels. Label rules are persisted as region rules +and are visible through APIs. + +### Affinity view + +`affinity.Manager` owns affinity groups, region-to-group cache, key ranges, +group availability, and synchronization with region labels. It uses separate +locks for metadata and in-memory state; lock ordering is a correctness +constraint. + +## Process Lifecycle And Startup Sequencing + +Important anchors: + +1. `placement.NewRuleManager` +2. `RuleManager.Initialize` +3. `RuleManager.AdjustRule` +4. `placement.fitRegion` +5. `labeler.NewRegionLabeler` +6. `RegionLabeler.SetLabelRule` +7. `RegionLabeler.BuildRangeListLocked` +8. `affinity.NewManager` +9. `affinity.Manager.initialize` +10. `affinity.Manager.startAvailabilityCheckLoop` + +Startup order in cluster leadership: + +1. Initialize placement rules if enabled. +2. Create region labeler. +3. Create affinity manager with the region labeler. +4. Start scheduling and checker logic that consumes these policies. + +Maintenance rules: + +- Placement rules must be initialized before placement-aware region statistics + and checkers depend on them. +- Region label range lists must be rebuilt after rule mutations. +- Affinity `metaMutex` must not be taken while holding `RWMutex`. + +## Data Model And Metadata Contracts + +Hot contracts: + +- placement rule group ID, rule ID, role, count, key range, label constraints, + location labels, isolation level, and witness flag +- default placement rules generated from legacy replication config +- hex-encoded rule keys and key type validation +- `RegionFit`, `RuleFit`, orphan peers, isolation score, and witness score +- region label ID, index, labels, TTL, start time, rule type, and key ranges +- affinity group ID, leader store, voter stores, availability, affinity version, + and region label mapping +- store conditions that degrade or expire affinity groups + +Maintenance rule: + +- Policy metadata is persisted and user visible. Validate input strictly, keep + deterministic fit behavior, and preserve upgrade behavior from legacy config. + +## Observability And Operational Signals + +Open these first: + +- `pkg/schedule/placement` +- `pkg/schedule/labeler/metrics.go` +- `pkg/schedule/affinity/metrics.go` +- `server/api/rule.go` +- `server/api/region_label.go` +- `server/apiv2/handlers/affinity.go` + +Signals to preserve: + +- rule load and validation logs +- region labeler creation and GC metrics +- affinity group count and affinity region count metrics +- affinity availability logs +- placement status metrics from statistics collection + +## Change Management Guidance + +- Placement rule changes need tests for fit determinism, role matching, + isolation score, witness behavior, and orphan handling. +- Region label changes need tests for TTL, key-range parsing, range-list split + keys, and patch behavior. +- Affinity changes need tests for lock ordering, key-range sync, availability + transitions, degraded/expired state, and store-condition interpretation. +- API changes need compatibility review because rule and label structs are + exported through HTTP. + +## Must-Read File Order + +1. `pkg/schedule/placement/rule_manager.go` +2. `pkg/schedule/placement/rule.go` +3. `pkg/schedule/placement/rule_list.go` +4. `pkg/schedule/placement/fit.go` +5. `pkg/schedule/placement/label_constraint.go` +6. `pkg/schedule/labeler/labeler.go` +7. `pkg/schedule/labeler/rules.go` +8. `pkg/schedule/rangelist/range_list.go` +9. `pkg/schedule/affinity/manager.go` +10. `pkg/schedule/affinity/policy.go` +11. `pkg/schedule/affinity/group.go` +12. `pkg/schedule/checker/rule_checker.go` + +## Glossary + +- Placement rule: + declarative replica placement requirement. +- Rule group: + namespace and ordering container for placement rules. +- `RegionFit`: + fit result describing which peers satisfy rules and which are orphan peers. +- Orphan peer: + peer that does not satisfy any placement rule. +- Region label: + key-range label persisted as label-rule metadata. +- Affinity group: + policy object that binds regions and stores for availability intent. +- Range list: + indexed key-range structure used by region label matching. + +## Review Checklist + +- Are rule and label inputs validated before persistence? +- Is fit behavior deterministic across peer ordering and store ordering? +- Does the change preserve default-rule migration from legacy config? +- Are key ranges encoded and compared in the correct key mode? +- Are expired region labels removed without breaking range-list state? +- Does affinity preserve lock ordering and metadata/cache consistency? +- Do checkers and schedulers observe the intended policy state? diff --git a/docs/maintenance-guides/repo-overview.md b/docs/maintenance-guides/repo-overview.md new file mode 100644 index 00000000000..816be276138 --- /dev/null +++ b/docs/maintenance-guides/repo-overview.md @@ -0,0 +1,491 @@ +# Repository Overview + +This file is the top-level maintenance map for the PD repository. It is the +first file maintainers and reviewers should read before working on a +cross-component change. + +## Purpose And Scope + +- Describe PD at the maintenance boundary level, not at the user feature level. +- Explain how major subsystems fit together and where ownership boundaries are. +- Highlight repository-wide invariants, sequencing rules, and failure modes. +- Serve as an index into the deeper subsystem guides in this directory. +- Provide stable retrieval anchors for agents before they search subsystem code. + +## Agent Retrieval Notes + +- Use this file after [README](./README.md) and before opening a subsystem guide. +- Match the change to the closest ownership boundary in + [Core Components And Boundaries](#core-components-and-boundaries). +- Use [Core Concepts](#core-concepts) and [Glossary](#glossary) to disambiguate + overloaded PD terms such as leader, primary, region, keyspace, fallback, and + resource group. +- Use [Change-Impact Matrix](#change-impact-matrix) to decide which additional + subsystem guides should be read before reviewing or editing code. + +## System Context And External Dependencies + +PD is the Placement Driver for TiKV clusters. It normally runs with: + +- TiKV: + reports store and region heartbeats, receives scheduling operators, and asks + PD for split IDs and timestamps. +- TiDB: + requests TSO, uses PD client service discovery, reads cluster metadata, and + manages GC safe points and resource groups. +- etcd: + embedded by PD for membership, leader election, cluster metadata, dynamic + config, and service discovery. +- TiFlash and other TiDB components: + report store metadata and depend on placement and scheduling decisions. +- PD microservices: + optional split services for TSO, scheduling, resource manager, router, and + meta storage. + +Operationally significant dependencies include: + +- filesystem state under the configured PD data directory +- local LevelDB region metadata storage when enabled +- TLS material and auth configuration +- Prometheus metric scraping and HTTP/gRPC health probes +- failpoint instrumentation in tests + +## Core Concepts + +- PD is the cluster control plane. It owns metadata, scheduling decisions, TSO, + service discovery, and resource-control metadata, but TiKV and TiDB execute + the data path. +- Ownership state is split across several concepts: embedded etcd leader, PD + leader, TSO primary, scheduling primary, resource-manager primary, and plain + follower are not interchangeable. +- Most correctness-sensitive state is either persisted in etcd, cached in + memory, watched from etcd, or exposed over gRPC/HTTP. A change can be local in + code but still change a metadata contract. +- Request hot paths include TSO allocation, region heartbeat, store heartbeat, + scheduler operator dispatch, API forwarding, and resource-manager token + acquisition. +- Fallback paths matter in microservice mode. Classic PD can provide TSO, + scheduling, and resource-manager behavior when independent services are absent + or disabled by configuration. +- Region and store metadata are the shared input to statistics, placement, + scheduling, APIs, and follower sync. Preserve clone/snapshot semantics when + moving data across subsystem boundaries. +- Startup order, leader lease state, and shutdown cleanup are correctness + concerns because serving state and background jobs are gated by leadership. + +## Core Components And Boundaries + +### Process bootstrap and service wiring + +`server/` owns embedded etcd startup, PD member initialization, gRPC/HTTP +registration, leader election, service loops, callbacks, and shutdown. + +### Configuration and runtime options + +`server/config/` owns static config, persisted options, dynamic config access, +and config API compatibility. Runtime side effects belong to the consuming +subsystem but must stay aligned with persisted config. + +### Member and election + +`pkg/member/`, `pkg/election/`, and `pkg/utils/etcdutil/` own PD member +identity, PD leader election, service primary election, etcd client helpers, +health checks, and watch loops. + +### API and client compatibility + +`server/grpc_service.go`, `server/api/`, `server/apiv2/`, and `client/` own the +external request boundary. Changes here can affect TiKV, TiDB, operators, and +PD clients even when the implementation change is internal. + +### Cluster metadata and heartbeats + +`server/cluster/` owns `RaftCluster`, store and region heartbeat handling, +background jobs, cluster bootstrap state, and coordination with scheduling, +replication mode, GC, and statistics. `pkg/core/` owns the in-memory store and +region data model used by schedulers and APIs. + +### Scheduling + +`pkg/schedule/` owns the coordinator, checkers, schedulers, placement rules, +operators, filters, scatter/split helpers, scheduler plugins, and the heartbeat +stream used to deliver operators to TiKV. + +### Statistics and placement policy + +`pkg/statistics/` owns region/store status, store load, hot region, hot peer, +and bucket statistics used by scheduling decisions. `pkg/schedule/placement/`, +`pkg/schedule/labeler/`, and `pkg/schedule/affinity/` own persisted policy, +region labels, rule fitting, and affinity availability. + +### Metadata storage + +`pkg/storage/` owns the storage interfaces and backends. Most metadata is +etcd-backed through endpoint-specific interfaces. Region metadata can be routed +to a specialized local LevelDB storage by `coreStorage`. + +### Timestamp service + +`pkg/tso/` owns the timestamp oracle and allocator used by the embedded PD TSO +path. `pkg/mcs/tso/` owns the independent TSO microservice path. + +### Keyspace and microservices + +`pkg/keyspace/` owns keyspace metadata, keyspace states, keyspace groups, and +meta-service group assignment. `pkg/mcs/` owns split service servers, +registration, discovery, primary election, forwarding, and fallback behavior. + +### Resource manager + +`pkg/mcs/resourcemanager/` owns resource groups, RU token buckets, service +limits, controller config, resource group metadata watches, metering, and the +independent resource manager service. + +## Architectural Views + +### Layered view + +1. Process bootstrap, config, and embedded etcd +2. gRPC/HTTP request edge and client compatibility surface +3. Cluster metadata cache and storage contracts +4. Statistics and placement policy inputs +5. Scheduling decision engine and operator dispatch +6. TSO, keyspace, resource group, and service-discovery control planes +7. Metrics, health, diagnostics, and maintenance APIs + +### Runtime ownership view + +- Main startup and stop orchestration: `server.Server` +- PD leader ownership: `pkg/member` through `server.leaderLoop` +- Config ownership: `server/config.Config` and `server/config.PersistOptions` +- Cluster state ownership: `server/cluster.RaftCluster` +- Scheduler ownership: `pkg/schedule.Coordinator` +- Store/region cache ownership: `pkg/core.BasicCluster` +- Statistics ownership: `pkg/statistics` +- Placement policy ownership: `pkg/schedule/placement`, + `pkg/schedule/labeler`, and `pkg/schedule/affinity` +- Timestamp ownership: `pkg/tso.Allocator` +- Keyspace ownership: `pkg/keyspace.Manager` and `pkg/keyspace.GroupManager` +- Resource group ownership: `pkg/mcs/resourcemanager/server.Manager` +- Microservice ownership: `pkg/mcs/*/server` + +### Deployment view + +- A classic PD process embeds etcd and can provide PD, TSO, scheduling, and + resource-manager behavior in one process. +- In microservice mode, TSO, scheduling, router, resource manager, and meta + storage can run as independent services discovered through etcd. +- Fallback paths let PD provide some services when the corresponding + microservice is absent or disabled by configuration. + +## Process Lifecycle And Startup Sequencing + +The high-level classic PD startup order is: + +1. Build `server.Server` from configuration and service builders. +2. Start embedded etcd and wait for readiness. +3. Create etcd clients and initialize the PD member. +4. Initialize cluster ID, member metadata, ID allocator, encryption manager, + storage, TSO allocator, `BasicCluster`, `RaftCluster`, keyspace managers, + metering, GC state manager, heartbeat streams, and hot-region storage. +5. Mark the server running and start service loops. +6. Campaign for PD leader. +7. As leader, keep the leader lease, reload persisted config, initialize TSO, + run leader callbacks, and start or resume cluster background jobs. + +Shutdown runs the inverse ownership path: stop server loops, close keyspace and +TSO state, close clients, member, heartbeat streams, storage, rate limiters, and +registered callbacks. + +Maintenance rule: + +- Startup and shutdown ordering are correctness concerns. Do not treat them as + operational details only. + +## Data Model And Metadata Contracts + +Repository-wide metadata and state contracts include: + +- cluster identity: + cluster ID, bootstrap timestamp, PD member ID, advertised URLs +- store metadata: + `metapb.Store`, labels, state, address fields, deployment info, slow-store + state, and store limit settings +- region metadata: + `metapb.Region`, epoch, peers, leader, range, buckets, approximate size, flow + statistics, and source freshness +- scheduling metadata: + placement rules, operators, scheduler config, checker state, pending regions, + and region label rules +- timestamp metadata: + TSO physical/logical window, save interval, leader/primary lease, and + keyspace group ownership +- keyspace metadata: + keyspace name, ID, state, config, keyspace group assignment, and region-bound + labels +- service discovery metadata: + registered service entries, primary keys, expected primary flags, and + microservice fallback state + +Cross-component rule: + +- Any change to a persisted, network-exposed, or dynamically watched metadata + contract should trigger review of the matching guide and usually a doc update + in this directory. + +## Hot Request Paths + +### TSO + +1. Clients call `PD.Tso` or the TSO microservice. +2. The request is served by `pkg/tso.Allocator` if the local member is serving. +3. The allocator updates and persists timestamp windows through + `timestampOracle`. +4. In microservice or forwarding mode, requests may be redirected or proxied to + the current primary. + +Review rule: + +- TSO changes must preserve monotonicity, lease semantics, logical overflow + behavior, and leader/primary failure handling. + +### Region heartbeat + +1. TiKV streams region heartbeats to PD or scheduling service. +2. The service boundary validates leadership and forwards if needed. +3. `server/cluster` converts the heartbeat to `core.RegionInfo`. +4. `RaftCluster` validates epoch/range freshness, updates cache, persists + metadata when required, collects statistics, and syncs followers. +5. Schedulers and checkers create operators that are sent back through + heartbeat streams. + +Review rule: + +- Region heartbeat changes must preserve stale-region rejection, overlap + handling, async persistence semantics, and operator response behavior. + +### Store heartbeat + +1. TiKV reports store state and statistics. +2. PD updates `core.StoreInfo`, store limits, slow-store status, node state, and + scheduling inputs. +3. Background jobs and schedulers consume the updated store view. + +Review rule: + +- Store heartbeat changes can affect scheduling fairness, slow-node handling, + health checks, and operator target selection. + +## Background Jobs And Determinism + +Important background jobs include: + +- PD leader election and leader watch +- etcd leader watch and server metrics collection +- encryption key manager loop +- raft cluster service checks +- metrics collection, node state checks, store stats updates +- region sync to followers +- replication mode management +- min resolved TS persistence +- store config sync +- GC state manager and GC tuner +- hot region and storage size collection + +Important rule: + +- Background work should not change request ordering, leader ownership, or + metadata freshness guarantees expected by foreground logic. + +## Observability And Operational Signals + +Important observability surfaces include: + +- Prometheus metrics in `server/metrics.go`, `server/cluster/metrics.go`, + `pkg/schedule/metrics.go`, `pkg/core/metrics.go`, `pkg/storage/kv/metrics.go`, + `pkg/tso/metrics.go`, and `pkg/keyspace/metrics.go` +- HTTP API status, health, readiness, config, region, store, scheduler, and + maintenance endpoints +- gRPC response headers and error fields +- structured logs with member, region, store, scheduler, and service fields +- pprof and diagnostic APIs + +Maintenance rule: + +- If behavior changes the operator-facing explanation of a failure or delay, + update metrics, logs, and API responses together. + +## Failure Modes And Triage Playbook + +Common repository-wide failure modes: + +- PD leader and etcd leader diverge longer than expected +- TSO allocator resets or serves from the wrong ownership state +- region metadata becomes stale, overlaps are mishandled, or follower sync lags +- runtime config is persisted but not applied in memory +- statistics drift changes scheduler behavior unexpectedly +- placement rules, region labels, or affinity metadata reject valid placements +- microservice fallback flips unexpectedly +- resource manager token state or metadata watcher state diverges from storage +- API forwarding loops or returns incompatible errors +- scheduler operator creation ignores a store/region invariant +- storage backend switch causes region metadata warm-up drift + +Triage guidance: + +1. Identify the first boundary where the wrong behavior becomes visible: + client, API, cluster cache, scheduler, storage, TSO, or microservice. +2. Check leader/primary ownership before reasoning about state mutation. +3. Check whether the failing path is classic embedded PD or microservice mode. +4. Check metrics and logs before editing code. +5. Re-check whether the current maintenance guide is stale relative to the code. + +## Change Management Guidance + +- Treat this file and the matching subsystem guide as required context for any + non-trivial change. +- If a change modifies startup order, ownership boundaries, metadata contracts, + path-specific behavior, invariants, observability, or reading maps, update the + relevant guide in the same change. +- If a fix touches an API boundary, explicitly evaluate TiKV, TiDB, PD client, + and microservice compatibility. +- If the guide is missing facts needed for a safe review, improve the guide as + part of the maintenance work. + +## Change-Impact Matrix + +- Startup, shutdown, or leader election changes: + read [server](./server.md), [member-election](./member-election.md), + [storage](./storage.md), [tso](./tso.md), and + [keyspace-and-microservices](./keyspace-and-microservices.md). +- Region or store heartbeat changes: + read [api-and-client](./api-and-client.md), [cluster](./cluster.md), and + [statistics](./statistics.md), then [scheduling](./scheduling.md) if the + change affects operator creation or heartbeat responses. +- Scheduler, checker, operator, or placement changes: + read [scheduling](./scheduling.md), [placement-policy](./placement-policy.md), + [statistics](./statistics.md), [cluster](./cluster.md), and + [storage](./storage.md). +- Persisted config or metadata changes: + read [config](./config.md), [storage](./storage.md), [server](./server.md), + and the owning subsystem guide. +- TSO changes: + read [tso](./tso.md), [server](./server.md), [api-and-client](./api-and-client.md), + and [keyspace-and-microservices](./keyspace-and-microservices.md). +- Keyspace or microservice changes: + read [keyspace-and-microservices](./keyspace-and-microservices.md), + [storage](./storage.md), [tso](./tso.md), and [api-and-client](./api-and-client.md). +- Resource group or RU changes: + read [resource-manager](./resource-manager.md), [config](./config.md), + [keyspace-and-microservices](./keyspace-and-microservices.md), and + [api-and-client](./api-and-client.md). + +## Must-Read File Order + +Use this faster file order when the goal is review or change-impact analysis: + +1. `server/server.go` +2. `server/grpc_service.go` +3. `server/config/config.go` +4. `server/config/persist_options.go` +5. `pkg/member/member.go` +6. `server/cluster/cluster.go` +7. `server/cluster/cluster_worker.go` +8. `pkg/core/region.go` +9. `pkg/core/store.go` +10. `pkg/statistics/hot_cache.go` +11. `pkg/schedule/placement/rule_manager.go` +12. `pkg/schedule/coordinator.go` +13. `pkg/schedule/operator/operator_controller.go` +14. `pkg/storage/storage.go` +15. `pkg/tso/allocator.go` +16. `pkg/keyspace/keyspace.go` +17. `pkg/keyspace/tso_keyspace_group.go` +18. `pkg/mcs/resourcemanager/server/manager.go` +19. `pkg/mcs/discovery/register.go` +20. `client/client.go` + +## Reading Map And Companion Docs + +Suggested reading order for new maintainers: + +1. `README.md` +2. `docs/development.md` +3. `server/server.go` +4. `server/grpc_service.go` +5. `server/config/config.go` +6. `pkg/member/member.go` +7. `server/cluster/cluster.go` +8. `pkg/core/basic_cluster.go` +9. `pkg/core/region.go` +10. `pkg/core/store.go` +11. `pkg/statistics/region_collection.go` +12. `pkg/statistics/store_collection.go` +13. `pkg/schedule/placement/rule_manager.go` +14. `pkg/schedule/coordinator.go` +15. `pkg/schedule/checker/checker_controller.go` +16. `pkg/schedule/schedulers/scheduler_controller.go` +17. `pkg/schedule/operator/operator_controller.go` +18. `pkg/storage/storage.go` +19. `pkg/tso/allocator.go` +20. `pkg/keyspace/keyspace.go` +21. the matching deeper guide in `docs/maintenance-guides/` + +Useful companion docs in this repo: + +- `README.md` +- `CONTRIBUTING.md` +- `docs/development.md` +- `docs/development-workflow.md` + +## Glossary + +- PD leader: + the PD member that owns cluster mutations and serves leader-only behavior. +- etcd leader: + the embedded etcd raft leader. PD tries to keep it aligned with the PD leader. +- TSO: + timestamp oracle used by TiDB transactions. +- Keyspace: + logical metadata namespace used by serverless and multi-tenant paths. +- Keyspace group: + a grouping of keyspaces assigned to TSO service ownership. +- Region: + TiKV key-range shard and scheduling unit. +- Store: + TiKV node or storage service identity reported to PD. +- Operator: + scheduler-produced action plan delivered to TiKV through heartbeat response. + Operators have type, steps, status, influence, and timeout semantics. +- Placement rule: + declarative replica placement contract. +- PD member: + one PD process with embedded etcd identity, advertise URLs, and member + metadata. +- Follower: + a PD member that is not the current PD leader. It may serve some read or + forwarded APIs but must not run leader-owned mutations. +- Service primary: + the elected owner for a split microservice such as TSO, scheduling, or + resource manager. +- TSO primary: + the service owner allowed to initialize allocators and serve timestamps for + its keyspace groups. +- Epoch: + version metadata on regions used to reject stale region state and preserve + split/merge correctness. +- Resource group: + TiDB-facing resource-control object whose RU settings are managed by PD. +- RU: + request unit consumed and refilled by resource-manager token buckets. +- Metadata contract: + persisted, watched, or API-visible state whose key path, value shape, timing, + or compatibility matters across components. +- Hot path: + request or heartbeat path where extra storage IO, metric child creation, broad + locking, or logging can change cluster latency. +- Fallback: + mode where classic PD serves behavior that could otherwise be served by an + independent microservice. +- MCS: + microservice mode components under `pkg/mcs/`. diff --git a/docs/maintenance-guides/resource-manager.md b/docs/maintenance-guides/resource-manager.md new file mode 100644 index 00000000000..5162c1d3e58 --- /dev/null +++ b/docs/maintenance-guides/resource-manager.md @@ -0,0 +1,189 @@ +# Resource Manager + +This guide covers resource manager code under `pkg/mcs/resourcemanager/` and +its integration with the classic PD process. + +## Purpose And Scope + +Resource manager owns: + +- resource group metadata and state +- RU token bucket calculation +- service limits per keyspace +- controller config and RU version policy +- resource group metadata watching +- resource manager gRPC and REST service registration +- independent resource manager primary election +- embedded-PD fallback and proxy behavior +- RU consumption metrics and metering collection + +It does not own TiDB-side admission control, but TiDB relies on these contracts +for resource group behavior. + +## Core Concepts + +- Resource manager owns PD-side resource-group metadata, RU token buckets, + service limits, and metering collection. +- Token bucket state is runtime behavior; resource group metadata and states are + persisted contracts. +- Metadata-write and state-write roles are separate and depend on embedded PD + versus independent resource-manager mode. +- Metadata watcher initialization must produce the same cache meaning as full + storage load. +- The default resource group and reserved keyspace behavior must stay + idempotent across startup, watch replay, and API writes. + +## Architectural Views + +### Manager view + +`Manager` owns controller config, keyspace resource group managers, resource +group metadata, service limiters, consumption dispatch, metrics, and RU +collector registration. It initializes through server lifecycle callbacks in +embedded mode or service startup in independent mode. + +### Resource group view + +`ResourceGroup` wraps resource group settings, priority, runaway/background +settings, RU consumption, and a `GroupTokenBucket`. Settings can be patched or +applied depending on whether token deltas should be preserved. + +### Token bucket view + +`GroupTokenBucket` and token slots split RU across clients in the same resource +group, apply burst modes, preserve reserved burst/service tokens, and produce +minimum trickle times for clients. + +### Metadata watcher view + +Independent resource manager can initialize through a metadata watcher instead +of full storage load. The watcher applies controller config, resource group +settings, resource group states, and service limits from etcd events. + +## Process Lifecycle And Startup Sequencing + +Important anchors: + +1. `pkg/mcs/resourcemanager/server.NewManager` +2. `Manager.Init` +3. `Manager.initMetadata` +4. `Manager.initializeMetadataWatcher` +5. `Manager.backgroundMetricsFlush` +6. `Manager.persistLoop` +7. `Service.AcquireTokenBuckets` +8. `pkg/mcs/resourcemanager/server.Server.Run` +9. `Server.primaryElectionLoop` +10. `server/resource_group_proxy_service.go` + +Embedded PD sequence: + +1. Server start callback creates storage and registers the RU collector. +2. Service-ready callback initializes metadata after PD becomes serving. +3. Background metrics flushing starts. +4. State persistence starts when the write role allows state writes. + +Independent service sequence: + +1. Resource manager service starts and registers with service discovery. +2. It campaigns for resource manager primary. +3. Primary callbacks initialize manager state. +4. Metadata watcher or full load populates local caches. + +Maintenance rules: + +- Metadata writes and state writes are controlled separately by write role. +- gRPC metadata writes may be rejected in modes where PD owns metadata. +- The default resource group and reserved keyspace behavior must remain + idempotent. + +## Data Model And Metadata Contracts + +Hot contracts: + +- `resource_manager.ResourceGroup` +- group mode, priority, RU settings, token limit settings, runaway settings, + background settings, and RU consumption +- keyspace ID extraction and null/default keyspace behavior +- controller config and RU version policy +- resource group settings and states in storage +- service limits per keyspace +- token bucket fill rate, burst limit, override fill rate, override burst + limit, token slots, reserved tokens, and last update time +- metadata watcher key parsing under resource group prefixes + +Maintenance rule: + +- Resource group APIs are control-plane contracts with TiDB. Changes must keep + metadata persistence, runtime token behavior, and client compatibility aligned. + +## Observability And Operational Signals + +Open these first: + +- `pkg/mcs/resourcemanager/server/metrics.go` +- `pkg/mcs/resourcemanager/server/metering.go` +- `pkg/mcs/resourcemanager/server/grpc_service.go` +- `pkg/mcs/resourcemanager/server/manager.go` +- `pkg/mcs/resourcemanager/redirector` + +Signals to preserve: + +- token bucket request metrics +- consumption and RU metrics +- background metrics flushing +- resource manager primary election logs +- metadata watcher logs +- service limit and controller config update logs +- metering collector output + +## Change Management Guidance + +- Token bucket changes need tests for burst modes, slot balancing, refill, + reserved tokens, and trickle time. +- Metadata changes need watcher, full-load, write-role, and persistence tests. +- gRPC changes need TiDB compatibility review. +- Service limit changes need keyspace behavior and override-token review. +- Primary election changes need failure-path tests that avoid half-initialized + primaries. + +## Must-Read File Order + +1. `pkg/mcs/resourcemanager/server/manager.go` +2. `pkg/mcs/resourcemanager/server/resource_group.go` +3. `pkg/mcs/resourcemanager/server/token_buckets.go` +4. `pkg/mcs/resourcemanager/server/service_limit.go` +5. `pkg/mcs/resourcemanager/server/metadata_watcher.go` +6. `pkg/mcs/resourcemanager/server/grpc_service.go` +7. `pkg/mcs/resourcemanager/server/server.go` +8. `pkg/mcs/resourcemanager/server/config.go` +9. `pkg/mcs/resourcemanager/redirector/redirector.go` +10. `pkg/mcs/resourcemanager/metadataapi/config_service.go` +11. `server/resource_group_metadata_manager.go` +12. `server/resource_group_proxy_service.go` + +## Glossary + +- Resource group: + TiDB-facing control-plane object defining RU settings and behavior. +- RU: + request unit consumed by SQL workload and refilled by token buckets. +- Token bucket: + runtime structure that meters RU allocation to clients. +- Service limit: + per-keyspace limit state used to bound resource-manager service behavior. +- Write role: + policy deciding whether this manager can write metadata, state, or both. +- Metadata watcher: + etcd watcher that keeps resource group caches in sync from events. +- Metering: + collection and writing of resource usage records. + +## Review Checklist + +- Does the change respect metadata-write and state-write roles? +- Are embedded PD and independent resource manager paths both correct? +- Does metadata watcher behavior match full-load behavior? +- Is the default resource group initialized idempotently? +- Are token bucket calculations stable across setting changes and time jumps? +- Are keyspace-specific and legacy resource group paths both handled? +- Are metrics and metering updated with behavior changes? diff --git a/docs/maintenance-guides/scheduling.md b/docs/maintenance-guides/scheduling.md new file mode 100644 index 00000000000..3c2e36d803b --- /dev/null +++ b/docs/maintenance-guides/scheduling.md @@ -0,0 +1,169 @@ +# Scheduling + +This guide covers scheduling in `pkg/schedule/`. + +## Purpose And Scope + +Scheduling owns: + +- coordinator lifecycle +- schedulers and checker controllers +- placement rule evaluation at the scheduler boundary +- operator creation, tracking, dispatch, and TTL recording +- store and region filters +- scatter and split helpers +- scheduler plugins +- scheduling diagnostics + +It consumes cluster metadata from `server/cluster` and `pkg/core`. It should not +own persistence details outside scheduler config and placement metadata. +Detailed placement-rule, region-label, and affinity-policy maintenance lives in +[placement-policy](./placement-policy.md). + +## Core Concepts + +- `Coordinator` owns the scheduler runtime: checkers, schedulers, operators, + heartbeat streams, scatter/split helpers, and diagnostics. +- Checkers repair or preserve correctness; schedulers optimize balance or + policy goals. +- Filters are part of correctness because they encode store health, labels, + state, and special eviction constraints. +- Operators are stateful plans delivered through heartbeat responses and tracked + until finish, timeout, cancel, or replacement. +- Classic PD scheduling and independent scheduling-service mode share concepts + but differ in lifecycle and fallback routing. + +## Architectural Views + +### Coordinator view + +`Coordinator` wires checkers, schedulers, operator controller, scatterer, +splitter, heartbeat streams, plugin interface, and diagnostics. + +### Decision view + +Checkers patrol regions for correctness and rule compliance. Schedulers produce +balancing or policy-driven operators. Filters decide candidate eligibility. +Operators encode concrete actions delivered to TiKV. + +### Dispatch view + +Operators are pushed through heartbeat streams or actively driven by +`drivePushOperator`. Operator status and TTL records explain whether an action +was created, running, finished, timed out, or canceled. + +## Process Lifecycle And Startup Sequencing + +Important anchors: + +1. `schedule.NewCoordinator` +2. `Coordinator.RunUntilStop` +3. `Coordinator.PatrolRegions` +4. `Coordinator.drivePushOperator` +5. `schedulers.Controller` +6. `checker.Controller` +7. `operator.Controller` + +In classic mode, `RaftCluster.checkSchedulingService` starts scheduling jobs +inside PD. In microservice mode, scheduling can be independent and PD may use +fallback behavior depending on config and service discovery. + +Maintenance rules: + +- Do not create operators before the cluster has enough prepared region state. +- Do not bypass filters unless the caller has a narrowly documented reason. +- Operator lifecycle changes must preserve status tracking and metrics. + +## Data Model And Metadata Contracts + +Hot contracts: + +- scheduler names and args +- placement rules and rule groups +- store filters and candidate comparison +- operator kind, step, status, and influence +- pending processed regions +- scatter and split group semantics +- scheduler config persisted through storage + +Maintenance rule: + +- Scheduler decisions are only as correct as the cluster snapshot and filters + they use. Changes that alter `StoreInfo` or `RegionInfo` interpretation + require scheduling review. + +## Observability And Operational Signals + +Open these first: + +- `pkg/schedule/metrics.go` +- `pkg/schedule/checker/metrics.go` +- `pkg/schedule/operator/metrics.go` +- `pkg/schedule/schedulers/metrics.go` +- `pkg/schedule/filter/metrics.go` +- `pkg/schedule/scatter/metrics.go` + +Signals to preserve: + +- scheduler run counters and durations +- checker patrol duration and pending region state +- operator create, finish, timeout, cancel, and replace metrics +- filter reason counters +- scatter/split metrics +- diagnostic records + +## Change Management Guidance + +- New schedulers should define config, tests, metrics, and operator behavior. +- New filters should include tests for positive and negative candidate cases. +- Placement rule changes should be reviewed with + [placement-policy](./placement-policy.md) and tested with fit and + rule-manager tests. +- Operator changes need tests for status transitions and step semantics. +- Microservice scheduling changes need review with `pkg/mcs/scheduling`. + +## Must-Read File Order + +1. `pkg/schedule/coordinator.go` +2. `pkg/schedule/prepare_checker.go` +3. `pkg/schedule/checker/checker_controller.go` +4. `pkg/schedule/schedulers/scheduler_controller.go` +5. `pkg/schedule/schedulers/scheduler.go` +6. `pkg/schedule/operator/operator_controller.go` +7. `pkg/schedule/operator/operator.go` +8. `pkg/schedule/operator/step.go` +9. `pkg/schedule/filter/filters.go` +10. `pkg/schedule/placement/rule_manager.go` +11. `pkg/schedule/placement/fit.go` +12. `pkg/schedule/hbstream/heartbeat_streams.go` + +## Glossary + +- Coordinator: + scheduler runtime coordinator under `pkg/schedule`. +- Checker: + component that scans cluster state for correctness work such as replicas, + learners, splits, and rules. +- Scheduler: + component that creates balancing or policy-driven operators. +- Filter: + candidate eligibility rule for store or region selection. +- Operator: + ordered action plan with steps, kind, status, influence, and timeout. +- Heartbeat stream: + channel used to send operator responses back to TiKV. +- Pending processed region: + recently handled region tracked to avoid conflicting scheduling work. + +## Review Checklist + +- Does the scheduler or checker run only when the required cluster state is + prepared? +- Are candidate stores filtered for health, state, labels, placement, and + special eviction flags? +- Does the operator remain idempotent across heartbeat retries? +- Are operator status transitions and TTL records still correct? +- Does the change alter balancing fairness, hot-region behavior, or slow-store + behavior? +- Are placement rule changes compatible with existing rule persistence? +- Are microservice and classic scheduling paths both considered? diff --git a/docs/maintenance-guides/server.md b/docs/maintenance-guides/server.md new file mode 100644 index 00000000000..d3f677f5207 --- /dev/null +++ b/docs/maintenance-guides/server.md @@ -0,0 +1,183 @@ +# Server + +This guide covers the classic PD server process in `server/`. + +## Purpose And Scope + +`server.Server` owns process-level orchestration: + +- embedded etcd startup and client creation +- PD member initialization and leader election +- gRPC and HTTP service registration +- storage, TSO, keyspace, cluster, metering, GC, and heartbeat stream wiring +- service loops and leader callbacks +- shutdown ordering and resource cleanup + +It does not own scheduler algorithms, region cache internals, or timestamp +oracle rules. Those are covered by the scheduling, cluster, storage, and TSO +guides. + +## Core Concepts + +- The server process is the orchestration root. It wires embedded etcd, member + identity, storage, TSO, keyspace managers, cluster state, APIs, and shutdown. +- `startServer` initializes process-owned dependencies but does not mean this + member is the PD leader. +- `leaderLoop` and `campaignLeader` gate leader-owned behavior through the PD + leader lease. +- Service labels and rate limiters are initialized during startup and are part + of the external API boundary. +- `Close` must handle partial startup and stop loops before closing shared + clients, storage, hot-region storage, metering, and callbacks. + +## Architectural Views + +### Bootstrap view + +`CreateServer` builds an uninitialized server from config, handler builders, +service labels, rate limiters, registry hooks, and embedded etcd configuration. +`Run` starts embedded etcd, initializes the runtime services, starts monitoring, +and launches server loops. + +### Leadership view + +`leaderLoop` and `campaignLeader` coordinate PD leadership. The PD leader lease +gates leader-only services, TSO initialization, cluster startup, and leader +callbacks. + +### Service view + +The embedded etcd gRPC server registers PD, keyspace, diagnostics, resource +manager proxy, and any installed microservice gRPC services. HTTP handlers are +installed through the normal PD API routers and microservice registry hooks. + +## Process Lifecycle And Startup Sequencing + +Important startup anchors: + +1. `CreateServer` +2. `Run` +3. `startEtcd` +4. `startClient` +5. `initMember` +6. `startServer` +7. `startServerLoop` +8. `leaderLoop` +9. `campaignLeader` + +`startServer` initializes: + +- cluster ID and member metadata +- ID allocator +- encryption manager +- etcd-backed default storage +- LevelDB-backed region storage +- core storage wrapper +- embedded TSO allocator +- `BasicCluster` and `RaftCluster` +- keyspace and keyspace group managers +- metering writer +- GC state manager +- heartbeat streams +- hot-region storage +- HTTP and gRPC rate limiters + +Shutdown is centered on `Close`, which stops server loops first, then closes +owned managers, clients, embedded etcd member state, heartbeat streams, storage, +rate limiters, callbacks, and client connections. + +Maintenance rules: + +- Do not add leader-only behavior before the leader lease is kept. +- Do not start background loops without a matching cancellation path and + `WaitGroup` ownership. +- Do not assume `startServer` means this member is serving as PD leader. + +## Data Model And Metadata Contracts + +Server-owned contracts include: + +- member name, ID, advertise client URL, advertise peer URL +- member deploy path, binary version, and git hash +- cluster ID initialization +- PD leader key and lease +- service labels and rate limiter state +- service primary watchers in keyspace group mode +- server start timestamp and running state + +Changing these contracts can affect API responses, service discovery, PD client +behavior, and upgrade compatibility. + +## Observability And Operational Signals + +Open these first: + +- `server/metrics.go` +- `server/server.go` +- `server/api/health.go` +- `server/api/status.go` +- `server/apiv2/handlers/ready.go` +- `pkg/basicserver` + +Signals to preserve: + +- PD server info and start timestamp gauges +- embedded etcd state gauges +- leader election logs and lease timing logs +- service forwarding and rate limiter metrics +- health and readiness API behavior + +## Change Management Guidance + +- Startup and shutdown changes should include a failure-path review. +- Leader election changes should be reviewed with TSO and cluster startup + behavior. +- Service registration changes should be reviewed with API and client + compatibility. +- Config changes should be reviewed with persisted config loading and runtime + side effects. + +## Must-Read File Order + +1. `server/server.go` +2. `server/handler.go` +3. `server/grpc_service.go` +4. `server/api/router.go` +5. `server/apiv2/router.go` +6. `server/config/config.go` +7. `server/config/persist_options.go` +8. `pkg/member/member.go` +9. `pkg/member/election.go` +10. `pkg/utils/etcdutil` + +## Glossary + +- `CreateServer`: + builds an uninitialized server and installs handler/service hooks. +- `Run`: + starts embedded etcd, initializes runtime dependencies, and starts loops. +- `startServer`: + creates storage, allocators, managers, clusters, heartbeat streams, and + rate-limit state before serving. +- `leaderLoop`: + long-running loop that observes leadership and campaigns when appropriate. +- PD leader: + member with lease-backed ownership for cluster mutations and leader-only + services. +- Embedded etcd: + etcd server hosted inside each classic PD process. +- Close callback: + cleanup hook run after server-owned loops and core resources are stopped. + +## Review Checklist + +- Does the change run only on the intended ownership state: + follower, PD leader, etcd leader, service primary, or all members? +- Are contexts, timers, watchers, and goroutines stopped on every exit path? +- Can `Close` run after a partial startup failure? +- Does the change preserve leader lease timing and TSO readiness? +- Are HTTP/gRPC handlers registered before metrics or label initialization uses + them? +- Does the change need a config reload or persisted option migration? +- Are failpoint-aware tests used for startup, leader transfer, and shutdown + paths? diff --git a/docs/maintenance-guides/statistics.md b/docs/maintenance-guides/statistics.md new file mode 100644 index 00000000000..d498301258c --- /dev/null +++ b/docs/maintenance-guides/statistics.md @@ -0,0 +1,168 @@ +# Statistics + +This guide covers cluster statistics in `pkg/statistics/`. + +## Purpose And Scope + +Statistics owns: + +- region status classification +- store status and store load observation +- hot region and hot peer caches +- bucket-level hot statistics +- CPU and store load dimensions +- metrics derived from store, region, and scheduling config state +- data used by hot-region, balance, slow-store, and checker logic + +Statistics consumes `pkg/core` snapshots. It should not mutate the authoritative +cluster cache. + +## Core Concepts + +- Statistics turns store and region snapshots into scheduler-visible signals. +- `RegionStatistics` classifies health and placement status by region. +- Store statistics aggregate capacity, leader/region counts, load, slow-store + state, and score inputs. +- `HotCache` maintains read/write hot peer state asynchronously to avoid blocking + heartbeat paths. +- Collector behavior can differ by engine or store type, especially TiKV versus + TiFlash. + +## Architectural Views + +### Region statistics view + +`RegionStatistics` records region health classes such as missing peers, extra +peers, down peers, pending peers, offline peers, learners, empty regions, +oversized regions, undersized regions, and witness leaders. It uses placement +rules when placement is enabled. + +### Store statistics view + +Store statistics aggregate node state, storage capacity, region and leader +counts, label distribution, slow-store state, scores, store limits, and rolling +load dimensions from store heartbeat input. + +### Hot cache view + +`HotCache` maintains read and write `HotPeerCache` instances and processes +updates asynchronously through queues. Schedulers query hot peer stats, +thresholds, and hotness state from these caches. + +### Collector view + +Collectors translate TiKV and TiFlash store/peer load signals into comparable +dimensions. TiFlash handling differs from TiKV because TiFlash has no leader and +may derive write peer load differently. + +## Process Lifecycle And Startup Sequencing + +Important anchors: + +1. `statistics.NewRegionStatistics` +2. `statistics.RegionStatistics.Observe` +3. `statistics.StoreStatisticsMap` +4. `statistics.NewHotCache` +5. `statistics.HotCache.CheckReadAsync` +6. `statistics.HotCache.CheckWriteAsync` +7. `server/cluster.processRegionHeartbeat` +8. `server/cluster.runMetricsCollectionJob` +9. `server/cluster.runUpdateStoreStats` + +Maintenance rules: + +- Statistics should be updated from cloned or immutable cluster snapshots. +- Hot cache queue behavior is part of scheduler correctness and latency. +- Region stat refresh must be triggered when special status changes even if + region metadata does not otherwise need cache persistence. + +## Data Model And Metadata Contracts + +Hot contracts: + +- `RegionStatisticType` bit flags +- region status maps and region ID index +- store load dimensions in `pkg/statistics/utils` +- read/write hot peer thresholds and hot degree +- rolling store stats and instant store stats +- TiKV versus TiFlash collector behavior +- bucket statistics under `pkg/statistics/buckets` +- metrics label names for cluster status and hot cache status + +Maintenance rule: + +- Scheduler decisions often depend on statistical meaning, not only raw values. + Changing a dimension, threshold, interval, or collector can change placement + and balance behavior. + +## Observability And Operational Signals + +Open these first: + +- `pkg/statistics/metrics.go` +- `pkg/statistics/store.go` +- `pkg/statistics/hot_cache.go` +- `pkg/statistics/hot_peer_cache.go` +- `pkg/statistics/buckets/metric.go` +- `pkg/schedule/schedulers/hot_region.go` + +Signals to preserve: + +- region status gauges +- store status gauges +- store load rates and instant rates +- hot cache status gauges +- hot peer metrics +- bucket hot statistics +- scheduler config metrics emitted with store statistics + +## Change Management Guidance + +- New statistics dimensions need scheduler review before being used for + decisions. +- Hot cache changes need tests for read and write paths, queue saturation, + expiration, and threshold calculation. +- Region status changes need API and checker review. +- Store status changes need filter and slow-store scheduler review. +- Metric changes should avoid high-cardinality labels. + +## Must-Read File Order + +1. `pkg/statistics/region_collection.go` +2. `pkg/statistics/store_collection.go` +3. `pkg/statistics/hot_cache.go` +4. `pkg/statistics/hot_peer_cache.go` +5. `pkg/statistics/hot_peer.go` +6. `pkg/statistics/store_load.go` +7. `pkg/statistics/collector.go` +8. `pkg/statistics/buckets/hot_bucket_cache.go` +9. `pkg/statistics/utils/kind.go` +10. `server/cluster/cluster.go` +11. `pkg/schedule/schedulers/hot_region.go` + +## Glossary + +- Region statistic: + classification of a region into health or placement buckets. +- Store statistic: + aggregate view of store capacity, status, score, and load. +- Hot cache: + read/write cache tracking hot peers and hotness transitions. +- Hot peer: + region peer whose read or write flow exceeds hot thresholds. +- Dimension: + named load component such as bytes, keys, query count, or CPU. +- Collector: + translator from heartbeat input into comparable statistics. +- Threshold: + calculated boundary used to decide whether a peer is hot or cold. + +## Review Checklist + +- Does the change alter scheduler-visible meaning of load or hotness? +- Are read and write paths handled symmetrically where required? +- Are TiKV and TiFlash semantics both considered? +- Does async hot cache work avoid blocking heartbeat hot paths? +- Are stale or missing region/store snapshots handled safely? +- Are metric labels bounded and consistent? +- Do tests cover threshold, expiration, and special status transitions? diff --git a/docs/maintenance-guides/storage.md b/docs/maintenance-guides/storage.md new file mode 100644 index 00000000000..23f2d374709 --- /dev/null +++ b/docs/maintenance-guides/storage.md @@ -0,0 +1,158 @@ +# Storage + +This guide covers PD metadata storage in `pkg/storage/`. + +## Purpose And Scope + +Storage owns: + +- the composite `Storage` interface +- etcd-backed metadata storage +- memory storage for tests +- local LevelDB-backed region metadata storage +- endpoint-specific metadata contracts +- region storage switching and one-time region loading behavior +- hot-region storage + +Storage does not decide scheduling, leadership, or API semantics. It provides +the persistence contracts used by those subsystems. + +## Core Concepts + +- Most PD metadata is stored through etcd-backed storage and endpoint-specific + interfaces. +- Region metadata can be routed through `coreStorage` to a local LevelDB-backed + region storage. +- Key paths under `pkg/utils/keypath` are compatibility contracts, not local + implementation details. +- `TryLoadRegionsOnce` prevents repeated full local-region loads after the + local backend has been selected. +- Hot-region storage is a separate local persistence surface for historical hot + region data. + +## Architectural Views + +### Backend view + +Most metadata uses an etcd-backed `Storage`. Tests can use memory storage. Region +metadata can be stored in a specialized local LevelDB-backed `RegionStorage` +through `coreStorage`. + +### Endpoint view + +`pkg/storage/endpoint` splits metadata contracts by owner: config, meta, +placement rules, replication status, GC state, min resolved TS, external TS, +keyspace, resource group, TSO, keyspace group, meta-service group, maintenance, +and affinity. + +### Region storage view + +`NewCoreStorage` wraps a default storage and region storage. Region load/save +calls route to etcd or local region storage depending on +`TrySwitchRegionStorage`. `TryLoadRegionsOnce` prevents repeated full loads from +local region storage. + +## Process Lifecycle And Startup Sequencing + +Important anchors: + +1. `storage.NewStorageWithEtcdBackend` +2. `storage.NewRegionStorageWithLevelDBBackend` +3. `storage.NewCoreStorage` +4. `storage.TrySwitchRegionStorage` +5. `storage.TryLoadRegionsOnce` +6. `Storage.Close` + +`server.startServer` creates the default etcd storage, the region LevelDB +storage, then wraps them in `coreStorage`. `RaftCluster` loads cluster metadata +from storage during leader startup and persists updates during heartbeat +processing and background jobs. + +Maintenance rules: + +- Region storage routing must be explicit. Do not assume every metadata path + uses etcd. +- Local region storage load state is intentionally tracked to avoid repeated + full loads. +- Storage close currently closes the region storage in `coreStorage`. + +## Data Model And Metadata Contracts + +Hot contracts: + +- key paths under `pkg/utils/keypath` +- endpoint interfaces in `pkg/storage/endpoint` +- region metadata load/save/delete behavior +- persisted scheduler and placement config +- TSO and keyspace group metadata +- GC state, external TS, and min resolved TS +- service middleware and maintenance metadata + +Maintenance rule: + +- If a metadata key changes, review all readers, writers, migrations, clients, + and downgrade behavior. + +## Observability And Operational Signals + +Open these first: + +- `pkg/storage/kv/metrics.go` +- `pkg/storage/hot_region_storage.go` +- `pkg/storage/region_storage.go` +- `server/cluster/metrics.go` + +Signals to preserve: + +- storage operation metrics +- logs for failed region save/delete +- hot-region storage lifecycle signals +- cluster warm-up duration and load errors + +## Change Management Guidance + +- New metadata should be added through endpoint interfaces, not ad hoc key + access from unrelated packages. +- Persistence changes need tests against etcd/memory behavior when practical. +- Region storage changes need load-once, switch, flush, and close-path tests. +- Any persisted format change needs compatibility review. + +## Must-Read File Order + +1. `pkg/storage/storage.go` +2. `pkg/storage/etcd_backend.go` +3. `pkg/storage/memory_backend.go` +4. `pkg/storage/leveldb_backend.go` +5. `pkg/storage/region_storage.go` +6. `pkg/storage/hot_region_storage.go` +7. `pkg/storage/kv/kv.go` +8. `pkg/storage/kv/etcd_kv.go` +9. `pkg/storage/endpoint/endpoint.go` +10. `pkg/utils/keypath` + +## Glossary + +- Default storage: + usually the etcd-backed `Storage` implementation used for most metadata. +- Endpoint storage: + owner-specific storage interface under `pkg/storage/endpoint`. +- `coreStorage`: + wrapper that routes region metadata to etcd or local region storage. +- Region storage: + storage interface for loading, saving, deleting, and flushing region metadata. +- Local region storage: + LevelDB-backed region metadata store under the PD data directory. +- Key path: + etcd key layout helper that acts as a persisted compatibility boundary. +- Hot-region storage: + local store for historical hot region records used by API and diagnostics. + +## Review Checklist + +- Is the metadata persisted under the right ownership prefix? +- Are readers and writers using the same endpoint contract? +- Does the change work with both memory and etcd-backed tests? +- Does region metadata route correctly when local region storage is enabled? +- Are load, save, delete, flush, and close semantics preserved? +- Does the change need migration, cleanup, or compatibility handling? +- Are storage errors propagated or intentionally logged-and-continued? diff --git a/docs/maintenance-guides/tso.md b/docs/maintenance-guides/tso.md new file mode 100644 index 00000000000..7cd4fcdf2dd --- /dev/null +++ b/docs/maintenance-guides/tso.md @@ -0,0 +1,158 @@ +# TSO + +This guide covers the timestamp oracle in `pkg/tso/` and the TSO microservice +surface under `pkg/mcs/tso/`. + +## Purpose And Scope + +TSO owns: + +- timestamp allocation +- timestamp physical/logical update windows +- allocator initialization and reset +- PD leader or TSO primary ownership checks +- keyspace group TSO routing +- TSO request handling in embedded and microservice modes +- TSO metrics and failure signals + +TSO does not own transaction semantics in TiDB or storage MVCC in TiKV, but +those systems depend on TSO monotonicity and availability. + +## Core Concepts + +- TSO correctness is monotonicity across leader changes, service-primary changes, + process restarts, and retries. +- `Allocator` serves timestamps only after ownership and initialization are both + valid. +- `timestampOracle` maintains the in-memory timestamp and persisted upper-bound + window. +- Keyspace group mode routes TSO requests by keyspace group ownership, not only + by PD leadership. +- Reset and logical overflow paths are correctness-sensitive because they change + the timestamp window. + +## Architectural Views + +### Embedded PD view + +Classic PD creates a `tso.Allocator` in `server.startServer`. When the PD member +becomes leader, `RaftCluster` initializes the allocator and `GrpcServer.Tso` +serves requests from it. + +### Microservice view + +The TSO service in `pkg/mcs/tso/server` runs primary election independently and +uses keyspace group metadata to decide which allocator serves a request. +Classic PD can fall back to embedded TSO depending on keyspace group mode, +service discovery, and dynamic switching config. + +### Timestamp oracle view + +`timestampOracle` owns the persisted timestamp window and in-memory TSO object. +The allocator updater periodically advances the window while serving. + +## Process Lifecycle And Startup Sequencing + +Important anchors: + +1. `tso.NewAllocator` +2. `Allocator.allocatorUpdater` +3. `Allocator.Initialize` +4. `Allocator.UpdateTSO` +5. `Allocator.GenerateTSO` +6. `Allocator.Reset` +7. `Allocator.primaryElectionLoop` +8. `pkg/tso/keyspace_group_manager.go` +9. `pkg/mcs/tso/server/server.go` +10. `pkg/mcs/tso/server/grpc_service.go` + +Maintenance rules: + +- Serving requires both ownership and allocator initialization. +- Reset must clear the allocator role and timestamp state before leadership is + released or retried. +- Physical time updates are correctness-sensitive and should not block on + unrelated work. + +## Data Model And Metadata Contracts + +Hot contracts: + +- physical and logical timestamp components +- TSO save interval and update physical interval +- max reset TS gap +- keyspace group ID +- expected primary flags in microservice mode +- TSO service discovery entries +- persisted timestamp upper bound + +Maintenance rule: + +- TSO must be monotonic across leader changes, primary changes, process restarts, + and network retries. + +## Observability And Operational Signals + +Open these first: + +- `pkg/tso/metrics.go` +- `pkg/mcs/tso/server/metrics.go` +- `server/grpc_service.go` +- `server/metrics.go` + +Signals to preserve: + +- allocator role gauge +- not-leader and forwarding failures +- TSO update errors and reset logs +- proxy stream timeout behavior +- microservice primary election logs + +## Change Management Guidance + +- TSO behavior changes need tests for leader transfer, reset, and logical + overflow when applicable. +- Microservice changes need both embedded fallback and independent primary + review. +- Request boundary changes need client and gRPC compatibility review. +- Config changes need persisted config and runtime side-effect review. + +## Must-Read File Order + +1. `pkg/tso/allocator.go` +2. `pkg/tso/tso.go` +3. `pkg/tso/keyspace_group_manager.go` +4. `pkg/tso/config.go` +5. `server/grpc_service.go` +6. `server/cluster/cluster.go` +7. `pkg/keyspace/tso_keyspace_group.go` +8. `pkg/mcs/tso/server/server.go` +9. `pkg/mcs/tso/server/grpc_service.go` +10. `client/client.go` + +## Glossary + +- TSO: + timestamp oracle used by TiDB transaction ordering. +- Allocator: + component that initializes timestamp state and generates timestamps. +- `timestampOracle`: + lower-level owner of persisted timestamp window and in-memory TSO state. +- Physical time: + wall-clock component of a timestamp. +- Logical time: + counter component used when multiple timestamps share the same physical time. +- TSO primary: + elected service owner allowed to serve TSO in microservice mode. +- Keyspace group: + group of keyspaces assigned to a TSO ownership unit. + +## Review Checklist + +- Can this member serve TSO only in the correct leader or primary state? +- Is timestamp monotonicity preserved across retries and resets? +- Are persisted timestamp windows advanced before serving new timestamps? +- Does the change affect keyspace group routing? +- Does dynamic switching between embedded PD and TSO service remain safe? +- Are client timeout, forwarding, and stream-close semantics unchanged? +- Do tests use failpoint-aware make targets when failpoints are involved?