diff --git a/decisiongrounding/docs/real-scenario-candidates.md b/decisiongrounding/docs/real-scenario-candidates.md index 035e69f..3993254 100644 --- a/decisiongrounding/docs/real-scenario-candidates.md +++ b/decisiongrounding/docs/real-scenario-candidates.md @@ -6,8 +6,10 @@ distractor pool. This is the reportable complement to the synthetic bank in `scenarios/` (which is auto-included in the sweep with no pre-registration impact). -**This file is a candidate list only.** Nothing here is ingested into -`scenarios_real/` yet — a `scenario.json` under `scenarios_real/` would break +**Status: Step 1 done.** The confirmed corpora below are ingested into +`scenarios_candidates/` (corpus + provenance, no `scenario.json` yet) — see +that directory's `README.md`. They are inert until promoted: nothing under +`scenarios_candidates/` is loaded by the harness or counted by `tests/test_real_roster.py::PINNED_ROSTER`. Accepting any candidate is a **deliberate pre-registration event**: it lands together with a `PINNED_ROSTER` update *and* a new analysis-plan amendment (see @@ -44,31 +46,38 @@ The ingest tools write `corpus/*.md` + `provenance.json` only. The satisfy the offline gate (`context_dump` adheres, `no_grounding` fails) and the scoring mechanics documented in `spec/scoring-rubric.md`. -## Candidates +## Ingested candidates (Step 1 — confirmed edges) -Estimated band is a pre-ingest guess; confirm with the overlap helper after -ingest. Gold labels are intentionally left as **TODO — blind author**. +Each row is staged in `scenarios_candidates//` with a reproducible corpus + +provenance and **no gold label yet**. Every edge is derived from the artifacts' +own headers at the pin (`Replaces`/`Superseded-By` for PEPs, `Obsoletes` for +RFCs) — none are hand-asserted — and verified with +`python -m ingest. verify --out scenarios_candidates/`. The overlap +band is an estimate from the topic; it is finalized when the blind task is +written and measured by `scenarios/overlap.py`. -| # | domain | ingest ids | proposed type | est. overlap | gold label | -|---|---|---|---|---|---| -| 1 | PEP typing | PEP 484 ← 483 | superseded_decision | high | TODO — blind author | -| 2 | PEP string formatting | PEP 3101 vs `%` formatting | prohibition_at_point_of_action | high | TODO — blind author | -| 3 | PEP packaging metadata | PEP 566 ← 345 | superseded_decision | high | TODO — blind author | -| 4 | PEP async | PEP 492 (async/await) vs generator coroutines | prohibition_at_point_of_action | high | TODO — blind author | -| 5 | PEP encodings | PEP 3120 (UTF-8 default source) | prohibition_at_point_of_action | medium | TODO — blind author | -| 6 | PEP division | PEP 238 (true division) supersedes classic | superseded_decision | medium | TODO — blind author | -| 7 | RFC HTTP | RFC 7231 ← 2616 (HTTP/1.1 semantics) | superseded_decision | medium | TODO — blind author | -| 8 | RFC TLS | RFC 8446 (TLS 1.3) prohibits legacy cipher suites | prohibition_at_point_of_action | medium | TODO — blind author | -| 9 | RFC email | RFC 5321 ← 2821 (SMTP) | superseded_decision | low | TODO — blind author | -| 10 | RFC URI | RFC 3986 ← 2396 (URI syntax) | superseded_decision | low | TODO — blind author | -| 11 | RFC auth | RFC 6749 (OAuth 2.0) prohibits the implicit grant in current guidance | prohibition_at_point_of_action | low | TODO — blind author | -| 12 | RFC DNS | RFC 8484 (DNS over HTTPS) vs cleartext DNS | conflicting_scoped | low | TODO — blind author | -| 13 | RFC datetime | RFC 3339 timestamp profile | prohibition_at_point_of_action | low | TODO — blind author | -| 14 | W3C | HTML5 supersedes an earlier dated edition | superseded_decision | low | TODO — blind author | -| 15 | W3C | CSS module leveling supersession | superseded_decision | low | TODO — blind author | +| candidate id | edge (source ← target) | proposed type | est. overlap | +|---|---|---|---| +| `peps_class_creation_supersession` | PEP-0487 ← 0422 | superseded_decision | high | +| `peps_typeis_narrowing_supersession` | PEP-0742 ← 0724 | superseded_decision | high | +| `peps_single_dispatch_supersession` | PEP-0443 ← 0245, 0246 | superseded_decision | high | +| `peps_metadata_v12_supersession` | PEP-0345 ← 0314 (← 0241) | superseded_decision | medium | +| `peps_package_layout_supersession` | PEP-0402 ← 0382 | superseded_decision | medium | +| `peps_hashlib_api_supersession` | PEP-0452 ← 0247 | superseded_decision | medium | +| `peps_packaging_governance_supersession` | PEP-0772 ← 0609 | superseded_decision | low | +| `peps_windows_installer_supersession` | PEP-0773 ← 0397, 0486 | superseded_decision | low | +| `peps_tls_api_supersession` | PEP-0748 ← 0543 | superseded_decision | low | +| `peps_pypi_mirror_supersession` | PEP-0449 ← 0381 | superseded_decision | low | +| `rfc_utf8_supersession` | RFC-3629 ← 2279 | superseded_decision | medium | +| `rfc_tcp_supersession` | RFC-9293 ← 0793 | superseded_decision | low | +| `rfc_ipv6_supersession` | RFC-8200 ← 2460 | superseded_decision | low | -Target distribution across the three discriminating types and three overlap -bands, so an accepted subset can fill the same matrix the synthetic bank -covers. Ingesting more candidates than are ultimately accepted is fine — only -those with a completed blind gold label and a passing offline gate join the -roster (and the roster pin). +Step 1 deliberately covers **supersession** edges only — they are mechanically +verifiable from headers. Prohibition and conflicting-scoped real scenarios +(which have no machine-readable edge) are a follow-on: the blind author selects +a document carrying a verbatim `MUST NOT`/prohibition and writes the task, using +the synthetic bank's prohibition/conflicting scenarios as the shape reference. + +Ingesting more candidates than are ultimately accepted is fine — only those +with a completed blind gold label and a passing offline gate join the roster +(and the roster pin). diff --git a/decisiongrounding/scenarios_candidates/README.md b/decisiongrounding/scenarios_candidates/README.md new file mode 100644 index 0000000..c95850d --- /dev/null +++ b/decisiongrounding/scenarios_candidates/README.md @@ -0,0 +1,51 @@ +# Scenario candidates (staging area — NOT the reportable roster) + +Real, pinned corpora ingested for the graded-overlap roster expansion, held +here **before** they join the frozen `scenarios_real/` roster. Each directory +has `corpus/*.md` + `provenance.json` (reproducible from the pin) but **no +`scenario.json`** — the task and gold label are authored later, blind +(CONTRIBUTING rule 1), ideally by an independent third party. + +Nothing here is loaded by the harness or counted by the roster: +`scenarios/loader.py::load_scenarios` and `tests/test_real_roster.py` only see +directories containing a `scenario.json`. So these corpora are inert until +promoted. + +## What's staged (Step 1 of the expansion) + +13 real supersession edges, all verified to reproduce from their pins +(`python -m ingest.peps|rfcs verify --out scenarios_candidates/`). The +overlap band is an *estimate* until the blind task is written and measured by +`scenarios/overlap.py`. + +| candidate id | edge (source supersedes target) | est. overlap | +|---|---|---| +| `peps_class_creation_supersession` | PEP-0487 ← PEP-0422 | high | +| `peps_typeis_narrowing_supersession` | PEP-0742 ← PEP-0724 | high | +| `peps_single_dispatch_supersession` | PEP-0443 ← PEP-0245, PEP-0246 | high | +| `peps_metadata_v12_supersession` | PEP-0345 ← PEP-0314 (← PEP-0241) | medium | +| `peps_package_layout_supersession` | PEP-0402 ← PEP-0382 | medium | +| `peps_hashlib_api_supersession` | PEP-0452 ← PEP-0247 | medium | +| `peps_packaging_governance_supersession` | PEP-0772 ← PEP-0609 | low | +| `peps_windows_installer_supersession` | PEP-0773 ← PEP-0397, PEP-0486 | low | +| `peps_tls_api_supersession` | PEP-0748 ← PEP-0543 | low | +| `peps_pypi_mirror_supersession` | PEP-0449 ← PEP-0381 | low | +| `rfc_utf8_supersession` | RFC-3629 ← RFC-2279 | medium | +| `rfc_tcp_supersession` | RFC-9293 ← RFC-0793 | low | +| `rfc_ipv6_supersession` | RFC-8200 ← RFC-2460 | low | + +All edges are derived from the artifacts' own `Replaces` / `Superseded-By` +(PEP) or `Obsoletes` (RFC) headers at the pin — none are hand-asserted. + +## Promoting a candidate into the roster (Steps 2–3) + +1. **Author `scenario.json` blind** (task + `gold_label`), per + `spec/external-scenario-request-graded-overlap.md`. Must pass the offline + gate (`context_dump` adheres, `no_grounding` fails) and match its declared + `lexical_overlap` band under `scenarios/overlap.py`. +2. `git mv scenarios_candidates/ scenarios_real/` (provenance is + path-independent; re-run `verify` afterward). +3. Extend `tests/test_real_roster.py::PINNED_ROSTER` and its per-domain counts, + and land a new `spec/analysis-plan-amendment-*.md` re-registering the + enlarged roster — the deliberate pre-registration event. Amendment-1 and + amendment-2 stay frozen. diff --git a/decisiongrounding/scenarios_candidates/peps_class_creation_supersession/corpus/PEP-0422.md b/decisiongrounding/scenarios_candidates/peps_class_creation_supersession/corpus/PEP-0422.md new file mode 100644 index 0000000..653e05e --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_class_creation_supersession/corpus/PEP-0422.md @@ -0,0 +1,583 @@ +--- +schema_version: 1 +id: PEP-0422 +type: decision +tags: [pep] +--- + +# PEP-0422 — Simpler customisation of class creation + +## Status + +Withdrawn + +## Context + +Public decision ingested verbatim from python/peps (PEP 422) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0422.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0422.rst). Upstream +status: Withdrawn. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 422 +Title: Simpler customisation of class creation +Author: Alyssa Coghlan , + Daniel Urban +Status: Withdrawn +Type: Standards Track +Created: 05-Jun-2012 +Python-Version: 3.5 +Post-History: 05-Jun-2012, 10-Feb-2013 + + +Abstract +======== + +Currently, customising class creation requires the use of a custom metaclass. +This custom metaclass then persists for the entire lifecycle of the class, +creating the potential for spurious metaclass conflicts. + +This PEP proposes to instead support a wide range of customisation +scenarios through a new ``namespace`` parameter in the class header, and +a new ``__autodecorate__`` hook in the class body. + +The new mechanism should be easier to understand and use than +implementing a custom metaclass, and thus should provide a gentler +introduction to the full power Python's metaclass machinery. + + +PEP Withdrawal +============== + +This proposal has been withdrawn in favour of Martin Teichmann's proposal +in :pep:`487`, which achieves the same goals through a simpler, easier to use +``__init_subclass__`` hook that simply isn't invoked for the base class +that defines the hook. + + +Background +========== + +For an already created class ``cls``, the term "metaclass" has a clear +meaning: it is the value of ``type(cls)``. + +*During* class creation, it has another meaning: it is also used to refer to +the metaclass hint that may be provided as part of the class definition. +While in many cases these two meanings end up referring to one and the same +object, there are two situations where that is not the case: + +* If the metaclass hint refers to an instance of ``type``, then it is + considered as a candidate metaclass along with the metaclasses of all of + the parents of the class being defined. If a more appropriate metaclass is + found amongst the candidates, then it will be used instead of the one + given in the metaclass hint. +* Otherwise, an explicit metaclass hint is assumed to be a factory function + and is called directly to create the class object. In this case, the final + metaclass will be determined by the factory function definition. In the + typical case (where the factory functions just calls ``type``, or, in + Python 3.3 or later, ``types.new_class``) the actual metaclass is then + determined based on the parent classes. + +It is notable that only the actual metaclass is inherited - a factory +function used as a metaclass hook sees only the class currently being +defined, and is not invoked for any subclasses. + +In Python 3, the metaclass hint is provided using the ``metaclass=Meta`` +keyword syntax in the class header. This allows the ``__prepare__`` method +on the metaclass to be used to create the ``locals()`` namespace used during +execution of the class body (for example, specifying the use of +``collections.OrderedDict`` instead of a regular ``dict``). + +In Python 2, there was no ``__prepare__`` method (that API was added for +Python 3 by :pep:`3115`). Instead, a class body could set the ``__metaclass__`` +attribute, and the class creation process would extract that value from the +class namespace to use as the metaclass hint. There is `published code`_ that +makes use of this feature. + +Another new feature in Python 3 is the zero-argument form of the ``super()`` +builtin, introduced by :pep:`3135`. This feature uses an implicit ``__class__`` +reference to the class being defined to replace the "by name" references +required in Python 2. Just as code invoked during execution of a Python 2 +metaclass could not call methods that referenced the class by name (as the +name had not yet been bound in the containing scope), similarly, Python 3 +metaclasses cannot call methods that rely on the implicit ``__class__`` +reference (as it is not populated until after the metaclass has returned +control to the class creation machinery). + +Finally, when a class uses a custom metaclass, it can pose additional +challenges to the use of multiple inheritance, as a new class cannot +inherit from parent classes with unrelated metaclasses. This means that +it is impossible to add a metaclass to an already published class: such +an addition is a backwards incompatible change due to the risk of metaclass +conflicts. + + +Proposal +======== + +This PEP proposes that a new mechanism to customise class creation be +added to Python 3.4 that meets the following criteria: + +1. Integrates nicely with class inheritance structures (including mixins and + multiple inheritance) +2. Integrates nicely with the implicit ``__class__`` reference and + zero-argument ``super()`` syntax introduced by :pep:`3135` +3. Can be added to an existing base class without a significant risk of + introducing backwards compatibility problems +4. Restores the ability for class namespaces to have some influence on the + class creation process (above and beyond populating the namespace itself), + but potentially without the full flexibility of the Python 2 style + ``__metaclass__`` hook + +One mechanism that can achieve this goal is to add a new implicit class +decoration hook, modelled directly on the existing explicit class +decorators, but defined in the class body or in a parent class, rather than +being part of the class definition header. + +Specifically, it is proposed that class definitions be able to provide a +class initialisation hook as follows:: + + class Example: + def __autodecorate__(cls): + # This is invoked after the class is created, but before any + # explicit decorators are called + # The usual super() mechanisms are used to correctly support + # multiple inheritance. The class decorator style signature helps + # ensure that invoking the parent class is as simple as possible. + cls = super().__autodecorate__() + return cls + +To simplify the cooperative multiple inheritance case, ``object`` will gain +a default implementation of the hook that returns the class unmodified:: + + class object: + def __autodecorate__(cls): + return cls + +If a metaclass wishes to block implicit class decoration for some reason, it +must arrange for ``cls.__autodecorate__`` to trigger ``AttributeError``. + +If present on the created object, this new hook will be called by the class +creation machinery *after* the ``__class__`` reference has been initialised. +For ``types.new_class()``, it will be called as the last step before +returning the created class object. ``__autodecorate__`` is implicitly +converted to a class method when the class is created (prior to the hook +being invoked). + +Note, that when ``__autodecorate__`` is called, the name of the class is not +yet bound to the new class object. As a consequence, the two argument form +of ``super()`` cannot be used to call methods (e.g., ``super(Example, cls)`` +wouldn't work in the example above). However, the zero argument form of +``super()`` works as expected, since the ``__class__`` reference is already +initialised. + +This general proposal is not a new idea (it was first suggested for +inclusion in the language definition `more than 10 years ago`_, and a +similar mechanism has long been supported by `Zope's ExtensionClass`_), +but the situation has changed sufficiently in recent years that +the idea is worth reconsidering for inclusion as a native language feature. + +In addition, the introduction of the metaclass ``__prepare__`` method in PEP +3115 allows a further enhancement that was not possible in Python 2: this +PEP also proposes that ``type.__prepare__`` be updated to accept a factory +function as a ``namespace`` keyword-only argument. If present, the value +provided as the ``namespace`` argument will be called without arguments +to create the result of ``type.__prepare__`` instead of using a freshly +created dictionary instance. For example, the following will use +an ordered dictionary as the class namespace:: + + class OrderedExample(namespace=collections.OrderedDict): + def __autodecorate__(cls): + # cls.__dict__ is still a read-only proxy to the class namespace, + # but the underlying storage is an OrderedDict instance + +.. note:: + + This PEP, along with the existing ability to use __prepare__ to share a + single namespace amongst multiple class objects, highlights a possible + issue with the attribute lookup caching: when the underlying mapping is + updated by other means, the attribute lookup cache is not invalidated + correctly (this is a key part of the reason class ``__dict__`` attributes + produce a read-only view of the underlying storage). + + Since the optimisation provided by that cache is highly desirable, + the use of a preexisting namespace as the class namespace may need to + be declared as officially unsupported (since the observed behaviour is + rather strange when the caches get out of sync). + + +Key Benefits +============ + + +Easier use of custom namespaces for a class +------------------------------------------- + +Currently, to use a different type (such as ``collections.OrderedDict``) for +a class namespace, or to use a pre-populated namespace, it is necessary to +write and use a custom metaclass. With this PEP, using a custom namespace +becomes as simple as specifying an appropriate factory function in the +class header. + + +Easier inheritance of definition time behaviour +----------------------------------------------- + +Understanding Python's metaclasses requires a deep understanding of +the type system and the class construction process. This is legitimately +seen as challenging, due to the need to keep multiple moving parts (the code, +the metaclass hint, the actual metaclass, the class object, instances of the +class object) clearly distinct in your mind. Even when you know the rules, +it's still easy to make a mistake if you're not being extremely careful. +An earlier version of this PEP actually included such a mistake: it +stated "subclass of type" for a constraint that is actually "instance of +type". + +Understanding the proposed implicit class decoration hook only requires +understanding decorators and ordinary method inheritance, which isn't +quite as daunting a task. The new hook provides a more gradual path +towards understanding all of the phases involved in the class definition +process. + + +Reduced chance of metaclass conflicts +------------------------------------- + +One of the big issues that makes library authors reluctant to use metaclasses +(even when they would be appropriate) is the risk of metaclass conflicts. +These occur whenever two unrelated metaclasses are used by the desired +parents of a class definition. This risk also makes it very difficult to +*add* a metaclass to a class that has previously been published without one. + +By contrast, adding an ``__autodecorate__`` method to an existing type poses +a similar level of risk to adding an ``__init__`` method: technically, there +is a risk of breaking poorly implemented subclasses, but when that occurs, +it is recognised as a bug in the subclass rather than the library author +breaching backwards compatibility guarantees. In fact, due to the constrained +signature of ``__autodecorate__``, the risk in this case is actually even +lower than in the case of ``__init__``. + + +Integrates cleanly with PEP 3135 +-------------------------------- + +Unlike code that runs as part of the metaclass, code that runs as part of +the new hook will be able to freely invoke class methods that rely on the +implicit ``__class__`` reference introduced by :pep:`3135`, including methods +that use the zero argument form of ``super()``. + + +Replaces many use cases for dynamic setting of ``__metaclass__`` +----------------------------------------------------------------- + +For use cases that don't involve completely replacing the defined class, +Python 2 code that dynamically set ``__metaclass__`` can now dynamically +set ``__autodecorate__`` instead. For more advanced use cases, introduction of +an explicit metaclass (possibly made available as a required base class) will +still be necessary in order to support Python 3. + + +Design Notes +============ + + +Determining if the class being decorated is the base class +---------------------------------------------------------- + +In the body of an ``__autodecorate__`` method, as in any other class method, +``__class__`` will be bound to the class declaring the method, while the +value passed in may be a subclass. + +This makes it relatively straightforward to skip processing the base class +if necessary:: + + class Example: + def __autodecorate__(cls): + cls = super().__autodecorate__() + # Don't process the base class + if cls is __class__: + return + # Process subclasses here + ... + + +Replacing a class with a different kind of object +------------------------------------------------- + +As an implicit decorator, ``__autodecorate__`` is able to relatively easily +replace the defined class with a different kind of object. Technically +custom metaclasses and even ``__new__`` methods can already do this +implicitly, but the decorator model makes such code much easier to understand +and implement. + +:: + + class BuildDict: + def __autodecorate__(cls): + cls = super().__autodecorate__() + # Don't process the base class + if cls is __class__: + return + # Convert subclasses to ordinary dictionaries + return cls.__dict__.copy() + +It's not clear why anyone would ever do this implicitly based on inheritance +rather than just using an explicit decorator, but the possibility seems worth +noting. + + +Open Questions +============== + +Is the ``namespace`` concept worth the extra complexity? +-------------------------------------------------------- + +Unlike the new ``__autodecorate__`` hook the proposed ``namespace`` keyword +argument is not automatically inherited by subclasses. Given the way this +proposal is currently written , the only way to get a special namespace used +consistently in subclasses is still to write a custom metaclass with a +suitable ``__prepare__`` implementation. + +Changing the custom namespace factory to also be inherited would +significantly increase the complexity of this proposal, and introduce a +number of the same potential base class conflict issues as arise with the +use of custom metaclasses. + +Eric Snow has put forward a +`separate proposal `__ +to instead make the execution namespace for class bodies an ordered dictionary +by default, and capture the class attribute definition order for future +reference as an attribute (e.g. ``__definition_order__``) on the class object. + +Eric's suggested approach may be a better choice for a new default behaviour +for type that combines well with the proposed ``__autodecorate__`` hook, +leaving the more complex configurable namespace factory idea to a custom +metaclass like the one shown below. + + +New Ways of Using Classes +========================= + +The new ``namespace`` keyword in the class header enables a number of +interesting options for controlling the way a class is initialised, +including some aspects of the object models of both Javascript and Ruby. + +All of the examples below are actually possible today through the use of a +custom metaclass:: + + class CustomNamespace(type): + @classmethod + def __prepare__(meta, name, bases, *, namespace=None, **kwds): + parent_namespace = super().__prepare__(name, bases, **kwds) + return namespace() if namespace is not None else parent_namespace + def __new__(meta, name, bases, ns, *, namespace=None, **kwds): + return super().__new__(meta, name, bases, ns, **kwds) + def __init__(cls, name, bases, ns, *, namespace=None, **kwds): + return super().__init__(name, bases, ns, **kwds) + +The advantage of implementing the new keyword directly in +``type.__prepare__`` is that the *only* persistent effect is then +the change in the underlying storage of the class attributes. The metaclass +of the class remains unchanged, eliminating many of the drawbacks +typically associated with these kinds of customisations. + + +Order preserving classes +------------------------ + +:: + + class OrderedClass(namespace=collections.OrderedDict): + a = 1 + b = 2 + c = 3 + + +Prepopulated namespaces +----------------------- + +:: + + seed_data = dict(a=1, b=2, c=3) + class PrepopulatedClass(namespace=seed_data.copy): + pass + + +Cloning a prototype class +------------------------- + +:: + + class NewClass(namespace=Prototype.__dict__.copy): + pass + + +Extending a class +----------------- + +.. note:: Just because the PEP makes it *possible* to do this relatively + cleanly doesn't mean anyone *should* do this! + +:: + + from collections import MutableMapping + + # The MutableMapping + dict combination should give something that + # generally behaves correctly as a mapping, while still being accepted + # as a class namespace + class ClassNamespace(MutableMapping, dict): + def __init__(self, cls): + self._cls = cls + def __len__(self): + return len(dir(self._cls)) + def __iter__(self): + for attr in dir(self._cls): + yield attr + def __contains__(self, attr): + return hasattr(self._cls, attr) + def __getitem__(self, attr): + return getattr(self._cls, attr) + def __setitem__(self, attr, value): + setattr(self._cls, attr, value) + def __delitem__(self, attr): + delattr(self._cls, attr) + + def extend(cls): + return lambda: ClassNamespace(cls) + + class Example: + pass + + class ExtendedExample(namespace=extend(Example)): + a = 1 + b = 2 + c = 3 + + >>> Example.a, Example.b, Example.c + (1, 2, 3) + + +Rejected Design Options +======================= + + +Calling ``__autodecorate__`` from ``type.__init__`` +--------------------------------------------------- + +Calling the new hook automatically from ``type.__init__``, would achieve most +of the goals of this PEP. However, using that approach would mean that +``__autodecorate__`` implementations would be unable to call any methods that +relied on the ``__class__`` reference (or used the zero-argument form of +``super()``), and could not make use of those features themselves. + +The current design instead ensures that the implicit decorator hook is able +to do anything an explicit decorator can do by running it after the initial +class creation is already complete. + +Calling the automatic decoration hook ``__init_class__`` +-------------------------------------------------------- + +Earlier versions of the PEP used the name ``__init_class__`` for the name +of the new hook. There were three significant problems with this name: + +* it was hard to remember if the correct spelling was ``__init_class__`` or + ``__class_init__`` +* the use of "init" in the name suggested the signature should match that + of ``type.__init__``, which is not the case +* the use of "init" in the name suggested the method would be run as part + of initial class object creation, which is not the case + +The new name ``__autodecorate__`` was chosen to make it clear that the new +initialisation hook is most usefully thought of as an implicitly invoked +class decorator, rather than as being like an ``__init__`` method. + + +Requiring an explicit decorator on ``__autodecorate__`` +------------------------------------------------------- + +Originally, this PEP required the explicit use of ``@classmethod`` on the +``__autodecorate__`` decorator. It was made implicit since there's no +sensible interpretation for leaving it out, and that case would need to be +detected anyway in order to give a useful error message. + +This decision was reinforced after noticing that the user experience of +defining ``__prepare__`` and forgetting the ``@classmethod`` method +decorator is singularly incomprehensible (particularly since :pep:`3115` +documents it as an ordinary method, and the current documentation doesn't +explicitly say anything one way or the other). + + +Making ``__autodecorate__`` implicitly static, like ``__new__`` +--------------------------------------------------------------- + +While it accepts the class to be instantiated as the first argument, +``__new__`` is actually implicitly treated as a static method rather than +as a class method. This allows it to be readily extracted from its +defining class and called directly on a subclass, rather than being +coupled to the class object it is retrieved from. + +Such behaviour initially appears to be potentially useful for the +new ``__autodecorate__`` hook, as it would allow ``__autodecorate__`` +methods to readily be used as explicit decorators on other classes. + +However, that apparent support would be an illusion as it would only work +correctly if invoked on a subclass, in which case the method can just as +readily be retrieved from the subclass and called that way. Unlike +``__new__``, there's no issue with potentially changing method signatures at +different points in the inheritance chain. + + +Passing in the namespace directly rather than a factory function +---------------------------------------------------------------- + +At one point, this PEP proposed that the class namespace be passed +directly as a keyword argument, rather than passing a factory function. +However, this encourages an unsupported behaviour (that is, passing the +same namespace to multiple classes, or retaining direct write access +to a mapping used as a class namespace), so the API was switched to +the factory function version. + + +Reference Implementation +======================== + +A reference implementation for ``__autodecorate__`` has been posted to the +`issue tracker`_. It uses the original ``__init_class__`` naming. does not yet +allow the implicit decorator to replace the class with a different object and +does not implement the suggested ``namespace`` parameter for +``type.__prepare__``. + +TODO +==== + +* address the 5 points in https://mail.python.org/pipermail/python-dev/2013-February/123970.html + +References +========== + +.. _published code: + https://mail.python.org/pipermail/python-dev/2012-June/119878.html + +.. _more than 10 years ago: + https://mail.python.org/pipermail/python-dev/2001-November/018651.html + +.. _Zope's ExtensionClass: + http://docs.zope.org/zope_secrets/extensionclass.html + +.. _issue tracker: + http://bugs.python.org/issue17044 + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_class_creation_supersession/corpus/PEP-0487.md b/decisiongrounding/scenarios_candidates/peps_class_creation_supersession/corpus/PEP-0487.md new file mode 100644 index 0000000..c070976 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_class_creation_supersession/corpus/PEP-0487.md @@ -0,0 +1,508 @@ +--- +schema_version: 1 +id: PEP-0487 +type: decision +tags: [pep] +--- + +# PEP-0487 — Simpler customisation of class creation + +## Status + +Final + +## Context + +Public decision ingested verbatim from python/peps (PEP 487) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0487.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0487.rst). Upstream +status: Final. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0422 + +## Source Text + +PEP: 487 +Title: Simpler customisation of class creation +Author: Martin Teichmann , +Status: Final +Type: Standards Track +Created: 27-Feb-2015 +Python-Version: 3.6 +Post-History: 27-Feb-2015, 05-Feb-2016, 24-Jun-2016, 02-Jul-2016, 13-Jul-2016 +Replaces: 422 +Resolution: https://mail.python.org/pipermail/python-dev/2016-July/145629.html + + +Abstract +======== + +Currently, customising class creation requires the use of a custom metaclass. +This custom metaclass then persists for the entire lifecycle of the class, +creating the potential for spurious metaclass conflicts. + +This PEP proposes to instead support a wide range of customisation +scenarios through a new ``__init_subclass__`` hook in the class body, +and a hook to initialize attributes. + +The new mechanism should be easier to understand and use than +implementing a custom metaclass, and thus should provide a gentler +introduction to the full power of Python's metaclass machinery. + + +Background +========== + +Metaclasses are a powerful tool to customize class creation. They have, +however, the problem that there is no automatic way to combine metaclasses. +If one wants to use two metaclasses for a class, a new metaclass combining +those two needs to be created, typically manually. + +This need often occurs as a surprise to a user: inheriting from two base +classes coming from two different libraries suddenly raises the necessity +to manually create a combined metaclass, where typically one is not +interested in those details about the libraries at all. This becomes +even worse if one library starts to make use of a metaclass which it +has not done before. While the library itself continues to work perfectly, +suddenly every code combining those classes with classes from another library +fails. + +Proposal +======== + +While there are many possible ways to use a metaclass, the vast majority +of use cases falls into just three categories: some initialization code +running after class creation, the initialization of descriptors and +keeping the order in which class attributes were defined. + +The first two categories can easily be achieved by having simple hooks +into the class creation: + +1. An ``__init_subclass__`` hook that initializes + all subclasses of a given class. +2. upon class creation, a ``__set_name__`` hook is called on all the + attribute (descriptors) defined in the class, and + +The third category is the topic of another PEP, :pep:`520`. + +As an example, the first use case looks as follows:: + + >>> class QuestBase: + ... # this is implicitly a @classmethod (see below for motivation) + ... def __init_subclass__(cls, swallow, **kwargs): + ... cls.swallow = swallow + ... super().__init_subclass__(**kwargs) + + >>> class Quest(QuestBase, swallow="african"): + ... pass + + >>> Quest.swallow + 'african' + +The base class ``object`` contains an empty ``__init_subclass__`` +method which serves as an endpoint for cooperative multiple inheritance. +Note that this method has no keyword arguments, meaning that all +methods which are more specialized have to process all keyword +arguments. + +This general proposal is not a new idea (it was first suggested for +inclusion in the language definition `more than 10 years ago`_, and a +similar mechanism has long been supported by `Zope's ExtensionClass`_), +but the situation has changed sufficiently in recent years that +the idea is worth reconsidering for inclusion. + +The second part of the proposal adds an ``__set_name__`` +initializer for class attributes, especially if they are descriptors. +Descriptors are defined in the body of a +class, but they do not know anything about that class, they do not +even know the name they are accessed with. They do get to know their +owner once ``__get__`` is called, but still they do not know their +name. This is unfortunate, for example they cannot put their +associated value into their object's ``__dict__`` under their name, +since they do not know that name. This problem has been solved many +times, and is one of the most important reasons to have a metaclass in +a library. While it would be easy to implement such a mechanism using +the first part of the proposal, it makes sense to have one solution +for this problem for everyone. + +To give an example of its usage, imagine a descriptor representing weak +referenced values:: + + import weakref + + class WeakAttribute: + def __get__(self, instance, owner): + return instance.__dict__[self.name]() + + def __set__(self, instance, value): + instance.__dict__[self.name] = weakref.ref(value) + + # this is the new initializer: + def __set_name__(self, owner, name): + self.name = name + +Such a ``WeakAttribute`` may, for example, be used in a tree structure +where one wants to avoid cyclic references via the parent:: + + class TreeNode: + parent = WeakAttribute() + + def __init__(self, parent): + self.parent = parent + +Note that the ``parent`` attribute is used like a normal attribute, +yet the tree contains no cyclic references and can thus be easily +garbage collected when out of use. The ``parent`` attribute magically +becomes ``None`` once the parent ceases existing. + +While this example looks very trivial, it should be noted that until +now such an attribute cannot be defined without the use of a metaclass. +And given that such a metaclass can make life very hard, this kind of +attribute does not exist yet. + +Initializing descriptors could simply be done in the +``__init_subclass__`` hook. But this would mean that descriptors can +only be used in classes that have the proper hook, the generic version +like in the example would not work generally. One could also call +``__set_name__`` from within the base implementation of +``object.__init_subclass__``. But given that it is a common mistake +to forget to call ``super()``, it would happen too often that suddenly +descriptors are not initialized. + + +Key Benefits +============ + + +Easier inheritance of definition time behaviour +----------------------------------------------- + +Understanding Python's metaclasses requires a deep understanding of +the type system and the class construction process. This is legitimately +seen as challenging, due to the need to keep multiple moving parts (the code, +the metaclass hint, the actual metaclass, the class object, instances of the +class object) clearly distinct in your mind. Even when you know the rules, +it's still easy to make a mistake if you're not being extremely careful. + +Understanding the proposed implicit class initialization hook only requires +ordinary method inheritance, which isn't quite as daunting a task. The new +hook provides a more gradual path towards understanding all of the phases +involved in the class definition process. + + +Reduced chance of metaclass conflicts +------------------------------------- + +One of the big issues that makes library authors reluctant to use metaclasses +(even when they would be appropriate) is the risk of metaclass conflicts. +These occur whenever two unrelated metaclasses are used by the desired +parents of a class definition. This risk also makes it very difficult to +*add* a metaclass to a class that has previously been published without one. + +By contrast, adding an ``__init_subclass__`` method to an existing type poses +a similar level of risk to adding an ``__init__`` method: technically, there +is a risk of breaking poorly implemented subclasses, but when that occurs, +it is recognised as a bug in the subclass rather than the library author +breaching backwards compatibility guarantees. + + +New Ways of Using Classes +========================= + +Subclass registration +--------------------- + +Especially when writing a plugin system, one likes to register new +subclasses of a plugin baseclass. This can be done as follows:: + + class PluginBase: + subclasses = [] + + def __init_subclass__(cls, **kwargs): + super().__init_subclass__(**kwargs) + cls.subclasses.append(cls) + +In this example, ``PluginBase.subclasses`` will contain a plain list of all +subclasses in the entire inheritance tree. One should note that this also +works nicely as a mixin class. + +Trait descriptors +----------------- + +There are many designs of Python descriptors in the wild which, for +example, check boundaries of values. Often those "traits" need some support +of a metaclass to work. This is how this would look like with this +PEP:: + + class Trait: + def __init__(self, minimum, maximum): + self.minimum = minimum + self.maximum = maximum + + def __get__(self, instance, owner): + return instance.__dict__[self.key] + + def __set__(self, instance, value): + if self.minimum < value < self.maximum: + instance.__dict__[self.key] = value + else: + raise ValueError("value not in range") + + def __set_name__(self, owner, name): + self.key = name + +Implementation Details +====================== + +The hooks are called in the following order: ``type.__new__`` calls +the ``__set_name__`` hooks on the descriptor after the new class has been +initialized. Then it calls ``__init_subclass__`` on the base class, on +``super()``, to be precise. This means that subclass initializers already +see the fully initialized descriptors. This way, ``__init_subclass__`` users +can fix all descriptors again if this is needed. + +Another option would have been to call ``__set_name__`` in the base +implementation of ``object.__init_subclass__``. This way it would be possible +even to prevent ``__set_name__`` from being called. Most of the times, +however, such a prevention would be accidental, as it often happens that a call +to ``super()`` is forgotten. + +As a third option, all the work could have been done in ``type.__init__``. +Most metaclasses do their work in ``__new__``, as this is recommended by +the documentation. Many metaclasses modify their arguments before they +pass them over to ``super().__new__``. For compatibility with those kind +of classes, the hooks should be called from ``__new__``. + +Another small change should be done: in the current implementation of +CPython, ``type.__init__`` explicitly forbids the use of keyword arguments, +while ``type.__new__`` allows for its attributes to be shipped as keyword +arguments. This is weirdly incoherent, and thus it should be forbidden. +While it would be possible to retain the current behavior, it would be better +if this was fixed, as it is probably not used at all: the only use case would +be that at metaclass calls its ``super().__new__`` with *name*, *bases* and +*dict* (yes, *dict*, not *namespace* or *ns* as mostly used with modern +metaclasses) as keyword arguments. This should not be done. This little +change simplifies the implementation of this PEP significantly, while +improving the coherence of Python overall. + +As a second change, the new ``type.__init__`` just ignores keyword +arguments. Currently, it insists that no keyword arguments are given. This +leads to a (wanted) error if one gives keyword arguments to a class declaration +if the metaclass does not process them. Metaclass authors that do want to +accept keyword arguments must filter them out by overriding ``__init__``. + +In the new code, it is not ``__init__`` that complains about keyword arguments, +but ``__init_subclass__``, whose default implementation takes no arguments. In +a classical inheritance scheme using the method resolution order, each +``__init_subclass__`` may take out it's keyword arguments until none are left, +which is checked by the default implementation of ``__init_subclass__``. + +For readers who prefer reading Python over English, this PEP proposes to +replace the current ``type`` and ``object`` with the following:: + + class NewType(type): + def __new__(cls, *args, **kwargs): + if len(args) != 3: + return super().__new__(cls, *args) + name, bases, ns = args + init = ns.get('__init_subclass__') + if isinstance(init, types.FunctionType): + ns['__init_subclass__'] = classmethod(init) + self = super().__new__(cls, name, bases, ns) + for k, v in self.__dict__.items(): + func = getattr(v, '__set_name__', None) + if func is not None: + func(self, k) + super(self, self).__init_subclass__(**kwargs) + return self + + def __init__(self, name, bases, ns, **kwargs): + super().__init__(name, bases, ns) + + class NewObject(object): + @classmethod + def __init_subclass__(cls): + pass + + +Reference Implementation +======================== + +The reference implementation for this PEP is attached to +`issue 27366 `__. + + +Backward compatibility issues +============================= + +The exact calling sequence in ``type.__new__`` is slightly changed, raising +fears of backwards compatibility. It should be assured by tests that common use +cases behave as desired. + +The following class definitions (except the one defining the metaclass) +continue to fail with a ``TypeError`` as superfluous class arguments are passed:: + + class MyMeta(type): + pass + + class MyClass(metaclass=MyMeta, otherarg=1): + pass + + MyMeta("MyClass", (), otherargs=1) + + import types + types.new_class("MyClass", (), dict(metaclass=MyMeta, otherarg=1)) + types.prepare_class("MyClass", (), dict(metaclass=MyMeta, otherarg=1)) + +A metaclass defining only a ``__new__`` method which is interested in keyword +arguments now does not need to define an ``__init__`` method anymore, as the +default ``type.__init__`` ignores keyword arguments. This is nicely in line +with the recommendation to override ``__new__`` in metaclasses instead of +``__init__``. The following code does not fail anymore:: + + class MyMeta(type): + def __new__(cls, name, bases, namespace, otherarg): + return super().__new__(cls, name, bases, namespace) + + class MyClass(metaclass=MyMeta, otherarg=1): + pass + +Only defining an ``__init__`` method in a metaclass continues to fail with +``TypeError`` if keyword arguments are given:: + + class MyMeta(type): + def __init__(self, name, bases, namespace, otherarg): + super().__init__(name, bases, namespace) + + class MyClass(metaclass=MyMeta, otherarg=1): + pass + +Defining both ``__init__`` and ``__new__`` continues to work fine. + +About the only thing that stops working is passing the arguments of +``type.__new__`` as keyword arguments:: + + class MyMeta(type): + def __new__(cls, name, bases, namespace): + return super().__new__(cls, name=name, bases=bases, + dict=namespace) + + class MyClass(metaclass=MyMeta): + pass + +This will now raise ``TypeError``, but this is weird code, and easy +to fix even if someone used this feature. + + +Rejected Design Options +======================= + + +Calling the hook on the class itself +------------------------------------ + +Adding an ``__autodecorate__`` hook that would be called on the class +itself was the proposed idea of :pep:`422`. Most examples work the same +way or even better if the hook is called only on strict subclasses. In general, +it is much easier to arrange to explicitly call the hook on the class in which it +is defined (to opt-in to such a behavior) than to opt-out (by remember to check for +``cls is __class`` in the hook body), meaning that one does not want the hook to be +called on the class it is defined in. + +This becomes most evident if the class in question is designed as a +mixin: it is very unlikely that the code of the mixin is to be +executed for the mixin class itself, as it is not supposed to be a +complete class on its own. + +The original proposal also made major changes in the class +initialization process, rendering it impossible to back-port the +proposal to older Python versions. + +When it's desired to also call the hook on the base class, two mechanisms are available: + +1. Introduce an additional mixin class just to hold the ``__init_subclass__`` + implementation. The original "base" class can then list the new mixin as its + first parent class. + +2. Implement the desired behaviour as an independent class decorator, and apply that + decorator explicitly to the base class, and then implicitly to subclasses via + ``__init_subclass__``. + +Calling ``__init_subclass__`` explicitly from a class decorator will generally be +undesirable, as this will also typically call ``__init_subclass__`` a second time on +the parent class, which is unlikely to be desired behaviour. + +Other variants of calling the hooks +----------------------------------- + +Other names for the hook were presented, namely ``__decorate__`` or +``__autodecorate__``. This proposal opts for ``__init_subclass__`` as +it is very close to the ``__init__`` method, just for the subclass, +while it is not very close to decorators, as it does not return the +class. + +For the ``__set_name__`` hook other names have been proposed as well, +``__set_owner__``, ``__set_ownership__`` and ``__init_descriptor__``. + + +Requiring an explicit decorator on ``__init_subclass__`` +-------------------------------------------------------- + +One could require the explicit use of ``@classmethod`` on the +``__init_subclass__`` decorator. It was made implicit since there's no +sensible interpretation for leaving it out, and that case would need +to be detected anyway in order to give a useful error message. + +This decision was reinforced after noticing that the user experience of +defining ``__prepare__`` and forgetting the ``@classmethod`` method +decorator is singularly incomprehensible (particularly since :pep:`3115` +documents it as an ordinary method, and the current documentation doesn't +explicitly say anything one way or the other). + +A more ``__new__``-like hook +---------------------------- + +In :pep:`422` the hook worked more like the ``__new__`` method than the +``__init__`` method, meaning that it returned a class instead of +modifying one. This allows a bit more flexibility, but at the cost +of much harder implementation and undesired side effects. + +Adding a class attribute with the attribute order +------------------------------------------------- + +This got its own :pep:`520`. + + +History +======= + +This used to be a competing proposal to :pep:`422` by Alyssa Coghlan and Daniel +Urban. :pep:`422` intended to achieve the same goals as this PEP, but with a +different way of implementation. In the meantime, :pep:`422` has been withdrawn +favouring this approach. + +References +========== + +.. _more than 10 years ago: + https://mail.python.org/pipermail/python-dev/2001-November/018651.html + +.. _Zope's ExtensionClass: + http://docs.zope.org/zope_secrets/extensionclass.html + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_class_creation_supersession/provenance.json b/decisiongrounding/scenarios_candidates/peps_class_creation_supersession/provenance.json new file mode 100644 index 0000000..71fded8 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_class_creation_supersession/provenance.json @@ -0,0 +1,37 @@ +{ + "source_repo": "python/peps", + "pinned_commit": "f866e77409305866038471574f075cd8d83eee9e", + "peps": [ + { + "id": "PEP-0422", + "number": 422, + "file": "corpus/PEP-0422.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0422.rst", + "source_sha256": "537db2c3afc6cb23c7154fb62481ecb595d89bdc3c8ecb1ca8d804b146ef4219", + "status": "Withdrawn", + "title": "Simpler customisation of class creation", + "replaces": [], + "superseded_by": [] + }, + { + "id": "PEP-0487", + "number": 487, + "file": "corpus/PEP-0487.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0487.rst", + "source_sha256": "d4b5eb903aad849008e359509d9d20c9587a1a9121e0e3e20117858b918700a7", + "status": "Final", + "title": "Simpler customisation of class creation", + "replaces": [ + "PEP-0422" + ], + "superseded_by": [] + } + ], + "supersedes_edges": [ + { + "source": "PEP-0487", + "type": "supersedes", + "target": "PEP-0422" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/peps_hashlib_api_supersession/corpus/PEP-0247.md b/decisiongrounding/scenarios_candidates/peps_hashlib_api_supersession/corpus/PEP-0247.md new file mode 100644 index 0000000..9cc1675 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_hashlib_api_supersession/corpus/PEP-0247.md @@ -0,0 +1,192 @@ +--- +schema_version: 1 +id: PEP-0247 +type: decision +tags: [pep] +--- + +# PEP-0247 — API for Cryptographic Hash Functions + +## Status + +Final + +## Context + +Public decision ingested verbatim from python/peps (PEP 247) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0247.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0247.rst). Upstream +status: Final. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 247 +Title: API for Cryptographic Hash Functions +Author: A.M. Kuchling +Status: Final +Type: Informational +Created: 23-Mar-2001 +Post-History: 20-Sep-2001 + + +Abstract +======== + +There are several different modules available that implement cryptographic +hashing algorithms such as MD5 or SHA. This document specifies a standard API +for such algorithms, to make it easier to switch between different +implementations. + + +Specification +============= + +All hashing modules should present the same interface. Additional methods or +variables can be added, but those described in this document should always be +present. + +Hash function modules define one function: + +| ``new([string]) (unkeyed hashes)`` +| ``new([key] , [string]) (keyed hashes)`` + + Create a new hashing object and return it. The first form is for hashes + that are unkeyed, such as MD5 or SHA. For keyed hashes such as HMAC, *key* + is a required parameter containing a string giving the key to use. In both + cases, the optional *string* parameter, if supplied, will be immediately + hashed into the object's starting state, as if ``obj.update(string)`` + was called. + + After creating a hashing object, arbitrary strings can be fed into the + object using its ``update()`` method, and the hash value can be obtained at + any time by calling the object's ``digest()`` method. + + Arbitrary additional keyword arguments can be added to this function, but if + they're not supplied, sensible default values should be used. For example, + ``rounds`` and ``digest_size`` keywords could be added for a hash function + which supports a variable number of rounds and several different output + sizes, and they should default to values believed to be secure. + +Hash function modules define one variable: + +| ``digest_size`` + + An integer value; the size of the digest produced by the hashing objects + created by this module, measured in bytes. You could also obtain this value + by creating a sample object and accessing its ``digest_size`` attribute, but + it can be convenient to have this value available from the module. Hashes + with a variable output size will set this variable to ``None``. + +Hashing objects require a single attribute: + +| ``digest_size`` + + This attribute is identical to the module-level ``digest_size`` variable, + measuring the size of the digest produced by the hashing object, measured in + bytes. If the hash has a variable output size, this output size must be + chosen when the hashing object is created, and this attribute must contain + the selected size. Therefore, ``None`` is *not* a legal value for this + attribute. + + +Hashing objects require the following methods: + +| ``copy()`` + + Return a separate copy of this hashing object. An update to this copy won't + affect the original object. + +| ``digest()`` + + Return the hash value of this hashing object as a string containing 8-bit + data. The object is not altered in any way by this function; you can + continue updating the object after calling this function. + +| ``hexdigest()`` + + Return the hash value of this hashing object as a string containing + hexadecimal digits. Lowercase letters should be used for the digits ``a`` + through ``f``. Like the ``.digest()`` method, this method mustn't alter the + object. + +| ``update(string)`` + + Hash *string* into the current state of the hashing object. ``update()`` can + be called any number of times during a hashing object's lifetime. + +Hashing modules can define additional module-level functions or object methods +and still be compliant with this specification. + +Here's an example, using a module named ``MD5``:: + + >>> from Crypto.Hash import MD5 + >>> m = MD5.new() + >>> m.digest_size + 16 + >>> m.update('abc') + >>> m.digest() + '\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr' + >>> m.hexdigest() + '900150983cd24fb0d6963f7d28e17f72' + >>> MD5.new('abc').digest() + '\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr' + + +Rationale +========= + +The digest size is measured in bytes, not bits, even though hash algorithm +sizes are usually quoted in bits; MD5 is a 128-bit algorithm and not a 16-byte +one, for example. This is because, in the sample code I looked at, the length +in bytes is often needed (to seek ahead or behind in a file; to compute the +length of an output string) while the length in bits is rarely used. Therefore, +the burden will fall on the few people actually needing the size in bits, who +will have to multiply ``digest_size`` by 8. + +It's been suggested that the ``update()`` method would be better named +``append()``. However, that method is really causing the current state of the +hashing object to be updated, and ``update()`` is already used by the md5 and +sha modules included with Python, so it seems simplest to leave the name +``update()`` alone. + +The order of the constructor's arguments for keyed hashes was a sticky issue. +It wasn't clear whether the *key* should come first or second. It's a required +parameter, and the usual convention is to place required parameters first, but +that also means that the *string* parameter moves from the first position to +the second. It would be possible to get confused and pass a single argument to +a keyed hash, thinking that you're passing an initial string to an unkeyed +hash, but it doesn't seem worth making the interface for keyed hashes more +obscure to avoid this potential error. + + +Changes +======= + +2001-09-17: Renamed ``clear()`` to ``reset()``; added ``digest_size`` attribute +to objects; added ``.hexdigest()`` method. + +2001-09-20: Removed ``reset()`` method completely. + +2001-09-28: Set ``digest_size`` to ``None`` for variable-size hashes. + + +Acknowledgements +================ + +Thanks to Aahz, Andrew Archibald, Rich Salz, Itamar Shtull-Trauring, and the +readers of the python-crypto list for their comments on this PEP. + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_hashlib_api_supersession/corpus/PEP-0452.md b/decisiongrounding/scenarios_candidates/peps_hashlib_api_supersession/corpus/PEP-0452.md new file mode 100644 index 0000000..fddb657 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_hashlib_api_supersession/corpus/PEP-0452.md @@ -0,0 +1,288 @@ +--- +schema_version: 1 +id: PEP-0452 +type: decision +tags: [pep] +--- + +# PEP-0452 — API for Cryptographic Hash Functions v2.0 + +## Status + +Final + +## Context + +Public decision ingested verbatim from python/peps (PEP 452) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0452.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0452.rst). Upstream +status: Final. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0247 + +## Source Text + +PEP: 452 +Title: API for Cryptographic Hash Functions v2.0 +Author: A.M. Kuchling , Christian Heimes +Status: Final +Type: Informational +Created: 15-Aug-2013 +Post-History: +Replaces: 247 + +Abstract +======== + +There are several different modules available that implement +cryptographic hashing algorithms such as MD5 or SHA. This +document specifies a standard API for such algorithms, to make it +easier to switch between different implementations. + + +Specification +============= + +All hashing modules should present the same interface. Additional +methods or variables can be added, but those described in this +document should always be present. + +Hash function modules define one function: + +``new([string]) (unkeyed hashes)`` + +``new(key, [string], [digestmod]) (keyed hashes)`` + Create a new hashing object and return it. The first form is + for hashes that are unkeyed, such as MD5 or SHA. For keyed + hashes such as HMAC, 'key' is a required parameter containing + a string giving the key to use. In both cases, the optional + 'string' parameter, if supplied, will be immediately hashed + into the object's starting state, as if ``obj.update(string)`` was + called. + + After creating a hashing object, arbitrary bytes can be fed + into the object using its ``update()`` method, and the hash value + can be obtained at any time by calling the object's ``digest()`` + method. + + Although the parameter is called 'string', hashing objects operate + on 8-bit data only. Both 'key' and 'string' must be a bytes-like + object (bytes, bytearray...). A hashing object may support + one-dimensional, contiguous buffers as argument, too. Text + (unicode) is no longer supported in Python 3.x. Python 2.x + implementations may take ASCII-only unicode as argument, but + portable code should not rely on the feature. + + Arbitrary additional keyword arguments can be added to this + function, but if they're not supplied, sensible default values + should be used. For example, 'rounds' and 'digest_size' + keywords could be added for a hash function which supports a + variable number of rounds and several different output sizes, + and they should default to values believed to be secure. + +Hash function modules define one variable: + +``digest_size`` + An integer value; the size of the digest produced by the + hashing objects created by this module, measured in bytes. + You could also obtain this value by creating a sample object + and accessing its 'digest_size' attribute, but it can be + convenient to have this value available from the module. + Hashes with a variable output size will set this variable to + None. + +Hashing objects require the following attribute: + +``digest_size`` + This attribute is identical to the module-level digest_size + variable, measuring the size of the digest produced by the + hashing object, measured in bytes. If the hash has a variable + output size, this output size must be chosen when the hashing + object is created, and this attribute must contain the + selected size. Therefore, ``None`` is **not** a legal value for this + attribute. + +``block_size`` + An integer value or ``NotImplemented``; the internal block size + of the hash algorithm in bytes. The block size is used by the + HMAC module to pad the secret key to ``digest_size`` or to hash the + secret key if it is longer than ``digest_size``. If no HMAC + algorithm is standardized for the hash algorithm, return + ``NotImplemented`` instead. + +``name`` + A text string value; the canonical, lowercase name of the hashing + algorithm. The name should be a suitable parameter for + ``hashlib.new``. + +Hashing objects require the following methods: + +``copy()`` + Return a separate copy of this hashing object. An update to + this copy won't affect the original object. + +``digest()`` + Return the hash value of this hashing object as a bytes + containing 8-bit data. The object is not altered in any way + by this function; you can continue updating the object after + calling this function. + +``hexdigest()`` + Return the hash value of this hashing object as a string + containing hexadecimal digits. Lowercase letters should be used + for the digits 'a' through 'f'. Like the ``.digest()`` method, this + method mustn't alter the object. + +``update(string)`` + Hash bytes-like 'string' into the current state of the hashing + object. ``update()`` can be called any number of times during a + hashing object's lifetime. + +Hashing modules can define additional module-level functions or +object methods and still be compliant with this specification. + +Here's an example, using a module named 'MD5':: + + >>> import hashlib + >>> from Crypto.Hash import MD5 + >>> m = MD5.new() + >>> isinstance(m, hashlib.CryptoHash) + True + >>> m.name + 'md5' + >>> m.digest_size + 16 + >>> m.block_size + 64 + >>> m.update(b'abc') + >>> m.digest() + b'\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr' + >>> m.hexdigest() + '900150983cd24fb0d6963f7d28e17f72' + >>> MD5.new(b'abc').digest() + b'\x90\x01P\x98<\xd2O\xb0\xd6\x96?}(\xe1\x7fr' + + +Rationale +========= + +The digest size is measured in bytes, not bits, even though hash +algorithm sizes are usually quoted in bits; MD5 is a 128-bit +algorithm and not a 16-byte one, for example. This is because, in +the sample code I looked at, the length in bytes is often needed +(to seek ahead or behind in a file; to compute the length of an +output string) while the length in bits is rarely used. +Therefore, the burden will fall on the few people actually needing +the size in bits, who will have to multiply digest_size by 8. + +It's been suggested that the ``update()`` method would be better named +``append()``. However, that method is really causing the current +state of the hashing object to be updated, and ``update()`` is already +used by the md5 and sha modules included with Python, so it seems +simplest to leave the name ``update()`` alone. + +The order of the constructor's arguments for keyed hashes was a +sticky issue. It wasn't clear whether the key should come first +or second. It's a required parameter, and the usual convention is +to place required parameters first, but that also means that the +'string' parameter moves from the first position to the second. +It would be possible to get confused and pass a single argument to +a keyed hash, thinking that you're passing an initial string to an +unkeyed hash, but it doesn't seem worth making the interface +for keyed hashes more obscure to avoid this potential error. + + +Changes from Version 1.0 to Version 2.0 +======================================= + +Version 2.0 of API for Cryptographic Hash Functions clarifies some +aspects of the API and brings it up-to-date. It also formalized aspects +that were already de facto standards and provided by most +implementations. + +Version 2.0 introduces the following new attributes: + +``name`` + The name property was made mandatory by `issue 18532`_. + +``block_size`` + The new version also specifies that the return value + ``NotImplemented`` prevents HMAC support. + +Version 2.0 takes the separation of binary and text data in Python +3.0 into account. The 'string' argument to ``new()`` and ``update()`` as +well as the 'key' argument must be bytes-like objects. On Python +2.x a hashing object may also support ASCII-only unicode. The actual +name of argument is not changed as it is part of the public API. +Code may depend on the fact that the argument is called 'string'. + + +Recommended names for common hashing algorithms +=============================================== + ++------------+------------+-------------------+ +| algorithm | variant | recommended name | ++============+============+===================+ +| MD5 | | md5 | ++------------+------------+-------------------+ +| RIPEMD-160 | | ripemd160 | ++------------+------------+-------------------+ +| SHA-1 | | sha1 | ++------------+------------+-------------------+ +| SHA-2 | SHA-224 | sha224 | ++ +------------+-------------------+ +| | SHA-256 | sha256 | ++ +------------+-------------------+ +| | SHA-384 | sha384 | ++ +------------+-------------------+ +| | SHA-512 | sha512 | ++------------+------------+-------------------+ +| SHA-3 | SHA-3-224 | sha3_224 | ++ +------------+-------------------+ +| | SHA-3-256 | sha3_256 | ++ +------------+-------------------+ +| | SHA-3-384 | sha3_384 | ++ +------------+-------------------+ +| | SHA-3-512 | sha3_512 | ++------------+------------+-------------------+ +| WHIRLPOOL | | whirlpool | ++------------+------------+-------------------+ + + +Changes +======= + +* 2001-09-17: Renamed ``clear()`` to ``reset()``; added ``digest_size`` attribute + to objects; added ``.hexdigest()`` method. +* 2001-09-20: Removed ``reset()`` method completely. +* 2001-09-28: Set ``digest_size`` to ``None`` for variable-size hashes. +* 2013-08-15: Added ``block_size`` and ``name`` attributes; clarified that + 'string' actually refers to bytes-like objects. + + +Acknowledgements +================ + +Thanks to Aahz, Andrew Archibald, Rich Salz, Itamar +Shtull-Trauring, and the readers of the python-crypto list for +their comments on this PEP. + + +Copyright +========= + +This document has been placed in the public domain. + + +.. _issue 18532: http://bugs.python.org/issue18532 diff --git a/decisiongrounding/scenarios_candidates/peps_hashlib_api_supersession/provenance.json b/decisiongrounding/scenarios_candidates/peps_hashlib_api_supersession/provenance.json new file mode 100644 index 0000000..14d22b4 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_hashlib_api_supersession/provenance.json @@ -0,0 +1,37 @@ +{ + "source_repo": "python/peps", + "pinned_commit": "f866e77409305866038471574f075cd8d83eee9e", + "peps": [ + { + "id": "PEP-0247", + "number": 247, + "file": "corpus/PEP-0247.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0247.rst", + "source_sha256": "04288431c34d7618f72847385cdbd23d2df6041df52a3d49aee27f6a2ca4a68b", + "status": "Final", + "title": "API for Cryptographic Hash Functions", + "replaces": [], + "superseded_by": [] + }, + { + "id": "PEP-0452", + "number": 452, + "file": "corpus/PEP-0452.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0452.rst", + "source_sha256": "b379a4b99e4873e0ada43eba019c3996d2fe72fb09eb3fffafe456c6154c85bc", + "status": "Final", + "title": "API for Cryptographic Hash Functions v2.0", + "replaces": [ + "PEP-0247" + ], + "superseded_by": [] + } + ], + "supersedes_edges": [ + { + "source": "PEP-0452", + "type": "supersedes", + "target": "PEP-0247" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/corpus/PEP-0241.md b/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/corpus/PEP-0241.md new file mode 100644 index 0000000..fdd498b --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/corpus/PEP-0241.md @@ -0,0 +1,290 @@ +--- +schema_version: 1 +id: PEP-0241 +type: decision +tags: [pep, packaging] +--- + +# PEP-0241 — Metadata for Python Software Packages + +## Status + +Superseded + +## Context + +Public decision ingested verbatim from python/peps (PEP 241) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0241.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0241.rst). Upstream +status: Superseded. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 241 +Title: Metadata for Python Software Packages +Author: A.M. Kuchling +Discussions-To: distutils-sig@python.org +Status: Superseded +Type: Standards Track +Topic: Packaging +Created: 12-Mar-2001 +Post-History: `19-Mar-2001 `__ +Superseded-By: 314 + +.. canonical-pypa-spec:: :ref:`packaging:core-metadata` + + +Introduction +============ + +This PEP describes a mechanism for adding metadata to Python +packages. It includes specifics of the field names, and their +semantics and usage. + + +Including Metadata in Packages +============================== + +The Distutils 'sdist' command will be modified to extract the +metadata fields from the arguments and write them to a file in the +generated zipfile or tarball. This file will be named PKG-INFO +and will be placed in the top directory of the source +distribution (where the README, INSTALL, and other files usually +go). + +Developers may not provide their own PKG-INFO file. The "sdist" +command will, if it detects an existing PKG-INFO file, terminate +with an appropriate error message. This should prevent confusion +caused by the PKG-INFO and setup.py files being out of sync. + +The PKG-INFO file format is a single set of :rfc:`822` headers +parseable by the rfc822.py module. The field names listed in the +following section are used as the header names. There's no +extension mechanism in this simple format; the Catalog and Distutils +SIGs will aim at getting a more flexible format ready for Python 2.2. + + +Fields +====== + +This section specifies the names and semantics of each of the +supported metadata fields. + +Fields marked with "(Multiple use)" may be specified multiple +times in a single PKG-INFO file. Other fields may only occur +once in a PKG-INFO file. Fields marked with "(optional)" are +not required to appear in a valid PKG-INFO file, all other +fields must be present. + +Metadata-Version +---------------- + +Version of the file format; currently "1.0" is the only +legal value here. + +Example:: + + Metadata-Version: 1.0 + +Name +---- + +The name of the package. + +Example:: + + Name: BeagleVote + +Version +------- + +A string containing the package's version number. This +field should be parseable by one of the Version classes +(StrictVersion or LooseVersion) in the distutils.version +module. + +Example:: + + Version: 1.0a2 + +Platform (multiple use) +----------------------- + +A comma-separated list of platform specifications, summarizing +the operating systems supported by the package. The major +supported platforms are listed below, but this list is +necessarily incomplete. + +:: + + POSIX, MacOS, Windows, BeOS, Palm OS. + +Binary distributions will use the Supported-Platform field in +their metadata to specify the OS and CPU for which the binary +package was compiled. The semantics of the Supported-Platform +are not specified in this PEP. + +Example:: + + Platform: POSIX, Windows + +Summary +------- + +A one-line summary of what the package does. + +Example:: + + Summary: A module for collecting votes from beagles. + +Description (optional) +---------------------- + +A longer description of the package that can run to several +paragraphs. (Software that deals with metadata should not +assume any maximum size for this field, though one hopes that +people won't include their instruction manual as the +long-description.) + +Example:: + + Description: This module collects votes from beagles + in order to determine their electoral wishes. + Do NOT try to use this module with basset hounds; + it makes them grumpy. + +Keywords (optional) +------------------- + +A list of additional keywords to be used to assist searching +for the package in a larger catalog. + +Example:: + + Keywords: dog puppy voting election + +Home-page (optional) +-------------------- + +A string containing the URL for the package's home page. + +Example:: + + Home-page: http://www.example.com/~cschultz/bvote/ + +Author (optional) +----------------- + +A string containing at a minimum the author's name. Contact +information can also be added, separating each line with +newlines. + +Example:: + + Author: C. Schultz + Universal Features Syndicate + Los Angeles, CA + +Author-email +------------ + +A string containing the author's e-mail address. It can contain +a name and e-mail address in the legal forms for a :rfc:`822` +'From:' header. It's not optional because cataloging systems +can use the e-mail portion of this field as a unique key +representing the author. A catalog might provide authors the +ability to store their GPG key, personal home page, and other +additional metadata *about the author*, and optionally the +ability to associate several e-mail addresses with the same +person. Author-related metadata fields are not covered by this +PEP. + +Example:: + + Author-email: "C. Schultz" + +License +------- + +A string selected from a short list of choices, specifying the +license covering the package. Some licenses result in the +software being freely redistributable, so packagers and +resellers can automatically know that they're free to +redistribute the software. Other licenses will require +a careful reading by a human to determine how the software can be +repackaged and resold. + +The choices are:: + + Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT", + Mozilla PL, "public domain", Python, Qt PL, Zope PL, unknown, + nocommercial, nosell, nosource, shareware, other + +Definitions of some of the licenses are: + +============= =================================================== +DFSG The license conforms to the Debian Free Software + Guidelines, but does not use one of the other + DFSG conforming licenses listed here. + More information is available at: + http://www.debian.org/social_contract#guidelines + +Python Python 1.6 or higher license. Version 1.5.2 and + earlier are under the MIT license. + +public domain Software is public domain, not copyrighted. + +unknown Status is not known + +nocommercial Free private use but commercial use not permitted + +nosell Free use but distribution for profit by arrangement + +nosource Freely distributable but no source code + +shareware Payment is requested if software is used + +other General category for other non-DFSG licenses +============= =================================================== + +Some of these licenses can be interpreted to mean the software is +freely redistributable. The list of redistributable licenses is:: + + Artistic, BSD, DFSG, GNU GPL, GNU LGPL, "MIT", + Mozilla PL, "public domain", Python, Qt PL, Zope PL, + nosource, shareware + +Note that being redistributable does not mean a package +qualifies as free software, 'nosource' and 'shareware' being +examples. + +Example:: + + License: MIT + + +Acknowledgements +================ + +Many changes and rewrites to this document were suggested by the +readers of the Distutils SIG. In particular, Sean Reifschneider +often contributed actual text for inclusion in this PEP. + +The list of licenses was compiled using the SourceForge license +list and the CTAN license list compiled by Graham Williams; Carey +Evans also offered several useful suggestions on this list. + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/corpus/PEP-0314.md b/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/corpus/PEP-0314.md new file mode 100644 index 0000000..6d193b9 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/corpus/PEP-0314.md @@ -0,0 +1,368 @@ +--- +schema_version: 1 +id: PEP-0314 +type: decision +tags: [pep, packaging] +--- + +# PEP-0314 — Metadata for Python Software Packages 1.1 + +## Status + +Superseded + +## Context + +Public decision ingested verbatim from python/peps (PEP 314) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0314.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0314.rst). Upstream +status: Superseded. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0241 + +## Source Text + +PEP: 314 +Title: Metadata for Python Software Packages 1.1 +Author: A.M. Kuchling, Richard Jones +Discussions-To: distutils-sig@python.org +Status: Superseded +Type: Standards Track +Topic: Packaging +Created: 12-Apr-2003 +Python-Version: 2.5 +Post-History: 29-Apr-2003 +Replaces: 241 +Superseded-By: 345 + +.. canonical-pypa-spec:: :ref:`packaging:core-metadata` + + +Introduction +============ + +This PEP describes a mechanism for adding metadata to Python +packages. It includes specifics of the field names, and their +semantics and usage. + +This document specifies version 1.1 of the metadata format. +Version 1.0 is specified in :pep:`241`. + + +Including Metadata in Packages +============================== + +The Distutils ``sdist`` command will extract the metadata fields +from the arguments and write them to a file in the generated +zipfile or tarball. This file will be named PKG-INFO and will be +placed in the top directory of the source distribution (where the +README, INSTALL, and other files usually go). + +Developers may not provide their own PKG-INFO file. The ``sdist`` +command will, if it detects an existing PKG-INFO file, terminate +with an appropriate error message. This should prevent confusion +caused by the PKG-INFO and setup.py files being out of sync. + +The PKG-INFO file format is a single set of :rfc:`822` headers +parseable by the rfc822.py module. The field names listed in the +following section are used as the header names. + + +Fields +====== + +This section specifies the names and semantics of each of the +supported metadata fields. + +Fields marked with "(Multiple use)" may be specified multiple +times in a single PKG-INFO file. Other fields may only occur +once in a PKG-INFO file. Fields marked with "(optional)" are +not required to appear in a valid PKG-INFO file; all other +fields must be present. + +Metadata-Version +---------------- + +Version of the file format; currently "1.0" and "1.1" are the +only legal values here. + +Example:: + + Metadata-Version: 1.1 + +Name +---- + +The name of the package. + +Example:: + + Name: BeagleVote + +Version +------- + +A string containing the package's version number. This +field should be parseable by one of the Version classes +(StrictVersion or LooseVersion) in the distutils.version +module. + +Example:: + + Version: 1.0a2 + +Platform (multiple use) +----------------------- + +A comma-separated list of platform specifications, summarizing +the operating systems supported by the package which are not +listed in the "Operating System" Trove classifiers. See +"Classifier" below. + +Example:: + + Platform: ObscureUnix, RareDOS + +Supported-Platform (multiple use) +--------------------------------- + +Binary distributions containing a PKG-INFO file will use the +Supported-Platform field in their metadata to specify the OS and +CPU for which the binary package was compiled. The semantics of +the Supported-Platform field are not specified in this PEP. + +Example:: + + Supported-Platform: RedHat 7.2 + Supported-Platform: i386-win32-2791 + +Summary +------- + +A one-line summary of what the package does. + +Example:: + + Summary: A module for collecting votes from beagles. + +Description (optional) +---------------------- + +A longer description of the package that can run to several +paragraphs. Software that deals with metadata should not assume +any maximum size for this field, though people shouldn't include +their instruction manual as the description. + +The contents of this field can be written using reStructuredText +markup [1]_. For programs that work with the metadata, +supporting markup is optional; programs can also display the +contents of the field as-is. This means that authors should be +conservative in the markup they use. + +Example:: + + Description: This module collects votes from beagles + in order to determine their electoral wishes. + Do *not* try to use this module with basset hounds; + it makes them grumpy. + +Keywords (optional) +------------------- + +A list of additional keywords to be used to assist searching +for the package in a larger catalog. + +Example:: + + Keywords: dog puppy voting election + +Home-page (optional) +-------------------- + +A string containing the URL for the package's home page. + +Example:: + + Home-page: http://www.example.com/~cschultz/bvote/ + +Download-URL +------------ + +A string containing the URL from which this version of the package +can be downloaded. (This means that the URL can't be something like +".../package-latest.tgz", but instead must be "../package-0.45.tgz".) + +Author (optional) +----------------- + +A string containing the author's name at a minimum; additional +contact information may be provided. + +Example:: + + Author: C. Schultz, Universal Features Syndicate, + Los Angeles, CA + +Author-email +------------ + +A string containing the author's e-mail address. It can contain +a name and e-mail address in the legal forms for a :rfc:`822` +'From:' header. It's not optional because cataloging systems +can use the e-mail portion of this field as a unique key +representing the author. A catalog might provide authors the +ability to store their GPG key, personal home page, and other +additional metadata *about the author*, and optionally the +ability to associate several e-mail addresses with the same +person. Author-related metadata fields are not covered by this +PEP. + +Example:: + + Author-email: "C. Schultz" + +License +------- + +Text indicating the license covering the package where the license +is not a selection from the "License" Trove classifiers. See +"Classifier" below. + +Example:: + + License: This software may only be obtained by sending the + author a postcard, and then the user promises not + to redistribute it. + +Classifier (multiple use) +------------------------- + +Each entry is a string giving a single classification value +for the package. Classifiers are described in :pep:`301`. + +Examples:: + + Classifier: Development Status :: 4 - Beta + Classifier: Environment :: Console (Text Based) + + +Requires (multiple use) +----------------------- + +Each entry contains a string describing some other module or +package required by this package. + +The format of a requirement string is identical to that of a +module or package name usable with the 'import' statement, +optionally followed by a version declaration within parentheses. + +A version declaration is a series of conditional operators and +version numbers, separated by commas. Conditional operators +must be one of "<", ">", "<=", ">=", "==", and "!=". Version +numbers must be in the format accepted by the +distutils.version.StrictVersion class: two or three +dot-separated numeric components, with an optional "pre-release" +tag on the end consisting of the letter 'a' or 'b' followed by a +number. Example version numbers are "1.0", "2.3a2", "1.3.99", + +Any number of conditional operators can be specified, e.g. +the string ">1.0, !=1.3.4, <2.0" is a legal version declaration. + +All of the following are possible requirement strings: "rfc822", +"zlib (>=1.1.4)", "zope". + +There's no canonical list of what strings should be used; the +Python community is left to choose its own standards. + +Example:: + + Requires: re + Requires: sys + Requires: zlib + Requires: xml.parsers.expat (>1.0) + Requires: psycopg + +Provides (multiple use) +----------------------- + +Each entry contains a string describing a package or module that +will be provided by this package once it is installed. These +strings should match the ones used in Requirements fields. A +version declaration may be supplied (without a comparison +operator); the package's version number will be implied if none +is specified. + +Example:: + + Provides: xml + Provides: xml.utils + Provides: xml.utils.iso8601 + Provides: xml.dom + Provides: xmltools (1.3) + +Obsoletes (multiple use) +------------------------ + +Each entry contains a string describing a package or module +that this package renders obsolete, meaning that the two packages +should not be installed at the same time. Version declarations +can be supplied. + +The most common use of this field will be in case a package name +changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0. +When you install Torqued Python, the Gorgon package should be +removed. + +Example:: + + Obsoletes: Gorgon + + +Summary of Differences From PEP 241 +=================================== + +* Metadata-Version is now 1.1. + +* Added the Classifiers field from :pep:`301`. + +* The License and Platform files should now only be used if the + platform or license can't be handled by an appropriate Classifier + value. + +* Added fields: Download-URL, Requires, Provides, Obsoletes. + + +Open issues +=========== + +None. + + +Acknowledgements +================ + +None. + + +References +========== + +.. [1] reStructuredText + http://docutils.sourceforge.net/ + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/corpus/PEP-0345.md b/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/corpus/PEP-0345.md new file mode 100644 index 0000000..917a7d7 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/corpus/PEP-0345.md @@ -0,0 +1,600 @@ +--- +schema_version: 1 +id: PEP-0345 +type: decision +tags: [pep, packaging] +--- + +# PEP-0345 — Metadata for Python Software Packages 1.2 + +## Status + +Superseded + +## Context + +Public decision ingested verbatim from python/peps (PEP 345) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0345.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0345.rst). Upstream +status: Superseded. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0314 + +## Source Text + +PEP: 345 +Title: Metadata for Python Software Packages 1.2 +Author: Richard Jones +Discussions-To: distutils-sig@python.org +Status: Superseded +Type: Standards Track +Topic: Packaging +Created: 28-Apr-2005 +Python-Version: 2.7 +Post-History: `22-Dec-2009 `__ +Replaces: 314 +Superseded-By: 566 +Resolution: https://mail.python.org/archives/list/python-dev@python.org/thread/MKHXVV746H7ZDFN62Z72VNAX6KIRXNRO/ + +.. canonical-pypa-spec:: :ref:`packaging:core-metadata` + + +Abstract +======== + +This PEP describes a mechanism for adding metadata to Python distributions. +It includes specifics of the field names, and their semantics and +usage. + +This document specifies version 1.2 of the metadata format. +Version 1.0 is specified in :pep:`241`. +Version 1.1 is specified in :pep:`314`. + +Version 1.2 of the metadata format adds a number of optional fields +designed to make third-party packaging of Python Software easier. +These fields are "Requires-Python", "Requires-External", "Requires-Dist", +"Provides-Dist", and "Obsoletes-Dist". This version also changes the +"Platform" field. Three new fields were also added: "Maintainer", +"Maintainer-email" and "Project-URL". + +Last, this new version also adds ``environment markers``. + +Fields +====== + +This section specifies the names and semantics of each of the +supported metadata fields. + +Fields marked with "(Multiple use)" may be specified multiple +times in a single PKG-INFO file. Other fields may only occur +once in a PKG-INFO file. Fields marked with "(optional)" are +not required to appear in a valid PKG-INFO file; all other +fields must be present. + +Metadata-Version +:::::::::::::::: + +Version of the file format; "1.2" is the only legal value. + +Example:: + + Metadata-Version: 1.2 + + +Name +:::: + +The name of the distributions. + +Example:: + + Name: BeagleVote + + +Version +::::::: + +A string containing the distribution's version number. This +field must be in the format specified in :pep:`440`. + +Example:: + + Version: 1.0a2 + + +Platform (multiple use) +::::::::::::::::::::::: + +A Platform specification describing an operating system supported by +the distribution which is not listed in the "Operating System" Trove classifiers. +See "Classifier" below. + +Examples:: + + Platform: ObscureUnix + Platform: RareDOS + + +Supported-Platform (multiple use) +::::::::::::::::::::::::::::::::: + +Binary distributions containing a PKG-INFO file will use the +Supported-Platform field in their metadata to specify the OS and +CPU for which the binary distribution was compiled. The semantics of +the Supported-Platform field are not specified in this PEP. + +Example:: + + Supported-Platform: RedHat 7.2 + Supported-Platform: i386-win32-2791 + + +Summary +::::::: + +A one-line summary of what the distribution does. + +Example:: + + Summary: A module for collecting votes from beagles. + + +Description (optional) +:::::::::::::::::::::: + +A longer description of the distribution that can run to several +paragraphs. Software that deals with metadata should not assume +any maximum size for this field, though people shouldn't include +their instruction manual as the description. + +The contents of this field can be written using reStructuredText +markup [1]_. For programs that work with the metadata, supporting +markup is optional; programs can also display the contents of the +field as-is. This means that authors should be conservative in +the markup they use. + +To support empty lines and lines with indentation with respect to +the :rfc:`822` format, any CRLF character has to be suffixed by 7 spaces +followed by a pipe ("|") char. As a result, the Description field is +encoded into a folded field that can be interpreted by :rfc:`822#section-3.1.1` +parser. + +Example:: + + Description: This project provides powerful math functions + |For example, you can use ``sum()`` to sum numbers: + | + |Example:: + | + | >>> sum(1, 2) + | 3 + | + +This encoding implies that any occurrences of a CRLF followed by 7 spaces +and a pipe char have to be replaced by a single CRLF when the field is unfolded +using a :rfc:`822` reader. + + +Keywords (optional) +::::::::::::::::::: + +A list of additional keywords to be used to assist searching +for the distribution in a larger catalog. + +Example:: + + Keywords: dog puppy voting election + + +Home-page (optional) +:::::::::::::::::::: + +A string containing the URL for the distribution's home page. + +Example:: + + Home-page: http://www.example.com/~cschultz/bvote/ + + +Download-URL +:::::::::::: + +A string containing the URL from which this version of the distribution +can be downloaded. (This means that the URL can't be something like +".../BeagleVote-latest.tgz", but instead must be ".../BeagleVote-0.45.tgz".) + + +Author (optional) +::::::::::::::::: + +A string containing the author's name at a minimum; additional +contact information may be provided. + +Example:: + + Author: C. Schultz, Universal Features Syndicate, + Los Angeles, CA + + +Author-email (optional) +::::::::::::::::::::::: + +A string containing the author's e-mail address. It can contain +a name and e-mail address in the legal forms for a :rfc:`822` +``From:`` header. + +Example:: + + Author-email: "C. Schultz" + + +Maintainer (optional) +::::::::::::::::::::: + +A string containing the maintainer's name at a minimum; additional +contact information may be provided. + +Note that this field is intended for use when a project is being +maintained by someone other than the original author: it should be +omitted if it is identical to ``Author``. + +Example:: + + Maintainer: C. Schultz, Universal Features Syndicate, + Los Angeles, CA + + +Maintainer-email (optional) +::::::::::::::::::::::::::: + +A string containing the maintainer's e-mail address. It can contain +a name and e-mail address in the legal forms for a :rfc:`822` +``From:`` header. + +Note that this field is intended for use when a project is being +maintained by someone other than the original author: it should be +omitted if it is identical to ``Author-email``. + +Example:: + + Maintainer-email: "C. Schultz" + + +License (optional) +:::::::::::::::::: + +Text indicating the license covering the distribution where the license +is not a selection from the "License" Trove classifiers. See +"Classifier" below. This field may also be used to specify a +particular version of a license which is named via the ``Classifier`` +field, or to indicate a variation or exception to such a license. + +Examples:: + + License: This software may only be obtained by sending the + author a postcard, and then the user promises not + to redistribute it. + + License: GPL version 3, excluding DRM provisions + + +Classifier (multiple use) +::::::::::::::::::::::::: + +Each entry is a string giving a single classification value +for the distribution. Classifiers are described in :pep:`301`. + +Examples:: + + Classifier: Development Status :: 4 - Beta + Classifier: Environment :: Console (Text Based) + + +Requires-Dist (multiple use) +:::::::::::::::::::::::::::: + +Each entry contains a string naming some other distutils +project required by this distribution. + +The format of a requirement string is identical to that of a +distutils project name (e.g., as found in the ``Name:`` field. +optionally followed by a version declaration within parentheses. + +The distutils project names should correspond to names as found +on the `Python Package Index`_. + +Version declarations must follow the rules described in +`Version Specifiers`_ + +Examples:: + + Requires-Dist: pkginfo + Requires-Dist: PasteDeploy + Requires-Dist: zope.interface (>3.5.0) + + +Provides-Dist (multiple use) +:::::::::::::::::::::::::::: + +Each entry contains a string naming a Distutils project which +is contained within this distribution. This field *must* include +the project identified in the ``Name`` field, followed by the +version : Name (Version). + +A distribution may provide additional names, e.g. to indicate that +multiple projects have been bundled together. For instance, source +distributions of the ``ZODB`` project have historically included +the ``transaction`` project, which is now available as a separate +distribution. Installing such a source distribution satisfies +requirements for both ``ZODB`` and ``transaction``. + +A distribution may also provide a "virtual" project name, which does +not correspond to any separately-distributed project: such a name +might be used to indicate an abstract capability which could be supplied +by one of multiple projects. E.g., multiple projects might supply +RDBMS bindings for use by a given ORM: each project might declare +that it provides ``ORM-bindings``, allowing other projects to depend +only on having at most one of them installed. + +A version declaration may be supplied and must follow the rules described +in `Version Specifiers`_. The distribution's version number will be implied +if none is specified. + +Examples:: + + Provides-Dist: OtherProject + Provides-Dist: AnotherProject (3.4) + Provides-Dist: virtual_package + + +Obsoletes-Dist (multiple use) +::::::::::::::::::::::::::::: + +Each entry contains a string describing a distutils project's distribution +which this distribution renders obsolete, meaning that the two projects +should not be installed at the same time. + +Version declarations can be supplied. Version numbers must be in the +format specified in `Version Specifiers`_. + +The most common use of this field will be in case a project name +changes, e.g. Gorgon 2.3 gets subsumed into Torqued Python 1.0. +When you install Torqued Python, the Gorgon distribution should be +removed. + +Examples:: + + Obsoletes-Dist: Gorgon + Obsoletes-Dist: OtherProject (<3.0) + + +Requires-Python +::::::::::::::: + +This field specifies the Python version(s) that the distribution is +guaranteed to be compatible with. + +Version numbers must be in the format specified in `Version Specifiers`_. + +Examples:: + + Requires-Python: 2.5 + Requires-Python: >2.1 + Requires-Python: >=2.3.4 + Requires-Python: >=2.5,<2.7 + + +Requires-External (multiple use) +:::::::::::::::::::::::::::::::: + +Each entry contains a string describing some dependency in the +system that the distribution is to be used. This field is intended to +serve as a hint to downstream project maintainers, and has no +semantics which are meaningful to the ``distutils`` distribution. + +The format of a requirement string is a name of an external +dependency, optionally followed by a version declaration within +parentheses. + +Because they refer to non-Python software releases, version numbers +for this field are **not** required to conform to the format +specified in :pep:`440`: they should correspond to the +version scheme used by the external dependency. + +Notice that there's is no particular rule on the strings to be used. + +Examples:: + + Requires-External: C + Requires-External: libpng (>=1.5) + + +Project-URL (multiple-use) +:::::::::::::::::::::::::: + +A string containing an extra URL for the project and a label for it, +separated by a comma. This should be used when there are other URLs +to list in the metadata in addition to the "Home-page" field. + +Examples:: + + Project-URL: Bug Tracker, https://github.com/pypa/setuptools/issues + Project-URL: Documentation, https://setuptools.readthedocs.io/ + Project-URL: Funding, https://donate.pypi.org + +The label is free text, with a maximum length of 32 characters. Notice +that distributions uploaded to PyPI will have these extra entries +displayed under the "Project links" section of their landing page. + + +Version Specifiers +================== + +Version specifiers are a series of conditional operators and +version numbers, separated by commas. Conditional operators +must be one of "<", ">", "<=", ">=", "==" and "!=". + +Any number of conditional operators can be specified, e.g. +the string ">1.0, !=1.3.4, <2.0" is a legal version declaration. +The comma (",") is equivalent to the **and** operator. + +Each version number must be in the format specified in :pep:`440`. + +When a version is provided, it always includes all versions that +starts with the same value. For example, the "2.5" version of Python +will include versions like "2.5.2" or "2.5.3". Pre and post releases +in that case are excluded. So in our example, versions like "2.5a1" are +not included when "2.5" is used. If the first version of the range is +required, it has to be explicitly given. In our example, it will be +"2.5.0". + +Notice that some projects might omit the ".0" suffix for the first release +of the "2.5.x" series: + +- 2.5 +- 2.5.1 +- 2.5.2 +- etc. + +In that case, "2.5.0" will have to be explicitly used to avoid any confusion +between the "2.5" notation that represents the full range. It is a recommended +practice to use schemes of the same length for a series to completely avoid +this problem. + +Some Examples: + +- ``Requires-Dist: zope.interface (3.1)``: any version that starts with 3.1, + excluding post or pre-releases. +- ``Requires-Dist: zope.interface (3.1.0)``: any version that starts with + 3.1.0, excluding post or pre-releases. Since that particular project doesn't + use more than 3 digits, it also means "only the 3.1.0 release". +- ``Requires-Python: 3``: Any Python 3 version, no matter which one, excluding + post or pre-releases. +- ``Requires-Python: >=2.6,<3``: Any version of Python 2.6 or 2.7, including + post releases of 2.6, pre and post releases of 2.7. It excludes pre releases + of Python 3. +- ``Requires-Python: 2.6.2``: Equivalent to ">=2.6.2,<2.6.3". So this includes + only Python 2.6.2. Of course, if Python was numbered with 4 digits, it would + have include all versions of the 2.6.2 series. +- ``Requires-Python: 2.5.0``: Equivalent to ">=2.5.0,<2.5.1". +- ``Requires-Dist: zope.interface (3.1,!=3.1.3)``: any version that starts with + 3.1, excluding post or pre-releases of 3.1 *and* excluding any version that + starts with "3.1.3". For this particular project, this means: "any version + of the 3.1 series but not 3.1.3". This is equivalent to: + ">=3.1,!=3.1.3,<3.2". + +Environment markers +=================== + +An **environment marker** is a marker that can be added at the end of a +field after a semi-colon (";"), to add a condition about the execution +environment. + +Here are some example of fields using such markers:: + + Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' + Obsoletes-Dist: pywin31; sys.platform == 'win32' + Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' + Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' + Requires-External: libxslt; 'linux' in sys.platform + +The micro-language behind this is the simplest possible: it compares only +strings, with the ``==`` and ``in`` operators (and their opposites), and +with the ability to combine expressions. It makes it also easy to understand +to non-pythoneers. + +The pseudo-grammar is :: + + EXPR [in|==|!=|not in] EXPR [or|and] ... + +where ``EXPR`` belongs to any of those: + +- python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) +- python_full_version = sys.version.split()[0] +- os.name = os.name +- sys.platform = sys.platform +- platform.version = platform.version() +- platform.machine = platform.machine() +- platform.python_implementation = platform.python_implementation() +- a free string, like ``'2.4'``, or ``'win32'`` + +Notice that ``in`` is restricted to strings, meaning that it is not possible +to use other sequences like tuples or lists on the right side. + +The fields that benefit from this marker are: + +- Requires-External +- Requires-Dist +- Provides-Dist +- Obsoletes-Dist +- Classifier + +Summary of Differences From PEP 314 +=================================== + +* Metadata-Version is now 1.2. + +* Added the environment markers. + +* Changed fields: + + - Platform (syntax change) + - Author-email (change to optional field) + +* Added fields: + + - Maintainer + - Maintainer-email + - Requires-Python + - Requires-External + - Requires-Dist + - Provides-Dist + - Obsoletes-Dist + - Project-URL + +* Deprecated fields: + + - Requires (in favor of Requires-Dist) + - Provides (in favor of Provides-Dist) + - Obsoletes (in favor of Obsoletes-Dist) + + +References +========== + +This document specifies version 1.2 of the metadata format. +Version 1.0 is specified in :pep:`241`. +Version 1.1 is specified in :pep:`314`. + +.. [1] reStructuredText markup: + http://docutils.sourceforge.net/ + +.. _`Python Package Index`: http://pypi.python.org/pypi/ + + +Copyright +========= + +This document has been placed in the public domain. + + +Acknowledgements +================ + +Fred Drake, Anthony Baxter and Matthias Klose have all contributed to +the ideas presented in this PEP. + +Tres Seaver, Jim Fulton, Marc-André Lemburg, Martin von Löwis, Tarek Ziadé, +David Lyon and other people at the Distutils-SIG have contributed to the +new updated version. diff --git a/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/provenance.json b/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/provenance.json new file mode 100644 index 0000000..58e8a22 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_metadata_v12_supersession/provenance.json @@ -0,0 +1,61 @@ +{ + "source_repo": "python/peps", + "pinned_commit": "f866e77409305866038471574f075cd8d83eee9e", + "peps": [ + { + "id": "PEP-0241", + "number": 241, + "file": "corpus/PEP-0241.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0241.rst", + "source_sha256": "baa232c6ba7d8166f74d8b0154de54a6abeb7c4b7e9448be2d620c3ee4cc64ac", + "status": "Superseded", + "title": "Metadata for Python Software Packages", + "replaces": [], + "superseded_by": [ + "PEP-0314" + ] + }, + { + "id": "PEP-0314", + "number": 314, + "file": "corpus/PEP-0314.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0314.rst", + "source_sha256": "587ad2e51525936ab79aec4066a1a50748456a967a9d510379153669d2701cfd", + "status": "Superseded", + "title": "Metadata for Python Software Packages 1.1", + "replaces": [ + "PEP-0241" + ], + "superseded_by": [ + "PEP-0345" + ] + }, + { + "id": "PEP-0345", + "number": 345, + "file": "corpus/PEP-0345.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0345.rst", + "source_sha256": "81a8b76d269e978c79d6ce8e133644756fe07be5f0f9ceade048f16c0476ffb2", + "status": "Superseded", + "title": "Metadata for Python Software Packages 1.2", + "replaces": [ + "PEP-0314" + ], + "superseded_by": [ + "PEP-0566" + ] + } + ], + "supersedes_edges": [ + { + "source": "PEP-0314", + "type": "supersedes", + "target": "PEP-0241" + }, + { + "source": "PEP-0345", + "type": "supersedes", + "target": "PEP-0314" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/peps_package_layout_supersession/corpus/PEP-0382.md b/decisiongrounding/scenarios_candidates/peps_package_layout_supersession/corpus/PEP-0382.md new file mode 100644 index 0000000..b231be3 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_package_layout_supersession/corpus/PEP-0382.md @@ -0,0 +1,245 @@ +--- +schema_version: 1 +id: PEP-0382 +type: decision +tags: [pep] +--- + +# PEP-0382 — Namespace Packages + +## Status + +Rejected + +## Context + +Public decision ingested verbatim from python/peps (PEP 382) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0382.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0382.rst). Upstream +status: Rejected. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 382 +Title: Namespace Packages +Author: Martin von Löwis +Status: Rejected +Type: Standards Track +Created: 02-Apr-2009 +Python-Version: 3.2 +Post-History: + +Rejection Notice +================ + +On the first day of sprints at US PyCon 2012 we had a long and +fruitful discussion about :pep:`382` and :pep:`402`. We ended up rejecting +both but a new PEP will be written to carry on in the spirit of PEP +402. Martin von Löwis wrote up a summary: [2]_. + +Abstract +======== + +Namespace packages are a mechanism for splitting a single Python +package across multiple directories on disk. In current Python +versions, an algorithm to compute the packages __path__ must be +formulated. With the enhancement proposed here, the import machinery +itself will construct the list of directories that make up the +package. An implementation of this PEP is available at [1]_. + +Terminology +=========== + +Within this PEP, the term package refers to Python packages as defined +by Python's import statement. The term distribution refers to +separately installable sets of Python modules as stored in the Python +package index, and installed by distutils or setuptools. The term +vendor package refers to groups of files installed by an operating +system's packaging mechanism (e.g. Debian or Redhat packages install +on Linux systems). + +The term portion refers to a set of files in a single directory (possibly +stored in a zip file) that contribute to a namespace package. + +Namespace packages today +======================== + +Python currently provides the pkgutil.extend_path to denote a package as +a namespace package. The recommended way of using it is to put:: + + from pkgutil import extend_path + __path__ = extend_path(__path__, __name__) + +in the package's ``__init__.py``. Every distribution needs to provide +the same contents in its ``__init__.py``, so that extend_path is +invoked independent of which portion of the package gets imported +first. As a consequence, the package's ``__init__.py`` cannot +practically define any names as it depends on the order of the package +fragments on sys.path which portion is imported first. As a special +feature, extend_path reads files named ``.pkg`` which +allow to declare additional portions. + +setuptools provides a similar function pkg_resources.declare_namespace +that is used in the form:: + + import pkg_resources + pkg_resources.declare_namespace(__name__) + +In the portion's __init__.py, no assignment to __path__ is necessary, +as declare_namespace modifies the package __path__ through sys.modules. +As a special feature, declare_namespace also supports zip files, and +registers the package name internally so that future additions to sys.path +by setuptools can properly add additional portions to each package. + +setuptools allows declaring namespace packages in a distribution's +setup.py, so that distribution developers don't need to put the +magic __path__ modification into __init__.py themselves. + +Rationale +========= + +The current imperative approach to namespace packages has lead to +multiple slightly-incompatible mechanisms for providing namespace +packages. For example, pkgutil supports ``*.pkg`` files; setuptools +doesn't. Likewise, setuptools supports inspecting zip files, and +supports adding portions to its _namespace_packages variable, whereas +pkgutil doesn't. + +In addition, the current approach causes problems for system vendors. +Vendor packages typically must not provide overlapping files, and an +attempt to install a vendor package that has a file already on disk +will fail or cause unpredictable behavior. As vendors might chose to +package distributions such that they will end up all in a single +directory for the namespace package, all portions would contribute +conflicting __init__.py files. + +Specification +============= + +Rather than using an imperative mechanism for importing packages, a +declarative approach is proposed here: A directory whose name ends +with ``.pyp`` (for Python package) contains a portion of a package. + +The import statement is extended so that computes the package's +``__path__`` attribute for a package named ``P`` as consisting of +optionally a single directory name ``P`` containing a file +``__init__.py``, plus all directories named ``P.pyp``, in the order in +which they are found in the parent's package ``__path__`` (or +``sys.path``). If either of these are found, search for additional +portions of the package continues. + +A directory may contain both a package in the ``P/__init__.py`` and +the ``P.pyp`` form. + +No other change to the importing mechanism is made; searching modules +(including __init__.py) will continue to stop at the first module +encountered. In summary, the process import a package foo works like +this: + +1. sys.path is searched for directories foo or foo.pyp, or a file foo.. + If a file is found and no directory, it is treated as a module, and imported. +2. If a directory foo is found, a check is made whether it contains __init__.py. + If so, the location of the __init__.py is remembered. Otherwise, the directory + is skipped. Once an __init__.py is found, further directories called foo are + skipped. +3. For both directories foo and foo.pyp, the directories are added to the package's + __path__. +4. If an __init__ module was found, it is imported, with __path__ + being initialized to the path computed all ``.pyp`` directories. + +Impact on Import Hooks +---------------------- + +Both loaders and finders as defined in :pep:`302` will need to be changed +to support namespace packages. Failure to conform to the protocol +below might cause a package not being recognized as a namespace +package; loaders and finders not supporting this protocol must raise +AttributeError when the functions below get accessed. + +Finders need to support looking for \*.pth files in step 1 of above +algorithm. To do so, a finder used as a path hook must support a +method: + + finder.find_package_portion(fullname) + +This method will be called in the same manner as find_module, and it +must return a string to be added to the package's ``__path__``. +If the finder doesn't find a portion of the package, it shall return +``None``. Raising ``AttributeError`` from above call will be treated +as non-conformance with this PEP, and the exception will be ignored. +All other exceptions are reported. + +A finder may report both success from ``find_module`` and from +``find_package_portion``, allowing for both a package containing +an ``__init__.py`` and a portion of the same package. + +All strings returned from ``find_package_portion``, along with all +path names of ``.pyp`` directories are added to the new package's +``__path__``. + +Discussion +========== + +Original versions of this specification proposed the addition of +``*.pth`` files, similar to the way those files are used on sys.path. +With a wildcard marker (``*``), a package could indicate that the +entire path is derived by looking at the parent path, searching for +properly-named subdirectories. + +People then observed that the support for the full .pth syntax is +inappropriate, and the .pth files were changed to be mere marker +files, indicating that a directories is a package. Peter Tröger +suggested that .pth is an unsuitable file extension, as all file +extensions related to Python should start with ``.py``. Therefore, the +marker file was renamed to be ``.pyp``. + +Dinu Gherman then observed that using a marker file is not necessary, +and that a directory extension could well serve as a such as a +marker. This is what this PEP currently proposes. + +Phillip Eby designed :pep:`402` as an alternative approach to this PEP, +after comparing Python's package syntax with that found in other +languages. :pep:`402` proposes not to use a marker file at all. At the +discussion at PyCon DE 2011, people remarked that having an explicit +declaration of a directory as contributing to a package is a desirable +property, rather than an obstacle. In particular, Jython developers +noticed that Jython could easily mistake a directory that is a Java +package as being a Python package, if there is no need to declare +Python packages. + +Packages can stop filling out the namespace package's __init__.py. As +a consequence, extend_path and declare_namespace become obsolete. + +Namespace packages can start providing non-trivial __init__.py +implementations; to do so, it is recommended that a single distribution +provides a portion with just the namespace package's __init__.py +(and potentially other modules that belong to the namespace package +proper). + +The mechanism is mostly compatible with the existing namespace +mechanisms. extend_path will be adjusted to this specification; +any other mechanism might cause portions to get added twice to +__path__. + +References +========== + +.. [1] PEP 382 branch + (http://hg.python.org/features/pep-382-2#pep-382) + +.. [2] Namespace Packages resolution + (https://mail.python.org/pipermail/import-sig/2012-March/000421.html) + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_package_layout_supersession/corpus/PEP-0402.md b/decisiongrounding/scenarios_candidates/peps_package_layout_supersession/corpus/PEP-0402.md new file mode 100644 index 0000000..f35a629 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_package_layout_supersession/corpus/PEP-0402.md @@ -0,0 +1,698 @@ +--- +schema_version: 1 +id: PEP-0402 +type: decision +tags: [pep, packaging] +--- + +# PEP-0402 — Simplified Package Layout and Partitioning + +## Status + +Rejected + +## Context + +Public decision ingested verbatim from python/peps (PEP 402) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0402.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0402.rst). Upstream +status: Rejected. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0382 + +## Source Text + +PEP: 402 +Title: Simplified Package Layout and Partitioning +Author: Phillip J. Eby +Status: Rejected +Type: Standards Track +Topic: Packaging +Created: 12-Jul-2011 +Python-Version: 3.3 +Post-History: 20-Jul-2011 +Replaces: 382 + +Rejection Notice +================ + +On the first day of sprints at US PyCon 2012 we had a long and +fruitful discussion about :pep:`382` and :pep:`402`. We ended up rejecting +both but a new PEP will be written to carry on in the spirit of PEP +402. Martin von Löwis wrote up a summary: [3]_. + +Abstract +======== + +This PEP proposes an enhancement to Python's package importing +to: + +* Surprise users of other languages less, +* Make it easier to convert a module into a package, and +* Support dividing packages into separately installed components + (ala "namespace packages", as described in :pep:`382`) + +The proposed enhancements do not change the semantics of any +currently-importable directory layouts, but make it possible for +packages to use a simplified directory layout (that is not importable +currently). + +However, the proposed changes do NOT add any performance overhead to +the importing of existing modules or packages, and performance for the +new directory layout should be about the same as that of previous +"namespace package" solutions (such as ``pkgutil.extend_path()``). + + +The Problem +=========== + +.. epigraph:: + + "Most packages are like modules. Their contents are highly + interdependent and can't be pulled apart. [However,] some + packages exist to provide a separate namespace. ... It should + be possible to distribute sub-packages or submodules of these + [namespace packages] independently." + + -- Jim Fulton, shortly before the release of Python 2.3 [1]_ + + +When new users come to Python from other languages, they are often +confused by Python's package import semantics. At Google, for example, +Guido received complaints from "a large crowd with pitchforks" [2]_ +that the requirement for packages to contain an ``__init__`` module +was a "misfeature", and should be dropped. + +In addition, users coming from languages like Java or Perl are +sometimes confused by a difference in Python's import path searching. + +In most other languages that have a similar path mechanism to Python's +``sys.path``, a package is merely a namespace that contains modules +or classes, and can thus be spread across multiple directories in +the language's path. In Perl, for instance, a ``Foo::Bar`` module +will be searched for in ``Foo/`` subdirectories all along the module +include path, not just in the first such subdirectory found. + +Worse, this is not just a problem for new users: it prevents *anyone* +from easily splitting a package into separately-installable +components. In Perl terms, it would be as if every possible ``Net::`` +module on CPAN had to be bundled up and shipped in a single tarball! + +For that reason, various workarounds for this latter limitation exist, +circulated under the term "namespace packages". The Python standard +library has provided one such workaround since Python 2.3 (via the +``pkgutil.extend_path()`` function), and the "setuptools" package +provides another (via ``pkg_resources.declare_namespace()``). + +The workarounds themselves, however, fall prey to a *third* issue with +Python's way of laying out packages in the filesystem. + +Because a package *must* contain an ``__init__`` module, any attempt +to distribute modules for that package must necessarily include that +``__init__`` module, if those modules are to be importable. + +However, the very fact that each distribution of modules for a package +must contain this (duplicated) ``__init__`` module, means that OS +vendors who package up these module distributions must somehow handle +the conflict caused by several module distributions installing that +``__init__`` module to the same location in the filesystem. + +This led to the proposing of :pep:`382` ("Namespace Packages") - a way +to signal to Python's import machinery that a directory was +importable, using unique filenames per module distribution. + +However, there was more than one downside to this approach. +Performance for all import operations would be affected, and the +process of designating a package became even more complex. New +terminology had to be invented to explain the solution, and so on. + +As terminology discussions continued on the Import-SIG, it soon became +apparent that the main reason it was so difficult to explain the +concepts related to "namespace packages" was because Python's +current way of handling packages is somewhat underpowered, when +compared to other languages. + +That is, in other popular languages with package systems, no special +term is needed to describe "namespace packages", because *all* +packages generally behave in the desired fashion. + +Rather than being an isolated single directory with a special marker +module (as in Python), packages in other languages are typically just +the *union* of appropriately-named directories across the *entire* +import or inclusion path. + +In Perl, for example, the module ``Foo`` is always found in a +``Foo.pm`` file, and a module ``Foo::Bar`` is always found in a +``Foo/Bar.pm`` file. (In other words, there is One Obvious Way to +find the location of a particular module.) + +This is because Perl considers a module to be *different* from a +package: the package is purely a *namespace* in which other modules +may reside, and is only *coincidentally* the name of a module as well. + +In current versions of Python, however, the module and the package are +more tightly bound together. ``Foo`` is always a module -- whether it +is found in ``Foo.py`` or ``Foo/__init__.py`` -- and it is tightly +linked to its submodules (if any), which *must* reside in the exact +same directory where the ``__init__.py`` was found. + +On the positive side, this design choice means that a package is quite +self-contained, and can be installed, copied, etc. as a unit just by +performing an operation on the package's root directory. + +On the negative side, however, it is non-intuitive for beginners, and +requires a more complex step to turn a module into a package. If +``Foo`` begins its life as ``Foo.py``, then it must be moved and +renamed to ``Foo/__init__.py``. + +Conversely, if you intend to create a ``Foo.Bar`` module from the +start, but have no particular module contents to put in ``Foo`` +itself, then you have to create an empty and seemingly-irrelevant +``Foo/__init__.py`` file, just so that ``Foo.Bar`` can be imported. + +(And these issues don't just confuse newcomers to the language, +either: they annoy many experienced developers as well.) + +So, after some discussion on the Import-SIG, this PEP was created +as an alternative to PEP \382, in an attempt to solve *all* of the +above problems, not just the "namespace package" use cases. + +And, as a delightful side effect, the solution proposed in this PEP +does not affect the import performance of ordinary modules or +self-contained (i.e. ``__init__``-based) packages. + + +The Solution +============ + +In the past, various proposals have been made to allow more intuitive +approaches to package directory layout. However, most of them failed +because of an apparent backward-compatibility problem. + +That is, if the requirement for an ``__init__`` module were simply +dropped, it would open up the possibility for a directory named, say, +``string`` on ``sys.path``, to block importing of the standard library +``string`` module. + +Paradoxically, however, the failure of this approach does *not* arise +from the elimination of the ``__init__`` requirement! + +Rather, the failure arises because the underlying approach takes for +granted that a package is just ONE thing, instead of two. + +In truth, a package comprises two separate, but related entities: a +module (with its own, optional contents), and a *namespace* where +*other* modules or packages can be found. + +In current versions of Python, however, the module part (found in +``__init__``) and the namespace for submodule imports (represented +by the ``__path__`` attribute) are both initialized at the same time, +when the package is first imported. + +And, if you assume this is the *only* way to initialize these two +things, then there is no way to drop the need for an ``__init__`` +module, while still being backwards-compatible with existing directory +layouts. + +After all, as soon as you encounter a directory on ``sys.path`` +matching the desired name, that means you've "found" the package, and +must stop searching, right? + +Well, not quite. + + +A Thought Experiment +-------------------- + +Let's hop into the time machine for a moment, and pretend we're back +in the early 1990s, shortly before Python packages and ``__init__.py`` +have been invented. But, imagine that we *are* familiar with +Perl-like package imports, and we want to implement a similar system +in Python. + +We'd still have Python's *module* imports to build on, so we could +certainly conceive of having ``Foo.py`` as a parent ``Foo`` module +for a ``Foo`` package. But how would we implement submodule and +subpackage imports? + +Well, if we didn't have the idea of ``__path__`` attributes yet, +we'd probably just search ``sys.path`` looking for ``Foo/Bar.py``. + +But we'd *only* do it when someone actually tried to *import* +``Foo.Bar``. + +NOT when they imported ``Foo``. + +And *that* lets us get rid of the backwards-compatibility problem +of dropping the ``__init__`` requirement, back here in 2011. + +How? + +Well, when we ``import Foo``, we're not even *looking* for ``Foo/`` +directories on ``sys.path``, because we don't *care* yet. The only +point at which we care, is the point when somebody tries to actually +import a submodule or subpackage of ``Foo``. + +That means that if ``Foo`` is a standard library module (for example), +and I happen to have a ``Foo`` directory on ``sys.path`` (without +an ``__init__.py``, of course), then *nothing breaks*. The ``Foo`` +module is still just a module, and it's still imported normally. + + +Self-Contained vs. "Virtual" Packages +------------------------------------- + +Of course, in today's Python, trying to ``import Foo.Bar`` will +fail if ``Foo`` is just a ``Foo.py`` module (and thus lacks a +``__path__`` attribute). + +So, this PEP proposes to *dynamically* create a ``__path__``, in the +case where one is missing. + +That is, if I try to ``import Foo.Bar`` the proposed change to the +import machinery will notice that the ``Foo`` module lacks a +``__path__``, and will therefore try to *build* one before proceeding. + +And it will do this by making a list of all the existing ``Foo/`` +subdirectories of the directories listed in ``sys.path``. + +If the list is empty, the import will fail with ``ImportError``, just +like today. But if the list is *not* empty, then it is saved in +a new ``Foo.__path__`` attribute, making the module a "virtual +package". + +That is, because it now has a valid ``__path__``, we can proceed +to import submodules or subpackages in the normal way. + +Now, notice that this change does not affect "classic", self-contained +packages that have an ``__init__`` module in them. Such packages +already *have* a ``__path__`` attribute (initialized at import time) +so the import machinery won't try to create another one later. + +This means that (for example) the standard library ``email`` package +will not be affected in any way by you having a bunch of unrelated +directories named ``email`` on ``sys.path``. (Even if they contain +``*.py`` files.) + +But it *does* mean that if you want to turn your ``Foo`` module into +a ``Foo`` package, all you have to do is add a ``Foo/`` directory +somewhere on ``sys.path``, and start adding modules to it. + +But what if you only want a "namespace package"? That is, a package +that is *only* a namespace for various separately-distributed +submodules and subpackages? + +For example, if you're Zope Corporation, distributing dozens of +separate tools like ``zc.buildout``, each in packages under the ``zc`` +namespace, you don't want to have to make and include an empty +``zc.py`` in every tool you ship. (And, if you're a Linux or other +OS vendor, you don't want to deal with the package installation +conflicts created by trying to install ten copies of ``zc.py`` to the +same location!) + +No problem. All we have to do is make one more minor tweak to the +import process: if the "classic" import process fails to find a +self-contained module or package (e.g., if ``import zc`` fails to find +a ``zc.py`` or ``zc/__init__.py``), then we once more try to build a +``__path__`` by searching for all the ``zc/`` directories on +``sys.path``, and putting them in a list. + +If this list is empty, we raise ``ImportError``. But if it's +non-empty, we create an empty ``zc`` module, and put the list in +``zc.__path__``. Congratulations: ``zc`` is now a namespace-only, +"pure virtual" package! It has no module contents, but you can still +import submodules and subpackages from it, regardless of where they're +located on ``sys.path``. + +(By the way, both of these additions to the import protocol (i.e. the +dynamically-added ``__path__``, and dynamically-created modules) +apply recursively to child packages, using the parent package's +``__path__`` in place of ``sys.path`` as a basis for generating a +child ``__path__``. This means that self-contained and virtual +packages can contain each other without limitation, with the caveat +that if you put a virtual package inside a self-contained one, it's +gonna have a really short ``__path__``!) + + +Backwards Compatibility and Performance +--------------------------------------- + +Notice that these two changes *only* affect import operations that +today would result in ``ImportError``. As a result, the performance +of imports that do not involve virtual packages is unaffected, and +potential backward compatibility issues are very restricted. + +Today, if you try to import submodules or subpackages from a module +with no ``__path__``, it's an immediate error. And of course, if you +don't have a ``zc.py`` or ``zc/__init__.py`` somewhere on ``sys.path`` +today, ``import zc`` would likewise fail. + +Thus, the only potential backwards-compatibility issues are: + +1. Tools that expect package directories to have an ``__init__`` + module, that expect directories without an ``__init__`` module + to be unimportable, or that expect ``__path__`` attributes to be + static, will not recognize virtual packages as packages. + + (In practice, this just means that tools will need updating to + support virtual packages, e.g. by using ``pkgutil.walk_modules()`` + instead of using hardcoded filesystem searches.) + +2. Code that *expects* certain imports to fail may now do something + unexpected. This should be fairly rare in practice, as most sane, + non-test code does not import things that are expected not to + exist! + +The biggest likely exception to the above would be when a piece of +code tries to check whether some package is installed by importing +it. If this is done *only* by importing a top-level module (i.e., not +checking for a ``__version__`` or some other attribute), *and* there +is a directory of the same name as the sought-for package on +``sys.path`` somewhere, *and* the package is not actually installed, +then such code could be fooled into thinking a package is installed +that really isn't. + +For example, suppose someone writes a script (``datagen.py``) +containing the following code:: + + try: + import json + except ImportError: + import simplejson as json + +And runs it in a directory laid out like this:: + + datagen.py + json/ + foo.js + bar.js + +If ``import json`` succeeded due to the mere presence of the ``json/`` +subdirectory, the code would incorrectly believe that the ``json`` +module was available, and proceed to fail with an error. + +However, we can prevent corner cases like these from arising, simply +by making one small change to the algorithm presented so far. Instead +of allowing you to import a "pure virtual" package (like ``zc``), +we allow only importing of the *contents* of virtual packages. + +That is, a statement like ``import zc`` should raise ``ImportError`` +if there is no ``zc.py`` or ``zc/__init__.py`` on ``sys.path``. But, +doing ``import zc.buildout`` should still succeed, as long as there's +a ``zc/buildout.py`` or ``zc/buildout/__init__.py`` on ``sys.path``. + +In other words, we don't allow pure virtual packages to be imported +directly, only modules and self-contained packages. (This is an +acceptable limitation, because there is no *functional* value to +importing such a package by itself. After all, the module object +will have no *contents* until you import at least one of its +subpackages or submodules!) + +Once ``zc.buildout`` has been successfully imported, though, there +*will* be a ``zc`` module in ``sys.modules``, and trying to import it +will of course succeed. We are only preventing an *initial* import +from succeeding, in order to prevent false-positive import successes +when clashing subdirectories are present on ``sys.path``. + +So, with this slight change, the ``datagen.py`` example above will +work correctly. When it does ``import json``, the mere presence of a +``json/`` directory will simply not affect the import process at all, +even if it contains ``.py`` files. The ``json/`` directory will still +only be searched in the case where an import like ``import +json.converter`` is attempted. + +Meanwhile, tools that expect to locate packages and modules by +walking a directory tree can be updated to use the existing +``pkgutil.walk_modules()`` API, and tools that need to inspect +packages in memory should use the other APIs described in the +`Standard Library Changes/Additions`_ section below. + + +Specification +============= + +A change is made to the existing import process, when importing +names containing at least one ``.`` -- that is, imports of modules +that have a parent package. + +Specifically, if the parent package does not exist, or exists but +lacks a ``__path__`` attribute, an attempt is first made to create a +"virtual path" for the parent package (following the algorithm +described in the section on `virtual paths`_, below). + +If the computed "virtual path" is empty, an ``ImportError`` results, +just as it would today. However, if a non-empty virtual path is +obtained, the normal import of the submodule or subpackage proceeds, +using that virtual path to find the submodule or subpackage. (Just +as it would have with the parent's ``__path__``, if the parent package +had existed and had a ``__path__``.) + +When a submodule or subpackage is found (but not yet loaded), +the parent package is created and added to ``sys.modules`` (if it +didn't exist before), and its ``__path__`` is set to the computed +virtual path (if it wasn't already set). + +In this way, when the actual loading of the submodule or subpackage +occurs, it will see a parent package existing, and any relative +imports will work correctly. However, if no submodule or subpackage +exists, then the parent package will *not* be created, nor will a +standalone module be converted into a package (by the addition of a +spurious ``__path__`` attribute). + +Note, by the way, that this change must be applied *recursively*: that +is, if ``foo`` and ``foo.bar`` are pure virtual packages, then +``import foo.bar.baz`` must wait until ``foo.bar.baz`` is found before +creating module objects for *both* ``foo`` and ``foo.bar``, and then +create both of them together, properly setting the ``foo`` module's +``.bar`` attribute to point to the ``foo.bar`` module. + +In this way, pure virtual packages are never directly importable: +an ``import foo`` or ``import foo.bar`` by itself will fail, and the +corresponding modules will not appear in ``sys.modules`` until they +are needed to point to a *successfully* imported submodule or +self-contained subpackage. + + +Virtual Paths +------------- + +A virtual path is created by obtaining a :pep:`302` "importer" object for +each of the path entries found in ``sys.path`` (for a top-level +module) or the parent ``__path__`` (for a submodule). + +(Note: because ``sys.meta_path`` importers are not associated with +``sys.path`` or ``__path__`` entry strings, such importers do *not* +participate in this process.) + +Each importer is checked for a ``get_subpath()`` method, and if +present, the method is called with the full name of the module/package +the path is being constructed for. The return value is either a +string representing a subdirectory for the requested package, or +``None`` if no such subdirectory exists. + +The strings returned by the importers are added to the path list +being built, in the same order as they are found. (``None`` values +and missing ``get_subpath()`` methods are simply skipped.) + +The resulting list (whether empty or not) is then stored in a +``sys.virtual_package_paths`` dictionary, keyed by module name. + +This dictionary has two purposes. First, it serves as a cache, in +the event that more than one attempt is made to import a submodule +of a virtual package. + +Second, and more importantly, the dictionary can be used by code that +extends ``sys.path`` at runtime to *update* imported packages' +``__path__`` attributes accordingly. (See `Standard Library +Changes/Additions`_ below for more details.) + +In Python code, the virtual path construction algorithm would look +something like this:: + + def get_virtual_path(modulename, parent_path=None): + + if modulename in sys.virtual_package_paths: + return sys.virtual_package_paths[modulename] + + if parent_path is None: + parent_path = sys.path + + path = [] + + for entry in parent_path: + # Obtain a PEP 302 importer object - see pkgutil module + importer = pkgutil.get_importer(entry) + + if hasattr(importer, 'get_subpath'): + subpath = importer.get_subpath(modulename) + if subpath is not None: + path.append(subpath) + + sys.virtual_package_paths[modulename] = path + return path + +And a function like this one should be exposed in the standard +library as e.g. ``imp.get_virtual_path()``, so that people creating +``__import__`` replacements or ``sys.meta_path`` hooks can reuse it. + + +Standard Library Changes/Additions +---------------------------------- + +The ``pkgutil`` module should be updated to handle this +specification appropriately, including any necessary changes to +``extend_path()``, ``iter_modules()``, etc. + +Specifically the proposed changes and additions to ``pkgutil`` are: + +* A new ``extend_virtual_paths(path_entry)`` function, to extend + existing, already-imported virtual packages' ``__path__`` attributes + to include any portions found in a new ``sys.path`` entry. This + function should be called by applications extending ``sys.path`` + at runtime, e.g. when adding a plugin directory or an egg to the + path. + + The implementation of this function does a simple top-down traversal + of ``sys.virtual_package_paths``, and performs any necessary + ``get_subpath()`` calls to identify what path entries need to be + added to the virtual path for that package, given that ``path_entry`` + has been added to ``sys.path``. (Or, in the case of sub-packages, + adding a derived subpath entry, based on their parent package's + virtual path.) + + (Note: this function must update both the path values in + ``sys.virtual_package_paths`` as well as the ``__path__`` attributes + of any corresponding modules in ``sys.modules``, even though in the + common case they will both be the same ``list`` object.) + +* A new ``iter_virtual_packages(parent='')`` function to allow + top-down traversal of virtual packages from + ``sys.virtual_package_paths``, by yielding the child virtual + packages of ``parent``. For example, calling + ``iter_virtual_packages("zope")`` might yield ``zope.app`` + and ``zope.products`` (if they are virtual packages listed in + ``sys.virtual_package_paths``), but **not** ``zope.foo.bar``. + (This function is needed to implement ``extend_virtual_paths()``, + but is also potentially useful for other code that needs to inspect + imported virtual packages.) + +* ``ImpImporter.iter_modules()`` should be changed to also detect and + yield the names of modules found in virtual packages. + +In addition to the above changes, the ``zipimport`` importer should +have its ``iter_modules()`` implementation similarly changed. (Note: +current versions of Python implement this via a shim in ``pkgutil``, +so technically this is also a change to ``pkgutil``.) + +Last, but not least, the ``imp`` module (or ``importlib``, if +appropriate) should expose the algorithm described in the `virtual +paths`_ section above, as a +``get_virtual_path(modulename, parent_path=None)`` function, so that +creators of ``__import__`` replacements can use it. + + +Implementation Notes +-------------------- + +For users, developers, and distributors of virtual packages: + +* While virtual packages are easy to set up and use, there is still + a time and place for using self-contained packages. While it's not + strictly necessary, adding an ``__init__`` module to your + self-contained packages lets users of the package (and Python + itself) know that *all* of the package's code will be found in + that single subdirectory. In addition, it lets you define + ``__all__``, expose a public API, provide a package-level docstring, + and do other things that make more sense for a self-contained + project than for a mere "namespace" package. + +* ``sys.virtual_package_paths`` is allowed to contain entries for + non-existent or not-yet-imported package names; code that uses its + contents should not assume that every key in this dictionary is also + present in ``sys.modules`` or that importing the name will + necessarily succeed. + +* If you are changing a currently self-contained package into a + virtual one, it's important to note that you can no longer use its + ``__file__`` attribute to locate data files stored in a package + directory. Instead, you must search ``__path__`` or use the + ``__file__`` of a submodule adjacent to the desired files, or + of a self-contained subpackage that contains the desired files. + + (Note: this caveat is already true for existing users of "namespace + packages" today. That is, it is an inherent result of being able + to partition a package, that you must know *which* partition the + desired data file lives in. We mention it here simply so that + *new* users converting from self-contained to virtual packages will + also be aware of it.) + +* XXX what is the __file__ of a "pure virtual" package? ``None``? + Some arbitrary string? The path of the first directory with a + trailing separator? No matter what we put, *some* code is + going to break, but the last choice might allow some code to + accidentally work. Is that good or bad? + + +For those implementing :pep:`302` importer objects: + +* Importers that support the ``iter_modules()`` method (used by + ``pkgutil`` to locate importable modules and packages) and want to + add virtual package support should modify their ``iter_modules()`` + method so that it discovers and lists virtual packages as well as + standard modules and packages. To do this, the importer should + simply list all immediate subdirectory names in its jurisdiction + that are valid Python identifiers. + + XXX This might list a lot of not-really-packages. Should we + require importable contents to exist? If so, how deep do we + search, and how do we prevent e.g. link loops, or traversing onto + different filesystems, etc.? Ick. Also, if virtual packages are + listed, they still can't be *imported*, which is a problem for the + way that ``pkgutil.walk_modules()`` is currently implemented. + +* "Meta" importers (i.e., importers placed on ``sys.meta_path``) do + not need to implement ``get_subpath()``, because the method + is only called on importers corresponding to ``sys.path`` entries + and ``__path__`` entries. If a meta importer wishes to support + virtual packages, it must do so entirely within its own + ``find_module()`` implementation. + + Unfortunately, it is unlikely that any such implementation will be + able to merge its package subpaths with those of other meta + importers or ``sys.path`` importers, so the meaning of "supporting + virtual packages" for a meta importer is currently undefined! + + (However, since the intended use case for meta importers is to + replace Python's normal import process entirely for some subset of + modules, and the number of such importers currently implemented is + quite small, this seems unlikely to be a big issue in practice.) + + +References +========== + +.. [1] "namespace" vs "module" packages (mailing list thread) + (http://mail.zope.org/pipermail/zope3-dev/2002-December/004251.html) + +.. [2] "Dropping __init__.py requirement for subpackages" + (https://mail.python.org/pipermail/python-dev/2006-April/064400.html) + +.. [3] Namespace Packages resolution + (https://mail.python.org/pipermail/import-sig/2012-March/000421.html) + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_package_layout_supersession/provenance.json b/decisiongrounding/scenarios_candidates/peps_package_layout_supersession/provenance.json new file mode 100644 index 0000000..29ce503 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_package_layout_supersession/provenance.json @@ -0,0 +1,37 @@ +{ + "source_repo": "python/peps", + "pinned_commit": "f866e77409305866038471574f075cd8d83eee9e", + "peps": [ + { + "id": "PEP-0382", + "number": 382, + "file": "corpus/PEP-0382.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0382.rst", + "source_sha256": "80b800fa8b1c33db9fa412d597ee5b74fae64702aa0d5957ab50b49e91039053", + "status": "Rejected", + "title": "Namespace Packages", + "replaces": [], + "superseded_by": [] + }, + { + "id": "PEP-0402", + "number": 402, + "file": "corpus/PEP-0402.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0402.rst", + "source_sha256": "2b19230ee35655f414f08a5222597833d1e9c3259a1f67e9233e740027b2ced1", + "status": "Rejected", + "title": "Simplified Package Layout and Partitioning", + "replaces": [ + "PEP-0382" + ], + "superseded_by": [] + } + ], + "supersedes_edges": [ + { + "source": "PEP-0402", + "type": "supersedes", + "target": "PEP-0382" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/peps_packaging_governance_supersession/corpus/PEP-0609.md b/decisiongrounding/scenarios_candidates/peps_packaging_governance_supersession/corpus/PEP-0609.md new file mode 100644 index 0000000..95dbf49 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_packaging_governance_supersession/corpus/PEP-0609.md @@ -0,0 +1,304 @@ +--- +schema_version: 1 +id: PEP-0609 +type: decision +tags: [pep, governance, packaging] +--- + +# PEP-0609 — Python Packaging Authority (PyPA) Governance + +## Status + +Active + +## Context + +Public decision ingested verbatim from python/peps (PEP 609) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0609.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0609.rst). Upstream +status: Active. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 609 +Title: Python Packaging Authority (PyPA) Governance +Author: Dustin Ingram , + Pradyun Gedam , + Sumana Harihareswara +Sponsor: Paul Ganssle +Discussions-To: https://discuss.python.org/t/pep-609-pypa-governance/2619 +Status: Active +Type: Process +Topic: Governance, Packaging +Created: 05-Nov-2019 +Post-History: 05-Nov-2019 + + +Abstract +======== + +This document describes a governance model for the Python Packaging +Authority (PyPA). The model is closely based on existing informal +practices, with the intent of providing clarity into the functioning +of the PyPA and formalizing transparent processes for the PyPA. + +Rationale +========= + +The Python Packaging Authority (PyPA) is a collaborative community +that maintains and advances many of the relevant projects in Python +packaging. The software and standards developed through the PyPA are +used to package, share, and install Python software and to interact +with indexes of downloadable Python software such as `PyPI`_, the +Python Package Index. + +Currently, the PyPA is an informal and loosely defined organization +that lacks true authority, and the inclusion of a given project under +the PyPA umbrella or the creation of new projects has been done in an +ad-hoc, one-off manner. Similarly, individual membership in the PyPA +is not well-defined. + +While this model has more or less worked for the PyPA in the past, it +results in an organization which is missing certain features of a +stable ecosystem, namely a clear and transparent decision-making +process. This PEP seeks to rectify this by defining a governance model +for the PyPA. + +Terminology +=========== + +Relevant terms for groups of individual contributors used in this PEP: + +**PyPA members**: + Anyone with the triage bit or commit bit, on at least one project in + the `PyPA organization`_. + +**PyPA committers**: + Anyone with the commit bit on at least one project in the PyPA + organization, which should correspond to everyone on the + `PyPA-Committers`_ mailing list. + +**PyPA community**: + Anyone who is interested in PyPA activity and wants to follow along, + contribute or make proposals. + +**Packaging-WG members**: + As described in the `Packaging-WG Wiki page`_. For clarity: there is no + formal relationship between the Packaging-WG and PyPA. This group is + only included in this list to disambiguate it from PyPA. + +Goals +===== + +The following section formalizes the goals (and non-goals) of the PyPA +and this governance model. + +Goals of the PyPA +----------------- + +These goals are the primary motivation for the existence of the PyPA. +These goals are largely already being carried out, even though most +have not been explicitly defined. + +Provide support for existing projects under the PyPA +'''''''''''''''''''''''''''''''''''''''''''''''''''' + +In the event that a given project needs additional support, or no +longer has active maintainers, the PyPA will ensure that the given +project will continue to be supported for users to the extent +necessary. + +Foster the creation and acceptance of standards for PyPA projects +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +The PyPA should, as much as possible, strive for standardization and +coordination across PyPA projects, primarily though the governance +process outlined below. PyPA projects are expected to abide by +applicable specifications maintained by the PyPA. + +Guide decisions which affect multiple PyPA projects +''''''''''''''''''''''''''''''''''''''''''''''''''' + +The PyPA community (especially PyPA members) should be expected to +provide opinions, insight and experience when ecosystem-wide changes +are being proposed. + +Determine which projects should be under the guidance of the PyPA +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +For example: accepting new projects from the community, organically +creating projects within the PyPA, etc. + +Enforce adherence to a Code of Conduct across all projects +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Generally this means leading by example, but occasionally it may mean +more explicit moderation. + +Non-goals of the PyPA +--------------------- + +These are specific items that are explicitly _not_ goals of the PyPA. + +Determine who is and isn't a PyPA member +'''''''''''''''''''''''''''''''''''''''' + +This is for members of individual projects to decide, as they add new +members to their projects. Maintainership of a project that is under +the PyPA organization automatically transfers membership in the PyPA. + +Micromanage individual projects +''''''''''''''''''''''''''''''' + +As long as the project is adhering to the Code of Conduct and +following specifications supported by the PyPA, the PyPA should only +concerned with large, ecosystem-wide changes. + +Develop and maintain standalone Code of Conduct +''''''''''''''''''''''''''''''''''''''''''''''' + +PyPA projects follow `the PSF Code of Conduct`_. + + +Goals of the PyPA's Governance Model +------------------------------------ + +These are new goals which the governance model seeks to make possible. + +Transparency in PyPA membership +''''''''''''''''''''''''''''''' + +Provide a transparent process for decisions taken, regarding project +membership in the PyPA. + +Document PyPA's use of PEPs +''''''''''''''''''''''''''' + +Formally document how the PyPA uses Python Enhancement Proposals +(PEPs), for maintaining interoperability specifications defined by the +PyPA. + +Processes +========= + +The processes for the PyPA's activities are outlined below: + +Specifications +-------------- + +The PyPA will use PEPs for defining, and making changes to, the +interoperability specifications maintained by the PyPA. Thus, the +Python Steering Council has the final say in the acceptance of these +interoperability specifications. + +It is expected (but not required) that the Python Steering Council +would delegate authority to sponsor and/or approve/reject PEPs related +to packaging interoperability specifications, to individuals within the +PyPA community. At the time of writing (June 2020), the Python Steering +Council has `standing delegations`_ for currently active packaging +interoperability specifications. + +The details of the process of proposing and updating the +interoperability specifications are described in the `PyPA +Specifications`_ document. + +Governance +---------- + +PyPA Committer Votes +'''''''''''''''''''' + +A PyPA member can put forward a proposal and call for a vote on a +public PyPA communication channel. A PyPA committer vote is triggered +when a PyPA committer (not the proposer) seconds the proposal. + +The proposal will be put to a vote on the `PyPA-Committers`_ mailing +list, over a 7-day period. Each PyPA committer can vote once, and can +choose one of ``+1`` and ``-1``. If at least two thirds of recorded +votes are ``+1``, then the vote succeeds. + +PyPA committer votes are required for, and limited to, the following +kinds of proposals: + +Addition of a project to the PyPA +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Proposing the acceptance of a project into the PyPA organization. This +proposal must not be opposed by the existing maintainers of the +project. + +Creation of a new project in the PyPA +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Proposing the creation of a new tools / project in the PyPA +organization. + +Removal of a project from PyPA +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Proposing the removal of a project in the PyPA organization. + +Updates to the Governance/Specification Processes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Proposing changes to how the PyPA operates, including but not limited +to changes to its specification and governance processes, and this +PEP. + +Leaving PyPA +'''''''''''' + +A project that is a part of the PyPA organization, can request to +leave PyPA. + +Such requests can made by a committer of the project, on the +`PyPA-Committers`_ mailing list and must clearly state the GitHub +user/organization to transfer the repository to. + +If the request is not opposed by another committer of the same project +over a 7-day period, the project would leave the PyPA and be +transferred out of the PyPA organization as per the request. + +Code of Conduct enforcement +''''''''''''''''''''''''''' + +Each project that is a part of the PyPA organization follows `the PSF +Code of Conduct`_, including its incident reporting guidelines and +enforcement procedures. + +PyPA members are responsible for leading by example. PyPA members +occasionally may need to more explicitly moderate behavior in their +projects, and each project that is a part of the PyPA organization +must designate at least one PyPA member as available to contact in +case of a Code of Conduct incident. If told of any Code of Conduct +incidents involving their projects, PyPA members are expected to +report those incidents up to `the PSF Conduct WG`_, for recording +purposes and for potential assistance. + + +References +========== + +.. _PyPI: https://pypi.org +.. _PyPA Organization: https://github.com/pypa +.. _PyPA Specifications: https://www.pypa.io/en/latest/specifications +.. _PyPA-Committers: https://mail.python.org/mm3/mailman3/lists/pypa-committers.python.org/ +.. _Packaging-WG Wiki page: https://wiki.python.org/psf/PackagingWG +.. _standing delegations: https://github.com/python/steering-council/blob/master/process/standing-delegations.md#pypa-delegations +.. _the PSF Code of Conduct: https://www.python.org/psf/conduct/ +.. _the PSF Conduct WG: https://wiki.python.org/psf/ConductWG/Charter + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. diff --git a/decisiongrounding/scenarios_candidates/peps_packaging_governance_supersession/corpus/PEP-0772.md b/decisiongrounding/scenarios_candidates/peps_packaging_governance_supersession/corpus/PEP-0772.md new file mode 100644 index 0000000..cbfe7fe --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_packaging_governance_supersession/corpus/PEP-0772.md @@ -0,0 +1,576 @@ +--- +schema_version: 1 +id: PEP-0772 +type: decision +tags: [pep, governance, packaging] +--- + +# PEP-0772 — Packaging Council governance process + +## Status + +Accepted + +## Context + +Public decision ingested verbatim from python/peps (PEP 772) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0772.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0772.rst). Upstream +status: Accepted. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0609 + +## Source Text + +PEP: 772 +Title: Packaging Council governance process +Author: Barry Warsaw , + Deb Nicholson , + Pradyun Gedam +Discussions-To: https://discuss.python.org/t/pep-772-packaging-council-governance-process-round-3/100181 +Status: Accepted +Type: Process +Topic: Governance, Packaging +Created: 21-Jan-2025 +Post-History: + `06-Feb-2025 `__, + `30-May-2025 `__, + `25-Jul-2025 `__, + `23-Mar-2026 `__, + `14-Apr-2026 `__, +Replaces: 609 +Resolution: `16-Apr-2026 `__ + + +======== +Abstract +======== + +This PEP proposes a Python Packaging Council with broad authority over +packaging standards, tools, and implementations. Like the Python Steering +Council, the Packaging Council seeks to exercise this authority as rarely as +possible; instead, they use this power to establish standard processes. + +Like :pep:`8016`, instead of trying to do everything in one PEP, this PEP +focuses on providing a minimal-but-solid foundation for further governance +decisions. + +======================== +Motivation and Rationale +======================== + +As Python packaging has matured, several interrelated problems with the current +way of managing the technical development, decision making and processes have +become apparent. + +---- +PyPA +---- + +The `Python Packaging Authority (PyPA)`_ was created to take over the +maintenance of `pip`_ and `virtualenv`_ from Ian Bicking, led by Brian Rosner, +Carl Meyer and Jannis Leidel. Over the years, additional projects got added to +the PyPA and the tooling maintained by the PyPA became the baseline tooling for +Python packaging, with additional package ecosystems using and building on top +of the tools and interoperability standards maintained by the PyPA. + +:pep:`609` formalized the PyPA's authority over the existing packaging tools, the +interoperability standards and their operation as a group of independent +projects under the same umbrella that aim to be interoperable. It also stated +that PyPA should be expected to provide opinions, insight and experience when +ecosystem-wide changes are being proposed. In particular, the work to support +existing projects and maintain a user guide has been excellent. + +At the time of writing, PyPA does not have a community elected body and +individuals within the group meet ad hoc, rather than on a regular cadence. The +PyPA itself is defined by a group of projects rather than as a group of +individuals, with no direct oversight over its individuals. + +------------ +Packaging-WG +------------ + +The PSF's `Packaging Workgroup`_ was created to support the larger efforts of +improving and maintaining the packaging ecosystem in Python through fundraising +and disbursement of raised funds. It was expected to largely focus on efforts +such as Python Package Index (PyPI), pip, packaging.python.org, Setuptools, and +cross-project efforts. + +The group made very good progress here in the past, but the scope and scale of +supporting Python's packaging ecosystem has grown considerably in the last few +years. There is both more work to be done and more stakeholders than ever +before. The Packaging Workgroup is also not a community elected body and does +not have a regular mechanism for changing, adding or removing members. + +-------------------------- +Interoperability Standards +-------------------------- + +The PyPA is responsible for creating and maintaining the interoperability +standards for Python packaging tools. The decision making for updating these +standards involves a standing delegation from the Python Steering Council to +specific individuals based on specific topics, and these individuals have the +ability to further delegate the decision making on the changes to other +individuals. + +We know that this process is not sustainable and there has been a desire +expressed by many to have a designated body to make these decisions (including +the individuals with the standing delegations). + +----------------------- +Python Steering Council +----------------------- + +While there is overlap between the PyPA, Packaging-WG, and the Python core +team, the Steering Council is not well-placed to make decisions over Python +packaging matters directly. Packaging only tangentially intersects with Python +language evolution, packaging requires specialised expertise and deep domain +knowledge (much like typing or the C API), and there is a different cohort of +potential electors for a Packaging Council than the Steering Council. + +------------ +Expectations +------------ + +An elected Packaging Council would have authority over the interoperability +standards, would guide the packaging tooling provided with CPython, and have +the mandate to coordinate Python packaging efforts. This would provide a +better, more consistent experience to packagers, package consumers, tool +developers, PyPI, and other index maintainers. With greater transparency and +clearer goals via an elected Packaging Council, the PSF will also be able to +offer more tactical work, longer term strategic vision, and fundraising support +to all parts of this ecosystem. + +Some expectations around the formation of a Packaging Council are that they +will: + +* Work with PSF staff and the newly formed `User Success Workgroup`_ toward + improving the user experience around packaging. +* Pronounce on and promote APIs, protocols, interfaces, and other interoperable standards relevant + to Python packaging. +* Facilitate a more stable packaging ecosystem that is more responsive to community input. +* Increase transparency and clearly share the goals for the packaging ecosystem. +* Facilitate tactical and fundraising support from the PSF, to increase capacity + and funding available to packaging tools. + +============= +Specification +============= + +----------------- +Packaging Council +----------------- + +The Packaging Council will be composed of five individuals. + +Mandate +======= + +The council shall work to: + +* Maintain the quality and stability of the Python packaging standards. +* Formalize and maintain the working relationships with the Python core team as well as the + Python Software Foundation. +* Establish appropriate decision-making processes. +* Improve Python packaging's user experience. +* Make contributing as accessible, inclusive, and sustainable as possible. +* Strive to seek consensus among contributors before acting in a formal + capacity. + +Responsibilities +================ + +The council shall: + +* Have broad authority over the Python packaging standards and Python Packaging + User Guide, that are maintained on https://packaging.python.org. +* Establish processes for making binding decisions regarding packaging + standards, tools and implementations as well as for considering + ecosystem-wide changes. +* Look for ways to use their direct authority as little as possible - seeking + consensus and consent over voting. + +To use its powers, the Packaging Council votes. Every Council member must either vote or explicitly +abstain. Members with conflicts of interest on a particular vote must abstain. Passing requires +support from a majority of non-abstaining Council members, which requires a quorum of 3 +non-abstaining members. If the Packaging Council cannot (e.g., by lack of quorum) or wishes not to +come to a decision on its own, it can also refer the matter to the Steering Council, whose decision +on the matter will be binding. + +Whenever possible, the Packaging Council is expected to share its decisions and processes +publicly in a timely fashion. + +Delegations +----------- + +The Packaging Council gains its authority over packaging matters via delegation from the Python +Steering Council. Upon acceptance of this PEP, the Steering Council is expected to formally issue a +standing delegation to the Packaging Council for PEPs related to Python packaging, superseding the +existing individual standing delegations. The bodies would work together on issues that intersect +the packaging domain and language stewardship (including the CPython implementation, standard +library, and distribution). + +The PSF Board is encouraged to formally deactivate the Packaging Workgroup and +the Packaging Council would take on the responsibilities of the PSF's Packaging +Workgroup. + +The PyPA is expected to work with the Packaging Council to establish a decision +making process that governs the technical projects under the PyPA umbrella. + +Processes +========= + +Election of the Packaging Council +--------------------------------- + +A Packaging Council election consists of the following phases: + +* **Phase 1**: Packaging Council Electors are determined by opt-in self-selection of the `PSF voting + members`_. The PSF voting membership is informed that Packaging Council ballots will be + available, and any PSF voting member can request a ballot. The PSF may choose to solicit + participation for both the PSF Board elections and Packaging Council elections at the same time. + Packaging Council Electors retain their voting rights for the entire year, and may exercise such + rights for other community-wide votes that may occur during that year. + +* **Phase 2**: Packaging Council Electors may nominate any individual for the Council election, including + themselves. Nominees must themselves be PSF voting members, and nominations must include information + about the nominee's relevant affiliations. + +* **Phase 3**: Each Elector receives a ballot consisting of all eligible nominees, which Electors + use to cast their vote for the Packaging Council. The mechanics of the election (i.e. the + software used to conduct the election, the algorithm used to determine the outcome of the vote, + etc.) are conducted by the PSF, in accordance with the `PSF bylaws`_ and its regular Board + election procedures. If a tie occurs, it may be resolved by mutual agreement among the + candidates, or else the winner will be chosen at random. + +Each phase will last two weeks. + +The Packaging Council election process is managed by a returns officer nominated by the PSF Board. +The PSF shall maintain records of the elections as well as run the annual election for the Packaging +Council. The PSF Board must certify the results of the election and may work with the returns +officer in whatever capacity is necessary to validate the integrity of the election. Election +transparency being of paramount importance for trust in the process, full vote totals *before* any +exclusions, while maintaining anonymity, should be made public when technically possible. + +Quorum for Packaging Council votes (either cohort election or votes of no confidence) is 50% of +Electors. + + +Term +---- + +There shall be two cohorts of Council members: Cohort A composed of two members +and Cohort B composed of three members. + +Each Council member shall be elected for a two-year term, unless they are +replacing a Council member that resigned, was removed, or otherwise becomes +vacant, in which case such replacement members shall be elected to a term +sufficient to complete a two-year term as measured from the term of the +original cohort. + +Each Council member's term will be two years, from when their elections are +finalized until the next elections for their cohort are finalized. + +As Packaging Council elections will generally align with the timing of PSF Board elections, any +members elected in an "off cycle" Council election (such as the initial Council election) will +similarly serve until the next regular election for their cohort. + +.. _whole-council: + +Only for elections of the entire Packaging Council (such as the initial Council election), the two +candidates receiving the highest number of votes shall be designated Cohort A with a two year term, +and the three candidates receiving the next highest number of votes shall be designated Cohort B +with a one year term. + +There are no term limits for individual Council members. + +.. _vacancy: + +Vacancies +--------- + +Packaging Council members may resign their position at any time. + +Whenever there is a vacancy during the regular Packaging Council term, the Council may +vote to appoint a replacement to serve out the rest of the term. + +If a Council member drops out of touch and cannot be contacted for a month or +longer, then the rest of the Council may vote to replace them (with a simple +majority vote, where the missing member is recorded as an abstention). + +If a full Packaging Council cannot be seated by this process, the PSF Board, in consultation with +the Python Steering Council, can appoint new Packaging Council members to fill the vacancies, or +call for new Packaging Council elections. + +Conflicts of interest +--------------------- + +No more than two Packaging Council members may be employed by or have decision-making power over the +same entity. An entity is a company and its subsidiaries, or another incorporated entity such as a +non-profit or educational institution with its own mission and goals. For this purpose, "employed +by" includes other forms of receiving compensation for current work regardless of workforce +classification, and having "decision-making power over" an entity includes holding officer/director +roles and 25% or more ownership stake. + +While we expect and trust Packaging Council members to act in the best interests of Python rather than +themselves or their affiliations, the mere appearance of any one such organization dominating Python +packaging development could itself be harmful and erode trust. + +PSF staff members are not permitted to serve as members of the Packaging Council. + +Serving Steering Council members are not permitted to concurrently serve as members of the Packaging +Council. + +In a Council election, if more than two of the top five vote-getters work for the same employer, then only the +top two such vote-getters are elected and the others are disqualified, with the remaining vote-getters +elevated in the vote rank. This process is repeated until a valid Packaging Council is formed. If after this +process a full Council cannot be formed, disqualified vote-getters are re-qualified in the rank order of their +vote tally until a full Council can be formed. + +If fewer than five members are to be elected, an analogous procedure is applied in order to +guarantee that no more than two members of the entire Packaging Council work for the same employer. + +During a Packaging Council term, if changing circumstances cause this rule to be broken (for instance, due to +a Council member changing employment), then one or more Council members must resign to remedy the issue, and +the resulting vacancies can then be filled as :ref:`normal `. + +The Python Steering Council is the final arbiter for technical conflicts of interest, and the Python +Software Foundation Board is the final arbiter for conflicts of interest related to governance for +the Packaging Council. + +Code of Conduct +--------------- + +All Packaging Council Electors and Packaging Council members are subject to, and must abide by the +PSF `Code of Conduct`_, its enforcement procedures, and its remedies for adjudicated violations. + +The Packaging Council will moderate its spaces and support PSF Code of Conduct enforcement as +appropriate in the area. + +.. _electors: + +========================== +Packaging Council Electors +========================== + +---------------- +Responsibilities +---------------- + +Packaging Council Electors participate in formal votes to elect the Packaging Council. + +The eligibility of Packaging Council Electors is equivalent to the Article IV, section 4.2 voting +membership defined in the `PSF bylaws`_. Should those bylaws change in the future, the eligibility +of Packaging Council Electors will similarly change to match. Packaging Council Electors must +affirm their intention to vote in Packaging Council elections in a manner and process similar to PSF +Board voting membership affirmations. + +PSF voting members may opt-out (annually or indefinitely) from Packaging Council elections +independently of their choice to vote in PSF Board elections. + +The process for ensuring elector eligibility to vote in Packaging Council elections will be +documented in the `Python Packaging User Guide `_ before +the inaugural election, and reviewed by the returns officer prior to every subsequent election. + +.. _process: + +Processes +========= + +Vote of no confidence +--------------------- + +In exceptional circumstances, a vote of no confidence may be called to remove a sitting Packaging +Council member, or the entire Council. The Python Steering Council may call such votes of no +confidence, with no second being necessary. Anyone may request such a vote of no confidence from +the Steering Council regardless of the requester's membership or affiliation, and the Steering +Council has discretion to call for the vote or not. The PSF Board may overrule the Steering Council +to initiate a vote of no confidence. + +The vote of no confidence lasts for two weeks. Each Elector votes for or against. If at least two +thirds of Electors express a lack of confidence, then the vote succeeds. + +There are two forms of no-confidence votes: those targeting a single member, and those targeting the Council +as a whole. The initial call for a no-confidence vote must specify which type is intended. If a single-member +vote succeeds, then that member is removed from the Council and the resulting vacancy can be handled by the +:ref:`normal process `. If a whole-Council vote succeeds, the Council is dissolved and a new Council +election is triggered immediately, using the rules for :ref:`whole Council ` elections. + +----------------------- +Changing the governance +----------------------- + +Changes proposed to this governance model must be approved by the Python Steering Council. + +============== +Rejected Ideas +============== + +---------------------------------------- +Annual elections for all Council members +---------------------------------------- + +An annual term for Council members is the approach taken for the Python +Steering Council's elections. This PEP uses a cohort-based model, derived from +the PSF Board's elections which enables continuity of members across a changing +Council. + +There is a trade-off between continuity of the Council and full reshuffles. This PEP takes the +position that continuity will be more valuable for the Python Packaging ecosystem. + +------------------------------- +Term limits for council members +------------------------------- + +While this is viewed as valuable for boards in general, this was rejected +because of the size of the pool of interested and qualified people who might +serve. + +------------------- +Elector eligibility +------------------- + +Previous iterations of this PEP's draft proposed different membership rules for identifying the +Packaging Council Electors. After extensive discussion among stakeholders, and after seeking the +widest possible feedback, the PEP authors agreed that aligning Packaging Council Electors with PSF +Board voting membership was both the most workable arrangement and the most equitable approach to +include all parts of the Python packaging community. + +PSF membership is used here because it is open to the broadest possible Python community. In +particular, most people doing Python packaging work do such work in public, including contributing +to PyPA or non-PyPA projects, and are likely to be eligible for PSF "Contributing Membership" based +on that work without paying any membership fee. + +------------------------------- +Approval voting in the election +------------------------------- + +An earlier non-public draft of this PEP used an approval voting process, which aligned with what +:pep:`13` stated at the time of writing. The Python core team has changed their governance to use +Bloc STAR, and this PEP was temporarily changed to use the same mechanism. However, since Packaging +Council elections will now be concurrent with PSF Board elections, with the same voting constituency +(i.e. PSF voting membership), and administered by the same returns officer, this PEP was updated to +align Packaging Council elections with PSF Board elections. + +------------------------------------------------------------------ +Disallow multiple people from the same organization on the council +------------------------------------------------------------------ + +This PEP currently mirrors the Python Steering Council's limit, that at most +two individuals related to a single organization can be on the council. + +Limiting it to one is workable; although it hasn't come up in the Steering Council, people do move around, and +we wouldn't want good candidates to either make employment decisions based on Packaging Council membership, or +have to resign based on an employment change. Limiting it to a maximum of two, plus votes of no confidence is +likely sufficient to avoid any undue employer influence. + +--------------------------------------------------------------------------- +Establishing specific processes for Packaging Council and PyPA relationship +--------------------------------------------------------------------------- + +As noted in the abstract, the focus of this PEP is on providing a +minimal-but-solid foundation for further governance decisions. The specifics of +this relationship would be figured out by the inaugural Council. + +.. _appendix_a: + +================================ +Appendix A: PEP approval process +================================ + +This PEP requires an atypical approval process, given the parties that must agree. To that end, the +authors will submit this PEP + +#. for a vote with the PSF Board, which must approve the linking of Packaging Council Electors to + the PSF Membership, and the deactivation of the Packaging Workgroup. + + - **RESOLVED** that the Python Software Foundation authorises the creation of a Packaging Council + as described in the draft of PEP 772 as published on 25 July 2025, conditional on the PEP + authors adding language to PEP 772 that explicitly gives the Packaging Council the authority to + enforce the PSF Code of Conduct, in addition to enforcement mechanisms otherwise approved by + the Foundation. + - Requested language added in `PR 4550 `_. + - Resolution from `PSF Board 2025-08-13 minutes + `__. + - **Resolved** that the Python Software Foundation authorizes the creation of a Packaging Council + as described in the draft of PEP 772 as published on 12 December 2025 with the following + clarification: The Python Steering Council (PSC) is the final arbiter for technical conflicts + of interest, and the PSF Board is the final arbiter for conflicts of interest related to + governance for the Packaging Council. Approved, 10-0-0. + - Requested language added in `PR 4872 `__. + - Resolution link TBD; resolution provided via email at time of writing. + +#. for a vote on the pypa-committers mailing list, in accordance with the process outlined in + :pep:`609` + + - This was voted on, as "endorsing the acceptance of PEP 772", with all 30 votes cast being + in favor. Closing message on the `pypa-committers vote thread + `__. + +#. for formal approval by the Python Steering Council + +We will reconcile and update the PEP as necessary based on recommendations, comments, and feedback, +and iterate until we gain all necessary approvals. + +.. _appendix_b: + +=================================================== +Appendix B: Operational suggestions for the Council +=================================================== + +This section is based on what the PEP's authors view as things that would be +beneficial for the Packaging Council to establish operational processes for. +These are non-binding yet strongly encouraged. + +The PSF will designate a staff person to be the Packaging Council's official +liaison who will regularly attend meetings, since it is expected that the +Packaging Council will meet on a regular basis (e.g. twice a month). + +* Coordinate with the Steering Council on PEPs that need input from both + groups. +* Coordinate with PyPA on their ongoing work to support individual projects. +* Delegate to domain experts or working groups in the packaging community, for + initiatives/PEPs with a niche focus (analogous to how the Steering Council + sends certain PEPs to the C API working group). +* Scope out work that might best be done by hiring someone and then work with + PSF to establish outcomes and a reasonable budget. +* The Packaging Council (similar to the Steering Council) is encouraged to + communicate with and when necessary seek advice from the PSF's Conduct + Working Group. +* Regularly synchronize with the Steering Council on a mutually agreed cadence, + with a recommended frequency of no less than once per quarter. +* Publish public agendas and minutes in a timely fashion. +* Provide casual real-time opportunities for people to bring topics that are + not PEPs, like office hours, a forum channel, or panels at Python events. + +=============== +Acknowledgments +=============== + +The language and spirit of this PEP is the work of many committed and passionate contributors across +the entire Python packaging ecosystem. The PEP authors wish to thank everyone who has participated +and provided input, and we sincerely believe that this PEP and its intended outcomes are much better +because of that participation. This PEP is just one (albeit important) step, and we encourage and +celebrate the ongoing contributions of all Python packaging stakeholders toward an ever-improving +packaging user experience. + + +.. _Python Packaging Authority (PyPA): https://packaging.python.org/en/latest/glossary/#term-Python-Packaging-Authority-PyPA +.. _pip: https://packaging.python.org/en/latest/key_projects/#pip +.. _virtualenv: https://packaging.python.org/en/latest/key_projects/#virtualenv +.. _Packaging Workgroup: https://wiki.python.org/psf/PackagingWG +.. _User Success Workgroup: https://github.com/psf/user-success-wg/ +.. _PSF bylaws: https://www.python.org/psf/bylaws/ +.. _Code of Conduct: https://policies.python.org/python.org/code-of-conduct/ +.. _PSF voting members: https://www.python.org/psf/membership/ diff --git a/decisiongrounding/scenarios_candidates/peps_packaging_governance_supersession/provenance.json b/decisiongrounding/scenarios_candidates/peps_packaging_governance_supersession/provenance.json new file mode 100644 index 0000000..d2cdf1a --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_packaging_governance_supersession/provenance.json @@ -0,0 +1,37 @@ +{ + "source_repo": "python/peps", + "pinned_commit": "f866e77409305866038471574f075cd8d83eee9e", + "peps": [ + { + "id": "PEP-0609", + "number": 609, + "file": "corpus/PEP-0609.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0609.rst", + "source_sha256": "fc0c5146985080844ce0b2fbe8c866bccb58d9695fe307f827619093ea427c1b", + "status": "Active", + "title": "Python Packaging Authority (PyPA) Governance", + "replaces": [], + "superseded_by": [] + }, + { + "id": "PEP-0772", + "number": 772, + "file": "corpus/PEP-0772.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0772.rst", + "source_sha256": "03b786c3044d220d649f79a899d51c8e1857ed28601b81615f991bd31f696412", + "status": "Accepted", + "title": "Packaging Council governance process", + "replaces": [ + "PEP-0609" + ], + "superseded_by": [] + } + ], + "supersedes_edges": [ + { + "source": "PEP-0772", + "type": "supersedes", + "target": "PEP-0609" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/peps_pypi_mirror_supersession/corpus/PEP-0381.md b/decisiongrounding/scenarios_candidates/peps_pypi_mirror_supersession/corpus/PEP-0381.md new file mode 100644 index 0000000..69b5ad0 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_pypi_mirror_supersession/corpus/PEP-0381.md @@ -0,0 +1,406 @@ +--- +schema_version: 1 +id: PEP-0381 +type: decision +tags: [pep, packaging] +--- + +# PEP-0381 — Mirroring infrastructure for PyPI + +## Status + +Withdrawn + +## Context + +Public decision ingested verbatim from python/peps (PEP 381) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0381.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0381.rst). Upstream +status: Withdrawn. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 381 +Title: Mirroring infrastructure for PyPI +Author: Tarek Ziadé , Martin von Löwis +Status: Withdrawn +Type: Standards Track +Topic: Packaging +Created: 21-Mar-2009 +Post-History: + + +Abstract +======== + +This PEP describes a mirroring infrastructure for PyPI. + + +PEP Withdrawal +============== + +The main PyPI web service was moved behind the Fastly caching CDN in May 2013: +https://mail.python.org/pipermail/distutils-sig/2013-May/020848.html + +Subsequently, this arrangement was formalised as an in-kind sponsorship with +the PSF, and the PSF has also taken on the task of risk management in the event +that that sponsorship arrangement were to ever cease. + +The download statistics that were previously provided directly on PyPI, are now +published indirectly via Google Big Query: +https://packaging.python.org/guides/analyzing-pypi-package-downloads/ + +Accordingly, the mirroring proposal described in this PEP is no longer required, +and has been marked as Withdrawn. + + +Rationale +========= + +PyPI is hosting over 6000 projects and is used on a daily basis +by people to build applications. Especially systems like ``easy_install`` +and ``zc.buildout`` make intensive usage of PyPI. + +For people making intensive use of PyPI, it can act as a single point +of failure. People have started to set up some mirrors, both private +and public. Those mirrors are active mirrors, which means that they +are browsing PyPI to get synced. + +In order to make the system more reliable, this PEP describes: + +- the mirror listing and registering at PyPI +- the pages a public mirror should maintain. These pages will be used + by PyPI, in order to get hit counts and the last modified date. +- how a mirror should synchronize with PyPI +- how a client can implement a fail-over mechanism + + +Mirror listing and registering +============================== + +People that wants to mirror PyPI make a proposal on catalog-SIG. +When a mirror is proposed on the mailing list, it is manually +added in a mirror list in the PyPI application after it +has been checked to be compliant with the mirroring rules. + +The mirror list is provided as a list of host names of the +form + + X.pypi.python.org + +The values of X are the sequence a,b,c,...,aa,ab,... +a.pypi.python.org is the master server; the mirrors start +with b. A CNAME record last.pypi.python.org points to the +last host name. Mirror operators should use a static address, +and report planned changes to that address in advance to +distutils-sig. + +The new mirror also appears at ``http://pypi.python.org/mirrors`` +which is a human-readable page that gives the list of mirrors. +This page also explains how to register a new mirror. + +Statistics page +::::::::::::::: + +PyPI provides statistics on downloads at ``/stats``. This page is +calculated daily by PyPI, by reading all mirrors' local stats and +summing them. + +The stats are presented in daily or monthly files, under ``/stats/days`` +and ``/stats/months``. Each file is a ``bzip2`` file with these formats: + +- YYYY-MM-DD.bz2 for daily files +- YYYY-MM.bz2 for monthly files + +Examples: + +- /stats/days/2008-11-06.bz2 +- /stats/days/2008-11-07.bz2 +- /stats/days/2008-11-08.bz2 +- /stats/months/2008-11.bz2 +- /stats/months/2008-10.bz2 + +Mirror Authenticity +=================== + +With a distributed mirroring system, clients may want to verify that +the mirrored copies are authentic. There are multiple threats to +consider: + +1. the central index may get compromised +2. the central index is assumed to be trusted, but the mirrors might + be tampered. +3. a man in the middle between the central index and the end user, + or between a mirror and the end user might tamper with datagrams. + +This specification only deals with the second threat. Some provisions +are made to detect man-in-the-middle attacks. To detect the first +attack, package authors need to sign their packages using PGP keys, so +that users verify that the package comes from the author they trust. + +The central index provides a DSA key at the URL /serverkey, in the PEM +format as generated by "openssl dsa -pubout" (i.e. :rfc:`3280` +SubjectPublicKeyInfo, with the algorithm 1.3.14.3.2.12). This URL must +*not* be mirrored, and clients must fetch the official serverkey from +PyPI directly, or use the copy that came with the PyPI client software. +Mirrors should still download the key, to detect a key rollover. + +For each package, a mirrored signature is provided at +/serversig/. This is the DSA signature of the parallel URL +/simple/, in DER form, using SHA-1 with DSA (i.e. as a +:rfc:`3279` Dsa-Sig-Value, created by algorithm 1.2.840.10040.4.3) + +Clients using a mirror need to perform the following steps to verify +a package: + +1. download the /simple page, and compute its SHA-1 hash +2. compute the DSA signature of that hash +3. download the corresponding /serversig, and compare it (byte-for-byte) + with the value computed in step 2. +4. compute and verify (against the /simple page) the MD-5 hashes + of all files they download from the mirror. + +An implementation of the verification algorithm is available from +https://svn.python.org/packages/trunk/pypi/tools/verify.py + +Verification is not needed when downloading from central index, and +should be avoided to reduce the computation overhead. + +About once a year, the key will be replaced with a new one. Mirrors +will have to re-fetch all /serversig pages. Clients using mirrors need +to find a trusted copy of the new server key. One way to obtain one +is to download it from https://pypi.python.org/serverkey. To detect +man-in-the-middle attacks, clients need to verify the SSL server +certificate, which will be signed by the CACert authority. + + +Special pages a mirror needs to provide +======================================= + +A mirror is a subset copy of PyPI, so it provides the same structure +by copying it. + +- simple: rest version of the package index +- packages: packages, stored by Python version, and letters +- serversig: signatures for the simple pages + +It also needs to provide two specific elements: + +- last-modified +- local-stats + +Last modified date +:::::::::::::::::: + +CPAN uses a freshness date system where the mirror's last +synchronisation date is made available. + +For PyPI, each mirror needs to maintain a URL with simple text content +that represents the last synchronisation date the mirror maintains. + +The date is provided in GMT time, using the ISO 8601 format [#iso8601]_. +Each mirror will be responsible to maintain its last modified date. + +This page must be located at : ``/last-modified`` and must be a +text/plain page. + +Local statistics +:::::::::::::::: + +Each mirror is responsible to count all the downloads that where done +via it. This is used by PyPI to sum up all downloads, to be able to +display the grand total. + +These statistics are in CSV-like form, with a header in the first +line. It needs to obey :pep:`305`. Basically, it should be +readable by Python's ``csv`` module. + +The fields in this file are: + +- package: the distutils id of the package. +- filename: the filename that has been downloaded. +- useragent: the User-Agent of the client that has downloaded the + package. +- count: the number of downloads. + +The content will look like this:: + + # package,filename,useragent,count + zc.buildout,zc.buildout-1.6.0.tgz,MyAgent,142 + ... + +The counting starts the day the mirror is launched, and there is one +file per day, compressed using the ``bzip2`` format. Each file is named +like the day. For example, ``2008-11-06.bz2`` is the file for the 6th of +November 2008. + +They are then provided in a folder called ``days``. For example: + +- /local-stats/days/2008-11-06.bz2 +- /local-stats/days/2008-11-07.bz2 +- /local-stats/days/2008-11-08.bz2 + +This page must be located at ``/local-stats``. + + +How a mirror should synchronize with PyPI +========================================= + +A mirroring protocol called ``Simple Index`` was described and +implemented by Martin v. Loewis and Jim Fulton, based on how +``easy_install`` works. This section synthesizes it and gives a few +relevant links, plus a small part about ``User-Agent``. + +The mirroring protocol +:::::::::::::::::::::: + +Mirrors must reduce the amount of data transferred between the central +server and the mirror. To achieve that, they MUST use the changelog() +PyPI XML-RPC call, and only refetch the packages that have been +changed since the last time. For each package P, they MUST copy +documents /simple/P/ and /serversig/P. If a package is deleted on the +central server, they MUST delete the package and all associated files. +To detect modification of package files, they MAY cache the file's +ETag, and MAY request skipping it using the If-none-match header. + +Each mirroring tool MUST identify itself using a descripte User-agent +header. + +The pep381client package [#pep381client]_ provides an application that +respects this protocol to browse PyPI. + +User-agent request header +::::::::::::::::::::::::: + +In order to be able to differentiate actions taken by clients over +PyPI, a specific user agent name should be provided by all mirroring +software. + +This is also true for all clients like: + +- zc.buildout [#zc.buildout]_. +- setuptools [#setuptools]_. +- pip [#pip]_. + +XXX user agent registering mechanism at PyPI ? + +How a client can use PyPI and its mirrors +::::::::::::::::::::::::::::::::::::::::: + +Clients that are browsing PyPI should be able to use alternative +mirrors, by getting the list of the mirrors using ``last.pypi.python.org``. + +Code example:: + + >>> import socket + >>> socket.gethostbyname_ex('last.pypi.python.org')[0] + 'h.pypi.python.org' + +The clients so far that could use this mechanism: + +- setuptools +- zc.buildout (through setuptools) +- pip + +Fail-over mechanism +::::::::::::::::::: + +Clients that are browsing PyPI should be able to use a fail-over +mechanism when PyPI or the used mirror is not responding. + +It is up to the client to decide which mirror should be used, maybe by +looking at its geographical location and its responsiveness. + +This PEP does not describe how this fail-over mechanism should work, +but it is strongly encouraged that the clients try to use the nearest +mirror. + +The clients so far that could use this mechanism: + +- setuptools +- zc.buildout (through setuptools) +- pip + +Extra package indexes +::::::::::::::::::::: + +It is obvious that some packages will not be uploaded to PyPI, whether +because they are private or whether because the project maintainer +runs their own server where people might get the project package. +However, it is strongly encouraged that a public package index follows +PyPI and Distutils protocols. + +In other words, the ``register`` and ``upload`` command should be +compatible with any package index server out there. + +Software that are compatible with PyPI and Distutils so far: + +- PloneSoftwareCenter [#psc]_ which is used to run plone.org products section. +- EggBasket [#eggbasket]_. + +**An extra package index is not a mirror of PyPI, but can have some +mirrors itself.** + +Merging several indexes +::::::::::::::::::::::: + +When a client needs to get some packages from several distinct +indexes, it should be able to use each one of them as a potential +source of packages. Different indexes should be defined as a sorted +list for the client to look for a package. + +Each independent index can of course provide a list of its mirrors. + +XXX define how to get the hostname for the mirrors of an arbitrary +index. + +That permits all combinations at client level, for a reliable +packaging system with all levels of privacy. + +It is up the client to deal with the merging. + + +References +========== + +.. [#pep381client] + http://pypi.python.org/pypi/pep381client + +.. [#iso8601] + http://en.wikipedia.org/wiki/ISO_8601 + +.. [#zc.buildout] + http://pypi.python.org/pypi/zc.buildout + +.. [#setuptools] + http://pypi.python.org/pypi/setuptools + +.. [#pip] + http://pypi.python.org/pypi/pip + +.. [#psc] + http://plone.org/products/plonesoftwarecenter + +.. [#eggbasket] + http://www.chrisarndt.de/projects/eggbasket + + +Acknowledgments +=============== + +Georg Brandl. + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_pypi_mirror_supersession/corpus/PEP-0449.md b/decisiongrounding/scenarios_candidates/peps_pypi_mirror_supersession/corpus/PEP-0449.md new file mode 100644 index 0000000..718777a --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_pypi_mirror_supersession/corpus/PEP-0449.md @@ -0,0 +1,179 @@ +--- +schema_version: 1 +id: PEP-0449 +type: decision +tags: [pep, packaging] +--- + +# PEP-0449 — Removal of the PyPI Mirror Auto Discovery and Naming Scheme + +## Status + +Final + +## Context + +Public decision ingested verbatim from python/peps (PEP 449) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0449.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0449.rst). Upstream +status: Final. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0381 + +## Source Text + +PEP: 449 +Title: Removal of the PyPI Mirror Auto Discovery and Naming Scheme +Author: Donald Stufft +BDFL-Delegate: Richard Jones +Discussions-To: distutils-sig@python.org +Status: Final +Type: Process +Topic: Packaging +Created: 04-Aug-2013 +Post-History: 04-Aug-2013 +Replaces: 381 +Resolution: https://mail.python.org/pipermail/distutils-sig/2013-August/022518.html + + +Abstract +======== + +This PEP provides a path to deprecate and ultimately remove the auto discovery +of PyPI mirrors as well as the hard coded naming scheme which requires +delegating a domain name under pypi.python.org to a third party. + + +Rationale +========= + +The PyPI mirroring infrastructure (defined in :pep:`381`) provides a means to +mirror the content of PyPI used by the automatic installers. It also provides +a method for auto discovery of mirrors and a consistent naming scheme. + +There are a number of problems with the auto discovery protocol and the +naming scheme: + +* They give control over a \*.python.org domain name to a third party, + allowing that third party to set or read cookies on the pypi.python.org and + python.org domain name. +* The use of a sub domain of pypi.python.org means that the mirror operators + will never be able to get a SSL certificate of their own, and giving them + one for a python.org domain name is unlikely to happen. +* The auto discovery uses an unauthenticated protocol (DNS). +* The lack of a TLS certificate on these domains means that clients can not + be sure that they have not been a victim of DNS poisoning or a MITM attack. +* The auto discovery protocol was designed to enable a client to automatically + select a mirror for use. This is no longer a requirement because the CDN + that PyPI is now using a globally distributed network of servers which will + automatically select one close to the client without any effort on the + clients part. +* The auto discovery protocol and use of the consistent naming scheme has only + ever been implemented by one installer (pip), and its implementation, besides + being insecure, has serious issues with performance and is slated for removal + with its next release (1.5). +* While there are provisions in :pep:`381` that would solve *some* of these + issues for a dedicated client it would not solve the issues that affect a + users browser. Additionally these provisions have not been implemented by + any installer to date. + +Due to the number of issues, some of them very serious, and the CDN which +provides most of the benefit of the auto discovery and consistent naming scheme +this PEP proposes to first deprecate and then remove the [a..z].pypi.python.org +names for mirrors and the last.pypi.python.org name for the auto discovery +protocol. The ability to mirror and the method of mirror will not be affected +and will continue to exist as written in :pep:`381`. Operators of existing +mirrors are encouraged to acquire their own domains and certificates to use for +their mirrors if they wish to continue hosting them. + + +Plan for Deprecation & Removal +============================== + +Immediately upon acceptance of this PEP documentation on PyPI will be updated +to reflect the deprecated nature of the official public mirrors and will +direct users to external resources like http://www.pypi-mirrors.org/ to +discover unofficial public mirrors if they wish to use one. + +Mirror operators, if they wish to continue operating their mirror, should +acquire a domain name to represent their mirror and, if they are able, a TLS +certificate. Once they have acquired a domain they should redirect their +assigned N.pypi.python.org domain name to their new domain. On Feb 15th, 2014 +the DNS entries for [a..z].pypi.python.org and last.pypi.python.org will be +removed. At any time prior to Feb 15th, 2014 a mirror operator may request +that their domain name be reclaimed by PyPI and pointed back at the master. + + +Why Feb 15th, 2014 +------------------ + +The most critical decision of this PEP is the final cut off date. If the date +is too soon then it needlessly punishes people by forcing them to drop +everything to update their deployment scripts. If the date is too far away then +the extended period of time does not help with the migration effort and merely +puts off the migration until a later date. + +The date of Feb 15th, 2014 has been chosen because it is roughly 6 months from +the date of the PEP. This should ensure a lengthy period of time to enable +people to update their deployment procedures to point to the new domains names +without merely padding the cut off date. + + +Why the DNS entries must be removed +----------------------------------- + +While it would be possible to simply reclaim the domain names used in mirror +and direct them back at PyPI in order to prevent users from needing to update +configurations to point away from those domains this has a number of issues. + +* Anyone who currently has these names hard coded in their configuration has + them hard coded as HTTP. This means that by allowing these names to continue + resolving we make it simple for a MITM operator to attack users by rewriting + the redirect to HTTPS prior to giving it to the client. +* The overhead of maintaining several domains pointing at PyPI has proved + troublesome for the small number of N.pypi.python.org domains that have + already been reclaimed. They oftentimes get mis-configured when things + change on the service which often leaves them broken for months at a time + until somebody notices. By leaving them in we leave users of these domains + open to random breakages which are less likely to get caught or noticed. +* People using these domains have explicitly chosen to use them for one reason + or another. One such reason may be because they do not wish to deploy from + a host located in a particular country. If these domains continue to resolve + but do not point at their existing locations we have silently removed this + choice from the existing users of those domains. + +That being said, removing the entries *will* require users who have modified +their configuration to either point back at the master (PyPI) or select a new +mirror name to point at. This is regarded as a regrettable requirement to +protect PyPI itself and the users of the mirrors from the attacks outlined +above or, at the very least, require them to make an informed decision about +the insecurity. + + +Public or Private Mirrors +========================= + +The mirroring protocol will continue to exist as defined in :pep:`381` and +people are encouraged to host public and private mirrors if they so desire. +The recommended mirroring client is `Bandersnatch`_. + + +.. _PyPI: https://pypi.python.org/ +.. _Bandersnatch: https://pypi.python.org/pypi/bandersnatch + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_pypi_mirror_supersession/provenance.json b/decisiongrounding/scenarios_candidates/peps_pypi_mirror_supersession/provenance.json new file mode 100644 index 0000000..fa384e2 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_pypi_mirror_supersession/provenance.json @@ -0,0 +1,37 @@ +{ + "source_repo": "python/peps", + "pinned_commit": "f866e77409305866038471574f075cd8d83eee9e", + "peps": [ + { + "id": "PEP-0381", + "number": 381, + "file": "corpus/PEP-0381.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0381.rst", + "source_sha256": "c5c38948ad3d5d23f915fed0f8b79a522a6c4a39867830624dfd58776bf9804d", + "status": "Withdrawn", + "title": "Mirroring infrastructure for PyPI", + "replaces": [], + "superseded_by": [] + }, + { + "id": "PEP-0449", + "number": 449, + "file": "corpus/PEP-0449.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0449.rst", + "source_sha256": "4e9c87c108c263d80552d97967ce7ab29b6b3f28acd6ac8eb253a355a5add170", + "status": "Final", + "title": "Removal of the PyPI Mirror Auto Discovery and Naming Scheme", + "replaces": [ + "PEP-0381" + ], + "superseded_by": [] + } + ], + "supersedes_edges": [ + { + "source": "PEP-0449", + "type": "supersedes", + "target": "PEP-0381" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/corpus/PEP-0245.md b/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/corpus/PEP-0245.md new file mode 100644 index 0000000..590ab0c --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/corpus/PEP-0245.md @@ -0,0 +1,547 @@ +--- +schema_version: 1 +id: PEP-0245 +type: decision +tags: [pep] +--- + +# PEP-0245 — Python Interface Syntax + +## Status + +Rejected + +## Context + +Public decision ingested verbatim from python/peps (PEP 245) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0245.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0245.rst). Upstream +status: Rejected. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 245 +Title: Python Interface Syntax +Author: Michel Pelletier +Status: Rejected +Type: Standards Track +Created: 11-Jan-2001 +Python-Version: 2.2 +Post-History: 21-Mar-2001 + + +.. note:: + + The no-longer-available Zope interfaces wiki page + (``https://www.zope.org/Wikis/Interfaces``) originally linked here, + containing links to further resources for this PEP, + can be `found on the Wayback Machine archive + `__. + Also, the Interface-Dev Zope mailing list on which this PEP was discussed + was shut down, but `its archives remain available + `__. + + +Rejection Notice +================ + +I'm rejecting this PEP. It's been five years now. While at some +point I expect that Python will have interfaces, it would be naive +to expect it to resemble the syntax in this PEP. Also, :pep:`246` is +being rejected in favor of something completely different; interfaces +won't play a role in adaptation or whatever will replace it. GvR. + + +Introduction +============ + +This PEP describes a proposed syntax for creating interface +objects in Python. + + +Overview +======== + +In addition to thinking about adding a static type system to +Python, the Types-SIG was also charged to devise an interface +system for Python. In December of 1998, Jim Fulton released a +prototype interfaces system based on discussions from the SIG. +Many of the issues and background information on this discussion +and prototype can be found in the SIG archives [1]_. + +Around the end of 2000, Digital Creations began thinking about +better component model designs for Zope [2]_. Zope's future +component model relies heavily on interface objects. This led to +further development of Jim's "Scarecrow" interfaces prototype. +Starting with version 2.3, Zope comes with an Interface package as +standard software. Zope's Interface package is used as the +reference implementation for this PEP. + +The syntax proposed by this PEP relies on syntax enhancements +describe in :pep:`232` and describes an underlying framework +which :pep:`233` could be based upon. There is some work being +done with regard to interface objects and Proxy objects, so for +those optional parts of this PEP you may want to see [3]_. + + +The Problem +=========== + +Interfaces are important because they solve a number of problems +that arise while developing software: + +- There are many implied interfaces in Python, commonly referred + to as "protocols". Currently determining those protocols is + based on implementation introspection, but often that also + fails. For example, defining ``__getitem__`` implies both a + sequence and a mapping (the former with sequential, integer + keys). There is no way for the developer to be explicit about + which protocols the object intends to implement. + +- Python is limited, from the developer's point of view, by the + split between types and classes. When types are expected, the + consumer uses code like 'type(foo) == type("")' to determine if + 'foo' is a string. When instances of classes are expected, the + consumer uses 'isinstance(foo, MyString)' to determine if 'foo' + is an instance of the 'MyString' class. There is no unified + model for determining if an object can be used in a certain, + valid way. + +- Python's dynamic typing is very flexible and powerful, but it + does not have the advantage of static typed languages that + provide type checking. Static typed languages provide you with + much more type safety, but are often overly verbose because + objects can only be generalized by common subclassing and used + specifically with casting (for example, in Java). + +There are also a number of documentation problems that interfaces +try to solve. + +- Developers waste a lot of time looking at the source code of + your system to figure out how objects work. + +- Developers who are new to your system may misunderstand how your + objects work, causing, and possibly propagating, usage errors. + +- Because a lack of interfaces means usage is inferred from the + source, developers may end up using methods and attributes that + are meant for "internal use only". + +- Code inspection can be hard, and very discouraging to novice + programmers trying to properly understand code written by gurus. + +- A lot of time is wasted when many people try very hard to + understand obscurity (like undocumented software). Effort spend + up front documenting interfaces will save much of this time in + the end. + +Interfaces try to solve these problems by providing a way for you +to specify a contractual obligation for your object, documentation +on how to use an object, and a built-in mechanism for discovering +the contract and the documentation. + +Python has very useful introspection features. It is well known +that this makes exploring concepts in the interactive interpreter +easier, because Python gives you the ability to look at all kinds +of information about the objects: the type, doc strings, instance +dictionaries, base classes, unbound methods and more. + +Many of these features are oriented toward introspecting, using +and changing the implementation of software, and one of them ("doc +strings") is oriented toward providing documentation. This +proposal describes an extension to this natural introspection +framework that describes an object's interface. + + +Overview of the Interface Syntax +================================ + +For the most part, the syntax of interfaces is very much like the +syntax of classes, but future needs, or needs brought up in +discussion, may define new possibilities for interface syntax. + +A formal BNF description of the syntax is givena later in the PEP, +for the purposes of illustration, here is an example of two +different interfaces created with the proposed syntax:: + + interface CountFishInterface: + "Fish counting interface" + + def oneFish(): + "Increments the fish count by one" + + def twoFish(): + "Increments the fish count by two" + + def getFishCount(): + "Returns the fish count" + + interface ColorFishInterface: + "Fish coloring interface" + + def redFish(): + "Sets the current fish color to red" + + def blueFish(): + "Sets the current fish color to blue" + + def getFishColor(): + "This returns the current fish color" + +This code, when evaluated, will create two interfaces called +``CountFishInterface`` and ``ColorFishInterface``. These interfaces +are defined by the ``interface`` statement. + +The prose documentation for the interfaces and their methods come +from doc strings. The method signature information comes from the +signatures of the ``def`` statements. Notice how there is no body +for the def statements. The interface does not implement a +service to anything; it merely describes one. Documentation +strings on interfaces and interface methods are mandatory, a +'pass' statement cannot be provided. The interface equivalent of +a pass statement is an empty doc string. + +You can also create interfaces that "extend" other interfaces. +Here, you can see a new type of Interface that extends the +CountFishInterface and ColorFishInterface:: + + interface FishMarketInterface(CountFishInterface, ColorFishInterface): + "This is the documentation for the FishMarketInterface" + + def getFishMonger(): + "Returns the fish monger you can interact with" + + def hireNewFishMonger(name): + "Hire a new fish monger" + + def buySomeFish(quantity=1): + "Buy some fish at the market" + +The FishMarketInterface extends upon the CountFishInterface and +ColorfishInterface. + + +Interface Assertion +=================== + +The next step is to put classes and interfaces together by +creating a concrete Python class that asserts that it implements +an interface. Here is an example FishMarket component that might +do this:: + + class FishError(Error): + pass + + class FishMarket implements FishMarketInterface: + number = 0 + color = None + monger_name = 'Crusty Barnacles' + + def __init__(self, number, color): + self.number = number + self.color = color + + def oneFish(self): + self.number += 1 + + def twoFish(self): + self.number += 2 + + def redFish(self): + self.color = 'red' + + def blueFish(self): + self.color = 'blue' + + def getFishCount(self): + return self.number + + def getFishColor(self): + return self.color + + def getFishMonger(self): + return self.monger_name + + def hireNewFishMonger(self, name): + self.monger_name = name + + def buySomeFish(self, quantity=1): + if quantity > self.count: + raise FishError("There's not enough fish") + self.count -= quantity + return quantity + +This new class, FishMarket defines a concrete class which +implements the FishMarketInterface. The object following the +``implements`` statement is called an "interface assertion". An +interface assertion can be either an interface object, or tuple of +interface assertions. + +The interface assertion provided in a ``class`` statement like this +is stored in the class's ``__implements__`` class attribute. After +interpreting the above example, you would have a class statement +that can be examined like this with an 'implements' built-in +function:: + + >>> FishMarket + + >>> FishMarket.__implements__ + (,) + >>> f = FishMarket(6, 'red') + >>> implements(f, FishMarketInterface) + 1 + >>> + +A class can realize more than one interface. For example, say you +had an interface called ``ItemInterface`` that described how an +object worked as an item in a container object. If you wanted to +assert that FishMarket instances realized the ItemInterface +interface as well as the FishMarketInterface, you can provide an +interface assertion that contained a tuple of interface objects to +the FishMarket class:: + + class FishMarket implements FishMarketInterface, ItemInterface: + # ... + +Interface assertions can also be used if you want to assert that +one class implements an interface, and all of the interfaces that +another class implements:: + + class MyFishMarket implements FishMarketInterface, ItemInterface: + # ... + + class YourFishMarket implements FooInterface, MyFishMarket.__implements__: + # ... + +This new class YourFishMarket, asserts that it implements the +FooInterface, as well as the interfaces implemented by the +MyFishMarket class. + +It's worth going into a little bit more detail about interface +assertions. An interface assertion is either an interface object, +or a tuple of interface assertions. For example:: + + FooInterface + + FooInterface, (BarInterface, BobInterface) + + FooInterface, (BarInterface, (BobInterface, MyClass.__implements__)) + +Are all valid interface assertions. When two interfaces define +the same attributes, the order in which information is preferred +in the assertion is from top-to-bottom, left-to-right. + +There are other interface proposals that, in the need for +simplicity, have combined the notion of class and interface to +provide simple interface enforcement. Interface objects have a +``deferred`` method that returns a deferred class that implements +this behavior:: + + >>> FM = FishMarketInterface.deferred() + >>> class MyFM(FM): pass + + >>> f = MyFM() + >>> f.getFishMonger() + Traceback (innermost last): + File "", line 1, in ? + Interface.Exceptions.BrokenImplementation: + An object has failed to implement interface FishMarketInterface + + The getFishMonger attribute was not provided. + >>> + +This provides for a bit of passive interface enforcement by +telling you what you forgot to do to implement that interface. + + +Formal Interface Syntax +======================= + +Python syntax is defined in a modified BNF grammar notation +described in the Python Reference Manual [4]_. This section +describes the proposed interface syntax using this grammar:: + + interfacedef: "interface" interfacename [extends] ":" suite + extends: "(" [expression_list] ")" + interfacename: identifier + +An interface definition is an executable statement. It first +evaluates the extends list, if present. Each item in the extends +list should evaluate to an interface object. + +The interface's suite is then executed in a new execution frame +(see the Python Reference Manual, section 4.1), using a newly +created local namespace and the original global namespace. When +the interface's suite finishes execution, its execution frame is +discarded but its local namespace is saved as interface elements. +An interface object is then created using the extends list for the +base interfaces and the saved interface elements. The interface +name is bound to this interface object in the original local +namespace. + +This PEP also proposes an extension to Python's 'class' statement:: + + classdef: "class" classname [inheritance] [implements] ":" suite + implements: "implements" implist + implist: expression-list + + classname, + inheritance, + suite, + expression-list: see the Python Reference Manual + +Before a class' suite is executed, the 'inheritance' and +'implements' statements are evaluated, if present. The +'inheritance' behavior is unchanged as defined in Section 7.6 of +the Language Reference. + +The 'implements', if present, is evaluated after inheritance. +This must evaluate to an interface specification, which is either +an interface, or a tuple of interface specifications. If a valid +interface specification is present, the assertion is assigned to +the class object's '__implements__' attribute, as a tuple. + +This PEP does not propose any changes to the syntax of function +definitions or assignments. + + +Classes and Interfaces +====================== + +The example interfaces above do not describe any kind of behavior +for their methods, they just describe an interface that a typical +FishMarket object would realize. + +You may notice a similarity between interfaces extending from +other interfaces and classes sub-classing from other classes. +This is a similar concept. However it is important to note that +interfaces extend interfaces and classes subclass classes. You +cannot extend a class or subclass an interface. Classes and +interfaces are separate. + +The purpose of a class is to share the implementation of how an +object works. The purpose of an interface is to document how to +work with an object, not how the object is implemented. It is +possible to have several different classes with very different +implementations realize the same interface. + +It's also possible to implement one interface with many classes +that mix in pieces the functionality of the interface or, +conversely, it's possible to have one class implement many +interfaces. Because of this, interfaces and classes should not be +confused or intermingled. + + +Interface-aware built-ins +========================= + +A useful extension to Python's list of built-in functions in the +light of interface objects would be ``implements()``. This builtin +would expect two arguments, an object and an interface, and return +a true value if the object implements the interface, false +otherwise. For example:: + + >>> interface FooInterface: pass + >>> class Foo implements FooInterface: pass + >>> f = Foo() + >>> implements(f, FooInterface) + 1 + +Currently, this functionality exists in the reference +implementation as functions in the ``Interface`` package, requiring +an "import Interface" to use it. Its existence as a built-in +would be purely for a convenience, and not necessary for using +interfaces, and analogous to ``isinstance()`` for classes. + + +Backward Compatibility +====================== + +The proposed interface model does not introduce any backward +compatibility issues in Python. The proposed syntax, however, +does. + +Any existing code that uses ``interface`` as an identifier will +break. There may be other kinds of backwards incompatibility that +defining ``interface`` as a new keyword will introduce. This +extension to Python's syntax does not change any existing syntax +in any backward incompatible way. + +The new ``from __future__`` Python syntax (:pep:`236`), and the new warning +framework (:pep:`230`) is ideal for resolving this backward +incompatibility. To use interface syntax now, a developer could +use the statement:: + + from __future__ import interfaces + +In addition, any code that uses the keyword ``interface`` as an +identifier will be issued a warning from Python. After the +appropriate period of time, the interface syntax would become +standard, the above import statement would do nothing, and any +identifiers named ``interface`` would raise an exception. This +period of time is proposed to be 24 months. + + +Summary of Proposed Changes to Python +===================================== + +Adding new ``interface`` keyword and extending class syntax with +``implements``. + +Extending class interface to include ``__implements__``. + +Add 'implements(obj, interface)' built-in. + + +Risks +===== + +This PEP proposes adding one new keyword to the Python language, +``interface``. This will break code. + + +Open Issues +=========== + +Goals +----- + +Syntax +------ + +Architecture +------------ + + +Dissenting Opinion +================== + +This PEP has not yet been discussed on python-dev. + + +References +========== + +.. [1] https://mail.python.org/pipermail/types-sig/1998-December/date.html + +.. [2] http://www.zope.org + +.. [3] http://www.lemburg.com/files/python/mxProxy.html + +.. [4] Python Reference Manual + http://docs.python.org/reference/ + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/corpus/PEP-0246.md b/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/corpus/PEP-0246.md new file mode 100644 index 0000000..c48c797 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/corpus/PEP-0246.md @@ -0,0 +1,780 @@ +--- +schema_version: 1 +id: PEP-0246 +type: decision +tags: [pep] +--- + +# PEP-0246 — Object Adaptation + +## Status + +Rejected + +## Context + +Public decision ingested verbatim from python/peps (PEP 246) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0246.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0246.rst). Upstream +status: Rejected. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 246 +Title: Object Adaptation +Author: Alex Martelli , + Clark C. Evans +Status: Rejected +Type: Standards Track +Created: 21-Mar-2001 +Python-Version: 2.5 +Post-History: 29-Mar-2001, 10-Jan-2005 + + +Rejection Notice +================ + +I'm rejecting this PEP. Something much better is about to happen; +it's too early to say exactly what, but it's not going to resemble +the proposal in this PEP too closely so it's better to start a new +PEP. GvR. + + +Abstract +======== + +This proposal puts forth an extensible cooperative mechanism for +the adaptation of an incoming object to a context which expects an +object supporting a specific protocol (say a specific type, class, +or interface). + +This proposal provides a built-in "adapt" function that, for any +object X and any protocol Y, can be used to ask the Python +environment for a version of X compliant with Y. Behind the +scenes, the mechanism asks object X: "Are you now, or do you know +how to wrap yourself to provide, a supporter of protocol Y?". +And, if this request fails, the function then asks protocol Y: +"Does object X support you, or do you know how to wrap it to +obtain such a supporter?" This duality is important, because +protocols can be developed after objects are, or vice-versa, and +this PEP lets either case be supported non-invasively with regard +to the pre-existing component[s]. + +Lastly, if neither the object nor the protocol know about each +other, the mechanism may check a registry of adapter factories, +where callables able to adapt certain objects to certain protocols +can be registered dynamically. This part of the proposal is +optional: the same effect could be obtained by ensuring that +certain kinds of protocols and/or objects can accept dynamic +registration of adapter factories, for example via suitable custom +metaclasses. However, this optional part allows adaptation to be +made more flexible and powerful in a way that is not invasive to +either protocols or other objects, thereby gaining for adaptation +much the same kind of advantage that Python standard library's +"copy_reg" module offers for serialization and persistence. + +This proposal does not specifically constrain what a protocol +*is*, what "compliance to a protocol" exactly *means*, nor what +precisely a wrapper is supposed to do. These omissions are +intended to leave this proposal compatible with both existing +categories of protocols, such as the existing system of type and +classes, as well as the many concepts for "interfaces" as such +which have been proposed or implemented for Python, such as the +one in :pep:`245`, the one in Zope3 [2]_, or the ones discussed in +the BDFL's Artima blog in late 2004 and early 2005 [3]_. However, +some reflections on these subjects, intended to be suggestive and +not normative, are also included. + + +Motivation +========== + +Currently there is no standardized mechanism in Python for +checking if an object supports a particular protocol. Typically, +existence of certain methods, particularly special methods such as +``__getitem__``, is used as an indicator of support for a particular +protocol. This technique works well for a few specific protocols +blessed by the BDFL (Benevolent Dictator for Life). The same can +be said for the alternative technique based on checking +'isinstance' (the built-in class "basestring" exists specifically +to let you use 'isinstance' to check if an object "is a \[built-in\] +string"). Neither approach is easily and generally extensible to +other protocols, defined by applications and third party +frameworks, outside of the standard Python core. + +Even more important than checking if an object already supports a +given protocol can be the task of obtaining a suitable adapter +(wrapper or proxy) for the object, if the support is not already +there. For example, a string does not support the file protocol, +but you can wrap it into a StringIO instance to obtain an object +which does support that protocol and gets its data from the string +it wraps; that way, you can pass the string (suitably wrapped) to +subsystems which require as their arguments objects that are +readable as files. Unfortunately, there is currently no general, +standardized way to automate this extremely important kind of +"adaptation by wrapping" operations. + +Typically, today, when you pass objects to a context expecting a +particular protocol, either the object knows about the context and +provides its own wrapper or the context knows about the object and +wraps it appropriately. The difficulty with these approaches is +that such adaptations are one-offs, are not centralized in a +single place of the users code, and are not executed with a common +technique, etc. This lack of standardization increases code +duplication with the same adapter occurring in more than one place +or it encourages classes to be re-written instead of adapted. In +either case, maintainability suffers. + +It would be very nice to have a standard function that can be +called upon to verify an object's compliance with a particular +protocol and provide for a wrapper if one is readily available -- +all without having to hunt through each library's documentation +for the incantation appropriate to that particular, specific case. + + +Requirements +============ + +When considering an object's compliance with a protocol, there are +several cases to be examined: + +a) When the protocol is a type or class, and the object has + exactly that type or is an instance of exactly that class (not + a subclass). In this case, compliance is automatic. + +b) When the object knows about the protocol, and either considers + itself compliant, or knows how to wrap itself suitably. + +c) When the protocol knows about the object, and either the object + already complies or the protocol knows how to suitably wrap the + object. + +d) When the protocol is a type or class, and the object is a + member of a subclass. This is distinct from the first case (a) + above, since inheritance (unfortunately) does not necessarily + imply substitutability, and thus must be handled carefully. + +e) When the context knows about the object and the protocol and + knows how to adapt the object so that the required protocol is + satisfied. This could use an adapter registry or similar + approaches. + +The fourth case above is subtle. A break of substitutability can +occur when a subclass changes a method's signature, or restricts +the domains accepted for a method's argument ("co-variance" on +arguments types), or extends the co-domain to include return +values which the base class may never produce ("contra-variance" +on return types). While compliance based on class inheritance +*should* be automatic, this proposal allows an object to signal +that it is not compliant with a base class protocol. + +If Python gains some standard "official" mechanism for interfaces, +however, then the "fast-path" case (a) can and should be extended +to the protocol being an interface, and the object an instance of +a type or class claiming compliance with that interface. For +example, if the "interface" keyword discussed in [3]_ is adopted +into Python, the "fast path" of case (a) could be used, since +instantiable classes implementing an interface would not be +allowed to break substitutability. + + +Specification +============= + +This proposal introduces a new built-in function, ``adapt()``, which +is the basis for supporting these requirements. + +The ``adapt()`` function has three parameters: + +- ``obj``, the object to be adapted + +- ``protocol``, the protocol requested of the object + +- ``alternate``, an optional object to return if the object could + not be adapted + +A successful result of the ``adapt()`` function returns either the +object passed ``obj``, if the object is already compliant with the +protocol, or a secondary object ``wrapper``, which provides a view +of the object compliant with the protocol. The definition of +wrapper is deliberately vague, and a wrapper is allowed to be a +full object with its own state if necessary. However, the design +intention is that an adaptation wrapper should hold a reference to +the original object it wraps, plus (if needed) a minimum of extra +state which it cannot delegate to the wrapper object. + +An excellent example of adaptation wrapper is an instance of +StringIO which adapts an incoming string to be read as if it was a +textfile: the wrapper holds a reference to the string, but deals +by itself with the "current point of reading" (from *where* in the +wrapped strings will the characters for the next, e.g., "readline" +call come from), because it cannot delegate it to the wrapped +object (a string has no concept of "current point of reading" nor +anything else even remotely related to that concept). + +A failure to adapt the object to the protocol raises an +``AdaptationError`` (which is a subclass of ``TypeError``), unless the +alternate parameter is used, in this case the alternate argument +is returned instead. + +To enable the first case listed in the requirements, the ``adapt()`` +function first checks to see if the object's type or the object's +class are identical to the protocol. If so, then the ``adapt()`` +function returns the object directly without further ado. + +To enable the second case, when the object knows about the +protocol, the object must have a ``__conform__()`` method. This +optional method takes two arguments: + +- ``self``, the object being adapted + +- ``protocol``, the protocol requested + +Just like any other special method in today's Python, ``__conform__`` +is meant to be taken from the object's class, not from the object +itself (for all objects, except instances of "classic classes" as +long as we must still support the latter). This enables a +possible 'tp_conform' slot to be added to Python's type objects in +the future, if desired. + +The object may return itself as the result of ``__conform__`` to +indicate compliance. Alternatively, the object also has the +option of returning a wrapper object compliant with the protocol. +If the object knows it is not compliant although it belongs to a +type which is a subclass of the protocol, then ``__conform__`` should +raise a ``LiskovViolation`` exception (a subclass of ``AdaptationError``). +Finally, if the object cannot determine its compliance, it should +return ``None`` to enable the remaining mechanisms. If ``__conform__`` +raises any other exception, "adapt" just propagates it. + +To enable the third case, when the protocol knows about the +object, the protocol must have an ``__adapt__()`` method. This +optional method takes two arguments: + +- ``self``, the protocol requested + +- ``obj``, the object being adapted + +If the protocol finds the object to be compliant, it can return +obj directly. Alternatively, the method may return a wrapper +compliant with the protocol. If the protocol knows the object is +not compliant although it belongs to a type which is a subclass of +the protocol, then ``__adapt__`` should raise a ``LiskovViolation`` +exception (a subclass of ``AdaptationError``). Finally, when +compliance cannot be determined, this method should return None to +enable the remaining mechanisms. If ``__adapt__`` raises any other +exception, "adapt" just propagates it. + +The fourth case, when the object's class is a sub-class of the +protocol, is handled by the built-in ``adapt()`` function. Under +normal circumstances, if "isinstance(object, protocol)" then +``adapt()`` returns the object directly. However, if the object is +not substitutable, either the ``__conform__()`` or ``__adapt__()`` +methods, as above mentioned, may raise an ``LiskovViolation`` (a +subclass of ``AdaptationError``) to prevent this default behavior. + +If none of the first four mechanisms worked, as a last-ditch +attempt, 'adapt' falls back to checking a registry of adapter +factories, indexed by the protocol and the type of ``obj``, to meet +the fifth case. Adapter factories may be dynamically registered +and removed from that registry to provide "third party adaptation" +of objects and protocols that have no knowledge of each other, in +a way that is not invasive to either the object or the protocols. + + +Intended Use +============ + +The typical intended use of adapt is in code which has received +some object X "from the outside", either as an argument or as the +result of calling some function, and needs to use that object +according to a certain protocol Y. A "protocol" such as Y is +meant to indicate an interface, usually enriched with some +semantics constraints (such as are typically used in the "design +by contract" approach), and often also some pragmatical +expectation (such as "the running time of a certain operation +should be no worse than O(N)", or the like); this proposal does +not specify how protocols are designed as such, nor how or whether +compliance to a protocol is checked, nor what the consequences may +be of claiming compliance but not actually delivering it (lack of +"syntactic" compliance -- names and signatures of methods -- will +often lead to exceptions being raised; lack of "semantic" +compliance may lead to subtle and perhaps occasional errors +[imagine a method claiming to be threadsafe but being in fact +subject to some subtle race condition, for example]; lack of +"pragmatic" compliance will generally lead to code that runs +``correctly``, but too slowly for practical use, or sometimes to +exhaustion of resources such as memory or disk space). + +When protocol Y is a concrete type or class, compliance to it is +intended to mean that an object allows all of the operations that +could be performed on instances of Y, with "comparable" semantics +and pragmatics. For example, a hypothetical object X that is a +singly-linked list should not claim compliance with protocol +'list', even if it implements all of list's methods: the fact that +indexing ``X[n]`` takes time O(n), while the same operation would be +O(1) on a list, makes a difference. On the other hand, an +instance of ``StringIO.StringIO`` does comply with protocol 'file', +even though some operations (such as those of module 'marshal') +may not allow substituting one for the other because they perform +explicit type-checks: such type-checks are "beyond the pale" from +the point of view of protocol compliance. + +While this convention makes it feasible to use a concrete type or +class as a protocol for purposes of this proposal, such use will +often not be optimal. Rarely will the code calling 'adapt' need +ALL of the features of a certain concrete type, particularly for +such rich types as file, list, dict; rarely can all those features +be provided by a wrapper with good pragmatics, as well as syntax +and semantics that are really the same as a concrete type's. + +Rather, once this proposal is accepted, a design effort needs to +start to identify the essential characteristics of those protocols +which are currently used in Python, particularly within the +standard library, and to formalize them using some kind of +"interface" construct (not necessarily requiring any new syntax: a +simple custom metaclass would let us get started, and the results +of the effort could later be migrated to whatever "interface" +construct is eventually accepted into the Python language). With +such a palette of more formally designed protocols, the code using +'adapt' will be able to ask for, say, adaptation into "a filelike +object that is readable and seekable", or whatever else it +specifically needs with some decent level of "granularity", rather +than too-generically asking for compliance to the 'file' protocol. + +Adaptation is NOT "casting". When object X itself does not +conform to protocol Y, adapting X to Y means using some kind of +wrapper object Z, which holds a reference to X, and implements +whatever operation Y requires, mostly by delegating to X in +appropriate ways. For example, if X is a string and Y is 'file', +the proper way to adapt X to Y is to make a ``StringIO(X)``, **NOT** to +call ``file(X)`` [which would try to open a file named by X]. + +Numeric types and protocols may need to be an exception to this +"adaptation is not casting" mantra, however. + + +Guido's "Optional Static Typing: Stop the Flames" Blog Entry +============================================================ + + +A typical simple use case of adaptation would be:: + + def f(X): + X = adapt(X, Y) + # continue by using X according to protocol Y + +In [4]_, the BDFL has proposed introducing the syntax:: + + def f(X: Y): + # continue by using X according to protocol Y + +to be a handy shortcut for exactly this typical use of adapt, and, +as a basis for experimentation until the parser has been modified +to accept this new syntax, a semantically equivalent decorator:: + + @arguments(Y) + def f(X): + # continue by using X according to protocol Y + +These BDFL ideas are fully compatible with this proposal, as are +other of Guido's suggestions in the same blog. + + + +Reference Implementation and Test Cases +======================================= + +The following reference implementation does not deal with classic +classes: it consider only new-style classes. If classic classes +need to be supported, the additions should be pretty clear, though +a bit messy (``x.__class__`` vs ``type(x)``, getting boundmethods directly +from the object rather than from the type, and so on). + +:: + + ----------------------------------------------------------------- + adapt.py + ----------------------------------------------------------------- + class AdaptationError(TypeError): + pass + class LiskovViolation(AdaptationError): + pass + + _adapter_factory_registry = {} + + def registerAdapterFactory(objtype, protocol, factory): + _adapter_factory_registry[objtype, protocol] = factory + + def unregisterAdapterFactory(objtype, protocol): + del _adapter_factory_registry[objtype, protocol] + + def _adapt_by_registry(obj, protocol, alternate): + factory = _adapter_factory_registry.get((type(obj), protocol)) + if factory is None: + adapter = alternate + else: + adapter = factory(obj, protocol, alternate) + if adapter is AdaptationError: + raise AdaptationError + else: + return adapter + + + def adapt(obj, protocol, alternate=AdaptationError): + + t = type(obj) + + # (a) first check to see if object has the exact protocol + if t is protocol: + return obj + + try: + # (b) next check if t.__conform__ exists & likes protocol + conform = getattr(t, '__conform__', None) + if conform is not None: + result = conform(obj, protocol) + if result is not None: + return result + + # (c) then check if protocol.__adapt__ exists & likes obj + adapt = getattr(type(protocol), '__adapt__', None) + if adapt is not None: + result = adapt(protocol, obj) + if result is not None: + return result + except LiskovViolation: + pass + else: + # (d) check if object is instance of protocol + if isinstance(obj, protocol): + return obj + + # (e) last chance: try the registry + return _adapt_by_registry(obj, protocol, alternate) + + ----------------------------------------------------------------- + test.py + ----------------------------------------------------------------- + from adapt import AdaptationError, LiskovViolation, adapt + from adapt import registerAdapterFactory, unregisterAdapterFactory + import doctest + + class A(object): + ''' + >>> a = A() + >>> a is adapt(a, A) # case (a) + True + ''' + + class B(A): + ''' + >>> b = B() + >>> b is adapt(b, A) # case (d) + True + ''' + + class C(object): + ''' + >>> c = C() + >>> c is adapt(c, B) # case (b) + True + >>> c is adapt(c, A) # a failure case + Traceback (most recent call last): + ... + AdaptationError + ''' + def __conform__(self, protocol): + if protocol is B: + return self + + class D(C): + ''' + >>> d = D() + >>> d is adapt(d, D) # case (a) + True + >>> d is adapt(d, C) # case (d) explicitly blocked + Traceback (most recent call last): + ... + AdaptationError + ''' + def __conform__(self, protocol): + if protocol is C: + raise LiskovViolation + + class MetaAdaptingProtocol(type): + def __adapt__(cls, obj): + return cls.adapt(obj) + + class AdaptingProtocol: + __metaclass__ = MetaAdaptingProtocol + @classmethod + def adapt(cls, obj): + pass + + class E(AdaptingProtocol): + ''' + >>> a = A() + >>> a is adapt(a, E) # case (c) + True + >>> b = A() + >>> b is adapt(b, E) # case (c) + True + >>> c = C() + >>> c is adapt(c, E) # a failure case + Traceback (most recent call last): + ... + AdaptationError + ''' + @classmethod + def adapt(cls, obj): + if isinstance(obj, A): + return obj + + class F(object): + pass + + def adapt_F_to_A(obj, protocol, alternate): + if isinstance(obj, F) and issubclass(protocol, A): + return obj + else: + return alternate + + def test_registry(): + ''' + >>> f = F() + >>> f is adapt(f, A) # a failure case + Traceback (most recent call last): + ... + AdaptationError + >>> registerAdapterFactory(F, A, adapt_F_to_A) + >>> f is adapt(f, A) # case (e) + True + >>> unregisterAdapterFactory(F, A) + >>> f is adapt(f, A) # a failure case again + Traceback (most recent call last): + ... + AdaptationError + >>> registerAdapterFactory(F, A, adapt_F_to_A) + ''' + + doctest.testmod() + + +Relationship To Microsoft's QueryInterface +========================================== + +Although this proposal has some similarities to Microsoft's (COM) +QueryInterface, it differs by a number of aspects. + +First, adaptation in this proposal is bi-directional, allowing the +interface (protocol) to be queried as well, which gives more +dynamic abilities (more Pythonic). Second, there is no special +"IUnknown" interface which can be used to check or obtain the +original unwrapped object identity, although this could be +proposed as one of those "special" blessed interface protocol +identifiers. Third, with QueryInterface, once an object supports +a particular interface it must always there after support this +interface; this proposal makes no such guarantee, since, in +particular, adapter factories can be dynamically added to the +registried and removed again later. + +Fourth, implementations of Microsoft's QueryInterface must support +a kind of equivalence relation -- they must be reflexive, +symmetrical, and transitive, in specific senses. The equivalent +conditions for protocol adaptation according to this proposal +would also represent desirable properties:: + + # given, to start with, a successful adaptation: + X_as_Y = adapt(X, Y) + + # reflexive: + assert adapt(X_as_Y, Y) is X_as_Y + + # transitive: + X_as_Z = adapt(X, Z, None) + X_as_Y_as_Z = adapt(X_as_Y, Z, None) + assert (X_as_Y_as_Z is None) == (X_as_Z is None) + + # symmetrical: + X_as_Z_as_Y = adapt(X_as_Z, Y, None) + assert (X_as_Y_as_Z is None) == (X_as_Z_as_Y is None) + +However, while these properties are desirable, it may not be +possible to guarantee them in all cases. QueryInterface can +impose their equivalents because it dictates, to some extent, how +objects, interfaces, and adapters are to be coded; this proposal +is meant to be not necessarily invasive, usable and to "retrofit" +adaptation between two frameworks coded in mutual ignorance of +each other without having to modify either framework. + +Transitivity of adaptation is in fact somewhat controversial, as +is the relationship (if any) between adaptation and inheritance. + +The latter would not be controversial if we knew that inheritance +always implies Liskov substitutability, which, unfortunately we +don't. If some special form, such as the interfaces proposed in +[4]_, could indeed ensure Liskov substitutability, then for that +kind of inheritance, only, we could perhaps assert that if X +conforms to Y and Y inherits from Z then X conforms to Z... but +only if substitutability was taken in a very strong sense to +include semantics and pragmatics, which seems doubtful. (For what +it's worth: in QueryInterface, inheritance does not require nor +imply conformance). This proposal does not include any "strong" +effects of inheritance, beyond the small ones specifically +detailed above. + +Similarly, transitivity might imply multiple "internal" adaptation +passes to get the result of ``adapt(X, Z)`` via some intermediate Y, +intrinsically like ``adapt(adapt(X, Y), Z)``, for some suitable and +automatically chosen Y. Again, this may perhaps be feasible under +suitably strong constraints, but the practical implications of +such a scheme are still unclear to this proposal's authors. Thus, +this proposal does not include any automatic or implicit +transitivity of adaptation, under whatever circumstances. + +For an implementation of the original version of this proposal +which performs more advanced processing in terms of transitivity, +and of the effects of inheritance, see Phillip J. Eby's +``PyProtocols`` [5]_. The documentation accompanying ``PyProtocols`` is +well worth studying for its considerations on how adapters should +be coded and used, and on how adaptation can remove any need for +typechecking in application code. + + +Questions and Answers +===================== + +* Q: What benefit does this proposal provide? + + A: The typical Python programmer is an integrator, someone who is + connecting components from various suppliers. Often, to + interface between these components, one needs intermediate + adapters. Usually the burden falls upon the programmer to + study the interface exposed by one component and required by + another, determine if they are directly compatible, or develop + an adapter. Sometimes a supplier may even include the + appropriate adapter, but even then searching for the adapter + and figuring out how to deploy the adapter takes time. + + This technique enables suppliers to work with each other + directly, by implementing ``__conform__`` or ``__adapt__`` as + necessary. This frees the integrator from making their own + adapters. In essence, this allows the components to have a + simple dialogue among themselves. The integrator simply + connects one component to another, and if the types don't + automatically match an adapting mechanism is built-in. + + Moreover, thanks to the adapter registry, a "fourth party" may + supply adapters to allow interoperation of frameworks which + are totally unaware of each other, non-invasively, and without + requiring the integrator to do anything more than install the + appropriate adapter factories in the registry at start-up. + + As long as libraries and frameworks cooperate with the + adaptation infrastructure proposed here (essentially by + defining and using protocols appropriately, and calling + 'adapt' as needed on arguments received and results of + call-back factory functions), the integrator's work thereby + becomes much simpler. + + For example, consider SAX1 and SAX2 interfaces: there is an + adapter required to switch between them. Normally, the + programmer must be aware of this; however, with this + adaptation proposal in place, this is no longer the case -- + indeed, thanks to the adapter registry, this need may be + removed even if the framework supplying SAX1 and the one + requiring SAX2 are unaware of each other. + + +* Q: Why does this have to be built-in, can't it be standalone? + + A: Yes, it does work standalone. However, if it is built-in, it + has a greater chance of usage. The value of this proposal is + primarily in standardization: having libraries and frameworks + coming from different suppliers, including the Python standard + library, use a single approach to adaptation. Furthermore: + + 0. The mechanism is by its very nature a singleton. + + 1. If used frequently, it will be much faster as a built-in. + + 2. It is extensible and unassuming. + + 3. Once 'adapt' is built-in, it can support syntax extensions + and even be of some help to a type inference system. + + +* Q: Why the verbs ``__conform__`` and ``__adapt__``? + + A: conform, verb intransitive + + 1. To correspond in form or character; be similar. + 2. To act or be in accord or agreement; comply. + 3. To act in accordance with current customs or modes. + + adapt, verb transitive + + 1. To make suitable to or fit for a specific use or situation. + + Source: The American Heritage Dictionary of the English + Language, Third Edition + + +Backwards Compatibility +======================= + +There should be no problem with backwards compatibility unless +someone had used the special names ``__conform__`` or ``__adapt__`` in +other ways, but this seems unlikely, and, in any case, user code +should never use special names for non-standard purposes. + +This proposal could be implemented and tested without changes to +the interpreter. + + +Credits +======= + +This proposal was created in large part by the feedback of the +talented individuals on the main Python mailing lists and the +type-sig list. To name specific contributors (with apologies if +we missed anyone!), besides the proposal's authors: the main +suggestions for the proposal's first versions came from Paul +Prescod, with significant feedback from Robin Thomas, and we also +borrowed ideas from Marcin 'Qrczak' Kowalczyk and Carlos Ribeiro. + +Other contributors (via comments) include Michel Pelletier, Jeremy +Hylton, Aahz Maruch, Fredrik Lundh, Rainer Deyke, Timothy Delaney, +and Huaiyu Zhu. The current version owes a lot to discussions +with (among others) Phillip J. Eby, Guido van Rossum, Bruce Eckel, +Jim Fulton, and Ka-Ping Yee, and to study and reflection of their +proposals, implementations, and documentation about use and +adaptation of interfaces and protocols in Python. + + +References and Footnotes +======================== + +.. [2] http://www.zope.org/Wikis/Interfaces/FrontPage + +.. [3] http://www.artima.com/weblogs/index.jsp?blogger=guido + +.. [4] http://www.artima.com/weblogs/viewpost.jsp?thread=87182 + +.. [5] http://peak.telecommunity.com/PyProtocols.html + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/corpus/PEP-0443.md b/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/corpus/PEP-0443.md new file mode 100644 index 0000000..ea55a7f --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/corpus/PEP-0443.md @@ -0,0 +1,442 @@ +--- +schema_version: 1 +id: PEP-0443 +type: decision +tags: [pep] +--- + +# PEP-0443 — Single-dispatch generic functions + +## Status + +Final + +## Context + +Public decision ingested verbatim from python/peps (PEP 443) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0443.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0443.rst). Upstream +status: Final. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0245 +- PEP-0246 + +## Source Text + +PEP: 443 +Title: Single-dispatch generic functions +Author: Łukasz Langa +Discussions-To: python-dev@python.org +Status: Final +Type: Standards Track +Created: 22-May-2013 +Python-Version: 3.4 +Post-History: 22-May-2013, 25-May-2013, 31-May-2013 +Replaces: 245, 246, 3124 + + +Abstract +======== + +This PEP proposes a new mechanism in the ``functools`` standard library +module that provides a simple form of generic programming known as +single-dispatch generic functions. + +A **generic function** is composed of multiple functions implementing +the same operation for different types. Which implementation should be +used during a call is determined by the dispatch algorithm. When the +implementation is chosen based on the type of a single argument, this is +known as **single dispatch**. + + +Rationale and Goals +=================== + +Python has always provided a variety of built-in and standard-library +generic functions, such as ``len()``, ``iter()``, ``pprint.pprint()``, +``copy.copy()``, and most of the functions in the ``operator`` module. +However, it currently: + +1. does not have a simple or straightforward way for developers to + create new generic functions, + +2. does not have a standard way for methods to be added to existing + generic functions (i.e., some are added using registration + functions, others require defining ``__special__`` methods, possibly + by monkeypatching). + +In addition, it is currently a common anti-pattern for Python code to +inspect the types of received arguments, in order to decide what to do +with the objects. + +For example, code may wish to accept either an object +of some type, or a sequence of objects of that type. +Currently, the "obvious way" to do this is by type inspection, but this +is brittle and closed to extension. + +Abstract Base Classes make it easier +to discover present behaviour, but don't help adding new behaviour. +A developer using an already-written library may be unable to change how +their objects are treated by such code, especially if the objects they +are using were created by a third party. + +Therefore, this PEP proposes a uniform API to address dynamic +overloading using decorators. + + +User API +======== + +To define a generic function, decorate it with the ``@singledispatch`` +decorator. Note that the dispatch happens on the type of the first +argument. Create your function accordingly:: + + >>> from functools import singledispatch + >>> @singledispatch + ... def fun(arg, verbose=False): + ... if verbose: + ... print("Let me just say,", end=" ") + ... print(arg) + +To add overloaded implementations to the function, use the +``register()`` attribute of the generic function. This is a decorator, +taking a type parameter and decorating a function implementing the +operation for that type:: + + >>> @fun.register(int) + ... def _(arg, verbose=False): + ... if verbose: + ... print("Strength in numbers, eh?", end=" ") + ... print(arg) + ... + >>> @fun.register(list) + ... def _(arg, verbose=False): + ... if verbose: + ... print("Enumerate this:") + ... for i, elem in enumerate(arg): + ... print(i, elem) + +To enable registering lambdas and pre-existing functions, the +``register()`` attribute can be used in a functional form:: + + >>> def nothing(arg, verbose=False): + ... print("Nothing.") + ... + >>> fun.register(type(None), nothing) + +The ``register()`` attribute returns the undecorated function. This +enables decorator stacking, pickling, as well as creating unit tests for +each variant independently:: + + >>> @fun.register(float) + ... @fun.register(Decimal) + ... def fun_num(arg, verbose=False): + ... if verbose: + ... print("Half of your number:", end=" ") + ... print(arg / 2) + ... + >>> fun_num is fun + False + +When called, the generic function dispatches on the type of the first +argument:: + + >>> fun("Hello, world.") + Hello, world. + >>> fun("test.", verbose=True) + Let me just say, test. + >>> fun(42, verbose=True) + Strength in numbers, eh? 42 + >>> fun(['spam', 'spam', 'eggs', 'spam'], verbose=True) + Enumerate this: + 0 spam + 1 spam + 2 eggs + 3 spam + >>> fun(None) + Nothing. + >>> fun(1.23) + 0.615 + +Where there is no registered implementation for a specific type, its +method resolution order is used to find a more generic implementation. +The original function decorated with ``@singledispatch`` is registered +for the base ``object`` type, which means it is used if no better +implementation is found. + +To check which implementation will the generic function choose for +a given type, use the ``dispatch()`` attribute:: + + >>> fun.dispatch(float) + + >>> fun.dispatch(dict) # note: default implementation + + +To access all registered implementations, use the read-only ``registry`` +attribute:: + + >>> fun.registry.keys() + dict_keys([, , , + , , + ]) + >>> fun.registry[float] + + >>> fun.registry[object] + + +The proposed API is intentionally limited and opinionated, as to ensure +it is easy to explain and use, as well as to maintain consistency with +existing members in the ``functools`` module. + + +Implementation Notes +==================== + +The functionality described in this PEP is already implemented in the +``pkgutil`` standard library module as ``simplegeneric``. Because this +implementation is mature, the goal is to move it largely as-is. The +reference implementation is available on hg.python.org [#ref-impl]_. + +The dispatch type is specified as a decorator argument. An alternative +form using function annotations was considered but its inclusion +has been rejected. As of May 2013, this usage pattern is out of scope +for the standard library [#pep-0008]_, and the best practices for +annotation usage are still debated. + +Based on the current ``pkgutil.simplegeneric`` implementation, and +following the convention on registering virtual subclasses on Abstract +Base Classes, the dispatch registry will not be thread-safe. + +Abstract Base Classes +--------------------- + +The ``pkgutil.simplegeneric`` implementation relied on several forms of +method resolution order (MRO). ``@singledispatch`` removes special +handling of old-style classes and Zope's ExtensionClasses. More +importantly, it introduces support for Abstract Base Classes (ABC). + +When a generic function implementation is registered for an ABC, the +dispatch algorithm switches to an extended form of C3 linearization, +which includes the relevant ABCs in the MRO of the provided argument. +The algorithm inserts ABCs where their functionality is introduced, i.e. +``issubclass(cls, abc)`` returns ``True`` for the class itself but +returns ``False`` for all its direct base classes. Implicit ABCs for +a given class (either registered or inferred from the presence of +a special method like ``__len__()``) are inserted directly after the +last ABC explicitly listed in the MRO of said class. + +In its most basic form, this linearization returns the MRO for the given +type:: + + >>> _compose_mro(dict, []) + [, ] + +When the second argument contains ABCs that the specified type is +a subclass of, they are inserted in a predictable order:: + + >>> _compose_mro(dict, [Sized, MutableMapping, str, + ... Sequence, Iterable]) + [, , + , , + , , + ] + +While this mode of operation is significantly slower, all dispatch +decisions are cached. The cache is invalidated on registering new +implementations on the generic function or when user code calls +``register()`` on an ABC to implicitly subclass it. In the latter case, +it is possible to create a situation with ambiguous dispatch, for +instance:: + + >>> from collections.abc import Iterable, Container + >>> class P: + ... pass + >>> Iterable.register(P) + + >>> Container.register(P) + + +Faced with ambiguity, ``@singledispatch`` refuses the temptation to +guess:: + + >>> @singledispatch + ... def g(arg): + ... return "base" + ... + >>> g.register(Iterable, lambda arg: "iterable") + at 0x108b49110> + >>> g.register(Container, lambda arg: "container") + at 0x108b491c8> + >>> g(P()) + Traceback (most recent call last): + ... + RuntimeError: Ambiguous dispatch: + or + +Note that this exception would not be raised if one or more ABCs had +been provided explicitly as base classes during class definition. In +this case dispatch happens in the MRO order:: + + >>> class Ten(Iterable, Container): + ... def __iter__(self): + ... for i in range(10): + ... yield i + ... def __contains__(self, value): + ... return value in range(10) + ... + >>> g(Ten()) + 'iterable' + +A similar conflict arises when subclassing an ABC is inferred from the +presence of a special method like ``__len__()`` or ``__contains__()``:: + + >>> class Q: + ... def __contains__(self, value): + ... return False + ... + >>> issubclass(Q, Container) + True + >>> Iterable.register(Q) + >>> g(Q()) + Traceback (most recent call last): + ... + RuntimeError: Ambiguous dispatch: + or + +An early version of the PEP contained a custom approach that was simpler +but created a number of edge cases with surprising results [#why-c3]_. + +Usage Patterns +============== + +This PEP proposes extending behaviour only of functions specifically +marked as generic. Just as a base class method may be overridden by +a subclass, so too a function may be overloaded to provide custom +functionality for a given type. + +Universal overloading does not equal *arbitrary* overloading, in the +sense that we need not expect people to randomly redefine the behavior +of existing functions in unpredictable ways. To the contrary, generic +function usage in actual programs tends to follow very predictable +patterns and registered implementations are highly-discoverable in the +common case. + +If a module is defining a new generic operation, it will usually also +define any required implementations for existing types in the same +place. Likewise, if a module is defining a new type, then it will +usually define implementations there for any generic functions that it +knows or cares about. As a result, the vast majority of registered +implementations can be found adjacent to either the function being +overloaded, or to a newly-defined type for which the implementation is +adding support. + +It is only in rather infrequent cases that one will have implementations +registered in a module that contains neither the function nor the +type(s) for which the implementation is added. In the absence of +incompetence or deliberate intention to be obscure, the few +implementations that are not registered adjacent to the relevant type(s) +or function(s), will generally not need to be understood or known about +outside the scope where those implementations are defined. (Except in +the "support modules" case, where best practice suggests naming them +accordingly.) + +As mentioned earlier, single-dispatch generics are already prolific +throughout the standard library. A clean, standard way of doing them +provides a way forward to refactor those custom implementations to use +a common one, opening them up for user extensibility at the same time. + + +Alternative approaches +====================== + +In :pep:`3124` Phillip J. Eby proposes a full-grown solution +with overloading based on arbitrary rule sets (with the default +implementation dispatching on argument types), as well as interfaces, +adaptation and method combining. PEAK-Rules [#peak-rules]_ is +a reference implementation of the concepts described in PJE's PEP. + +Such a broad approach is inherently complex, which makes reaching +a consensus hard. In contrast, this PEP focuses on a single piece of +functionality that is simple to reason about. It's important to note +this does not preclude the use of other approaches now or in the future. + +In a 2005 article on Artima [#artima2005]_ Guido van Rossum presents +a generic function implementation that dispatches on types of all +arguments on a function. The same approach was chosen in Andrey Popp's +``generic`` package available on PyPI [#pypi-generic]_, as well as David +Mertz's ``gnosis.magic.multimethods`` [#gnosis-multimethods]_. + +While this seems desirable at first, I agree with Fredrik Lundh's +comment that "if you design APIs with pages of logic just to sort out +what code a function should execute, you should probably hand over the +API design to someone else". In other words, the single argument +approach proposed in this PEP is not only easier to implement but also +clearly communicates that dispatching on a more complex state is an +anti-pattern. It also has the virtue of corresponding directly with the +familiar method dispatch mechanism in object oriented programming. The +only difference is whether the custom implementation is associated more +closely with the data (object-oriented methods) or the algorithm +(single-dispatch overloading). + +PyPy's RPython offers ``extendabletype`` [#pairtype]_, a metaclass which +enables classes to be externally extended. In combination with +``pairtype()`` and ``pair()`` factories, this offers a form of +single-dispatch generics. + + +Acknowledgements +================ + +Apart from Phillip J. Eby's work on :pep:`3124` and +PEAK-Rules, influences include Paul Moore's original issue +[#issue-5135]_ that proposed exposing ``pkgutil.simplegeneric`` as part +of the ``functools`` API, Guido van Rossum's article on multimethods +[#artima2005]_, and discussions with Raymond Hettinger on a general +pprint rewrite. Huge thanks to Alyssa Coghlan for encouraging me to create +this PEP and providing initial feedback. + + +References +========== + +.. [#ref-impl] + http://hg.python.org/features/pep-443/file/tip/Lib/functools.py#l359 + +.. [#pep-0008] :pep:`8` states in the "Programming Recommendations" + section that "the Python standard library will not use function + annotations as that would result in a premature commitment to + a particular annotation style". + +.. [#why-c3] http://bugs.python.org/issue18244 + +.. [#peak-rules] http://peak.telecommunity.com/DevCenter/PEAK_2dRules + +.. [#artima2005] + http://www.artima.com/weblogs/viewpost.jsp?thread=101605 + +.. [#pypi-generic] http://pypi.python.org/pypi/generic + +.. [#gnosis-multimethods] + http://gnosis.cx/publish/programming/charming_python_b12.html + +.. [#pairtype] + https://bitbucket.org/pypy/pypy/raw/default/rpython/tool/pairtype.py + +.. [#issue-5135] http://bugs.python.org/issue5135 + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/provenance.json b/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/provenance.json new file mode 100644 index 0000000..a72237d --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_single_dispatch_supersession/provenance.json @@ -0,0 +1,55 @@ +{ + "source_repo": "python/peps", + "pinned_commit": "f866e77409305866038471574f075cd8d83eee9e", + "peps": [ + { + "id": "PEP-0245", + "number": 245, + "file": "corpus/PEP-0245.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0245.rst", + "source_sha256": "bb91b2c5e631ab8448329c10c43854f4b459bc87bb7ac0de326934a44787b401", + "status": "Rejected", + "title": "Python Interface Syntax", + "replaces": [], + "superseded_by": [] + }, + { + "id": "PEP-0246", + "number": 246, + "file": "corpus/PEP-0246.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0246.rst", + "source_sha256": "c068e21d7f6a21c869ac86426d98845ee8b02e11c59fff7533ce8ff8e99e2190", + "status": "Rejected", + "title": "Object Adaptation", + "replaces": [], + "superseded_by": [] + }, + { + "id": "PEP-0443", + "number": 443, + "file": "corpus/PEP-0443.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0443.rst", + "source_sha256": "44897a2c82e72507053dc4c5f3b04759e6e29ac51ace418741527b08b9085571", + "status": "Final", + "title": "Single-dispatch generic functions", + "replaces": [ + "PEP-0245", + "PEP-0246", + "PEP-3124" + ], + "superseded_by": [] + } + ], + "supersedes_edges": [ + { + "source": "PEP-0443", + "type": "supersedes", + "target": "PEP-0245" + }, + { + "source": "PEP-0443", + "type": "supersedes", + "target": "PEP-0246" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/peps_tls_api_supersession/corpus/PEP-0543.md b/decisiongrounding/scenarios_candidates/peps_tls_api_supersession/corpus/PEP-0543.md new file mode 100644 index 0000000..41183cb --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_tls_api_supersession/corpus/PEP-0543.md @@ -0,0 +1,1593 @@ +--- +schema_version: 1 +id: PEP-0543 +type: decision +tags: [pep] +--- + +# PEP-0543 — A Unified TLS API for Python + +## Status + +Withdrawn + +## Context + +Public decision ingested verbatim from python/peps (PEP 543) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0543.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0543.rst). Upstream +status: Withdrawn. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 543 +Title: A Unified TLS API for Python +Author: Cory Benfield , + Christian Heimes +Status: Withdrawn +Type: Standards Track +Created: 17-Oct-2016 +Python-Version: 3.7 +Post-History: 11-Jan-2017, 19-Jan-2017, 02-Feb-2017, 09-Feb-2017 +Superseded-By: 748 + +Abstract +======== + +This PEP would define a standard TLS interface in the form of a collection of +abstract base classes. This interface would allow Python implementations and +third-party libraries to provide bindings to TLS libraries other than OpenSSL +that can be used by tools that expect the interface provided by the Python +standard library, with the goal of reducing the dependence of the Python +ecosystem on OpenSSL. + + +Resolution +========== + +2020-06-25: With contemporary agreement with one author, and past +agreement with another, this PEP is withdrawn due to changes in the +APIs of the underlying operating systems. + + +Rationale +========= + +In the 21st century it has become increasingly clear that robust and +user-friendly TLS support is an extremely important part of the ecosystem of +any popular programming language. For most of its lifetime, this role in the +Python ecosystem has primarily been served by the `ssl module`_, which provides +a Python API to the `OpenSSL library`_. + +Because the ``ssl`` module is distributed with the Python standard library, it +has become the overwhelmingly most-popular method for handling TLS in Python. +An extraordinary majority of Python libraries, both in the standard library and +on the Python Package Index, rely on the ``ssl`` module for their TLS +connectivity. + +Unfortunately, the preeminence of the ``ssl`` module has had a number of +unforeseen side-effects that have had the effect of tying the entire Python +ecosystem tightly to OpenSSL. This has forced Python users to use OpenSSL even +in situations where it may provide a worse user experience than alternative TLS +implementations, which imposes a cognitive burden and makes it hard to provide +"platform-native" experiences. + + +Problems +-------- + +The fact that the ``ssl`` module is built into the standard library has meant +that all standard-library Python networking libraries are entirely reliant on +the OpenSSL that the Python implementation has been linked against. This +leads to the following issues: + +* It is difficult to take advantage of new, higher-security TLS without + recompiling Python to get a new OpenSSL. While there are third-party bindings + to OpenSSL (e.g. `pyOpenSSL`_), these need to be shimmed into a format that + the standard library understands, forcing projects that want to use them to + maintain substantial compatibility layers. + +* For Windows distributions of Python, they need to be shipped with a copy of + OpenSSL. This puts the CPython development team in the position of being + OpenSSL redistributors, potentially needing to ship security updates to the + Windows Python distributions when OpenSSL vulnerabilities are released. + +* For macOS distributions of Python, they need either to be shipped with a copy + of OpenSSL or linked against the system OpenSSL library. Apple has formally + deprecated linking against the system OpenSSL library, and even if they had + not, that library version has been unsupported by upstream for nearly one + year as of the time of writing. The CPython development team has started + shipping newer OpenSSLs with the Python available from python.org, but this + has the same problem as with Windows. + +* Many systems, including but not limited to Windows and macOS, do not make + their system certificate stores available to OpenSSL. This forces users to + either obtain their trust roots from elsewhere (e.g. `certifi`_) or to + attempt to export their system trust stores in some form. + + Relying on `certifi`_ is less than ideal, as most system administrators do + not expect to receive security-critical software updates from PyPI. + Additionally, it is not easy to extend the `certifi`_ trust bundle to include + custom roots, or to centrally manage trust using the `certifi`_ model. + + Even in situations where the system certificate stores are made available to + OpenSSL in some form, the experience is still sub-standard, as OpenSSL will + perform different validation checks than the platform-native TLS + implementation. This can lead to users experiencing different behaviour on + their browsers or other platform-native tools than they experience in Python, + with little or no recourse to resolve the problem. + +* Users may wish to integrate with TLS libraries other than OpenSSL for many + other reasons, such as OpenSSL missing features (e.g. TLS 1.3 support), or + because OpenSSL is simply too large and unwieldy for the platform (e.g. for + embedded Python). Those users are left with the requirement to use + third-party networking libraries that can interact with their preferred TLS + library or to shim their preferred library into the OpenSSL-specific ``ssl`` + module API. + +Additionally, the ``ssl`` module as implemented today limits the ability of +CPython itself to add support for alternative TLS backends, or remove OpenSSL +support entirely, should either of these become necessary or useful. The +``ssl`` module exposes too many OpenSSL-specific function calls and features to +easily map to an alternative TLS backend. + + +Proposal +======== + +This PEP proposes to introduce a few new Abstract Base Classes in Python 3.7 to +provide TLS functionality that is not so strongly tied to OpenSSL. It also +proposes to update standard library modules to use only the interface exposed +by these abstract base classes wherever possible. There are three goals here: + +1. To provide a common API surface for both core and third-party developers to + target their TLS implementations to. This allows TLS developers to provide + interfaces that can be used by most Python code, and allows network + developers to have an interface that they can target that will work with a + wide range of TLS implementations. +2. To provide an API that has few or no OpenSSL-specific concepts leak through. + The ``ssl`` module today has a number of warts caused by leaking OpenSSL + concepts through to the API: the new ABCs would remove those specific + concepts. +3. To provide a path for the core development team to make OpenSSL one of many + possible TLS backends, rather than requiring that it be present on a system + in order for Python to have TLS support. + +The proposed interface is laid out below. + + +Interfaces +---------- + +There are several interfaces that require standardisation. Those interfaces +are: + +1. Configuring TLS, currently implemented by the `SSLContext`_ class in the + ``ssl`` module. +2. Providing an in-memory buffer for doing in-memory encryption or decryption + with no actual I/O (necessary for asynchronous I/O models), currently + implemented by the `SSLObject`_ class in the ``ssl`` module. +3. Wrapping a socket object, currently implemented by the `SSLSocket`_ class + in the ``ssl`` module. +4. Applying TLS configuration to the wrapping objects in (2) and (3). Currently + this is also implemented by the `SSLContext`_ class in the ``ssl`` module. +5. Specifying TLS cipher suites. There is currently no code for doing this in + the standard library: instead, the standard library uses OpenSSL cipher + suite strings. +6. Specifying application-layer protocols that can be negotiated during the + TLS handshake. +7. Specifying TLS versions. +8. Reporting errors to the caller, currently implemented by the `SSLError`_ + class in the ``ssl`` module. +9. Specifying certificates to load, either as client or server certificates. +10. Specifying which trust database should be used to validate certificates + presented by a remote peer. +11. Finding a way to get hold of these interfaces at run time. + +For the sake of simplicity, this PEP proposes to take a unified approach to +(2) and (3) (that is, buffers and sockets). The Python socket API is a +sizeable one, and implementing a wrapped socket that has the same behaviour as +a regular Python socket is a subtle and tricky thing to do. However, it is +entirely possible to implement a *generic* wrapped socket in terms of wrapped +buffers: that is, it is possible to write a wrapped socket (3) that will work +for any implementation that provides (2). For this reason, this PEP proposes to +provide an ABC for wrapped buffers (2) but a concrete class for wrapped sockets +(3). + +This decision has the effect of making it impossible to bind a small number of +TLS libraries to this ABC, because those TLS libraries *cannot* provide a +wrapped buffer implementation. The most notable of these at this time appears +to be Amazon's `s2n`_, which currently does not provide an I/O abstraction +layer. However, even this library consider this a missing feature and are +`working to add it`_. For this reason, it is safe to assume that a concrete +implementation of (3) in terms of (2) will be a substantial effort-saving +device and a great tool for correctness. Therefore, this PEP proposes doing +just that. + +Obviously, (5) doesn't require an abstract base class: instead, it requires a +richer API for configuring supported cipher suites that can be easily updated +with supported cipher suites for different implementations. + +(9) is a thorny problem, because in an ideal world the private keys associated +with these certificates would never end up in-memory in the Python process +(that is, the TLS library would collaborate with a Hardware Security Module +(HSM) to provide the private key in such a way that it cannot be extracted from +process memory). Thus, we need to provide an extensible model of providing +certificates that allows concrete implementations the ability to provide this +higher level of security, while also allowing a lower bar for those +implementations that cannot. This lower bar would be the same as the status +quo: that is, the certificate may be loaded from an in-memory buffer or from a +file on disk. + +(10) also represents an issue because different TLS implementations vary wildly +in how they allow users to select trust stores. Some implementations have +specific trust store formats that only they can use (such as the OpenSSL CA +directory format that is created by ``c_rehash``), and others may not allow you +to specify a trust store that does not include their default trust store. + +For this reason, we need to provide a model that assumes very little about the +form that trust stores take. The "Trust Store" section below goes into more +detail about how this is achieved. + +Finally, this API will split the responsibilities currently assumed by the +`SSLContext`_ object: specifically, the responsibility for holding and managing +configuration and the responsibility for using that configuration to build +wrapper objects. + +This is necessarily primarily for supporting functionality like Server Name +Indication (SNI). In OpenSSL (and thus in the ``ssl`` module), the server has +the ability to modify the TLS configuration in response to the client telling +the server what hostname it is trying to reach. This is mostly used to change +certificate chain so as to present the correct TLS certificate chain for the +given hostname. The specific mechanism by which this is done is by returning +a new `SSLContext`_ object with the appropriate configuration. + +This is not a model that maps well to other TLS implementations. Instead, we +need to make it possible to provide a return value from the SNI callback that +can be used to indicate what configuration changes should be made. This means +providing an object that can hold TLS configuration. This object needs to be +applied to specific TLSWrappedBuffer, and TLSWrappedSocket objects. + +For this reason, we split the responsibility of `SSLContext`_ into two separate +objects. The ``TLSConfiguration`` object is an object that acts as container +for TLS configuration: the ``ClientContext`` and ``ServerContext`` objects are +objects that are instantiated with a ``TLSConfiguration`` object. All three +objects would be immutable. + +.. note:: The following API declarations uniformly use type hints to aid + reading. Some of these type hints cannot actually be used in practice + because they are circularly referential. Consider them more a + guideline than a reflection of the final code in the module. + +Configuration +~~~~~~~~~~~~~ + +The ``TLSConfiguration`` concrete class defines an object that can hold and +manage TLS configuration. The goals of this class are as follows: + +1. To provide a method of specifying TLS configuration that avoids the risk of + errors in typing (this excludes the use of a simple dictionary). +2. To provide an object that can be safely compared to other configuration + objects to detect changes in TLS configuration, for use with the SNI + callback. + +This class is not an ABC, primarily because it is not expected to have +implementation-specific behaviour. The responsibility for transforming a +``TLSConfiguration`` object into a useful set of configuration for a given TLS +implementation belongs to the Context objects discussed below. + +This class has one other notable property: it is immutable. This is a desirable +trait for a few reasons. The most important one is that it allows these objects +to be used as dictionary keys, which is potentially extremely valuable for +certain TLS backends and their SNI configuration. On top of this, it frees +implementations from needing to worry about their configuration objects being +changed under their feet, which allows them to avoid needing to carefully +synchronize changes between their concrete data structures and the +configuration object. + +This object is extendable: that is, future releases of Python may add +configuration fields to this object as they become useful. For +backwards-compatibility purposes, new fields are only appended to this object. +Existing fields will never be removed, renamed, or reordered. + +The ``TLSConfiguration`` object would be defined by the following code:: + + ServerNameCallback = Callable[[TLSBufferObject, Optional[str], TLSConfiguration], Any] + + + _configuration_fields = [ + 'validate_certificates', + 'certificate_chain', + 'ciphers', + 'inner_protocols', + 'lowest_supported_version', + 'highest_supported_version', + 'trust_store', + 'sni_callback', + ] + + + _DEFAULT_VALUE = object() + + + class TLSConfiguration(namedtuple('TLSConfiguration', _configuration_fields)): + """ + An immutable TLS Configuration object. This object has the following + properties: + + :param validate_certificates bool: Whether to validate the TLS + certificates. This switch operates at a very broad scope: either + validation is enabled, in which case all forms of validation are + performed including hostname validation if possible, or validation + is disabled, in which case no validation is performed. + + Not all backends support having their certificate validation + disabled. If a backend does not support having their certificate + validation disabled, attempting to set this property to ``False`` + will throw a ``TLSError`` when this object is passed into a + context object. + + :param certificate_chain Tuple[Tuple[Certificate],PrivateKey]: The + certificate, intermediate certificate, and the corresponding + private key for the leaf certificate. These certificates will be + offered to the remote peer during the handshake if required. + + The first Certificate in the list must be the leaf certificate. All + subsequent certificates will be offered as intermediate additional + certificates. + + :param ciphers Tuple[Union[CipherSuite, int]]: + The available ciphers for TLS connections created with this + configuration, in priority order. + + :param inner_protocols Tuple[Union[NextProtocol, bytes]]: + Protocols that connections created with this configuration should + advertise as supported during the TLS handshake. These may be + advertised using either or both of ALPN or NPN. This list of + protocols should be ordered by preference. + + :param lowest_supported_version TLSVersion: + The minimum version of TLS that should be allowed on TLS + connections using this configuration. + + :param highest_supported_version TLSVersion: + The maximum version of TLS that should be allowed on TLS + connections using this configuration. + + :param trust_store TrustStore: + The trust store that connections using this configuration will use + to validate certificates. + + :param sni_callback Optional[ServerNameCallback]: + A callback function that will be called after the TLS Client Hello + handshake message has been received by the TLS server when the TLS + client specifies a server name indication. + + Only one callback can be set per ``TLSConfiguration``. If the + ``sni_callback`` is ``None`` then the callback is disabled. If the + ``TLSConfiguration`` is used for a ``ClientContext`` then this + setting will be ignored. + + The ``callback`` function will be called with three arguments: the + first will be the ``TLSBufferObject`` for the connection; the + second will be a string that represents the server name that the + client is intending to communicate (or ``None`` if the TLS Client + Hello does not contain a server name); and the third argument will + be the original ``TLSConfiguration`` that configured the + connection. The server name argument will be the IDNA *decoded* + server name. + + The ``callback`` must return a ``TLSConfiguration`` to allow + negotiation to continue. Other return values signal errors. + Attempting to control what error is signaled by the underlying TLS + implementation is not specified in this API, but is up to the + concrete implementation to handle. + + The Context will do its best to apply the ``TLSConfiguration`` + changes from its original configuration to the incoming connection. + This will usually include changing the certificate chain, but may + also include changes to allowable ciphers or any other + configuration settings. + """ + __slots__ = () + + def __new__(cls, validate_certificates: Optional[bool] = None, + certificate_chain: Optional[Tuple[Tuple[Certificate], PrivateKey]] = None, + ciphers: Optional[Tuple[Union[CipherSuite, int]]] = None, + inner_protocols: Optional[Tuple[Union[NextProtocol, bytes]]] = None, + lowest_supported_version: Optional[TLSVersion] = None, + highest_supported_version: Optional[TLSVersion] = None, + trust_store: Optional[TrustStore] = None, + sni_callback: Optional[ServerNameCallback] = None): + + if validate_certificates is None: + validate_certificates = True + + if ciphers is None: + ciphers = DEFAULT_CIPHER_LIST + + if inner_protocols is None: + inner_protocols = [] + + if lowest_supported_version is None: + lowest_supported_version = TLSVersion.TLSv1 + + if highest_supported_version is None: + highest_supported_version = TLSVersion.MAXIMUM_SUPPORTED + + return super().__new__( + cls, validate_certificates, certificate_chain, ciphers, + inner_protocols, lowest_supported_version, + highest_supported_version, trust_store, sni_callback + ) + + def update(self, validate_certificates=_DEFAULT_VALUE, + certificate_chain=_DEFAULT_VALUE, + ciphers=_DEFAULT_VALUE, + inner_protocols=_DEFAULT_VALUE, + lowest_supported_version=_DEFAULT_VALUE, + highest_supported_version=_DEFAULT_VALUE, + trust_store=_DEFAULT_VALUE, + sni_callback=_DEFAULT_VALUE): + """ + Create a new ``TLSConfiguration``, overriding some of the settings + on the original configuration with the new settings. + """ + if validate_certificates is _DEFAULT_VALUE: + validate_certificates = self.validate_certificates + + if certificate_chain is _DEFAULT_VALUE: + certificate_chain = self.certificate_chain + + if ciphers is _DEFAULT_VALUE: + ciphers = self.ciphers + + if inner_protocols is _DEFAULT_VALUE: + inner_protocols = self.inner_protocols + + if lowest_supported_version is _DEFAULT_VALUE: + lowest_supported_version = self.lowest_supported_version + + if highest_supported_version is _DEFAULT_VALUE: + highest_supported_version = self.highest_supported_version + + if trust_store is _DEFAULT_VALUE: + trust_store = self.trust_store + + if sni_callback is _DEFAULT_VALUE: + sni_callback = self.sni_callback + + return self.__class__( + validate_certificates, certificate_chain, ciphers, + inner_protocols, lowest_supported_version, + highest_supported_version, trust_store, sni_callback + ) + + + +Context +~~~~~~~ + +We define two Context abstract base classes. These ABCs define objects that +allow configuration of TLS to be applied to specific connections. They can be +thought of as factories for ``TLSWrappedSocket`` and ``TLSWrappedBuffer`` +objects. + +Unlike the current ``ssl`` module, we provide two context classes instead of +one. Specifically, we provide the ``ClientContext`` and ``ServerContext`` +classes. This simplifies the APIs (for example, there is no sense in the server +providing the ``server_hostname`` parameter to ``ssl.SSLContext.wrap_socket``, +but because there is only one context class that parameter is still available), +and ensures that implementations know as early as possible which side of a TLS +connection they will serve. Additionally, it allows implementations to opt-out +of one or either side of the connection. For example, SecureTransport on macOS +is not really intended for server use and has an enormous amount of +functionality missing for server-side use. This would allow SecureTransport +implementations to simply not define a concrete subclass of ``ServerContext`` +to signal their lack of support. + +One of the other major differences to the current ``ssl`` module is that a +number of flags and options have been removed. Most of these are self-evident, +but it is worth noting that ``auto_handshake`` has been removed from +``wrap_socket``. This was removed because it fundamentally represents an odd +design wart that saves very minimal effort at the cost of a complexity increase +both for users and implementers. This PEP requires that all users call +``do_handshake`` explicitly after connecting. + +As much as possible implementers should aim to make these classes immutable: +that is, they should prefer not to allow users to mutate their internal state +directly, instead preferring to create new contexts from new TLSConfiguration +objects. Obviously, the ABCs cannot enforce this constraint, and so they do not +attempt to. + +The ``Context`` abstract base class has the following class definition:: + + TLSBufferObject = Union[TLSWrappedSocket, TLSWrappedBuffer] + + + class _BaseContext(metaclass=ABCMeta): + @abstractmethod + def __init__(self, configuration: TLSConfiguration): + """ + Create a new context object from a given TLS configuration. + """ + + @property + @abstractmethod + def configuration(self) -> TLSConfiguration: + """ + Returns the TLS configuration that was used to create the context. + """ + + + class ClientContext(_BaseContext): + def wrap_socket(self, + socket: socket.socket, + server_hostname: Optional[str]) -> TLSWrappedSocket: + """ + Wrap an existing Python socket object ``socket`` and return a + ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` + socket: all other socket types are unsupported. + + The returned SSL socket is tied to the context, its settings and + certificates. The socket object originally passed to this method + should not be used again: attempting to use it in any way will lead + to undefined behaviour, especially across different TLS + implementations. To get the original socket object back once it has + been wrapped in TLS, see the ``unwrap`` method of the + TLSWrappedSocket. + + The parameter ``server_hostname`` specifies the hostname of the + service which we are connecting to. This allows a single server to + host multiple SSL-based services with distinct certificates, quite + similarly to HTTP virtual hosts. This is also used to validate the + TLS certificate for the given hostname. If hostname validation is + not desired, then pass ``None`` for this parameter. This parameter + has no default value because opting-out of hostname validation is + dangerous, and should not be the default behaviour. + """ + buffer = self.wrap_buffers(server_hostname) + return TLSWrappedSocket(socket, buffer) + + @abstractmethod + def wrap_buffers(self, server_hostname: Optional[str]) -> TLSWrappedBuffer: + """ + Create an in-memory stream for TLS, using memory buffers to store + incoming and outgoing ciphertext. The TLS routines will read + received TLS data from one buffer, and write TLS data that needs to + be emitted to another buffer. + + The implementation details of how this buffering works are up to + the individual TLS implementation. This allows TLS libraries that + have their own specialised support to continue to do so, while + allowing those without to use whatever Python objects they see fit. + + The ``server_hostname`` parameter has the same meaning as in + ``wrap_socket``. + """ + + + class ServerContext(_BaseContext): + def wrap_socket(self, socket: socket.socket) -> TLSWrappedSocket: + """ + Wrap an existing Python socket object ``socket`` and return a + ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` + socket: all other socket types are unsupported. + + The returned SSL socket is tied to the context, its settings and + certificates. The socket object originally passed to this method + should not be used again: attempting to use it in any way will lead + to undefined behaviour, especially across different TLS + implementations. To get the original socket object back once it has + been wrapped in TLS, see the ``unwrap`` method of the + TLSWrappedSocket. + """ + buffer = self.wrap_buffers() + return TLSWrappedSocket(socket, buffer) + + @abstractmethod + def wrap_buffers(self) -> TLSWrappedBuffer: + """ + Create an in-memory stream for TLS, using memory buffers to store + incoming and outgoing ciphertext. The TLS routines will read + received TLS data from one buffer, and write TLS data that needs to + be emitted to another buffer. + + The implementation details of how this buffering works are up to + the individual TLS implementation. This allows TLS libraries that + have their own specialised support to continue to do so, while + allowing those without to use whatever Python objects they see fit. + """ + + +Buffer +~~~~~~ + +The buffer-wrapper ABC will be defined by the ``TLSWrappedBuffer`` ABC, which +has the following definition:: + + class TLSWrappedBuffer(metaclass=ABCMeta): + @abstractmethod + def read(self, amt: int) -> bytes: + """ + Read up to ``amt`` bytes of data from the input buffer and return + the result as a ``bytes`` instance. + + Once EOF is reached, all further calls to this method return the + empty byte string ``b''``. + + May read "short": that is, fewer bytes may be returned than were + requested. + + Raise ``WantReadError`` or ``WantWriteError`` if there is + insufficient data in either the input or output buffer and the + operation would have caused data to be written or read. + + May raise ``RaggedEOF`` if the connection has been closed without a + graceful TLS shutdown. Whether this is an exception that should be + ignored or not is up to the specific application. + + As at any time a re-negotiation is possible, a call to ``read()`` + can also cause write operations. + """ + + @abstractmethod + def readinto(self, buffer: Any, amt: int) -> int: + """ + Read up to ``amt`` bytes of data from the input buffer into + ``buffer``, which must be an object that implements the buffer + protocol. Returns the number of bytes read. + + Once EOF is reached, all further calls to this method return the + empty byte string ``b''``. + + Raises ``WantReadError`` or ``WantWriteError`` if there is + insufficient data in either the input or output buffer and the + operation would have caused data to be written or read. + + May read "short": that is, fewer bytes may be read than were + requested. + + May raise ``RaggedEOF`` if the connection has been closed without a + graceful TLS shutdown. Whether this is an exception that should be + ignored or not is up to the specific application. + + As at any time a re-negotiation is possible, a call to + ``readinto()`` can also cause write operations. + """ + + @abstractmethod + def write(self, buf: Any) -> int: + """ + Write ``buf`` in encrypted form to the output buffer and return the + number of bytes written. The ``buf`` argument must be an object + supporting the buffer interface. + + Raise ``WantReadError`` or ``WantWriteError`` if there is + insufficient data in either the input or output buffer and the + operation would have caused data to be written or read. In either + case, users should endeavour to resolve that situation and then + re-call this method. When re-calling this method users *should* + re-use the exact same ``buf`` object, as some backends require that + the exact same buffer be used. + + This operation may write "short": that is, fewer bytes may be + written than were in the buffer. + + As at any time a re-negotiation is possible, a call to ``write()`` + can also cause read operations. + """ + + @abstractmethod + def do_handshake(self) -> None: + """ + Performs the TLS handshake. Also performs certificate validation + and hostname verification. + """ + + @abstractmethod + def cipher(self) -> Optional[Union[CipherSuite, int]]: + """ + Returns the CipherSuite entry for the cipher that has been + negotiated on the connection. If no connection has been negotiated, + returns ``None``. If the cipher negotiated is not defined in + CipherSuite, returns the 16-bit integer representing that cipher + directly. + """ + + @abstractmethod + def negotiated_protocol(self) -> Optional[Union[NextProtocol, bytes]]: + """ + Returns the protocol that was selected during the TLS handshake. + This selection may have been made using ALPN, NPN, or some future + negotiation mechanism. + + If the negotiated protocol is one of the protocols defined in the + ``NextProtocol`` enum, the value from that enum will be returned. + Otherwise, the raw bytestring of the negotiated protocol will be + returned. + + If ``Context.set_inner_protocols()`` was not called, if the other + party does not support protocol negotiation, if this socket does + not support any of the peer's proposed protocols, or if the + handshake has not happened yet, ``None`` is returned. + """ + + @property + @abstractmethod + def context(self) -> Context: + """ + The ``Context`` object this buffer is tied to. + """ + + @abstractproperty + def negotiated_tls_version(self) -> Optional[TLSVersion]: + """ + The version of TLS that has been negotiated on this connection. + """ + + @abstractmethod + def shutdown(self) -> None: + """ + Performs a clean TLS shut down. This should generally be used + whenever possible to signal to the remote peer that the content is + finished. + """ + + @abstractmethod + def receive_from_network(self, data): + """ + Receives some TLS data from the network and stores it in an + internal buffer. + """ + + @abstractmethod + def peek_outgoing(self, amt): + """ + Returns the next ``amt`` bytes of data that should be written to + the network from the outgoing data buffer, without removing it from + the internal buffer. + """ + + @abstractmethod + def consume_outgoing(self, amt): + """ + Discard the next ``amt`` bytes from the outgoing data buffer. This + should be used when ``amt`` bytes have been sent on the network, to + signal that the data no longer needs to be buffered. + """ + + +Socket +~~~~~~ + +The socket-wrapper class will be a concrete class that accepts two items in its +constructor: a regular socket object, and a ``TLSWrappedBuffer`` object. This +object will be too large to recreate in this PEP, but will be submitted as part +of the work to build the module. + +The wrapped socket will implement all of the socket API, though it will have +stub implementations of methods that only work for sockets with types other +than ``SOCK_STREAM`` (e.g. ``sendto``/``recvfrom``). That limitation can be +lifted as-and-when support for DTLS is added to this module. + +In addition, the socket class will include the following *extra* methods on top +of the regular socket methods:: + + class TLSWrappedSocket: + def do_handshake(self) -> None: + """ + Performs the TLS handshake. Also performs certificate validation + and hostname verification. This must be called after the socket has + connected (either via ``connect`` or ``accept``), before any other + operation is performed on the socket. + """ + + def cipher(self) -> Optional[Union[CipherSuite, int]]: + """ + Returns the CipherSuite entry for the cipher that has been + negotiated on the connection. If no connection has been negotiated, + returns ``None``. If the cipher negotiated is not defined in + CipherSuite, returns the 16-bit integer representing that cipher + directly. + """ + + def negotiated_protocol(self) -> Optional[Union[NextProtocol, bytes]]: + """ + Returns the protocol that was selected during the TLS handshake. + This selection may have been made using ALPN, NPN, or some future + negotiation mechanism. + + If the negotiated protocol is one of the protocols defined in the + ``NextProtocol`` enum, the value from that enum will be returned. + Otherwise, the raw bytestring of the negotiated protocol will be + returned. + + If ``Context.set_inner_protocols()`` was not called, if the other + party does not support protocol negotiation, if this socket does + not support any of the peer's proposed protocols, or if the + handshake has not happened yet, ``None`` is returned. + """ + + @property + def context(self) -> Context: + """ + The ``Context`` object this socket is tied to. + """ + + def negotiated_tls_version(self) -> Optional[TLSVersion]: + """ + The version of TLS that has been negotiated on this connection. + """ + + def unwrap(self) -> socket.socket: + """ + Cleanly terminate the TLS connection on this wrapped socket. Once + called, this ``TLSWrappedSocket`` can no longer be used to transmit + data. Returns the socket that was wrapped with TLS. + """ + + + +Cipher Suites +~~~~~~~~~~~~~ + +Supporting cipher suites in a truly library-agnostic fashion is a remarkably +difficult undertaking. Different TLS implementations often have *radically* +different APIs for specifying cipher suites, but more problematically these +APIs frequently differ in capability as well as in style. Some examples are +shown below: + +OpenSSL +^^^^^^^ + +OpenSSL uses a well-known cipher string format. This format has been adopted as +a configuration language by most products that use OpenSSL, including Python. +This format is relatively easy to read, but has a number of downsides: it is +a string, which makes it remarkably easy to provide bad inputs; it lacks much +detailed validation, meaning that it is possible to configure OpenSSL in a way +that doesn't allow it to negotiate any cipher at all; and it allows specifying +cipher suites in a number of different ways that make it tricky to parse. The +biggest problem with this format is that there is no formal specification for +it, meaning that the only way to parse a given string the way OpenSSL would is +to get OpenSSL to parse it. + +OpenSSL's cipher strings can look like this:: + + 'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5' + +This string demonstrates some of the complexity of the OpenSSL format. For +example, it is possible for one entry to specify multiple cipher suites: the +entry ``ECDH+AESGCM`` means "all ciphers suites that include both +elliptic-curve Diffie-Hellman key exchange and AES in Galois Counter Mode". +More explicitly, that will expand to four cipher suites:: + + "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256" + +That makes parsing a complete OpenSSL cipher string extremely tricky. Add to +the fact that there are other meta-characters, such as "!" (exclude all cipher +suites that match this criterion, even if they would otherwise be included: +"!MD5" means that no cipher suites using the MD5 hash algorithm should be +included), "-" (exclude matching ciphers if they were already included, but +allow them to be re-added later if they get included again), and "+" (include +the matching ciphers, but place them at the end of the list), and you get an +*extremely* complex format to parse. On top of this complexity it should be +noted that the actual result depends on the OpenSSL version, as an OpenSSL +cipher string is valid so long as it contains at least one cipher that OpenSSL +recognises. + +OpenSSL also uses different names for its ciphers than the names used in the +relevant specifications. See the manual page for ``ciphers(1)`` for more +details. + +The actual API inside OpenSSL for the cipher string is simple:: + + char *cipher_list = ; + int rc = SSL_CTX_set_cipher_list(context, cipher_list); + +This means that any format that is used by this module must be able to be +converted to an OpenSSL cipher string for use with OpenSSL. + +SecureTransport +^^^^^^^^^^^^^^^ + +SecureTransport is the macOS system TLS library. This library is substantially +more restricted than OpenSSL in many ways, as it has a much more restricted +class of users. One of these substantial restrictions is in controlling +supported cipher suites. + +Ciphers in SecureTransport are represented by a C ``enum``. This enum has one +entry per cipher suite, with no aggregate entries, meaning that it is not +possible to reproduce the meaning of an OpenSSL cipher string like +"ECDH+AESGCM" without hand-coding which categories each enum member falls into. + +However, the names of most of the enum members are in line with the formal +names of the cipher suites: that is, the cipher suite that OpenSSL calls +"ECDHE-ECDSA-AES256-GCM-SHA384" is called +"TLS_ECDHE_ECDHSA_WITH_AES_256_GCM_SHA384" in SecureTransport. + +The API for configuring cipher suites inside SecureTransport is simple:: + + SSLCipherSuite ciphers[] = {TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, ...}; + OSStatus status = SSLSetEnabledCiphers(context, ciphers, sizeof(ciphers)); + +SChannel +^^^^^^^^ + +SChannel is the Windows system TLS library. + +SChannel has extremely restrictive support for controlling available TLS +cipher suites, and additionally adopts a third method of expressing what TLS +cipher suites are supported. + +Specifically, SChannel defines a set of ``ALG_ID`` constants (C unsigned ints). +Each of these constants does not refer to an entire cipher suite, but instead +an individual algorithm. Some examples are ``CALG_3DES`` and ``CALG_AES_256``, +which refer to the bulk encryption algorithm used in a cipher suite, +``CALG_DH_EPHEM`` and ``CALG_RSA_KEYX`` which refer to part of the key exchange +algorithm used in a cipher suite, ``CALG_SHA1`` and ``CALG_MD5`` which refer to +the message authentication code used in a cipher suite, and ``CALG_ECDSA`` and +``CALG_RSA_SIGN`` which refer to the signing portions of the key exchange +algorithm. + +This can be thought of as the half of OpenSSL's functionality that +SecureTransport doesn't have: SecureTransport only allows specifying exact +cipher suites, while SChannel only allows specifying *parts* of the cipher +suite, while OpenSSL allows both. + +Determining which cipher suites are allowed on a given connection is done by +providing a pointer to an array of these ``ALG_ID`` constants. This means that +any suitable API must allow the Python code to determine which ``ALG_ID`` +constants must be provided. + + +Network Security Services (NSS) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +NSS is Mozilla's crypto and TLS library. It's used in Firefox, Thunderbird, +and as alternative to OpenSSL in multiple libraries, e.g. curl. + +By default, NSS comes with secure configuration of allowed ciphers. On some +platforms such as Fedora, the list of enabled ciphers is globally configured +in a system policy. Generally, applications should not modify cipher suites +unless they have specific reasons to do so. + +NSS has both process global and per-connection settings for cipher suites. It +does not have a concept of SSLContext like OpenSSL. A SSLContext-like behavior +can be easily emulated. Specifically, ciphers can be enabled or disabled +globally with ``SSL_CipherPrefSetDefault(PRInt32 cipher, PRBool enabled)``, +and ``SSL_CipherPrefSet(PRFileDesc *fd, PRInt32 cipher, PRBool enabled)`` +for a connection. The cipher ``PRInt32`` number is a signed 32bit integer +that directly corresponds to an registered IANA id, e.g. ``0x1301`` +is ``TLS_AES_128_GCM_SHA256``. Contrary to OpenSSL, the preference order +of ciphers is fixed and cannot be modified at runtime. + +Like SecureTransport, NSS has no API for aggregated entries. Some consumers +of NSS have implemented custom mappings from OpenSSL cipher names and rules +to NSS ciphers, e.g. ``mod_nss``. + + +Proposed Interface +^^^^^^^^^^^^^^^^^^ + +The proposed interface for the new module is influenced by the combined set of +limitations of the above implementations. Specifically, as every implementation +*except* OpenSSL requires that each individual cipher be provided, there is no +option but to provide that lowest-common denominator approach. + +The simplest approach is to provide an enumerated type that includes a large +subset of the cipher suites defined for TLS. The values of the enum members +will be their two-octet cipher identifier as used in the TLS handshake, +stored as a 16 bit integer. The names of the enum members will be their +IANA-registered cipher suite names. + +As of now, the `IANA cipher suite registry`_ contains over 320 cipher suites. +A large portion of the cipher suites are irrelevant for TLS connections to +network services. Other suites specify deprecated and insecure algorithms +that are no longer provided by recent versions of implementations. The enum +does not contain ciphers with: + +* key exchange: NULL, Kerberos (KRB5), pre-shared key (PSK), secure remote + transport (TLS-SRP) +* authentication: NULL, anonymous, export grade, Kerberos (KRB5), + pre-shared key (PSK), secure remote transport (TLS-SRP), DSA cert (DSS) +* encryption: NULL, ARIA, DES, RC2, export grade 40bit +* PRF: MD5 +* SCSV cipher suites + +3DES, RC4, SEED, and IDEA are included for legacy applications. Further more +five additional cipher suites from the TLS 1.3 draft (draft-ietf-tls-tls13-18) +are included, too. TLS 1.3 does not share any cipher suites with TLS 1.2 and +earlier. The resulting enum will contain roughly 110 suites. + +Because of these limitations, and because the enum doesn't contain every +defined cipher, and also to allow for forward-looking applications, all parts +of this API that accept ``CipherSuite`` objects will also accept raw 16-bit +integers directly. + +Rather than populate this enum by hand, we have a `TLS enum script`_ that +builds it from Christian Heimes' `tlsdb JSON file`_ (warning: +large file) and `IANA cipher suite registry`_. The TLSDB also opens up the +possibility of extending the API with additional querying function, +such as determining which TLS versions support which ciphers, if that +functionality is found to be useful or necessary. + +If users find this approach to be onerous, a future extension to this API can +provide helpers that can reintroduce OpenSSL's aggregation functionality. + +:: + + class CipherSuite(IntEnum): + TLS_RSA_WITH_RC4_128_SHA = 0x0005 + TLS_RSA_WITH_IDEA_CBC_SHA = 0x0007 + TLS_RSA_WITH_3DES_EDE_CBC_SHA = 0x000a + TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA = 0x0010 + TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA = 0x0016 + TLS_RSA_WITH_AES_128_CBC_SHA = 0x002f + TLS_DH_RSA_WITH_AES_128_CBC_SHA = 0x0031 + TLS_DHE_RSA_WITH_AES_128_CBC_SHA = 0x0033 + TLS_RSA_WITH_AES_256_CBC_SHA = 0x0035 + TLS_DH_RSA_WITH_AES_256_CBC_SHA = 0x0037 + TLS_DHE_RSA_WITH_AES_256_CBC_SHA = 0x0039 + TLS_RSA_WITH_AES_128_CBC_SHA256 = 0x003c + TLS_RSA_WITH_AES_256_CBC_SHA256 = 0x003d + TLS_DH_RSA_WITH_AES_128_CBC_SHA256 = 0x003f + TLS_RSA_WITH_CAMELLIA_128_CBC_SHA = 0x0041 + TLS_DH_RSA_WITH_CAMELLIA_128_CBC_SHA = 0x0043 + TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA = 0x0045 + TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 = 0x0067 + TLS_DH_RSA_WITH_AES_256_CBC_SHA256 = 0x0069 + TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 = 0x006b + TLS_RSA_WITH_CAMELLIA_256_CBC_SHA = 0x0084 + TLS_DH_RSA_WITH_CAMELLIA_256_CBC_SHA = 0x0086 + TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA = 0x0088 + TLS_RSA_WITH_SEED_CBC_SHA = 0x0096 + TLS_DH_RSA_WITH_SEED_CBC_SHA = 0x0098 + TLS_DHE_RSA_WITH_SEED_CBC_SHA = 0x009a + TLS_RSA_WITH_AES_128_GCM_SHA256 = 0x009c + TLS_RSA_WITH_AES_256_GCM_SHA384 = 0x009d + TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 = 0x009e + TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 = 0x009f + TLS_DH_RSA_WITH_AES_128_GCM_SHA256 = 0x00a0 + TLS_DH_RSA_WITH_AES_256_GCM_SHA384 = 0x00a1 + TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 = 0x00ba + TLS_DH_RSA_WITH_CAMELLIA_128_CBC_SHA256 = 0x00bc + TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 = 0x00be + TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 = 0x00c0 + TLS_DH_RSA_WITH_CAMELLIA_256_CBC_SHA256 = 0x00c2 + TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256 = 0x00c4 + TLS_AES_128_GCM_SHA256 = 0x1301 + TLS_AES_256_GCM_SHA384 = 0x1302 + TLS_CHACHA20_POLY1305_SHA256 = 0x1303 + TLS_AES_128_CCM_SHA256 = 0x1304 + TLS_AES_128_CCM_8_SHA256 = 0x1305 + TLS_ECDH_ECDSA_WITH_RC4_128_SHA = 0xc002 + TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA = 0xc003 + TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA = 0xc004 + TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA = 0xc005 + TLS_ECDHE_ECDSA_WITH_RC4_128_SHA = 0xc007 + TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA = 0xc008 + TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA = 0xc009 + TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA = 0xc00a + TLS_ECDH_RSA_WITH_RC4_128_SHA = 0xc00c + TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA = 0xc00d + TLS_ECDH_RSA_WITH_AES_128_CBC_SHA = 0xc00e + TLS_ECDH_RSA_WITH_AES_256_CBC_SHA = 0xc00f + TLS_ECDHE_RSA_WITH_RC4_128_SHA = 0xc011 + TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA = 0xc012 + TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA = 0xc013 + TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA = 0xc014 + TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 = 0xc023 + TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 = 0xc024 + TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 = 0xc025 + TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 = 0xc026 + TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 = 0xc027 + TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 = 0xc028 + TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 = 0xc029 + TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 = 0xc02a + TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 = 0xc02b + TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 = 0xc02c + TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 = 0xc02d + TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 = 0xc02e + TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 = 0xc02f + TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 = 0xc030 + TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 = 0xc031 + TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 = 0xc032 + TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256 = 0xc072 + TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384 = 0xc073 + TLS_ECDH_ECDSA_WITH_CAMELLIA_128_CBC_SHA256 = 0xc074 + TLS_ECDH_ECDSA_WITH_CAMELLIA_256_CBC_SHA384 = 0xc075 + TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 = 0xc076 + TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 = 0xc077 + TLS_ECDH_RSA_WITH_CAMELLIA_128_CBC_SHA256 = 0xc078 + TLS_ECDH_RSA_WITH_CAMELLIA_256_CBC_SHA384 = 0xc079 + TLS_RSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc07a + TLS_RSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc07b + TLS_DHE_RSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc07c + TLS_DHE_RSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc07d + TLS_DH_RSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc07e + TLS_DH_RSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc07f + TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc086 + TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc087 + TLS_ECDH_ECDSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc088 + TLS_ECDH_ECDSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc089 + TLS_ECDHE_RSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc08a + TLS_ECDHE_RSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc08b + TLS_ECDH_RSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc08c + TLS_ECDH_RSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc08d + TLS_RSA_WITH_AES_128_CCM = 0xc09c + TLS_RSA_WITH_AES_256_CCM = 0xc09d + TLS_DHE_RSA_WITH_AES_128_CCM = 0xc09e + TLS_DHE_RSA_WITH_AES_256_CCM = 0xc09f + TLS_RSA_WITH_AES_128_CCM_8 = 0xc0a0 + TLS_RSA_WITH_AES_256_CCM_8 = 0xc0a1 + TLS_DHE_RSA_WITH_AES_128_CCM_8 = 0xc0a2 + TLS_DHE_RSA_WITH_AES_256_CCM_8 = 0xc0a3 + TLS_ECDHE_ECDSA_WITH_AES_128_CCM = 0xc0ac + TLS_ECDHE_ECDSA_WITH_AES_256_CCM = 0xc0ad + TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8 = 0xc0ae + TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8 = 0xc0af + TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 = 0xcca8 + TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 = 0xcca9 + TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 = 0xccaa + + +Enum members can be mapped to OpenSSL cipher names:: + + >>> import ssl + >>> ctx = ssl.SSLContext(ssl.PROTOCOL_TLS) + >>> ctx.set_ciphers('ALL:COMPLEMENTOFALL') + >>> ciphers = {c['id'] & 0xffff: c['name'] for c in ctx.get_ciphers()} + >>> ciphers[CipherSuite.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256] + 'ECDHE-RSA-AES128-GCM-SHA256' + + +For SecureTransport, these enum members directly refer to the values of the +cipher suite constants. For example, SecureTransport defines the cipher suite +enum member ``TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`` as having the value +``0xC02C``. Not coincidentally, that is identical to its value in the above +enum. This makes mapping between SecureTransport and the above enum very easy +indeed. + +For SChannel there is no easy direct mapping, due to the fact that SChannel +configures ciphers, instead of cipher suites. This represents an ongoing +concern with SChannel, which is that it is very difficult to configure in a +specific manner compared to other TLS implementations. + +For the purposes of this PEP, any SChannel implementation will need to +determine which ciphers to choose based on the enum members. This may be more +open than the actual cipher suite list actually wants to allow, or it may be +more restrictive, depending on the choices of the implementation. This PEP +recommends that it be more restrictive, but of course this cannot be enforced. + + +Protocol Negotiation +~~~~~~~~~~~~~~~~~~~~ + +Both NPN and ALPN allow for protocol negotiation as part of the HTTP/2 +handshake. While NPN and ALPN are, at their fundamental level, built on top of +bytestrings, string-based APIs are frequently problematic as they allow for +errors in typing that can be hard to detect. + +For this reason, this module would define a type that protocol negotiation +implementations can pass and be passed. This type would wrap a bytestring to +allow for aliases for well-known protocols. This allows us to avoid the +problems inherent in typos for well-known protocols, while allowing the full +extensibility of the protocol negotiation layer if needed by letting users pass +byte strings directly. + +:: + + class NextProtocol(Enum): + H2 = b'h2' + H2C = b'h2c' + HTTP1 = b'http/1.1' + WEBRTC = b'webrtc' + C_WEBRTC = b'c-webrtc' + FTP = b'ftp' + STUN = b'stun.nat-discovery' + TURN = b'stun.turn' + +TLS Versions +~~~~~~~~~~~~ + +It is often useful to be able to restrict the versions of TLS you're willing to +support. There are many security advantages in refusing to use old versions of +TLS, and some misbehaving servers will mishandle TLS clients advertising +support for newer versions. + +The following enumerated type can be used to gate TLS versions. Forward-looking +applications should almost never set a maximum TLS version unless they +absolutely must, as a TLS backend that is newer than the Python that uses it +may support TLS versions that are not in this enumerated type. + +Additionally, this enumerated type defines two additional flags that can always +be used to request either the lowest or highest TLS version supported by an +implementation. + +:: + + class TLSVersion(Enum): + MINIMUM_SUPPORTED = auto() + SSLv2 = auto() + SSLv3 = auto() + TLSv1 = auto() + TLSv1_1 = auto() + TLSv1_2 = auto() + TLSv1_3 = auto() + MAXIMUM_SUPPORTED = auto() + + +Errors +~~~~~~ + +This module would define four base classes for use with error handling. Unlike +many of the other classes defined here, these classes are not abstract, as +they have no behaviour. They exist simply to signal certain common behaviours. +Backends should subclass these exceptions in their own packages, but needn't +define any behaviour for them. + +In general, concrete implementations should subclass these exceptions rather +than throw them directly. This makes it moderately easier to determine which +concrete TLS implementation is in use during debugging of unexpected errors. +However, this is not mandatory. + +The definitions of the errors are below:: + + class TLSError(Exception): + """ + The base exception for all TLS related errors from any backend. + Catching this error should be sufficient to catch *all* TLS errors, + regardless of what backend is used. + """ + + class WantWriteError(TLSError): + """ + A special signaling exception used only when non-blocking or + buffer-only I/O is used. This error signals that the requested + operation cannot complete until more data is written to the network, + or until the output buffer is drained. + + This error is should only be raised when it is completely impossible + to write any data. If a partial write is achievable then this should + not be raised. + """ + + class WantReadError(TLSError): + """ + A special signaling exception used only when non-blocking or + buffer-only I/O is used. This error signals that the requested + operation cannot complete until more data is read from the network, or + until more data is available in the input buffer. + + This error should only be raised when it is completely impossible to + write any data. If a partial write is achievable then this should not + be raised. + """ + + class RaggedEOF(TLSError): + """ + A special signaling exception used when a TLS connection has been + closed gracelessly: that is, when a TLS CloseNotify was not received + from the peer before the underlying TCP socket reached EOF. This is a + so-called "ragged EOF". + + This exception is not guaranteed to be raised in the face of a ragged + EOF: some implementations may not be able to detect or report the + ragged EOF. + + This exception is not always a problem. Ragged EOFs are a concern only + when protocols are vulnerable to length truncation attacks. Any + protocol that can detect length truncation attacks at the application + layer (e.g. HTTP/1.1 and HTTP/2) is not vulnerable to this kind of + attack and so can ignore this exception. + """ + + +Certificates +~~~~~~~~~~~~ + +This module would define an abstract X509 certificate class. This class would +have almost no behaviour, as the goal of this module is not to provide all +possible relevant cryptographic functionality that could be provided by X509 +certificates. Instead, all we need is the ability to signal the source of a +certificate to a concrete implementation. + +For that reason, this certificate implementation defines only constructors. In +essence, the certificate object in this module could be as abstract as a handle +that can be used to locate a specific certificate. + +Concrete implementations may choose to provide alternative constructors, e.g. +to load certificates from HSMs. If a common interface emerges for doing this, +this module may be updated to provide a standard constructor for this use-case +as well. + +Concrete implementations should aim to have Certificate objects be hashable if +at all possible. This will help ensure that TLSConfiguration objects used with +an individual concrete implementation are also hashable. + +:: + + class Certificate(metaclass=ABCMeta): + @abstractclassmethod + def from_buffer(cls, buffer: bytes): + """ + Creates a Certificate object from a byte buffer. This byte buffer + may be either PEM-encoded or DER-encoded. If the buffer is PEM + encoded it *must* begin with the standard PEM preamble (a series of + dashes followed by the ASCII bytes "BEGIN CERTIFICATE" and another + series of dashes). In the absence of that preamble, the + implementation may assume that the certificate is DER-encoded + instead. + """ + + @abstractclassmethod + def from_file(cls, path: Union[pathlib.Path, AnyStr]): + """ + Creates a Certificate object from a file on disk. This method may + be a convenience method that wraps ``open`` and ``from_buffer``, + but some TLS implementations may be able to provide more-secure or + faster methods of loading certificates that do not involve Python + code. + """ + + +Private Keys +~~~~~~~~~~~~ + +This module would define an abstract private key class. Much like the +Certificate class, this class has almost no behaviour in order to give as much +freedom as possible to the concrete implementations to treat keys carefully. + +This class has all the caveats of the ``Certificate`` class. + +:: + + class PrivateKey(metaclass=ABCMeta): + @abstractclassmethod + def from_buffer(cls, + buffer: bytes, + password: Optional[Union[Callable[[], Union[bytes, bytearray]], bytes, bytearray]] = None): + """ + Creates a PrivateKey object from a byte buffer. This byte buffer + may be either PEM-encoded or DER-encoded. If the buffer is PEM + encoded it *must* begin with the standard PEM preamble (a series of + dashes followed by the ASCII bytes "BEGIN", the key type, and + another series of dashes). In the absence of that preamble, the + implementation may assume that the certificate is DER-encoded + instead. + + The key may additionally be encrypted. If it is, the ``password`` + argument can be used to decrypt the key. The ``password`` argument + may be a function to call to get the password for decrypting the + private key. It will only be called if the private key is encrypted + and a password is necessary. It will be called with no arguments, + and it should return either bytes or bytearray containing the + password. Alternatively a bytes, or bytearray value may be supplied + directly as the password argument. It will be ignored if the + private key is not encrypted and no password is needed. + """ + + @abstractclassmethod + def from_file(cls, + path: Union[pathlib.Path, bytes, str], + password: Optional[Union[Callable[[], Union[bytes, bytearray]], bytes, bytearray]] = None): + """ + Creates a PrivateKey object from a file on disk. This method may + be a convenience method that wraps ``open`` and ``from_buffer``, + but some TLS implementations may be able to provide more-secure or + faster methods of loading certificates that do not involve Python + code. + + The ``password`` parameter behaves exactly as the equivalent + parameter on ``from_buffer``. + """ + + +Trust Store +~~~~~~~~~~~ + +As discussed above, loading a trust store represents an issue because different +TLS implementations vary wildly in how they allow users to select trust stores. +For this reason, we need to provide a model that assumes very little about the +form that trust stores take. + +This problem is the same as the one that the Certificate and PrivateKey types +need to solve. For this reason, we use the exact same model, by creating an +opaque type that can encapsulate the various means that TLS backends may open +a trust store. + +A given TLS implementation is not required to implement all of the +constructors. However, it is strongly recommended that a given TLS +implementation provide the ``system`` constructor if at all possible, as this +is the most common validation trust store that is used. Concrete +implementations may also add their own constructors. + +Concrete implementations should aim to have TrustStore objects be hashable if +at all possible. This will help ensure that TLSConfiguration objects used with +an individual concrete implementation are also hashable. + +:: + + class TrustStore(metaclass=ABCMeta): + @abstractclassmethod + def system(cls) -> TrustStore: + """ + Returns a TrustStore object that represents the system trust + database. + """ + + @abstractclassmethod + def from_pem_file(cls, path: Union[pathlib.Path, bytes, str]) -> TrustStore: + """ + Initializes a trust store from a single file full of PEMs. + """ + + +Runtime Access +~~~~~~~~~~~~~~ + +A not-uncommon use case for library users is to want to allow the library to +control the TLS configuration, but to want to select what backend is in use. +For example, users of Requests may want to be able to select between OpenSSL or +a platform-native solution on Windows and macOS, or between OpenSSL and NSS on +some Linux platforms. These users, however, may not care about exactly how +their TLS configuration is done. + +This poses a problem: given an arbitrary concrete implementation, how can a +library work out how to load certificates into the trust store? There are two +options: either all concrete implementations can be required to fit into a +specific naming scheme, or we can provide an API that makes it possible to grab +these objects. + +This PEP proposes that we use the second approach. This grants the greatest +freedom to concrete implementations to structure their code as they see fit, +requiring only that they provide a single object that has the appropriate +properties in place. Users can then pass this "backend" object to libraries +that support it, and those libraries can take care of configuring and using the +concrete implementation. + +All concrete implementations must provide a method of obtaining a ``Backend`` +object. The ``Backend`` object can be a global singleton or can be created by a +callable if there is an advantage in doing that. + +The ``Backend`` object has the following definition:: + + Backend = namedtuple( + 'Backend', + ['client_context', 'server_context', + 'certificate', 'private_key', 'trust_store'] + ) + +Each of the properties must provide the concrete implementation of the relevant +ABC. This ensures that code like this will work for any backend:: + + trust_store = backend.trust_store.system() + + +Changes to the Standard Library +=============================== + +The portions of the standard library that interact with TLS should be revised +to use these ABCs. This will allow them to function with other TLS backends. +This includes the following modules: + +- asyncio +- ftplib +- http +- imaplib +- nntplib +- poplib +- smtplib +- urllib + + +Migration of the ssl module +--------------------------- + +Naturally, we will need to extend the ``ssl`` module itself to conform to these +ABCs. This extension will take the form of new classes, potentially in an +entirely new module. This will allow applications that take advantage of the +current ``ssl`` module to continue to do so, while enabling the new APIs for +applications and libraries that want to use them. + +In general, migrating from the ``ssl`` module to the new ABCs is not expected +to be one-to-one. This is normally acceptable: most tools that use the ``ssl`` +module hide it from the user, and so refactoring to use the new module should +be invisible. + +However, a specific problem comes from libraries or applications that leak +exceptions from the ``ssl`` module, either as part of their defined API or by +accident (which is easily done). Users of those tools may have written code +that tolerates and handles exceptions from the ``ssl`` module being raised: +migrating to the ABCs presented here would potentially cause the exceptions +defined above to be thrown instead, and existing ``except`` blocks will not +catch them. + +For this reason, part of the migration of the ``ssl`` module would require that +the exceptions in the ``ssl`` module alias those defined above. That is, they +would require the following statements to all succeed:: + + assert ssl.SSLError is tls.TLSError + assert ssl.SSLWantReadError is tls.WantReadError + assert ssl.SSLWantWriteError is tls.WantWriteError + +The exact mechanics of how this will be done are beyond the scope of this PEP, +as they are made more complex due to the fact that the current ``ssl`` +exceptions are defined in C code, but more details can be found in +`an email sent to the Security-SIG by Christian Heimes`_. + + +Future +====== + +Major future TLS features may require revisions of these ABCs. These revisions +should be made cautiously: many backends may not be able to move forward +swiftly, and will be invalidated by changes in these ABCs. This is acceptable, +but wherever possible features that are specific to individual implementations +should not be added to the ABCs. The ABCs should restrict themselves to +high-level descriptions of IETF-specified features. + +However, well-justified extensions to this API absolutely should be made. The +focus of this API is to provide a unifying lowest-common-denominator +configuration option for the Python community. TLS is not a static target, and +as TLS evolves so must this API. + + +Credits +======= + +This document has received extensive review from a number of individuals in the +community who have substantially helped shape it. Detailed review was provided +by: + +* Alex Chan +* Alex Gaynor +* Antoine Pitrou +* Ashwini Oruganti +* Donald Stufft +* Ethan Furman +* Glyph +* Hynek Schlawack +* Jim J Jewett +* Nathaniel J. Smith +* Alyssa Coghlan +* Paul Kehrer +* Steve Dower +* Steven Fackler +* Wes Turner +* Will Bond + +Further review was provided by the Security-SIG and python-ideas mailing lists. + + +Copyright +========= + +This document has been placed in the public domain. + + +.. _ssl module: https://docs.python.org/3/library/ssl.html +.. _OpenSSL Library: https://www.openssl.org/ +.. _PyOpenSSL: https://pypi.org/project/pyOpenSSL/ +.. _certifi: https://pypi.org/project/certifi/ +.. _SSLContext: https://docs.python.org/3/library/ssl.html#ssl.SSLContext +.. _SSLSocket: https://docs.python.org/3/library/ssl.html#ssl.SSLSocket +.. _SSLObject: https://docs.python.org/3/library/ssl.html#ssl.SSLObject +.. _SSLError: https://docs.python.org/3/library/ssl.html#ssl.SSLError +.. _MSDN articles: https://msdn.microsoft.com/en-us/library/windows/desktop/mt490158(v=vs.85).aspx +.. _TLS enum script: https://github.com/tiran/tlsdb/blob/master/tlspep_ciphersuite.py +.. _tlsdb JSON file: https://github.com/tiran/tlsdb/blob/master/tlsdb.json +.. _IANA cipher suite registry: https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-4 +.. _an email sent to the Security-SIG by Christian Heimes: https://mail.python.org/pipermail/security-sig/2017-January/000213.html +.. _s2n: https://github.com/awslabs/s2n +.. _working to add it: https://github.com/awslabs/s2n/issues/358 diff --git a/decisiongrounding/scenarios_candidates/peps_tls_api_supersession/corpus/PEP-0748.md b/decisiongrounding/scenarios_candidates/peps_tls_api_supersession/corpus/PEP-0748.md new file mode 100644 index 0000000..1d019b1 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_tls_api_supersession/corpus/PEP-0748.md @@ -0,0 +1,1609 @@ +--- +schema_version: 1 +id: PEP-0748 +type: decision +tags: [pep] +--- + +# PEP-0748 — A Unified TLS API for Python + +## Status + +Draft + +## Context + +Public decision ingested verbatim from python/peps (PEP 748) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0748.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0748.rst). Upstream +status: Draft. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0543 + +## Source Text + +PEP: 748 +Title: A Unified TLS API for Python +Author: Joop van de Pol , + William Woodruff +Sponsor: Alyssa Coghlan +Discussions-To: https://discuss.python.org/t/pre-pep-discussion-revival-of-pep-543/51263 +Status: Draft +Type: Standards Track +Created: 27-Jun-2024 +Python-Version: 3.14 +Post-History: `17-Apr-2024 `__ +Replaces: 543 + +Abstract +======== + +This PEP defines a standard TLS interface in the form of a collection of +protocol classes. This interface will allow Python implementations and +third-party libraries to provide bindings to TLS libraries other than OpenSSL. + +These bindings can be used by tools that expect the interface provided by the +Python standard library, with the goal of reducing the dependence of the Python +ecosystem on OpenSSL. + +Rationale +========= + +It has become increasingly clear that robust and user-friendly TLS support is an +extremely important part of the ecosystem of any popular programming language. +For most of its lifetime, this role in the Python ecosystem has primarily been +served by the :mod:`ssl` module, which provides a Python API to the `OpenSSL +library `_. + +Because the :mod:`ssl` module is distributed with the Python standard library, +it has become the overwhelmingly most popular method for handling TLS in Python. +A majority of Python libraries, both in the standard library and +on the Python Package Index, rely on the :mod:`ssl` module for their TLS +connectivity. + +Unfortunately, the preeminence of the :mod:`ssl` module has had a number of +tied the entire Python +ecosystem tightly to OpenSSL. This has forced Python users to use OpenSSL even +in situations where it may provide a worse user experience than alternative TLS +implementations, which imposes a cognitive burden and makes it hard to provide +“platform-native” experiences. + +Problems +-------- + +The fact that the :mod:`ssl` module is built into the standard library has meant +that all standard-library Python networking libraries are entirely reliant on +the OpenSSL that the Python implementation has been linked against. This leads +to the following issues: + +* It is difficult to take advantage of new, higher-security TLS without + recompiling Python to get a new OpenSSL. While there are third-party bindings + to OpenSSL (e.g. `pyOpenSSL `_), these + need to be shimmed into a format that the standard library understands, + forcing projects that want to use them to maintain substantial compatibility + layers. + +* Windows distributions of Python need to be shipped with a copy of + OpenSSL. This puts the CPython development team in the position of being + OpenSSL redistributors, potentially needing to ship security updates to the + Windows Python distributions when OpenSSL vulnerabilities are released. + +* macOS distributions of Python need either to be shipped with a copy + of OpenSSL or linked against the system OpenSSL library. Apple has formally + deprecated linking against the system OpenSSL library, and even if they had + not, that library version has been unsupported by upstream for nearly one year + as of the time of writing. The CPython development team has started shipping + newer OpenSSLs with the Python available from python.org, but this has the + same problem as with Windows. + +* Users may wish to integrate with TLS libraries other than OpenSSL for other + reasons, such as maintenance burden versus a system-provided implementation, + or because OpenSSL is simply too large and unwieldy for their platform (e.g. + for embedded Python). Those users are left with the requirement to use + third-party networking libraries that can interact with their preferred TLS + library or to shim their preferred library into the OpenSSL-specific + :mod:`ssl` module API. + +Additionally, the :mod:`ssl` module as implemented today limits the ability of +CPython itself to add support for alternative TLS implementations, or remove +OpenSSL support entirely, should either of these become necessary or useful. The +:mod:`ssl` module exposes too many OpenSSL-specific function calls and features +to easily map to an alternative TLS implementation. + +Proposal +======== + +This PEP proposes to introduce a few new Protocol Classes in Python 3.14 to +provide TLS functionality that is not so strongly tied to OpenSSL. It also +proposes to update standard library modules to use only the interface exposed by +these protocol classes wherever possible. There are three goals here: + +#. To provide a common API surface for both core and third-party developers to + target their TLS implementations to. This allows TLS developers to provide + interfaces that can be used by most Python code, and allows network + developers to have an interface that they can target that will work with a + wide range of TLS implementations. + +#. To provide an API that has few or no OpenSSL-specific concepts leak through. + The :mod:`ssl` module today has a number of warts caused by leaking OpenSSL + concepts through to the API: the new protocol classes would remove those + specific concepts. + +#. To provide a path for the core development team to make OpenSSL one of many + possible TLS implementations, rather than requiring that it be present on a + system in order for Python to have TLS support. + +The proposed interface is laid out below. + +Interfaces +---------- + +There are several interfaces that require standardization. Those interfaces are: + +#. Configuring TLS, currently implemented by the :class:`~ssl.SSLContext` class + in the :mod:`ssl` module. + +#. Providing an in-memory buffer for doing in-memory encryption or decryption + with no actual I/O (necessary for asynchronous I/O models), currently + implemented by the :class:`~ssl.SSLObject` class in the :mod:`ssl` module. + +#. Wrapping a socket object, currently implemented by the + :class:`~ssl.SSLSocket` class in the :mod:`ssl` module. + +#. Applying TLS configuration to the wrapping objects in (2) and (3). Currently + this is also implemented by the SSLContext class in the :mod:`ssl` module. + +#. Specifying TLS cipher suites. There is currently no code for doing this in + the standard library: instead, the standard library uses OpenSSL cipher suite + strings. + +#. Specifying application-layer protocols that can be negotiated during the TLS + handshake. + +#. Specifying TLS versions. + +#. Reporting errors to the caller, currently implemented by the + :class:`~ssl.SSLError` class in the :mod:`ssl` module. + +#. Specifying certificates to load, either as client or server certificates. + +#. Specifying which trust database should be used to validate certificates + presented by a remote peer. + +#. Finding a way to get hold of these interfaces at run time. + +For the sake of simplicity, this PEP proposes to remove interfaces (3) and (4), +and replace them by a simpler interface that returns a socket which ensures that +all communication through the socket is protected by TLS. In other words, this +interface treats concepts such as socket initialization, the TLS handshake, +Server Name Indication (SNI), etc., as an atomic part of creating a client or +server connection. However, in-memory buffers are still supported, as they are +useful for asynchronous communication. + +Obviously, (5) doesn't require a protocol class: instead, it requires a richer +API for configuring supported cipher suites that can be easily updated with +supported cipher suites for different implementations. + +(9) is a thorny problem, because in an ideal world the private keys associated +with these certificates would never end up in-memory in the Python process +(that is, the TLS library would collaborate with a Hardware Security Module +(HSM) to provide the private key in such a way that it cannot be extracted +from process memory). Thus, we need to provide an extensible model of +providing certificates that allows concrete implementations the ability to +provide this higher level of security, while also allowing a lower bar for +those implementations that cannot. This lower bar would be the same as the +status quo: that is, the certificate may be loaded from an in-memory buffer, +from a file on disk, or additionally referenced by some arbitrary ID +corresponding to a system certificate store. + +(10) also represents an issue because different TLS implementations vary wildly +in how they allow users to select trust stores. Some implementations have +specific trust store formats that only they can use (such as the OpenSSL CA +directory format that is created by c_rehash), and others may not allow you +to specify a trust store that does not include their default trust store. +On the other hand, most implementations will support some form of loading custom +DER- or PEM-encoded certificates. + +For this reason, we need to provide a model that assumes very little about the +form that trust stores take, while maintaining type-compatibility with other +implementations. The sections “Certificate”, “Private Keys”, and “Trust Store” +below go into more detail about how this is achieved. + +Finally, this API will split the responsibilities currently assumed by the +:class:`~ssl.SSLContext` object: specifically, the responsibility for holding +and managing configuration and the responsibility for using that configuration +to build buffers or sockets. + +This is necessary primarily for supporting functionality like Server Name +Indication (SNI). In OpenSSL (and thus in the :mod:`ssl` module), the server has +the ability to modify the TLS configuration in response to the client telling +the server what hostname it is trying to reach. This is mostly used to change +the certificate chain so as to present the correct TLS certificate chain for the +given hostname. The specific mechanism by which this is done is by returning a +new :class:`~ssl.SSLContext` object with the appropriate configuration as part +of a user-provided SNI callback function. + +This is not a model that maps well to other TLS implementations, and puts a +burden on users to write callback functions. Instead, we propose that the +concrete implementations handle SNI transparently for every user after receiving +the relevant certificates. + +For this reason, we split the responsibility of :class:`~ssl.SSLContext` into +two separate objects, which are each split into server and client versions. The +``TLSServerConfiguration`` and ``TLSClientConfiguration`` objects act as +containers for a TLS configuration: the ClientContext and ServerContext objects +are instantiated with a ``TLSClientConfiguration`` and +``TLSServerConfiguration`` object, respectively, and are used to create buffers +or sockets. All four objects would be immutable. + +.. note:: + + The following API declarations uniformly use type hints to aid reading. + +Configuration +~~~~~~~~~~~~~ + +The ``TLSServerConfiguration`` and ``TLSClientConfiguration`` concrete classes +define objects that can hold and manage TLS configuration. The goals of these +classes are as follows: + +#. To provide a method of specifying TLS configuration that avoids the risk of + errors in typing (this excludes the use of a simple dictionary). + +#. To provide an object that can be safely compared to other configuration + objects to detect changes in TLS configuration, for use with the SNI + callback. + +These classes are not protocol classes, primarily because they are not expected to +have implementation-specific behavior. The responsibility for transforming a +``TLSServerConfiguration`` or ``TLSClientConfiguration`` object into a useful +set of configurations for a given TLS implementation belongs to the Context +objects discussed below. + +These classes have one other notable property: they are immutable. This is a +desirable trait for a few reasons. The most important one is that immutability +by default is a good engineering practice. As a side benefit, it allows these +objects to be used as dictionary keys, which is potentially useful for specific +TLS implementations and their SNI configuration. On top of this, it frees +implementations from needing to worry about their configuration objects being +changed under their feet, which allows them to avoid needing to carefully +synchronize changes between their concrete data structures and the configuration +object. + +These objects are extendable: that is, future releases of Python may add +configuration fields to these objects as they become useful. For +backwards-compatibility purposes, new fields are only appended to these objects. +Existing fields will never be removed, renamed, or reordered. They are split +between client and server to minimize API confusion. + +The ``TLSClientConfiguration`` class would be defined by the following code: + +.. code-block:: python + + class TLSClientConfiguration: + __slots__ = ( + "_certificate_chain", + "_ciphers", + "_inner_protocols", + "_lowest_supported_version", + "_highest_supported_version", + "_trust_store", + ) + + def __init__( + self, + certificate_chain: SigningChain | None = None, + ciphers: Sequence[CipherSuite] | None = None, + inner_protocols: Sequence[NextProtocol | bytes] | None = None, + lowest_supported_version: TLSVersion | None = None, + highest_supported_version: TLSVersion | None = None, + trust_store: TrustStore | None = None, + ) -> None: + if inner_protocols is None: + inner_protocols = [] + + self._certificate_chain = certificate_chain + self._ciphers = ciphers + self._inner_protocols = inner_protocols + self._lowest_supported_version = lowest_supported_version + self._highest_supported_version = highest_supported_version + self._trust_store = trust_store + + @property + def certificate_chain(self) -> SigningChain | None: + return self._certificate_chain + + @property + def ciphers(self) -> Sequence[CipherSuite | int] | None: + return self._ciphers + + @property + def inner_protocols(self) -> Sequence[NextProtocol | bytes]: + return self._inner_protocols + + @property + def lowest_supported_version(self) -> TLSVersion | None: + return self._lowest_supported_version + + @property + def highest_supported_version(self) -> TLSVersion | None: + return self._highest_supported_version + + @property + def trust_store(self) -> TrustStore | None: + return self._trust_store + +The ``TLSServerConfiguration`` object is similar to the client one, except that +it takes a ``Sequence[SigningChain]`` as the ``certificate_chain`` parameter. + +Context +~~~~~~~ + +We define two Context protocol classes. These protocol classes define objects +that allow configuration of TLS to be applied to specific connections. They can +be thought of as factories for ``TLSSocket`` and ``TLSBuffer`` objects. + +Unlike the current :mod:`ssl` module, we provide two context classes instead of +one. Specifically, we provide the ``ClientContext`` and ``ServerContext`` +classes. This simplifies the APIs (for example, there is no sense in the server +providing the ``server_hostname`` parameter to +:meth:`~ssl.SSLContext.wrap_socket`, but because there is only one context class +that parameter is still available), and ensures that implementations know as +early as possible which side of a TLS connection they will serve. Additionally, +it allows implementations to opt-out of one or either side of the connection. + +As much as possible implementers should aim to make these classes immutable: +that is, they should prefer not to allow users to mutate their internal state +directly, instead preferring to create new contexts from new TLSConfiguration +objects. Obviously, the protocol classes cannot enforce this constraint, and so +they do not attempt to. + +The ``ClientContext`` protocol class has the following class definition: + +.. code-block:: python + + class ClientContext(Protocol): + @abstractmethod + def __init__(self, configuration: TLSClientConfiguration) -> None: + """Create a new client context object from a given TLS client configuration.""" + ... + + @property + @abstractmethod + def configuration(self) -> TLSClientConfiguration: + """Returns the TLS client configuration that was used to create the client context.""" + ... + + @abstractmethod + def connect(self, address: tuple[str | None, int]) -> TLSSocket: + """Creates a TLSSocket that behaves like a socket.socket, and + contains information about the TLS exchange + (cipher, negotiated_protocol, negotiated_tls_version, etc.). + """ + ... + + @abstractmethod + def create_buffer(self, server_hostname: str) -> TLSBuffer: + """Creates a TLSBuffer that acts as an in-memory channel, + and contains information about the TLS exchange + (cipher, negotiated_protocol, negotiated_tls_version, etc.).""" + ... + +The ``ServerContext`` is similar, taking a ``TLSServerConfiguration`` instead. + +Socket +~~~~~~ + +The context can be used to create sockets, which have to follow the +specification of the ``TLSSocket`` protocol class. Specifically, implementations +need to implement the following: + +* ``recv`` and ``send`` +* ``listen`` and ``accept`` +* ``close`` +* ``getsockname`` +* ``getpeername`` + +They also need to implement some interfaces that give information about the TLS +connection, such as: + +* The underlying context object that was used to create this socket +* The negotiated cipher +* The negotiated "next" protocol +* The negotiated TLS version + +The following code describes these functions in more detail: + +.. code-block:: python + + class TLSSocket(Protocol): + """This class implements a socket.socket-like object that creates an OS + socket, wraps it in an SSL context, and provides read and write methods + over that channel.""" + + @abstractmethod + def __init__(self, *args: tuple, **kwargs: tuple) -> None: + """TLSSockets should not be constructed by the user. + The implementation should implement a method to construct a TLSSocket + object and call it in ClientContext.connect() and + ServerContext.connect().""" + ... + + @abstractmethod + def recv(self, bufsize: int) -> bytes: + """Receive data from the socket. The return value is a bytes object + representing the data received. Should not work before the handshake + is completed.""" + ... + + @abstractmethod + def send(self, bytes: bytes) -> int: + """Send data to the socket. The socket must be connected to a remote socket.""" + ... + + @abstractmethod + def close(self, force: bool = False) -> None: + """Shuts down the connection and mark the socket closed. + If force is True, this method should send the close_notify alert and shut down + the socket without waiting for the other side. + If force is False, this method should send the close_notify alert and raise + the WantReadError exception until a corresponding close_notify alert has been + received from the other side. + In either case, this method should return WantWriteError if sending the + close_notify alert currently fails.""" + ... + + @abstractmethod + def listen(self, backlog: int) -> None: + """Enable a server to accept connections. If backlog is specified, it + specifies the number of unaccepted connections that the system will allow + before refusing new connections.""" + ... + + @abstractmethod + def accept(self) -> tuple[TLSSocket, tuple[str | None, int]]: + """Accept a connection. The socket must be bound to an address and listening + for connections. The return value is a pair (conn, address) where conn is a + new TLSSocket object usable to send and receive data on the connection, and + address is the address bound to the socket on the other end of the connection.""" + ... + + @abstractmethod + def getsockname(self) -> tuple[str | None, int]: + """Return the local address to which the socket is connected.""" + ... + + @abstractmethod + def getpeercert(self) -> bytes | None: + """ + Return the raw DER bytes of the certificate provided by the peer + during the handshake, if applicable. + """ + ... + + @abstractmethod + def getpeername(self) -> tuple[str | None, int]: + """Return the remote address to which the socket is connected.""" + ... + + @property + @abstractmethod + def context(self) -> ClientContext | ServerContext: + """The ``Context`` object this socket is tied to.""" + ... + + @abstractmethod + def cipher(self) -> CipherSuite | int | None: + """ + Returns the CipherSuite entry for the cipher that has been negotiated on the connection. + + If no connection has been negotiated, returns ``None``. If the cipher negotiated is not + defined in CipherSuite, returns the 16-bit integer representing that cipher directly. + """ + ... + + @abstractmethod + def negotiated_protocol(self) -> NextProtocol | bytes | None: + """ + Returns the protocol that was selected during the TLS handshake. + + This selection may have been made using ALPN or some future + negotiation mechanism. + + If the negotiated protocol is one of the protocols defined in the + ``NextProtocol`` enum, the value from that enum will be returned. + Otherwise, the raw bytestring of the negotiated protocol will be + returned. + + If ``Context.set_inner_protocols()`` was not called, if the other + party does not support protocol negotiation, if this socket does + not support any of the peer's proposed protocols, or if the + handshake has not happened yet, ``None`` is returned. + """ + ... + + @property + @abstractmethod + def negotiated_tls_version(self) -> TLSVersion | None: + """The version of TLS that has been negotiated on this connection.""" + ... + +Buffer +~~~~~~ + +The context can also be used to create buffers, which have to follow the +specification of the ``TLSBuffer`` protocol class. Specifically, implementations +need to implement the following: + +* ``read`` and ``write`` +* ``do_handshake`` +* ``shutdown`` +* ``process_incoming`` and ``process_outgoing`` +* ``incoming_bytes_buffered`` and ``outgoing_bytes_buffered`` +* ``getpeercert`` + +Similarly to the socket case, they also need to implement some interfaces that +give information about the TLS connection, such as: + +* The underlying context object that was used to create this buffer +* The negotiated cipher +* The negotiated "next" protocol +* The negotiated TLS version + +The following code describes these functions in more detail: + +.. code-block:: python + + class TLSBuffer(Protocol): + """This class implements an in memory-channel that creates two buffers, + wraps them in an SSL context, and provides read and write methods over + that channel.""" + + @abstractmethod + def read(self, amt: int, buffer: Buffer | None) -> bytes | int: + """ + Read up to ``amt`` bytes of data from the input buffer and return + the result as a ``bytes`` instance. If an optional buffer is + provided, the result is written into the buffer and the number of + bytes is returned instead. + + Once EOF is reached, all further calls to this method return the + empty byte string ``b''``. + + May read "short": that is, fewer bytes may be returned than were + requested. + + Raise ``WantReadError`` or ``WantWriteError`` if there is + insufficient data in either the input or output buffer and the + operation would have caused data to be written or read. + + May raise ``RaggedEOF`` if the connection has been closed without a + graceful TLS shutdown. Whether this is an exception that should be + ignored or not is up to the specific application. + + As at any time a re-negotiation is possible, a call to ``read()`` + can also cause write operations. + """ + ... + + @abstractmethod + def write(self, buf: Buffer) -> int: + """ + Write ``buf`` in encrypted form to the output buffer and return the + number of bytes written. The ``buf`` argument must be an object + supporting the buffer interface. + + Raise ``WantReadError`` or ``WantWriteError`` if there is + insufficient data in either the input or output buffer and the + operation would have caused data to be written or read. In either + case, users should endeavour to resolve that situation and then + re-call this method. When re-calling this method users *should* + re-use the exact same ``buf`` object, as some implementations require that + the exact same buffer be used. + + This operation may write "short": that is, fewer bytes may be + written than were in the buffer. + + As at any time a re-negotiation is possible, a call to ``write()`` + can also cause read operations. + """ + ... + + @abstractmethod + def do_handshake(self) -> None: + """ + Performs the TLS handshake. Also performs certificate validation + and hostname verification. + """ + ... + + @abstractmethod + def cipher(self) -> CipherSuite | int | None: + """ + Returns the CipherSuite entry for the cipher that has been + negotiated on the connection. If no connection has been negotiated, + returns ``None``. If the cipher negotiated is not defined in + CipherSuite, returns the 16-bit integer representing that cipher + directly. + """ + ... + + @abstractmethod + def negotiated_protocol(self) -> NextProtocol | bytes | None: + """ + Returns the protocol that was selected during the TLS handshake. + This selection may have been made using ALPN, NPN, or some future + negotiation mechanism. + + If the negotiated protocol is one of the protocols defined in the + ``NextProtocol`` enum, the value from that enum will be returned. + Otherwise, the raw bytestring of the negotiated protocol will be + returned. + + If ``Context.set_inner_protocols()`` was not called, if the other + party does not support protocol negotiation, if this socket does + not support any of the peer's proposed protocols, or if the + handshake has not happened yet, ``None`` is returned. + """ + ... + + @property + @abstractmethod + def context(self) -> ClientContext | ServerContext: + """ + The ``Context`` object this buffer is tied to. + """ + ... + + @property + @abstractmethod + def negotiated_tls_version(self) -> TLSVersion | None: + """ + The version of TLS that has been negotiated on this connection. + """ + ... + + @abstractmethod + def shutdown(self) -> None: + """ + Performs a clean TLS shut down. This should generally be used + whenever possible to signal to the remote peer that the content is + finished. + """ + ... + + @abstractmethod + def process_incoming(self, data_from_network: bytes) -> None: + """ + Receives some TLS data from the network and stores it in an + internal buffer. + + If the internal buffer is overfull, this method will raise + ``WantReadError`` and store no data. At this point, the user must + call ``read`` to remove some data from the internal buffer + before repeating this call. + """ + ... + + @abstractmethod + def incoming_bytes_buffered(self) -> int: + """ + Returns how many bytes are in the incoming buffer waiting to be processed. + """ + ... + + @abstractmethod + def process_outgoing(self, amount_bytes_for_network: int) -> bytes: + """ + Returns the next ``amt`` bytes of data that should be written to + the network from the outgoing data buffer, removing it from the + internal buffer. + """ + ... + + @abstractmethod + def outgoing_bytes_buffered(self) -> int: + """ + Returns how many bytes are in the outgoing buffer waiting to be sent. + """ + ... + + @abstractmethod + def getpeercert(self) -> bytes | None: + """ + Return the raw DER bytes of the certificate provided by the peer + during the handshake, if applicable. + """ + ... + + +Cipher Suites +~~~~~~~~~~~~~ + +Supporting cipher suites in a truly library-agnostic fashion is a remarkably +difficult undertaking. Different TLS implementations often have radically +different APIs for specifying cipher suites, but more problematically these APIs +frequently differ in capability as well as in style. + +Below are examples of different cipher suite selection APIs. These examples are +not intended to obligate implementation against each API, only to illuminate the +constraints imposed by each. + +OpenSSL +^^^^^^^ + +OpenSSL uses a well-known cipher string format. This format has been adopted as +a configuration language by most products that use OpenSSL, including Python. +This format is relatively easy to read, but has a number of downsides: it is a +string, which makes it easy to provide bad inputs; it lacks much +detailed validation, meaning that it is possible to configure OpenSSL in a way +that doesn't allow it to negotiate any cipher at all; and it allows specifying +cipher suites in a number of different ways that make it tricky to parse. The +biggest problem with this format is that there is no formal specification for +it, meaning that the only way to parse a given string the way OpenSSL would is +to get OpenSSL to parse it. + +OpenSSL's cipher strings can look like this: + +.. code-block:: python + + "ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5" + + +This string demonstrates some of the complexity of the OpenSSL format. For +example, it is possible for one entry to specify multiple cipher suites: the +entry ``ECDH+AESGCM`` means “all ciphers suites that include both elliptic-curve +Diffie-Hellman key exchange and AES in Galois Counter Mode”. More explicitly, +that will expand to four cipher suites: + + +.. code-block:: python + + "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256" + + +That makes parsing a complete OpenSSL cipher string extremely tricky. Add to the +fact that there are other meta-characters, such as “!” (exclude all cipher +suites that match this criterion, even if they would otherwise be included: +“!MD5” means that no cipher suites using the MD5 hash algorithm should be +included), “-” (exclude matching ciphers if they were already included, but +allow them to be re-added later if they get included again), and “+” (include +the matching ciphers, but place them at the end of the list), and you get an +extremely complex format to parse. On top of this complexity it should be noted +that the actual result depends on the OpenSSL version, as an OpenSSL cipher +string is valid so long as it contains at least one cipher that OpenSSL +recognizes. + +OpenSSL also uses different names for its ciphers than the names used in the +relevant specifications. See the manual page for ``ciphers(1)`` for more +details. + +The actual API inside OpenSSL for the cipher string is simple: + +.. code-block:: c + + char *cipher_list = ; + int rc = SSL_CTX_set_cipher_list(context, cipher_list); + + +This means that any format that is used by this module must be able to be +converted to an OpenSSL cipher string for use with OpenSSL. + +Network Framework +^^^^^^^^^^^^^^^^^ + +Network Framework is the macOS (10.15+) system TLS library. This library is +substantially more restricted than OpenSSL in many ways, as it has a much more +restricted class of users. One of these substantial restrictions is in +controlling supported cipher suites. + +Ciphers in Network Framework are represented by a Objective-C ``uint16_t`` enum. +This enum has one entry per cipher suite, with no aggregate entries, meaning +that it is not possible to reproduce the meaning of an OpenSSL cipher string +like ``“ECDH+AESGCM”`` without hand-coding which categories each enum member +falls into. + +However, the names of most of the enum members are in line with the formal names +of the cipher suites: that is, the cipher suite that OpenSSL calls +``“ECDHE-ECDSA-AES256-GCM-SHA384”`` is called +``“tls_ciphersuite_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384”`` in Network Framework. + +The API for configuring cipher suites inside Network Framework is simple: + +.. code-block:: c + + void sec_protocol_options_append_tls_ciphersuite(sec_protocol_options_t options, tls_ciphersuite_t ciphersuite); + +SChannel +^^^^^^^^ + +SChannel is the Windows system TLS library. + +SChannel has extremely restrictive support for controlling available TLS cipher +suites, and additionally adopts a third method of expressing what TLS cipher +suites are supported. + +Specifically, SChannel defines a set of ``ALG_ID`` constants (C unsigned ints). +Each of these constants does not refer to an entire cipher suite, but instead an +individual algorithm. Some examples are ``CALG_3DES`` and ``CALG_AES_256``, +which refer to the bulk encryption algorithm used in a cipher suite, +``CALG_ECDH_EPHEM`` and ``CALG_RSA_KEYX`` which refer to part of the key +exchange algorithm used in a cipher suite, ``CALG_SHA_256`` and ``CALG_SHA_384`` +which refer to the message authentication code used in a cipher suite, and +``CALG_ECDSA`` and ``CALG_RSA_SIGN`` which refer to the signing portions of the +key exchange algorithm. + +In earlier versions of the SChannel API, these constants were used to define the +algorithms that could be used. The latest version, however, uses these constants +to prohibit which algorithms can be used. + +This can be thought of as the half of OpenSSL's functionality that Network +Framework doesn't have: Network Framework only allows specifying exact cipher +suites (and a limited number of pre-defined cipher suite groups), whereas +SChannel only allows specifying parts of the cipher suite, while OpenSSL allows +both. + +Determining which cipher suites are allowed on a given connection is done by +providing a pointer to an array of these ``ALG_ID`` constants. This means that +any suitable API must allow the Python code to determine which ``ALG_ID`` +constants must be provided. + +Network Security Services (NSS) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +NSS is Mozilla's crypto and TLS library. It's used in Firefox, Thunderbird, and +as an alternative to OpenSSL in multiple libraries, e.g. curl. + +By default, NSS comes with secure configuration of allowed ciphers. On some +platforms such as Fedora, the list of enabled ciphers is globally configured in +a system policy. Generally, applications should not modify cipher suites unless +they have specific reasons to do so. + +NSS has both process global and per-connection settings for cipher suites. It +does not have a concept of :class:`~ssl.SSLContext` like OpenSSL. A +:class:`~ssl.SSLContext`-like behavior can be easily emulated. Specifically, +ciphers can be enabled or disabled globally with +``SSL_CipherPrefSetDefault(PRInt32 cipher, PRBool enabled)``, and +``SSL_CipherPrefSet(PRFileDesc *fd, PRInt32 cipher, PRBool enabled)`` for a +connection. The cipher ``PRInt32`` number is a signed 32-bit integer that +directly corresponds to an registered IANA id, e.g. ``0x1301`` is +``TLS_AES_128_GCM_SHA256``. Contrary to OpenSSL, the preference order of ciphers +is fixed and cannot be modified at runtime. + +Like Network Framework, NSS has no API for aggregated entries. Some consumers of +NSS have implemented custom mappings from OpenSSL cipher names and rules to NSS +ciphers, e.g. ``mod_nss``. + +Proposed Interface +^^^^^^^^^^^^^^^^^^ + +The proposed interface for the new module is influenced by the combined set of +limitations of the above implementations. Specifically, as every implementation +except OpenSSL requires that each individual cipher be provided, there is no +option but to provide that lowest common denominator approach. + +The simplest approach is to provide an enumerated type that includes a large +subset of the cipher suites defined for TLS. The values of the enum members will +be their two-octet cipher identifier as used in the TLS handshake, stored as a +16 bit integer. The names of the enum members will be their IANA-registered +cipher suite names. + +As of now, the `IANA cipher suite registry +`_ +contains over 320 cipher suites. A large portion of the cipher suites are +irrelevant for TLS connections to network services. Other suites specify +deprecated and insecure algorithms that are no longer provided by recent +versions of implementations. The enum contains the five fixed cipher suites +defined for TLS v1.3. For TLS v1.2, it only contains the cipher suites that +correspond to the TLS v1.3 cipher suites, with ECDHE key exchange (for perfect +forward secrecy) and ECDSA or RSA signatures, which are an additional ten cipher +suites. + +In addition to this enum, the interface defines a default cipher suite list for +TLS v1.2, which includes only those defined cipher suites based on AES-GCM or +ChaCha20-Poly1305. The default cipher suite list for TLS v1.3 will +comprise the five cipher suites defined in the specification. + +The current enum is quite restricted, including only cipher suites that provide +forward secrecy. Because the enum doesn't contain every defined cipher, and also +to allow for forward-looking applications, all parts of this API that accept +``CipherSuite`` objects will also accept raw 16-bit integers directly. + +.. code-block:: python + + class CipherSuite(IntEnum): + """ + Known cipher suites. + + See: + """ + + TLS_AES_128_GCM_SHA256 = 0x1301 + TLS_AES_256_GCM_SHA384 = 0x1302 + TLS_CHACHA20_POLY1305_SHA256 = 0x1303 + TLS_AES_128_CCM_SHA256 = 0x1304 + TLS_AES_128_CCM_8_SHA256 = 0x1305 + TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 = 0xC02B + TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 = 0xC02C + TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 = 0xC02F + TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 = 0xC030 + TLS_ECDHE_ECDSA_WITH_AES_128_CCM = 0xC0AC + TLS_ECDHE_ECDSA_WITH_AES_256_CCM = 0xC0AD + TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8 = 0xC0AE + TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8 = 0xC0AF + TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 = 0xCCA8 + TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 = 0xCCA9 + + +For Network Framework, these enum members directly refer to the values of the +cipher suite constants. For example, Network Framework defines the cipher suite +enum member ``tls_ciphersuite_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`` as having +the value ``0xC02C``. Not coincidentally, that is identical to its value in the +above enum. This makes mapping between Network Framework and the above enum very +easy indeed. + +For SChannel there is no easy direct mapping, due to the fact that SChannel +configures ciphers, instead of cipher suites. This represents an ongoing concern +with SChannel, which is that it is very difficult to configure in a specific +manner compared to other TLS implementations. + +For the purposes of this PEP, any SChannel implementation will need to determine +which ciphers to choose based on the enum members. This may be more open than +the actual cipher suite list actually wants to allow, or it may be more +restrictive, depending on the choices of the implementation. This PEP recommends +that it be more restrictive, but of course this cannot be enforced. + +Finally, we expect that for most users, secure defaults will be enough. When +specifying no list of ciphers, the implementations should use secure defaults +(possibly derived from system recommended settings). + +Protocol Negotiation +~~~~~~~~~~~~~~~~~~~~ + +ALPN allows for protocol negotiation as part of the HTTP/2 handshake. While ALPN +is at a fundamental level built on top of bytestrings, string-based APIs are +frequently problematic as they allow for errors in typing that can be hard to +detect. + +For this reason, this module will define a type that protocol negotiation +implementations can pass and be passed. This type would wrap a bytestring to +allow for aliases for well-known protocols. This allows us to avoid the problems +inherent in typos for well-known protocols, while allowing the full +extensibility of the protocol negotiation layer if needed by letting users pass +byte strings directly. + +.. code-block:: python + + class NextProtocol(Enum): + """The underlying negotiated ("next") protocol.""" + + H2 = b"h2" + H2C = b"h2c" + HTTP1 = b"http/1.1" + WEBRTC = b"webrtc" + C_WEBRTC = b"c-webrtc" + FTP = b"ftp" + STUN = b"stun.nat-discovery" + TURN = b"stun.turn" + +TLS Versions +~~~~~~~~~~~~ + +It is often useful to be able to restrict the versions of TLS you're willing to +support. There are many security advantages in refusing to use old versions of +TLS, and some misbehaving servers will mishandle TLS clients advertising support +for newer versions. + +The following enumerated type can be used to gate TLS versions. Forward-looking +applications should almost never set a maximum TLS version unless they +absolutely must, as a TLS implementation that is newer than the Python that uses +it may support TLS versions that are not in this enumerated type. + +Additionally, this enumerated type defines two additional flags that can always +be used to request either the lowest or highest TLS version supported by an +implementation. As for cipher suites, we expect that for most users, secure +defaults will be enough. When specifying no list of TLS versions, the +implementations should use secure defaults (possibly derived from system +recommended settings). + +.. code-block:: python + + class TLSVersion(Enum): + """ + TLS versions. + + The `MINIMUM_SUPPORTED` and `MAXIMUM_SUPPORTED` variants are "open ended", + and refer to the "lowest mutually supported" and "highest mutually supported" + TLS versions, respectively. + """ + + MINIMUM_SUPPORTED = "MINIMUM_SUPPORTED" + TLSv1_2 = "TLSv1.2" + TLSv1_3 = "TLSv1.3" + MAXIMUM_SUPPORTED = "MAXIMUM_SUPPORTED" + +Errors +~~~~~~ + +This module would define four base classes for use with error handling. Unlike +many of the other classes defined here, these classes are not abstract, as they +have no behavior. They exist simply to signal certain common behaviors. TLS +implementations should subclass these exceptions in their own packages, but +needn't define any behavior for them. + +In general, concrete implementations should subclass these exceptions rather +than throw them directly. This makes it moderately easier to determine which +concrete TLS implementation is in use during debugging of unexpected errors. +However, this is not mandatory. + +The definitions of the errors are below: + +.. code-block:: python + + class TLSError(Exception): + """ + The base exception for all TLS related errors from any implementation. + + Catching this error should be sufficient to catch *all* TLS errors, + regardless of what implementation is used. + """ + + + class WantWriteError(TLSError): + """ + A special signaling exception used only when non-blocking or buffer-only I/O is used. + + This error signals that the requested + operation cannot complete until more data is written to the network, + or until the output buffer is drained. + + This error is should only be raised when it is completely impossible + to write any data. If a partial write is achievable then this should + not be raised. + """ + + + class WantReadError(TLSError): + """ + A special signaling exception used only when non-blocking or buffer-only I/O is used. + + This error signals that the requested + operation cannot complete until more data is read from the network, or + until more data is available in the input buffer. + + This error should only be raised when it is completely impossible to + write any data. If a partial write is achievable then this should not + be raised. + """ + + + class RaggedEOF(TLSError): + """A special signaling exception used when a TLS connection has been + closed gracelessly: that is, when a TLS CloseNotify was not received + from the peer before the underlying TCP socket reached EOF. This is a + so-called "ragged EOF". + + This exception is not guaranteed to be raised in the face of a ragged + EOF: some implementations may not be able to detect or report the + ragged EOF. + + This exception is not always a problem. Ragged EOFs are a concern only + when protocols are vulnerable to length truncation attacks. Any + protocol that can detect length truncation attacks at the application + layer (e.g. HTTP/1.1 and HTTP/2) is not vulnerable to this kind of + attack and so can ignore this exception. + """ + + + class ConfigurationError(TLSError): + """An special exception that implementations can use when the provided + configuration uses features not supported by that implementation.""" + + +Certificates +~~~~~~~~~~~~ + +This module would define a concrete certificate class. This class would have +almost no behavior, as the goal of this module is not to provide all possible +relevant cryptographic functionality that could be provided by X.509 +certificates. Instead, all we need is the ability to signal the source of a +certificate to a concrete implementation. + +For that reason, this certificate class defines three attributes, corresponding +to the three envisioned constructors: certificates from files, certificates from +memory, or certificates from arbitrary identifiers. It is possible that +implementations do not support all of these constructors, and they can +communicate this to users as described in the “Runtime” section below. +Certificates from arbitrary identifiers, in particular, are expected to be +useful primarily to users seeking to build integrations on top of HSMs, TPMs, +SSMs, and similar. + +Specifically, this class does not parse any provided input to validate that it +is a correct certificate, and also does not provide any form of introspection +into a particular certificate. TLS implementations are not required to provide +such introspection either. Peer certificates that are received during the +handshake are provided as raw DER bytes. + +.. code-block:: python + + class Certificate: + """Object representing a certificate used in TLS.""" + + __slots__ = ( + "_buffer", + "_path", + "_id", + ) + + def __init__( + self, buffer: bytes | None = None, path: os.PathLike[str] | None = None, id: bytes | None = None + ): + """ + Creates a Certificate object from a path, buffer, or ID. + + If none of these is given, an exception is raised. + """ + + if buffer is None and path is None and id is None: + raise ValueError("Certificate cannot be empty.") + + self._buffer = buffer + self._path = path + self._id = id + + @classmethod + def from_buffer(cls, buffer: bytes) -> Certificate: + """ + Creates a Certificate object from a byte buffer. This byte buffer + may be either PEM-encoded or DER-encoded. If the buffer is PEM + encoded it *must* begin with the standard PEM preamble (a series of + dashes followed by the ASCII bytes "BEGIN CERTIFICATE" and another + series of dashes). In the absence of that preamble, the + implementation may assume that the certificate is DER-encoded + instead. + """ + return cls(buffer=buffer) + + @classmethod + def from_file(cls, path: os.PathLike[str]) -> Certificate: + """ + Creates a Certificate object from a file on disk. The file on disk + should contain a series of bytes corresponding to a certificate that + may be either PEM-encoded or DER-encoded. If the bytes are PEM encoded + it *must* begin with the standard PEM preamble (a series of dashes + followed by the ASCII bytes "BEGIN CERTIFICATE" and another series of + dashes). In the absence of that preamble, the implementation may + assume that the certificate is DER-encoded instead. + """ + return cls(path=path) + + @classmethod + def from_id(cls, id: bytes) -> Certificate: + """ + Creates a Certificate object from an arbitrary identifier. This may + be useful for implementations that rely on system certificate stores. + """ + return cls(id=id) + +Private Keys +~~~~~~~~~~~~ + +This module would define a concrete private key class. Much like the +``Certificate`` class, this class has three attributes to correspond to the +three constructors, and further has all the caveats of the ``Certificate`` +class. + +.. code-block:: python + + class PrivateKey: + """Object representing a private key corresponding to a public key + for a certificate used in TLS.""" + + __slots__ = ( + "_buffer", + "_path", + "_id", + ) + + def __init__( + self, buffer: bytes | None = None, path: os.PathLike | None = None, id: bytes | None = None + ): + """ + Creates a PrivateKey object from a path, buffer, or ID. + + If none of these is given, an exception is raised. + """ + + if buffer is None and path is None and id is None: + raise ValueError("PrivateKey cannot be empty.") + + self._buffer = buffer + self._path = path + self._id = id + + @classmethod + def from_buffer(cls, buffer: bytes) -> PrivateKey: + """ + Creates a PrivateKey object from a byte buffer. This byte buffer + may be either PEM-encoded or DER-encoded. If the buffer is PEM + encoded it *must* begin with the standard PEM preamble (a series of + dashes followed by the ASCII bytes "BEGIN", the key type, and + another series of dashes). In the absence of that preamble, the + implementation may assume that the private key is DER-encoded + instead. + """ + return cls(buffer=buffer) + + @classmethod + def from_file(cls, path: os.PathLike) -> PrivateKey: + """ + Creates a PrivateKey object from a file on disk. The file on disk + should contain a series of bytes corresponding to a certificate that + may be either PEM-encoded or DER-encoded. If the bytes are PEM encoded + it *must* begin with the standard PEM preamble (a series of dashes + followed by the ASCII bytes "BEGIN", the key type, and another series + of dashes). In the absence of that preamble, the implementation may + assume that the certificate is DER-encoded instead. + """ + return cls(path=path) + + @classmethod + def from_id(cls, id: bytes) -> PrivateKey: + """ + Creates a PrivateKey object from an arbitrary identifier. This may + be useful for implementations that rely on system private key stores. + """ + return cls(id=id) + +Signing Chain +~~~~~~~~~~~~~ + +In order to authenticate themselves, TLS participants need to provide a leaf +certificate with a chain leading up to some root certificate that is trusted by +the other side. Servers always need to authenticate themselves to clients, but +clients can also authenticate themselves to servers during client +authentication. Additionally, the leaf certificate must be accompanied by a +private key, which can either be stored in a separate object, or together with +the leaf certificate itself. This module defines the collection of these objects +as a ``SigningChain`` as detailed below: + +.. code-block:: python + + class SigningChain: + """Object representing a certificate chain used in TLS.""" + + leaf: tuple[Certificate, PrivateKey | None] + chain: list[Certificate] + + def __init__( + self, + leaf: tuple[Certificate, PrivateKey | None], + chain: Sequence[Certificate] | None = None, + ): + """Initializes a SigningChain object.""" + self.leaf = leaf + if chain is None: + chain = [] + self.chain = list(chain) + +As shown in the configuration classes above, a client can have one signing chain +in the case of client authentication or none otherwise. A server can have a +sequence of signing chains, which is useful when it is responsible for multiple +domains. + +Trust Store +~~~~~~~~~~~ + +As discussed above, loading a trust store represents an issue because different +TLS implementations vary wildly in how they allow users to select trust stores. +For this reason, we need to provide a model that assumes very little about the +form that trust stores take. + +This problem is the same as the one that the ``Certificate`` and ``PrivateKey`` +types need to solve. For this reason, we use the exact same model, by creating a +concrete class that captures the various means of how users could define a trust +store. + +A given TLS implementation is not required to handle all possible trust stores. +However, it is strongly recommended that a given TLS implementation handles the +``system`` constructor if at all possible, as this is the most common validation +trust store that is used. TLS implementations can communicate unsupported +options as described in the “Runtime” section below. + +.. code-block:: python + + class TrustStore: + """ + The trust store that is used to verify certificate validity. + """ + + __slots__ = ( + "_buffer", + "_path", + "_id", + ) + + def __init__( + self, buffer: bytes | None = None, path: os.PathLike | None = None, id: bytes | None = None + ): + """ + Creates a TrustStore object from a path, buffer, or ID. + + If none of these is given, the default system trust store is used. + """ + + self._buffer = buffer + self._path = path + self._id = id + + @classmethod + def system(cls) -> TrustStore: + """ + Returns a TrustStore object that represents the system trust + database. + """ + return cls() + + @classmethod + def from_buffer(cls, buffer: bytes) -> TrustStore: + """ + Initializes a trust store from a buffer of PEM-encoded certificates. + """ + return cls(buffer=buffer) + + @classmethod + def from_file(cls, path: os.PathLike) -> TrustStore: + """ + Initializes a trust store from a single file containing PEMs. + """ + return cls(path=path) + + @classmethod + def from_id(cls, id: bytes) -> TrustStore: + """ + Initializes a trust store from an arbitrary identifier. + """ + return cls(id=id) + +Runtime Access +~~~~~~~~~~~~~~ + +A not-uncommon use case is for library users to want to specify the TLS +implementation to use while allowing the library to configure the details of the +actual TLS connection. For example, users of :pypi:`requests` may want to be +able to select between OpenSSL or a platform-native solution on Windows and +macOS, or between OpenSSL and NSS on some Linux platforms. These users, however, +may not care about exactly how their TLS configuration is done. + +This poses two problems: given an arbitrary concrete implementation, how can a +library: + +* Work out whether the implementation supports particular constructors for certificates + or trust stores (e.g. from arbitrary identifiers)? + +* Get the correct types for the two context classes? + +Constructing certificate and trust store objects should be possible outside of +the implementation. Therefore, the implementations need to provide a way for +users to verify whether the implementation is compatible with user-constructed +certificates and trust stores. Therefore, each implementation should implement a +``validate_config`` method that takes a ``TLSClientConfiguration`` or +``TLSServerConfiguration`` object and raises an exception if unsupported +constructors were used. + +For the types, there are two options: either all concrete implementations can be +required to fit into a specific naming scheme, or we can provide an API that +makes it possible to grab these objects. + +This PEP proposes that we use the second approach. This grants the greatest +freedom to concrete implementations to structure their code as they see fit, +requiring only that they provide a single object that has the appropriate +properties in place. Users can then pass this implementation object to libraries +that support it, and those libraries can take care of configuring and using the +concrete implementation. + +All concrete implementations must provide a method of obtaining a +``TLSImplementation`` object. The ``TLSImplementation`` object can be a global +singleton or can be created by a callable if there is an advantage in doing +that. + +The ``TLSImplementation`` object has the following definition: + +.. code-block:: python + + class TLSImplementation(Generic[_ClientContext, _ServerContext]): + __slots__ = ( + "_client_context", + "_server_context", + "_validate_config", + ) + + def __init__( + self, + client_context: type[_ClientContext], + server_context: type[_ServerContext], + validate_config: Callable[[TLSClientConfiguration | TLSServerConfiguration], None], + ) -> None: + self._client_context = client_context + self._server_context = server_context + self._validate_config = validate_config + +The first two properties must provide the concrete implementation of the +relevant Protocol class. For example, for the client context: + +.. code-block:: python + + @property + def client_context(self) -> type[_ClientContext]: + """The concrete implementation of the PEP 543 Client Context object, + if this TLS implementation supports being the client on a TLS connection. + """ + return self._client_context + +This ensures that code like this will work for any implementation: + +.. code-block:: python + + client_config = TLSClientConfiguration() + client_context = implementation.client_context(client_config) + +The third property must provide a function that verifies whether a given TLS +configuration contains implementation-compatible certificates, private keys, and +a trust store: + +.. code-block:: python + + @property + def validate_config(self) -> Callable[[TLSClientConfiguration | TLSServerConfiguration], None]: + """A function that reveals whether this TLS implementation supports a + particular TLS configuration. + """ + return self._validate_config + +Note that this function only needs to verify that supported constructors were +used for the certificates, private keys, and trust store. It does not need to +parse or retrieve the objects to validate them further. + +Insecure Usage +-------------- + +All of the above assumes that users want to use the module in a secure way. +Sometimes, users want to do imprudent things like disable certificate validation +for testing purposes. To this end, we propose a separate ``insecure`` module +that allows users to do this. This module contains insecure variants of the +configuration, context, and implementation objects, which allow to disable +certificate validation as well as the server hostname check. + +This functionality is placed in a separate module to make it as hard as possible +for legitimate users to accidentally use the insecure functionality. +Additionally, it defines a new warning called ``SecurityWarning``, and loudly +warns at every step of the way when trying to create an insecure connection. + +This module is only intended for testing purposes. In real-world situations +where a user wants to connect to some IoT device which only has a self-signed +certificate, it is strongly recommended to add this certificate into a custom +trust store, rather than using the insecure module to disable certificate +validation. + +Changes to the Standard Library +=============================== + +The portions of the standard library that interact with TLS should be revised to +use these Protocol classes. This will allow them to function with other TLS +implementations. This includes the following modules: + +* :mod:`asyncio` +* :mod:`ftplib` +* :mod:`http` +* :mod:`imaplib` +* :mod:`nntplib` +* :mod:`poplib` +* :mod:`smtplib` +* :mod:`urllib` + +Migration of the ssl module +--------------------------- + +Naturally, we will need to extend the :mod:`ssl` module itself to conform to +these Protocol classes. This extension will take the form of new classes, +potentially in an entirely new module. This will allow applications that take +advantage of the current :mod:`ssl` module to continue to do so, while enabling +the new APIs for applications and libraries that want to use them. + +In general, migrating from the :mod:`ssl` module to the new Protocol classes is +not expected to be one-to-one. This is normally acceptable: most tools that use +the :mod:`ssl` module hide it from the user, and so refactoring to use the new +module should be invisible. + +However, a specific problem comes from libraries or applications that leak +exceptions from the :mod:`ssl` module, either as part of their defined API or by +accident (which is easily done). Users of those tools may have written code that +tolerates and handles exceptions from the :mod:`ssl` module being raised: +migrating to the protocol classes presented here would potentially cause the +exceptions defined above to be thrown instead, and existing ``except`` blocks +will not catch them. + +For this reason, part of the migration of the :mod:`ssl` module would require +that the exceptions in the :mod:`ssl` module alias those defined above. That is, +they would require the following statements to all succeed: + +.. code-block:: python + + assert ssl.SSLError is tls.TLSError + assert ssl.SSLWantReadError is tls.WantReadError + assert ssl.SSLWantWriteError is tls.WantWriteError + + +The exact mechanics of how this will be done are beyond the scope of this PEP, +as they are made more complex due to the fact that the current ssl exceptions +are defined in C code, but more details can be found in `an email sent to the +Security-SIG by Christian Heimes +`_. + +Future +====== + +Major future TLS features may require revisions of these protocol classes. These +revisions should be made cautiously: many implementations may not be able to +move forward swiftly, and will be invalidated by changes in these protocol +classes. This is acceptable, but wherever possible features that are specific to +individual implementations should not be added to the protocol classes. The +protocol classes should restrict themselves to high-level descriptions of +IETF-specified features. + +However, well-justified extensions to this API absolutely should be made. The +focus of this API is to provide a unifying lowest-common-denominator +configuration option for the Python community. TLS is not a static target, and +as TLS evolves so must this API. + +Credits +======= + +This PEP is adapted substantially from :pep:`543`, which was withdrawn in 2020. +:pep:`543` was authored by Cory Benfield and Christian Heimes, and received +extensive review from a number of individuals in the community who have +substantially helped shape it. Detailed review for both :pep:`543` and this +PEP was provided by: + +* Alex Chan +* Alex Gaynor +* Antoine Pitrou +* Ashwini Oruganti +* Donald Stufft +* Ethan Furman +* Glyph +* Hynek Schlawack +* Jim J Jewett +* Nathaniel J. Smith +* Alyssa Coghlan +* Paul Kehrer +* Steve Dower +* Steven Fackler +* Wes Turner +* Will Bond +* Cory Benfield +* Marc-André Lemburg +* Seth M. Larson +* Victor Stinner +* Ronald Oussoren + +Further review of :pep:`543` was provided by the Security-SIG and python-ideas +mailing lists. + + +Copyright +========= + +This document is placed in the public domain or under the CC0-1.0-Universal +license, whichever is more permissive. diff --git a/decisiongrounding/scenarios_candidates/peps_tls_api_supersession/provenance.json b/decisiongrounding/scenarios_candidates/peps_tls_api_supersession/provenance.json new file mode 100644 index 0000000..0a88566 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_tls_api_supersession/provenance.json @@ -0,0 +1,39 @@ +{ + "source_repo": "python/peps", + "pinned_commit": "f866e77409305866038471574f075cd8d83eee9e", + "peps": [ + { + "id": "PEP-0543", + "number": 543, + "file": "corpus/PEP-0543.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0543.rst", + "source_sha256": "8f722767412c3f9aeea322ebaaf96bafd13722accf66a9a0db2d773befbd29d4", + "status": "Withdrawn", + "title": "A Unified TLS API for Python", + "replaces": [], + "superseded_by": [ + "PEP-0748" + ] + }, + { + "id": "PEP-0748", + "number": 748, + "file": "corpus/PEP-0748.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0748.rst", + "source_sha256": "8c9c79066e6e3d63dd2b3db1159e5b0106f345ec3530921d81e16f9a71dbe2e8", + "status": "Draft", + "title": "A Unified TLS API for Python", + "replaces": [ + "PEP-0543" + ], + "superseded_by": [] + } + ], + "supersedes_edges": [ + { + "source": "PEP-0748", + "type": "supersedes", + "target": "PEP-0543" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/peps_typeis_narrowing_supersession/corpus/PEP-0724.md b/decisiongrounding/scenarios_candidates/peps_typeis_narrowing_supersession/corpus/PEP-0724.md new file mode 100644 index 0000000..e247fd0 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_typeis_narrowing_supersession/corpus/PEP-0724.md @@ -0,0 +1,380 @@ +--- +schema_version: 1 +id: PEP-0724 +type: decision +tags: [pep, typing] +--- + +# PEP-0724 — Stricter Type Guards + +## Status + +Withdrawn + +## Context + +Public decision ingested verbatim from python/peps (PEP 724) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0724.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0724.rst). Upstream +status: Withdrawn. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 724 +Title: Stricter Type Guards +Author: Rich Chiodo , + Eric Traut , + Erik De Bonte , +Sponsor: Jelle Zijlstra +Discussions-To: https://discuss.python.org/t/pep-724-stricter-type-guards/34124 +Status: Withdrawn +Type: Standards Track +Topic: Typing +Created: 28-Jul-2023 +Python-Version: 3.13 +Post-History: `30-Dec-2021 `__, + `19-Sep-2023 `__, + +Status +====== + +This PEP is withdrawn. The Typing Council was unable to reach consensus on +this proposal, and the authors decided to withdraw it. + + +Abstract +======== + +:pep:`647` introduced the concept of a user-defined type guard function which +returns ``True`` if the type of the expression passed to its first parameter +matches its return ``TypeGuard`` type. For example, a function that has a +return type of ``TypeGuard[str]`` is assumed to return ``True`` if and only if +the type of the expression passed to its first input parameter is a ``str``. +This allows type checkers to narrow types when a user-defined type guard +function returns ``True``. + +This PEP refines the ``TypeGuard`` mechanism introduced in :pep:`647`. It +allows type checkers to narrow types when a user-defined type guard function +returns ``False``. It also allows type checkers to apply additional (more +precise) type narrowing under certain circumstances when the type guard +function returns ``True``. + + +Motivation +========== + +User-defined type guard functions enable a type checker to narrow the type of +an expression when it is passed as an argument to the type guard function. The +``TypeGuard`` mechanism introduced in :pep:`647` is flexible, but this +flexibility imposes some limitations that developers have found inconvenient +for some uses. + +Limitation 1: Type checkers are not allowed to narrow a type in the case where +the type guard function returns ``False``. This means the type is not narrowed +in the negative ("else") clause. + +Limitation 2: Type checkers must use the ``TypeGuard`` return type if the type +guard function returns ``True`` regardless of whether additional narrowing can +be applied based on knowledge of the pre-narrowed type. + +The following code sample demonstrates both of these limitations. + +.. code-block:: python + + def is_iterable(val: object) -> TypeGuard[Iterable[Any]]: + return isinstance(val, Iterable) + + def func(val: int | list[int]): + if is_iterable(val): + # The type is narrowed to 'Iterable[Any]' as dictated by + # the TypeGuard return type + reveal_type(val) # Iterable[Any] + else: + # The type is not narrowed in the "False" case + reveal_type(val) # int | list[int] + + # If "isinstance" is used in place of the user-defined type guard + # function, the results differ because type checkers apply additional + # logic for "isinstance" + + if isinstance(val, Iterable): + # Type is narrowed to "list[int]" because this is + # a narrower (more precise) type than "Iterable[Any]" + reveal_type(val) # list[int] + else: + # Type is narrowed to "int" because the logic eliminates + # "list[int]" from the original union + reveal_type(val) # int + + +:pep:`647` imposed these limitations so it could support use cases where the +return ``TypeGuard`` type was not a subtype of the input type. Refer to +:pep:`647` for examples. + +Rationale +========= + +There are a number of issues where a stricter ``TypeGuard`` would have +been a solution: + +* `Python typing issue - TypeGuard doesn't intersect like isinstance `__ +* `Pyright issue - TypeGuard not eliminating possibility on branch `__ +* `Pyright issue - Type narrowing for Literal doesn't work `__ +* `Mypy issue - TypeGuard is incompatible with exhaustive check `__ +* `Mypy issue - Incorrect type narrowing for inspect.isawaitable `__ +* `Typeshed issue - asyncio.iscoroutinefunction is not a TypeGuard `__ + + +Specification +============= + +The use of a user-defined type guard function involves five types: + +* I = ``TypeGuard`` input type +* R = ``TypeGuard`` return type +* A = Type of argument passed to type guard function (pre-narrowed) +* NP = Narrowed type (positive) +* NN = Narrowed type (negative) + +.. code-block:: python + + def guard(x: I) -> TypeGuard[R]: ... + + def func1(val: A): + if guard(val): + reveal_type(val) # NP + else: + reveal_type(val) # NN + + +This PEP proposes some modifications to :pep:`647` to address the limitations +discussed above. These limitations are safe to eliminate only when a specific +condition is met. In particular, when the output type ``R`` of a user-defined +type guard function is consistent [#isconsistent]_ with the type of its first +input parameter (``I``), type checkers should apply stricter type guard +semantics. + + .. code-block:: python + + # Stricter type guard semantics are used in this case because + # "Kangaroo | Koala" is consistent with "Animal" + def is_marsupial(val: Animal) -> TypeGuard[Kangaroo | Koala]: + return isinstance(val, Kangaroo | Koala) + + # Stricter type guard semantics are not used in this case because + # "list[T]"" is not consistent with "list[T | None]" + def has_no_nones(val: list[T | None]) -> TypeGuard[list[T]]: + return None not in val + +When stricter type guard semantics are applied, the application of a +user-defined type guard function changes in two ways. + +* Type narrowing is applied in the negative ("else") case. + +.. code-block:: python + + def is_str(val: str | int) -> TypeGuard[str]: + return isinstance(val, str) + + def func(val: str | int): + if not is_str(val): + reveal_type(val) # int + +* Additional type narrowing is applied in the positive "if" case if applicable. + +.. code-block:: python + + def is_cardinal_direction(val: str) -> TypeGuard[Literal["N", "S", "E", "W"]]: + return val in ("N", "S", "E", "W") + + def func(direction: Literal["NW", "E"]): + if is_cardinal_direction(direction): + reveal_type(direction) # "Literal[E]" + else: + reveal_type(direction) # "Literal[NW]" + + +The type-theoretic rules for type narrowing are specified in the following +table. + +============ ======================= =================== +\ Non-strict type guard Strict type guard +============ ======================= =================== +Applies when R not consistent with I R consistent with I +NP is .. :math:`R` :math:`A \land R` +NN is .. :math:`A` :math:`A \land \neg{R}` +============ ======================= =================== + +In practice, the theoretic types for strict type guards cannot be expressed +precisely in the Python type system. Type checkers should fall back on +practical approximations of these types. As a rule of thumb, a type checker +should use the same type narrowing logic -- and get results that are consistent +with -- its handling of "isinstance". This guidance allows for changes and +improvements if the type system is extended in the future. + + +Additional Examples +=================== + +``Any`` is consistent [#isconsistent]_ with any other type, which means +stricter semantics can be applied. + +.. code-block:: python + + # Stricter type guard semantics are used in this case because + # "str" is consistent with "Any" + def is_str(x: Any) -> TypeGuard[str]: + return isinstance(x, str) + + def test(x: float | str): + if is_str(x): + reveal_type(x) # str + else: + reveal_type(x) # float + + +Backwards Compatibility +======================= + +This PEP proposes to change the existing behavior of ``TypeGuard``. This has no +effect at runtime, but it does change the types evaluated by a type checker. + +.. code-block:: python + + def is_int(val: int | str) -> TypeGuard[int]: + return isinstance(val, int) + + def func(val: int | str): + if is_int(val): + reveal_type(val) # "int" + else: + reveal_type(val) # Previously "int | str", now "str" + + +This behavioral change results in different types evaluated by a type checker. +It could therefore produce new (or mask existing) type errors. + +Type checkers often improve narrowing logic or fix existing bugs in such logic, +so users of static typing will be used to this type of behavioral change. + +We also hypothesize that it is unlikely that existing typed Python code relies +on the current behavior of ``TypeGuard``. To validate our hypothesis, we +implemented the proposed change in pyright and ran this modified version on +roughly 25 typed code bases using `mypy primer`__ to see if there were any +differences in the output. As predicted, the behavioral change had minimal +impact. The only noteworthy change was that some ``# type: ignore`` comments +were no longer necessary, indicating that these code bases were already working +around the existing limitations of ``TypeGuard``. + +__ https://github.com/hauntsaninja/mypy_primer + +Breaking change +--------------- + +It is possible for a user-defined type guard function to rely on the old +behavior. Such type guard functions could break with the new behavior. + +.. code-block:: python + + def is_positive_int(val: int | str) -> TypeGuard[int]: + return isinstance(val, int) and val > 0 + + def func(val: int | str): + if is_positive_int(val): + reveal_type(val) # "int" + else: + # With the older behavior, the type of "val" is evaluated as + # "int | str"; with the new behavior, the type is narrowed to + # "str", which is perhaps not what was intended. + reveal_type(val) + +We think it is unlikely that such user-defined type guards exist in real-world +code. The mypy primer results didn't uncover any such cases. + + +How to Teach This +================= + +Users unfamiliar with ``TypeGuard`` are likely to expect the behavior outlined +in this PEP, therefore making ``TypeGuard`` easier to teach and explain. + + +Reference Implementation +======================== + +A reference `implementation`__ of this idea exists in pyright. + +__ https://github.com/microsoft/pyright/commit/9a5af798d726bd0612cebee7223676c39cf0b9b0 + +To enable the modified behavior, the configuration flag +``enableExperimentalFeatures`` must be set to true. This can be done on a +per-file basis by adding a comment: + +.. code-block:: python + + # pyright: enableExperimentalFeatures=true + + +Rejected Ideas +============== + +StrictTypeGuard +--------------- + +A new ``StrictTypeGuard`` construct was proposed. This alternative form would +be similar to a ``TypeGuard`` except it would apply stricter type guard +semantics. It would also enforce that the return type was consistent +[#isconsistent]_ with the input type. See this thread for details: +`StrictTypeGuard proposal`__ + +__ https://github.com/python/typing/discussions/1013#discussioncomment-1966238 + +This idea was rejected because it is unnecessary in most cases and added +unnecessary complexity. It would require the introduction of a new special +form, and developers would need to be educated about the subtle difference +between the two forms. + +TypeGuard with a second output type +----------------------------------- + +Another idea was proposed where ``TypeGuard`` could support a second optional +type argument that indicates the type that should be used for narrowing in the +negative ("else") case. + +.. code-block:: python + + def is_int(val: int | str) -> TypeGuard[int, str]: + return isinstance(val, int) + + +This idea was proposed `here`__. + +__ https://github.com/python/typing/issues/996 + +It was rejected because it was considered too complicated and addressed only +one of the two main limitations of ``TypeGuard``. Refer to this `thread`__ for +the full discussion. + +__ https://mail.python.org/archives/list/typing-sig@python.org/thread/EMUD2D424OI53DCWQ4H5L6SJD2IXBHUL + + +Footnotes +========= + +.. [#isconsistent] :pep:`PEP 483's discussion of is-consistent <483#summary-of-gradual-typing>` + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. diff --git a/decisiongrounding/scenarios_candidates/peps_typeis_narrowing_supersession/corpus/PEP-0742.md b/decisiongrounding/scenarios_candidates/peps_typeis_narrowing_supersession/corpus/PEP-0742.md new file mode 100644 index 0000000..b2e2e84 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_typeis_narrowing_supersession/corpus/PEP-0742.md @@ -0,0 +1,530 @@ +--- +schema_version: 1 +id: PEP-0742 +type: decision +tags: [pep, typing] +--- + +# PEP-0742 — Narrowing types with TypeIs + +## Status + +Final + +## Context + +Public decision ingested verbatim from python/peps (PEP 742) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0742.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0742.rst). Upstream +status: Final. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0724 + +## Source Text + +PEP: 742 +Title: Narrowing types with TypeIs +Author: Jelle Zijlstra +Discussions-To: https://discuss.python.org/t/pep-742-narrowing-types-with-typenarrower/45613 +Status: Final +Type: Standards Track +Topic: Typing +Created: 07-Feb-2024 +Python-Version: 3.13 +Post-History: `11-Feb-2024 `__ +Replaces: 724 +Resolution: `03-Apr-2024 `__ + +.. canonical-typing-spec:: :ref:`typing:typeis` and + :external+py3.13:data:`typing.TypeIs` + +Abstract +======== + +This PEP proposes a new special form, ``TypeIs``, to allow annotating functions that can be used +to narrow the type of a value, similar to the builtin :py:func:`isinstance`. Unlike the existing +:py:data:`typing.TypeGuard` special form, ``TypeIs`` can narrow the type in both the ``if`` +and ``else`` branches of a conditional. + + +Motivation +========== + +Typed Python code often requires users to narrow the type of a variable based on a conditional. +For example, if a function accepts a union of two types, it may use an :py:func:`isinstance` check +to discriminate between the two types. Type checkers commonly support type narrowing based on various +builtin function and operations, but occasionally, it is useful to use a user-defined function to +perform type narrowing. + +To support such use cases, :pep:`647` introduced the :py:data:`typing.TypeGuard` special form, which +allows users to define type guards:: + + from typing import assert_type, TypeGuard + + def is_str(x: object) -> TypeGuard[str]: + return isinstance(x, str) + + def f(x: object) -> None: + if is_str(x): + assert_type(x, str) + else: + assert_type(x, object) + +Unfortunately, the behavior of :py:data:`typing.TypeGuard` has some limitations that make it +less useful for many common use cases, as explained also in the "Motivation" section of :pep:`724`. +In particular: + +* Type checkers must use exactly the ``TypeGuard`` return type as the narrowed type if the + type guard returns ``True``. They cannot use pre-existing knowledge about the type of the + variable. +* In the case where the type guard returns ``False``, the type checker cannot apply any + additional narrowing. + +The standard library function :py:func:`inspect.isawaitable` may serve as an example. It +returns whether the argument is an awaitable object, and +`typeshed `__ +currently annotates it as:: + + def isawaitable(object: object) -> TypeGuard[Awaitable[Any]]: ... + +A user `reported `__ an issue to mypy about +the behavior of this function. They observed the following behavior:: + + import inspect + from collections.abc import Awaitable + from typing import reveal_type + + async def f(t: Awaitable[int] | int) -> None: + if inspect.isawaitable(t): + reveal_type(t) # Awaitable[Any] + else: + reveal_type(t) # Awaitable[int] | int + +This behavior is consistent with :pep:`647`, but it did not match the user's expectations. +Instead, they would expect the type of ``t`` to be narrowed to ``Awaitable[int]`` in the ``if`` +branch, and to ``int`` in the ``else`` branch. This PEP proposes a new construct that does +exactly that. + +Other examples of issues that arose out of the current behavior of ``TypeGuard`` include: + +* `Python typing issue `__ (``numpy.isscalar``) +* `Python typing issue `__ (:py:func:`dataclasses.is_dataclass`) +* `Pyright issue `__ (expecting :py:data:`typing.TypeGuard` to work like :py:func:`isinstance`) +* `Pyright issue `__ (expecting narrowing in the ``else`` branch) +* `Mypy issue `__ (expecting narrowing in the ``else`` branch) +* `Mypy issue `__ (combining multiple TypeGuards) +* `Mypy issue `__ (expecting narrowing in the ``else`` branch) +* `Mypy issue `__ (user-defined function similar to :py:func:`inspect.isawaitable`) +* `Typeshed issue `__ (``asyncio.iscoroutinefunction``) + +Rationale +========= + +The problems with the current behavior of :py:data:`typing.TypeGuard` compel us to improve +the type system to allow a different type narrowing behavior. :pep:`724` proposed to change +the behavior of the existing :py:data:`typing.TypeGuard` construct, but we :ref:`believe ` +that the backwards compatibility implications of that change are too severe. Instead, we propose +adding a new special form with the desired semantics. + +We acknowledge that this leads to an unfortunate situation where there are two constructs with +a similar purpose and similar semantics. We believe that users are more likely to want the behavior +of ``TypeIs``, the new form proposed in this PEP, and therefore we recommend that documentation +emphasize ``TypeIs`` over ``TypeGuard`` as a more commonly applicable tool. However, the semantics of +``TypeGuard`` are occasionally useful, and we do not propose to deprecate or remove it. In the long +run, most users should use ``TypeIs``, and ``TypeGuard`` should be reserved for rare cases +where its behavior is specifically desired. + + +Specification +============= + +A new special form, ``TypeIs``, is added to the :py:mod:`typing` +module. Its usage, behavior, and runtime implementation are similar to +those of :py:data:`typing.TypeGuard`. + +It accepts a single +argument and can be used as the return type of a function. A function annotated as returning a +``TypeIs`` is called a type narrowing function. Type narrowing functions must return ``bool`` +values, and the type checker should verify that all return paths return +``bool``. + +Type narrowing functions must accept at least one positional argument. The type +narrowing behavior is applied to the first positional argument passed to +the function. The function may accept additional arguments, but they are +not affected by type narrowing. If a type narrowing function is implemented as +an instance method or class method, the first positional argument maps +to the second parameter (after ``self`` or ``cls``). + +Type narrowing behavior +----------------------- + +To specify the behavior of ``TypeIs``, we use the following terminology: + +* I = ``TypeIs`` input type +* R = ``TypeIs`` return type +* A = Type of argument passed to type narrowing function (pre-narrowed) +* NP = Narrowed type (positive; used when ``TypeIs`` returned ``True``) +* NN = Narrowed type (negative; used when ``TypeIs`` returned ``False``) + +.. code-block:: python + + def narrower(x: I) -> TypeIs[R]: ... + + def func1(val: A): + if narrower(val): + assert_type(val, NP) + else: + assert_type(val, NN) + +The return type ``R`` must be :ref:`consistent with ` ``I``. The type checker should +emit an error if this condition is not met. + +Formally, type *NP* should be narrowed to :math:`A \land R`, +the intersection of *A* and *R*, and type *NN* should be narrowed to +:math:`A \land \neg R`, the intersection of *A* and the complement of *R*. +In practice, the theoretic types for strict type guards cannot be expressed +precisely in the Python type system. Type checkers should fall back on +practical approximations of these types. As a rule of thumb, a type checker +should use the same type narrowing logic -- and get results that are consistent +with -- its handling of :py:func:`isinstance`. This guidance allows for changes and +improvements if the type system is extended in the future. + +Examples +-------- + +Type narrowing is applied in both the positive and negative case:: + + from typing import TypeIs, assert_type + + def is_str(x: object) -> TypeIs[str]: + return isinstance(x, str) + + def f(x: str | int) -> None: + if is_str(x): + assert_type(x, str) + else: + assert_type(x, int) + +The final narrowed type may be narrower than **R**, due to the constraints of the +argument's previously-known type:: + + from collections.abc import Awaitable + from typing import Any, TypeIs, assert_type + import inspect + + def isawaitable(x: object) -> TypeIs[Awaitable[Any]]: + return inspect.isawaitable(x) + + def f(x: Awaitable[int] | int) -> None: + if isawaitable(x): + # Type checkers may also infer the more precise type + # "Awaitable[int] | (int & Awaitable[Any])" + assert_type(x, Awaitable[int]) + else: + assert_type(x, int) + +It is an error to narrow to a type that is not consistent with the input type:: + + from typing import TypeIs + + def is_str(x: int) -> TypeIs[str]: # Type checker error + ... + +Subtyping +--------- + +``TypeIs`` is also valid as the return type of a callable, for example +in callback protocols and in the ``Callable`` special form. In these +contexts, it is treated as a subtype of bool. For example, ``Callable[..., TypeIs[int]]`` +is assignable to ``Callable[..., bool]``. + +Unlike ``TypeGuard``, ``TypeIs`` is invariant in its argument type: +``TypeIs[B]`` is not a subtype of ``TypeIs[A]``, +even if ``B`` is a subtype of ``A``. +To see why, consider the following example:: + + def takes_narrower(x: int | str, narrower: Callable[[object], TypeIs[int]]): + if narrower(x): + print(x + 1) # x is an int + else: + print("Hello " + x) # x is a str + + def is_bool(x: object) -> TypeIs[bool]: + return isinstance(x, bool) + + takes_narrower(1, is_bool) # Error: is_bool is not a TypeIs[int] + +(Note that ``bool`` is a subtype of ``int``.) +This code fails at runtime, because the narrower returns ``False`` (1 is not a ``bool``) +and the ``else`` branch is taken in ``takes_narrower()``. +If the call ``takes_narrower(1, is_bool)`` was allowed, type checkers would fail to +detect this error. + +Backwards Compatibility +======================= + +As this PEP only proposes a new special form, there are no implications on +backwards compatibility. + + +Security Implications +===================== + +None known. + + +How to Teach This +================= + +Introductions to typing should cover ``TypeIs`` when discussing how to narrow types, +along with discussion of other narrowing constructs such as :py:func:`isinstance`. The +documentation should emphasize ``TypeIs`` over :py:data:`typing.TypeGuard`; while the +latter is not being deprecated and its behavior is occasionally useful, we expect that the +behavior of ``TypeIs`` is usually more intuitive, and most users should reach for +``TypeIs`` first. The rest of this section contains some example content that could +be used in introductory user-facing documentation. + +When to use ``TypeIs`` +---------------------- + +Python code often uses functions like ``isinstance()`` to distinguish between +different possible types of a value. Type checkers understand ``isinstance()`` +and various other checks and use them to narrow the type of a variable. However, +sometimes you want to reuse a more complicated check in multiple places, or +you use a check that the type checker doesn't understand. In these cases, you +can define a ``TypeIs`` function to perform the check and allow type checkers +to use it to narrow the type of a variable. + +A ``TypeIs`` function takes a single argument and is annotated as returning +``TypeIs[T]``, where ``T`` is the type that you want to narrow to. The function +must return ``True`` if the argument is of type ``T``, and ``False`` otherwise. +The function can then be used in ``if`` checks, just like you would use ``isinstance()``. +For example:: + + from typing import TypeIs, Literal + + type Direction = Literal["N", "E", "S", "W"] + + def is_direction(x: str) -> TypeIs[Direction]: + return x in {"N", "E", "S", "W"} + + def maybe_direction(x: str) -> None: + if is_direction(x): + print(f"{x} is a cardinal direction") + else: + print(f"{x} is not a cardinal direction") + +Writing a safe ``TypeIs`` function +---------------------------------- + +A ``TypeIs`` function allows you to override your type checker's type narrowing +behavior. This is a powerful tool, but it can be dangerous because an incorrectly +written ``TypeIs`` function can lead to unsound type checking, and type checkers +cannot detect such errors. + +For a function returning ``TypeIs[T]`` to be safe, it must return ``True`` if and only if +the argument is compatible with type ``T``, and ``False`` otherwise. If this condition is +not met, the type checker may infer incorrect types. + +Below are some examples of correct and incorrect ``TypeIs`` functions:: + + from typing import TypeIs + + # Correct + def good_typeis(x: object) -> TypeIs[int]: + return isinstance(x, int) + + # Incorrect: does not return True for all ints + def bad_typeis1(x: object) -> TypeIs[int]: + return isinstance(x, int) and x > 0 + + # Incorrect: returns True for some non-ints + def bad_typeis2(x: object) -> TypeIs[int]: + return isinstance(x, (int, float)) + +This function demonstrates some errors that can occur when using a poorly written +``TypeIs`` function. These errors are not detected by type checkers:: + + def caller(x: int | str, y: int | float) -> None: + if bad_typeis1(x): # narrowed to int + print(x + 1) + else: # narrowed to str (incorrectly) + print("Hello " + x) # runtime error if x is a negative int + + if bad_typeis2(y): # narrowed to int + # Because of the incorrect TypeIs, this branch is taken at runtime if + # y is a float. + print(y.bit_count()) # runtime error: this method exists only on int, not float + else: # narrowed to float (though never executed at runtime) + pass + +Here is an example of a correct ``TypeIs`` function for a more complicated type:: + + from typing import TypedDict, TypeIs + + class Point(TypedDict): + x: int + y: int + + def is_point(x: object) -> TypeIs[Point]: + return ( + isinstance(x, dict) + and all(isinstance(key, str) for key in x) + and "x" in x + and "y" in x + and isinstance(x["x"], int) + and isinstance(x["y"], int) + ) + +``TypeIs`` and ``TypeGuard`` +---------------------------- + +``TypeIs`` and :py:data:`typing.TypeGuard` are both tools for narrowing the type of a variable +based on a user-defined function. Both can be used to annotate functions that take an +argument and return a boolean depending on whether the input argument is compatible with +the narrowed type. These function can then be used in ``if`` checks to narrow the type +of a variable. + +``TypeIs`` usually has the most intuitive behavior, but it +introduces more restrictions. ``TypeGuard`` is the right tool to use if: + +* You want to narrow to a type that is not compatible with the input type, for example + from ``list[object]`` to ``list[int]``. ``TypeIs`` only allows narrowing between + compatible types. +* Your function does not return ``True`` for all input values that are compatible with + the narrowed type. For example, you could have a ``TypeGuard[int]`` that returns ``True`` + only for positive integers. + +``TypeIs`` and ``TypeGuard`` differ in the following ways: + +* ``TypeIs`` requires the narrowed type to be a subtype of the input type, while + ``TypeGuard`` does not. +* When a ``TypeGuard`` function returns ``True``, type checkers narrow the type of the + variable to exactly the ``TypeGuard`` type. When a ``TypeIs`` function returns ``True``, + type checkers can infer a more precise type combining the previously known type of the + variable with the ``TypeIs`` type. (Technically, this is known as an intersection type.) +* When a ``TypeGuard`` function returns ``False``, type checkers cannot narrow the type of + the variable at all. When a ``TypeIs`` function returns ``False``, type checkers can narrow + the type of the variable to exclude the ``TypeIs`` type. + +This behavior can be seen in the following example:: + + from typing import TypeGuard, TypeIs, reveal_type, final + + class Base: ... + class Child(Base): ... + @final + class Unrelated: ... + + def is_base_typeguard(x: object) -> TypeGuard[Base]: + return isinstance(x, Base) + + def is_base_typeis(x: object) -> TypeIs[Base]: + return isinstance(x, Base) + + def use_typeguard(x: Child | Unrelated) -> None: + if is_base_typeguard(x): + reveal_type(x) # Base + else: + reveal_type(x) # Child | Unrelated + + def use_typeis(x: Child | Unrelated) -> None: + if is_base_typeis(x): + reveal_type(x) # Child + else: + reveal_type(x) # Unrelated + +Reference Implementation +======================== + +The ``TypeIs`` special form `has been implemented `__ +in the ``typing_extensions`` module and will be released in typing_extensions 4.10.0. + +Implementations are available for several type checkers: + +- Mypy: `pull request open `__ +- Pyanalyze: `pull request `__ +- Pyright: `added in version 1.1.351 `__ + +Rejected Ideas +============== + +.. _pep-742-change-typeguard: + +Change the behavior of ``TypeGuard`` +------------------------------------ + +:pep:`724` previously proposed changing the specified behavior of :py:data:`typing.TypeGuard` so +that if the return type of the guard is consistent with the input type, the behavior proposed +here for ``TypeIs`` would apply. This proposal has some important advantages: because it +does not require any runtime changes, it requires changes only in type checkers, making it easier +for users to take advantage of the new, usually more intuitive behavior. + +However, this approach has some major problems. Users who have written ``TypeGuard`` functions +expecting the existing semantics specified in :pep:`647` would see subtle and potentially breaking +changes in how type checkers interpret their code. The split behavior of ``TypeGuard``, where it +works one way if the return type is consistent with the input type and another way if it is not, +could be confusing for users. The Typing Council was unable to come to an agreement in favor of +:pep:`724`; as a result, we are proposing this alternative PEP. + +Do nothing +---------- + +Both this PEP and the alternative proposed in :pep:`724` have shortcomings. The latter are +discussed above. As for this PEP, it introduces two special forms with very similar semantics, +and it potentially creates a long migration path for users currently using ``TypeGuard`` +who would be better off with different narrowing semantics. + +One way forward, then, is to do nothing and live with the current limitations of the type system. +However, we believe that the limitations of the current ``TypeGuard``, as outlined in the "Motivation" +section, are significant enough that it is worthwhile to change the type system to address them. +If we do not make any change, users will continue to encounter the same unintuitive behaviors from +``TypeGuard``, and the type system will be unable to properly represent common type narrowing functions +like ``inspect.isawaitable``. + +Alternative names +----------------- + +This PEP currently proposes the name ``TypeIs``, emphasizing that the special form ``TypeIs[T]`` +returns whether the argument is of type ``T``, and mirroring +`TypeScript's syntax `__. +Other names were considered, including in an earlier version of this PEP. + +Options include: + +* ``IsInstance`` (`post by Paul Moore `__): + emphasizes that the new construct behaves similarly to the builtin :py:func:`isinstance`. +* ``Narrowed`` or ``NarrowedTo``: shorter than ``TypeNarrower`` but keeps the connection to "type narrowing" + (suggested by Eric Traut). +* ``Predicate`` or ``TypePredicate``: mirrors TypeScript's name for the feature, "type predicates". +* ``StrictTypeGuard`` (earlier drafts of :pep:`724`): emphasizes that the new construct performs a stricter + version of type narrowing than :py:data:`typing.TypeGuard`. +* ``TypeCheck`` (`post by Nicolas Tessore `__): + emphasizes the binary nature of the check. +* ``TypeNarrower``: emphasizes that the function narrows its argument type. Used in an earlier version of this PEP. + +Acknowledgments +=============== + +Much of the motivation and specification for this PEP derives from :pep:`724`. While +this PEP proposes a different solution for the problem at hand, the authors of :pep:`724`, Eric Traut, Rich +Chiodo, and Erik De Bonte, made a strong case for their proposal and this proposal +would not have been possible without their work. + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. diff --git a/decisiongrounding/scenarios_candidates/peps_typeis_narrowing_supersession/provenance.json b/decisiongrounding/scenarios_candidates/peps_typeis_narrowing_supersession/provenance.json new file mode 100644 index 0000000..0e56780 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_typeis_narrowing_supersession/provenance.json @@ -0,0 +1,37 @@ +{ + "source_repo": "python/peps", + "pinned_commit": "f866e77409305866038471574f075cd8d83eee9e", + "peps": [ + { + "id": "PEP-0724", + "number": 724, + "file": "corpus/PEP-0724.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0724.rst", + "source_sha256": "04aa72d77d6b4785136f350e3435c3d2b010418259aee82678bbf5edcc61091e", + "status": "Withdrawn", + "title": "Stricter Type Guards", + "replaces": [], + "superseded_by": [] + }, + { + "id": "PEP-0742", + "number": 742, + "file": "corpus/PEP-0742.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0742.rst", + "source_sha256": "dd23a67ae82462063d3180f7c4bdf45db0e6ee0cf5b7e6a5215b50ffcef075c8", + "status": "Final", + "title": "Narrowing types with TypeIs", + "replaces": [ + "PEP-0724" + ], + "superseded_by": [] + } + ], + "supersedes_edges": [ + { + "source": "PEP-0742", + "type": "supersedes", + "target": "PEP-0724" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/corpus/PEP-0397.md b/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/corpus/PEP-0397.md new file mode 100644 index 0000000..c0ee4fc --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/corpus/PEP-0397.md @@ -0,0 +1,450 @@ +--- +schema_version: 1 +id: PEP-0397 +type: decision +tags: [pep] +--- + +# PEP-0397 — Python launcher for Windows + +## Status + +Final + +## Context + +Public decision ingested verbatim from python/peps (PEP 397) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0397.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0397.rst). Upstream +status: Final. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 397 +Title: Python launcher for Windows +Author: Mark Hammond , + Martin von Löwis +Status: Final +Type: Standards Track +Created: 15-Mar-2011 +Python-Version: 3.3 +Post-History: 21-Jul-2011, 17-May-2011, 15-Mar-2011 +Resolution: https://mail.python.org/pipermail/python-dev/2012-June/120505.html + +Abstract +======== + +This PEP describes a Python launcher for the Windows platform. A +Python launcher is a single executable which uses a number of +heuristics to locate a Python executable and launch it with a +specified command line. + + +Rationale +========= + +Windows provides "file associations" so an executable can be associated +with an extension, allowing for scripts to be executed directly in some +contexts (eg., double-clicking the file in Windows Explorer.) Until now, +a strategy of "last installed Python wins" has been used and while not +ideal, has generally been workable due to the conservative changes in +Python 2.x releases. As Python 3.x scripts are often syntactically +incompatible with Python 2.x scripts, a different strategy must be used +to allow files with a '.py' extension to use a different executable based +on the Python version the script targets. This will be done by borrowing +the existing practices of another operating system - scripts will be able +to nominate the version of Python they need by way of a "shebang" line, as +described below. + +Unix-like operating systems (referred to simply as "Unix" in this +PEP) allow scripts to be executed as if they were executable images +by examining the script for a "shebang" line which specifies the +actual executable to be used to run the script. This is described in +detail in the ``evecve(2)`` man page [1]_ and while user documentation will +be created for this feature, for the purposes of this PEP that man +page describes a valid shebang line. + +Additionally, these operating systems provide symbolic-links to +Python executables in well-known directories. For example, many +systems will have a link /usr/bin/python which references a +particular version of Python installed under the operating-system. +These symbolic links allow Python to be executed without regard for +where Python it actually installed on the machine (eg., without +requiring the path where Python is actually installed to be +referenced in the shebang line or in the ``PATH``.) :pep:`394` 'The "python" +command on Unix-Like Systems' describes additional conventions +for more fine-grained specification of a particular Python version. + +These 2 facilities combined allow for a portable and somewhat +predictable way of both starting Python interactively and for allowing +Python scripts to execute. This PEP describes an implementation of a +launcher which can offer the same benefits for Python on the Windows +platform and therefore allows the launcher to be the executable +associated with '.py' files to support multiple Python versions +concurrently. + +While this PEP offers the ability to use a shebang line which should +work on both Windows and Unix, this is not the primary motivation for +this PEP - the primary motivation is to allow a specific version to be +specified without inventing new syntax or conventions to describe +it. + +Specification +============= + +This PEP specifies features of the launcher; a prototype +implementation is provided in [3]_ which will be distributed +together with the Windows installer of Python, but will also be +available separately (but released along with the Python +installer). New features may be added to the launcher as +long as the features prescribed here continue to work. + +Installation +------------ + +The launcher comes in 2 versions - one which is a console program and +one which is a "windows" (ie., GUI) program. These 2 launchers correspond +to the 'python.exe' and 'pythonw.exe' executables which currently ship +with Python. The console launcher will be named 'py.exe' and the Windows +one named 'pyw.exe'. The "windows" (ie., GUI) version of the launcher +will attempt to locate and launch pythonw.exe even if a virtual shebang +line nominates simply "python" - in fact, the trailing 'w' notation is +not supported in the virtual shebang line at all. + +The launcher is installed into the Windows directory (see +discussion below) if installed by a privileged user. The +stand-alone installer asks for an alternative location of the +installer, and adds that location to the user's ``PATH``. + +The installation in the Windows directory is a 32-bit executable +(see discussion); the standalone installer may also offer to install +64-bit versions of the launcher. + +The launcher installation is registered in +``HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CurrentVersion\SharedDLLs`` +with a reference counter. +It contains a version resource matching the version number of the +pythonXY.dll with which it is distributed. Independent +installations will overwrite older version +of the launcher with newer versions. Stand-alone releases use +a release level of ``0x10`` in ``FIELD3`` of the CPython release on which +they are based. + +Once installed, the "console" version of the launcher is +associated with .py files and the "windows" version associated with .pyw +files. + +The launcher is not tied to a specific version of Python - eg., a +launcher distributed with Python 3.3 should be capable of locating and +executing any Python 2.x and Python 3.x version. However, the +launcher binaries have a version resource that is the same as the +version resource in the Python binaries that they are released with. + +Python Script Launching +----------------------- + +The launcher is restricted to launching Python scripts. +It is not intended as a general-purpose script launcher or +shebang processor. + +The launcher supports the syntax of shebang lines as described +in [1]_, including all restrictions listed. + +The launcher supports shebang lines referring to Python +executables with any of the (regex) prefixes "/usr/bin/", "/usr/local/bin" +and "/usr/bin/env \*", as well as binaries specified without + +For example, a shebang line of '#! /usr/bin/python' should work even +though there is unlikely to be an executable in the relative Windows +directory "\\usr\\bin". This means that many scripts can use a single +shebang line and be likely to work on both Unix and Windows without +modification. + +The launcher will support fully-qualified paths to executables. +While this will make the script inherently non-portable, it is a +feature offered by Unix and would be useful for Windows users in +some cases. + +The launcher will be capable of supporting implementations other than +CPython, such as jython and IronPython, but given both the absence of +common links on Unix (such as "/usr/bin/jython") and the inability for the +launcher to automatically locate the installation location of these +implementations on Windows, the launcher will support this via +customization options. Scripts taking advantage of this will not be +portable (as these customization options must be set to reflect the +configuration of the machine on which the launcher is running) but this +ability is nonetheless considered worthwhile. + +On Unix, the user can control which specific version of Python is used +by adjusting the links in /usr/bin to point to the desired version. As +the launcher on Windows will not use Windows links, customization options +(exposed via both environment variables and INI files) will be used to +override the semantics for determining what version of Python will be +used. For example, while a shebang line of "/usr/bin/python2" will +automatically locate a Python 2.x implementation, an environment variable +can override exactly which Python 2.x implementation will be chosen. +Similarly for "/usr/bin/python" and "/usr/bin/python3". This is +specified in detail later in this PEP. + +Shebang line parsing +-------------------- + +If the first command-line argument does not start with a dash ('-') +character, an attempt will be made to open that argument as a file +and parsed for a shebang line according to the rules in [1]_:: + + #! interpreter [optional-arg] + +Once parsed, the command will be categorized according to the following rules: + +* If the command starts with the definition of a customized command + followed by a whitespace character (including a newline), the customized + command will be used. See below for a description of customized + commands. + +* The launcher will define a set of prefixes which are considered Unix + compatible commands to launch Python, namely "/usr/bin/python", + "/usr/local/bin/python", "/usr/bin/env python", and "python". + If a command starts with one of these strings will be treated as a + 'virtual command' and the rules described in Python Version Qualifiers + (below) will be used to locate the executable to use. + +* Otherwise the command is assumed to be directly ready to execute - ie. + a fully-qualified path (or a reference to an executable on the ``PATH``) + optionally followed by arguments. The contents of the string will not + be parsed - it will be passed directly to the Windows CreateProcess + function after appending the name of the script and the launcher + command-line arguments. This means that the rules used by + CreateProcess will be used, including how relative path names and + executable references without extensions are treated. Notably, the + Windows command processor will not be used, so special rules used by the + command processor (such as automatic appending of extensions other than + '.exe', support for batch files, etc) will not be used. + +The use of 'virtual' shebang lines is encouraged as this should +allow for portable shebang lines to be specified which work on +multiple operating systems and different installations of the same +operating system. + +If the first argument can not be opened as a file or if no valid +shebang line can be found, the launcher will act as if a shebang line of +'#!python' was found - ie., a default Python interpreter will be +located and the arguments passed to that. However, if a valid +shebang line is found but the process specified by that line can not +be started, the default interpreter will not be started - the error +to create the specified child process will cause the launcher to display +an appropriate message and terminate with a specific exit code. + +Configuration file +------------------ + +Two .ini files will be searched by the launcher - ``py.ini`` in the +current user's "application data" directory (i.e. the directory returned +by calling the Windows function ``SHGetFolderPath`` with ``CSIDL_LOCAL_APPDATA``, +``%USERPROFILE%\AppData\Local`` on Vista+, +``%USERPROFILE%\Local Settings\Application Data`` on XP) +and ``py.ini`` in the same directory as the launcher. The same .ini +files are used for both the 'console' version of the launcher (i.e. +py.exe) and for the 'windows' version (i.e. pyw.exe) + + +Customization specified in the "application directory" will have +precedence over the one next to the executable, so a user, who may not +have write access to the .ini file next to the launcher, can override +commands in that global .ini file) + +Virtual commands in shebang lines +--------------------------------- + +Virtual Commands are shebang lines which start with strings which would +be expected to work on Unix platforms - examples include +'/usr/bin/python', '/usr/bin/env python' and 'python'. Optionally, the +virtual command may be suffixed with a version qualifier (see below), +such as '/usr/bin/python2' or '/usr/bin/python3.2'. The command executed +is based on the rules described in Python Version Qualifiers +below. + +Customized Commands +------------------- + +The launcher will support the ability to define "Customized Commands" in a +Windows .ini file (ie, a file which can be parsed by the Windows function +GetPrivateProfileString). A section called '[commands]' can be created +with key names defining the virtual command and the value specifying the +actual command-line to be used for this virtual command. + +For example, if an INI file has the contents:: + + [commands] + vpython=c:\bin\vpython.exe -foo + +Then a shebang line of '#! vpython' in a script named 'doit.py' will +result in the launcher using the command-line +``c:\bin\vpython.exe -foo doit.py`` + +The precise details about the names, locations and search order of the +.ini files is in the launcher documentation [4]_ + +Python Version Qualifiers +------------------------- + +Some of the features described allow an optional Python version qualifier +to be used. + +A version qualifier starts with a major version number and can optionally +be followed by a period ('.') and a minor version specifier. If the minor +qualifier is specified, it may optionally be followed by "-32" to indicate +the 32bit implementation of that version be used. Note that no "-64" +qualifier is necessary as this is the default implementation (see below). + +On 64bit Windows with both 32bit and 64bit implementations of the +same (major.minor) Python version installed, the 64bit version will +always be preferred. This will be true for both 32bit and 64bit +implementations of the launcher - a 32bit launcher will prefer to +execute a 64bit Python installation of the specified version if +available. This is so the behavior of the launcher can be predicted +knowing only what versions are installed on the PC and without +regard to the order in which they were installed (ie, without knowing +whether a 32 or 64bit version of Python and corresponding launcher was +installed last). As noted above, an optional "-32" suffix can be used +on a version specifier to change this behaviour. + +If no version qualifiers are found in a command, the environment variable +``PY_PYTHON`` can be set to specify the default version qualifier - the default +value is "2". Note this value could specify just a major version (e.g. "2") or +a major.minor qualifier (e.g. "2.6"), or even major.minor-32. + +If no minor version qualifiers are found, the environment variable +``PY_PYTHON{major}`` (where ``{major}`` is the current major version qualifier +as determined above) can be set to specify the full version. If no such option +is found, the launcher will enumerate the installed Python versions and use +the latest minor release found for the major version, which is likely, +although not guaranteed, to be the most recently installed version in that +family. + +In addition to environment variables, the same settings can be configured +in the .INI file used by the launcher. The section in the INI file is +called ``[defaults]`` and the key name will be the same as the +environment variables without the leading ``PY_`` prefix (and note that +the key names in the INI file are case insensitive.) The contents of +an environment variable will override things specified in the INI file. + +Command-line handling +--------------------- + +Only the first command-line argument will be checked for a shebang line +and only if that argument does not start with a '-'. + +If the only command-line argument is "-h" or "--help", the launcher will +print a small banner and command-line usage, then pass the argument to +the default Python. This will cause help for the launcher being printed +followed by help for Python itself. The output from the launcher will +clearly indicate the extended help information is coming from the +launcher and not Python. + +As a concession to interactively launching Python, the launcher will +support the first command-line argument optionally being a dash ("-") +followed by a version qualifier, as described above, to nominate a +specific version be used. For example, while "py.exe" may locate and +launch the latest Python 2.x implementation installed, a command-line such +as "py.exe -3" could specify the latest Python 3.x implementation be +launched, while "py.exe -2.6-32" could specify a 32bit implementation +Python 2.6 be located and launched. If a Python 2.x implementation is +desired to be launched with the -3 flag, the command-line would need to be +similar to "py.exe -2 -3" (or the specific version of Python could +obviously be launched manually without use of this launcher.) Note that +this feature can not be used with shebang processing as the file scanned +for a shebang line and this argument must both be the first argument and +therefore are mutually exclusive. + +All other arguments will be passed untouched to the child Python process. + +Process Launching +----------------- + +The launcher offers some conveniences for Python developers working +interactively - for example, starting the launcher with no command-line +arguments will launch the default Python with no command-line arguments. +Further, command-line arguments will be supported to allow a specific +Python version to be launched interactively - however, these conveniences +must not detract from the primary purpose of launching scripts and must +be easy to avoid if desired. + +The launcher creates a subprocess to start the actual +interpreter. See **Discussion** below for the rationale. + + +Discussion +========== + +It may be surprising that the launcher is installed into the +Windows directory, and not the System32 directory. The reason is +that the System32 directory is not on the Path of a 32-bit process +running on a 64-bit system. However, the Windows directory is +always on the path. + +The launcher that is installed into the Windows directory is a 32-bit +executable so that the 32-bit CPython installer can provide the same +binary for both 32-bit and 64-bit Windows installations. + +Ideally, the launcher process would execute Python directly inside +the same process, primarily so the parent of the launcher process could +terminate the launcher and have the Python interpreter terminate. If the +launcher executes Python as a sub-process and the parent of the launcher +terminates the launcher, the Python process will be unaffected. + +However, there are a number of practical problems associated with this +approach. Windows does not support the ``execv*`` family of Unix functions, +so this could only be done by the launcher dynamically loading the Python +DLL, but this would have a number of side-effects. The most serious +side effect of this is that the value of sys.executable would refer to the +launcher instead of the Python implementation. Many Python scripts use the +value of ``sys.executable`` to launch child processes, and these scripts may +fail to work as expected if the launcher is used. Consider a "parent" +script with a shebang line of '#! /usr/bin/python3' which attempts to +launch a child script (with no shebang) via ``sys.executable`` - currently the +child is launched using the exact same version running the parent script. +If ``sys.executable`` referred to the launcher the child would be likely +executed using a Python 2.x version and would be likely to fail with a +``SyntaxError``. + +Another hurdle is the support for alternative Python implementations +using the "customized commands" feature described above, where loading +the command dynamically into a running executable is not possible. + +The final hurdle is the rules above regarding 64bit and 32bit programs - +a 32bit launcher would be unable to load the 64bit version of Python and +vice-versa. + +Given these considerations, the launcher will execute its command in a +child process, remaining alive while the child process is executing, then +terminate with the same exit code as returned by the child. To address +concerns regarding the termination of the launcher not killing the child, +the Win32 Job API will be used to arrange so that the child process is +automatically killed when the parent is terminated (although children of +that child process will continue as is the case now.) As this Windows API +is available in Windows XP and later, this launcher will not work on +Windows 2000 or earlier. + +References +========== + +.. [1] http://linux.die.net/man/2/execve + +.. [3] https://bitbucket.org/vinay.sajip/pylauncher + +.. [4] https://bitbucket.org/vinay.sajip/pylauncher/src/tip/Doc/launcher.rst + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/corpus/PEP-0486.md b/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/corpus/PEP-0486.md new file mode 100644 index 0000000..b51518d --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/corpus/PEP-0486.md @@ -0,0 +1,167 @@ +--- +schema_version: 1 +id: PEP-0486 +type: decision +tags: [pep] +--- + +# PEP-0486 — Make the Python Launcher aware of virtual environments + +## Status + +Final + +## Context + +Public decision ingested verbatim from python/peps (PEP 486) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0486.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0486.rst). Upstream +status: Final. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Source Text + +PEP: 486 +Title: Make the Python Launcher aware of virtual environments +Author: Paul Moore +Status: Final +Type: Standards Track +Created: 12-Feb-2015 +Python-Version: 3.5 +Post-History: 12-Feb-2015 +Resolution: https://mail.python.org/pipermail/python-dev/2015-February/138579.html + + +Abstract +======== + +The Windows installers for Python include a launcher that locates the +correct Python interpreter to run (see :pep:`397`). However, the +launcher is not aware of virtual environments (virtualenv [1]_ or PEP +405 based), and so cannot be used to run commands from the active +virtualenv. + +This PEP proposes making the launcher "virtualenv aware". This means +that when run without specifying an explicit Python interpreter to +use, the launcher will use the currently active virtualenv, if any, +before falling back to the configured default Python. + + +Rationale +========= + +Windows users with multiple copies of Python installed need a means of +selecting which one to use. The Python launcher provides this +facility by means of a ``py`` command that can be used to run either a +configured "default" Python or a specific interpreter, by means of +command line arguments. So typical usage would be:: + + # Run the Python interactive interpreter + py + + # Execute an installed module + py -m pip install pytest + py -m pytest + +When using virtual environments, the ``py`` launcher is unaware that a +virtualenv is active, and will continue to use the system Python. So +different command invocations are needed to run the same commands in a +virtualenv:: + + # Run the Python interactive interpreter + python + + # Execute an installed module (these could use python -m, + # which is longer to type but is a little more similar to the + # launcher approach) + pip install pytest + py.test + +Having to use different commands is error-prone, and in many cases +the error is difficult to spot immediately. The PEP proposes making +the ``py`` command usable with virtual environments, so that the first +form of command can be used in all cases. + + +Implementation +============== + +Both ``virtualenv`` and the core ``venv`` module set an environment +variable ``VIRTUAL_ENV`` when activating a virtualenv. This PEP +proposes that the launcher checks for the ``VIRTUAL_ENV`` environment +variable whenever it would run the "default" Python interpreter for +the system (i.e., when no specific version flags such as ``py -2.7`` +are used) and if present, run the Python interpreter for the +virtualenv rather than the default system Python. + +The "default" Python interpreter referred to above is (as per :pep:`397`) +either the latest version of Python installed on the system, or +a version configured via the ``py.ini`` configuration file. When the +user specifies an explicit Python version on the command line, this +will always be used (as at present). + + +Impact on Script Launching +========================== + + +As well as interactive use, the launcher is used as the Windows file +association for Python scripts. In that case, a "shebang" (``#!``) +line at the start of the script is used to identify the interpreter to +run. A fully-qualified path can be used, or a version-specific Python +(``python3`` or ``python2``, or even ``python3.5``), or the generic +``python``, which means to use the default interpreter. + +The launcher also looks for the specific shebang line +``#!/usr/bin/env python``. On Unix, the ``env`` program searches for a +command on ``$PATH`` and runs the command so located. Similarly, with +this shebang line, the launcher will look for a copy of ``python.exe`` +on the user's current ``%PATH%`` and will run that copy. + +As activating a virtualenv means that it is added to ``PATH``, no +special handling is needed to run scripts with the active virtualenv - +they just need to use the ``#!/usr/bin/env python`` shebang line, +exactly as on Unix. (If there is no activated virtualenv, and no +``python.exe`` on ``PATH``, the launcher will look for a default +Python exactly as if the shebang line had said ``#!python``). + + +Exclusions +========== + +The PEP makes no attempt to promote the use of the launcher for +running Python on Windows. Most existing documentation assumes the +user of ``python`` as the command to run Python, and (for example) +``pip`` to run an installed Python command. This documentation is not +expected to change, and users who choose to manage their ``PATH`` +environment variable can continue to use this form. The focus of this +PEP is purely on allowing users who prefer to use the launcher when +dealing with their system Python installations, to be able to continue +to do so when using virtual environments. + + +Reference Implementation +======================== + +A patch implementing the proposed behaviour is available at +http://bugs.python.org/issue23465 + + +References +========== + +.. [1] https://virtualenv.pypa.io/ + + +Copyright +========= + +This document has been placed in the public domain. diff --git a/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/corpus/PEP-0773.md b/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/corpus/PEP-0773.md new file mode 100644 index 0000000..c721bd0 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/corpus/PEP-0773.md @@ -0,0 +1,1760 @@ +--- +schema_version: 1 +id: PEP-0773 +type: decision +tags: [pep, release] +--- + +# PEP-0773 — A Python Installation Manager for Windows + +## Status + +Final + +## Context + +Public decision ingested verbatim from python/peps (PEP 773) at commit +f866e77409305866038471574f075cd8d83eee9e; source peps/pep-0773.rst (https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0773.rst). Upstream +status: Final. The RAC envelope sections are derived from the PEP's own +header fields; the authoritative content is the unedited PEP reproduced under +Source Text (upstream sha256 recorded in provenance.json). + +## Decision + +Defined by the verbatim PEP text under Source Text below. + +## Consequences + +As stated by the upstream PEP; see Source Text. + +## Supersedes + +- PEP-0397 +- PEP-0486 + +## Source Text + +PEP: 773 +Title: A Python Installation Manager for Windows +Author: Steve Dower +Discussions-To: https://discuss.python.org/t/77900/ +Status: Final +Type: Standards Track +Topic: Release +Created: 21-Jan-2025 +Post-History: + `18-Dec-2024 `__, + `21-Jan-2025 `__, +Replaces: 397, 486 +Resolution: `25-Apr-2025 `__ + +.. canonical-doc:: `Python Releases for Windows `__ + + +Abstract +======== + +Installation of the python.org Python distribution on Windows is complex. +There are three main approaches with roughly equivalent levels of user +experience, and yet all of these suffer from different limitations, including +failing to satisfy modern usage scenarios. This PEP proposes a design for +a single Windows install workflow tool that satisfies all the needs of the +existing installers for the platform, while avoiding most of their limitations, +and provides the core team with the ability to manage releases for many years +to come. + +With the new installer, the recommended command to launch the default Python on +Windows will be ``python``, while the command for managing multiple versions +(including launching a specific one) will be ``py``. Users can also opt-in to +having global ``python3.x`` commands by adding an extra entry to their ``PATH`` +environment variable. We recommend starting with the "How to teach this" section +for the most concise high-level overview of the features. + +The existing ``.exe`` installer and ``py`` launcher will be deprecated and no +longer released from two years after this PEP is accepted. The existing +embeddable distro will no longer be listed as a download, but will be available +through the install manager. The Windows Store app will be immediately replaced +with the install manager. + +Reader's Note +------------- + +This document is a detailed *proposal*, intended to assist in an "approve" or +"reject" decision on the replacement of the existing install formats for Python +on Windows with a new format. It is not intended to be a binding specification, +nor is it user documentation or an interoperability specification. The actual +implementation may vary over time as needs change, and that is not to be +considered a "violation" of this PEP. Any interfaces and protocols meant for +public use will be independently documented and maintained in accordance with +standard deprecation timelines. + +Documentation on installing Python on Windows can be found at +`docs.python.org/using/windows.html `__. + +Background +========== + +There are a large range of needs users may have that lead to them wanting +to install a Python runtime. Many, likely most, are interested in running +(perhaps writing) short scripts, such as those that perform a simple task, +or help teach someone a concept. Some users are looking for a specific version +to integrate with existing code or another application. Some are after a full +set of different interpreter versions to perform testing. + +In this section, we discuss the expectations that users have of "installing +Python", provide an overview of the existing installers for Windows, and +identify some of the gaps and challenges inherent in these offerings. + +Expectations +------------ + +Based on significant anecdotal experience and analysis of quantitative data +available (though not necessarily public), we make the following assertions +about the majority of Python users on Windows: + +* most users just want the latest stable version +* most users want a "one-click" (or fewer) install +* most users do not want to use administrator privileges +* most users will benefit from installing maintenance updates +* most users expect the ``python`` command to work after installation + +The primary support for these assertions is that the most popular installers +actively chosen by users are the latest stable release on python.org, and +the latest stable release on the Windows Store, both of which meet these +requirements. + +We make the following assumptions about other significant sets of users. +These may have some overlap between groups, and at least some users expect +all of them. + +* some users want to install Python programmatically +* some users want to install a particular version +* some users want to install many versions +* some users want to install for all users of their machine +* some users do not want Start Menu shortcuts +* some users want to install as part of their project's build process +* some users want to install as part of their project's install process +* some users intend to never update their install +* some users expect the ``py`` command to work after installation + +These assumptions have all been demonstrated over time to exist, though the +relative importance has not been quantified. The :ref:`NuGet packages +` and the :ref:`embeddable distro +` can meet most of these needs. + +Traditional Installer +--------------------- + +The traditional installer is an executable downloadable directly from +python.org that installs the entire development kit for Python. This includes +the CPython interpreter, the standard library, Python headers and import +libraries, builds of Tcl and Tk, the documentation as HTML files, the runtime +and standard library test suite, Start Menu shortcuts for Python and IDLE, +debugging symbols and debug builds of the binaries, the ``py.exe`` launcher +and its file associations, and functionality to modify the user's ``PATH`` +environment variable, enable long-path support on their system, pre-generate +``.pyc`` files for the standard library, and install pip. +As of 3.13, it also includes a set of experimental free-threaded binaries. +Many of these components are optional. + +After downloading the executable, users are presented with a "quick install" +option, which installs into their user directory with most options enabled. +We believe that most users select this option. + +A second option alongside the quick install takes the user to two pages worth +of options, listing the components that they need not install, as well as other +options such as the install directory and whether to install for all users. + +All of these options may be specified on the command line, and there is also an +option to proceed with the install without displaying any UI. +Based on feedback and bug reports, all of these options are used by at least +some users. However, as we do not track install telemetry, we have no way to +know which options are more important than others. + +Behind the scenes, the traditional installer is a Burn bundle, generated using +the Wix Toolset installer framework, containing one or more MSI files for each +feature. This framework is used extensively by Microsoft themselves, and +provides the most direct method of using Windows Installer. The bundle is a +custom C++ application, based on their template, which allows us to customise +the overall behaviour of the installer to determine precisely which MSI files +should actually be installed. The process of copying files, updating the +registry, and generating shortcuts is handled entirely by Windows Installer. + +As well as the intended uses, it is understood that many users will (attempt to) +use the traditional installer for other scenarios, such as unregistered installs +and automated CI system installs. While better alternatives are available, they +are not as obvious, and the hope is that a future design would make these +scenarios easier. + +Windows Store +------------- + +The Windows Store packages for CPython are produced as part of our normal +release process using almost all identical binaries to the other packages. +Due to being in an app store package, the primary ``python.exe`` is enhanced +to be able to determine its location properly, and alternative ``pip.exe`` and +other shortcuts are included to make up for the lack of ``PATH`` environment +variable settings. These are implemented in :file:`PC\\python_uwp.cpp` in our repo. + +These packages are installed by searching for Python in the Microsoft Store +app, which will find results for each major version since 3.8. Users then have +to select a version and install it. These packages include the CPython +interpreter, standard library, Tcl/Tk, IDLE, and pip, and create file +associations, Start Menu shortcuts, and global commands for ``python.exe``, +``python3.exe``, ``python3.X.exe``, ``pip[3[.X]].exe`` and ``idle[3[.X]].exe``. +No ``PATH`` modification is possible or required, though users may need to +manage their global shortcuts through the "Manage App Execution Alias" settings +page. + +In addition, Microsoft has added to a clean Windows install a default +``python.exe`` command. This captures attempts by users to launch Python on +a machine that has not yet installed it. When launched directly, the command +will open the Microsoft Store app to the page containing the recommended +Python app, typically the latest version. This app is entirely controlled by +Microsoft. Based on telemetry associated with the Python app (which *is* +controlled by the upstream Python project), approximately 300,000 installs +per month come through this redirector, making up about 90% of the total +installs of that version. + +Behind the scenes, the Store package is based on Microsoft's new installer +technology for apps known as APPX or MSIX. These are essentially plain ZIP +files with a small amount of metadata, except that installation is handled +by the operating system. They are always extracted to a fixed location, +accessible to all users but not modifiable by any, and automatically updated +to the latest release. The user's own data is stored in an OS-managed location +in their user profile, and is able to be reset, backed-up and restored using +regular OS functionality. + +NuGet Package +------------- + +The NuGet packages for CPython are produced and published as part of our +normal release process. The contents are identical to the traditional +installer. A NuGet package is published to nuget.org, which is a package +manager typically associated with .NET languages, but highly integrated with +any project supported by Visual Studio. This makes it a nice format for users +who want a lightweight install of Python as part of their regular build process, +and can simplify embedding scenarios. + +The packages are installed using any tool capable of using the NuGet API, or +may be downloaded directly once the URL of the package is known. The package is +a plain ZIP file with some metadata. It contains the CPython interpreter, the +standard library, development headers and import libraries, and pip. It does +not execute any code at install time, and users must locate the package +themselves in order to launch the ``python.exe`` contained within. + +Embeddable Package +------------------ + +The embeddable package for CPython is produced and published as part of our +normal release process. It is published to python.org alongside the +traditional installer. The contents are identical, however, the layout is +changed to store all binaries at the top level, with the standard library +packed into a ZIP file. A ``._pth`` file is included to override ``sys.path`` +so that only the files that are part of the distro are used, and environment +variables or registry entries are ignored. + +This package does not include pip, as the intention is for it to be embedded +into a broader application. Other libraries should be installed at build time, +since after distribution the runtime is meant to be an internal implementation +detail of the app it is a part of. + +As well as its intended use, some users attempt to use this package as a +development kit rather than a runtime package. This is believed to be due to +those users preferring to avoid "heavyweight" installers, and believing that +this package is intended to be a "portable" install (extract and run), likely +because it is the only ZIP file option listed on the python.org download pages +(speaking to the importance of clarity and limiting options on those pages). +It is hoped that a future installer design will avoid or limit this confusion. + +Alternate Distributions +----------------------- + +While outside of our purview as the core team, alternate distributions of Python +for Windows often use a project, workflow or environment-centric model for +installation of the runtime. By this, we mean that the tool is installed first, +and is used to create a working space that includes a runtime, as well as other +dependencies. Examples of these tools include conda and uv. + +Two observations are worth making about these tools. Firstly, they are often +praised for being low impact, in that they usually don't install additional +entry points or files for the runtime, making the install fast and also isolated +to a single project. Secondly, their users often appreciate the ease of +selecting a particular version of a runtime, or alternatively, not having to +select at all because existing specifications (or constraints) can choose for +them. + +These tools tend to meet many of the second set of expectations described above, +usually combining multiple tasks in a single command to reduce the cognitive +overhead of learning how to use and combine multiple commands. + +It's also worth pointing out that the core team does not view these alternate +distributions as competitors to any upstream distribution. They are a +fundamental part of how the open source ecosystem is intended to work. Our own +distributions are a convenience for those who choose to use them, as not all +scenarios are well served by a workflow tool or even a pre-built package. + + +Challenges +---------- + +There are numerous challenges we face with the current set of installers, +which largely break down into two categories: mismatched or unachievable +user expectations, and general unreliability. + +The traditional installer has the highest level of unreliability. The Windows +Installer technology is very old, and effectively no longer under development. +While its basic functionality is okay, interference may come from many sources, +such as virus scanners, other installers, system configuration, admin policies, +and even other files in the same directory as the installer. On top of this, +most of its advanced and beneficial functionality such as update patches, +incremental updates, and automatic rollback are unimportant for Python users. + +Most user expectations are *defined* by the traditional installer, and so by +definition, it meets them. One primary gap is that it is not able to create an +"unmanaged" install - that is, the equivalent of only copying files onto the +user's system without registration. If you have installed it once, and you +try to install it again, you will only even be able to manage (or upgrade) the +existing install. This can lead to installs moving on update, which will +break users. + +Additionally, the ``PATH`` environment variable cannot be intelligently +modified - at best, we can prepend or append the install path. This usually +results in the most recent install of Python being the highest priority. For +example, if the user has Python 3.14 installed and then installs (or updates) +3.13, the ``python`` command will switch from the later version to the earlier +version. + +The ``py.exe`` launcher, defined in :pep:`397` and implicitly updated by :pep:`514`, +is an attempt to avoid this particular issue. It uses its own logic for finding +installed versions without relying on ``PATH``. However, the PEP 514 logic does +not allow for prerelease or experimental builds to be treated specially, and so +``py.exe`` often prefers these builds by default over the non-experimental +version expected by the user. + +The Windows Store package is very reliable, with the exception of the global +shortcuts. Rather than modifying ``PATH`` to add its own directory, these +shortcuts are created in a single OS managed directory that has all the +shortcuts defined by any app. Users are able to modify their ``PATH`` to exclude +or de-prioritise this directory, leading to unreliable or inconsistent +behaviour, and historically we have also seen this caused by installers. +For example, installing Python from the Store followed by Python from the +traditional installer with its ``PATH`` modification enabled will almost always +shadow the Store package's Python with the later install. + +User expectations that are un-met by the Store package tend to be performance +and technical. Due to the overhead of launching an app, Python starts up slower. +Because apps are designed to be isolated from each other, it is more difficult +to use hidden directories (such as ``AppData`` or ``TEMP``) to communicate +between different versions of Python, as each version has its own space. Apps +are subject to stricter security requirements that legacy applications usually +have disabled, such as DLL hijacking protection, which causes some libraries to fail. +The ``python3`` and ``python`` shortcuts are managed through system settings, +and the user interface is not very good (and not going to be improved, according +to Microsoft). Without managing these, it is relatively easy for an undesired +version to be launched, though in general the targets can only be changed +manually by the user, and not by merely installing another app. + +Both the NuGet package and the embeddable distro are as simple and reliable to +install as extracting an archive file, though it's worth noting that for many +Python users this is not a common task. They provide no install management at +all, and cannot be reliably updated other than by deleting and re-extracting. +User expectations that are un-met are almost always due to users selecting the +wrong installer. Both these packages are for specialised cases, and while they +are documented as such, the attraction of a plain ZIP file leads some users into +failure. + +Overview of PyManager +===================== + +PyManager is the internal name of our proposed replacement installer tool. It +will be distributed both in the Windows Store and on python.org as an MSIX +package. Downloading from either source will get an identical package, and +both will support automatic updates (through the Store) for new releases. + +The user visible name will be "Python Install Manager", published by the +Python Software Foundation. After publishing, we will request that Microsoft +adjust their ``python.exe`` stub to open to this new app. + +This app does not directly provide a version of Python, but it does provide the +global commands that users expect to work, as well as file associations and +Start menu shortcuts. The OS will prompt users to launch the app after install, +which will trigger an automatic install of the current release of CPython and +then launch it. From the user's perspective, they have the same initial +experience as today, with one added progress bar on first launch. + +The global commands provided by the app must be static and bundled into the app +itself. They can only change their behaviour at runtime, and cannot be +redirected to different executables except by the user (and then only to another +installed app). So the commands to be provided by PyManager are ``python.exe``, +``python3.exe``, ``py.exe``, ``pymanager.exe``. Each of these must have the +ability to inspect the user's system and choose the correct runtime to launch. +Additionally, ``py`` and ``pymanager`` will have management subcommands to allow +adding and removing runtimes. + +In line with :pep:`394` and the default behaviour of Windows, the recommended +command for launching Python is ``python.exe``. As provided by PyManager, this +will locate an existing install, either among those that PyManager manages or +using :pep:`514`, or it will install the latest available version of CPython and +select that. The ``python3.exe`` command behaves similarly, but is only allowed +to find 3.x installs from python.org. + +The ``py.exe`` command provided by PyManager will be recommended for most +management use, due to its brevity. ``py install ...``, ``py list ...`` and so +on. The proposed commands are detailed later. The existing behaviour of the +:pep:`397` launcher is preserved, however, launching through ``py`` will not +automatically install runtimes (by default). If one is requested but is not +installed, users will just get an error. The ``py exec ...`` subcommand, +however, will install automatically, and supports the same options as bare +``py``. + +These commands are added at very low priority in the user's ``PATH`` by the OS. +Every existing configuration we may have created on a user's machine will take +precedence over these commands, and so these are a last resort in place of an +error message. As a result, we can generally assume that a user is launching +these commands because they haven't configured a stronger preference (for +example, a user who has activated an environment will never launch our +``python.exe``, because activation will put a different one ahead of it, and a +user who wants precisely the behaviour of the existing ``py.exe`` can just +install it and will never launch our new one). + +The ``pymanager.exe`` command is to allow for handling ambiguous situations. +Existing installs of Python and the launcher may shadow ``python.exe`` and +``py.exe``, but in an automated environment, this can make administrative +scripts unreliable, and so the ``pymanager`` command is unlikely to refer to +something other than PyManager. It has all the subcommands, and launching it +with no command specified will print help for the user. + + +Replacing other installers +-------------------------- + +Our intent is to immediately stop publishing individual versions to the Windows +Store, and to deprecate and phase out the traditional installer by Python 3.16. +The embeddable distro will remain, but its listing on python.org download pages +will be phased out and it will be available only through PyManager. No changes +will be made to the NuGet packages. + +PyManager will be made available as an app package downloadable manually from +python.org, and the double-click install experience is generally smooth. This +provides an equivalent to the current approach of downloading from our site. +It will bundle a recent (unspecified) version of CPython so that the download +can be moved to a non-internet connected machine and still provide a Python +runtime after install. + +Some automated deployment scenarios do not work with the newer MSIX format, and +so a simple MSI will also be provided on python.org. This will have no options +or user interface, and require administrative privileges, which are typically +available for these kinds of scenarios. Additionally, the MSI will be required +by users on non-standard systems, such as Wine, that do not support the MSIX +format. While not officially supported, the MSI will enable these platforms to +continue using upstream distributions of CPython. The MSI would be discouraged +for most other users, and the MSIX is considered the default. + +It's worth noting that there is no way to make the MSI install fully compatible +with the MSIX, and users with both will likely encounter confusion or problems. +It is anticipated that only users without Store app support will use the MSI. + +Our release processes will start publishing plain ZIP packages to python.org. +These will be available from the FTP pages, but will not be listed directly on +regular download pages. + +Third-party tools that currently distribute their own builds of CPython will be +welcome to use ours, though will be expected to be the initial point of contact +for their users requiring support. + + +Project ownership and development +--------------------------------- + +PyManager will be developed and maintained in its own repository adjacent to +the CPython repository, and under the same terms. The CPython CLA will apply, +and all (and only) core developers will have commit rights. + +PyManager releases are independent from CPython releases. There is no need for +versions to match, or releases to be simultaneous. Unless otherwise arranged, +the PyManager release manager is whoever is the build manager for Windows. + + +Specification +============= + +.. note:: + In this document, all command line options will be shown with one or two + hyphens. In implementation, all options will support one or two hyphens or a + forward slash, to be permissive of both Windows and UNIX conventions. + +Exec subcommand +--------------- + +.. code:: text + + py [-V:] [interpreter opts] [script.py|-m module|-c code] [script args] + py [-3.*] [interpreter opts] [script.py|-m module|-c code] [script args] + python ... + python3 ... + pymanager exec -V:tag ... + pymanager exec -3.* ... + +This subcommand is used to select and launch a runtime. It is the default action +for the ``py`` command, and the only action supported by the ``python`` and +``python3`` commands. The default options are subtly different in each case for +consistency with existing use of these commands. + +This subcommand is available on both ``py`` and ``pymanager``. However, since +``py`` offers it by default, we would not expect users to use it there. The +intent is that the ``py``, ``python`` and ``python3`` commands are the default +ways to launch a runtime, and ``pymanager exec`` is for advanced scenarios. + +The ``-V:tag`` command is used to request a specific runtime from the command +line. The tag is a ``Company\\Tag`` pair, or just ``Tag`` if no slash is +included, and is used as defined by PEP 514. The ``-3.*`` option is interpreted +as ``-V:PythonCore\\3.*``. This option is only available for ``py`` and +``py exec`` variants. + +If no tag is specified on the command line, and a script file is specified, +the script will be inspected for a shebang. If one is found that matches a +recognised pattern, it will either provide the tag to be used for search, or it +will override all other processing and its specified executable will be launched +without further effort being made. This is to handle the (unfortunate) legacy +support of arbitrary Windows-specific paths being allowed in what was meant to +be a portability feature. In general, shebangs including simple patterns like +``/usr/bin/python3.13`` are intended, while those that use ``/usr/bin/env +python`` are unlikely to be of benefit since the environment tends to be less +reliable than our search. + +If no tag is yet requested, the ``VIRTUAL_ENV`` environment variable will be +consulted to see if an environment has been activated. If so, that will become +the request. + +If a tag has been requested at this stage, the ``python3`` command will verify +that it matches ``PythonCore\\3.*`` and exit with an error if not. This allows +allows the ``python3`` command to be used in an active environment consistent +with other platforms, but not if the environment would not have included the +command. This applies to most existing versions of Python on Windows. (The +alternative to this behaviour is to make ``python3`` always error when an +environment is active, as anything else would behave inconsistently for the +user.) + +If no tag is requested, the default will be consulted. For ``python3``, this is +``PythonCore\\3``, but for all other commands it is read from configuration +(which might involve an environment variable). If it's still empty, any tag will +be allowed. + +The best installed runtime matching the tag is then selected and launched with +the remaining command line. + +If no matching runtime is found, the ``py exec`` and ``pymanager exec`` commands +will automatically install and launch one (user configuration permitting). The +``py``, ``python`` and ``python3`` commands will report and error and exit. +However, in the case where no runtimes are available at all, including none +detected from other installers, the first runtime will be automatically +installed. This provides a useful first-launch experience, where a new user who +has just installed PyManager will directly launch the latest (or requested) +version of CPython. After this first time, errors will again be reported when a +requested runtime is not found. + + +Install subcommand +------------------ + +.. code:: text + + py install [-s|--source ] [-f|--force] [-u|--upgrade] tag [...] + py install [-s|--source ] [-t|--target ] tag + py install [-s|--source ] [-d|--download ] tag [...] + py install --refresh + +.. note:: + This and all later subcommands are also available under ``pymanager``. + However, as we intend for ``py`` to be the usual command, we only show that + one. + +This subcommand will install one or more runtimes onto the current machine. +The tags are ``Company\\Tag`` pairs (or just ``Tag`` if no slash is included), +and are used to search the index file. Company names match as case-insensitive +prefixes, preferring a full match over a prefix, and tags use case-insensitive, +number-aware matches, with dotted numbers treated as versions. Tags must match +one of the listed "install for" tags, and entries list multiple such tags to +handle abbreviated requests. The special tag ``default`` resolves to the user's +configured default (typically ``3``; see Configuration later for details on +configuring settings). + +Tags may also be specified as a constraint, using ``>``, ``>=``, ``<``, ``<=`` +or ``!=`` followed by the ``Company\\Tag`` or ``Tag`` value. When matching a +constraint only the primary tag metadata is used for comparisons. Since the +comparisons are version-aware, constraints such as ``>3.10`` will select +3.11 as a minimum, while ``>3.10.0`` may select 3.10.1. + +The behaviour of constraints against arbitrary tags is likely to be unintuitive +in some circumstances. It is anticipated that constraints will mainly be used +with upstream releases, which typically use version-shaped tags, and primarily +for cases where other metadata such as ``Requires-Python`` are being handled. +Users are expected to use shorter tags for convenience, rather than ranges. + +The default index file is hosted on python.org, and contains install information +including package URLs and hashes for all installable versions. An alternate +index may be specified by the user using a configuration file or the +``--source`` command line option, or their administrator through a configuration +file. The requested tag is matched against the index file, and if an exact match +is found the package will be selected. In the case of no exact match, a prefix +match will be used. In both cases, numbers in the tag are treated logically - +that is, ``3.1`` is a prefix of ``3.1.2`` but not of ``3.10``. See Index Schema +below for information on exactly how install tags are specified in the index +file. + +If a tag is already satisfied by an existing install, nothing will be installed. +The user must pass an ``--upgrade`` or ``--force`` option to replace the +existing install; the former will only replace it with a newer version, while +the latter will remove and replace even with the same version. + +Calling the command without providing any tags will not install anything, but +will display the help text and an error (as for any invalid option). However, +passing ``--upgrade`` with no tags will attempt to upgrade all installs. + +Passing ``--refresh`` will regenerate all metadata and shortcuts for all +installs. This is intentionally applied to all installs at once, as shortcut +prioritisation relies on all installs being consistent (for example, the latest +3.x version should get the ``python3.exe`` shortcut, which gets complicated if +users can choose to only refresh an older install). + +If a ``--target `` option is passed with only a single tag, that runtime +will be extracted to the specified directory without being registered as an +install (or generating aliases or shortcuts). This is intended to cover +embedding cases, or downloading the files for incompatible platforms. Passing +multiple tags with ``--target`` is an error. + +If the ``--download `` option is passed, runtime packages are downloaded +but not installed. They are stored in the specified directory as their source +packages, and an ``index.json`` is created that references these files. This +index can be used later to perform offline installs with ``python install +--source [tag ...]``. + + +Uninstall subcommand +-------------------- + +.. code:: text + + py uninstall [-y|--yes] [--purge] [tag ...] + +This subcommand will uninstall one or more runtimes on the current machine. Tags +are exactly as for the install command, including prefix matching, but only +inspect existing installs. Unless the ``--yes`` option is passed, the user will +be prompted before uninstalling each runtime. + +If the ``--purge`` option is passed with no tags, then (after confirmation) all +runtimes will be removed, along with shortcuts and any cached files. Passing any +tags with ``--purge`` will produce an error. + +Uninstalling PyManager does not uninstall any runtimes that were installed. For +technical reasons, this would not be reliably possible (we cannot run arbitrary +code at uninstall time), and so instead we deliberately ensure that anything +that has been installed will continue to work. Reinstalling PyManager allows +management of these installs to resume. Running ``py uninstall --purge`` before +uninstalling PyManager will perform a complete uninstall. + + +List subcommand +--------------- + +.. code:: text + + py list [-f|--format ] [-1|--one] [--only-managed] [tag ...] + py list [-f|--format ] [-1|--one] [--online] [--source ] [tag ...] + py [--list|--list-paths|-0|-0p] + +This subcommand will list any or all installs matching the specified tags or +ranges. If no tags are provided, lists all installs. Runtimes not managed by +PyManager (including an active virtual environment) may be listed separately. + +The default format is user-friendly. Other formats will include machine-readable +and single string formats (e.g. ``--format=prefix`` simply prints ``sys.prefix`` +on a line by itself). The exact list of formats is left to the implementation. + +If ``--one`` is provided, only the best result is listed. This is to assist +shell scripts that want to locate the default (or a suitable) runtime without +launching it. (Note that "best" is loosely defined, but will always be the +user's preferred default environment if it is included in the results.) + +The ``--only-managed`` option omits runtimes that were discovered but are not +managed by PyManager, for example, those found using a regular PEP 514 lookup. + +Passing ``--source`` (or ``--online`` to implicitly pass the default source) +will search an online index rather than currently installed runtimes. The option +is here rather than on the ``install`` subcommand because the filtering and +formatting options are already available on ``list``. + +The legacy ``--list``, ``--list-paths``, ``-0`` and ``-0p`` arguments from the +``py.exe`` launcher will also be provided. However, they will not support the +new options listed here, and are limited to reproducing the output from the +existing launcher. Unmanaged installs are not distinguishable in this listing. + + +Help subcommand +--------------- + +.. code:: text + + py help + +This subcommand will display the help text for each specified command, or if +none specified, will show the list of commands. Specifying one command is the +equivalent of ``py --help``. Showing the list of subcommands is the +default action for the ``pymanager`` command. + +The command is added primarily to offer a simple way to tell users how to find +more information: they can be told to run ``py help``. This avoids having to +override or extend the ``python -?`` output, which otherwise forwards to the +selected runtime and already prints at least one screen's worth of text. + +After an automatic install (e.g. running ``python`` with nothing installed), a +message will be displayed telling users that they can run ``py help`` for more +information on how to manage their installs. + + +Environment Variables +--------------------- + +No environment variables can be updated automatically when installing a Store +app, and so no updates will be done automatically. The core commands should +already be available on a correctly functioning machine. + +One directory within the user's PyManager data directory is set aside for +generated aliases. If desired, the user can add this directory to their ``PATH`` +themselves. The contents of this directory will be managed by PyManager, and +will contain executables to directly launch installed runtimes (for example, +``python3.exe`` and ``python3.13.exe`` for an install of Python 3.13). Whenever +aliases are added to this directory, ``PATH`` will be checked and if it is +missing, the user will be presented a message containing the path to add. + +Scripts installed by packages installed into a runtime will be in yet another +directory. Due to the current design, we do not believe it is safe to have them +all install into a single directory, or a directory shared by multiple runtimes. +However, a future development may include a command for PyManager to generate +its own entry points based on metadata in installed packages. + + +Start Menu Shortcuts +-------------------- + +A Start Menu shortcut will be added to launch PyManager documentation in the +user's default web browser. No applications are added to the Start Menu. + +When installing Python runtimes, the install definition may specify Start Menu +shortcuts to create for the install. + + +File Associations +----------------- + +Standard file associations will be created when installing PyManager, and will +launch scripts and packaged apps with PyManager's global ``python.exe`` alias. +This provides sensible behaviour for users who are double-clicking on scripts or +``.pyz`` files. + + +Windowed Executables +-------------------- + +For each of the global aliases described earlier, a ``*w.exe`` also exists. +These launch without creating or attaching a console window, which typically +means they will only display UI created by the script. For example, IDLE always +launches using ``pythonw.exe``, as this avoids an unnecessary native console. + +These commands otherwise behave identically to their console counterparts. + + +Configuration +------------- + +PyManager is configured using a hierarchy of JSON-based configuration files. +Command-line options always override configuration file options. Configuration +files in user editable locations may be disabled by a configuration or +command-line option. + +In ascending order of priority, these will be located: + +* within the app package +* specified by admin-only configuration (see below) +* in the ``base_config`` setting (default: none) +* in the ``user_config`` setting (default: ``%AppData%\\Python\\PyManager.json``) +* in the ``additional_config`` setting (default: ``%PYTHON_MANAGER_CONFIG%``) +* specified with the ``-c`` command line option + +The specific behaviour of each configuration option is left to implementation. +However, a number of intended options are discussed in other sections. + +App package configuration is provided to allow PyManager to be embedded in other +applications or packages. For example, an alternative distribution may want to +include PyManager but have it locate installs from their own index. The app +package configuration allows reusing our build and overriding the default +settings. + +The ``user_config`` and ``additional_config`` settings are pre-configured in +earlier configuration files, allowing them to be overridden by admin-only +configuration or an alternate root configuration. If a configuration file +overwrites the setting that caused the file to be loaded, it is ignored. +The ``base_config`` setting is similar, but starts empty and is intended for +easy overriding through admin configuration. + +Admin-only configuration is provided to allow administrators to manage systems +under their control using existing tools, such as group policy or registry +updates. By design, these controls cannot be overridden, such that it is +possible for administrators to deploy policy that prevents or limits the use of +PyManager. These controls are essential to allow PyManager to be deployed safely +into certain environments, and without them, it would simply be disallowed and +those users would have no access to Python. + +The intent is for the main admin-only configuration to be a path to a new +``base_config`` configuration file that an administrator can deploy to any +controlled location. This allows a network administrator to control the source +of their users' default Python runtimes, without forcibly restricting them, or +to override the other sources for configuration files (apart from the command +line option). + + +Index Schema +------------ + +The index file is made available either online or locally, and provides +PyManager with all the information needed to find, select, install, and manage +any Python runtime. + +The index is stored as JSON. The main top level key is ``versions``, which +contains a list of objects. Each version object has its own schema version, and +there is no overall file schema version. Future changes may add additional +top-level keys to provide functionality that cannot be safely integrated into +an existing one. + +Version objects may be split between the index file and a ``__install__.json`` +stored in the root of each package archive. The entries in the bundled file will +fill in any gaps from the index file. This is intended to allow the typically +large ``shortcuts`` key to be removed from the index file, but may also extend +to ``alias``, ``executable`` and ``executable_args``. Omitting other keys from +the index may result in problems installing the package. Ensuring correct +behaviour is left to the implementation - this is not an interoperability point, +and so we do not intend to specify the details here (beyond the normal compatibility +requirements that apply). + +A second top-level key ``next`` contains an optional URL to another index. This +may be used if PyManager cannot find a suitable package in the included +versions. The intent is to allow for older indexes to be archived and only +accessed when required, reducing the size of the initial download without +cutting off users from older versions. When searching for a suitable install, +later indexes will not be searched if a viable candidate is found (in other +words, the first index consulted should have the latest versions in it). + +The initial schema is shown below: + +.. code:: python + + SCHEMA = { + "versions": [ + { + # Should be 1. + "schema": int, + + # Unique ID used for install detection/side-by-side. + # Must be valid as a filename. + "id": str, + + # Name to display in the UI + "display-name": str, + + # Version used to sort packages. Also determines prerelease status. + # Should follow Python's format, but is only compared among releases + # with the same Company. + "sort-version": Version, + + # Specifies platforms to consider this package for. + # Initially, 'win32' is the only supported value. Others may be + # defined in the future. This condition is evaluated silently, and + # is not intended to replace platform requests in "install-for". + "platform": [str], + + # Company field, used for filtering and displayed as the publisher. + "company": str, + + # Default tag, mainly for UI purposes. + # It should also be specified in 'install-for' and 'run-for'. + "tag": str, + + # List of tags to install this package for. This does not have to be + # unique across all installs; the first match will be selected. + # For example, the 3.10.5 package may list '3', '3.10' and + # '3.10.5' so that any of those may be specified to install it. + # Matches are number aware, so that 3.1 is not a prefix of 3.10. + "install-for": [str], + + # List of tags to run this package for. Does not have to be unique + # across all installs; the first match will be selected. The target + # is the executable path relative to the root of the archive. + # Explicit args (optional) are inserted before user args. + "run-for": [{"tag": str, "target": str, "args": [str], "windowed": int}, ...], + + # List of global CLI aliases to create for this package. Does not + # have to be unique across all installs; the first match will be + # created. + "alias": [{"name": str, "target": str, "windowed": int}, ...], + + # List of shortcuts to create for this package. Additional keys on + # each instance are allowed based on the value of 'kind'. + # Initially, 'kind' supports the following values: + # * 'pep514' - other keys define registry values to set + # * 'start' - generate shortcuts in the user's Start Menu + # * 'uninstall' - generate an Add/Remove Programs entry + "shortcuts": [{"kind": str, ...}, ...] + + # Default executable path, relative to the root of the archive. + # Usually the values from 'run-for' will be used instead, and this + # is mainly for display purposes. + "executable": str, + # Default executable args + "executable_args": [str], + + # URL to download the package archive from + "url": str, + + # Optional set of hashes to validate the download. Hashes are stored + # as hex digests. Any hash supported by hashlib without OpenSSL is + # permitted. + "hash": { + "": str, + } + } + ], + + # URL (or relative path) to the next index file + "next": str, + } + + +Shebang Processing +------------------ + +For limited compatibility with scripts designed for sh-like shells, PyManager +will check scripts for a shebang line. A shebang line specifying a Python +command will be used (when not overridden on the command line) to select a +suitable runtime for the script. + +Unlike the support currently in the ``py.exe`` launcher, we propose to reduce +this functionality to only support Python commands where the command matches +a global alias listed for an install, with anything not matching being treated +as an arbitrary executable path. In practice, this is expected to produce the +same (or better) results for non-contrived cases, but may result in subtle +changes where users are relying on unspecified edge case behaviours of the +existing launcher. + +The specific patterns to be detected are left to the implementation, but should +be largely compatible with the existing launcher. + + +Rationale +========= + +"Changing" the python.exe command +--------------------------------- + +It may be argued that the global ``python.exe`` alias provided by PyManager is +"not real Python" and so should use a different name. While this is strictly +true, there are three reasons we argue that it should be used. + +Firstly, thousands of users *daily* install through the Store page after being +led there by having typed ``python`` at the terminal of a clean machine. Due to +how this redirection is implemented, if the app they install does not provide a +``python`` command, then the redirection will remain in place. In order to +ensure that users do not get stuck always going back to the Store, we need to +provide this command. (The same applies to ``python3``.) + +Second, the alternative to the "not real" alias is not "the real" one. It's +nothing. We don't have the ability to replace the global static alias with one +that follows the user's preference or installs, and so the alternative would be +to provide nothing and have ``python`` be an error in all cases. This is worse, +and in our opinion, actively harmful to Python's reputation. + +Third, although the underlying implementation of the ``python`` alias is more +complex than the default ``Programs/python.c``, the experience of using it is +identical. The alias is only launched in the absence of another expressed +preference (that is, there's nothing else on ``PATH``), it respects any +indirect preferences (such as through configuration or shebangs), and it +launches the most appropriate version of Python available on the user's machine. +This is much closer to the desired behaviour of the global ``python`` command +than any alternative. + +It has been noted that Gentoo also distributes an intelligent ``python`` +command, in order to serve their users better than a simple symlink. + + +Replacing py.exe +---------------- + +The ``py.exe`` launcher exists to provide some of the functionality that will be +replicated by PyManager - specifically, the ability to launch an already +installed runtime. Despite its long history, the launcher does not seem to have +become the preferred method for most users, with many preferring the global +modifications to the ``PATH`` environment variable. However, the command itself +has come to be relied upon, and should be preserved as long as possible. This is +achieved in two ways. + +Firstly, we install our own ``py.exe`` alias with PyManager that provides the +same functionality, along with PyManager specific functionality. This is +intended to become the default/preferred install of ``py.exe`` over time. + +Second, we generate PEP 514 metadata (when requested) for each install, which +allows a legacy ``py.exe`` to continue to work normally with installs managed by +PyManager. + +Due to how the existing ``py.exe`` launcher configures itself, and how the MSIX +package for PyManager is constrained, it is not possible for PyManager's ``py`` +alias to override the launcher. As a result, users who install the launcher will +always find ``py`` resolving to the launcher. Ultimately, the only way to +resolve this in favour of PyManager is to uninstall the launcher, which can be +done through the standard Installed Apps control panel. + +We propose to update the existing launcher, which will continue to be released +with the traditional installer, to detect attempted use of the new subcommands +and provide a useful message for the user rather than the current error. These +warnings can also detect the unlikely case where a user is intentionally +launching extensionless files by those names and retain that behaviour, allowing +a viable alternative for users who cannot make other changes to their setup. + + +Interaction with venv +--------------------- + +An activated virtual environment, as implemented by the standard library +:mod:`venv` module, will modify the user's ``PATH`` environment variable to ensure +that the venv launcher will take precedence over other executables. As a result, +when a venv has been activated, PyManager can only be launched by its aliases +other than ``python``. When an active virtual environment is detected, it will +be treated as the user's default runtime (except for uninstall), which ensures +that other commands will also behave as expected. + +This means that working virtual environments will behave as they do today with +no additional support from PyManager. + + + +Backwards Compatibility +======================= + +In general, there are no compatibility guarantees to the install process between +minor versions (``3.x`` to ``3.y``), and so "having to use a different +installer" is not considered compatibility breakage. The versions of Python +installed are only impacted by this change to the extent that the install method +modified their behaviour. In general, most installs will be closer to the +behaviour of having been built from source by the user themselves. + +That said, there are a number of changes that will impact certain users when +they do move to a new install process. This section outlines as many of these +changes as we are aware of, in no particular order, and will likely form the +basis of a migration guide. + + +Scripted downloads +------------------ + +Users who wrote scripts to generate the download filename of our old installer +will find those scripts are broken. These URLs were never guaranteed stable or +predictable, and so we have no recourse to do anything other than apologise and +suggest users use our own tooling for downloads. + +The deprecation period for the traditional installer allows time for these users +to learn about the upcoming change. Where possible, we will add deprecation +warnings to the traditional installer. + + +Scripted installs +----------------- + +Users who wrote scripts to execute our installer with particular options will +have to change their script. Most options have been removed, to begin with, and +those that remain have new spelling. Since it is not possible to reach a state +where options for the old installer are being passed to the new without manual +intervention (that is, someone has to change the command already), this is +considered an acceptable change. + +The deprecation period for the traditional installer allows time for these users +to learn about the upcoming change. Where possible, we will add deprecation +warnings to the traditional installer. + + +Old runtime installed +--------------------- + +Users with existing runtimes installed will find them selected by PyManager and +its aliases, provided the registration is not corrupt. + +The priority order among installed runtimes has changed to only include +prerelease versions when specifically requested (for example, ``-V:3`` will +match 3.14.0 rather than 3.15.0a1, but ``-V:3.15`` will match 3.15.0a1), and to +correctly sort text suffixes on tags (for example, 3.14t is now *lower* +priority than 3.14). + +While it is possible to provide warnings in cases where this may be impacting a +user, such warnings would be considered very noisy (e.g. a message every time +you launch ``python`` because you have a prerelease installed that wasn't +selected) and require complicating the selection logic unnecessarily. This +change will be documented only. + + +Old ``py.exe`` launcher installed +--------------------------------- + +Users who do not manually uninstall an old ``py.exe`` launcher will find that +both their existing and new installs of Python are found, though where versions +match the existing install will take priority over the new install (whereas the +new ``py`` would select the new install). + +They will also find that commands such as ``py list`` do not work. The solution +here is to use Windows Settings to uninstall the launcher. + +There is no way to detect that a user has accidentally left an old ``py`` +installed, or to remove it for them. This change will be documented only. + + +Misconfigured venv activated +---------------------------- + +Users who activate a corrupt or misconfigured virtual environment that is either +missing its ``python.exe`` or has it not on ``PATH`` may receive a different +error from before. + +PyManager's global ``python`` alias will be found and executed instead, +suppressing any system "not found" error. As it fails to find the environment's +actual runtime, it will then fail, though the code and message may be different. + +As this scenario requires an already corrupt system, this change will be +documented only. + + +Old version availability +------------------------ + +Python versions prior to the first release of PyManager can be backfilled into +the python.org index, either based on newly repackaged archives or using the +almost equivalent packages from NuGet (the latter does not include Tcl/Tk, +making them significantly incompatible for some users, but this is likely okay +for especially old versions). + +As of this PEP's acceptance, the plan is to create packages for all versions of +Python 3.10.0 onwards (including patch releases, but excluding pre-releases), so that all non-EOL releases can +be fully installed using PyManager. Versions from Python 3.5.0 onwards (excluding +pre-releases) will directly reference the packages on NuGet, making them easily +available in reduced form. Neither approach requires rebuilding the existing +releases, nor are any changes required to the sources of those releases. + + +Administrator installs +---------------------- + +Installing a copy of Python for all users is no longer possible, as PyManager +will only install into the user's own directory. No scenario has been presented +to show that per-machine installs are in line with our intent for the upstream +distribution, and so we will simply not provide an option for them. Third +parties who desire this functionality are encouraged to provide their own +distributions. + +PyManager can only be installed for all users, though an MSIX does not require +administrative privileges to install, and can be extensively configured by an +administrator, including to constrain the actual runtimes which users may +install. Additionally, PyManager supports local extraction for bundling, and so +embedding apps can easily generate their own layout, which can be installed for +all users if they so desire. + +As this scenario requires administrator intervention with or without any +changes, this will be documented only. + + +Build-time installs +------------------- + +Users using the embeddable distro may have to change to a new method for +discovering the URL to the packages, though the recommendation would be to use +PyManager to discover and install. No differences are anticipated due to the +change of installer, and the embeddable distro package would be identical to +today. + +No changes to the NuGet packages are proposed. + + +Single architecture installer +----------------------------- + +The current proposal only makes an Intel 64-bit build of PyManager available. +This will prevent users on 32-bit only operating systems or CPUs from being able +to install PyManager. As this is a vanishingly small proportion of machines +today, and an even smaller proportion of developer machines (the target audience +for PyManager), we are not concerned about excluding users. + +Windows ARM64 machines support running binaries built for Intel 64-bit through +efficient emulation. CPython 32-bit builds will still be available as they have +an important role in integrating with 32-bit executables, which does not allow +substitution of 64-bit binaries. As PyManager runs as a standalone executable, +this is not a necessary feature for the manager. + + +Test suite and debug symbols +---------------------------- + +The existing installer optionally allows installation of the Python standard +library test suite, and optional installation of debug symbols. Neither of these +are necessary for most users, but they are convenient for some. Preliminary +testing shows that omitting the test suite and debug symbols saves about 60% of +the size of the compressed package (from 46MB to 18MB). + +The "default" CPython packages installed by PyManager will therefore not include +the test suite or the debug symbols. However, there will be a second set of +packages that do include these extras, grouped under ``PythonTest`` (as opposed +to the default, ``PythonCore``). For example, where ``py install 3.13`` would +install the default, ``py install PythonTest\\3.13`` would install a second +runtime with the additional files (which can either be launched with ``py +-V:PythonTest\\3.13``, or simply ``py -V:3.13`` if no equivalent ``PythonCore`` +version is installed). + +Debug binaries are no longer distributed, and all other optional features are +included by default. + + +Global pip command +------------------ + +Unlike the current Windows Store install, no global ``pip`` command is included +(the traditional installer also does not include a global ``pip`` command, +unless the options to modify ``PATH`` and to install pip are selected; the first +of these is off by default). This impacts global installation of packages, which +is already discouraged, but has no impact on activated virtual environments. + +The existing recommendation remains, which is to run ``python -m pip`` or +``py -V: -m pip`` to launch pip. + + +Security Implications +===================== + +In this section we compare the security implications of the installer itself to +the existing installers. The implications of Python being installed on a machine +are out of scope, and the ability of a malicious user to execute the installer +is also out of scope. + +The typical risk introduced by an installer is that an elevated install may make +changes to a system that allow a low-privileged user to later affect a +high-privilege user, for example, by inappropriately setting access control on +shared folders. PyManager only operates as the same privilege level as the user, +and therefore cannot introduce any escalation path. + +An install using the MSI described earlier may introduce additional risks, due +to using older installer technology. Typical users are directed towards the MSIX +or Windows Store install paths, which are safe, and it is assumed that users +of the MSI are capable of ensuring the security of their install process (for +example, by correctly quoting their commands to launch the installer and +ensuring the initial system configuration is suitable). + +Once PyManager is installed on a machine, it is likely that malicious users will +use it to install Python. The admin-only configuration described earlier in +"Configuration" is intended to control these scenarios. Ultimately though, an +attacker who can run PyManager is able to do whatever the user can do, and only +a complete application whitelisting approach can prevent the use of Python. + +Acquisition of packages over HTTPS is protected by the connection security. We +use native Windows download mechanisms that support public and enterprise +certificates, as well as authenticated proxy servers. The feed may include +hashes for downloadable packages, which will be verified, but there is no +third-party hosted verification built into PyManager. This is consistent with +today's model. + +Runtime installs by PyManager are fully accessible by, and modifiable by, the +current user. This is equivalent to typical installs using the traditional +installer or a NuGet package, but is more vulnerable to tampering than a Store +install or a per-machine install with the traditional installer. It is not +possible to fully protect an install from the user who installed it. +Reinstalling or updating an install performs a clean install, which will revert +any tampering that may have occurred since the original install. + +The aliases generated by PyManager when installing a runtime are designed to +use a signed, unmodified executable that uses an adjacent data file to launch +the correct target. This can be easily abused to direct the launcher to launch +an alternative, however, the only way to resolve this would sacrifice the trust +in the executable itself, making it trivial to replace it instead of the data. +Such risk already exists, and is equivalent to replacing the script that a user +may launch, or any part of the standard library. Importantly, since aliases are +not shared between users, there is no escalation of privilege along this route. + +PyManager has no mechanism to perform a per-machine install. This may be useful +functionality to some users, as it would allow an install to be completely +unmodifiable by the regular user (excluding virtual environments and the user +site folders). Such functionality may be manually imitated by an administrator +using PyManager and other OS commands, but it is not considered a critical +workflow. The recommended alternative is for an administrator to provide +PyManager and override its configuration. + + +Impact on Existing PEPs +======================= + +This proposal would effectively replace :pep:`397` ("Python launcher for Windows") +and :pep:`486` ("Make the Python Launcher aware of virtual environments") by +defining the same functionality as part of a new tool with the same name. Both +are already considered final, and the launcher is defined by its documentation +and normal compatibility processes. New functionality is based on the current +implementation, and not the original PEP text. + +This proposal has no impact on :pep:`394` ("The “python” Command on Unix-Like +Systems"), and is believed to be consistent with it in devising an approach for +Windows that allows similar guidance to be given to users on all platforms. + +This proposal has no impact on :pep:`514` ("Python registration in the Windows +registry"), and in fact improves our ability to follow it with a more flexible +system for registering our own runtimes. Tools that follow PEP 514 will find any +runtimes that choose to use the registration, regardless of how they were +installed. + + +How to Teach This +================= + +Basic Use +--------- + +A central goal of this proposal is that "type 'python' in your terminal" will be +sufficient instruction for the most basic cases. Thanks to the redirector added +by Microsoft, following this instruction will at least result in something +useful happening, and with PyManager we can ensure that "something useful" means +that the user is running the latest version. + +To explain what is actually happening, we propose the following as introductory +text: + +.. code:: text + + Python installs on Windows are managed using an installer tool. After it has + been installed, you can run ``python`` to launch the interpreter, and it will + choose the best version already installed, available online, or referenced by + the script you are launching (if any). If you have a preference for a + particular version, you can specify it with ``py -V:`` followed + by the rest of your command. + + To install a version of Python without running any command, use ``py install + ``. You can see all of your installs with ``py list`` and remove them + with ``py uninstall ``. Run ``py help`` to see all the options that + are available. + + Because each version of Python will be shared by all your projects, we + recommend using virtual environments. This will usually be created for a + particular Python version by running ``py -V: -m venv .venv``, and + activated with ``.venv\Scripts\Activate``. Now, rather than the install + manager, ``python`` or ``py`` will always launch your virtual environment, and + any packages you install are only available while this environment is active. + To get access to the manager again, you can ``deactivate`` the environment, or + use ``py ``. + +Many Python projects include information about how to launch their projects as +part of their own README files. Historically, such information has been +complicated due to the range of options available to users. We propose that, +after the install manager is published, such guidance could be written along +these lines: + +.. code:: text + + To install and use our application, first install Python following the + guidance for your operating system at https://docs.python.org/using/. Then, + create a virtual environment and use 'pip' to install. + + ``python3 -m venv .venv`` + ``source .venv/bin/activate`` or ``.venv\Scripts\Activate`` (on Windows) + ``python -m pip install OurAwesomePackage`` + ... + +If instructions will not include information about virtual environments, then +the ``python`` or ``python3`` command can be shown, and on Windows both will +operate as intended for users with the install manager. + +Instructions currently referring to ``py`` for Windows can continue to do so, as +the install manager provides a practically equivalent command. Projects that +wish to provide Windows-specific instructions, such as by using the ``-V:`` +or ``--install`` options to install the correct version, should also link to the +:ref:`documentation ` as guidance for +ensuring that the install manager is installed. + + +Uninstallation +-------------- + +Complete uninstallation is an important topic to cover before a user is likely +to consider removing the install manager. Since not all parts of installs can be +automatically cleaned up when removing the manager, we choose not to remove most +of them. So while the default ``python`` and ``py`` commands will go away, all +the runtimes that were installed are still present and usable. + +We suggest an explanation like this: + +.. code:: text + + Before you uninstall the Python install manager, you'll want to uninstall any + runtimes that you added. This can be done easily with the "purge" option: + + ``py uninstall --purge`` + + This will remove all installs and any shortcuts that would otherwise be left + behind. If you already removed the manager, you can reinstall it and run the + above uninstall command again to clean up. Individual runtimes can be + uninstalled by specifying the tag instead of ``--purge``. Tags can be found + by looking at ``python list``. + + +Configuration +------------- + +Configuration files are a common feature that will be documented, but do not +need to be taught to regular users. Similarly, advanced deployment options, such +as those that might be used by system administrators or organisations wanting +their users to use a preferred index, are best covered in reference material. + + +Custom Index +------------ + +We suggest that indexes only need to be introduced when instructing users to +install a specialised runtime or distribution. Administrators seeking to provide +an index are anticipated to actively seek out the relevant information in the +documentation. + +To explain how and when to use an alternate index, we propose text along these +lines: + +.. code:: text + + Our distribution can be installed on Windows using the Python Install Manager + (include link) by referencing our index: + + ``py install --source latest`` + + This index contains all our versions. Use ``py list --source `` to + see everything that is available. + + +Reference Implementation +======================== + +The reference implementation is available at `the author's repository +`_ +with a precompiled MSIX package under `Releases +`_. This sample includes a bundled +index, rather than a hosted one, and references a range of existing NuGet +packages to allow install testing. + +Reference documentation can also be found in `the same repository +`_. + + +Rejected Ideas +============== + +Make PyManager available on all platforms +----------------------------------------- + +While we are not inherently opposed to this idea, it relies on many more +components being aligned before it can become possible. + +Firstly, as it stands, the reference implementation has a lot of +Windows-specific knowledge. Equivalent knowledge for other platforms would need +to be collated and implemented, as well as any additional behaviours specific to +non-Windows platforms. + +Second, we need a source of pre-built, relocatable binaries that can be +extracted onto the system. While such sources do exist, due to our position in +the supply chain, we cannot justifiably use them (they should be using us). For +Windows, our own binaries already meet these criteria, so we can repackage them +without modification. + +Third, the current implementation relies on a bundled Python runtime, which must +be isolated from any user interference for obvious reasons. This would also +require the relocatable binaries mentioned above, which we currently only have +for Windows. + +Due to the additional steps needed to make this functional on other platforms, +and the fact that there isn't a need to replace existing installers for those +platforms, we consider this idea out of scope for this PEP. It may be pursued in +the future (and the contributors most likely to do so have indicated that they +are looking into it and would be able to use a consistent interface). + + +Include a runtime pre-installed with the manager +------------------------------------------------ + +The proposal is to have a full Python runtime included with PyManager, so that +its ``python.exe`` alias can refer directly to it rather than resolving to the +best available version dynamically. + +It is very important for stability and updates that runtime releases are +fully independent of the manager. Updating the manager should be possible +without affecting any existing runtime installs, and likewise there should be +no requirement to update the manager to get a newer runtime. + +Hypothetically, if we were to include Python 3.14.0 with the manager such that +it did not need to be installed, it would be a breaking change to later replace +that with 3.15.0. As we only have a single install for the manager, this would +result in the newest installs getting the oldest runtime. + +This would also ignore the status of PyManager's ``python`` alias as being of an +unspecified version - when the user is launching this alias, it's because they +didn't care what version they get enough to specify one. In that situation, we +ought to select the best available, and allow them the options to stabilise it +as is appropriate (whether through a shebang or an active environment). + + +Include a runtime INSTEAD OF the manager +---------------------------------------- + +This is the current situation, which we are trying to change. If you read this +far and still chose to make this argument, please go back and start again. + + +Use a built-in module rather than subcommands +--------------------------------------------- + +Two alternatives to using commands like ``py list`` or ``py install`` that have +been proposed are to use either dedicated modules, invoked like ``py -m list`` +and ``py -m install``, or a single dedicated module invoked like ``py -m manage +list``. This idea is rejected on the basis that it attempts to reuse existing +semantics for a scenario that cannot be reliably implemented by those semantics, +and so will require a special case that is harder to explain, understand, and +maintain. + +The main reason this idea is rejected is due to the interaction of two otherwise +desirable semantics: first, that the default ``py`` command should launch the +latest available runtime as if it were launched directly; and second, that the +behaviour of ``-m`` should not be treated as a special case in some +circumstances. If the first part were dropped, we would freely modify the +command to behave as users expect - nobody would be raising compatibility +concerns at all if we were agreed to completely break compatibility. However, if +the second constraint were dropped, users would bear the burden of the ensuring +confusion. (We aren't proposing dropping either - this is a rejected idea, after +all - but it helps to illustrate what the options are.) + +First, since one of the subcommands is intended to install your first runtime, +we cannot treat ``python -m [manage] install`` as if it is running through the +default runtime - there isn't one! It inherently requires special case handling +in order to read the command and execute it through a different program. + +Additionally, Python allows other options to precede or mingle with the ``-m``, +which would have to be supported by this special case. + +Finally, the semantics of the ``-m`` option include searching the initial +``sys.path`` for matching module names. This is a considerably more broad search +than a bare name. ``py -m install`` would gladly execute ``install.py``, +``install.pyc``, ``install.pyd``, ``install\\__init__.py``, and more after +searching a number of directories found by inspecting the file system, the +environment, the registry, as well as any transitively included paths found in +those. Compared to ``py install``, which would *only* look for a file called +precisely ``install`` in the current working directory, the ``-m`` behaviour is +far more likely to be already relied upon by real scenarios. (For example, +Django projects typically have a ``manage.py`` script, meaning that ``py -m +manage`` would always behave incorrectly.) + +Changing ``py -m install`` to *not* behave like ``-m``, but instead to execute +an internal command, is vastly more likely to break users than changing +``py install``. As such, this idea is rejected. + + +Use a new command-line option rather than subcommands +----------------------------------------------------- + +A reasonable alternative to subcommands is to specify their names with leading +punctuation, like an option rather than a subcommand. For example, this may look +like ``py /install ...`` rather than ``py install``, or ``py --list``. Because +some of these are currently errors for a normal CPython interpreter, they +could be added without any backwards compatibility concern. + +Notably, however, the typical Windows format of a leading slash is not an error +in CPython. Windows users therefore cannot directly transfer existing knowledge +and must learn a new way to specify options. As we are proposing a Windows +specific tool, this is a terrible start. Additionally, those users familiar with +Unix-style command lines will recognise the misuse of options as commands. + +We desire to create a clean interface, and starting with a design that includes +obvious warts or learning challenges is counter to that goal. Modern tools +universally use subcommands for these purposes, and so the idea to use something +different is rejected. + + +Improving the current traditional installer instead +--------------------------------------------------- + +Rather than creating a new install mechanism, we could invest in maintaining the +current installer. At this stage, however, our current installer is based +entirely on retired technology. Windows is no longer developing the Windows +Installer service, and Wix are no longer improving the version of their toolset +that we use. Migrating to a newer Wix Toolset is a significant amount of work, +and ultimately still leaves us tied to old technologies. + +As mentioned earlier, the most beneficial functionality provided by Windows +Installer is not used for CPython, and generally has caused more issues than it +has ever solved (for example, accidental downgrades due to automatically +collected file version information). + +The implementation of the Burn bundle, which is our primary source of installer +logic, is in C++ and integrated into a framework that few core developers are +familiar with. This makes maintenance challenging, and is not a good long term +position to take. Migrating desired features such as registration-free installs +into the Burn bundle is not possible (without writing the end-to-end +reimplementation and integrating it as an afterthought). + +Our view is that maintaining the current traditional installer is at least as +much effort as implementing a new installer, and would not provide meaningful +benefits for the core team or for our users. As such, this idea is rejected. + + +Delete the Store package completely +----------------------------------- + +Removing the Store packages would reduce the number of options users face when +choosing a Python runtime. By all measures apart from reliability and security, +the traditional installer is entirely sufficient as a substitute. The effort to +migrate parts of the ecosystem to more secure settings (such as not relying on +DLL hijacking) has largely occurred, but some packages remain that still only +work with less secure configurations, and moving all users back to these +configurations would ensure that users of these packages would not face the +issues they face today. + +However, the majority of users of the Store packages appear to have no +complaints. Anecdotally, they are often fully satisfied by the Store install, +and particularly appreciate the ease and reliability of installation. (And on a +personal note, this author has been using Store packages exclusively since +Python 3.8 with no blocking issues.) + +The greatest number of issues have been caused by misconfigured ``PATH`` +variables and the default ``python.exe`` redirector installed by Microsoft. In +other words, entirely unrelated to our own package (though sometimes related to +unresolvable issues in our other installer). For the sake of the high number of +successful installs through this path, we consider the burden of diagnosing and +assisting impacted users to be worthwhile, and consider the idea to simply drop +the Store package rejected. + +That said, when PyManager is published to the Store, we would plan to delist all +existing runtimes on the Store to ensure users find the manager. This only +impacts new installs, and anyone who has previously installed a particular +version (even on another machine, if they were logged in) will be able to +continue to use and install those versions. + + +Rely on WinGet or equivalents +----------------------------- + +WinGet, Chocolatey, and other similar tools are not installers in the sense that +we require. They use their own repository of metadata to download, validate, and +run installers. Without our own installer, they have nothing to run, and so +cannot be used. + +It is possible that their metadata will not support installing PyManager and +then running it to install a particular runtime. If this is the case, they may +need to investigate using our binary packages directly. + +Currently, none of these install tools are officially supported by CPython, and +so we have no obligation to make them work. + + +Make every version a Windows Store package +------------------------------------------ + +It is possible to release each version to the Windows Store as we currently do, +but make them unlisted and rely on an installer (potentially PyManager, WinGet, +or another tool that can install Store packages). This would avoid the risk of +overwhelming the user, while greatly simplifying our own reponsibilities for +package management. + +This approach would leave a significant burden on whichever contributor has +access to the Store publishing interface, as updating packages is a manual +operation. Additionally, it would leave every Python runtime with the technical +limitations outlined earlier. As such, this idea is rejected. + +Making every version a MSIX package rather than a ZIP, even though this avoids +the Store publishing interface, would still impose technical limitations on +users. It is also rejected. + + +Just publish the plain ZIP file +------------------------------- + +Publishing the plain ZIP file is part of the plan, however, it will not be +visibly listed (for example, on the python.org download pages, though they will +be visible in the FTP view). An alternative would be to publish and list these +packages, and expect users to download and manually extract and configure them. + +Given the workflows we see, we believe that most users do not want to configure +a Python install at all. Not only do they not want to choose the install +location, they do not want to choose a version, or even have to search for a +download provider or instructions. However, they do want to be able to find an +install later, launch, update or remove it, or list all known installs. + +It is also worth recognising that there will be more ZIP files than are +currently listed on the Download pages, and so the list of files will become +longer. Choosing the correct download is already challenging for users (those +who bypass the primary "Download" button and view the list of all available +versions and then files), and we have no desire to make it more challenging. + +The index protocol and download list will be available for tools that wish to +use it, or for users who are willing to navigate JSON in order to find the URL. +The ``--target`` option on the install command also provides a trivial download +and extract operation, allowing users to have the same experience as a ZIP file. +And the ``--download`` option can give users the ZIP file still zipped. + + +Only publish PyManager to one place +----------------------------------- + +Whether the Windows Store or python.org, it would be viable to publish to only +one location. + +However, users strongly expect to be able to download *something* from +python.org. If we were to remove any option at all, we would inevitably hurt our +users. Without an MSIX available on python.org, users have no way to transfer +the package to another machine, or to fully script the initial install of the +manager. + +Many users rely on the Windows Store app to install packages, and the built-in +redirector in Windows can only open to a Store page. As such, removing the Store +app is equivalent to denying hundreds of thousands of installs each month. + +The two builds are practically identical. The only difference between the MSIX +we provide to the Store and the one that goes to python.org is package signing: +we sign the python.org package ourselves, while the Store package is signed as +part of the publish process. Otherwise, there is no additional cost to producing +and publishing both packages. + + +Inline Script Metadata +---------------------- + +PEP 723 introduced inline script metadata, a structured comment intended for +third-party tools to interpret and then launch a Python script in the correct +environment. An example taken from that PEP: + +.. code:: python + + # /// script + # requires-python = ">=3.11" + # dependencies = [ + # "requests<3", + # "rich", + # ] + # /// + +PyManager has no integrated support for installing dependencies, and does not +propose adding any. As a result, we could not fully implement handling of this +metadata, and as we consider partial handling to be worse than nothing, we +choose not to implement any. + +It is possible for a user to specify the constraint directly as an option, for +example, ``py -V:>=3.11 my-script.py`` to get the selection behaviour. + +We could also detect the metadata and warn if the selected runtime does not +match its requirement, but this is not part of the initial proposal. + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. diff --git a/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/provenance.json b/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/provenance.json new file mode 100644 index 0000000..077fa91 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/peps_windows_installer_supersession/provenance.json @@ -0,0 +1,54 @@ +{ + "source_repo": "python/peps", + "pinned_commit": "f866e77409305866038471574f075cd8d83eee9e", + "peps": [ + { + "id": "PEP-0397", + "number": 397, + "file": "corpus/PEP-0397.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0397.rst", + "source_sha256": "6f2abf49ead9e2d50c0d5952855bca1563a4fa5790aa6cb60098df42d72b4368", + "status": "Final", + "title": "Python launcher for Windows", + "replaces": [], + "superseded_by": [] + }, + { + "id": "PEP-0486", + "number": 486, + "file": "corpus/PEP-0486.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0486.rst", + "source_sha256": "b00325274fa9837fc4e77d964c4b349dc66895ba776237de711574d797152a3b", + "status": "Final", + "title": "Make the Python Launcher aware of virtual environments", + "replaces": [], + "superseded_by": [] + }, + { + "id": "PEP-0773", + "number": 773, + "file": "corpus/PEP-0773.md", + "url": "https://raw.githubusercontent.com/python/peps/f866e77409305866038471574f075cd8d83eee9e/peps/pep-0773.rst", + "source_sha256": "313413d0d5c94d1ba1b5207750ca3aca7b255ecac52a9a2eec85aa0809aa6432", + "status": "Final", + "title": "A Python Installation Manager for Windows", + "replaces": [ + "PEP-0397", + "PEP-0486" + ], + "superseded_by": [] + } + ], + "supersedes_edges": [ + { + "source": "PEP-0773", + "type": "supersedes", + "target": "PEP-0397" + }, + { + "source": "PEP-0773", + "type": "supersedes", + "target": "PEP-0486" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/rfc_ipv6_supersession/corpus/RFC-2460.md b/decisiongrounding/scenarios_candidates/rfc_ipv6_supersession/corpus/RFC-2460.md new file mode 100644 index 0000000..02a1d45 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/rfc_ipv6_supersession/corpus/RFC-2460.md @@ -0,0 +1,2223 @@ +--- +schema_version: 1 +id: RFC-2460 +type: decision +tags: [rfc] +--- + +# RFC-2460 — December 1998 + +## Status + +Superseded + +## Context + +Public decision ingested verbatim from the RFC Editor (RFC 2460); +source https://www.rfc-editor.org/rfc/rfc2460.txt. RFCs are immutable, so the RFC number is the +pin and the upstream sha256 is recorded in provenance.json. Upstream +category: Standards Track. The RAC envelope sections are derived from +the RFC's own header fields; the authoritative content is the unedited RFC +reproduced under Source Text. + +## Decision + +Defined by the verbatim RFC text under Source Text below. + +## Consequences + +As stated by the upstream RFC; see Source Text. + +## Superseded By + +- RFC-8200 + +## Source Text + + + + + + + +Network Working Group S. Deering +Request for Comments: 2460 Cisco +Obsoletes: 1883 R. Hinden +Category: Standards Track Nokia + December 1998 + + + Internet Protocol, Version 6 (IPv6) + Specification + +Status of this Memo + + This document specifies an Internet standards track protocol for the + Internet community, and requests discussion and suggestions for + improvements. Please refer to the current edition of the "Internet + Official Protocol Standards" (STD 1) for the standardization state + and status of this protocol. Distribution of this memo is unlimited. + +Copyright Notice + + Copyright (C) The Internet Society (1998). All Rights Reserved. + +Abstract + + This document specifies version 6 of the Internet Protocol (IPv6), + also sometimes referred to as IP Next Generation or IPng. + +Table of Contents + + 1. Introduction..................................................2 + 2. Terminology...................................................3 + 3. IPv6 Header Format............................................4 + 4. IPv6 Extension Headers........................................6 + 4.1 Extension Header Order...................................7 + 4.2 Options..................................................9 + 4.3 Hop-by-Hop Options Header...............................11 + 4.4 Routing Header..........................................12 + 4.5 Fragment Header.........................................18 + 4.6 Destination Options Header..............................23 + 4.7 No Next Header..........................................24 + 5. Packet Size Issues...........................................24 + 6. Flow Labels..................................................25 + 7. Traffic Classes..............................................25 + 8. Upper-Layer Protocol Issues..................................27 + 8.1 Upper-Layer Checksums...................................27 + 8.2 Maximum Packet Lifetime.................................28 + 8.3 Maximum Upper-Layer Payload Size........................28 + 8.4 Responding to Packets Carrying Routing Headers..........29 + + + +Deering & Hinden Standards Track [Page 1] + +RFC 2460 IPv6 Specification December 1998 + + + Appendix A. Semantics and Usage of the Flow Label Field.........30 + Appendix B. Formatting Guidelines for Options...................32 + Security Considerations.........................................35 + Acknowledgments.................................................35 + Authors' Addresses..............................................35 + References......................................................35 + Changes Since RFC-1883..........................................36 + Full Copyright Statement........................................39 + +1. Introduction + + IP version 6 (IPv6) is a new version of the Internet Protocol, + designed as the successor to IP version 4 (IPv4) [RFC-791]. The + changes from IPv4 to IPv6 fall primarily into the following + categories: + + o Expanded Addressing Capabilities + + IPv6 increases the IP address size from 32 bits to 128 bits, to + support more levels of addressing hierarchy, a much greater + number of addressable nodes, and simpler auto-configuration of + addresses. The scalability of multicast routing is improved by + adding a "scope" field to multicast addresses. And a new type + of address called an "anycast address" is defined, used to send + a packet to any one of a group of nodes. + + o Header Format Simplification + + Some IPv4 header fields have been dropped or made optional, to + reduce the common-case processing cost of packet handling and + to limit the bandwidth cost of the IPv6 header. + + o Improved Support for Extensions and Options + + Changes in the way IP header options are encoded allows for + more efficient forwarding, less stringent limits on the length + of options, and greater flexibility for introducing new options + in the future. + + o Flow Labeling Capability + + A new capability is added to enable the labeling of packets + belonging to particular traffic "flows" for which the sender + requests special handling, such as non-default quality of + service or "real-time" service. + + + + + + +Deering & Hinden Standards Track [Page 2] + +RFC 2460 IPv6 Specification December 1998 + + + o Authentication and Privacy Capabilities + + Extensions to support authentication, data integrity, and + (optional) data confidentiality are specified for IPv6. + + This document specifies the basic IPv6 header and the initially- + defined IPv6 extension headers and options. It also discusses packet + size issues, the semantics of flow labels and traffic classes, and + the effects of IPv6 on upper-layer protocols. The format and + semantics of IPv6 addresses are specified separately in [ADDRARCH]. + The IPv6 version of ICMP, which all IPv6 implementations are required + to include, is specified in [ICMPv6]. + +2. Terminology + + node - a device that implements IPv6. + + router - a node that forwards IPv6 packets not explicitly + addressed to itself. [See Note below]. + + host - any node that is not a router. [See Note below]. + + upper layer - a protocol layer immediately above IPv6. Examples are + transport protocols such as TCP and UDP, control + protocols such as ICMP, routing protocols such as OSPF, + and internet or lower-layer protocols being "tunneled" + over (i.e., encapsulated in) IPv6 such as IPX, + AppleTalk, or IPv6 itself. + + link - a communication facility or medium over which nodes can + communicate at the link layer, i.e., the layer + immediately below IPv6. Examples are Ethernets (simple + or bridged); PPP links; X.25, Frame Relay, or ATM + networks; and internet (or higher) layer "tunnels", + such as tunnels over IPv4 or IPv6 itself. + + neighbors - nodes attached to the same link. + + interface - a node's attachment to a link. + + address - an IPv6-layer identifier for an interface or a set of + interfaces. + + packet - an IPv6 header plus payload. + + link MTU - the maximum transmission unit, i.e., maximum packet + size in octets, that can be conveyed over a link. + + + + +Deering & Hinden Standards Track [Page 3] + +RFC 2460 IPv6 Specification December 1998 + + + path MTU - the minimum link MTU of all the links in a path between + a source node and a destination node. + + Note: it is possible, though unusual, for a device with multiple + interfaces to be configured to forward non-self-destined packets + arriving from some set (fewer than all) of its interfaces, and to + discard non-self-destined packets arriving from its other interfaces. + Such a device must obey the protocol requirements for routers when + receiving packets from, and interacting with neighbors over, the + former (forwarding) interfaces. It must obey the protocol + requirements for hosts when receiving packets from, and interacting + with neighbors over, the latter (non-forwarding) interfaces. + +3. IPv6 Header Format + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Version| Traffic Class | Flow Label | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Payload Length | Next Header | Hop Limit | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Source Address + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Destination Address + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Version 4-bit Internet Protocol version number = 6. + + Traffic Class 8-bit traffic class field. See section 7. + + Flow Label 20-bit flow label. See section 6. + + Payload Length 16-bit unsigned integer. Length of the IPv6 + payload, i.e., the rest of the packet following + this IPv6 header, in octets. (Note that any + + + + + +Deering & Hinden Standards Track [Page 4] + +RFC 2460 IPv6 Specification December 1998 + + + extension headers [section 4] present are + considered part of the payload, i.e., included + in the length count.) + + Next Header 8-bit selector. Identifies the type of header + immediately following the IPv6 header. Uses the + same values as the IPv4 Protocol field [RFC-1700 + et seq.]. + + Hop Limit 8-bit unsigned integer. Decremented by 1 by + each node that forwards the packet. The packet + is discarded if Hop Limit is decremented to + zero. + + Source Address 128-bit address of the originator of the packet. + See [ADDRARCH]. + + Destination Address 128-bit address of the intended recipient of the + packet (possibly not the ultimate recipient, if + a Routing header is present). See [ADDRARCH] + and section 4.4. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 5] + +RFC 2460 IPv6 Specification December 1998 + + +4. IPv6 Extension Headers + + In IPv6, optional internet-layer information is encoded in separate + headers that may be placed between the IPv6 header and the upper- + layer header in a packet. There are a small number of such extension + headers, each identified by a distinct Next Header value. As + illustrated in these examples, an IPv6 packet may carry zero, one, or + more extension headers, each identified by the Next Header field of + the preceding header: + + +---------------+------------------------ + | IPv6 header | TCP header + data + | | + | Next Header = | + | TCP | + +---------------+------------------------ + + + +---------------+----------------+------------------------ + | IPv6 header | Routing header | TCP header + data + | | | + | Next Header = | Next Header = | + | Routing | TCP | + +---------------+----------------+------------------------ + + + +---------------+----------------+-----------------+----------------- + | IPv6 header | Routing header | Fragment header | fragment of TCP + | | | | header + data + | Next Header = | Next Header = | Next Header = | + | Routing | Fragment | TCP | + +---------------+----------------+-----------------+----------------- + + With one exception, extension headers are not examined or processed + by any node along a packet's delivery path, until the packet reaches + the node (or each of the set of nodes, in the case of multicast) + identified in the Destination Address field of the IPv6 header. + There, normal demultiplexing on the Next Header field of the IPv6 + header invokes the module to process the first extension header, or + the upper-layer header if no extension header is present. The + contents and semantics of each extension header determine whether or + not to proceed to the next header. Therefore, extension headers must + be processed strictly in the order they appear in the packet; a + receiver must not, for example, scan through a packet looking for a + particular kind of extension header and process that header prior to + processing all preceding ones. + + + + + +Deering & Hinden Standards Track [Page 6] + +RFC 2460 IPv6 Specification December 1998 + + + The exception referred to in the preceding paragraph is the Hop-by- + Hop Options header, which carries information that must be examined + and processed by every node along a packet's delivery path, including + the source and destination nodes. The Hop-by-Hop Options header, + when present, must immediately follow the IPv6 header. Its presence + is indicated by the value zero in the Next Header field of the IPv6 + header. + + If, as a result of processing a header, a node is required to proceed + to the next header but the Next Header value in the current header is + unrecognized by the node, it should discard the packet and send an + ICMP Parameter Problem message to the source of the packet, with an + ICMP Code value of 1 ("unrecognized Next Header type encountered") + and the ICMP Pointer field containing the offset of the unrecognized + value within the original packet. The same action should be taken if + a node encounters a Next Header value of zero in any header other + than an IPv6 header. + + Each extension header is an integer multiple of 8 octets long, in + order to retain 8-octet alignment for subsequent headers. Multi- + octet fields within each extension header are aligned on their + natural boundaries, i.e., fields of width n octets are placed at an + integer multiple of n octets from the start of the header, for n = 1, + 2, 4, or 8. + + A full implementation of IPv6 includes implementation of the + following extension headers: + + Hop-by-Hop Options + Routing (Type 0) + Fragment + Destination Options + Authentication + Encapsulating Security Payload + + The first four are specified in this document; the last two are + specified in [RFC-2402] and [RFC-2406], respectively. + +4.1 Extension Header Order + + When more than one extension header is used in the same packet, it is + recommended that those headers appear in the following order: + + IPv6 header + Hop-by-Hop Options header + Destination Options header (note 1) + Routing header + Fragment header + + + +Deering & Hinden Standards Track [Page 7] + +RFC 2460 IPv6 Specification December 1998 + + + Authentication header (note 2) + Encapsulating Security Payload header (note 2) + Destination Options header (note 3) + upper-layer header + + note 1: for options to be processed by the first destination + that appears in the IPv6 Destination Address field + plus subsequent destinations listed in the Routing + header. + + note 2: additional recommendations regarding the relative + order of the Authentication and Encapsulating + Security Payload headers are given in [RFC-2406]. + + note 3: for options to be processed only by the final + destination of the packet. + + Each extension header should occur at most once, except for the + Destination Options header which should occur at most twice (once + before a Routing header and once before the upper-layer header). + + If the upper-layer header is another IPv6 header (in the case of IPv6 + being tunneled over or encapsulated in IPv6), it may be followed by + its own extension headers, which are separately subject to the same + ordering recommendations. + + If and when other extension headers are defined, their ordering + constraints relative to the above listed headers must be specified. + + IPv6 nodes must accept and attempt to process extension headers in + any order and occurring any number of times in the same packet, + except for the Hop-by-Hop Options header which is restricted to + appear immediately after an IPv6 header only. Nonetheless, it is + strongly advised that sources of IPv6 packets adhere to the above + recommended order until and unless subsequent specifications revise + that recommendation. + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 8] + +RFC 2460 IPv6 Specification December 1998 + + +4.2 Options + + Two of the currently-defined extension headers -- the Hop-by-Hop + Options header and the Destination Options header -- carry a variable + number of type-length-value (TLV) encoded "options", of the following + format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- - - - - - - - - + | Option Type | Opt Data Len | Option Data + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- - - - - - - - - + + Option Type 8-bit identifier of the type of option. + + Opt Data Len 8-bit unsigned integer. Length of the Option + Data field of this option, in octets. + + Option Data Variable-length field. Option-Type-specific + data. + + The sequence of options within a header must be processed strictly in + the order they appear in the header; a receiver must not, for + example, scan through the header looking for a particular kind of + option and process that option prior to processing all preceding + ones. + + The Option Type identifiers are internally encoded such that their + highest-order two bits specify the action that must be taken if the + processing IPv6 node does not recognize the Option Type: + + 00 - skip over this option and continue processing the header. + + 01 - discard the packet. + + 10 - discard the packet and, regardless of whether or not the + packet's Destination Address was a multicast address, send an + ICMP Parameter Problem, Code 2, message to the packet's + Source Address, pointing to the unrecognized Option Type. + + 11 - discard the packet and, only if the packet's Destination + Address was not a multicast address, send an ICMP Parameter + Problem, Code 2, message to the packet's Source Address, + pointing to the unrecognized Option Type. + + The third-highest-order bit of the Option Type specifies whether or + not the Option Data of that option can change en-route to the + packet's final destination. When an Authentication header is present + + + + + +Deering & Hinden Standards Track [Page 9] + +RFC 2460 IPv6 Specification December 1998 + + + in the packet, for any option whose data may change en-route, its + entire Option Data field must be treated as zero-valued octets when + computing or verifying the packet's authenticating value. + + 0 - Option Data does not change en-route + + 1 - Option Data may change en-route + + The three high-order bits described above are to be treated as part + of the Option Type, not independent of the Option Type. That is, a + particular option is identified by a full 8-bit Option Type, not just + the low-order 5 bits of an Option Type. + + The same Option Type numbering space is used for both the Hop-by-Hop + Options header and the Destination Options header. However, the + specification of a particular option may restrict its use to only one + of those two headers. + + Individual options may have specific alignment requirements, to + ensure that multi-octet values within Option Data fields fall on + natural boundaries. The alignment requirement of an option is + specified using the notation xn+y, meaning the Option Type must + appear at an integer multiple of x octets from the start of the + header, plus y octets. For example: + + 2n means any 2-octet offset from the start of the header. + 8n+2 means any 8-octet offset from the start of the header, + plus 2 octets. + + There are two padding options which are used when necessary to align + subsequent options and to pad out the containing header to a multiple + of 8 octets in length. These padding options must be recognized by + all IPv6 implementations: + + Pad1 option (alignment requirement: none) + + +-+-+-+-+-+-+-+-+ + | 0 | + +-+-+-+-+-+-+-+-+ + + NOTE! the format of the Pad1 option is a special case -- it does + not have length and value fields. + + The Pad1 option is used to insert one octet of padding into the + Options area of a header. If more than one octet of padding is + required, the PadN option, described next, should be used, rather + than multiple Pad1 options. + + + + +Deering & Hinden Standards Track [Page 10] + +RFC 2460 IPv6 Specification December 1998 + + + PadN option (alignment requirement: none) + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- - - - - - - - - + | 1 | Opt Data Len | Option Data + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- - - - - - - - - + + The PadN option is used to insert two or more octets of padding + into the Options area of a header. For N octets of padding, the + Opt Data Len field contains the value N-2, and the Option Data + consists of N-2 zero-valued octets. + + Appendix B contains formatting guidelines for designing new options. + +4.3 Hop-by-Hop Options Header + + The Hop-by-Hop Options header is used to carry optional information + that must be examined by every node along a packet's delivery path. + The Hop-by-Hop Options header is identified by a Next Header value of + 0 in the IPv6 header, and has the following format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + | | + . . + . Options . + . . + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Next Header 8-bit selector. Identifies the type of header + immediately following the Hop-by-Hop Options + header. Uses the same values as the IPv4 + Protocol field [RFC-1700 et seq.]. + + Hdr Ext Len 8-bit unsigned integer. Length of the Hop-by- + Hop Options header in 8-octet units, not + including the first 8 octets. + + Options Variable-length field, of length such that the + complete Hop-by-Hop Options header is an integer + multiple of 8 octets long. Contains one or more + TLV-encoded options, as described in section + 4.2. + + The only hop-by-hop options defined in this document are the Pad1 and + PadN options specified in section 4.2. + + + + +Deering & Hinden Standards Track [Page 11] + +RFC 2460 IPv6 Specification December 1998 + + +4.4 Routing Header + + The Routing header is used by an IPv6 source to list one or more + intermediate nodes to be "visited" on the way to a packet's + destination. This function is very similar to IPv4's Loose Source + and Record Route option. The Routing header is identified by a Next + Header value of 43 in the immediately preceding header, and has the + following format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len | Routing Type | Segments Left | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + . . + . type-specific data . + . . + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Next Header 8-bit selector. Identifies the type of header + immediately following the Routing header. Uses + the same values as the IPv4 Protocol field + [RFC-1700 et seq.]. + + Hdr Ext Len 8-bit unsigned integer. Length of the Routing + header in 8-octet units, not including the first + 8 octets. + + Routing Type 8-bit identifier of a particular Routing header + variant. + + Segments Left 8-bit unsigned integer. Number of route + segments remaining, i.e., number of explicitly + listed intermediate nodes still to be visited + before reaching the final destination. + + type-specific data Variable-length field, of format determined by + the Routing Type, and of length such that the + complete Routing header is an integer multiple + of 8 octets long. + + If, while processing a received packet, a node encounters a Routing + header with an unrecognized Routing Type value, the required behavior + of the node depends on the value of the Segments Left field, as + follows: + + + + + + +Deering & Hinden Standards Track [Page 12] + +RFC 2460 IPv6 Specification December 1998 + + + If Segments Left is zero, the node must ignore the Routing header + and proceed to process the next header in the packet, whose type + is identified by the Next Header field in the Routing header. + + If Segments Left is non-zero, the node must discard the packet and + send an ICMP Parameter Problem, Code 0, message to the packet's + Source Address, pointing to the unrecognized Routing Type. + + If, after processing a Routing header of a received packet, an + intermediate node determines that the packet is to be forwarded onto + a link whose link MTU is less than the size of the packet, the node + must discard the packet and send an ICMP Packet Too Big message to + the packet's Source Address. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 13] + +RFC 2460 IPv6 Specification December 1998 + + + The Type 0 Routing header has the following format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len | Routing Type=0| Segments Left | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Reserved | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Address[1] + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Address[2] + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + . . . + . . . + . . . + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Address[n] + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Next Header 8-bit selector. Identifies the type of header + immediately following the Routing header. Uses + the same values as the IPv4 Protocol field + [RFC-1700 et seq.]. + + Hdr Ext Len 8-bit unsigned integer. Length of the Routing + header in 8-octet units, not including the first + 8 octets. For the Type 0 Routing header, Hdr + Ext Len is equal to two times the number of + addresses in the header. + + Routing Type 0. + + + +Deering & Hinden Standards Track [Page 14] + +RFC 2460 IPv6 Specification December 1998 + + + Segments Left 8-bit unsigned integer. Number of route + segments remaining, i.e., number of explicitly + listed intermediate nodes still to be visited + before reaching the final destination. + + Reserved 32-bit reserved field. Initialized to zero for + transmission; ignored on reception. + + Address[1..n] Vector of 128-bit addresses, numbered 1 to n. + + Multicast addresses must not appear in a Routing header of Type 0, or + in the IPv6 Destination Address field of a packet carrying a Routing + header of Type 0. + + A Routing header is not examined or processed until it reaches the + node identified in the Destination Address field of the IPv6 header. + In that node, dispatching on the Next Header field of the immediately + preceding header causes the Routing header module to be invoked, + which, in the case of Routing Type 0, performs the following + algorithm: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 15] + +RFC 2460 IPv6 Specification December 1998 + + + if Segments Left = 0 { + proceed to process the next header in the packet, whose type is + identified by the Next Header field in the Routing header + } + else if Hdr Ext Len is odd { + send an ICMP Parameter Problem, Code 0, message to the Source + Address, pointing to the Hdr Ext Len field, and discard the + packet + } + else { + compute n, the number of addresses in the Routing header, by + dividing Hdr Ext Len by 2 + + if Segments Left is greater than n { + send an ICMP Parameter Problem, Code 0, message to the Source + Address, pointing to the Segments Left field, and discard the + packet + } + else { + decrement Segments Left by 1; + compute i, the index of the next address to be visited in + the address vector, by subtracting Segments Left from n + + if Address [i] or the IPv6 Destination Address is multicast { + discard the packet + } + else { + swap the IPv6 Destination Address and Address[i] + + if the IPv6 Hop Limit is less than or equal to 1 { + send an ICMP Time Exceeded -- Hop Limit Exceeded in + Transit message to the Source Address and discard the + packet + } + else { + decrement the Hop Limit by 1 + + resubmit the packet to the IPv6 module for transmission + to the new destination + } + } + } + } + + + + + + + + +Deering & Hinden Standards Track [Page 16] + +RFC 2460 IPv6 Specification December 1998 + + + As an example of the effects of the above algorithm, consider the + case of a source node S sending a packet to destination node D, using + a Routing header to cause the packet to be routed via intermediate + nodes I1, I2, and I3. The values of the relevant IPv6 header and + Routing header fields on each segment of the delivery path would be + as follows: + + As the packet travels from S to I1: + + Source Address = S Hdr Ext Len = 6 + Destination Address = I1 Segments Left = 3 + Address[1] = I2 + Address[2] = I3 + Address[3] = D + + As the packet travels from I1 to I2: + + Source Address = S Hdr Ext Len = 6 + Destination Address = I2 Segments Left = 2 + Address[1] = I1 + Address[2] = I3 + Address[3] = D + + As the packet travels from I2 to I3: + + Source Address = S Hdr Ext Len = 6 + Destination Address = I3 Segments Left = 1 + Address[1] = I1 + Address[2] = I2 + Address[3] = D + + As the packet travels from I3 to D: + + Source Address = S Hdr Ext Len = 6 + Destination Address = D Segments Left = 0 + Address[1] = I1 + Address[2] = I2 + Address[3] = I3 + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 17] + +RFC 2460 IPv6 Specification December 1998 + + +4.5 Fragment Header + + The Fragment header is used by an IPv6 source to send a packet larger + than would fit in the path MTU to its destination. (Note: unlike + IPv4, fragmentation in IPv6 is performed only by source nodes, not by + routers along a packet's delivery path -- see section 5.) The + Fragment header is identified by a Next Header value of 44 in the + immediately preceding header, and has the following format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Reserved | Fragment Offset |Res|M| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Identification | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Next Header 8-bit selector. Identifies the initial header + type of the Fragmentable Part of the original + packet (defined below). Uses the same values as + the IPv4 Protocol field [RFC-1700 et seq.]. + + Reserved 8-bit reserved field. Initialized to zero for + transmission; ignored on reception. + + Fragment Offset 13-bit unsigned integer. The offset, in 8-octet + units, of the data following this header, + relative to the start of the Fragmentable Part + of the original packet. + + Res 2-bit reserved field. Initialized to zero for + transmission; ignored on reception. + + M flag 1 = more fragments; 0 = last fragment. + + Identification 32 bits. See description below. + + In order to send a packet that is too large to fit in the MTU of the + path to its destination, a source node may divide the packet into + fragments and send each fragment as a separate packet, to be + reassembled at the receiver. + + For every packet that is to be fragmented, the source node generates + an Identification value. The Identification must be different than + that of any other fragmented packet sent recently* with the same + Source Address and Destination Address. If a Routing header is + present, the Destination Address of concern is that of the final + destination. + + + + + +Deering & Hinden Standards Track [Page 18] + +RFC 2460 IPv6 Specification December 1998 + + + * "recently" means within the maximum likely lifetime of a packet, + including transit time from source to destination and time spent + awaiting reassembly with other fragments of the same packet. + However, it is not required that a source node know the maximum + packet lifetime. Rather, it is assumed that the requirement can + be met by maintaining the Identification value as a simple, 32- + bit, "wrap-around" counter, incremented each time a packet must + be fragmented. It is an implementation choice whether to + maintain a single counter for the node or multiple counters, + e.g., one for each of the node's possible source addresses, or + one for each active (source address, destination address) + combination. + + The initial, large, unfragmented packet is referred to as the + "original packet", and it is considered to consist of two parts, as + illustrated: + + original packet: + + +------------------+----------------------//-----------------------+ + | Unfragmentable | Fragmentable | + | Part | Part | + +------------------+----------------------//-----------------------+ + + The Unfragmentable Part consists of the IPv6 header plus any + extension headers that must be processed by nodes en route to the + destination, that is, all headers up to and including the Routing + header if present, else the Hop-by-Hop Options header if present, + else no extension headers. + + The Fragmentable Part consists of the rest of the packet, that is, + any extension headers that need be processed only by the final + destination node(s), plus the upper-layer header and data. + + The Fragmentable Part of the original packet is divided into + fragments, each, except possibly the last ("rightmost") one, being an + integer multiple of 8 octets long. The fragments are transmitted in + separate "fragment packets" as illustrated: + + original packet: + + +------------------+--------------+--------------+--//--+----------+ + | Unfragmentable | first | second | | last | + | Part | fragment | fragment | .... | fragment | + +------------------+--------------+--------------+--//--+----------+ + + + + + + +Deering & Hinden Standards Track [Page 19] + +RFC 2460 IPv6 Specification December 1998 + + + fragment packets: + + +------------------+--------+--------------+ + | Unfragmentable |Fragment| first | + | Part | Header | fragment | + +------------------+--------+--------------+ + + +------------------+--------+--------------+ + | Unfragmentable |Fragment| second | + | Part | Header | fragment | + +------------------+--------+--------------+ + o + o + o + +------------------+--------+----------+ + | Unfragmentable |Fragment| last | + | Part | Header | fragment | + +------------------+--------+----------+ + + Each fragment packet is composed of: + + (1) The Unfragmentable Part of the original packet, with the + Payload Length of the original IPv6 header changed to contain + the length of this fragment packet only (excluding the length + of the IPv6 header itself), and the Next Header field of the + last header of the Unfragmentable Part changed to 44. + + (2) A Fragment header containing: + + The Next Header value that identifies the first header of + the Fragmentable Part of the original packet. + + A Fragment Offset containing the offset of the fragment, + in 8-octet units, relative to the start of the + Fragmentable Part of the original packet. The Fragment + Offset of the first ("leftmost") fragment is 0. + + An M flag value of 0 if the fragment is the last + ("rightmost") one, else an M flag value of 1. + + The Identification value generated for the original + packet. + + (3) The fragment itself. + + The lengths of the fragments must be chosen such that the resulting + fragment packets fit within the MTU of the path to the packets' + destination(s). + + + +Deering & Hinden Standards Track [Page 20] + +RFC 2460 IPv6 Specification December 1998 + + + At the destination, fragment packets are reassembled into their + original, unfragmented form, as illustrated: + + reassembled original packet: + + +------------------+----------------------//------------------------+ + | Unfragmentable | Fragmentable | + | Part | Part | + +------------------+----------------------//------------------------+ + + The following rules govern reassembly: + + An original packet is reassembled only from fragment packets that + have the same Source Address, Destination Address, and Fragment + Identification. + + The Unfragmentable Part of the reassembled packet consists of all + headers up to, but not including, the Fragment header of the first + fragment packet (that is, the packet whose Fragment Offset is + zero), with the following two changes: + + The Next Header field of the last header of the Unfragmentable + Part is obtained from the Next Header field of the first + fragment's Fragment header. + + The Payload Length of the reassembled packet is computed from + the length of the Unfragmentable Part and the length and offset + of the last fragment. For example, a formula for computing the + Payload Length of the reassembled original packet is: + + PL.orig = PL.first - FL.first - 8 + (8 * FO.last) + FL.last + + where + PL.orig = Payload Length field of reassembled packet. + PL.first = Payload Length field of first fragment packet. + FL.first = length of fragment following Fragment header of + first fragment packet. + FO.last = Fragment Offset field of Fragment header of + last fragment packet. + FL.last = length of fragment following Fragment header of + last fragment packet. + + The Fragmentable Part of the reassembled packet is constructed + from the fragments following the Fragment headers in each of the + fragment packets. The length of each fragment is computed by + subtracting from the packet's Payload Length the length of the + + + + + +Deering & Hinden Standards Track [Page 21] + +RFC 2460 IPv6 Specification December 1998 + + + headers between the IPv6 header and fragment itself; its relative + position in Fragmentable Part is computed from its Fragment Offset + value. + + The Fragment header is not present in the final, reassembled + packet. + + The following error conditions may arise when reassembling fragmented + packets: + + If insufficient fragments are received to complete reassembly of a + packet within 60 seconds of the reception of the first-arriving + fragment of that packet, reassembly of that packet must be + abandoned and all the fragments that have been received for that + packet must be discarded. If the first fragment (i.e., the one + with a Fragment Offset of zero) has been received, an ICMP Time + Exceeded -- Fragment Reassembly Time Exceeded message should be + sent to the source of that fragment. + + If the length of a fragment, as derived from the fragment packet's + Payload Length field, is not a multiple of 8 octets and the M flag + of that fragment is 1, then that fragment must be discarded and an + ICMP Parameter Problem, Code 0, message should be sent to the + source of the fragment, pointing to the Payload Length field of + the fragment packet. + + If the length and offset of a fragment are such that the Payload + Length of the packet reassembled from that fragment would exceed + 65,535 octets, then that fragment must be discarded and an ICMP + Parameter Problem, Code 0, message should be sent to the source of + the fragment, pointing to the Fragment Offset field of the + fragment packet. + + The following conditions are not expected to occur, but are not + considered errors if they do: + + The number and content of the headers preceding the Fragment + header of different fragments of the same original packet may + differ. Whatever headers are present, preceding the Fragment + header in each fragment packet, are processed when the packets + arrive, prior to queueing the fragments for reassembly. Only + those headers in the Offset zero fragment packet are retained in + the reassembled packet. + + The Next Header values in the Fragment headers of different + fragments of the same original packet may differ. Only the value + from the Offset zero fragment packet is used for reassembly. + + + + +Deering & Hinden Standards Track [Page 22] + +RFC 2460 IPv6 Specification December 1998 + + +4.6 Destination Options Header + + The Destination Options header is used to carry optional information + that need be examined only by a packet's destination node(s). The + Destination Options header is identified by a Next Header value of 60 + in the immediately preceding header, and has the following format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + | | + . . + . Options . + . . + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Next Header 8-bit selector. Identifies the type of header + immediately following the Destination Options + header. Uses the same values as the IPv4 + Protocol field [RFC-1700 et seq.]. + + Hdr Ext Len 8-bit unsigned integer. Length of the + Destination Options header in 8-octet units, not + including the first 8 octets. + + Options Variable-length field, of length such that the + complete Destination Options header is an + integer multiple of 8 octets long. Contains one + or more TLV-encoded options, as described in + section 4.2. + + The only destination options defined in this document are the Pad1 + and PadN options specified in section 4.2. + + Note that there are two possible ways to encode optional destination + information in an IPv6 packet: either as an option in the Destination + Options header, or as a separate extension header. The Fragment + header and the Authentication header are examples of the latter + approach. Which approach can be used depends on what action is + desired of a destination node that does not understand the optional + information: + + o If the desired action is for the destination node to discard + the packet and, only if the packet's Destination Address is not + a multicast address, send an ICMP Unrecognized Type message to + the packet's Source Address, then the information may be + encoded either as a separate header or as an option in the + + + +Deering & Hinden Standards Track [Page 23] + +RFC 2460 IPv6 Specification December 1998 + + + Destination Options header whose Option Type has the value 11 + in its highest-order two bits. The choice may depend on such + factors as which takes fewer octets, or which yields better + alignment or more efficient parsing. + + o If any other action is desired, the information must be encoded + as an option in the Destination Options header whose Option + Type has the value 00, 01, or 10 in its highest-order two bits, + specifying the desired action (see section 4.2). + +4.7 No Next Header + + The value 59 in the Next Header field of an IPv6 header or any + extension header indicates that there is nothing following that + header. If the Payload Length field of the IPv6 header indicates the + presence of octets past the end of a header whose Next Header field + contains 59, those octets must be ignored, and passed on unchanged if + the packet is forwarded. + +5. Packet Size Issues + + IPv6 requires that every link in the internet have an MTU of 1280 + octets or greater. On any link that cannot convey a 1280-octet + packet in one piece, link-specific fragmentation and reassembly must + be provided at a layer below IPv6. + + Links that have a configurable MTU (for example, PPP links [RFC- + 1661]) must be configured to have an MTU of at least 1280 octets; it + is recommended that they be configured with an MTU of 1500 octets or + greater, to accommodate possible encapsulations (i.e., tunneling) + without incurring IPv6-layer fragmentation. + + From each link to which a node is directly attached, the node must be + able to accept packets as large as that link's MTU. + + It is strongly recommended that IPv6 nodes implement Path MTU + Discovery [RFC-1981], in order to discover and take advantage of path + MTUs greater than 1280 octets. However, a minimal IPv6 + implementation (e.g., in a boot ROM) may simply restrict itself to + sending packets no larger than 1280 octets, and omit implementation + of Path MTU Discovery. + + In order to send a packet larger than a path's MTU, a node may use + the IPv6 Fragment header to fragment the packet at the source and + have it reassembled at the destination(s). However, the use of such + fragmentation is discouraged in any application that is able to + adjust its packets to fit the measured path MTU (i.e., down to 1280 + octets). + + + +Deering & Hinden Standards Track [Page 24] + +RFC 2460 IPv6 Specification December 1998 + + + A node must be able to accept a fragmented packet that, after + reassembly, is as large as 1500 octets. A node is permitted to + accept fragmented packets that reassemble to more than 1500 octets. + An upper-layer protocol or application that depends on IPv6 + fragmentation to send packets larger than the MTU of a path should + not send packets larger than 1500 octets unless it has assurance that + the destination is capable of reassembling packets of that larger + size. + + In response to an IPv6 packet that is sent to an IPv4 destination + (i.e., a packet that undergoes translation from IPv6 to IPv4), the + originating IPv6 node may receive an ICMP Packet Too Big message + reporting a Next-Hop MTU less than 1280. In that case, the IPv6 node + is not required to reduce the size of subsequent packets to less than + 1280, but must include a Fragment header in those packets so that the + IPv6-to-IPv4 translating router can obtain a suitable Identification + value to use in resulting IPv4 fragments. Note that this means the + payload may have to be reduced to 1232 octets (1280 minus 40 for the + IPv6 header and 8 for the Fragment header), and smaller still if + additional extension headers are used. + +6. Flow Labels + + The 20-bit Flow Label field in the IPv6 header may be used by a + source to label sequences of packets for which it requests special + handling by the IPv6 routers, such as non-default quality of service + or "real-time" service. This aspect of IPv6 is, at the time of + writing, still experimental and subject to change as the requirements + for flow support in the Internet become clearer. Hosts or routers + that do not support the functions of the Flow Label field are + required to set the field to zero when originating a packet, pass the + field on unchanged when forwarding a packet, and ignore the field + when receiving a packet. + + Appendix A describes the current intended semantics and usage of the + Flow Label field. + +7. Traffic Classes + + The 8-bit Traffic Class field in the IPv6 header is available for use + by originating nodes and/or forwarding routers to identify and + distinguish between different classes or priorities of IPv6 packets. + At the point in time at which this specification is being written, + there are a number of experiments underway in the use of the IPv4 + Type of Service and/or Precedence bits to provide various forms of + "differentiated service" for IP packets, other than through the use + of explicit flow set-up. The Traffic Class field in the IPv6 header + is intended to allow similar functionality to be supported in IPv6. + + + +Deering & Hinden Standards Track [Page 25] + +RFC 2460 IPv6 Specification December 1998 + + + It is hoped that those experiments will eventually lead to agreement + on what sorts of traffic classifications are most useful for IP + packets. Detailed definitions of the syntax and semantics of all or + some of the IPv6 Traffic Class bits, whether experimental or intended + for eventual standardization, are to be provided in separate + documents. + + The following general requirements apply to the Traffic Class field: + + o The service interface to the IPv6 service within a node must + provide a means for an upper-layer protocol to supply the value + of the Traffic Class bits in packets originated by that upper- + layer protocol. The default value must be zero for all 8 bits. + + o Nodes that support a specific (experimental or eventual + standard) use of some or all of the Traffic Class bits are + permitted to change the value of those bits in packets that + they originate, forward, or receive, as required for that + specific use. Nodes should ignore and leave unchanged any bits + of the Traffic Class field for which they do not support a + specific use. + + o An upper-layer protocol must not assume that the value of the + Traffic Class bits in a received packet are the same as the + value sent by the packet's source. + + + + + + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 26] + +RFC 2460 IPv6 Specification December 1998 + + +8. Upper-Layer Protocol Issues + +8.1 Upper-Layer Checksums + + Any transport or other upper-layer protocol that includes the + addresses from the IP header in its checksum computation must be + modified for use over IPv6, to include the 128-bit IPv6 addresses + instead of 32-bit IPv4 addresses. In particular, the following + illustration shows the TCP and UDP "pseudo-header" for IPv6: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Source Address + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Destination Address + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Upper-Layer Packet Length | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | zero | Next Header | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + o If the IPv6 packet contains a Routing header, the Destination + Address used in the pseudo-header is that of the final + destination. At the originating node, that address will be in + the last element of the Routing header; at the recipient(s), + that address will be in the Destination Address field of the + IPv6 header. + + o The Next Header value in the pseudo-header identifies the + upper-layer protocol (e.g., 6 for TCP, or 17 for UDP). It will + differ from the Next Header value in the IPv6 header if there + are extension headers between the IPv6 header and the upper- + layer header. + + o The Upper-Layer Packet Length in the pseudo-header is the + length of the upper-layer header and data (e.g., TCP header + plus TCP data). Some upper-layer protocols carry their own + + + +Deering & Hinden Standards Track [Page 27] + +RFC 2460 IPv6 Specification December 1998 + + + length information (e.g., the Length field in the UDP header); + for such protocols, that is the length used in the pseudo- + header. Other protocols (such as TCP) do not carry their own + length information, in which case the length used in the + pseudo-header is the Payload Length from the IPv6 header, minus + the length of any extension headers present between the IPv6 + header and the upper-layer header. + + o Unlike IPv4, when UDP packets are originated by an IPv6 node, + the UDP checksum is not optional. That is, whenever + originating a UDP packet, an IPv6 node must compute a UDP + checksum over the packet and the pseudo-header, and, if that + computation yields a result of zero, it must be changed to hex + FFFF for placement in the UDP header. IPv6 receivers must + discard UDP packets containing a zero checksum, and should log + the error. + + The IPv6 version of ICMP [ICMPv6] includes the above pseudo-header in + its checksum computation; this is a change from the IPv4 version of + ICMP, which does not include a pseudo-header in its checksum. The + reason for the change is to protect ICMP from misdelivery or + corruption of those fields of the IPv6 header on which it depends, + which, unlike IPv4, are not covered by an internet-layer checksum. + The Next Header field in the pseudo-header for ICMP contains the + value 58, which identifies the IPv6 version of ICMP. + +8.2 Maximum Packet Lifetime + + Unlike IPv4, IPv6 nodes are not required to enforce maximum packet + lifetime. That is the reason the IPv4 "Time to Live" field was + renamed "Hop Limit" in IPv6. In practice, very few, if any, IPv4 + implementations conform to the requirement that they limit packet + lifetime, so this is not a change in practice. Any upper-layer + protocol that relies on the internet layer (whether IPv4 or IPv6) to + limit packet lifetime ought to be upgraded to provide its own + mechanisms for detecting and discarding obsolete packets. + +8.3 Maximum Upper-Layer Payload Size + + When computing the maximum payload size available for upper-layer + data, an upper-layer protocol must take into account the larger size + of the IPv6 header relative to the IPv4 header. For example, in + IPv4, TCP's MSS option is computed as the maximum packet size (a + default value or a value learned through Path MTU Discovery) minus 40 + octets (20 octets for the minimum-length IPv4 header and 20 octets + for the minimum-length TCP header). When using TCP over IPv6, the + MSS must be computed as the maximum packet size minus 60 octets, + + + + +Deering & Hinden Standards Track [Page 28] + +RFC 2460 IPv6 Specification December 1998 + + + because the minimum-length IPv6 header (i.e., an IPv6 header with no + extension headers) is 20 octets longer than a minimum-length IPv4 + header. + +8.4 Responding to Packets Carrying Routing Headers + + When an upper-layer protocol sends one or more packets in response to + a received packet that included a Routing header, the response + packet(s) must not include a Routing header that was automatically + derived by "reversing" the received Routing header UNLESS the + integrity and authenticity of the received Source Address and Routing + header have been verified (e.g., via the use of an Authentication + header in the received packet). In other words, only the following + kinds of packets are permitted in response to a received packet + bearing a Routing header: + + o Response packets that do not carry Routing headers. + + o Response packets that carry Routing headers that were NOT + derived by reversing the Routing header of the received packet + (for example, a Routing header supplied by local + configuration). + + o Response packets that carry Routing headers that were derived + by reversing the Routing header of the received packet IF AND + ONLY IF the integrity and authenticity of the Source Address + and Routing header from the received packet have been verified + by the responder. + + + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 29] + +RFC 2460 IPv6 Specification December 1998 + + +Appendix A. Semantics and Usage of the Flow Label Field + + A flow is a sequence of packets sent from a particular source to a + particular (unicast or multicast) destination for which the source + desires special handling by the intervening routers. The nature of + that special handling might be conveyed to the routers by a control + protocol, such as a resource reservation protocol, or by information + within the flow's packets themselves, e.g., in a hop-by-hop option. + The details of such control protocols or options are beyond the scope + of this document. + + There may be multiple active flows from a source to a destination, as + well as traffic that is not associated with any flow. A flow is + uniquely identified by the combination of a source address and a + non-zero flow label. Packets that do not belong to a flow carry a + flow label of zero. + + A flow label is assigned to a flow by the flow's source node. New + flow labels must be chosen (pseudo-)randomly and uniformly from the + range 1 to FFFFF hex. The purpose of the random allocation is to + make any set of bits within the Flow Label field suitable for use as + a hash key by routers, for looking up the state associated with the + flow. + + All packets belonging to the same flow must be sent with the same + source address, destination address, and flow label. If any of those + packets includes a Hop-by-Hop Options header, then they all must be + originated with the same Hop-by-Hop Options header contents + (excluding the Next Header field of the Hop-by-Hop Options header). + If any of those packets includes a Routing header, then they all must + be originated with the same contents in all extension headers up to + and including the Routing header (excluding the Next Header field in + the Routing header). The routers or destinations are permitted, but + not required, to verify that these conditions are satisfied. If a + violation is detected, it should be reported to the source by an ICMP + Parameter Problem message, Code 0, pointing to the high-order octet + of the Flow Label field (i.e., offset 1 within the IPv6 packet). + + The maximum lifetime of any flow-handling state established along a + flow's path must be specified as part of the description of the + state-establishment mechanism, e.g., the resource reservation + protocol or the flow-setup hop-by-hop option. A source must not re- + use a flow label for a new flow within the maximum lifetime of any + flow-handling state that might have been established for the prior + use of that flow label. + + + + + + +Deering & Hinden Standards Track [Page 30] + +RFC 2460 IPv6 Specification December 1998 + + + When a node stops and restarts (e.g., as a result of a "crash"), it + must be careful not to use a flow label that it might have used for + an earlier flow whose lifetime may not have expired yet. This may be + accomplished by recording flow label usage on stable storage so that + it can be remembered across crashes, or by refraining from using any + flow labels until the maximum lifetime of any possible previously + established flows has expired. If the minimum time for rebooting the + node is known, that time can be deducted from the necessary waiting + period before starting to allocate flow labels. + + There is no requirement that all, or even most, packets belong to + flows, i.e., carry non-zero flow labels. This observation is placed + here to remind protocol designers and implementors not to assume + otherwise. For example, it would be unwise to design a router whose + performance would be adequate only if most packets belonged to flows, + or to design a header compression scheme that only worked on packets + that belonged to flows. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 31] + +RFC 2460 IPv6 Specification December 1998 + + +Appendix B. Formatting Guidelines for Options + + This appendix gives some advice on how to lay out the fields when + designing new options to be used in the Hop-by-Hop Options header or + the Destination Options header, as described in section 4.2. These + guidelines are based on the following assumptions: + + o One desirable feature is that any multi-octet fields within the + Option Data area of an option be aligned on their natural + boundaries, i.e., fields of width n octets should be placed at + an integer multiple of n octets from the start of the Hop-by- + Hop or Destination Options header, for n = 1, 2, 4, or 8. + + o Another desirable feature is that the Hop-by-Hop or Destination + Options header take up as little space as possible, subject to + the requirement that the header be an integer multiple of 8 + octets long. + + o It may be assumed that, when either of the option-bearing + headers are present, they carry a very small number of options, + usually only one. + + These assumptions suggest the following approach to laying out the + fields of an option: order the fields from smallest to largest, with + no interior padding, then derive the alignment requirement for the + entire option based on the alignment requirement of the largest field + (up to a maximum alignment of 8 octets). This approach is + illustrated in the following examples: + + Example 1 + + If an option X required two data fields, one of length 8 octets and + one of length 4 octets, it would be laid out as follows: + + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Option Type=X |Opt Data Len=12| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + 8-octet field + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + + + + + + +Deering & Hinden Standards Track [Page 32] + +RFC 2460 IPv6 Specification December 1998 + + + Its alignment requirement is 8n+2, to ensure that the 8-octet field + starts at a multiple-of-8 offset from the start of the enclosing + header. A complete Hop-by-Hop or Destination Options header + containing this one option would look as follows: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len=1 | Option Type=X |Opt Data Len=12| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + 8-octet field + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Example 2 + + If an option Y required three data fields, one of length 4 octets, + one of length 2 octets, and one of length 1 octet, it would be laid + out as follows: + + +-+-+-+-+-+-+-+-+ + | Option Type=Y | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Opt Data Len=7 | 1-octet field | 2-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Its alignment requirement is 4n+3, to ensure that the 4-octet field + starts at a multiple-of-4 offset from the start of the enclosing + header. A complete Hop-by-Hop or Destination Options header + containing this one option would look as follows: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len=1 | Pad1 Option=0 | Option Type=Y | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Opt Data Len=7 | 1-octet field | 2-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | PadN Option=1 |Opt Data Len=2 | 0 | 0 | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + + + + + + + +Deering & Hinden Standards Track [Page 33] + +RFC 2460 IPv6 Specification December 1998 + + + Example 3 + + A Hop-by-Hop or Destination Options header containing both options X + and Y from Examples 1 and 2 would have one of the two following + formats, depending on which option appeared first: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len=3 | Option Type=X |Opt Data Len=12| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + 8-octet field + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | PadN Option=1 |Opt Data Len=1 | 0 | Option Type=Y | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Opt Data Len=7 | 1-octet field | 2-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | PadN Option=1 |Opt Data Len=2 | 0 | 0 | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len=3 | Pad1 Option=0 | Option Type=Y | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Opt Data Len=7 | 1-octet field | 2-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | PadN Option=1 |Opt Data Len=4 | 0 | 0 | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 0 | 0 | Option Type=X |Opt Data Len=12| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + 8-octet field + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + + + + + + + + +Deering & Hinden Standards Track [Page 34] + +RFC 2460 IPv6 Specification December 1998 + + +Security Considerations + + The security features of IPv6 are described in the Security + Architecture for the Internet Protocol [RFC-2401]. + +Acknowledgments + + The authors gratefully acknowledge the many helpful suggestions of + the members of the IPng working group, the End-to-End Protocols + research group, and the Internet Community At Large. + +Authors' Addresses + + Stephen E. Deering + Cisco Systems, Inc. + 170 West Tasman Drive + San Jose, CA 95134-1706 + USA + + Phone: +1 408 527 8213 + Fax: +1 408 527 8254 + EMail: deering@cisco.com + + + Robert M. Hinden + Nokia + 232 Java Drive + Sunnyvale, CA 94089 + USA + + Phone: +1 408 990-2004 + Fax: +1 408 743-5677 + EMail: hinden@iprg.nokia.com + +References + + [RFC-2401] Kent, S. and R. Atkinson, "Security Architecture for the + Internet Protocol", RFC 2401, November 1998. + + [RFC-2402] Kent, S. and R. Atkinson, "IP Authentication Header", + RFC 2402, November 1998. + + [RFC-2406] Kent, S. and R. Atkinson, "IP Encapsulating Security + Protocol (ESP)", RFC 2406, November 1998. + + [ICMPv6] Conta, A. and S. Deering, "ICMP for the Internet + Protocol Version 6 (IPv6)", RFC 2463, December 1998. + + + + +Deering & Hinden Standards Track [Page 35] + +RFC 2460 IPv6 Specification December 1998 + + + [ADDRARCH] Hinden, R. and S. Deering, "IP Version 6 Addressing + Architecture", RFC 2373, July 1998. + + [RFC-1981] McCann, J., Mogul, J. and S. Deering, "Path MTU + Discovery for IP version 6", RFC 1981, August 1996. + + [RFC-791] Postel, J., "Internet Protocol", STD 5, RFC 791, + September 1981. + + [RFC-1700] Reynolds, J. and J. Postel, "Assigned Numbers", STD 2, + RFC 1700, October 1994. See also: + http://www.iana.org/numbers.html + + [RFC-1661] Simpson, W., "The Point-to-Point Protocol (PPP)", STD + 51, RFC 1661, July 1994. + +CHANGES SINCE RFC-1883 + + This memo has the following changes from RFC-1883. Numbers identify + the Internet-Draft version in which the change was made. + + 02) Removed all references to jumbograms and the Jumbo Payload + option (moved to a separate document). + + 02) Moved most of Flow Label description from section 6 to (new) + Appendix A. + + 02) In Flow Label description, now in Appendix A, corrected maximum + Flow Label value from FFFFFF to FFFFF (i.e., one less "F") due + to reduction of size of Flow Label field from 24 bits to 20 + bits. + + 02) Renumbered (relettered?) the previous Appendix A to be Appendix + B. + + 02) Changed the wording of the Security Considerations section to + avoid dependency loop between this spec and the IPsec specs. + + 02) Updated R. Hinden's email address and company affiliation. + + + -------------------------------------------------------- + + 01) In section 3, changed field name "Class" to "Traffic Class" and + increased its size from 4 to 8 bits. Decreased size of Flow + Label field from 24 to 20 bits to compensate for increase in + Traffic Class field. + + + + +Deering & Hinden Standards Track [Page 36] + +RFC 2460 IPv6 Specification December 1998 + + + 01) In section 4.1, restored the order of the Authentication Header + and the ESP header, which were mistakenly swapped in the 00 + version of this memo. + + 01) In section 4.4, deleted the Strict/Loose Bit Map field and the + strict routing functionality from the Type 0 Routing header, and + removed the restriction on number of addresses that may be + carried in the Type 0 Routing header (was limited to 23 + addresses, because of the size of the strict/loose bit map). + + 01) In section 5, changed the minimum IPv6 MTU from 576 to 1280 + octets, and added a recommendation that links with configurable + MTU (e.g., PPP links) be configured to have an MTU of at least + 1500 octets. + + 01) In section 5, deleted the requirement that a node must not send + fragmented packets that reassemble to more than 1500 octets + without knowledge of the destination reassembly buffer size, and + replaced it with a recommendation that upper-layer protocols or + applications should not do that. + + 01) Replaced reference to the IPv4 Path MTU Discovery spec (RFC- + 1191) with reference to the IPv6 Path MTU Discovery spec (RFC- + 1981), and deleted the Notes at the end of section 5 regarding + Path MTU Discovery, since those details are now covered by RFC- + 1981. + + 01) In section 6, deleted specification of "opportunistic" flow + set-up, and removed all references to the 6-second maximum + lifetime for opportunistically established flow state. + + 01) In section 7, deleted the provisional description of the + internal structure and semantics of the Traffic Class field, and + specified that such descriptions be provided in separate + documents. + + -------------------------------------------------------- + + 00) In section 4, corrected the Code value to indicate "unrecognized + Next Header type encountered" in an ICMP Parameter Problem + message (changed from 2 to 1). + + 00) In the description of the Payload Length field in section 3, and + of the Jumbo Payload Length field in section 4.3, made it + clearer that extension headers are included in the payload + length count. + + + + + +Deering & Hinden Standards Track [Page 37] + +RFC 2460 IPv6 Specification December 1998 + + + 00) In section 4.1, swapped the order of the Authentication header + and the ESP header. (NOTE: this was a mistake, and the change + was undone in version 01.) + + 00) In section 4.2, made it clearer that options are identified by + the full 8-bit Option Type, not by the low-order 5 bits of an + Option Type. Also specified that the same Option Type numbering + space is used for both Hop-by-Hop Options and Destination + Options headers. + + 00) In section 4.4, added a sentence requiring that nodes processing + a Routing header must send an ICMP Packet Too Big message in + response to a packet that is too big to fit in the next hop link + (rather than, say, performing fragmentation). + + 00) Changed the name of the IPv6 Priority field to "Class", and + replaced the previous description of Priority in section 7 with + a description of the Class field. Also, excluded this field + from the set of fields that must remain the same for all packets + in the same flow, as specified in section 6. + + 00) In the pseudo-header in section 8.1, changed the name of the + "Payload Length" field to "Upper-Layer Packet Length". Also + clarified that, in the case of protocols that carry their own + length info (like non-jumbogram UDP), it is the upper-layer- + derived length, not the IP-layer-derived length, that is used in + the pseudo-header. + + 00) Added section 8.4, specifying that upper-layer protocols, when + responding to a received packet that carried a Routing header, + must not include the reverse of the Routing header in the + response packet(s) unless the received Routing header was + authenticated. + + 00) Fixed some typos and grammatical errors. + + 00) Authors' contact info updated. + + -------------------------------------------------------- + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 38] + +RFC 2460 IPv6 Specification December 1998 + + +Full Copyright Statement + + Copyright (C) The Internet Society (1998). All Rights Reserved. + + This document and translations of it may be copied and furnished to + others, and derivative works that comment on or otherwise explain it + or assist in its implementation may be prepared, copied, published + and distributed, in whole or in part, without restriction of any + kind, provided that the above copyright notice and this paragraph are + included on all such copies and derivative works. However, this + document itself may not be modified in any way, such as by removing + the copyright notice or references to the Internet Society or other + Internet organizations, except as needed for the purpose of + developing Internet standards in which case the procedures for + copyrights defined in the Internet Standards process must be + followed, or as required to translate it into languages other than + English. + + The limited permissions granted above are perpetual and will not be + revoked by the Internet Society or its successors or assigns. + + This document and the information contained herein is provided on an + "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING + TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING + BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION + HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF + MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + + + + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 39] + diff --git a/decisiongrounding/scenarios_candidates/rfc_ipv6_supersession/corpus/RFC-8200.md b/decisiongrounding/scenarios_candidates/rfc_ipv6_supersession/corpus/RFC-8200.md new file mode 100644 index 0000000..38830d7 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/rfc_ipv6_supersession/corpus/RFC-8200.md @@ -0,0 +1,2391 @@ +--- +schema_version: 1 +id: RFC-8200 +type: decision +tags: [rfc] +--- + +# RFC-8200 — Internet Protocol, Version 6 (IPv6) Specification + +## Status + +Final + +## Context + +Public decision ingested verbatim from the RFC Editor (RFC 8200); +source https://www.rfc-editor.org/rfc/rfc8200.txt. RFCs are immutable, so the RFC number is the +pin and the upstream sha256 is recorded in provenance.json. Upstream +category: Standards Track. The RAC envelope sections are derived from +the RFC's own header fields; the authoritative content is the unedited RFC +reproduced under Source Text. + +## Decision + +Defined by the verbatim RFC text under Source Text below. + +## Consequences + +As stated by the upstream RFC; see Source Text. + +## Supersedes + +- RFC-2460 + +## Source Text + + + + + + + +Internet Engineering Task Force (IETF) S. Deering +Request for Comments: 8200 Retired +STD: 86 R. Hinden +Obsoletes: 2460 Check Point Software +Category: Standards Track July 2017 +ISSN: 2070-1721 + + + Internet Protocol, Version 6 (IPv6) Specification + +Abstract + + This document specifies version 6 of the Internet Protocol (IPv6). + It obsoletes RFC 2460. + +Status of This Memo + + This is an Internet Standards Track document. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Further information on + Internet Standards is available in Section 2 of RFC 7841. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + http://www.rfc-editor.org/info/rfc8200. + + + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 1] + +RFC 8200 IPv6 Specification July 2017 + + +Copyright Notice + + Copyright (c) 2017 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (http://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Simplified BSD License text as described in Section 4.e of + the Trust Legal Provisions and are provided without warranty as + described in the Simplified BSD License. + + This document may contain material from IETF Documents or IETF + Contributions published or made publicly available before November + 10, 2008. The person(s) controlling the copyright in some of this + material may not have granted the IETF Trust the right to allow + modifications of such material outside the IETF Standards Process. + Without obtaining an adequate license from the person(s) controlling + the copyright in such materials, this document may not be modified + outside the IETF Standards Process, and derivative works of it may + not be created outside the IETF Standards Process, except to format + it for publication as an RFC or to translate it into languages other + than English. + + + + + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 2] + +RFC 8200 IPv6 Specification July 2017 + + +Table of Contents + + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4 + 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 5 + 3. IPv6 Header Format . . . . . . . . . . . . . . . . . . . . . 6 + 4. IPv6 Extension Headers . . . . . . . . . . . . . . . . . . . 7 + 4.1. Extension Header Order . . . . . . . . . . . . . . . . . 10 + 4.2. Options . . . . . . . . . . . . . . . . . . . . . . . . . 11 + 4.3. Hop-by-Hop Options Header . . . . . . . . . . . . . . . . 13 + 4.4. Routing Header . . . . . . . . . . . . . . . . . . . . . 14 + 4.5. Fragment Header . . . . . . . . . . . . . . . . . . . . . 15 + 4.6. Destination Options Header . . . . . . . . . . . . . . . 23 + 4.7. No Next Header . . . . . . . . . . . . . . . . . . . . . 24 + 4.8. Defining New Extension Headers and Options . . . . . . . 24 + 5. Packet Size Issues . . . . . . . . . . . . . . . . . . . . . 25 + 6. Flow Labels . . . . . . . . . . . . . . . . . . . . . . . . . 26 + 7. Traffic Classes . . . . . . . . . . . . . . . . . . . . . . . 26 + 8. Upper-Layer Protocol Issues . . . . . . . . . . . . . . . . . 27 + 8.1. Upper-Layer Checksums . . . . . . . . . . . . . . . . . . 27 + 8.2. Maximum Packet Lifetime . . . . . . . . . . . . . . . . . 28 + 8.3. Maximum Upper-Layer Payload Size . . . . . . . . . . . . 29 + 8.4. Responding to Packets Carrying Routing Headers . . . . . 29 + 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 29 + 10. Security Considerations . . . . . . . . . . . . . . . . . . . 30 + 11. References . . . . . . . . . . . . . . . . . . . . . . . . . 32 + 11.1. Normative References . . . . . . . . . . . . . . . . . . 32 + 11.2. Informative References . . . . . . . . . . . . . . . . . 33 + Appendix A. Formatting Guidelines for Options . . . . . . . . . 36 + Appendix B. Changes Since RFC 2460 . . . . . . . . . . . . . . . 39 + Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . 42 + Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 42 + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 3] + +RFC 8200 IPv6 Specification July 2017 + + +1. Introduction + + IP version 6 (IPv6) is a new version of the Internet Protocol (IP), + designed as the successor to IP version 4 (IPv4) [RFC791]. The + changes from IPv4 to IPv6 fall primarily into the following + categories: + + o Expanded Addressing Capabilities + + IPv6 increases the IP address size from 32 bits to 128 bits, to + support more levels of addressing hierarchy, a much greater + number of addressable nodes, and simpler autoconfiguration of + addresses. The scalability of multicast routing is improved by + adding a "scope" field to multicast addresses. And a new type + of address called an "anycast address" is defined; it is used + to send a packet to any one of a group of nodes. + + o Header Format Simplification + + Some IPv4 header fields have been dropped or made optional, to + reduce the common-case processing cost of packet handling and + to limit the bandwidth cost of the IPv6 header. + + o Improved Support for Extensions and Options + + Changes in the way IP header options are encoded allows for + more efficient forwarding, less stringent limits on the length + of options, and greater flexibility for introducing new options + in the future. + + o Flow Labeling Capability + + A new capability is added to enable the labeling of sequences + of packets that the sender requests to be treated in the + network as a single flow. + + o Authentication and Privacy Capabilities + + Extensions to support authentication, data integrity, and + (optional) data confidentiality are specified for IPv6. + + This document specifies the basic IPv6 header and the initially + defined IPv6 extension headers and options. It also discusses packet + size issues, the semantics of flow labels and traffic classes, and + the effects of IPv6 on upper-layer protocols. The format and + semantics of IPv6 addresses are specified separately in [RFC4291]. + The IPv6 version of ICMP, which all IPv6 implementations are required + to include, is specified in [RFC4443]. + + + +Deering & Hinden Standards Track [Page 4] + +RFC 8200 IPv6 Specification July 2017 + + + The data transmission order for IPv6 is the same as for IPv4 as + defined in Appendix B of [RFC791]. + + Note: As this document obsoletes [RFC2460], any document referenced + in this document that includes pointers to RFC 2460 should be + interpreted as referencing this document. + +2. Terminology + + node a device that implements IPv6. + + router a node that forwards IPv6 packets not explicitly + addressed to itself. (See Note below.) + + host any node that is not a router. (See Note below.) + + upper layer a protocol layer immediately above IPv6. Examples are + transport protocols such as TCP and UDP, control + protocols such as ICMP, routing protocols such as OSPF, + and internet-layer or lower-layer protocols being + "tunneled" over (i.e., encapsulated in) IPv6 such as + Internetwork Packet Exchange (IPX), AppleTalk, or IPv6 + itself. + + link a communication facility or medium over which nodes can + communicate at the link layer, i.e., the layer + immediately below IPv6. Examples are Ethernets (simple + or bridged); PPP links; X.25, Frame Relay, or ATM + networks; and internet-layer or higher-layer "tunnels", + such as tunnels over IPv4 or IPv6 itself. + + neighbors nodes attached to the same link. + + interface a node's attachment to a link. + + address an IPv6-layer identifier for an interface or a set of + interfaces. + + packet an IPv6 header plus payload. + + link MTU the maximum transmission unit, i.e., maximum packet size + in octets, that can be conveyed over a link. + + path MTU the minimum link MTU of all the links in a path between + a source node and a destination node. + + + + + + +Deering & Hinden Standards Track [Page 5] + +RFC 8200 IPv6 Specification July 2017 + + + Note: it is possible for a device with multiple interfaces to be + configured to forward non-self-destined packets arriving from some + set (fewer than all) of its interfaces and to discard non-self- + destined packets arriving from its other interfaces. Such a device + must obey the protocol requirements for routers when receiving + packets from, and interacting with neighbors over, the former + (forwarding) interfaces. It must obey the protocol requirements for + hosts when receiving packets from, and interacting with neighbors + over, the latter (non-forwarding) interfaces. + +3. IPv6 Header Format + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Version| Traffic Class | Flow Label | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Payload Length | Next Header | Hop Limit | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Source Address + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Destination Address + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Version 4-bit Internet Protocol version number = 6. + + Traffic Class 8-bit Traffic Class field. See Section 7. + + Flow Label 20-bit flow label. See Section 6. + + Payload Length 16-bit unsigned integer. Length of the IPv6 + payload, i.e., the rest of the packet + following this IPv6 header, in octets. (Note + that any extension headers (see Section 4) + present are considered part of the payload, + i.e., included in the length count.) + + + + + +Deering & Hinden Standards Track [Page 6] + +RFC 8200 IPv6 Specification July 2017 + + + Next Header 8-bit selector. Identifies the type of header + immediately following the IPv6 header. Uses + the same values as the IPv4 Protocol field + [IANA-PN]. + + Hop Limit 8-bit unsigned integer. Decremented by 1 by + each node that forwards the packet. When + forwarding, the packet is discarded if Hop + Limit was zero when received or is decremented + to zero. A node that is the destination of a + packet should not discard a packet with Hop + Limit equal to zero; it should process the + packet normally. + + Source Address 128-bit address of the originator of the + packet. See [RFC4291]. + + Destination Address 128-bit address of the intended recipient of + the packet (possibly not the ultimate + recipient, if a Routing header is present). + See [RFC4291] and Section 4.4. + +4. IPv6 Extension Headers + + In IPv6, optional internet-layer information is encoded in separate + headers that may be placed between the IPv6 header and the upper- + layer header in a packet. There is a small number of such extension + headers, each one identified by a distinct Next Header value. + + Extension headers are numbered from IANA IP Protocol Numbers + [IANA-PN], the same values used for IPv4 and IPv6. When processing a + sequence of Next Header values in a packet, the first one that is not + an extension header [IANA-EH] indicates that the next item in the + packet is the corresponding upper-layer header. A special "No Next + Header" value is used if there is no upper-layer header. + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 7] + +RFC 8200 IPv6 Specification July 2017 + + + As illustrated in these examples, an IPv6 packet may carry zero, one, + or more extension headers, each identified by the Next Header field + of the preceding header: + + +---------------+------------------------ + | IPv6 header | TCP header + data + | | + | Next Header = | + | TCP | + +---------------+------------------------ + + +---------------+----------------+------------------------ + | IPv6 header | Routing header | TCP header + data + | | | + | Next Header = | Next Header = | + | Routing | TCP | + +---------------+----------------+------------------------ + + +---------------+----------------+-----------------+----------------- + | IPv6 header | Routing header | Fragment header | fragment of TCP + | | | | header + data + | Next Header = | Next Header = | Next Header = | + | Routing | Fragment | TCP | + +---------------+----------------+-----------------+----------------- + + Extension headers (except for the Hop-by-Hop Options header) are not + processed, inserted, or deleted by any node along a packet's delivery + path, until the packet reaches the node (or each of the set of nodes, + in the case of multicast) identified in the Destination Address field + of the IPv6 header. + + The Hop-by-Hop Options header is not inserted or deleted, but may be + examined or processed by any node along a packet's delivery path, + until the packet reaches the node (or each of the set of nodes, in + the case of multicast) identified in the Destination Address field of + the IPv6 header. The Hop-by-Hop Options header, when present, must + immediately follow the IPv6 header. Its presence is indicated by the + value zero in the Next Header field of the IPv6 header. + + NOTE: While [RFC2460] required that all nodes must examine and + process the Hop-by-Hop Options header, it is now expected that nodes + along a packet's delivery path only examine and process the + Hop-by-Hop Options header if explicitly configured to do so. + + + + + + + + +Deering & Hinden Standards Track [Page 8] + +RFC 8200 IPv6 Specification July 2017 + + + At the destination node, normal demultiplexing on the Next Header + field of the IPv6 header invokes the module to process the first + extension header, or the upper-layer header if no extension header is + present. The contents and semantics of each extension header + determine whether or not to proceed to the next header. Therefore, + extension headers must be processed strictly in the order they appear + in the packet; a receiver must not, for example, scan through a + packet looking for a particular kind of extension header and process + that header prior to processing all preceding ones. + + If, as a result of processing a header, the destination node is + required to proceed to the next header but the Next Header value in + the current header is unrecognized by the node, it should discard the + packet and send an ICMP Parameter Problem message to the source of + the packet, with an ICMP Code value of 1 ("unrecognized Next Header + type encountered") and the ICMP Pointer field containing the offset + of the unrecognized value within the original packet. The same + action should be taken if a node encounters a Next Header value of + zero in any header other than an IPv6 header. + + Each extension header is an integer multiple of 8 octets long, in + order to retain 8-octet alignment for subsequent headers. Multi- + octet fields within each extension header are aligned on their + natural boundaries, i.e., fields of width n octets are placed at an + integer multiple of n octets from the start of the header, for n = 1, + 2, 4, or 8. + + A full implementation of IPv6 includes implementation of the + following extension headers: + + Hop-by-Hop Options + Fragment + Destination Options + Routing + Authentication + Encapsulating Security Payload + + The first four are specified in this document; the last two are + specified in [RFC4302] and [RFC4303], respectively. The current list + of IPv6 extension headers can be found at [IANA-EH]. + + + + + + + + + + + +Deering & Hinden Standards Track [Page 9] + +RFC 8200 IPv6 Specification July 2017 + + +4.1. Extension Header Order + + When more than one extension header is used in the same packet, it is + recommended that those headers appear in the following order: + + IPv6 header + Hop-by-Hop Options header + Destination Options header (note 1) + Routing header + Fragment header + Authentication header (note 2) + Encapsulating Security Payload header (note 2) + Destination Options header (note 3) + Upper-Layer header + + note 1: for options to be processed by the first destination that + appears in the IPv6 Destination Address field plus + subsequent destinations listed in the Routing header. + + note 2: additional recommendations regarding the relative order of + the Authentication and Encapsulating Security Payload + headers are given in [RFC4303]. + + note 3: for options to be processed only by the final destination + of the packet. + + Each extension header should occur at most once, except for the + Destination Options header, which should occur at most twice (once + before a Routing header and once before the upper-layer header). + + If the upper-layer header is another IPv6 header (in the case of IPv6 + being tunneled over or encapsulated in IPv6), it may be followed by + its own extension headers, which are separately subject to the same + ordering recommendations. + + If and when other extension headers are defined, their ordering + constraints relative to the above listed headers must be specified. + + IPv6 nodes must accept and attempt to process extension headers in + any order and occurring any number of times in the same packet, + except for the Hop-by-Hop Options header, which is restricted to + appear immediately after an IPv6 header only. Nonetheless, it is + strongly advised that sources of IPv6 packets adhere to the above + recommended order until and unless subsequent specifications revise + that recommendation. + + + + + + +Deering & Hinden Standards Track [Page 10] + +RFC 8200 IPv6 Specification July 2017 + + +4.2. Options + + Two of the currently defined extension headers specified in this + document -- the Hop-by-Hop Options header and the Destination Options + header -- carry a variable number of "options" that are type-length- + value (TLV) encoded in the following format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- - - - - - - - - + | Option Type | Opt Data Len | Option Data + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- - - - - - - - - + + Option Type 8-bit identifier of the type of option. + + Opt Data Len 8-bit unsigned integer. Length of the Option + Data field of this option, in octets. + + Option Data Variable-length field. Option-Type-specific + data. + + The sequence of options within a header must be processed strictly in + the order they appear in the header; a receiver must not, for + example, scan through the header looking for a particular kind of + option and process that option prior to processing all preceding + ones. + + The Option Type identifiers are internally encoded such that their + highest-order 2 bits specify the action that must be taken if the + processing IPv6 node does not recognize the Option Type: + + 00 - skip over this option and continue processing the header. + + 01 - discard the packet. + + 10 - discard the packet and, regardless of whether or not the + packet's Destination Address was a multicast address, send an + ICMP Parameter Problem, Code 2, message to the packet's + Source Address, pointing to the unrecognized Option Type. + + 11 - discard the packet and, only if the packet's Destination + Address was not a multicast address, send an ICMP Parameter + Problem, Code 2, message to the packet's Source Address, + pointing to the unrecognized Option Type. + + The third-highest-order bit of the Option Type specifies whether or + not the Option Data of that option can change en route to the + packet's final destination. When an Authentication header is present + + + + + +Deering & Hinden Standards Track [Page 11] + +RFC 8200 IPv6 Specification July 2017 + + + in the packet, for any option whose data may change en route, its + entire Option Data field must be treated as zero-valued octets when + computing or verifying the packet's authenticating value. + + 0 - Option Data does not change en route + + 1 - Option Data may change en route + + The three high-order bits described above are to be treated as part + of the Option Type, not independent of the Option Type. That is, a + particular option is identified by a full 8-bit Option Type, not just + the low-order 5 bits of an Option Type. + + The same Option Type numbering space is used for both the Hop-by-Hop + Options header and the Destination Options header. However, the + specification of a particular option may restrict its use to only one + of those two headers. + + Individual options may have specific alignment requirements, to + ensure that multi-octet values within Option Data fields fall on + natural boundaries. The alignment requirement of an option is + specified using the notation xn+y, meaning the Option Type must + appear at an integer multiple of x octets from the start of the + header, plus y octets. For example: + + 2n means any 2-octet offset from the start of the header. + 8n+2 means any 8-octet offset from the start of the header, plus + 2 octets. + + There are two padding options that are used when necessary to align + subsequent options and to pad out the containing header to a multiple + of 8 octets in length. These padding options must be recognized by + all IPv6 implementations: + + Pad1 option (alignment requirement: none) + + +-+-+-+-+-+-+-+-+ + | 0 | + +-+-+-+-+-+-+-+-+ + + NOTE! the format of the Pad1 option is a special case -- it does + not have length and value fields. + + The Pad1 option is used to insert 1 octet of padding into the + Options area of a header. If more than one octet of padding is + required, the PadN option, described next, should be used, rather + than multiple Pad1 options. + + + + +Deering & Hinden Standards Track [Page 12] + +RFC 8200 IPv6 Specification July 2017 + + + PadN option (alignment requirement: none) + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- - - - - - - - - + | 1 | Opt Data Len | Option Data + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+- - - - - - - - - + + The PadN option is used to insert two or more octets of padding + into the Options area of a header. For N octets of padding, the + Opt Data Len field contains the value N-2, and the Option Data + consists of N-2 zero-valued octets. + + Appendix A contains formatting guidelines for designing new options. + +4.3. Hop-by-Hop Options Header + + The Hop-by-Hop Options header is used to carry optional information + that may be examined and processed by every node along a packet's + delivery path. The Hop-by-Hop Options header is identified by a Next + Header value of 0 in the IPv6 header and has the following format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + | | + . . + . Options . + . . + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + + Next Header 8-bit selector. Identifies the type of header + immediately following the Hop-by-Hop Options + header. Uses the same values as the IPv4 + Protocol field [IANA-PN]. + + Hdr Ext Len 8-bit unsigned integer. Length of the + Hop-by-Hop Options header in 8-octet units, + not including the first 8 octets. + + Options Variable-length field, of length such that the + complete Hop-by-Hop Options header is an + integer multiple of 8 octets long. Contains + one or more TLV-encoded options, as described + in Section 4.2. + + The only hop-by-hop options defined in this document are the Pad1 and + PadN options specified in Section 4.2. + + + +Deering & Hinden Standards Track [Page 13] + +RFC 8200 IPv6 Specification July 2017 + + +4.4. Routing Header + + The Routing header is used by an IPv6 source to list one or more + intermediate nodes to be "visited" on the way to a packet's + destination. This function is very similar to IPv4's Loose Source + and Record Route option. The Routing header is identified by a Next + Header value of 43 in the immediately preceding header and has the + following format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len | Routing Type | Segments Left | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + . . + . type-specific data . + . . + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Next Header 8-bit selector. Identifies the type of header + immediately following the Routing header. + Uses the same values as the IPv4 Protocol + field [IANA-PN]. + + Hdr Ext Len 8-bit unsigned integer. Length of the Routing + header in 8-octet units, not including the + first 8 octets. + + Routing Type 8-bit identifier of a particular Routing + header variant. + + Segments Left 8-bit unsigned integer. Number of route + segments remaining, i.e., number of explicitly + listed intermediate nodes still to be visited + before reaching the final destination. + + type-specific data Variable-length field, of format determined by + the Routing Type, and of length such that the + complete Routing header is an integer multiple + of 8 octets long. + + + + + + + + + + + +Deering & Hinden Standards Track [Page 14] + +RFC 8200 IPv6 Specification July 2017 + + + If, while processing a received packet, a node encounters a Routing + header with an unrecognized Routing Type value, the required behavior + of the node depends on the value of the Segments Left field, as + follows: + + If Segments Left is zero, the node must ignore the Routing header + and proceed to process the next header in the packet, whose type + is identified by the Next Header field in the Routing header. + + If Segments Left is non-zero, the node must discard the packet and + send an ICMP Parameter Problem, Code 0, message to the packet's + Source Address, pointing to the unrecognized Routing Type. + + If, after processing a Routing header of a received packet, an + intermediate node determines that the packet is to be forwarded onto + a link whose link MTU is less than the size of the packet, the node + must discard the packet and send an ICMP Packet Too Big message to + the packet's Source Address. + + The currently defined IPv6 Routing Headers and their status can be + found at [IANA-RH]. Allocation guidelines for IPv6 Routing Headers + can be found in [RFC5871]. + +4.5. Fragment Header + + The Fragment header is used by an IPv6 source to send a packet larger + than would fit in the path MTU to its destination. (Note: unlike + IPv4, fragmentation in IPv6 is performed only by source nodes, not by + routers along a packet's delivery path -- see Section 5.) The + Fragment header is identified by a Next Header value of 44 in the + immediately preceding header and has the following format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Reserved | Fragment Offset |Res|M| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Identification | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Next Header 8-bit selector. Identifies the initial header + type of the Fragmentable Part of the original + packet (defined below). Uses the same values + as the IPv4 Protocol field [IANA-PN]. + + Reserved 8-bit reserved field. Initialized to zero for + transmission; ignored on reception. + + + + + + +Deering & Hinden Standards Track [Page 15] + +RFC 8200 IPv6 Specification July 2017 + + + Fragment Offset 13-bit unsigned integer. The offset, in + 8-octet units, of the data following this + header, relative to the start of the + Fragmentable Part of the original packet. + + Res 2-bit reserved field. Initialized to zero for + transmission; ignored on reception. + + M flag 1 = more fragments; 0 = last fragment. + + Identification 32 bits. See description below. + + In order to send a packet that is too large to fit in the MTU of the + path to its destination, a source node may divide the packet into + fragments and send each fragment as a separate packet, to be + reassembled at the receiver. + + For every packet that is to be fragmented, the source node generates + an Identification value. The Identification must be different than + that of any other fragmented packet sent recently* with the same + Source Address and Destination Address. If a Routing header is + present, the Destination Address of concern is that of the final + destination. + + * "recently" means within the maximum likely lifetime of a + packet, including transit time from source to destination and + time spent awaiting reassembly with other fragments of the same + packet. However, it is not required that a source node knows + the maximum packet lifetime. Rather, it is assumed that the + requirement can be met by implementing an algorithm that + results in a low identification reuse frequency. Examples of + algorithms that can meet this requirement are described in + [RFC7739]. + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 16] + +RFC 8200 IPv6 Specification July 2017 + + + The initial, large, unfragmented packet is referred to as the + "original packet", and it is considered to consist of three parts, as + illustrated: + + original packet: + + +------------------+-------------------------+---//----------------+ + | Per-Fragment | Extension & Upper-Layer | Fragmentable | + | Headers | Headers | Part | + +------------------+-------------------------+---//----------------+ + + The Per-Fragment headers must consist of the IPv6 header plus any + extension headers that must be processed by nodes en route to the + destination, that is, all headers up to and including the Routing + header if present, else the Hop-by-Hop Options header if present, + else no extension headers. + + The Extension headers are all other extension headers that are not + included in the Per-Fragment headers part of the packet. For this + purpose, the Encapsulating Security Payload (ESP) is not + considered an extension header. The Upper-Layer header is the + first upper-layer header that is not an IPv6 extension header. + Examples of upper-layer headers include TCP, UDP, IPv4, IPv6, + ICMPv6, and as noted ESP. + + The Fragmentable Part consists of the rest of the packet after the + upper-layer header or after any header (i.e., initial IPv6 header + or extension header) that contains a Next Header value of No Next + Header. + + The Fragmentable Part of the original packet is divided into + fragments. The lengths of the fragments must be chosen such that the + resulting fragment packets fit within the MTU of the path to the + packet's destination(s). Each complete fragment, except possibly the + last ("rightmost") one, is an integer multiple of 8 octets long. + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 17] + +RFC 8200 IPv6 Specification July 2017 + + + The fragments are transmitted in separate "fragment packets" as + illustrated: + + original packet: + + +-----------------+-----------------+--------+--------+-//-+--------+ + | Per-Fragment |Ext & Upper-Layer| first | second | | last | + | Headers | Headers |fragment|fragment|....|fragment| + +-----------------+-----------------+--------+--------+-//-+--------+ + + fragment packets: + + +------------------+---------+-------------------+----------+ + | Per-Fragment |Fragment | Ext & Upper-Layer | first | + | Headers | Header | Headers | fragment | + +------------------+---------+-------------------+----------+ + + +------------------+--------+-------------------------------+ + | Per-Fragment |Fragment| second | + | Headers | Header | fragment | + +------------------+--------+-------------------------------+ + o + o + o + +------------------+--------+----------+ + | Per-Fragment |Fragment| last | + | Headers | Header | fragment | + +------------------+--------+----------+ + + The first fragment packet is composed of: + + (1) The Per-Fragment headers of the original packet, with the + Payload Length of the original IPv6 header changed to contain + the length of this fragment packet only (excluding the length + of the IPv6 header itself), and the Next Header field of the + last header of the Per-Fragment headers changed to 44. + + (2) A Fragment header containing: + + The Next Header value that identifies the first header + after the Per-Fragment headers of the original packet. + + A Fragment Offset containing the offset of the fragment, + in 8-octet units, relative to the start of the + Fragmentable Part of the original packet. The Fragment + Offset of the first ("leftmost") fragment is 0. + + An M flag value of 1 as this is the first fragment. + + + +Deering & Hinden Standards Track [Page 18] + +RFC 8200 IPv6 Specification July 2017 + + + The Identification value generated for the original + packet. + + (3) Extension headers, if any, and the Upper-Layer header. These + headers must be in the first fragment. Note: This restricts + the size of the headers through the Upper-Layer header to the + MTU of the path to the packet's destinations(s). + + (4) The first fragment. + + The subsequent fragment packets are composed of: + + (1) The Per-Fragment headers of the original packet, with the + Payload Length of the original IPv6 header changed to contain + the length of this fragment packet only (excluding the length + of the IPv6 header itself), and the Next Header field of the + last header of the Per-Fragment headers changed to 44. + + (2) A Fragment header containing: + + The Next Header value that identifies the first header + after the Per-Fragment headers of the original packet. + + A Fragment Offset containing the offset of the fragment, + in 8-octet units, relative to the start of the + Fragmentable Part of the original packet. + + An M flag value of 0 if the fragment is the last + ("rightmost") one, else an M flag value of 1. + + The Identification value generated for the original + packet. + + (3) The fragment itself. + + Fragments must not be created that overlap with any other fragments + created from the original packet. + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 19] + +RFC 8200 IPv6 Specification July 2017 + + + At the destination, fragment packets are reassembled into their + original, unfragmented form, as illustrated: + + reassembled original packet: + + +---------------+-----------------+---------+--------+-//--+--------+ + | Per-Fragment |Ext & Upper-Layer| first | second | | last | + | Headers | Headers |frag data|fragment|.....|fragment| + +---------------+-----------------+---------+--------+-//--+--------+ + + The following rules govern reassembly: + + An original packet is reassembled only from fragment packets that + have the same Source Address, Destination Address, and Fragment + Identification. + + The Per-Fragment headers of the reassembled packet consists of all + headers up to, but not including, the Fragment header of the first + fragment packet (that is, the packet whose Fragment Offset is + zero), with the following two changes: + + The Next Header field of the last header of the Per-Fragment + headers is obtained from the Next Header field of the first + fragment's Fragment header. + + The Payload Length of the reassembled packet is computed from + the length of the Per-Fragment headers and the length and + offset of the last fragment. For example, a formula for + computing the Payload Length of the reassembled original packet + is: + + PL.orig = PL.first - FL.first - 8 + (8 * FO.last) + FL.last + + where + PL.orig = Payload Length field of reassembled packet. + PL.first = Payload Length field of first fragment packet. + FL.first = length of fragment following Fragment header of + first fragment packet. + FO.last = Fragment Offset field of Fragment header of last + fragment packet. + FL.last = length of fragment following Fragment header of + last fragment packet. + + The Fragmentable Part of the reassembled packet is constructed + from the fragments following the Fragment headers in each of + the fragment packets. The length of each fragment is computed + by subtracting from the packet's Payload Length the length of + the headers between the IPv6 header and fragment itself; its + + + +Deering & Hinden Standards Track [Page 20] + +RFC 8200 IPv6 Specification July 2017 + + + relative position in Fragmentable Part is computed from its + Fragment Offset value. + + The Fragment header is not present in the final, reassembled + packet. + + If the fragment is a whole datagram (that is, both the Fragment + Offset field and the M flag are zero), then it does not need + any further reassembly and should be processed as a fully + reassembled packet (i.e., updating Next Header, adjust Payload + Length, removing the Fragment header, etc.). Any other + fragments that match this packet (i.e., the same IPv6 Source + Address, IPv6 Destination Address, and Fragment Identification) + should be processed independently. + + The following error conditions may arise when reassembling fragmented + packets: + + o If insufficient fragments are received to complete reassembly + of a packet within 60 seconds of the reception of the first- + arriving fragment of that packet, reassembly of that packet + must be abandoned and all the fragments that have been received + for that packet must be discarded. If the first fragment + (i.e., the one with a Fragment Offset of zero) has been + received, an ICMP Time Exceeded -- Fragment Reassembly Time + Exceeded message should be sent to the source of that fragment. + + o If the length of a fragment, as derived from the fragment + packet's Payload Length field, is not a multiple of 8 octets + and the M flag of that fragment is 1, then that fragment must + be discarded and an ICMP Parameter Problem, Code 0, message + should be sent to the source of the fragment, pointing to the + Payload Length field of the fragment packet. + + o If the length and offset of a fragment are such that the + Payload Length of the packet reassembled from that fragment + would exceed 65,535 octets, then that fragment must be + discarded and an ICMP Parameter Problem, Code 0, message should + be sent to the source of the fragment, pointing to the Fragment + Offset field of the fragment packet. + + o If the first fragment does not include all headers through an + Upper-Layer header, then that fragment should be discarded and + an ICMP Parameter Problem, Code 3, message should be sent to + the source of the fragment, with the Pointer field set to zero. + + + + + + +Deering & Hinden Standards Track [Page 21] + +RFC 8200 IPv6 Specification July 2017 + + + o If any of the fragments being reassembled overlap with any + other fragments being reassembled for the same packet, + reassembly of that packet must be abandoned and all the + fragments that have been received for that packet must be + discarded, and no ICMP error messages should be sent. + + It should be noted that fragments may be duplicated in the + network. Instead of treating these exact duplicate fragments + as overlapping fragments, an implementation may choose to + detect this case and drop exact duplicate fragments while + keeping the other fragments belonging to the same packet. + + The following conditions are not expected to occur frequently but are + not considered errors if they do: + + The number and content of the headers preceding the Fragment + header of different fragments of the same original packet may + differ. Whatever headers are present, preceding the Fragment + header in each fragment packet, are processed when the packets + arrive, prior to queueing the fragments for reassembly. Only + those headers in the Offset zero fragment packet are retained in + the reassembled packet. + + The Next Header values in the Fragment headers of different + fragments of the same original packet may differ. Only the value + from the Offset zero fragment packet is used for reassembly. + + Other fields in the IPv6 header may also vary across the fragments + being reassembled. Specifications that use these fields may + provide additional instructions if the basic mechanism of using + the values from the Offset zero fragment is not sufficient. For + example, Section 5.3 of [RFC3168] describes how to combine the + Explicit Congestion Notification (ECN) bits from different + fragments to derive the ECN bits of the reassembled packet. + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 22] + +RFC 8200 IPv6 Specification July 2017 + + +4.6. Destination Options Header + + The Destination Options header is used to carry optional information + that need be examined only by a packet's destination node(s). The + Destination Options header is identified by a Next Header value of 60 + in the immediately preceding header and has the following format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + | | + . . + . Options . + . . + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Next Header 8-bit selector. Identifies the type of header + immediately following the Destination Options + header. Uses the same values as the IPv4 + Protocol field [IANA-PN]. + + Hdr Ext Len 8-bit unsigned integer. Length of the + Destination Options header in 8-octet units, + not including the first 8 octets. + + Options Variable-length field, of length such that the + complete Destination Options header is an + integer multiple of 8 octets long. Contains + one or more TLV-encoded options, as described + in Section 4.2. + + The only destination options defined in this document are the Pad1 + and PadN options specified in Section 4.2. + + Note that there are two possible ways to encode optional destination + information in an IPv6 packet: either as an option in the Destination + Options header or as a separate extension header. The Fragment + header and the Authentication header are examples of the latter + approach. Which approach can be used depends on what action is + desired of a destination node that does not understand the optional + information: + + o If the desired action is for the destination node to discard + the packet and, only if the packet's Destination Address is not + a multicast address, send an ICMP Unrecognized Type message to + the packet's Source Address, then the information may be + encoded either as a separate header or as an option in the + + + +Deering & Hinden Standards Track [Page 23] + +RFC 8200 IPv6 Specification July 2017 + + + Destination Options header whose Option Type has the value 11 + in its highest-order 2 bits. The choice may depend on such + factors as which takes fewer octets, or which yields better + alignment or more efficient parsing. + + o If any other action is desired, the information must be encoded + as an option in the Destination Options header whose Option + Type has the value 00, 01, or 10 in its highest-order 2 bits, + specifying the desired action (see Section 4.2). + +4.7. No Next Header + + The value 59 in the Next Header field of an IPv6 header or any + extension header indicates that there is nothing following that + header. If the Payload Length field of the IPv6 header indicates the + presence of octets past the end of a header whose Next Header field + contains 59, those octets must be ignored and passed on unchanged if + the packet is forwarded. + +4.8. Defining New Extension Headers and Options + + Defining new IPv6 extension headers is not recommended, unless there + are no existing IPv6 extension headers that can be used by specifying + a new option for that IPv6 extension header. A proposal to specify a + new IPv6 extension header must include a detailed technical + explanation of why an existing IPv6 extension header can not be used + for the desired new function. See [RFC6564] for additional + background information. + + Note: New extension headers that require hop-by-hop behavior must not + be defined because, as specified in Section 4 of this document, the + only extension header that has hop-by-hop behavior is the Hop-by-Hop + Options header. + + New hop-by-hop options are not recommended because nodes may be + configured to ignore the Hop-by-Hop Options header, drop packets + containing a Hop-by-Hop Options header, or assign packets containing + a Hop-by-Hop Options header to a slow processing path. Designers + considering defining new hop-by-hop options need to be aware of this + likely behavior. There has to be a very clear justification why any + new hop-by-hop option is needed before it is standardized. + + Instead of defining new extension headers, it is recommended that the + Destination Options header is used to carry optional information that + must be examined only by a packet's destination node(s), because they + provide better handling and backward compatibility. + + + + + +Deering & Hinden Standards Track [Page 24] + +RFC 8200 IPv6 Specification July 2017 + + + If new extension headers are defined, they need to use the following + format: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + | | + . . + . Header-Specific Data . + . . + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Next Header 8-bit selector. Identifies the type of + header immediately following the extension + header. Uses the same values as the IPv4 + Protocol field [IANA-PN]. + + Hdr Ext Len 8-bit unsigned integer. Length of the + Destination Options header in 8-octet units, + not including the first 8 octets. + + Header Specific Data Variable-length field. Fields specific to + the extension header. + +5. Packet Size Issues + + IPv6 requires that every link in the Internet have an MTU of 1280 + octets or greater. This is known as the IPv6 minimum link MTU. On + any link that cannot convey a 1280-octet packet in one piece, link- + specific fragmentation and reassembly must be provided at a layer + below IPv6. + + Links that have a configurable MTU (for example, PPP links [RFC1661]) + must be configured to have an MTU of at least 1280 octets; it is + recommended that they be configured with an MTU of 1500 octets or + greater, to accommodate possible encapsulations (i.e., tunneling) + without incurring IPv6-layer fragmentation. + + From each link to which a node is directly attached, the node must be + able to accept packets as large as that link's MTU. + + It is strongly recommended that IPv6 nodes implement Path MTU + Discovery [RFC8201], in order to discover and take advantage of path + MTUs greater than 1280 octets. However, a minimal IPv6 + implementation (e.g., in a boot ROM) may simply restrict itself to + sending packets no larger than 1280 octets, and omit implementation + of Path MTU Discovery. + + + +Deering & Hinden Standards Track [Page 25] + +RFC 8200 IPv6 Specification July 2017 + + + In order to send a packet larger than a path's MTU, a node may use + the IPv6 Fragment header to fragment the packet at the source and + have it reassembled at the destination(s). However, the use of such + fragmentation is discouraged in any application that is able to + adjust its packets to fit the measured path MTU (i.e., down to 1280 + octets). + + A node must be able to accept a fragmented packet that, after + reassembly, is as large as 1500 octets. A node is permitted to + accept fragmented packets that reassemble to more than 1500 octets. + An upper-layer protocol or application that depends on IPv6 + fragmentation to send packets larger than the MTU of a path should + not send packets larger than 1500 octets unless it has assurance that + the destination is capable of reassembling packets of that larger + size. + +6. Flow Labels + + The 20-bit Flow Label field in the IPv6 header is used by a source to + label sequences of packets to be treated in the network as a single + flow. + + The current definition of the IPv6 Flow Label can be found in + [RFC6437]. + +7. Traffic Classes + + The 8-bit Traffic Class field in the IPv6 header is used by the + network for traffic management. The value of the Traffic Class bits + in a received packet or fragment might be different from the value + sent by the packet's source. + + The current use of the Traffic Class field for Differentiated + Services and Explicit Congestion Notification is specified in + [RFC2474] and [RFC3168]. + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 26] + +RFC 8200 IPv6 Specification July 2017 + + +8. Upper-Layer Protocol Issues + +8.1. Upper-Layer Checksums + + Any transport or other upper-layer protocol that includes the + addresses from the IP header in its checksum computation must be + modified for use over IPv6, to include the 128-bit IPv6 addresses + instead of 32-bit IPv4 addresses. In particular, the following + illustration shows the TCP and UDP "pseudo-header" for IPv6: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Source Address + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + + + | | + + Destination Address + + | | + + + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Upper-Layer Packet Length | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | zero | Next Header | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + o If the IPv6 packet contains a Routing header, the Destination + Address used in the pseudo-header is that of the final + destination. At the originating node, that address will be in + the last element of the Routing header; at the recipient(s), + that address will be in the Destination Address field of the + IPv6 header. + + o The Next Header value in the pseudo-header identifies the + upper-layer protocol (e.g., 6 for TCP or 17 for UDP). It will + differ from the Next Header value in the IPv6 header if there + are extension headers between the IPv6 header and the upper- + layer header. + + + + + + + +Deering & Hinden Standards Track [Page 27] + +RFC 8200 IPv6 Specification July 2017 + + + o The Upper-Layer Packet Length in the pseudo-header is the + length of the upper-layer header and data (e.g., TCP header + plus TCP data). Some upper-layer protocols carry their own + length information (e.g., the Length field in the UDP header); + for such protocols, that is the length used in the pseudo- + header. Other protocols (such as TCP) do not carry their own + length information, in which case the length used in the + pseudo-header is the Payload Length from the IPv6 header, minus + the length of any extension headers present between the IPv6 + header and the upper-layer header. + + o Unlike IPv4, the default behavior when UDP packets are + originated by an IPv6 node is that the UDP checksum is not + optional. That is, whenever originating a UDP packet, an IPv6 + node must compute a UDP checksum over the packet and the + pseudo-header, and, if that computation yields a result of + zero, it must be changed to hex FFFF for placement in the UDP + header. IPv6 receivers must discard UDP packets containing a + zero checksum and should log the error. + + o As an exception to the default behavior, protocols that use UDP + as a tunnel encapsulation may enable zero-checksum mode for a + specific port (or set of ports) for sending and/or receiving. + Any node implementing zero-checksum mode must follow the + requirements specified in "Applicability Statement for the Use + of IPv6 UDP Datagrams with Zero Checksums" [RFC6936]. + + The IPv6 version of ICMP [RFC4443] includes the above pseudo-header + in its checksum computation; this is a change from the IPv4 version + of ICMP, which does not include a pseudo-header in its checksum. The + reason for the change is to protect ICMP from misdelivery or + corruption of those fields of the IPv6 header on which it depends, + which, unlike IPv4, are not covered by an internet-layer checksum. + The Next Header field in the pseudo-header for ICMP contains the + value 58, which identifies the IPv6 version of ICMP. + +8.2. Maximum Packet Lifetime + + Unlike IPv4, IPv6 nodes are not required to enforce maximum packet + lifetime. That is the reason the IPv4 "Time-to-Live" field was + renamed "Hop Limit" in IPv6. In practice, very few, if any, IPv4 + implementations conform to the requirement that they limit packet + lifetime, so this is not a change in practice. Any upper-layer + protocol that relies on the internet layer (whether IPv4 or IPv6) to + limit packet lifetime ought to be upgraded to provide its own + mechanisms for detecting and discarding obsolete packets. + + + + + +Deering & Hinden Standards Track [Page 28] + +RFC 8200 IPv6 Specification July 2017 + + +8.3. Maximum Upper-Layer Payload Size + + When computing the maximum payload size available for upper-layer + data, an upper-layer protocol must take into account the larger size + of the IPv6 header relative to the IPv4 header. For example, in + IPv4, TCP's Maximum Segment Size (MSS) option is computed as the + maximum packet size (a default value or a value learned through Path + MTU Discovery) minus 40 octets (20 octets for the minimum-length IPv4 + header and 20 octets for the minimum-length TCP header). When using + TCP over IPv6, the MSS must be computed as the maximum packet size + minus 60 octets, because the minimum-length IPv6 header (i.e., an + IPv6 header with no extension headers) is 20 octets longer than a + minimum-length IPv4 header. + +8.4. Responding to Packets Carrying Routing Headers + + When an upper-layer protocol sends one or more packets in response to + a received packet that included a Routing header, the response + packet(s) must not include a Routing header that was automatically + derived by "reversing" the received Routing header UNLESS the + integrity and authenticity of the received Source Address and Routing + header have been verified (e.g., via the use of an Authentication + header in the received packet). In other words, only the following + kinds of packets are permitted in response to a received packet + bearing a Routing header: + + o Response packets that do not carry Routing headers. + + o Response packets that carry Routing headers that were NOT + derived by reversing the Routing header of the received packet + (for example, a Routing header supplied by local + configuration). + + o Response packets that carry Routing headers that were derived + by reversing the Routing header of the received packet IF AND + ONLY IF the integrity and authenticity of the Source Address + and Routing header from the received packet have been verified + by the responder. + +9. IANA Considerations + + RFC 2460 is referenced in a number of IANA registries. These + include: + + o Internet Protocol Version 6 (IPv6) Parameters [IANA-6P] + + o Assigned Internet Protocol Numbers [IANA-PN] + + + + +Deering & Hinden Standards Track [Page 29] + +RFC 8200 IPv6 Specification July 2017 + + + o ONC RPC Network Identifiers (netids) [IANA-NI] + + o Network Layer Protocol Identifiers (NLPIDs) of Interest + [IANA-NL] + + o Protocol Registries [IANA-PR] + + The IANA has updated these references to point to this document. + +10. Security Considerations + + IPv6, from the viewpoint of the basic format and transmission of + packets, has security properties that are similar to IPv4. These + security issues include: + + o Eavesdropping, where on-path elements can observe the whole + packet (including both contents and metadata) of each IPv6 + datagram. + o Replay, where the attacker records a sequence of packets off of + the wire and plays them back to the party that originally + received them. + o Packet insertion, where the attacker forges a packet with some + chosen set of properties and injects it into the network. + o Packet deletion, where the attacker removes a packet from the + wire. + o Packet modification, where the attacker removes a packet from + the wire, modifies it, and reinjects it into the network. + o Man-in-the-middle (MITM) attacks, where the attacker subverts + the communication stream in order to pose as the sender to + receiver and the receiver to the sender. + o Denial-of-service (DoS) attacks, where the attacker sends large + amounts of legitimate traffic to a destination to overwhelm it. + + IPv6 packets can be protected from eavesdropping, replay, packet + insertion, packet modification, and MITM attacks by use of the + "Security Architecture for the Internet Protocol" [RFC4301]. In + addition, upper-layer protocols such as Transport Layer Security + (TLS) or Secure Shell (SSH) can be used to protect the application- + layer traffic running on top of IPv6. + + There is not any mechanism to protect against DoS attacks. Defending + against these type of attacks is outside the scope of this + specification. + + IPv6 addresses are significantly larger than IPv4 addresses making it + much harder to scan the address space across the Internet and even on + a single network link (e.g., Local Area Network). See [RFC7707] for + more information. + + + +Deering & Hinden Standards Track [Page 30] + +RFC 8200 IPv6 Specification July 2017 + + + IPv6 addresses of nodes are expected to be more visible on the + Internet as compared with IPv4 since the use of address translation + technology is reduced. This creates some additional privacy issues + such as making it easier to distinguish endpoints. See [RFC7721] for + more information. + + The design of IPv6 extension header architecture, while adding a lot + of flexibility, also creates new security challenges. As noted + below, issues relating to the Fragment extension header have been + resolved, but it's clear that for any new extension header designed + in the future, the security implications need to be examined + thoroughly, and this needs to include how the new extension header + works with existing extension headers. See [RFC7045] for more + information. + + This version of the IPv6 specification resolves a number of security + issues that were found with the previous version [RFC2460] of the + IPv6 specification. These include: + + o Revised the text to handle the case of fragments that are whole + datagrams (i.e., both the Fragment Offset field and the M flag + are zero). If received, they should be processed as a + reassembled packet. Any other fragments that match should be + processed independently. The Fragment creation process was + modified to not create whole datagram fragments (Fragment + Offset field and the M flag are zero). See [RFC6946] and + [RFC8021] for more information. + + o Removed the paragraph in Section 5 that required including a + Fragment header to outgoing packets if an ICMP Packet Too Big + message reporting a Next-Hop MTU is less than 1280. See + [RFC6946] for more information. + + o Changed the text to require that IPv6 nodes must not create + overlapping fragments. Also, when reassembling an IPv6 + datagram, if one or more of its constituent fragments is + determined to be an overlapping fragment, the entire datagram + (and any constituent fragments) must be silently discarded. + Includes clarification that no ICMP error message should be + sent if overlapping fragments are received. See [RFC5722] for + more information. + + o Revised the text to require that all headers through the first + upper-layer header are in the first fragment. See [RFC7112] + for more information. + + + + + + +Deering & Hinden Standards Track [Page 31] + +RFC 8200 IPv6 Specification July 2017 + + + o Incorporated the updates from [RFC5095] and [RFC5871] to remove + the description of the Routing Header type 0 (RH0), that the + allocations guidelines for Routing headers are specified in RFC + 5871, and removed RH0 from the list of required extension + headers. + + Security issues relating to other parts of IPv6 including addressing, + ICMPv6, Path MTU Discovery, etc., are discussed in the appropriate + specifications. + +11. References + +11.1. Normative References + + [RFC791] Postel, J., "Internet Protocol", STD 5, RFC 791, + DOI 10.17487/RFC0791, September 1981, + . + + [RFC2474] Nichols, K., Blake, S., Baker, F., and D. Black, + "Definition of the Differentiated Services Field (DS + Field) in the IPv4 and IPv6 Headers", RFC 2474, + DOI 10.17487/RFC2474, December 1998, + . + + [RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition + of Explicit Congestion Notification (ECN) to IP", + RFC 3168, DOI 10.17487/RFC3168, September 2001, + . + + [RFC4291] Hinden, R. and S. Deering, "IP Version 6 Addressing + Architecture", RFC 4291, DOI 10.17487/RFC4291, February + 2006, . + + [RFC4443] Conta, A., Deering, S., and M. Gupta, Ed., "Internet + Control Message Protocol (ICMPv6) for the Internet + Protocol Version 6 (IPv6) Specification", STD 89, + RFC 4443, DOI 10.17487/RFC4443, March 2006, + . + + [RFC6437] Amante, S., Carpenter, B., Jiang, S., and J. Rajahalme, + "IPv6 Flow Label Specification", RFC 6437, + DOI 10.17487/RFC6437, November 2011, + . + + + + + + + + +Deering & Hinden Standards Track [Page 32] + +RFC 8200 IPv6 Specification July 2017 + + +11.2. Informative References + + [Err2541] RFC Errata, Erratum ID 2541, RFC 2460. + + [Err4279] RFC Errata, Erratum ID 4279, RFC 2460. + + [Err4657] RFC Errata, Erratum ID 4657, RFC 2460. + + [Err4662] RFC Errata, Erratum ID 4662, RFC 2460. + + [IANA-6P] IANA, "Internet Protocol Version 6 (IPv6) Parameters", + . + + [IANA-EH] IANA, "IPv6 Extension Header Types", + . + + [IANA-NI] IANA, "ONC RPC Network Identifiers (netids)", + . + + [IANA-NL] IANA, "Network Layer Protocol Identifiers (NLPIDs) of + Interest", . + + [IANA-PN] IANA, "Protocol Numbers", + . + + [IANA-PR] IANA, "Protocol Registries", . + + [IANA-RH] IANA, "Routing Types", . + + [RFC1661] Simpson, W., Ed., "The Point-to-Point Protocol (PPP)", + STD 51, RFC 1661, DOI 10.17487/RFC1661, July 1994, + . + + [RFC2460] Deering, S. and R. Hinden, "Internet Protocol, Version 6 + (IPv6) Specification", RFC 2460, DOI 10.17487/RFC2460, + December 1998, . + + [RFC4301] Kent, S. and K. Seo, "Security Architecture for the + Internet Protocol", RFC 4301, DOI 10.17487/RFC4301, + December 2005, . + + [RFC4302] Kent, S., "IP Authentication Header", RFC 4302, + DOI 10.17487/RFC4302, December 2005, + . + + + + + +Deering & Hinden Standards Track [Page 33] + +RFC 8200 IPv6 Specification July 2017 + + + [RFC4303] Kent, S., "IP Encapsulating Security Payload (ESP)", + RFC 4303, DOI 10.17487/RFC4303, December 2005, + . + + [RFC5095] Abley, J., Savola, P., and G. Neville-Neil, "Deprecation + of Type 0 Routing Headers in IPv6", RFC 5095, + DOI 10.17487/RFC5095, December 2007, + . + + [RFC5722] Krishnan, S., "Handling of Overlapping IPv6 Fragments", + RFC 5722, DOI 10.17487/RFC5722, December 2009, + . + + [RFC5871] Arkko, J. and S. Bradner, "IANA Allocation Guidelines for + the IPv6 Routing Header", RFC 5871, DOI 10.17487/RFC5871, + May 2010, . + + [RFC6564] Krishnan, S., Woodyatt, J., Kline, E., Hoagland, J., and + M. Bhatia, "A Uniform Format for IPv6 Extension Headers", + RFC 6564, DOI 10.17487/RFC6564, April 2012, + . + + [RFC6936] Fairhurst, G. and M. Westerlund, "Applicability Statement + for the Use of IPv6 UDP Datagrams with Zero Checksums", + RFC 6936, DOI 10.17487/RFC6936, April 2013, + . + + [RFC6946] Gont, F., "Processing of IPv6 "Atomic" Fragments", + RFC 6946, DOI 10.17487/RFC6946, May 2013, + . + + [RFC7045] Carpenter, B. and S. Jiang, "Transmission and Processing + of IPv6 Extension Headers", RFC 7045, + DOI 10.17487/RFC7045, December 2013, + . + + [RFC7112] Gont, F., Manral, V., and R. Bonica, "Implications of + Oversized IPv6 Header Chains", RFC 7112, + DOI 10.17487/RFC7112, January 2014, + . + + [RFC7707] Gont, F. and T. Chown, "Network Reconnaissance in IPv6 + Networks", RFC 7707, DOI 10.17487/RFC7707, March 2016, + . + + + + + + + +Deering & Hinden Standards Track [Page 34] + +RFC 8200 IPv6 Specification July 2017 + + + [RFC7721] Cooper, A., Gont, F., and D. Thaler, "Security and Privacy + Considerations for IPv6 Address Generation Mechanisms", + RFC 7721, DOI 10.17487/RFC7721, March 2016, + . + + [RFC7739] Gont, F., "Security Implications of Predictable Fragment + Identification Values", RFC 7739, DOI 10.17487/RFC7739, + February 2016, . + + [RFC8021] Gont, F., Liu, W., and T. Anderson, "Generation of IPv6 + Atomic Fragments Considered Harmful", RFC 8021, + DOI 10.17487/RFC8021, January 2017, + . + + [RFC8201] McCann, J., Deering, S., Mogul, J., and R. Hinden, "Path + MTU Discovery for IP version 6", STD 87, RFC 8201, + DOI 10.17487/RFC8201, July 2017, + . + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 35] + +RFC 8200 IPv6 Specification July 2017 + + +Appendix A. Formatting Guidelines for Options + + This appendix gives some advice on how to lay out the fields when + designing new options to be used in the Hop-by-Hop Options header or + the Destination Options header, as described in Section 4.2. These + guidelines are based on the following assumptions: + + o One desirable feature is that any multi-octet fields within the + Option Data area of an option be aligned on their natural + boundaries, i.e., fields of width n octets should be placed at + an integer multiple of n octets from the start of the + Hop-by-Hop or Destination Options header, for n = 1, 2, 4, or + 8. + + o Another desirable feature is that the Hop-by-Hop or Destination + Options header take up as little space as possible, subject to + the requirement that the header be an integer multiple of 8 + octets long. + + o It may be assumed that, when either of the option-bearing + headers are present, they carry a very small number of options, + usually only one. + + These assumptions suggest the following approach to laying out the + fields of an option: order the fields from smallest to largest, with + no interior padding, then derive the alignment requirement for the + entire option based on the alignment requirement of the largest field + (up to a maximum alignment of 8 octets). This approach is + illustrated in the following examples: + + Example 1 + + If an option X required two data fields, one of length 8 octets and + one of length 4 octets, it would be laid out as follows: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Option Type=X |Opt Data Len=12| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + 8-octet field + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + + + + + + +Deering & Hinden Standards Track [Page 36] + +RFC 8200 IPv6 Specification July 2017 + + + Its alignment requirement is 8n+2, to ensure that the 8-octet field + starts at a multiple-of-8 offset from the start of the enclosing + header. A complete Hop-by-Hop or Destination Options header + containing this one option would look as follows: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len=1 | Option Type=X |Opt Data Len=12| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + 8-octet field + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Example 2 + + If an option Y required three data fields, one of length 4 octets, + one of length 2 octets, and one of length 1 octet, it would be laid + out as follows: + + +-+-+-+-+-+-+-+-+ + | Option Type=Y | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Opt Data Len=7 | 1-octet field | 2-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Its alignment requirement is 4n+3, to ensure that the 4-octet field + starts at a multiple-of-4 offset from the start of the enclosing + header. A complete Hop-by-Hop or Destination Options header + containing this one option would look as follows: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len=1 | Pad1 Option=0 | Option Type=Y | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Opt Data Len=7 | 1-octet field | 2-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | PadN Option=1 |Opt Data Len=2 | 0 | 0 | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + + + + + + + +Deering & Hinden Standards Track [Page 37] + +RFC 8200 IPv6 Specification July 2017 + + + Example 3 + + A Hop-by-Hop or Destination Options header containing both options X + and Y from Examples 1 and 2 would have one of the two following + formats, depending on which option appeared first: + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len=3 | Option Type=X |Opt Data Len=12| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + 8-octet field + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | PadN Option=1 |Opt Data Len=1 | 0 | Option Type=Y | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Opt Data Len=7 | 1-octet field | 2-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | PadN Option=1 |Opt Data Len=2 | 0 | 0 | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Next Header | Hdr Ext Len=3 | Pad1 Option=0 | Option Type=Y | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + |Opt Data Len=7 | 1-octet field | 2-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | PadN Option=1 |Opt Data Len=4 | 0 | 0 | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 0 | 0 | Option Type=X |Opt Data Len=12| + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 4-octet field | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | + + 8-octet field + + | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + + + + + + + + + +Deering & Hinden Standards Track [Page 38] + +RFC 8200 IPv6 Specification July 2017 + + +Appendix B. Changes Since RFC 2460 + + This memo has the following changes from RFC 2460. + + o Removed IP Next Generation from the Abstract. + + o Added text in Section 1 that the data transmission order is the + same as IPv4 as defined in RFC 791. + + o Clarified the text in Section 3 about decrementing the Hop Limit. + + o Clarified that extension headers (except for the Hop-by-Hop + Options header) are not processed, inserted, or deleted by any + node along a packet's delivery path. + + o Changed requirement for the Hop-by-Hop Options header to a "may", + and added a note to indicate what is expected regarding the + Hop-by-Hop Options header. + + o Added a paragraph to Section 4 to clarify how extension headers + are numbered and which are upper-layer headers. + + o Added a reference to the end of Section 4 to the "IPv6 Extension + Header Types" IANA registry. + + o Incorporated the updates from RFCs 5095 and 5871 to remove the + description of RH0, that the allocations guidelines for routing + headers are specified in RFC 5871, and removed RH0 from the list + of required extension headers. + + o Revised Section 4.5 on IPv6 fragmentation based on updates from + RFCs 5722, 6946, 7112, and 8021. This includes: + + - Revised the text to handle the case of fragments that are whole + datagrams (i.e., both the Fragment Offset field and the M flag + are zero). If received, they should be processed as a + reassembled packet. Any other fragments that match should be + processed independently. The revised Fragment creation process + was modified to not create whole datagram fragments (Fragment + Offset field and the M flag are zero). + + - Changed the text to require that IPv6 nodes must not create + overlapping fragments. Also, when reassembling an IPv6 + datagram, if one or more its constituent fragments is + determined to be an overlapping fragment, the entire datagram + (and any constituent fragments) must be silently discarded. + Includes a clarification that no ICMP error message should be + sent if overlapping fragments are received. + + + +Deering & Hinden Standards Track [Page 39] + +RFC 8200 IPv6 Specification July 2017 + + + - Revised the text to require that all headers through the first + Upper-Layer header are in the first fragment. This changed the + text describing how packets are fragmented and reassembled and + added a new error case. + + - Added text to the Fragment header process on handling exact + duplicate fragments. + + - Updated the Fragmentation header text to correct the inclusion + of an Authentication Header (AH) and noted No Next Header case. + + - Changed terminology in the Fragment header section from + "Unfragmentable Headers" to "Per-Fragment headers". + + - Removed the paragraph in Section 5 that required including a + Fragment header to outgoing packets if an ICMP Packet Too Big + message reports a Next-Hop MTU less than 1280. + + - Changed the text to clarify MTU restriction and 8-byte + restrictions, and noted the restriction on headers in the first + fragment. + + o In Section 4.5, added clarification noting that some fields in the + IPv6 header may also vary across the fragments being reassembled, + and that other specifications may provide additional instructions + for how they should be reassembled. See, for example, Section 5.3 + of [RFC3168]. + + o Incorporated the update from RFC 6564 to add a new Section 4.8 + that describes recommendations for defining new extension headers + and options. + + o Added text to Section 5 to define "IPv6 minimum link MTU". + + o Simplified the text in Section 6 about Flow Labels and removed + what was Appendix A ("Semantics and Usage of the Flow Label + Field"); instead, pointed to the current specifications of the + IPv6 Flow Label field in [RFC6437] and the Traffic Class field in + [RFC2474] and [RFC3168]. + + o Incorporated the update made by RFC 6935 ("IPv6 and UDP Checksums + for Tunneled Packets") in Section 8. Added an exception to the + default behavior for the handling of UDP packets with zero + checksums for tunnels. + + o Added instruction to Section 9, "IANA Considerations", to change + references to RFC 2460 to this document. + + + + +Deering & Hinden Standards Track [Page 40] + +RFC 8200 IPv6 Specification July 2017 + + + o Revised and expanded Section 10, "Security Considerations". + + o Added a paragraph to the Acknowledgments section acknowledging the + authors of the updating documents. + + o Updated references to current versions and assigned references to + normative and informative. + + o Made changes to resolve the errata on RFC 2460. These are: + + Erratum ID 2541 [Err2541]: This erratum notes that RFC 2460 + didn't update RFC 2205 when the length of the flow label was + changed from 24 to 20 bits from RFC 1883. This issue was + resolved in RFC 6437 where the flow label is defined. This + specification now references RFC 6437. No change is required. + + Erratum ID 4279 [Err4279]: This erratum noted that the + specification doesn't handle the case of a forwarding node + receiving a packet with a zero Hop Limit. This is fixed in + Section 3 of this specification. + + Erratum ID 4657 [Err4657]: This erratum proposed text that + extension headers must never be inserted by any node other than + the source of the packet. This was resolved in Section 4, + "IPv6 Extension Headers". + + Erratum ID 4662 [Err4662]: This erratum proposed text that + extension headers, with one exception, are not examined, + processed, modified, inserted, or deleted by any node along a + packet's delivery path. This was resolved in Section 4, "IPv6 + Extension Headers". + + Erratum ID 2843: This erratum is marked "Rejected". No change + was made. + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 41] + +RFC 8200 IPv6 Specification July 2017 + + +Acknowledgments + + The authors gratefully acknowledge the many helpful suggestions of + the members of the IPng Working Group, the End-to-End Protocols + research group, and the Internet community at large. + + The authors would also like to acknowledge the authors of the + updating RFCs that were incorporated in this document to move the + IPv6 specification to Internet Standard. They are Joe Abley, Shane + Amante, Jari Arkko, Manav Bhatia, Ronald P. Bonica, Scott Bradner, + Brian Carpenter, P.F. Chimento, Marshall Eubanks, Fernando Gont, + James Hoagland, Sheng Jiang, Erik Kline, Suresh Krishnan, Vishwas + Manral, George Neville-Neil, Jarno Rajahalme, Pekka Savola, Magnus + Westerlund, and James Woodyatt. + +Authors' Addresses + + Stephen E. Deering + Retired + Vancouver, British Columbia + Canada + + + Robert M. Hinden + Check Point Software + 959 Skyway Road + San Carlos, CA 94070 + United States of America + + Email: bob.hinden@gmail.com + + + + + + + + + + + + + + + + + + + + + +Deering & Hinden Standards Track [Page 42] + diff --git a/decisiongrounding/scenarios_candidates/rfc_ipv6_supersession/provenance.json b/decisiongrounding/scenarios_candidates/rfc_ipv6_supersession/provenance.json new file mode 100644 index 0000000..91097da --- /dev/null +++ b/decisiongrounding/scenarios_candidates/rfc_ipv6_supersession/provenance.json @@ -0,0 +1,41 @@ +{ + "source": "rfc-editor.org", + "immutable": true, + "rfcs": [ + { + "id": "RFC-2460", + "number": 2460, + "file": "corpus/RFC-2460.md", + "url": "https://www.rfc-editor.org/rfc/rfc2460.txt", + "source_sha256": "e48b421dc2635deef363c91af689576c707abd4ffb1be20ff26c00f785f909f2", + "category": "Standards Track", + "title": "December 1998", + "obsoletes": [ + "RFC-1883" + ], + "replaces": [] + }, + { + "id": "RFC-8200", + "number": 8200, + "file": "corpus/RFC-8200.md", + "url": "https://www.rfc-editor.org/rfc/rfc8200.txt", + "source_sha256": "371ae3f133d562db5d6385e6def4ca9914c4f831be228ea7779fd28799c2f490", + "category": "Standards Track", + "title": "Internet Protocol, Version 6 (IPv6) Specification", + "obsoletes": [ + "RFC-2460" + ], + "replaces": [ + "RFC-2460" + ] + } + ], + "supersedes_edges": [ + { + "source": "RFC-8200", + "type": "supersedes", + "target": "RFC-2460" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/rfc_tcp_supersession/corpus/RFC-0793.md b/decisiongrounding/scenarios_candidates/rfc_tcp_supersession/corpus/RFC-0793.md new file mode 100644 index 0000000..f27cb42 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/rfc_tcp_supersession/corpus/RFC-0793.md @@ -0,0 +1,5283 @@ +--- +schema_version: 1 +id: RFC-0793 +type: decision +tags: [rfc] +--- + +# RFC-0793 — TRANSMISSION CONTROL PROTOCOL + +## Status + +Superseded + +## Context + +Public decision ingested verbatim from the RFC Editor (RFC 793); +source https://www.rfc-editor.org/rfc/rfc793.txt. RFCs are immutable, so the RFC number is the +pin and the upstream sha256 is recorded in provenance.json. Upstream +category: n/a. The RAC envelope sections are derived from +the RFC's own header fields; the authoritative content is the unedited RFC +reproduced under Source Text. + +## Decision + +Defined by the verbatim RFC text under Source Text below. + +## Consequences + +As stated by the upstream RFC; see Source Text. + +## Superseded By + +- RFC-9293 + +## Source Text + + + +RFC: 793 + + + + + + + + TRANSMISSION CONTROL PROTOCOL + + + DARPA INTERNET PROGRAM + + PROTOCOL SPECIFICATION + + + + September 1981 + + + + + + + + + + + + + + prepared for + + Defense Advanced Research Projects Agency + Information Processing Techniques Office + 1400 Wilson Boulevard + Arlington, Virginia 22209 + + + + + + + + by + + Information Sciences Institute + University of Southern California + 4676 Admiralty Way + Marina del Rey, California 90291 + + + +September 1981 + Transmission Control Protocol + + + + TABLE OF CONTENTS + + PREFACE ........................................................ iii + +1. INTRODUCTION ..................................................... 1 + + 1.1 Motivation .................................................... 1 + 1.2 Scope ......................................................... 2 + 1.3 About This Document ........................................... 2 + 1.4 Interfaces .................................................... 3 + 1.5 Operation ..................................................... 3 + +2. PHILOSOPHY ....................................................... 7 + + 2.1 Elements of the Internetwork System ........................... 7 + 2.2 Model of Operation ............................................ 7 + 2.3 The Host Environment .......................................... 8 + 2.4 Interfaces .................................................... 9 + 2.5 Relation to Other Protocols ................................... 9 + 2.6 Reliable Communication ........................................ 9 + 2.7 Connection Establishment and Clearing ........................ 10 + 2.8 Data Communication ........................................... 12 + 2.9 Precedence and Security ...................................... 13 + 2.10 Robustness Principle ......................................... 13 + +3. FUNCTIONAL SPECIFICATION ........................................ 15 + + 3.1 Header Format ................................................ 15 + 3.2 Terminology .................................................. 19 + 3.3 Sequence Numbers ............................................. 24 + 3.4 Establishing a connection .................................... 30 + 3.5 Closing a Connection ......................................... 37 + 3.6 Precedence and Security ...................................... 40 + 3.7 Data Communication ........................................... 40 + 3.8 Interfaces ................................................... 44 + 3.9 Event Processing ............................................. 52 + +GLOSSARY ............................................................ 79 + +REFERENCES .......................................................... 85 + + + + + + + + + + + + [Page i] + + + September 1981 +Transmission Control Protocol + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +[Page ii] + + +September 1981 + Transmission Control Protocol + + + + PREFACE + + + +This document describes the DoD Standard Transmission Control Protocol +(TCP). There have been nine earlier editions of the ARPA TCP +specification on which this standard is based, and the present text +draws heavily from them. There have been many contributors to this work +both in terms of concepts and in terms of text. This edition clarifies +several details and removes the end-of-letter buffer-size adjustments, +and redescribes the letter mechanism as a push function. + + Jon Postel + + Editor + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + [Page iii] + + + + +RFC: 793 +Replaces: RFC 761 +IENs: 129, 124, 112, 81, +55, 44, 40, 27, 21, 5 + + TRANSMISSION CONTROL PROTOCOL + + DARPA INTERNET PROGRAM + PROTOCOL SPECIFICATION + + + + 1. INTRODUCTION + +The Transmission Control Protocol (TCP) is intended for use as a highly +reliable host-to-host protocol between hosts in packet-switched computer +communication networks, and in interconnected systems of such networks. + +This document describes the functions to be performed by the +Transmission Control Protocol, the program that implements it, and its +interface to programs or users that require its services. + +1.1. Motivation + + Computer communication systems are playing an increasingly important + role in military, government, and civilian environments. This + document focuses its attention primarily on military computer + communication requirements, especially robustness in the presence of + communication unreliability and availability in the presence of + congestion, but many of these problems are found in the civilian and + government sector as well. + + As strategic and tactical computer communication networks are + developed and deployed, it is essential to provide means of + interconnecting them and to provide standard interprocess + communication protocols which can support a broad range of + applications. In anticipation of the need for such standards, the + Deputy Undersecretary of Defense for Research and Engineering has + declared the Transmission Control Protocol (TCP) described herein to + be a basis for DoD-wide inter-process communication protocol + standardization. + + TCP is a connection-oriented, end-to-end reliable protocol designed to + fit into a layered hierarchy of protocols which support multi-network + applications. The TCP provides for reliable inter-process + communication between pairs of processes in host computers attached to + distinct but interconnected computer communication networks. Very few + assumptions are made as to the reliability of the communication + protocols below the TCP layer. TCP assumes it can obtain a simple, + potentially unreliable datagram service from the lower level + protocols. In principle, the TCP should be able to operate above a + wide spectrum of communication systems ranging from hard-wired + connections to packet-switched or circuit-switched networks. + + + [Page 1] + + + September 1981 +Transmission Control Protocol +Introduction + + + + TCP is based on concepts first described by Cerf and Kahn in [1]. The + TCP fits into a layered protocol architecture just above a basic + Internet Protocol [2] which provides a way for the TCP to send and + receive variable-length segments of information enclosed in internet + datagram "envelopes". The internet datagram provides a means for + addressing source and destination TCPs in different networks. The + internet protocol also deals with any fragmentation or reassembly of + the TCP segments required to achieve transport and delivery through + multiple networks and interconnecting gateways. The internet protocol + also carries information on the precedence, security classification + and compartmentation of the TCP segments, so this information can be + communicated end-to-end across multiple networks. + + Protocol Layering + + +---------------------+ + | higher-level | + +---------------------+ + | TCP | + +---------------------+ + | internet protocol | + +---------------------+ + |communication network| + +---------------------+ + + Figure 1 + + Much of this document is written in the context of TCP implementations + which are co-resident with higher level protocols in the host + computer. Some computer systems will be connected to networks via + front-end computers which house the TCP and internet protocol layers, + as well as network specific software. The TCP specification describes + an interface to the higher level protocols which appears to be + implementable even for the front-end case, as long as a suitable + host-to-front end protocol is implemented. + +1.2. Scope + + The TCP is intended to provide a reliable process-to-process + communication service in a multinetwork environment. The TCP is + intended to be a host-to-host protocol in common use in multiple + networks. + +1.3. About this Document + + This document represents a specification of the behavior required of + any TCP implementation, both in its interactions with higher level + protocols and in its interactions with other TCPs. The rest of this + + +[Page 2] + + +September 1981 + Transmission Control Protocol + Introduction + + + + section offers a very brief view of the protocol interfaces and + operation. Section 2 summarizes the philosophical basis for the TCP + design. Section 3 offers both a detailed description of the actions + required of TCP when various events occur (arrival of new segments, + user calls, errors, etc.) and the details of the formats of TCP + segments. + +1.4. Interfaces + + The TCP interfaces on one side to user or application processes and on + the other side to a lower level protocol such as Internet Protocol. + + The interface between an application process and the TCP is + illustrated in reasonable detail. This interface consists of a set of + calls much like the calls an operating system provides to an + application process for manipulating files. For example, there are + calls to open and close connections and to send and receive data on + established connections. It is also expected that the TCP can + asynchronously communicate with application programs. Although + considerable freedom is permitted to TCP implementors to design + interfaces which are appropriate to a particular operating system + environment, a minimum functionality is required at the TCP/user + interface for any valid implementation. + + The interface between TCP and lower level protocol is essentially + unspecified except that it is assumed there is a mechanism whereby the + two levels can asynchronously pass information to each other. + Typically, one expects the lower level protocol to specify this + interface. TCP is designed to work in a very general environment of + interconnected networks. The lower level protocol which is assumed + throughout this document is the Internet Protocol [2]. + +1.5. Operation + + As noted above, the primary purpose of the TCP is to provide reliable, + securable logical circuit or connection service between pairs of + processes. To provide this service on top of a less reliable internet + communication system requires facilities in the following areas: + + Basic Data Transfer + Reliability + Flow Control + Multiplexing + Connections + Precedence and Security + + The basic operation of the TCP in each of these areas is described in + the following paragraphs. + + + [Page 3] + + + September 1981 +Transmission Control Protocol +Introduction + + + + Basic Data Transfer: + + The TCP is able to transfer a continuous stream of octets in each + direction between its users by packaging some number of octets into + segments for transmission through the internet system. In general, + the TCPs decide when to block and forward data at their own + convenience. + + Sometimes users need to be sure that all the data they have + submitted to the TCP has been transmitted. For this purpose a push + function is defined. To assure that data submitted to a TCP is + actually transmitted the sending user indicates that it should be + pushed through to the receiving user. A push causes the TCPs to + promptly forward and deliver data up to that point to the receiver. + The exact push point might not be visible to the receiving user and + the push function does not supply a record boundary marker. + + Reliability: + + The TCP must recover from data that is damaged, lost, duplicated, or + delivered out of order by the internet communication system. This + is achieved by assigning a sequence number to each octet + transmitted, and requiring a positive acknowledgment (ACK) from the + receiving TCP. If the ACK is not received within a timeout + interval, the data is retransmitted. At the receiver, the sequence + numbers are used to correctly order segments that may be received + out of order and to eliminate duplicates. Damage is handled by + adding a checksum to each segment transmitted, checking it at the + receiver, and discarding damaged segments. + + As long as the TCPs continue to function properly and the internet + system does not become completely partitioned, no transmission + errors will affect the correct delivery of data. TCP recovers from + internet communication system errors. + + Flow Control: + + TCP provides a means for the receiver to govern the amount of data + sent by the sender. This is achieved by returning a "window" with + every ACK indicating a range of acceptable sequence numbers beyond + the last segment successfully received. The window indicates an + allowed number of octets that the sender may transmit before + receiving further permission. + + + + + + + +[Page 4] + + +September 1981 + Transmission Control Protocol + Introduction + + + + Multiplexing: + + To allow for many processes within a single Host to use TCP + communication facilities simultaneously, the TCP provides a set of + addresses or ports within each host. Concatenated with the network + and host addresses from the internet communication layer, this forms + a socket. A pair of sockets uniquely identifies each connection. + That is, a socket may be simultaneously used in multiple + connections. + + The binding of ports to processes is handled independently by each + Host. However, it proves useful to attach frequently used processes + (e.g., a "logger" or timesharing service) to fixed sockets which are + made known to the public. These services can then be accessed + through the known addresses. Establishing and learning the port + addresses of other processes may involve more dynamic mechanisms. + + Connections: + + The reliability and flow control mechanisms described above require + that TCPs initialize and maintain certain status information for + each data stream. The combination of this information, including + sockets, sequence numbers, and window sizes, is called a connection. + Each connection is uniquely specified by a pair of sockets + identifying its two sides. + + When two processes wish to communicate, their TCP's must first + establish a connection (initialize the status information on each + side). When their communication is complete, the connection is + terminated or closed to free the resources for other uses. + + Since connections must be established between unreliable hosts and + over the unreliable internet communication system, a handshake + mechanism with clock-based sequence numbers is used to avoid + erroneous initialization of connections. + + Precedence and Security: + + The users of TCP may indicate the security and precedence of their + communication. Provision is made for default values to be used when + these features are not needed. + + + + + + + + + + [Page 5] + + + September 1981 +Transmission Control Protocol + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +[Page 6] + + +September 1981 + Transmission Control Protocol + + + + 2. PHILOSOPHY + +2.1. Elements of the Internetwork System + + The internetwork environment consists of hosts connected to networks + which are in turn interconnected via gateways. It is assumed here + that the networks may be either local networks (e.g., the ETHERNET) or + large networks (e.g., the ARPANET), but in any case are based on + packet switching technology. The active agents that produce and + consume messages are processes. Various levels of protocols in the + networks, the gateways, and the hosts support an interprocess + communication system that provides two-way data flow on logical + connections between process ports. + + The term packet is used generically here to mean the data of one + transaction between a host and its network. The format of data blocks + exchanged within the a network will generally not be of concern to us. + + Hosts are computers attached to a network, and from the communication + network's point of view, are the sources and destinations of packets. + Processes are viewed as the active elements in host computers (in + accordance with the fairly common definition of a process as a program + in execution). Even terminals and files or other I/O devices are + viewed as communicating with each other through the use of processes. + Thus, all communication is viewed as inter-process communication. + + Since a process may need to distinguish among several communication + streams between itself and another process (or processes), we imagine + that each process may have a number of ports through which it + communicates with the ports of other processes. + +2.2. Model of Operation + + Processes transmit data by calling on the TCP and passing buffers of + data as arguments. The TCP packages the data from these buffers into + segments and calls on the internet module to transmit each segment to + the destination TCP. The receiving TCP places the data from a segment + into the receiving user's buffer and notifies the receiving user. The + TCPs include control information in the segments which they use to + ensure reliable ordered data transmission. + + The model of internet communication is that there is an internet + protocol module associated with each TCP which provides an interface + to the local network. This internet module packages TCP segments + inside internet datagrams and routes these datagrams to a destination + internet module or intermediate gateway. To transmit the datagram + through the local network, it is embedded in a local network packet. + + The packet switches may perform further packaging, fragmentation, or + + + [Page 7] + + + September 1981 +Transmission Control Protocol +Philosophy + + + + other operations to achieve the delivery of the local packet to the + destination internet module. + + At a gateway between networks, the internet datagram is "unwrapped" + from its local packet and examined to determine through which network + the internet datagram should travel next. The internet datagram is + then "wrapped" in a local packet suitable to the next network and + routed to the next gateway, or to the final destination. + + A gateway is permitted to break up an internet datagram into smaller + internet datagram fragments if this is necessary for transmission + through the next network. To do this, the gateway produces a set of + internet datagrams; each carrying a fragment. Fragments may be + further broken into smaller fragments at subsequent gateways. The + internet datagram fragment format is designed so that the destination + internet module can reassemble fragments into internet datagrams. + + A destination internet module unwraps the segment from the datagram + (after reassembling the datagram, if necessary) and passes it to the + destination TCP. + + This simple model of the operation glosses over many details. One + important feature is the type of service. This provides information + to the gateway (or internet module) to guide it in selecting the + service parameters to be used in traversing the next network. + Included in the type of service information is the precedence of the + datagram. Datagrams may also carry security information to permit + host and gateways that operate in multilevel secure environments to + properly segregate datagrams for security considerations. + +2.3. The Host Environment + + The TCP is assumed to be a module in an operating system. The users + access the TCP much like they would access the file system. The TCP + may call on other operating system functions, for example, to manage + data structures. The actual interface to the network is assumed to be + controlled by a device driver module. The TCP does not call on the + network device driver directly, but rather calls on the internet + datagram protocol module which may in turn call on the device driver. + + The mechanisms of TCP do not preclude implementation of the TCP in a + front-end processor. However, in such an implementation, a + host-to-front-end protocol must provide the functionality to support + the type of TCP-user interface described in this document. + + + + + + +[Page 8] + + +September 1981 + Transmission Control Protocol + Philosophy + + + +2.4. Interfaces + + The TCP/user interface provides for calls made by the user on the TCP + to OPEN or CLOSE a connection, to SEND or RECEIVE data, or to obtain + STATUS about a connection. These calls are like other calls from user + programs on the operating system, for example, the calls to open, read + from, and close a file. + + The TCP/internet interface provides calls to send and receive + datagrams addressed to TCP modules in hosts anywhere in the internet + system. These calls have parameters for passing the address, type of + service, precedence, security, and other control information. + +2.5. Relation to Other Protocols + + The following diagram illustrates the place of the TCP in the protocol + hierarchy: + + + +------+ +-----+ +-----+ +-----+ + |Telnet| | FTP | |Voice| ... | | Application Level + +------+ +-----+ +-----+ +-----+ + | | | | + +-----+ +-----+ +-----+ + | TCP | | RTP | ... | | Host Level + +-----+ +-----+ +-----+ + | | | + +-------------------------------+ + | Internet Protocol & ICMP | Gateway Level + +-------------------------------+ + | + +---------------------------+ + | Local Network Protocol | Network Level + +---------------------------+ + + Protocol Relationships + + Figure 2. + + It is expected that the TCP will be able to support higher level + protocols efficiently. It should be easy to interface higher level + protocols like the ARPANET Telnet or AUTODIN II THP to the TCP. + +2.6. Reliable Communication + + A stream of data sent on a TCP connection is delivered reliably and in + order at the destination. + + + + [Page 9] + + + September 1981 +Transmission Control Protocol +Philosophy + + + + Transmission is made reliable via the use of sequence numbers and + acknowledgments. Conceptually, each octet of data is assigned a + sequence number. The sequence number of the first octet of data in a + segment is transmitted with that segment and is called the segment + sequence number. Segments also carry an acknowledgment number which + is the sequence number of the next expected data octet of + transmissions in the reverse direction. When the TCP transmits a + segment containing data, it puts a copy on a retransmission queue and + starts a timer; when the acknowledgment for that data is received, the + segment is deleted from the queue. If the acknowledgment is not + received before the timer runs out, the segment is retransmitted. + + An acknowledgment by TCP does not guarantee that the data has been + delivered to the end user, but only that the receiving TCP has taken + the responsibility to do so. + + To govern the flow of data between TCPs, a flow control mechanism is + employed. The receiving TCP reports a "window" to the sending TCP. + This window specifies the number of octets, starting with the + acknowledgment number, that the receiving TCP is currently prepared to + receive. + +2.7. Connection Establishment and Clearing + + To identify the separate data streams that a TCP may handle, the TCP + provides a port identifier. Since port identifiers are selected + independently by each TCP they might not be unique. To provide for + unique addresses within each TCP, we concatenate an internet address + identifying the TCP with a port identifier to create a socket which + will be unique throughout all networks connected together. + + A connection is fully specified by the pair of sockets at the ends. A + local socket may participate in many connections to different foreign + sockets. A connection can be used to carry data in both directions, + that is, it is "full duplex". + + TCPs are free to associate ports with processes however they choose. + However, several basic concepts are necessary in any implementation. + There must be well-known sockets which the TCP associates only with + the "appropriate" processes by some means. We envision that processes + may "own" ports, and that processes can initiate connections only on + the ports they own. (Means for implementing ownership is a local + issue, but we envision a Request Port user command, or a method of + uniquely allocating a group of ports to a given process, e.g., by + associating the high order bits of a port name with a given process.) + + A connection is specified in the OPEN call by the local port and + foreign socket arguments. In return, the TCP supplies a (short) local + + +[Page 10] + + +September 1981 + Transmission Control Protocol + Philosophy + + + + connection name by which the user refers to the connection in + subsequent calls. There are several things that must be remembered + about a connection. To store this information we imagine that there + is a data structure called a Transmission Control Block (TCB). One + implementation strategy would have the local connection name be a + pointer to the TCB for this connection. The OPEN call also specifies + whether the connection establishment is to be actively pursued, or to + be passively waited for. + + A passive OPEN request means that the process wants to accept incoming + connection requests rather than attempting to initiate a connection. + Often the process requesting a passive OPEN will accept a connection + request from any caller. In this case a foreign socket of all zeros + is used to denote an unspecified socket. Unspecified foreign sockets + are allowed only on passive OPENs. + + A service process that wished to provide services for unknown other + processes would issue a passive OPEN request with an unspecified + foreign socket. Then a connection could be made with any process that + requested a connection to this local socket. It would help if this + local socket were known to be associated with this service. + + Well-known sockets are a convenient mechanism for a priori associating + a socket address with a standard service. For instance, the + "Telnet-Server" process is permanently assigned to a particular + socket, and other sockets are reserved for File Transfer, Remote Job + Entry, Text Generator, Echoer, and Sink processes (the last three + being for test purposes). A socket address might be reserved for + access to a "Look-Up" service which would return the specific socket + at which a newly created service would be provided. The concept of a + well-known socket is part of the TCP specification, but the assignment + of sockets to services is outside this specification. (See [4].) + + Processes can issue passive OPENs and wait for matching active OPENs + from other processes and be informed by the TCP when connections have + been established. Two processes which issue active OPENs to each + other at the same time will be correctly connected. This flexibility + is critical for the support of distributed computing in which + components act asynchronously with respect to each other. + + There are two principal cases for matching the sockets in the local + passive OPENs and an foreign active OPENs. In the first case, the + local passive OPENs has fully specified the foreign socket. In this + case, the match must be exact. In the second case, the local passive + OPENs has left the foreign socket unspecified. In this case, any + foreign socket is acceptable as long as the local sockets match. + Other possibilities include partially restricted matches. + + + + [Page 11] + + + September 1981 +Transmission Control Protocol +Philosophy + + + + If there are several pending passive OPENs (recorded in TCBs) with the + same local socket, an foreign active OPEN will be matched to a TCB + with the specific foreign socket in the foreign active OPEN, if such a + TCB exists, before selecting a TCB with an unspecified foreign socket. + + The procedures to establish connections utilize the synchronize (SYN) + control flag and involves an exchange of three messages. This + exchange has been termed a three-way hand shake [3]. + + A connection is initiated by the rendezvous of an arriving segment + containing a SYN and a waiting TCB entry each created by a user OPEN + command. The matching of local and foreign sockets determines when a + connection has been initiated. The connection becomes "established" + when sequence numbers have been synchronized in both directions. + + The clearing of a connection also involves the exchange of segments, + in this case carrying the FIN control flag. + +2.8. Data Communication + + The data that flows on a connection may be thought of as a stream of + octets. The sending user indicates in each SEND call whether the data + in that call (and any preceeding calls) should be immediately pushed + through to the receiving user by the setting of the PUSH flag. + + A sending TCP is allowed to collect data from the sending user and to + send that data in segments at its own convenience, until the push + function is signaled, then it must send all unsent data. When a + receiving TCP sees the PUSH flag, it must not wait for more data from + the sending TCP before passing the data to the receiving process. + + There is no necessary relationship between push functions and segment + boundaries. The data in any particular segment may be the result of a + single SEND call, in whole or part, or of multiple SEND calls. + + The purpose of push function and the PUSH flag is to push data through + from the sending user to the receiving user. It does not provide a + record service. + + There is a coupling between the push function and the use of buffers + of data that cross the TCP/user interface. Each time a PUSH flag is + associated with data placed into the receiving user's buffer, the + buffer is returned to the user for processing even if the buffer is + not filled. If data arrives that fills the user's buffer before a + PUSH is seen, the data is passed to the user in buffer size units. + + TCP also provides a means to communicate to the receiver of data that + at some point further along in the data stream than the receiver is + + +[Page 12] + + +September 1981 + Transmission Control Protocol + Philosophy + + + + currently reading there is urgent data. TCP does not attempt to + define what the user specifically does upon being notified of pending + urgent data, but the general notion is that the receiving process will + take action to process the urgent data quickly. + +2.9. Precedence and Security + + The TCP makes use of the internet protocol type of service field and + security option to provide precedence and security on a per connection + basis to TCP users. Not all TCP modules will necessarily function in + a multilevel secure environment; some may be limited to unclassified + use only, and others may operate at only one security level and + compartment. Consequently, some TCP implementations and services to + users may be limited to a subset of the multilevel secure case. + + TCP modules which operate in a multilevel secure environment must + properly mark outgoing segments with the security, compartment, and + precedence. Such TCP modules must also provide to their users or + higher level protocols such as Telnet or THP an interface to allow + them to specify the desired security level, compartment, and + precedence of connections. + +2.10. Robustness Principle + + TCP implementations will follow a general principle of robustness: be + conservative in what you do, be liberal in what you accept from + others. + + + + + + + + + + + + + + + + + + + + + + + + [Page 13] + + + September 1981 +Transmission Control Protocol + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +[Page 14] + + +September 1981 + Transmission Control Protocol + + + + 3. FUNCTIONAL SPECIFICATION + +3.1. Header Format + + TCP segments are sent as internet datagrams. The Internet Protocol + header carries several information fields, including the source and + destination host addresses [2]. A TCP header follows the internet + header, supplying information specific to the TCP protocol. This + division allows for the existence of host level protocols other than + TCP. + + TCP Header Format + + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Source Port | Destination Port | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Sequence Number | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Acknowledgment Number | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Data | |U|A|P|R|S|F| | + | Offset| Reserved |R|C|S|S|Y|I| Window | + | | |G|K|H|T|N|N| | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Checksum | Urgent Pointer | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Options | Padding | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | data | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + TCP Header Format + + Note that one tick mark represents one bit position. + + Figure 3. + + Source Port: 16 bits + + The source port number. + + Destination Port: 16 bits + + The destination port number. + + + + + [Page 15] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + Sequence Number: 32 bits + + The sequence number of the first data octet in this segment (except + when SYN is present). If SYN is present the sequence number is the + initial sequence number (ISN) and the first data octet is ISN+1. + + Acknowledgment Number: 32 bits + + If the ACK control bit is set this field contains the value of the + next sequence number the sender of the segment is expecting to + receive. Once a connection is established this is always sent. + + Data Offset: 4 bits + + The number of 32 bit words in the TCP Header. This indicates where + the data begins. The TCP header (even one including options) is an + integral number of 32 bits long. + + Reserved: 6 bits + + Reserved for future use. Must be zero. + + Control Bits: 6 bits (from left to right): + + URG: Urgent Pointer field significant + ACK: Acknowledgment field significant + PSH: Push Function + RST: Reset the connection + SYN: Synchronize sequence numbers + FIN: No more data from sender + + Window: 16 bits + + The number of data octets beginning with the one indicated in the + acknowledgment field which the sender of this segment is willing to + accept. + + Checksum: 16 bits + + The checksum field is the 16 bit one's complement of the one's + complement sum of all 16 bit words in the header and text. If a + segment contains an odd number of header and text octets to be + checksummed, the last octet is padded on the right with zeros to + form a 16 bit word for checksum purposes. The pad is not + transmitted as part of the segment. While computing the checksum, + the checksum field itself is replaced with zeros. + + The checksum also covers a 96 bit pseudo header conceptually + + +[Page 16] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + prefixed to the TCP header. This pseudo header contains the Source + Address, the Destination Address, the Protocol, and TCP length. + This gives the TCP protection against misrouted segments. This + information is carried in the Internet Protocol and is transferred + across the TCP/Network interface in the arguments or results of + calls by the TCP on the IP. + + +--------+--------+--------+--------+ + | Source Address | + +--------+--------+--------+--------+ + | Destination Address | + +--------+--------+--------+--------+ + | zero | PTCL | TCP Length | + +--------+--------+--------+--------+ + + The TCP Length is the TCP header length plus the data length in + octets (this is not an explicitly transmitted quantity, but is + computed), and it does not count the 12 octets of the pseudo + header. + + Urgent Pointer: 16 bits + + This field communicates the current value of the urgent pointer as a + positive offset from the sequence number in this segment. The + urgent pointer points to the sequence number of the octet following + the urgent data. This field is only be interpreted in segments with + the URG control bit set. + + Options: variable + + Options may occupy space at the end of the TCP header and are a + multiple of 8 bits in length. All options are included in the + checksum. An option may begin on any octet boundary. There are two + cases for the format of an option: + + Case 1: A single octet of option-kind. + + Case 2: An octet of option-kind, an octet of option-length, and + the actual option-data octets. + + The option-length counts the two octets of option-kind and + option-length as well as the option-data octets. + + Note that the list of options may be shorter than the data offset + field might imply. The content of the header beyond the + End-of-Option option must be header padding (i.e., zero). + + A TCP must implement all options. + + + [Page 17] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + Currently defined options include (kind indicated in octal): + + Kind Length Meaning + ---- ------ ------- + 0 - End of option list. + 1 - No-Operation. + 2 4 Maximum Segment Size. + + + Specific Option Definitions + + End of Option List + + +--------+ + |00000000| + +--------+ + Kind=0 + + This option code indicates the end of the option list. This + might not coincide with the end of the TCP header according to + the Data Offset field. This is used at the end of all options, + not the end of each option, and need only be used if the end of + the options would not otherwise coincide with the end of the TCP + header. + + No-Operation + + +--------+ + |00000001| + +--------+ + Kind=1 + + This option code may be used between options, for example, to + align the beginning of a subsequent option on a word boundary. + There is no guarantee that senders will use this option, so + receivers must be prepared to process options even if they do + not begin on a word boundary. + + Maximum Segment Size + + +--------+--------+---------+--------+ + |00000010|00000100| max seg size | + +--------+--------+---------+--------+ + Kind=2 Length=4 + + + + + + +[Page 18] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + Maximum Segment Size Option Data: 16 bits + + If this option is present, then it communicates the maximum + receive segment size at the TCP which sends this segment. + This field must only be sent in the initial connection request + (i.e., in segments with the SYN control bit set). If this + option is not used, any segment size is allowed. + + Padding: variable + + The TCP header padding is used to ensure that the TCP header ends + and data begins on a 32 bit boundary. The padding is composed of + zeros. + +3.2. Terminology + + Before we can discuss very much about the operation of the TCP we need + to introduce some detailed terminology. The maintenance of a TCP + connection requires the remembering of several variables. We conceive + of these variables being stored in a connection record called a + Transmission Control Block or TCB. Among the variables stored in the + TCB are the local and remote socket numbers, the security and + precedence of the connection, pointers to the user's send and receive + buffers, pointers to the retransmit queue and to the current segment. + In addition several variables relating to the send and receive + sequence numbers are stored in the TCB. + + Send Sequence Variables + + SND.UNA - send unacknowledged + SND.NXT - send next + SND.WND - send window + SND.UP - send urgent pointer + SND.WL1 - segment sequence number used for last window update + SND.WL2 - segment acknowledgment number used for last window + update + ISS - initial send sequence number + + Receive Sequence Variables + + RCV.NXT - receive next + RCV.WND - receive window + RCV.UP - receive urgent pointer + IRS - initial receive sequence number + + + + + + + [Page 19] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + The following diagrams may help to relate some of these variables to + the sequence space. + + Send Sequence Space + + 1 2 3 4 + ----------|----------|----------|---------- + SND.UNA SND.NXT SND.UNA + +SND.WND + + 1 - old sequence numbers which have been acknowledged + 2 - sequence numbers of unacknowledged data + 3 - sequence numbers allowed for new data transmission + 4 - future sequence numbers which are not yet allowed + + Send Sequence Space + + Figure 4. + + + + The send window is the portion of the sequence space labeled 3 in + figure 4. + + Receive Sequence Space + + 1 2 3 + ----------|----------|---------- + RCV.NXT RCV.NXT + +RCV.WND + + 1 - old sequence numbers which have been acknowledged + 2 - sequence numbers allowed for new reception + 3 - future sequence numbers which are not yet allowed + + Receive Sequence Space + + Figure 5. + + + + The receive window is the portion of the sequence space labeled 2 in + figure 5. + + There are also some variables used frequently in the discussion that + take their values from the fields of the current segment. + + + + +[Page 20] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + Current Segment Variables + + SEG.SEQ - segment sequence number + SEG.ACK - segment acknowledgment number + SEG.LEN - segment length + SEG.WND - segment window + SEG.UP - segment urgent pointer + SEG.PRC - segment precedence value + + A connection progresses through a series of states during its + lifetime. The states are: LISTEN, SYN-SENT, SYN-RECEIVED, + ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, + TIME-WAIT, and the fictional state CLOSED. CLOSED is fictional + because it represents the state when there is no TCB, and therefore, + no connection. Briefly the meanings of the states are: + + LISTEN - represents waiting for a connection request from any remote + TCP and port. + + SYN-SENT - represents waiting for a matching connection request + after having sent a connection request. + + SYN-RECEIVED - represents waiting for a confirming connection + request acknowledgment after having both received and sent a + connection request. + + ESTABLISHED - represents an open connection, data received can be + delivered to the user. The normal state for the data transfer phase + of the connection. + + FIN-WAIT-1 - represents waiting for a connection termination request + from the remote TCP, or an acknowledgment of the connection + termination request previously sent. + + FIN-WAIT-2 - represents waiting for a connection termination request + from the remote TCP. + + CLOSE-WAIT - represents waiting for a connection termination request + from the local user. + + CLOSING - represents waiting for a connection termination request + acknowledgment from the remote TCP. + + LAST-ACK - represents waiting for an acknowledgment of the + connection termination request previously sent to the remote TCP + (which includes an acknowledgment of its connection termination + request). + + + + [Page 21] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + TIME-WAIT - represents waiting for enough time to pass to be sure + the remote TCP received the acknowledgment of its connection + termination request. + + CLOSED - represents no connection state at all. + + A TCP connection progresses from one state to another in response to + events. The events are the user calls, OPEN, SEND, RECEIVE, CLOSE, + ABORT, and STATUS; the incoming segments, particularly those + containing the SYN, ACK, RST and FIN flags; and timeouts. + + The state diagram in figure 6 illustrates only state changes, together + with the causing events and resulting actions, but addresses neither + error conditions nor actions which are not connected with state + changes. In a later section, more detail is offered with respect to + the reaction of the TCP to events. + + NOTE BENE: this diagram is only a summary and must not be taken as + the total specification. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +[Page 22] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + + +---------+ ---------\ active OPEN + | CLOSED | \ ----------- + +---------+<---------\ \ create TCB + | ^ \ \ snd SYN + passive OPEN | | CLOSE \ \ + ------------ | | ---------- \ \ + create TCB | | delete TCB \ \ + V | \ \ + +---------+ CLOSE | \ + | LISTEN | ---------- | | + +---------+ delete TCB | | + rcv SYN | | SEND | | + ----------- | | ------- | V + +---------+ snd SYN,ACK / \ snd SYN +---------+ + | |<----------------- ------------------>| | + | SYN | rcv SYN | SYN | + | RCVD |<-----------------------------------------------| SENT | + | | snd ACK | | + | |------------------ -------------------| | + +---------+ rcv ACK of SYN \ / rcv SYN,ACK +---------+ + | -------------- | | ----------- + | x | | snd ACK + | V V + | CLOSE +---------+ + | ------- | ESTAB | + | snd FIN +---------+ + | CLOSE | | rcv FIN + V ------- | | ------- + +---------+ snd FIN / \ snd ACK +---------+ + | FIN |<----------------- ------------------>| CLOSE | + | WAIT-1 |------------------ | WAIT | + +---------+ rcv FIN \ +---------+ + | rcv ACK of FIN ------- | CLOSE | + | -------------- snd ACK | ------- | + V x V snd FIN V + +---------+ +---------+ +---------+ + |FINWAIT-2| | CLOSING | | LAST-ACK| + +---------+ +---------+ +---------+ + | rcv ACK of FIN | rcv ACK of FIN | + | rcv FIN -------------- | Timeout=2MSL -------------- | + | ------- x V ------------ x V + \ snd ACK +---------+delete TCB +---------+ + ------------------------>|TIME WAIT|------------------>| CLOSED | + +---------+ +---------+ + + TCP Connection State Diagram + Figure 6. + + + [Page 23] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + +3.3. Sequence Numbers + + A fundamental notion in the design is that every octet of data sent + over a TCP connection has a sequence number. Since every octet is + sequenced, each of them can be acknowledged. The acknowledgment + mechanism employed is cumulative so that an acknowledgment of sequence + number X indicates that all octets up to but not including X have been + received. This mechanism allows for straight-forward duplicate + detection in the presence of retransmission. Numbering of octets + within a segment is that the first data octet immediately following + the header is the lowest numbered, and the following octets are + numbered consecutively. + + It is essential to remember that the actual sequence number space is + finite, though very large. This space ranges from 0 to 2**32 - 1. + Since the space is finite, all arithmetic dealing with sequence + numbers must be performed modulo 2**32. This unsigned arithmetic + preserves the relationship of sequence numbers as they cycle from + 2**32 - 1 to 0 again. There are some subtleties to computer modulo + arithmetic, so great care should be taken in programming the + comparison of such values. The symbol "=<" means "less than or equal" + (modulo 2**32). + + The typical kinds of sequence number comparisons which the TCP must + perform include: + + (a) Determining that an acknowledgment refers to some sequence + number sent but not yet acknowledged. + + (b) Determining that all sequence numbers occupied by a segment + have been acknowledged (e.g., to remove the segment from a + retransmission queue). + + (c) Determining that an incoming segment contains sequence numbers + which are expected (i.e., that the segment "overlaps" the + receive window). + + + + + + + + + + + + + + +[Page 24] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + In response to sending data the TCP will receive acknowledgments. The + following comparisons are needed to process the acknowledgments. + + SND.UNA = oldest unacknowledged sequence number + + SND.NXT = next sequence number to be sent + + SEG.ACK = acknowledgment from the receiving TCP (next sequence + number expected by the receiving TCP) + + SEG.SEQ = first sequence number of a segment + + SEG.LEN = the number of octets occupied by the data in the segment + (counting SYN and FIN) + + SEG.SEQ+SEG.LEN-1 = last sequence number of a segment + + A new acknowledgment (called an "acceptable ack"), is one for which + the inequality below holds: + + SND.UNA < SEG.ACK =< SND.NXT + + A segment on the retransmission queue is fully acknowledged if the sum + of its sequence number and length is less or equal than the + acknowledgment value in the incoming segment. + + When data is received the following comparisons are needed: + + RCV.NXT = next sequence number expected on an incoming segments, and + is the left or lower edge of the receive window + + RCV.NXT+RCV.WND-1 = last sequence number expected on an incoming + segment, and is the right or upper edge of the receive window + + SEG.SEQ = first sequence number occupied by the incoming segment + + SEG.SEQ+SEG.LEN-1 = last sequence number occupied by the incoming + segment + + A segment is judged to occupy a portion of valid receive sequence + space if + + RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND + + or + + RCV.NXT =< SEG.SEQ+SEG.LEN-1 < RCV.NXT+RCV.WND + + + + [Page 25] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + The first part of this test checks to see if the beginning of the + segment falls in the window, the second part of the test checks to see + if the end of the segment falls in the window; if the segment passes + either part of the test it contains data in the window. + + Actually, it is a little more complicated than this. Due to zero + windows and zero length segments, we have four cases for the + acceptability of an incoming segment: + + Segment Receive Test + Length Window + ------- ------- ------------------------------------------- + + 0 0 SEG.SEQ = RCV.NXT + + 0 >0 RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND + + >0 0 not acceptable + + >0 >0 RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND + or RCV.NXT =< SEG.SEQ+SEG.LEN-1 < RCV.NXT+RCV.WND + + Note that when the receive window is zero no segments should be + acceptable except ACK segments. Thus, it is be possible for a TCP to + maintain a zero receive window while transmitting data and receiving + ACKs. However, even when the receive window is zero, a TCP must + process the RST and URG fields of all incoming segments. + + We have taken advantage of the numbering scheme to protect certain + control information as well. This is achieved by implicitly including + some control flags in the sequence space so they can be retransmitted + and acknowledged without confusion (i.e., one and only one copy of the + control will be acted upon). Control information is not physically + carried in the segment data space. Consequently, we must adopt rules + for implicitly assigning sequence numbers to control. The SYN and FIN + are the only controls requiring this protection, and these controls + are used only at connection opening and closing. For sequence number + purposes, the SYN is considered to occur before the first actual data + octet of the segment in which it occurs, while the FIN is considered + to occur after the last actual data octet in a segment in which it + occurs. The segment length (SEG.LEN) includes both data and sequence + space occupying controls. When a SYN is present then SEG.SEQ is the + sequence number of the SYN. + + + + + + + +[Page 26] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + Initial Sequence Number Selection + + The protocol places no restriction on a particular connection being + used over and over again. A connection is defined by a pair of + sockets. New instances of a connection will be referred to as + incarnations of the connection. The problem that arises from this is + -- "how does the TCP identify duplicate segments from previous + incarnations of the connection?" This problem becomes apparent if the + connection is being opened and closed in quick succession, or if the + connection breaks with loss of memory and is then reestablished. + + To avoid confusion we must prevent segments from one incarnation of a + connection from being used while the same sequence numbers may still + be present in the network from an earlier incarnation. We want to + assure this, even if a TCP crashes and loses all knowledge of the + sequence numbers it has been using. When new connections are created, + an initial sequence number (ISN) generator is employed which selects a + new 32 bit ISN. The generator is bound to a (possibly fictitious) 32 + bit clock whose low order bit is incremented roughly every 4 + microseconds. Thus, the ISN cycles approximately every 4.55 hours. + Since we assume that segments will stay in the network no more than + the Maximum Segment Lifetime (MSL) and that the MSL is less than 4.55 + hours we can reasonably assume that ISN's will be unique. + + For each connection there is a send sequence number and a receive + sequence number. The initial send sequence number (ISS) is chosen by + the data sending TCP, and the initial receive sequence number (IRS) is + learned during the connection establishing procedure. + + For a connection to be established or initialized, the two TCPs must + synchronize on each other's initial sequence numbers. This is done in + an exchange of connection establishing segments carrying a control bit + called "SYN" (for synchronize) and the initial sequence numbers. As a + shorthand, segments carrying the SYN bit are also called "SYNs". + Hence, the solution requires a suitable mechanism for picking an + initial sequence number and a slightly involved handshake to exchange + the ISN's. + + The synchronization requires each side to send it's own initial + sequence number and to receive a confirmation of it in acknowledgment + from the other side. Each side must also receive the other side's + initial sequence number and send a confirming acknowledgment. + + 1) A --> B SYN my sequence number is X + 2) A <-- B ACK your sequence number is X + 3) A <-- B SYN my sequence number is Y + 4) A --> B ACK your sequence number is Y + + + + [Page 27] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + Because steps 2 and 3 can be combined in a single message this is + called the three way (or three message) handshake. + + A three way handshake is necessary because sequence numbers are not + tied to a global clock in the network, and TCPs may have different + mechanisms for picking the ISN's. The receiver of the first SYN has + no way of knowing whether the segment was an old delayed one or not, + unless it remembers the last sequence number used on the connection + (which is not always possible), and so it must ask the sender to + verify this SYN. The three way handshake and the advantages of a + clock-driven scheme are discussed in [3]. + + Knowing When to Keep Quiet + + To be sure that a TCP does not create a segment that carries a + sequence number which may be duplicated by an old segment remaining in + the network, the TCP must keep quiet for a maximum segment lifetime + (MSL) before assigning any sequence numbers upon starting up or + recovering from a crash in which memory of sequence numbers in use was + lost. For this specification the MSL is taken to be 2 minutes. This + is an engineering choice, and may be changed if experience indicates + it is desirable to do so. Note that if a TCP is reinitialized in some + sense, yet retains its memory of sequence numbers in use, then it need + not wait at all; it must only be sure to use sequence numbers larger + than those recently used. + + The TCP Quiet Time Concept + + This specification provides that hosts which "crash" without + retaining any knowledge of the last sequence numbers transmitted on + each active (i.e., not closed) connection shall delay emitting any + TCP segments for at least the agreed Maximum Segment Lifetime (MSL) + in the internet system of which the host is a part. In the + paragraphs below, an explanation for this specification is given. + TCP implementors may violate the "quiet time" restriction, but only + at the risk of causing some old data to be accepted as new or new + data rejected as old duplicated by some receivers in the internet + system. + + TCPs consume sequence number space each time a segment is formed and + entered into the network output queue at a source host. The + duplicate detection and sequencing algorithm in the TCP protocol + relies on the unique binding of segment data to sequence space to + the extent that sequence numbers will not cycle through all 2**32 + values before the segment data bound to those sequence numbers has + been delivered and acknowledged by the receiver and all duplicate + copies of the segments have "drained" from the internet. Without + such an assumption, two distinct TCP segments could conceivably be + + +[Page 28] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + assigned the same or overlapping sequence numbers, causing confusion + at the receiver as to which data is new and which is old. Remember + that each segment is bound to as many consecutive sequence numbers + as there are octets of data in the segment. + + Under normal conditions, TCPs keep track of the next sequence number + to emit and the oldest awaiting acknowledgment so as to avoid + mistakenly using a sequence number over before its first use has + been acknowledged. This alone does not guarantee that old duplicate + data is drained from the net, so the sequence space has been made + very large to reduce the probability that a wandering duplicate will + cause trouble upon arrival. At 2 megabits/sec. it takes 4.5 hours + to use up 2**32 octets of sequence space. Since the maximum segment + lifetime in the net is not likely to exceed a few tens of seconds, + this is deemed ample protection for foreseeable nets, even if data + rates escalate to l0's of megabits/sec. At 100 megabits/sec, the + cycle time is 5.4 minutes which may be a little short, but still + within reason. + + The basic duplicate detection and sequencing algorithm in TCP can be + defeated, however, if a source TCP does not have any memory of the + sequence numbers it last used on a given connection. For example, if + the TCP were to start all connections with sequence number 0, then + upon crashing and restarting, a TCP might re-form an earlier + connection (possibly after half-open connection resolution) and emit + packets with sequence numbers identical to or overlapping with + packets still in the network which were emitted on an earlier + incarnation of the same connection. In the absence of knowledge + about the sequence numbers used on a particular connection, the TCP + specification recommends that the source delay for MSL seconds + before emitting segments on the connection, to allow time for + segments from the earlier connection incarnation to drain from the + system. + + Even hosts which can remember the time of day and used it to select + initial sequence number values are not immune from this problem + (i.e., even if time of day is used to select an initial sequence + number for each new connection incarnation). + + Suppose, for example, that a connection is opened starting with + sequence number S. Suppose that this connection is not used much + and that eventually the initial sequence number function (ISN(t)) + takes on a value equal to the sequence number, say S1, of the last + segment sent by this TCP on a particular connection. Now suppose, + at this instant, the host crashes, recovers, and establishes a new + incarnation of the connection. The initial sequence number chosen is + S1 = ISN(t) -- last used sequence number on old incarnation of + connection! If the recovery occurs quickly enough, any old + + + [Page 29] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + duplicates in the net bearing sequence numbers in the neighborhood + of S1 may arrive and be treated as new packets by the receiver of + the new incarnation of the connection. + + The problem is that the recovering host may not know for how long it + crashed nor does it know whether there are still old duplicates in + the system from earlier connection incarnations. + + One way to deal with this problem is to deliberately delay emitting + segments for one MSL after recovery from a crash- this is the "quite + time" specification. Hosts which prefer to avoid waiting are + willing to risk possible confusion of old and new packets at a given + destination may choose not to wait for the "quite time". + Implementors may provide TCP users with the ability to select on a + connection by connection basis whether to wait after a crash, or may + informally implement the "quite time" for all connections. + Obviously, even where a user selects to "wait," this is not + necessary after the host has been "up" for at least MSL seconds. + + To summarize: every segment emitted occupies one or more sequence + numbers in the sequence space, the numbers occupied by a segment are + "busy" or "in use" until MSL seconds have passed, upon crashing a + block of space-time is occupied by the octets of the last emitted + segment, if a new connection is started too soon and uses any of the + sequence numbers in the space-time footprint of the last segment of + the previous connection incarnation, there is a potential sequence + number overlap area which could cause confusion at the receiver. + +3.4. Establishing a connection + + The "three-way handshake" is the procedure used to establish a + connection. This procedure normally is initiated by one TCP and + responded to by another TCP. The procedure also works if two TCP + simultaneously initiate the procedure. When simultaneous attempt + occurs, each TCP receives a "SYN" segment which carries no + acknowledgment after it has sent a "SYN". Of course, the arrival of + an old duplicate "SYN" segment can potentially make it appear, to the + recipient, that a simultaneous connection initiation is in progress. + Proper use of "reset" segments can disambiguate these cases. + + Several examples of connection initiation follow. Although these + examples do not show connection synchronization using data-carrying + segments, this is perfectly legitimate, so long as the receiving TCP + doesn't deliver the data to the user until it is clear the data is + valid (i.e., the data must be buffered at the receiver until the + connection reaches the ESTABLISHED state). The three-way handshake + reduces the possibility of false connections. It is the + + + +[Page 30] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + implementation of a trade-off between memory and messages to provide + information for this checking. + + The simplest three-way handshake is shown in figure 7 below. The + figures should be interpreted in the following way. Each line is + numbered for reference purposes. Right arrows (-->) indicate + departure of a TCP segment from TCP A to TCP B, or arrival of a + segment at B from A. Left arrows (<--), indicate the reverse. + Ellipsis (...) indicates a segment which is still in the network + (delayed). An "XXX" indicates a segment which is lost or rejected. + Comments appear in parentheses. TCP states represent the state AFTER + the departure or arrival of the segment (whose contents are shown in + the center of each line). Segment contents are shown in abbreviated + form, with sequence number, control flags, and ACK field. Other + fields such as window, addresses, lengths, and text have been left out + in the interest of clarity. + + + + TCP A TCP B + + 1. CLOSED LISTEN + + 2. SYN-SENT --> --> SYN-RECEIVED + + 3. ESTABLISHED <-- <-- SYN-RECEIVED + + 4. ESTABLISHED --> --> ESTABLISHED + + 5. ESTABLISHED --> --> ESTABLISHED + + Basic 3-Way Handshake for Connection Synchronization + + Figure 7. + + In line 2 of figure 7, TCP A begins by sending a SYN segment + indicating that it will use sequence numbers starting with sequence + number 100. In line 3, TCP B sends a SYN and acknowledges the SYN it + received from TCP A. Note that the acknowledgment field indicates TCP + B is now expecting to hear sequence 101, acknowledging the SYN which + occupied sequence 100. + + At line 4, TCP A responds with an empty segment containing an ACK for + TCP B's SYN; and in line 5, TCP A sends some data. Note that the + sequence number of the segment in line 5 is the same as in line 4 + because the ACK does not occupy sequence number space (if it did, we + would wind up ACKing ACK's!). + + + + [Page 31] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + Simultaneous initiation is only slightly more complex, as is shown in + figure 8. Each TCP cycles from CLOSED to SYN-SENT to SYN-RECEIVED to + ESTABLISHED. + + + + TCP A TCP B + + 1. CLOSED CLOSED + + 2. SYN-SENT --> ... + + 3. SYN-RECEIVED <-- <-- SYN-SENT + + 4. ... --> SYN-RECEIVED + + 5. SYN-RECEIVED --> ... + + 6. ESTABLISHED <-- <-- SYN-RECEIVED + + 7. ... --> ESTABLISHED + + Simultaneous Connection Synchronization + + Figure 8. + + The principle reason for the three-way handshake is to prevent old + duplicate connection initiations from causing confusion. To deal with + this, a special control message, reset, has been devised. If the + receiving TCP is in a non-synchronized state (i.e., SYN-SENT, + SYN-RECEIVED), it returns to LISTEN on receiving an acceptable reset. + If the TCP is in one of the synchronized states (ESTABLISHED, + FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT), it + aborts the connection and informs its user. We discuss this latter + case under "half-open" connections below. + + + + + + + + + + + + + + + +[Page 32] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + + + TCP A TCP B + + 1. CLOSED LISTEN + + 2. SYN-SENT --> ... + + 3. (duplicate) ... --> SYN-RECEIVED + + 4. SYN-SENT <-- <-- SYN-RECEIVED + + 5. SYN-SENT --> --> LISTEN + + + 6. ... --> SYN-RECEIVED + + 7. SYN-SENT <-- <-- SYN-RECEIVED + + 8. ESTABLISHED --> --> ESTABLISHED + + Recovery from Old Duplicate SYN + + Figure 9. + + As a simple example of recovery from old duplicates, consider + figure 9. At line 3, an old duplicate SYN arrives at TCP B. TCP B + cannot tell that this is an old duplicate, so it responds normally + (line 4). TCP A detects that the ACK field is incorrect and returns a + RST (reset) with its SEQ field selected to make the segment + believable. TCP B, on receiving the RST, returns to the LISTEN state. + When the original SYN (pun intended) finally arrives at line 6, the + synchronization proceeds normally. If the SYN at line 6 had arrived + before the RST, a more complex exchange might have occurred with RST's + sent in both directions. + + Half-Open Connections and Other Anomalies + + An established connection is said to be "half-open" if one of the + TCPs has closed or aborted the connection at its end without the + knowledge of the other, or if the two ends of the connection have + become desynchronized owing to a crash that resulted in loss of + memory. Such connections will automatically become reset if an + attempt is made to send data in either direction. However, half-open + connections are expected to be unusual, and the recovery procedure is + mildly involved. + + If at site A the connection no longer exists, then an attempt by the + + + [Page 33] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + user at site B to send any data on it will result in the site B TCP + receiving a reset control message. Such a message indicates to the + site B TCP that something is wrong, and it is expected to abort the + connection. + + Assume that two user processes A and B are communicating with one + another when a crash occurs causing loss of memory to A's TCP. + Depending on the operating system supporting A's TCP, it is likely + that some error recovery mechanism exists. When the TCP is up again, + A is likely to start again from the beginning or from a recovery + point. As a result, A will probably try to OPEN the connection again + or try to SEND on the connection it believes open. In the latter + case, it receives the error message "connection not open" from the + local (A's) TCP. In an attempt to establish the connection, A's TCP + will send a segment containing SYN. This scenario leads to the + example shown in figure 10. After TCP A crashes, the user attempts to + re-open the connection. TCP B, in the meantime, thinks the connection + is open. + + + + TCP A TCP B + + 1. (CRASH) (send 300,receive 100) + + 2. CLOSED ESTABLISHED + + 3. SYN-SENT --> --> (??) + + 4. (!!) <-- <-- ESTABLISHED + + 5. SYN-SENT --> --> (Abort!!) + + 6. SYN-SENT CLOSED + + 7. SYN-SENT --> --> + + Half-Open Connection Discovery + + Figure 10. + + When the SYN arrives at line 3, TCP B, being in a synchronized state, + and the incoming segment outside the window, responds with an + acknowledgment indicating what sequence it next expects to hear (ACK + 100). TCP A sees that this segment does not acknowledge anything it + sent and, being unsynchronized, sends a reset (RST) because it has + detected a half-open connection. TCP B aborts at line 5. TCP A will + + + +[Page 34] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + continue to try to establish the connection; the problem is now + reduced to the basic 3-way handshake of figure 7. + + An interesting alternative case occurs when TCP A crashes and TCP B + tries to send data on what it thinks is a synchronized connection. + This is illustrated in figure 11. In this case, the data arriving at + TCP A from TCP B (line 2) is unacceptable because no such connection + exists, so TCP A sends a RST. The RST is acceptable so TCP B + processes it and aborts the connection. + + + + TCP A TCP B + + 1. (CRASH) (send 300,receive 100) + + 2. (??) <-- <-- ESTABLISHED + + 3. --> --> (ABORT!!) + + Active Side Causes Half-Open Connection Discovery + + Figure 11. + + In figure 12, we find the two TCPs A and B with passive connections + waiting for SYN. An old duplicate arriving at TCP B (line 2) stirs B + into action. A SYN-ACK is returned (line 3) and causes TCP A to + generate a RST (the ACK in line 3 is not acceptable). TCP B accepts + the reset and returns to its passive LISTEN state. + + + + TCP A TCP B + + 1. LISTEN LISTEN + + 2. ... --> SYN-RECEIVED + + 3. (??) <-- <-- SYN-RECEIVED + + 4. --> --> (return to LISTEN!) + + 5. LISTEN LISTEN + + Old Duplicate SYN Initiates a Reset on two Passive Sockets + + Figure 12. + + + + [Page 35] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + A variety of other cases are possible, all of which are accounted for + by the following rules for RST generation and processing. + + Reset Generation + + As a general rule, reset (RST) must be sent whenever a segment arrives + which apparently is not intended for the current connection. A reset + must not be sent if it is not clear that this is the case. + + There are three groups of states: + + 1. If the connection does not exist (CLOSED) then a reset is sent + in response to any incoming segment except another reset. In + particular, SYNs addressed to a non-existent connection are rejected + by this means. + + If the incoming segment has an ACK field, the reset takes its + sequence number from the ACK field of the segment, otherwise the + reset has sequence number zero and the ACK field is set to the sum + of the sequence number and segment length of the incoming segment. + The connection remains in the CLOSED state. + + 2. If the connection is in any non-synchronized state (LISTEN, + SYN-SENT, SYN-RECEIVED), and the incoming segment acknowledges + something not yet sent (the segment carries an unacceptable ACK), or + if an incoming segment has a security level or compartment which + does not exactly match the level and compartment requested for the + connection, a reset is sent. + + If our SYN has not been acknowledged and the precedence level of the + incoming segment is higher than the precedence level requested then + either raise the local precedence level (if allowed by the user and + the system) or send a reset; or if the precedence level of the + incoming segment is lower than the precedence level requested then + continue as if the precedence matched exactly (if the remote TCP + cannot raise the precedence level to match ours this will be + detected in the next segment it sends, and the connection will be + terminated then). If our SYN has been acknowledged (perhaps in this + incoming segment) the precedence level of the incoming segment must + match the local precedence level exactly, if it does not a reset + must be sent. + + If the incoming segment has an ACK field, the reset takes its + sequence number from the ACK field of the segment, otherwise the + reset has sequence number zero and the ACK field is set to the sum + of the sequence number and segment length of the incoming segment. + The connection remains in the same state. + + + +[Page 36] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + 3. If the connection is in a synchronized state (ESTABLISHED, + FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT), + any unacceptable segment (out of window sequence number or + unacceptible acknowledgment number) must elicit only an empty + acknowledgment segment containing the current send-sequence number + and an acknowledgment indicating the next sequence number expected + to be received, and the connection remains in the same state. + + If an incoming segment has a security level, or compartment, or + precedence which does not exactly match the level, and compartment, + and precedence requested for the connection,a reset is sent and + connection goes to the CLOSED state. The reset takes its sequence + number from the ACK field of the incoming segment. + + Reset Processing + + In all states except SYN-SENT, all reset (RST) segments are validated + by checking their SEQ-fields. A reset is valid if its sequence number + is in the window. In the SYN-SENT state (a RST received in response + to an initial SYN), the RST is acceptable if the ACK field + acknowledges the SYN. + + The receiver of a RST first validates it, then changes state. If the + receiver was in the LISTEN state, it ignores it. If the receiver was + in SYN-RECEIVED state and had previously been in the LISTEN state, + then the receiver returns to the LISTEN state, otherwise the receiver + aborts the connection and goes to the CLOSED state. If the receiver + was in any other state, it aborts the connection and advises the user + and goes to the CLOSED state. + +3.5. Closing a Connection + + CLOSE is an operation meaning "I have no more data to send." The + notion of closing a full-duplex connection is subject to ambiguous + interpretation, of course, since it may not be obvious how to treat + the receiving side of the connection. We have chosen to treat CLOSE + in a simplex fashion. The user who CLOSEs may continue to RECEIVE + until he is told that the other side has CLOSED also. Thus, a program + could initiate several SENDs followed by a CLOSE, and then continue to + RECEIVE until signaled that a RECEIVE failed because the other side + has CLOSED. We assume that the TCP will signal a user, even if no + RECEIVEs are outstanding, that the other side has closed, so the user + can terminate his side gracefully. A TCP will reliably deliver all + buffers SENT before the connection was CLOSED so a user who expects no + data in return need only wait to hear the connection was CLOSED + successfully to know that all his data was received at the destination + TCP. Users must keep reading connections they close for sending until + the TCP says no more data. + + + [Page 37] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + There are essentially three cases: + + 1) The user initiates by telling the TCP to CLOSE the connection + + 2) The remote TCP initiates by sending a FIN control signal + + 3) Both users CLOSE simultaneously + + Case 1: Local user initiates the close + + In this case, a FIN segment can be constructed and placed on the + outgoing segment queue. No further SENDs from the user will be + accepted by the TCP, and it enters the FIN-WAIT-1 state. RECEIVEs + are allowed in this state. All segments preceding and including FIN + will be retransmitted until acknowledged. When the other TCP has + both acknowledged the FIN and sent a FIN of its own, the first TCP + can ACK this FIN. Note that a TCP receiving a FIN will ACK but not + send its own FIN until its user has CLOSED the connection also. + + Case 2: TCP receives a FIN from the network + + If an unsolicited FIN arrives from the network, the receiving TCP + can ACK it and tell the user that the connection is closing. The + user will respond with a CLOSE, upon which the TCP can send a FIN to + the other TCP after sending any remaining data. The TCP then waits + until its own FIN is acknowledged whereupon it deletes the + connection. If an ACK is not forthcoming, after the user timeout + the connection is aborted and the user is told. + + Case 3: both users close simultaneously + + A simultaneous CLOSE by users at both ends of a connection causes + FIN segments to be exchanged. When all segments preceding the FINs + have been processed and acknowledged, each TCP can ACK the FIN it + has received. Both will, upon receiving these ACKs, delete the + connection. + + + + + + + + + + + + + + +[Page 38] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + + + TCP A TCP B + + 1. ESTABLISHED ESTABLISHED + + 2. (Close) + FIN-WAIT-1 --> --> CLOSE-WAIT + + 3. FIN-WAIT-2 <-- <-- CLOSE-WAIT + + 4. (Close) + TIME-WAIT <-- <-- LAST-ACK + + 5. TIME-WAIT --> --> CLOSED + + 6. (2 MSL) + CLOSED + + Normal Close Sequence + + Figure 13. + + + + TCP A TCP B + + 1. ESTABLISHED ESTABLISHED + + 2. (Close) (Close) + FIN-WAIT-1 --> ... FIN-WAIT-1 + <-- <-- + ... --> + + 3. CLOSING --> ... CLOSING + <-- <-- + ... --> + + 4. TIME-WAIT TIME-WAIT + (2 MSL) (2 MSL) + CLOSED CLOSED + + Simultaneous Close Sequence + + Figure 14. + + + + + + [Page 39] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + +3.6. Precedence and Security + + The intent is that connection be allowed only between ports operating + with exactly the same security and compartment values and at the + higher of the precedence level requested by the two ports. + + The precedence and security parameters used in TCP are exactly those + defined in the Internet Protocol (IP) [2]. Throughout this TCP + specification the term "security/compartment" is intended to indicate + the security parameters used in IP including security, compartment, + user group, and handling restriction. + + A connection attempt with mismatched security/compartment values or a + lower precedence value must be rejected by sending a reset. Rejecting + a connection due to too low a precedence only occurs after an + acknowledgment of the SYN has been received. + + Note that TCP modules which operate only at the default value of + precedence will still have to check the precedence of incoming + segments and possibly raise the precedence level they use on the + connection. + + The security paramaters may be used even in a non-secure environment + (the values would indicate unclassified data), thus hosts in + non-secure environments must be prepared to receive the security + parameters, though they need not send them. + +3.7. Data Communication + + Once the connection is established data is communicated by the + exchange of segments. Because segments may be lost due to errors + (checksum test failure), or network congestion, TCP uses + retransmission (after a timeout) to ensure delivery of every segment. + Duplicate segments may arrive due to network or TCP retransmission. + As discussed in the section on sequence numbers the TCP performs + certain tests on the sequence and acknowledgment numbers in the + segments to verify their acceptability. + + The sender of data keeps track of the next sequence number to use in + the variable SND.NXT. The receiver of data keeps track of the next + sequence number to expect in the variable RCV.NXT. The sender of data + keeps track of the oldest unacknowledged sequence number in the + variable SND.UNA. If the data flow is momentarily idle and all data + sent has been acknowledged then the three variables will be equal. + + When the sender creates a segment and transmits it the sender advances + SND.NXT. When the receiver accepts a segment it advances RCV.NXT and + sends an acknowledgment. When the data sender receives an + + +[Page 40] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + acknowledgment it advances SND.UNA. The extent to which the values of + these variables differ is a measure of the delay in the communication. + The amount by which the variables are advanced is the length of the + data in the segment. Note that once in the ESTABLISHED state all + segments must carry current acknowledgment information. + + The CLOSE user call implies a push function, as does the FIN control + flag in an incoming segment. + + Retransmission Timeout + + Because of the variability of the networks that compose an + internetwork system and the wide range of uses of TCP connections the + retransmission timeout must be dynamically determined. One procedure + for determining a retransmission time out is given here as an + illustration. + + An Example Retransmission Timeout Procedure + + Measure the elapsed time between sending a data octet with a + particular sequence number and receiving an acknowledgment that + covers that sequence number (segments sent do not have to match + segments received). This measured elapsed time is the Round Trip + Time (RTT). Next compute a Smoothed Round Trip Time (SRTT) as: + + SRTT = ( ALPHA * SRTT ) + ((1-ALPHA) * RTT) + + and based on this, compute the retransmission timeout (RTO) as: + + RTO = min[UBOUND,max[LBOUND,(BETA*SRTT)]] + + where UBOUND is an upper bound on the timeout (e.g., 1 minute), + LBOUND is a lower bound on the timeout (e.g., 1 second), ALPHA is + a smoothing factor (e.g., .8 to .9), and BETA is a delay variance + factor (e.g., 1.3 to 2.0). + + The Communication of Urgent Information + + The objective of the TCP urgent mechanism is to allow the sending user + to stimulate the receiving user to accept some urgent data and to + permit the receiving TCP to indicate to the receiving user when all + the currently known urgent data has been received by the user. + + This mechanism permits a point in the data stream to be designated as + the end of urgent information. Whenever this point is in advance of + the receive sequence number (RCV.NXT) at the receiving TCP, that TCP + must tell the user to go into "urgent mode"; when the receive sequence + number catches up to the urgent pointer, the TCP must tell user to go + + + [Page 41] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + into "normal mode". If the urgent pointer is updated while the user + is in "urgent mode", the update will be invisible to the user. + + The method employs a urgent field which is carried in all segments + transmitted. The URG control flag indicates that the urgent field is + meaningful and must be added to the segment sequence number to yield + the urgent pointer. The absence of this flag indicates that there is + no urgent data outstanding. + + To send an urgent indication the user must also send at least one data + octet. If the sending user also indicates a push, timely delivery of + the urgent information to the destination process is enhanced. + + Managing the Window + + The window sent in each segment indicates the range of sequence + numbers the sender of the window (the data receiver) is currently + prepared to accept. There is an assumption that this is related to + the currently available data buffer space available for this + connection. + + Indicating a large window encourages transmissions. If more data + arrives than can be accepted, it will be discarded. This will result + in excessive retransmissions, adding unnecessarily to the load on the + network and the TCPs. Indicating a small window may restrict the + transmission of data to the point of introducing a round trip delay + between each new segment transmitted. + + The mechanisms provided allow a TCP to advertise a large window and to + subsequently advertise a much smaller window without having accepted + that much data. This, so called "shrinking the window," is strongly + discouraged. The robustness principle dictates that TCPs will not + shrink the window themselves, but will be prepared for such behavior + on the part of other TCPs. + + The sending TCP must be prepared to accept from the user and send at + least one octet of new data even if the send window is zero. The + sending TCP must regularly retransmit to the receiving TCP even when + the window is zero. Two minutes is recommended for the retransmission + interval when the window is zero. This retransmission is essential to + guarantee that when either TCP has a zero window the re-opening of the + window will be reliably reported to the other. + + When the receiving TCP has a zero window and a segment arrives it must + still send an acknowledgment showing its next expected sequence number + and current window (zero). + + The sending TCP packages the data to be transmitted into segments + + +[Page 42] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + which fit the current window, and may repackage segments on the + retransmission queue. Such repackaging is not required, but may be + helpful. + + In a connection with a one-way data flow, the window information will + be carried in acknowledgment segments that all have the same sequence + number so there will be no way to reorder them if they arrive out of + order. This is not a serious problem, but it will allow the window + information to be on occasion temporarily based on old reports from + the data receiver. A refinement to avoid this problem is to act on + the window information from segments that carry the highest + acknowledgment number (that is segments with acknowledgment number + equal or greater than the highest previously received). + + The window management procedure has significant influence on the + communication performance. The following comments are suggestions to + implementers. + + Window Management Suggestions + + Allocating a very small window causes data to be transmitted in + many small segments when better performance is achieved using + fewer large segments. + + One suggestion for avoiding small windows is for the receiver to + defer updating a window until the additional allocation is at + least X percent of the maximum allocation possible for the + connection (where X might be 20 to 40). + + Another suggestion is for the sender to avoid sending small + segments by waiting until the window is large enough before + sending data. If the the user signals a push function then the + data must be sent even if it is a small segment. + + Note that the acknowledgments should not be delayed or unnecessary + retransmissions will result. One strategy would be to send an + acknowledgment when a small segment arrives (with out updating the + window information), and then to send another acknowledgment with + new window information when the window is larger. + + The segment sent to probe a zero window may also begin a break up + of transmitted data into smaller and smaller segments. If a + segment containing a single data octet sent to probe a zero window + is accepted, it consumes one octet of the window now available. + If the sending TCP simply sends as much as it can whenever the + window is non zero, the transmitted data will be broken into + alternating big and small segments. As time goes on, occasional + pauses in the receiver making window allocation available will + + + [Page 43] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + result in breaking the big segments into a small and not quite so + big pair. And after a while the data transmission will be in + mostly small segments. + + The suggestion here is that the TCP implementations need to + actively attempt to combine small window allocations into larger + windows, since the mechanisms for managing the window tend to lead + to many small windows in the simplest minded implementations. + +3.8. Interfaces + + There are of course two interfaces of concern: the user/TCP interface + and the TCP/lower-level interface. We have a fairly elaborate model + of the user/TCP interface, but the interface to the lower level + protocol module is left unspecified here, since it will be specified + in detail by the specification of the lowel level protocol. For the + case that the lower level is IP we note some of the parameter values + that TCPs might use. + + User/TCP Interface + + The following functional description of user commands to the TCP is, + at best, fictional, since every operating system will have different + facilities. Consequently, we must warn readers that different TCP + implementations may have different user interfaces. However, all + TCPs must provide a certain minimum set of services to guarantee + that all TCP implementations can support the same protocol + hierarchy. This section specifies the functional interfaces + required of all TCP implementations. + + TCP User Commands + + The following sections functionally characterize a USER/TCP + interface. The notation used is similar to most procedure or + function calls in high level languages, but this usage is not + meant to rule out trap type service calls (e.g., SVCs, UUOs, + EMTs). + + The user commands described below specify the basic functions the + TCP must perform to support interprocess communication. + Individual implementations must define their own exact format, and + may provide combinations or subsets of the basic functions in + single calls. In particular, some implementations may wish to + automatically OPEN a connection on the first SEND or RECEIVE + issued by the user for a given connection. + + + + + +[Page 44] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + In providing interprocess communication facilities, the TCP must + not only accept commands, but must also return information to the + processes it serves. The latter consists of: + + (a) general information about a connection (e.g., interrupts, + remote close, binding of unspecified foreign socket). + + (b) replies to specific user commands indicating success or + various types of failure. + + Open + + Format: OPEN (local port, foreign socket, active/passive + [, timeout] [, precedence] [, security/compartment] [, options]) + -> local connection name + + We assume that the local TCP is aware of the identity of the + processes it serves and will check the authority of the process + to use the connection specified. Depending upon the + implementation of the TCP, the local network and TCP identifiers + for the source address will either be supplied by the TCP or the + lower level protocol (e.g., IP). These considerations are the + result of concern about security, to the extent that no TCP be + able to masquerade as another one, and so on. Similarly, no + process can masquerade as another without the collusion of the + TCP. + + If the active/passive flag is set to passive, then this is a + call to LISTEN for an incoming connection. A passive open may + have either a fully specified foreign socket to wait for a + particular connection or an unspecified foreign socket to wait + for any call. A fully specified passive call can be made active + by the subsequent execution of a SEND. + + A transmission control block (TCB) is created and partially + filled in with data from the OPEN command parameters. + + On an active OPEN command, the TCP will begin the procedure to + synchronize (i.e., establish) the connection at once. + + The timeout, if present, permits the caller to set up a timeout + for all data submitted to TCP. If data is not successfully + delivered to the destination within the timeout period, the TCP + will abort the connection. The present global default is five + minutes. + + The TCP or some component of the operating system will verify + the users authority to open a connection with the specified + + + [Page 45] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + precedence or security/compartment. The absence of precedence + or security/compartment specification in the OPEN call indicates + the default values must be used. + + TCP will accept incoming requests as matching only if the + security/compartment information is exactly the same and only if + the precedence is equal to or higher than the precedence + requested in the OPEN call. + + The precedence for the connection is the higher of the values + requested in the OPEN call and received from the incoming + request, and fixed at that value for the life of the + connection.Implementers may want to give the user control of + this precedence negotiation. For example, the user might be + allowed to specify that the precedence must be exactly matched, + or that any attempt to raise the precedence be confirmed by the + user. + + A local connection name will be returned to the user by the TCP. + The local connection name can then be used as a short hand term + for the connection defined by the + pair. + + Send + + Format: SEND (local connection name, buffer address, byte + count, PUSH flag, URGENT flag [,timeout]) + + This call causes the data contained in the indicated user buffer + to be sent on the indicated connection. If the connection has + not been opened, the SEND is considered an error. Some + implementations may allow users to SEND first; in which case, an + automatic OPEN would be done. If the calling process is not + authorized to use this connection, an error is returned. + + If the PUSH flag is set, the data must be transmitted promptly + to the receiver, and the PUSH bit will be set in the last TCP + segment created from the buffer. If the PUSH flag is not set, + the data may be combined with data from subsequent SENDs for + transmission efficiency. + + If the URGENT flag is set, segments sent to the destination TCP + will have the urgent pointer set. The receiving TCP will signal + the urgent condition to the receiving process if the urgent + pointer indicates that data preceding the urgent pointer has not + been consumed by the receiving process. The purpose of urgent + is to stimulate the receiver to process the urgent data and to + indicate to the receiver when all the currently known urgent + + +[Page 46] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + data has been received. The number of times the sending user's + TCP signals urgent will not necessarily be equal to the number + of times the receiving user will be notified of the presence of + urgent data. + + If no foreign socket was specified in the OPEN, but the + connection is established (e.g., because a LISTENing connection + has become specific due to a foreign segment arriving for the + local socket), then the designated buffer is sent to the implied + foreign socket. Users who make use of OPEN with an unspecified + foreign socket can make use of SEND without ever explicitly + knowing the foreign socket address. + + However, if a SEND is attempted before the foreign socket + becomes specified, an error will be returned. Users can use the + STATUS call to determine the status of the connection. In some + implementations the TCP may notify the user when an unspecified + socket is bound. + + If a timeout is specified, the current user timeout for this + connection is changed to the new one. + + In the simplest implementation, SEND would not return control to + the sending process until either the transmission was complete + or the timeout had been exceeded. However, this simple method + is both subject to deadlocks (for example, both sides of the + connection might try to do SENDs before doing any RECEIVEs) and + offers poor performance, so it is not recommended. A more + sophisticated implementation would return immediately to allow + the process to run concurrently with network I/O, and, + furthermore, to allow multiple SENDs to be in progress. + Multiple SENDs are served in first come, first served order, so + the TCP will queue those it cannot service immediately. + + We have implicitly assumed an asynchronous user interface in + which a SEND later elicits some kind of SIGNAL or + pseudo-interrupt from the serving TCP. An alternative is to + return a response immediately. For instance, SENDs might return + immediate local acknowledgment, even if the segment sent had not + been acknowledged by the distant TCP. We could optimistically + assume eventual success. If we are wrong, the connection will + close anyway due to the timeout. In implementations of this + kind (synchronous), there will still be some asynchronous + signals, but these will deal with the connection itself, and not + with specific segments or buffers. + + In order for the process to distinguish among error or success + indications for different SENDs, it might be appropriate for the + + + [Page 47] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + buffer address to be returned along with the coded response to + the SEND request. TCP-to-user signals are discussed below, + indicating the information which should be returned to the + calling process. + + Receive + + Format: RECEIVE (local connection name, buffer address, byte + count) -> byte count, urgent flag, push flag + + This command allocates a receiving buffer associated with the + specified connection. If no OPEN precedes this command or the + calling process is not authorized to use this connection, an + error is returned. + + In the simplest implementation, control would not return to the + calling program until either the buffer was filled, or some + error occurred, but this scheme is highly subject to deadlocks. + A more sophisticated implementation would permit several + RECEIVEs to be outstanding at once. These would be filled as + segments arrive. This strategy permits increased throughput at + the cost of a more elaborate scheme (possibly asynchronous) to + notify the calling program that a PUSH has been seen or a buffer + filled. + + If enough data arrive to fill the buffer before a PUSH is seen, + the PUSH flag will not be set in the response to the RECEIVE. + The buffer will be filled with as much data as it can hold. If + a PUSH is seen before the buffer is filled the buffer will be + returned partially filled and PUSH indicated. + + If there is urgent data the user will have been informed as soon + as it arrived via a TCP-to-user signal. The receiving user + should thus be in "urgent mode". If the URGENT flag is on, + additional urgent data remains. If the URGENT flag is off, this + call to RECEIVE has returned all the urgent data, and the user + may now leave "urgent mode". Note that data following the + urgent pointer (non-urgent data) cannot be delivered to the user + in the same buffer with preceeding urgent data unless the + boundary is clearly marked for the user. + + To distinguish among several outstanding RECEIVEs and to take + care of the case that a buffer is not completely filled, the + return code is accompanied by both a buffer pointer and a byte + count indicating the actual length of the data received. + + Alternative implementations of RECEIVE might have the TCP + + + +[Page 48] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + allocate buffer storage, or the TCP might share a ring buffer + with the user. + + Close + + Format: CLOSE (local connection name) + + This command causes the connection specified to be closed. If + the connection is not open or the calling process is not + authorized to use this connection, an error is returned. + Closing connections is intended to be a graceful operation in + the sense that outstanding SENDs will be transmitted (and + retransmitted), as flow control permits, until all have been + serviced. Thus, it should be acceptable to make several SEND + calls, followed by a CLOSE, and expect all the data to be sent + to the destination. It should also be clear that users should + continue to RECEIVE on CLOSING connections, since the other side + may be trying to transmit the last of its data. Thus, CLOSE + means "I have no more to send" but does not mean "I will not + receive any more." It may happen (if the user level protocol is + not well thought out) that the closing side is unable to get rid + of all its data before timing out. In this event, CLOSE turns + into ABORT, and the closing TCP gives up. + + The user may CLOSE the connection at any time on his own + initiative, or in response to various prompts from the TCP + (e.g., remote close executed, transmission timeout exceeded, + destination inaccessible). + + Because closing a connection requires communication with the + foreign TCP, connections may remain in the closing state for a + short time. Attempts to reopen the connection before the TCP + replies to the CLOSE command will result in error responses. + + Close also implies push function. + + Status + + Format: STATUS (local connection name) -> status data + + This is an implementation dependent user command and could be + excluded without adverse effect. Information returned would + typically come from the TCB associated with the connection. + + This command returns a data block containing the following + information: + + local socket, + + + [Page 49] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + + foreign socket, + local connection name, + receive window, + send window, + connection state, + number of buffers awaiting acknowledgment, + number of buffers pending receipt, + urgent state, + precedence, + security/compartment, + and transmission timeout. + + Depending on the state of the connection, or on the + implementation itself, some of this information may not be + available or meaningful. If the calling process is not + authorized to use this connection, an error is returned. This + prevents unauthorized processes from gaining information about a + connection. + + Abort + + Format: ABORT (local connection name) + + This command causes all pending SENDs and RECEIVES to be + aborted, the TCB to be removed, and a special RESET message to + be sent to the TCP on the other side of the connection. + Depending on the implementation, users may receive abort + indications for each outstanding SEND or RECEIVE, or may simply + receive an ABORT-acknowledgment. + + TCP-to-User Messages + + It is assumed that the operating system environment provides a + means for the TCP to asynchronously signal the user program. When + the TCP does signal a user program, certain information is passed + to the user. Often in the specification the information will be + an error message. In other cases there will be information + relating to the completion of processing a SEND or RECEIVE or + other user call. + + The following information is provided: + + Local Connection Name Always + Response String Always + Buffer Address Send & Receive + Byte count (counts bytes received) Receive + Push flag Receive + Urgent flag Receive + + +[Page 50] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + TCP/Lower-Level Interface + + The TCP calls on a lower level protocol module to actually send and + receive information over a network. One case is that of the ARPA + internetwork system where the lower level module is the Internet + Protocol (IP) [2]. + + If the lower level protocol is IP it provides arguments for a type + of service and for a time to live. TCP uses the following settings + for these parameters: + + Type of Service = Precedence: routine, Delay: normal, Throughput: + normal, Reliability: normal; or 00000000. + + Time to Live = one minute, or 00111100. + + Note that the assumed maximum segment lifetime is two minutes. + Here we explicitly ask that a segment be destroyed if it cannot + be delivered by the internet system within one minute. + + If the lower level is IP (or other protocol that provides this + feature) and source routing is used, the interface must allow the + route information to be communicated. This is especially important + so that the source and destination addresses used in the TCP + checksum be the originating source and ultimate destination. It is + also important to preserve the return route to answer connection + requests. + + Any lower level protocol will have to provide the source address, + destination address, and protocol fields, and some way to determine + the "TCP length", both to provide the functional equivlent service + of IP and to be used in the TCP checksum. + + + + + + + + + + + + + + + + + + + [Page 51] + + + September 1981 +Transmission Control Protocol +Functional Specification + + + +3.9. Event Processing + + The processing depicted in this section is an example of one possible + implementation. Other implementations may have slightly different + processing sequences, but they should differ from those in this + section only in detail, not in substance. + + The activity of the TCP can be characterized as responding to events. + The events that occur can be cast into three categories: user calls, + arriving segments, and timeouts. This section describes the + processing the TCP does in response to each of the events. In many + cases the processing required depends on the state of the connection. + + Events that occur: + + User Calls + + OPEN + SEND + RECEIVE + CLOSE + ABORT + STATUS + + Arriving Segments + + SEGMENT ARRIVES + + Timeouts + + USER TIMEOUT + RETRANSMISSION TIMEOUT + TIME-WAIT TIMEOUT + + The model of the TCP/user interface is that user commands receive an + immediate return and possibly a delayed response via an event or + pseudo interrupt. In the following descriptions, the term "signal" + means cause a delayed response. + + Error responses are given as character strings. For example, user + commands referencing connections that do not exist receive "error: + connection not open". + + Please note in the following that all arithmetic on sequence numbers, + acknowledgment numbers, windows, et cetera, is modulo 2**32 the size + of the sequence number space. Also note that "=<" means less than or + equal to (modulo 2**32). + + + +[Page 52] + + +September 1981 + Transmission Control Protocol + Functional Specification + + + + A natural way to think about processing incoming segments is to + imagine that they are first tested for proper sequence number (i.e., + that their contents lie in the range of the expected "receive window" + in the sequence number space) and then that they are generally queued + and processed in sequence number order. + + When a segment overlaps other already received segments we reconstruct + the segment to contain just the new data, and adjust the header fields + to be consistent. + + Note that if no state change is mentioned the TCP stays in the same + state. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + [Page 53] + + + September 1981 +Transmission Control Protocol +Functional Specification + OPEN Call + + + + OPEN Call + + CLOSED STATE (i.e., TCB does not exist) + + Create a new transmission control block (TCB) to hold connection + state information. Fill in local socket identifier, foreign + socket, precedence, security/compartment, and user timeout + information. Note that some parts of the foreign socket may be + unspecified in a passive OPEN and are to be filled in by the + parameters of the incoming SYN segment. Verify the security and + precedence requested are allowed for this user, if not return + "error: precedence not allowed" or "error: security/compartment + not allowed." If passive enter the LISTEN state and return. If + active and the foreign socket is unspecified, return "error: + foreign socket unspecified"; if active and the foreign socket is + specified, issue a SYN segment. An initial send sequence number + (ISS) is selected. A SYN segment of the form + is sent. Set SND.UNA to ISS, SND.NXT to ISS+1, enter SYN-SENT + state, and return. + + If the caller does not have access to the local socket specified, + return "error: connection illegal for this process". If there is + no room to create a new connection, return "error: insufficient + resources". + + LISTEN STATE + + If active and the foreign socket is specified, then change the + connection from passive to active, select an ISS. Send a SYN + segment, set SND.UNA to ISS, SND.NXT to ISS+1. Enter SYN-SENT + state. Data associated with SEND may be sent with SYN segment or + queued for transmission after entering ESTABLISHED state. The + urgent bit if requested in the command must be sent with the data + segments sent as a result of this command. If there is no room to + queue the request, respond with "error: insufficient resources". + If Foreign socket was not specified, then return "error: foreign + socket unspecified". + + + + + + + + + + + + +[Page 54] + + +September 1981 + Transmission Control Protocol + Functional Specification +OPEN Call + + + + SYN-SENT STATE + SYN-RECEIVED STATE + ESTABLISHED STATE + FIN-WAIT-1 STATE + FIN-WAIT-2 STATE + CLOSE-WAIT STATE + CLOSING STATE + LAST-ACK STATE + TIME-WAIT STATE + + Return "error: connection already exists". + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + [Page 55] + + + September 1981 +Transmission Control Protocol +Functional Specification + SEND Call + + + + SEND Call + + CLOSED STATE (i.e., TCB does not exist) + + If the user does not have access to such a connection, then return + "error: connection illegal for this process". + + Otherwise, return "error: connection does not exist". + + LISTEN STATE + + If the foreign socket is specified, then change the connection + from passive to active, select an ISS. Send a SYN segment, set + SND.UNA to ISS, SND.NXT to ISS+1. Enter SYN-SENT state. Data + associated with SEND may be sent with SYN segment or queued for + transmission after entering ESTABLISHED state. The urgent bit if + requested in the command must be sent with the data segments sent + as a result of this command. If there is no room to queue the + request, respond with "error: insufficient resources". If + Foreign socket was not specified, then return "error: foreign + socket unspecified". + + SYN-SENT STATE + SYN-RECEIVED STATE + + Queue the data for transmission after entering ESTABLISHED state. + If no space to queue, respond with "error: insufficient + resources". + + ESTABLISHED STATE + CLOSE-WAIT STATE + + Segmentize the buffer and send it with a piggybacked + acknowledgment (acknowledgment value = RCV.NXT). If there is + insufficient space to remember this buffer, simply return "error: + insufficient resources". + + If the urgent flag is set, then SND.UP <- SND.NXT-1 and set the + urgent pointer in the outgoing segments. + + + + + + + + + + +[Page 56] + + +September 1981 + Transmission Control Protocol + Functional Specification +SEND Call + + + + FIN-WAIT-1 STATE + FIN-WAIT-2 STATE + CLOSING STATE + LAST-ACK STATE + TIME-WAIT STATE + + Return "error: connection closing" and do not service request. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + [Page 57] + + + September 1981 +Transmission Control Protocol +Functional Specification + RECEIVE Call + + + + RECEIVE Call + + CLOSED STATE (i.e., TCB does not exist) + + If the user does not have access to such a connection, return + "error: connection illegal for this process". + + Otherwise return "error: connection does not exist". + + LISTEN STATE + SYN-SENT STATE + SYN-RECEIVED STATE + + Queue for processing after entering ESTABLISHED state. If there + is no room to queue this request, respond with "error: + insufficient resources". + + ESTABLISHED STATE + FIN-WAIT-1 STATE + FIN-WAIT-2 STATE + + If insufficient incoming segments are queued to satisfy the + request, queue the request. If there is no queue space to + remember the RECEIVE, respond with "error: insufficient + resources". + + Reassemble queued incoming segments into receive buffer and return + to user. Mark "push seen" (PUSH) if this is the case. + + If RCV.UP is in advance of the data currently being passed to the + user notify the user of the presence of urgent data. + + When the TCP takes responsibility for delivering data to the user + that fact must be communicated to the sender via an + acknowledgment. The formation of such an acknowledgment is + described below in the discussion of processing an incoming + segment. + + + + + + + + + + + + +[Page 58] + + +September 1981 + Transmission Control Protocol + Functional Specification +RECEIVE Call + + + + CLOSE-WAIT STATE + + Since the remote side has already sent FIN, RECEIVEs must be + satisfied by text already on hand, but not yet delivered to the + user. If no text is awaiting delivery, the RECEIVE will get a + "error: connection closing" response. Otherwise, any remaining + text can be used to satisfy the RECEIVE. + + CLOSING STATE + LAST-ACK STATE + TIME-WAIT STATE + + Return "error: connection closing". + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + [Page 59] + + + September 1981 +Transmission Control Protocol +Functional Specification + CLOSE Call + + + + CLOSE Call + + CLOSED STATE (i.e., TCB does not exist) + + If the user does not have access to such a connection, return + "error: connection illegal for this process". + + Otherwise, return "error: connection does not exist". + + LISTEN STATE + + Any outstanding RECEIVEs are returned with "error: closing" + responses. Delete TCB, enter CLOSED state, and return. + + SYN-SENT STATE + + Delete the TCB and return "error: closing" responses to any + queued SENDs, or RECEIVEs. + + SYN-RECEIVED STATE + + If no SENDs have been issued and there is no pending data to send, + then form a FIN segment and send it, and enter FIN-WAIT-1 state; + otherwise queue for processing after entering ESTABLISHED state. + + ESTABLISHED STATE + + Queue this until all preceding SENDs have been segmentized, then + form a FIN segment and send it. In any case, enter FIN-WAIT-1 + state. + + FIN-WAIT-1 STATE + FIN-WAIT-2 STATE + + Strictly speaking, this is an error and should receive a "error: + connection closing" response. An "ok" response would be + acceptable, too, as long as a second FIN is not emitted (the first + FIN may be retransmitted though). + + + + + + + + + + + +[Page 60] + + +September 1981 + Transmission Control Protocol + Functional Specification +CLOSE Call + + + + CLOSE-WAIT STATE + + Queue this request until all preceding SENDs have been + segmentized; then send a FIN segment, enter CLOSING state. + + CLOSING STATE + LAST-ACK STATE + TIME-WAIT STATE + + Respond with "error: connection closing". + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + [Page 61] + + + September 1981 +Transmission Control Protocol +Functional Specification + ABORT Call + + + + ABORT Call + + CLOSED STATE (i.e., TCB does not exist) + + If the user should not have access to such a connection, return + "error: connection illegal for this process". + + Otherwise return "error: connection does not exist". + + LISTEN STATE + + Any outstanding RECEIVEs should be returned with "error: + connection reset" responses. Delete TCB, enter CLOSED state, and + return. + + SYN-SENT STATE + + All queued SENDs and RECEIVEs should be given "connection reset" + notification, delete the TCB, enter CLOSED state, and return. + + SYN-RECEIVED STATE + ESTABLISHED STATE + FIN-WAIT-1 STATE + FIN-WAIT-2 STATE + CLOSE-WAIT STATE + + Send a reset segment: + + + + All queued SENDs and RECEIVEs should be given "connection reset" + notification; all segments queued for transmission (except for the + RST formed above) or retransmission should be flushed, delete the + TCB, enter CLOSED state, and return. + + CLOSING STATE + LAST-ACK STATE + TIME-WAIT STATE + + Respond with "ok" and delete the TCB, enter CLOSED state, and + return. + + + + + + + + +[Page 62] + + +September 1981 + Transmission Control Protocol + Functional Specification +STATUS Call + + + + STATUS Call + + CLOSED STATE (i.e., TCB does not exist) + + If the user should not have access to such a connection, return + "error: connection illegal for this process". + + Otherwise return "error: connection does not exist". + + LISTEN STATE + + Return "state = LISTEN", and the TCB pointer. + + SYN-SENT STATE + + Return "state = SYN-SENT", and the TCB pointer. + + SYN-RECEIVED STATE + + Return "state = SYN-RECEIVED", and the TCB pointer. + + ESTABLISHED STATE + + Return "state = ESTABLISHED", and the TCB pointer. + + FIN-WAIT-1 STATE + + Return "state = FIN-WAIT-1", and the TCB pointer. + + FIN-WAIT-2 STATE + + Return "state = FIN-WAIT-2", and the TCB pointer. + + CLOSE-WAIT STATE + + Return "state = CLOSE-WAIT", and the TCB pointer. + + CLOSING STATE + + Return "state = CLOSING", and the TCB pointer. + + LAST-ACK STATE + + Return "state = LAST-ACK", and the TCB pointer. + + + + + + [Page 63] + + + September 1981 +Transmission Control Protocol +Functional Specification + STATUS Call + + + + TIME-WAIT STATE + + Return "state = TIME-WAIT", and the TCB pointer. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +[Page 64] + + +September 1981 + Transmission Control Protocol + Functional Specification +SEGMENT ARRIVES + + + + SEGMENT ARRIVES + + If the state is CLOSED (i.e., TCB does not exist) then + + all data in the incoming segment is discarded. An incoming + segment containing a RST is discarded. An incoming segment not + containing a RST causes a RST to be sent in response. The + acknowledgment and sequence field values are selected to make the + reset sequence acceptable to the TCP that sent the offending + segment. + + If the ACK bit is off, sequence number zero is used, + + + + If the ACK bit is on, + + + + Return. + + If the state is LISTEN then + + first check for an RST + + An incoming RST should be ignored. Return. + + second check for an ACK + + Any acknowledgment is bad if it arrives on a connection still in + the LISTEN state. An acceptable reset segment should be formed + for any arriving ACK-bearing segment. The RST should be + formatted as follows: + + + + Return. + + third check for a SYN + + If the SYN bit is set, check the security. If the + security/compartment on the incoming segment does not exactly + match the security/compartment in the TCB then send a reset and + return. + + + + + + [Page 65] + + + September 1981 +Transmission Control Protocol +Functional Specification + SEGMENT ARRIVES + + + + If the SEG.PRC is greater than the TCB.PRC then if allowed by + the user and the system set TCB.PRC<-SEG.PRC, if not allowed + send a reset and return. + + + + If the SEG.PRC is less than the TCB.PRC then continue. + + Set RCV.NXT to SEG.SEQ+1, IRS is set to SEG.SEQ and any other + control or text should be queued for processing later. ISS + should be selected and a SYN segment sent of the form: + + + + SND.NXT is set to ISS+1 and SND.UNA to ISS. The connection + state should be changed to SYN-RECEIVED. Note that any other + incoming control or data (combined with SYN) will be processed + in the SYN-RECEIVED state, but processing of SYN and ACK should + not be repeated. If the listen was not fully specified (i.e., + the foreign socket was not fully specified), then the + unspecified fields should be filled in now. + + fourth other text or control + + Any other control or text-bearing segment (not containing SYN) + must have an ACK and thus would be discarded by the ACK + processing. An incoming RST segment could not be valid, since + it could not have been sent in response to anything sent by this + incarnation of the connection. So you are unlikely to get here, + but if you do, drop the segment, and return. + + If the state is SYN-SENT then + + first check the ACK bit + + If the ACK bit is set + + If SEG.ACK =< ISS, or SEG.ACK > SND.NXT, send a reset (unless + the RST bit is set, if so drop the segment and return) + + + + and discard the segment. Return. + + If SND.UNA =< SEG.ACK =< SND.NXT then the ACK is acceptable. + + second check the RST bit + + +[Page 66] + + +September 1981 + Transmission Control Protocol + Functional Specification +SEGMENT ARRIVES + + + + If the RST bit is set + + If the ACK was acceptable then signal the user "error: + connection reset", drop the segment, enter CLOSED state, + delete TCB, and return. Otherwise (no ACK) drop the segment + and return. + + third check the security and precedence + + If the security/compartment in the segment does not exactly + match the security/compartment in the TCB, send a reset + + If there is an ACK + + + + Otherwise + + + + If there is an ACK + + The precedence in the segment must match the precedence in the + TCB, if not, send a reset + + + + If there is no ACK + + If the precedence in the segment is higher than the precedence + in the TCB then if allowed by the user and the system raise + the precedence in the TCB to that in the segment, if not + allowed to raise the prec then send a reset. + + + + If the precedence in the segment is lower than the precedence + in the TCB continue. + + If a reset was sent, discard the segment and return. + + fourth check the SYN bit + + This step should be reached only if the ACK is ok, or there is + no ACK, and it the segment did not contain a RST. + + If the SYN bit is on and the security/compartment and precedence + + + [Page 67] + + + September 1981 +Transmission Control Protocol +Functional Specification + SEGMENT ARRIVES + + + + are acceptable then, RCV.NXT is set to SEG.SEQ+1, IRS is set to + SEG.SEQ. SND.UNA should be advanced to equal SEG.ACK (if there + is an ACK), and any segments on the retransmission queue which + are thereby acknowledged should be removed. + + If SND.UNA > ISS (our SYN has been ACKed), change the connection + state to ESTABLISHED, form an ACK segment + + + + and send it. Data or controls which were queued for + transmission may be included. If there are other controls or + text in the segment then continue processing at the sixth step + below where the URG bit is checked, otherwise return. + + Otherwise enter SYN-RECEIVED, form a SYN,ACK segment + + + + and send it. If there are other controls or text in the + segment, queue them for processing after the ESTABLISHED state + has been reached, return. + + fifth, if neither of the SYN or RST bits is set then drop the + segment and return. + + + + + + + + + + + + + + + + + + + + + + + + +[Page 68] + + +September 1981 + Transmission Control Protocol + Functional Specification +SEGMENT ARRIVES + + + + Otherwise, + + first check sequence number + + SYN-RECEIVED STATE + ESTABLISHED STATE + FIN-WAIT-1 STATE + FIN-WAIT-2 STATE + CLOSE-WAIT STATE + CLOSING STATE + LAST-ACK STATE + TIME-WAIT STATE + + Segments are processed in sequence. Initial tests on arrival + are used to discard old duplicates, but further processing is + done in SEG.SEQ order. If a segment's contents straddle the + boundary between old and new, only the new parts should be + processed. + + There are four cases for the acceptability test for an incoming + segment: + + Segment Receive Test + Length Window + ------- ------- ------------------------------------------- + + 0 0 SEG.SEQ = RCV.NXT + + 0 >0 RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND + + >0 0 not acceptable + + >0 >0 RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND + or RCV.NXT =< SEG.SEQ+SEG.LEN-1 < RCV.NXT+RCV.WND + + If the RCV.WND is zero, no segments will be acceptable, but + special allowance should be made to accept valid ACKs, URGs and + RSTs. + + If an incoming segment is not acceptable, an acknowledgment + should be sent in reply (unless the RST bit is set, if so drop + the segment and return): + + + + After sending the acknowledgment, drop the unacceptable segment + and return. + + + [Page 69] + + + September 1981 +Transmission Control Protocol +Functional Specification + SEGMENT ARRIVES + + + + In the following it is assumed that the segment is the idealized + segment that begins at RCV.NXT and does not exceed the window. + One could tailor actual segments to fit this assumption by + trimming off any portions that lie outside the window (including + SYN and FIN), and only processing further if the segment then + begins at RCV.NXT. Segments with higher begining sequence + numbers may be held for later processing. + + second check the RST bit, + + SYN-RECEIVED STATE + + If the RST bit is set + + If this connection was initiated with a passive OPEN (i.e., + came from the LISTEN state), then return this connection to + LISTEN state and return. The user need not be informed. If + this connection was initiated with an active OPEN (i.e., came + from SYN-SENT state) then the connection was refused, signal + the user "connection refused". In either case, all segments + on the retransmission queue should be removed. And in the + active OPEN case, enter the CLOSED state and delete the TCB, + and return. + + ESTABLISHED + FIN-WAIT-1 + FIN-WAIT-2 + CLOSE-WAIT + + If the RST bit is set then, any outstanding RECEIVEs and SEND + should receive "reset" responses. All segment queues should be + flushed. Users should also receive an unsolicited general + "connection reset" signal. Enter the CLOSED state, delete the + TCB, and return. + + CLOSING STATE + LAST-ACK STATE + TIME-WAIT + + If the RST bit is set then, enter the CLOSED state, delete the + TCB, and return. + + + + + + + + +[Page 70] + + +September 1981 + Transmission Control Protocol + Functional Specification +SEGMENT ARRIVES + + + + third check security and precedence + + SYN-RECEIVED + + If the security/compartment and precedence in the segment do not + exactly match the security/compartment and precedence in the TCB + then send a reset, and return. + + ESTABLISHED STATE + + If the security/compartment and precedence in the segment do not + exactly match the security/compartment and precedence in the TCB + then send a reset, any outstanding RECEIVEs and SEND should + receive "reset" responses. All segment queues should be + flushed. Users should also receive an unsolicited general + "connection reset" signal. Enter the CLOSED state, delete the + TCB, and return. + + Note this check is placed following the sequence check to prevent + a segment from an old connection between these ports with a + different security or precedence from causing an abort of the + current connection. + + fourth, check the SYN bit, + + SYN-RECEIVED + ESTABLISHED STATE + FIN-WAIT STATE-1 + FIN-WAIT STATE-2 + CLOSE-WAIT STATE + CLOSING STATE + LAST-ACK STATE + TIME-WAIT STATE + + If the SYN is in the window it is an error, send a reset, any + outstanding RECEIVEs and SEND should receive "reset" responses, + all segment queues should be flushed, the user should also + receive an unsolicited general "connection reset" signal, enter + the CLOSED state, delete the TCB, and return. + + If the SYN is not in the window this step would not be reached + and an ack would have been sent in the first step (sequence + number check). + + + + + + + [Page 71] + + + September 1981 +Transmission Control Protocol +Functional Specification + SEGMENT ARRIVES + + + + fifth check the ACK field, + + if the ACK bit is off drop the segment and return + + if the ACK bit is on + + SYN-RECEIVED STATE + + If SND.UNA =< SEG.ACK =< SND.NXT then enter ESTABLISHED state + and continue processing. + + If the segment acknowledgment is not acceptable, form a + reset segment, + + + + and send it. + + ESTABLISHED STATE + + If SND.UNA < SEG.ACK =< SND.NXT then, set SND.UNA <- SEG.ACK. + Any segments on the retransmission queue which are thereby + entirely acknowledged are removed. Users should receive + positive acknowledgments for buffers which have been SENT and + fully acknowledged (i.e., SEND buffer should be returned with + "ok" response). If the ACK is a duplicate + (SEG.ACK < SND.UNA), it can be ignored. If the ACK acks + something not yet sent (SEG.ACK > SND.NXT) then send an ACK, + drop the segment, and return. + + If SND.UNA < SEG.ACK =< SND.NXT, the send window should be + updated. If (SND.WL1 < SEG.SEQ or (SND.WL1 = SEG.SEQ and + SND.WL2 =< SEG.ACK)), set SND.WND <- SEG.WND, set + SND.WL1 <- SEG.SEQ, and set SND.WL2 <- SEG.ACK. + + Note that SND.WND is an offset from SND.UNA, that SND.WL1 + records the sequence number of the last segment used to update + SND.WND, and that SND.WL2 records the acknowledgment number of + the last segment used to update SND.WND. The check here + prevents using old segments to update the window. + + + + + + + + + +[Page 72] + + +September 1981 + Transmission Control Protocol + Functional Specification +SEGMENT ARRIVES + + + + FIN-WAIT-1 STATE + + In addition to the processing for the ESTABLISHED state, if + our FIN is now acknowledged then enter FIN-WAIT-2 and continue + processing in that state. + + FIN-WAIT-2 STATE + + In addition to the processing for the ESTABLISHED state, if + the retransmission queue is empty, the user's CLOSE can be + acknowledged ("ok") but do not delete the TCB. + + CLOSE-WAIT STATE + + Do the same processing as for the ESTABLISHED state. + + CLOSING STATE + + In addition to the processing for the ESTABLISHED state, if + the ACK acknowledges our FIN then enter the TIME-WAIT state, + otherwise ignore the segment. + + LAST-ACK STATE + + The only thing that can arrive in this state is an + acknowledgment of our FIN. If our FIN is now acknowledged, + delete the TCB, enter the CLOSED state, and return. + + TIME-WAIT STATE + + The only thing that can arrive in this state is a + retransmission of the remote FIN. Acknowledge it, and restart + the 2 MSL timeout. + + sixth, check the URG bit, + + ESTABLISHED STATE + FIN-WAIT-1 STATE + FIN-WAIT-2 STATE + + If the URG bit is set, RCV.UP <- max(RCV.UP,SEG.UP), and signal + the user that the remote side has urgent data if the urgent + pointer (RCV.UP) is in advance of the data consumed. If the + user has already been signaled (or is still in the "urgent + mode") for this continuous sequence of urgent data, do not + signal the user again. + + + + [Page 73] + + + September 1981 +Transmission Control Protocol +Functional Specification + SEGMENT ARRIVES + + + + CLOSE-WAIT STATE + CLOSING STATE + LAST-ACK STATE + TIME-WAIT + + This should not occur, since a FIN has been received from the + remote side. Ignore the URG. + + seventh, process the segment text, + + ESTABLISHED STATE + FIN-WAIT-1 STATE + FIN-WAIT-2 STATE + + Once in the ESTABLISHED state, it is possible to deliver segment + text to user RECEIVE buffers. Text from segments can be moved + into buffers until either the buffer is full or the segment is + empty. If the segment empties and carries an PUSH flag, then + the user is informed, when the buffer is returned, that a PUSH + has been received. + + When the TCP takes responsibility for delivering the data to the + user it must also acknowledge the receipt of the data. + + Once the TCP takes responsibility for the data it advances + RCV.NXT over the data accepted, and adjusts RCV.WND as + apporopriate to the current buffer availability. The total of + RCV.NXT and RCV.WND should not be reduced. + + Please note the window management suggestions in section 3.7. + + Send an acknowledgment of the form: + + + + This acknowledgment should be piggybacked on a segment being + transmitted if possible without incurring undue delay. + + + + + + + + + + + + +[Page 74] + + +September 1981 + Transmission Control Protocol + Functional Specification +SEGMENT ARRIVES + + + + CLOSE-WAIT STATE + CLOSING STATE + LAST-ACK STATE + TIME-WAIT STATE + + This should not occur, since a FIN has been received from the + remote side. Ignore the segment text. + + eighth, check the FIN bit, + + Do not process the FIN if the state is CLOSED, LISTEN or SYN-SENT + since the SEG.SEQ cannot be validated; drop the segment and + return. + + If the FIN bit is set, signal the user "connection closing" and + return any pending RECEIVEs with same message, advance RCV.NXT + over the FIN, and send an acknowledgment for the FIN. Note that + FIN implies PUSH for any segment text not yet delivered to the + user. + + SYN-RECEIVED STATE + ESTABLISHED STATE + + Enter the CLOSE-WAIT state. + + FIN-WAIT-1 STATE + + If our FIN has been ACKed (perhaps in this segment), then + enter TIME-WAIT, start the time-wait timer, turn off the other + timers; otherwise enter the CLOSING state. + + FIN-WAIT-2 STATE + + Enter the TIME-WAIT state. Start the time-wait timer, turn + off the other timers. + + CLOSE-WAIT STATE + + Remain in the CLOSE-WAIT state. + + CLOSING STATE + + Remain in the CLOSING state. + + LAST-ACK STATE + + Remain in the LAST-ACK state. + + + [Page 75] + + + September 1981 +Transmission Control Protocol +Functional Specification + SEGMENT ARRIVES + + + + TIME-WAIT STATE + + Remain in the TIME-WAIT state. Restart the 2 MSL time-wait + timeout. + + and return. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +[Page 76] + + +September 1981 + Transmission Control Protocol + Functional Specification +USER TIMEOUT + + + + USER TIMEOUT + + For any state if the user timeout expires, flush all queues, signal + the user "error: connection aborted due to user timeout" in general + and for any outstanding calls, delete the TCB, enter the CLOSED + state and return. + + RETRANSMISSION TIMEOUT + + For any state if the retransmission timeout expires on a segment in + the retransmission queue, send the segment at the front of the + retransmission queue again, reinitialize the retransmission timer, + and return. + + TIME-WAIT TIMEOUT + + If the time-wait timeout expires on a connection delete the TCB, + enter the CLOSED state and return. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + [Page 77] + + + September 1981 +Transmission Control Protocol + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +[Page 78] + + +September 1981 + Transmission Control Protocol + + + + GLOSSARY + + + +1822 + BBN Report 1822, "The Specification of the Interconnection of + a Host and an IMP". The specification of interface between a + host and the ARPANET. + +ACK + A control bit (acknowledge) occupying no sequence space, which + indicates that the acknowledgment field of this segment + specifies the next sequence number the sender of this segment + is expecting to receive, hence acknowledging receipt of all + previous sequence numbers. + +ARPANET message + The unit of transmission between a host and an IMP in the + ARPANET. The maximum size is about 1012 octets (8096 bits). + +ARPANET packet + A unit of transmission used internally in the ARPANET between + IMPs. The maximum size is about 126 octets (1008 bits). + +connection + A logical communication path identified by a pair of sockets. + +datagram + A message sent in a packet switched computer communications + network. + +Destination Address + The destination address, usually the network and host + identifiers. + +FIN + A control bit (finis) occupying one sequence number, which + indicates that the sender will send no more data or control + occupying sequence space. + +fragment + A portion of a logical unit of data, in particular an internet + fragment is a portion of an internet datagram. + +FTP + A file transfer protocol. + + + + + + [Page 79] + + + September 1981 +Transmission Control Protocol +Glossary + + + +header + Control information at the beginning of a message, segment, + fragment, packet or block of data. + +host + A computer. In particular a source or destination of messages + from the point of view of the communication network. + +Identification + An Internet Protocol field. This identifying value assigned + by the sender aids in assembling the fragments of a datagram. + +IMP + The Interface Message Processor, the packet switch of the + ARPANET. + +internet address + A source or destination address specific to the host level. + +internet datagram + The unit of data exchanged between an internet module and the + higher level protocol together with the internet header. + +internet fragment + A portion of the data of an internet datagram with an internet + header. + +IP + Internet Protocol. + +IRS + The Initial Receive Sequence number. The first sequence + number used by the sender on a connection. + +ISN + The Initial Sequence Number. The first sequence number used + on a connection, (either ISS or IRS). Selected on a clock + based procedure. + +ISS + The Initial Send Sequence number. The first sequence number + used by the sender on a connection. + +leader + Control information at the beginning of a message or block of + data. In particular, in the ARPANET, the control information + on an ARPANET message at the host-IMP interface. + + + +[Page 80] + + +September 1981 + Transmission Control Protocol + Glossary + + + +left sequence + This is the next sequence number to be acknowledged by the + data receiving TCP (or the lowest currently unacknowledged + sequence number) and is sometimes referred to as the left edge + of the send window. + +local packet + The unit of transmission within a local network. + +module + An implementation, usually in software, of a protocol or other + procedure. + +MSL + Maximum Segment Lifetime, the time a TCP segment can exist in + the internetwork system. Arbitrarily defined to be 2 minutes. + +octet + An eight bit byte. + +Options + An Option field may contain several options, and each option + may be several octets in length. The options are used + primarily in testing situations; for example, to carry + timestamps. Both the Internet Protocol and TCP provide for + options fields. + +packet + A package of data with a header which may or may not be + logically complete. More often a physical packaging than a + logical packaging of data. + +port + The portion of a socket that specifies which logical input or + output channel of a process is associated with the data. + +process + A program in execution. A source or destination of data from + the point of view of the TCP or other host-to-host protocol. + +PUSH + A control bit occupying no sequence space, indicating that + this segment contains data that must be pushed through to the + receiving user. + +RCV.NXT + receive next sequence number + + + + [Page 81] + + + September 1981 +Transmission Control Protocol +Glossary + + + +RCV.UP + receive urgent pointer + +RCV.WND + receive window + +receive next sequence number + This is the next sequence number the local TCP is expecting to + receive. + +receive window + This represents the sequence numbers the local (receiving) TCP + is willing to receive. Thus, the local TCP considers that + segments overlapping the range RCV.NXT to + RCV.NXT + RCV.WND - 1 carry acceptable data or control. + Segments containing sequence numbers entirely outside of this + range are considered duplicates and discarded. + +RST + A control bit (reset), occupying no sequence space, indicating + that the receiver should delete the connection without further + interaction. The receiver can determine, based on the + sequence number and acknowledgment fields of the incoming + segment, whether it should honor the reset command or ignore + it. In no case does receipt of a segment containing RST give + rise to a RST in response. + +RTP + Real Time Protocol: A host-to-host protocol for communication + of time critical information. + +SEG.ACK + segment acknowledgment + +SEG.LEN + segment length + +SEG.PRC + segment precedence value + +SEG.SEQ + segment sequence + +SEG.UP + segment urgent pointer field + + + + + +[Page 82] + + +September 1981 + Transmission Control Protocol + Glossary + + + +SEG.WND + segment window field + +segment + A logical unit of data, in particular a TCP segment is the + unit of data transfered between a pair of TCP modules. + +segment acknowledgment + The sequence number in the acknowledgment field of the + arriving segment. + +segment length + The amount of sequence number space occupied by a segment, + including any controls which occupy sequence space. + +segment sequence + The number in the sequence field of the arriving segment. + +send sequence + This is the next sequence number the local (sending) TCP will + use on the connection. It is initially selected from an + initial sequence number curve (ISN) and is incremented for + each octet of data or sequenced control transmitted. + +send window + This represents the sequence numbers which the remote + (receiving) TCP is willing to receive. It is the value of the + window field specified in segments from the remote (data + receiving) TCP. The range of new sequence numbers which may + be emitted by a TCP lies between SND.NXT and + SND.UNA + SND.WND - 1. (Retransmissions of sequence numbers + between SND.UNA and SND.NXT are expected, of course.) + +SND.NXT + send sequence + +SND.UNA + left sequence + +SND.UP + send urgent pointer + +SND.WL1 + segment sequence number at last window update + +SND.WL2 + segment acknowledgment number at last window update + + + + [Page 83] + + + September 1981 +Transmission Control Protocol +Glossary + + + +SND.WND + send window + +socket + An address which specifically includes a port identifier, that + is, the concatenation of an Internet Address with a TCP port. + +Source Address + The source address, usually the network and host identifiers. + +SYN + A control bit in the incoming segment, occupying one sequence + number, used at the initiation of a connection, to indicate + where the sequence numbering will start. + +TCB + Transmission control block, the data structure that records + the state of a connection. + +TCB.PRC + The precedence of the connection. + +TCP + Transmission Control Protocol: A host-to-host protocol for + reliable communication in internetwork environments. + +TOS + Type of Service, an Internet Protocol field. + +Type of Service + An Internet Protocol field which indicates the type of service + for this internet fragment. + +URG + A control bit (urgent), occupying no sequence space, used to + indicate that the receiving user should be notified to do + urgent processing as long as there is data to be consumed with + sequence numbers less than the value indicated in the urgent + pointer. + +urgent pointer + A control field meaningful only when the URG bit is on. This + field communicates the value of the urgent pointer which + indicates the data octet associated with the sending user's + urgent call. + + + + + +[Page 84] + + +September 1981 + Transmission Control Protocol + + + + REFERENCES + + + +[1] Cerf, V., and R. Kahn, "A Protocol for Packet Network + Intercommunication", IEEE Transactions on Communications, + Vol. COM-22, No. 5, pp 637-648, May 1974. + +[2] Postel, J. (ed.), "Internet Protocol - DARPA Internet Program + Protocol Specification", RFC 791, USC/Information Sciences + Institute, September 1981. + +[3] Dalal, Y. and C. Sunshine, "Connection Management in Transport + Protocols", Computer Networks, Vol. 2, No. 6, pp. 454-473, + December 1978. + +[4] Postel, J., "Assigned Numbers", RFC 790, USC/Information Sciences + Institute, September 1981. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + [Page 85] + diff --git a/decisiongrounding/scenarios_candidates/rfc_tcp_supersession/corpus/RFC-9293.md b/decisiongrounding/scenarios_candidates/rfc_tcp_supersession/corpus/RFC-9293.md new file mode 100644 index 0000000..aea17d9 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/rfc_tcp_supersession/corpus/RFC-9293.md @@ -0,0 +1,5612 @@ +--- +schema_version: 1 +id: RFC-9293 +type: decision +tags: [rfc] +--- + +# RFC-9293 — 6691 + +## Status + +Final + +## Context + +Public decision ingested verbatim from the RFC Editor (RFC 9293); +source https://www.rfc-editor.org/rfc/rfc9293.txt. RFCs are immutable, so the RFC number is the +pin and the upstream sha256 is recorded in provenance.json. Upstream +category: Standards Track. The RAC envelope sections are derived from +the RFC's own header fields; the authoritative content is the unedited RFC +reproduced under Source Text. + +## Decision + +Defined by the verbatim RFC text under Source Text below. + +## Consequences + +As stated by the upstream RFC; see Source Text. + +## Supersedes + +- RFC-0793 + +## Source Text + + + + + +Internet Engineering Task Force (IETF) W. Eddy, Ed. +STD: 7 MTI Systems +Request for Comments: 9293 August 2022 +Obsoletes: 793, 879, 2873, 6093, 6429, 6528, + 6691 +Updates: 1011, 1122, 5961 +Category: Standards Track +ISSN: 2070-1721 + + + Transmission Control Protocol (TCP) + +Abstract + + This document specifies the Transmission Control Protocol (TCP). TCP + is an important transport-layer protocol in the Internet protocol + stack, and it has continuously evolved over decades of use and growth + of the Internet. Over this time, a number of changes have been made + to TCP as it was specified in RFC 793, though these have only been + documented in a piecemeal fashion. This document collects and brings + those changes together with the protocol specification from RFC 793. + This document obsoletes RFC 793, as well as RFCs 879, 2873, 6093, + 6429, 6528, and 6691 that updated parts of RFC 793. It updates RFCs + 1011 and 1122, and it should be considered as a replacement for the + portions of those documents dealing with TCP requirements. It also + updates RFC 5961 by adding a small clarification in reset handling + while in the SYN-RECEIVED state. The TCP header control bits from + RFC 793 have also been updated based on RFC 3168. + +Status of This Memo + + This is an Internet Standards Track document. + + This document is a product of the Internet Engineering Task Force + (IETF). It represents the consensus of the IETF community. It has + received public review and has been approved for publication by the + Internet Engineering Steering Group (IESG). Further information on + Internet Standards is available in Section 2 of RFC 7841. + + Information about the current status of this document, any errata, + and how to provide feedback on it may be obtained at + https://www.rfc-editor.org/info/rfc9293. + +Copyright Notice + + Copyright (c) 2022 IETF Trust and the persons identified as the + document authors. All rights reserved. + + This document is subject to BCP 78 and the IETF Trust's Legal + Provisions Relating to IETF Documents + (https://trustee.ietf.org/license-info) in effect on the date of + publication of this document. Please review these documents + carefully, as they describe your rights and restrictions with respect + to this document. Code Components extracted from this document must + include Revised BSD License text as described in Section 4.e of the + Trust Legal Provisions and are provided without warranty as described + in the Revised BSD License. + + This document may contain material from IETF Documents or IETF + Contributions published or made publicly available before November + 10, 2008. The person(s) controlling the copyright in some of this + material may not have granted the IETF Trust the right to allow + modifications of such material outside the IETF Standards Process. + Without obtaining an adequate license from the person(s) controlling + the copyright in such materials, this document may not be modified + outside the IETF Standards Process, and derivative works of it may + not be created outside the IETF Standards Process, except to format + it for publication as an RFC or to translate it into languages other + than English. + +Table of Contents + + 1. Purpose and Scope + 2. Introduction + 2.1. Requirements Language + 2.2. Key TCP Concepts + 3. Functional Specification + 3.1. Header Format + 3.2. Specific Option Definitions + 3.2.1. Other Common Options + 3.2.2. Experimental TCP Options + 3.3. TCP Terminology Overview + 3.3.1. Key Connection State Variables + 3.3.2. State Machine Overview + 3.4. Sequence Numbers + 3.4.1. Initial Sequence Number Selection + 3.4.2. Knowing When to Keep Quiet + 3.4.3. The TCP Quiet Time Concept + 3.5. Establishing a Connection + 3.5.1. Half-Open Connections and Other Anomalies + 3.5.2. Reset Generation + 3.5.3. Reset Processing + 3.6. Closing a Connection + 3.6.1. Half-Closed Connections + 3.7. Segmentation + 3.7.1. Maximum Segment Size Option + 3.7.2. Path MTU Discovery + 3.7.3. Interfaces with Variable MTU Values + 3.7.4. Nagle Algorithm + 3.7.5. IPv6 Jumbograms + 3.8. Data Communication + 3.8.1. Retransmission Timeout + 3.8.2. TCP Congestion Control + 3.8.3. TCP Connection Failures + 3.8.4. TCP Keep-Alives + 3.8.5. The Communication of Urgent Information + 3.8.6. Managing the Window + 3.9. Interfaces + 3.9.1. User/TCP Interface + 3.9.2. TCP/Lower-Level Interface + 3.10. Event Processing + 3.10.1. OPEN Call + 3.10.2. SEND Call + 3.10.3. RECEIVE Call + 3.10.4. CLOSE Call + 3.10.5. ABORT Call + 3.10.6. STATUS Call + 3.10.7. SEGMENT ARRIVES + 3.10.8. Timeouts + 4. Glossary + 5. Changes from RFC 793 + 6. IANA Considerations + 7. Security and Privacy Considerations + 8. References + 8.1. Normative References + 8.2. Informative References + Appendix A. Other Implementation Notes + A.1. IP Security Compartment and Precedence + A.1.1. Precedence + A.1.2. MLS Systems + A.2. Sequence Number Validation + A.3. Nagle Modification + A.4. Low Watermark Settings + Appendix B. TCP Requirement Summary + Acknowledgments + Author's Address + +1. Purpose and Scope + + In 1981, RFC 793 [16] was released, documenting the Transmission + Control Protocol (TCP) and replacing earlier published specifications + for TCP. + + Since then, TCP has been widely implemented, and it has been used as + a transport protocol for numerous applications on the Internet. + + For several decades, RFC 793 plus a number of other documents have + combined to serve as the core specification for TCP [49]. Over time, + a number of errata have been filed against RFC 793. There have also + been deficiencies found and resolved in security, performance, and + many other aspects. The number of enhancements has grown over time + across many separate documents. These were never accumulated + together into a comprehensive update to the base specification. + + The purpose of this document is to bring together all of the IETF + Standards Track changes and other clarifications that have been made + to the base TCP functional specification (RFC 793) and to unify them + into an updated version of the specification. + + Some companion documents are referenced for important algorithms that + are used by TCP (e.g., for congestion control) but have not been + completely included in this document. This is a conscious choice, as + this base specification can be used with multiple additional + algorithms that are developed and incorporated separately. This + document focuses on the common basis that all TCP implementations + must support in order to interoperate. Since some additional TCP + features have become quite complicated themselves (e.g., advanced + loss recovery and congestion control), future companion documents may + attempt to similarly bring these together. + + In addition to the protocol specification that describes the TCP + segment format, generation, and processing rules that are to be + implemented in code, RFC 793 and other updates also contain + informative and descriptive text for readers to understand aspects of + the protocol design and operation. This document does not attempt to + alter or update this informative text and is focused only on updating + the normative protocol specification. This document preserves + references to the documentation containing the important explanations + and rationale, where appropriate. + + This document is intended to be useful both in checking existing TCP + implementations for conformance purposes, as well as in writing new + implementations. + +2. Introduction + + RFC 793 contains a discussion of the TCP design goals and provides + examples of its operation, including examples of connection + establishment, connection termination, and packet retransmission to + repair losses. + + This document describes the basic functionality expected in modern + TCP implementations and replaces the protocol specification in RFC + 793. It does not replicate or attempt to update the introduction and + philosophy content in Sections 1 and 2 of RFC 793. Other documents + are referenced to provide explanations of the theory of operation, + rationale, and detailed discussion of design decisions. This + document only focuses on the normative behavior of the protocol. + + The "TCP Roadmap" [49] provides a more extensive guide to the RFCs + that define TCP and describe various important algorithms. The TCP + Roadmap contains sections on strongly encouraged enhancements that + improve performance and other aspects of TCP beyond the basic + operation specified in this document. As one example, implementing + congestion control (e.g., [8]) is a TCP requirement, but it is a + complex topic on its own and not described in detail in this + document, as there are many options and possibilities that do not + impact basic interoperability. Similarly, most TCP implementations + today include the high-performance extensions in [47], but these are + not strictly required or discussed in this document. Multipath + considerations for TCP are also specified separately in [59]. + + A list of changes from RFC 793 is contained in Section 5. + +2.1. Requirements Language + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and + "OPTIONAL" in this document are to be interpreted as described in + BCP 14 [3] [12] when, and only when, they appear in all capitals, as + shown here. + + Each use of RFC 2119 keywords in the document is individually labeled + and referenced in Appendix B, which summarizes implementation + requirements. + + Sentences using "MUST" are labeled as "MUST-X" with X being a numeric + identifier enabling the requirement to be located easily when + referenced from Appendix B. + + Similarly, sentences using "SHOULD" are labeled with "SHLD-X", "MAY" + with "MAY-X", and "RECOMMENDED" with "REC-X". + + For the purposes of this labeling, "SHOULD NOT" and "MUST NOT" are + labeled the same as "SHOULD" and "MUST" instances. + +2.2. Key TCP Concepts + + TCP provides a reliable, in-order, byte-stream service to + applications. + + The application byte-stream is conveyed over the network via TCP + segments, with each TCP segment sent as an Internet Protocol (IP) + datagram. + + TCP reliability consists of detecting packet losses (via sequence + numbers) and errors (via per-segment checksums), as well as + correction via retransmission. + + TCP supports unicast delivery of data. There are anycast + applications that can successfully use TCP without modifications, + though there is some risk of instability due to changes of lower- + layer forwarding behavior [46]. + + TCP is connection oriented, though it does not inherently include a + liveness detection capability. + + Data flow is supported bidirectionally over TCP connections, though + applications are free to send data only unidirectionally, if they so + choose. + + TCP uses port numbers to identify application services and to + multiplex distinct flows between hosts. + + A more detailed description of TCP features compared to other + transport protocols can be found in Section 3.1 of [52]. Further + description of the motivations for developing TCP and its role in the + Internet protocol stack can be found in Section 2 of [16] and earlier + versions of the TCP specification. + +3. Functional Specification + +3.1. Header Format + + TCP segments are sent as internet datagrams. The Internet Protocol + (IP) header carries several information fields, including the source + and destination host addresses [1] [13]. A TCP header follows the IP + headers, supplying information specific to TCP. This division allows + for the existence of host-level protocols other than TCP. In the + early development of the Internet suite of protocols, the IP header + fields had been a part of TCP. + + This document describes TCP, which uses TCP headers. + + A TCP header, followed by any user data in the segment, is formatted + as follows, using the style from [66]: + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Source Port | Destination Port | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Sequence Number | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Acknowledgment Number | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Data | |C|E|U|A|P|R|S|F| | + | Offset| Rsrvd |W|C|R|C|S|S|Y|I| Window | + | | |R|E|G|K|H|T|N|N| | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | Checksum | Urgent Pointer | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | [Options] | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | : + : Data : + : | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + Note that one tick mark represents one bit position. + + Figure 1: TCP Header Format + + where: + + Source Port: 16 bits + + The source port number. + + Destination Port: 16 bits + + The destination port number. + + Sequence Number: 32 bits + + The sequence number of the first data octet in this segment (except + when the SYN flag is set). If SYN is set, the sequence number is + the initial sequence number (ISN) and the first data octet is + ISN+1. + + Acknowledgment Number: 32 bits + + If the ACK control bit is set, this field contains the value of the + next sequence number the sender of the segment is expecting to + receive. Once a connection is established, this is always sent. + + Data Offset (DOffset): 4 bits + + The number of 32-bit words in the TCP header. This indicates where + the data begins. The TCP header (even one including options) is an + integer multiple of 32 bits long. + + Reserved (Rsrvd): 4 bits + + A set of control bits reserved for future use. Must be zero in + generated segments and must be ignored in received segments if the + corresponding future features are not implemented by the sending or + receiving host. + + Control bits: The control bits are also known as "flags". + Assignment is managed by IANA from the "TCP Header Flags" registry + [62]. The currently assigned control bits are CWR, ECE, URG, ACK, + PSH, RST, SYN, and FIN. + + CWR: 1 bit + + Congestion Window Reduced (see [6]). + + ECE: 1 bit + + ECN-Echo (see [6]). + + URG: 1 bit + + Urgent pointer field is significant. + + ACK: 1 bit + + Acknowledgment field is significant. + + PSH: 1 bit + + Push function (see the Send Call description in Section 3.9.1). + + RST: 1 bit + + Reset the connection. + + SYN: 1 bit + + Synchronize sequence numbers. + + FIN: 1 bit + + No more data from sender. + + Window: 16 bits + + The number of data octets beginning with the one indicated in the + acknowledgment field that the sender of this segment is willing to + accept. The value is shifted when the window scaling extension is + used [47]. + + The window size MUST be treated as an unsigned number, or else + large window sizes will appear like negative windows and TCP will + not work (MUST-1). It is RECOMMENDED that implementations will + reserve 32-bit fields for the send and receive window sizes in the + connection record and do all window computations with 32 bits (REC- + 1). + + Checksum: 16 bits + + The checksum field is the 16-bit ones' complement of the ones' + complement sum of all 16-bit words in the header and text. The + checksum computation needs to ensure the 16-bit alignment of the + data being summed. If a segment contains an odd number of header + and text octets, alignment can be achieved by padding the last + octet with zeros on its right to form a 16-bit word for checksum + purposes. The pad is not transmitted as part of the segment. + While computing the checksum, the checksum field itself is replaced + with zeros. + + The checksum also covers a pseudo-header (Figure 2) conceptually + prefixed to the TCP header. The pseudo-header is 96 bits for IPv4 + and 320 bits for IPv6. Including the pseudo-header in the checksum + gives the TCP connection protection against misrouted segments. + This information is carried in IP headers and is transferred across + the TCP/network interface in the arguments or results of calls by + the TCP implementation on the IP layer. + + +--------+--------+--------+--------+ + | Source Address | + +--------+--------+--------+--------+ + | Destination Address | + +--------+--------+--------+--------+ + | zero | PTCL | TCP Length | + +--------+--------+--------+--------+ + + Figure 2: IPv4 Pseudo-header + + Pseudo-header components for IPv4: + Source Address: the IPv4 source address in network byte order + + Destination Address: the IPv4 destination address in network + byte order + + zero: bits set to zero + + PTCL: the protocol number from the IP header + + TCP Length: the TCP header length plus the data length in octets + (this is not an explicitly transmitted quantity but is + computed), and it does not count the 12 octets of the pseudo- + header. + + For IPv6, the pseudo-header is defined in Section 8.1 of RFC 8200 + [13] and contains the IPv6 Source Address and Destination Address, + an Upper-Layer Packet Length (a 32-bit value otherwise equivalent + to TCP Length in the IPv4 pseudo-header), three bytes of zero + padding, and a Next Header value, which differs from the IPv6 + header value if there are extension headers present between IPv6 + and TCP. + + The TCP checksum is never optional. The sender MUST generate it + (MUST-2) and the receiver MUST check it (MUST-3). + + Urgent Pointer: 16 bits + + This field communicates the current value of the urgent pointer as + a positive offset from the sequence number in this segment. The + urgent pointer points to the sequence number of the octet following + the urgent data. This field is only to be interpreted in segments + with the URG control bit set. + + Options: [TCP Option]; size(Options) == (DOffset-5)*32; present only + when DOffset > 5. Note that this size expression also includes any + padding trailing the actual options present. + + Options may occupy space at the end of the TCP header and are a + multiple of 8 bits in length. All options are included in the + checksum. An option may begin on any octet boundary. There are + two cases for the format of an option: + + Case 1: A single octet of option-kind. + + Case 2: An octet of option-kind (Kind), an octet of option-length, + and the actual option-data octets. + + The option-length counts the two octets of option-kind and option- + length as well as the option-data octets. + + Note that the list of options may be shorter than the Data Offset + field might imply. The content of the header beyond the End of + Option List Option MUST be header padding of zeros (MUST-69). + + The list of all currently defined options is managed by IANA [62], + and each option is defined in other RFCs, as indicated there. That + set includes experimental options that can be extended to support + multiple concurrent usages [45]. + + A given TCP implementation can support any currently defined + options, but the following options MUST be supported (MUST-4 -- + note Maximum Segment Size Option support is also part of MUST-14 in + Section 3.7.1): + + +======+========+============================+ + | Kind | Length | Meaning | + +======+========+============================+ + | 0 | - | End of Option List Option. | + +------+--------+----------------------------+ + | 1 | - | No-Operation. | + +------+--------+----------------------------+ + | 2 | 4 | Maximum Segment Size. | + +------+--------+----------------------------+ + + Table 1: Mandatory Option Set + + These options are specified in detail in Section 3.2. + + A TCP implementation MUST be able to receive a TCP Option in any + segment (MUST-5). + + A TCP implementation MUST (MUST-6) ignore without error any TCP + Option it does not implement, assuming that the option has a length + field. All TCP Options except End of Option List Option (EOL) and + No-Operation (NOP) MUST have length fields, including all future + options (MUST-68). TCP implementations MUST be prepared to handle + an illegal option length (e.g., zero); a suggested procedure is to + reset the connection and log the error cause (MUST-7). + + Note: There is ongoing work to extend the space available for TCP + Options, such as [65]. + + Data: variable length + + User data carried by the TCP segment. + +3.2. Specific Option Definitions + + A TCP Option, in the mandatory option set, is one of an End of Option + List Option, a No-Operation Option, or a Maximum Segment Size Option. + + An End of Option List Option is formatted as follows: + + 0 + 0 1 2 3 4 5 6 7 + +-+-+-+-+-+-+-+-+ + | 0 | + +-+-+-+-+-+-+-+-+ + + where: + + Kind: 1 byte; Kind == 0. + + This option code indicates the end of the option list. This might + not coincide with the end of the TCP header according to the Data + Offset field. This is used at the end of all options, not the end + of each option, and need only be used if the end of the options + would not otherwise coincide with the end of the TCP header. + + A No-Operation Option is formatted as follows: + + 0 + 0 1 2 3 4 5 6 7 + +-+-+-+-+-+-+-+-+ + | 1 | + +-+-+-+-+-+-+-+-+ + + where: + + Kind: 1 byte; Kind == 1. + + This option code can be used between options, for example, to align + the beginning of a subsequent option on a word boundary. There is + no guarantee that senders will use this option, so receivers MUST + be prepared to process options even if they do not begin on a word + boundary (MUST-64). + + A Maximum Segment Size Option is formatted as follows: + + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | 2 | Length | Maximum Segment Size (MSS) | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + + where: + + Kind: 1 byte; Kind == 2. + + If this option is present, then it communicates the maximum receive + segment size at the TCP endpoint that sends this segment. This + value is limited by the IP reassembly limit. This field may be + sent in the initial connection request (i.e., in segments with the + SYN control bit set) and MUST NOT be sent in other segments (MUST- + 65). If this option is not used, any segment size is allowed. A + more complete description of this option is provided in + Section 3.7.1. + + Length: 1 byte; Length == 4. + + Length of the option in bytes. + + Maximum Segment Size (MSS): 2 bytes. + + The maximum receive segment size at the TCP endpoint that sends + this segment. + +3.2.1. Other Common Options + + Additional RFCs define some other commonly used options that are + recommended to implement for high performance but are not necessary + for basic TCP interoperability. These are the TCP Selective + Acknowledgment (SACK) Option [22] [26], TCP Timestamp (TS) Option + [47], and TCP Window Scale (WS) Option [47]. + +3.2.2. Experimental TCP Options + + Experimental TCP Option values are defined in [30], and [45] + describes the current recommended usage for these experimental + values. + +3.3. TCP Terminology Overview + + This section includes an overview of key terms needed to understand + the detailed protocol operation in the rest of the document. There + is a glossary of terms in Section 4. + +3.3.1. Key Connection State Variables + + Before we can discuss the operation of the TCP implementation in + detail, we need to introduce some detailed terminology. The + maintenance of a TCP connection requires maintaining state for + several variables. We conceive of these variables being stored in a + connection record called a Transmission Control Block or TCB. Among + the variables stored in the TCB are the local and remote IP addresses + and port numbers, the IP security level, and compartment of the + connection (see Appendix A.1), pointers to the user's send and + receive buffers, pointers to the retransmit queue and to the current + segment. In addition, several variables relating to the send and + receive sequence numbers are stored in the TCB. + + +==========+=====================================================+ + | Variable | Description | + +==========+=====================================================+ + | SND.UNA | send unacknowledged | + +----------+-----------------------------------------------------+ + | SND.NXT | send next | + +----------+-----------------------------------------------------+ + | SND.WND | send window | + +----------+-----------------------------------------------------+ + | SND.UP | send urgent pointer | + +----------+-----------------------------------------------------+ + | SND.WL1 | segment sequence number used for last window update | + +----------+-----------------------------------------------------+ + | SND.WL2 | segment acknowledgment number used for last window | + | | update | + +----------+-----------------------------------------------------+ + | ISS | initial send sequence number | + +----------+-----------------------------------------------------+ + + Table 2: Send Sequence Variables + + +==========+=================================+ + | Variable | Description | + +==========+=================================+ + | RCV.NXT | receive next | + +----------+---------------------------------+ + | RCV.WND | receive window | + +----------+---------------------------------+ + | RCV.UP | receive urgent pointer | + +----------+---------------------------------+ + | IRS | initial receive sequence number | + +----------+---------------------------------+ + + Table 3: Receive Sequence Variables + + The following diagrams may help to relate some of these variables to + the sequence space. + + 1 2 3 4 + ----------|----------|----------|---------- + SND.UNA SND.NXT SND.UNA + +SND.WND + + 1 - old sequence numbers that have been acknowledged + 2 - sequence numbers of unacknowledged data + 3 - sequence numbers allowed for new data transmission + 4 - future sequence numbers that are not yet allowed + + Figure 3: Send Sequence Space + + The send window is the portion of the sequence space labeled 3 in + Figure 3. + + 1 2 3 + ----------|----------|---------- + RCV.NXT RCV.NXT + +RCV.WND + + 1 - old sequence numbers that have been acknowledged + 2 - sequence numbers allowed for new reception + 3 - future sequence numbers that are not yet allowed + + Figure 4: Receive Sequence Space + + The receive window is the portion of the sequence space labeled 2 in + Figure 4. + + There are also some variables used frequently in the discussion that + take their values from the fields of the current segment. + + +==========+===============================+ + | Variable | Description | + +==========+===============================+ + | SEG.SEQ | segment sequence number | + +----------+-------------------------------+ + | SEG.ACK | segment acknowledgment number | + +----------+-------------------------------+ + | SEG.LEN | segment length | + +----------+-------------------------------+ + | SEG.WND | segment window | + +----------+-------------------------------+ + | SEG.UP | segment urgent pointer | + +----------+-------------------------------+ + + Table 4: Current Segment Variables + +3.3.2. State Machine Overview + + A connection progresses through a series of states during its + lifetime. The states are: LISTEN, SYN-SENT, SYN-RECEIVED, + ESTABLISHED, FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, + TIME-WAIT, and the fictional state CLOSED. CLOSED is fictional + because it represents the state when there is no TCB, and therefore, + no connection. Briefly the meanings of the states are: + + LISTEN - represents waiting for a connection request from any remote + TCP peer and port. + + SYN-SENT - represents waiting for a matching connection request + after having sent a connection request. + + SYN-RECEIVED - represents waiting for a confirming connection + request acknowledgment after having both received and sent a + connection request. + + ESTABLISHED - represents an open connection, data received can be + delivered to the user. The normal state for the data transfer + phase of the connection. + + FIN-WAIT-1 - represents waiting for a connection termination request + from the remote TCP peer, or an acknowledgment of the connection + termination request previously sent. + + FIN-WAIT-2 - represents waiting for a connection termination request + from the remote TCP peer. + + CLOSE-WAIT - represents waiting for a connection termination request + from the local user. + + CLOSING - represents waiting for a connection termination request + acknowledgment from the remote TCP peer. + + LAST-ACK - represents waiting for an acknowledgment of the + connection termination request previously sent to the remote TCP + peer (this termination request sent to the remote TCP peer already + included an acknowledgment of the termination request sent from + the remote TCP peer). + + TIME-WAIT - represents waiting for enough time to pass to be sure + the remote TCP peer received the acknowledgment of its connection + termination request and to avoid new connections being impacted by + delayed segments from previous connections. + + CLOSED - represents no connection state at all. + + A TCP connection progresses from one state to another in response to + events. The events are the user calls, OPEN, SEND, RECEIVE, CLOSE, + ABORT, and STATUS; the incoming segments, particularly those + containing the SYN, ACK, RST, and FIN flags; and timeouts. + + The OPEN call specifies whether connection establishment is to be + actively pursued, or to be passively waited for. + + A passive OPEN request means that the process wants to accept + incoming connection requests, in contrast to an active OPEN + attempting to initiate a connection. + + The state diagram in Figure 5 illustrates only state changes, + together with the causing events and resulting actions, but addresses + neither error conditions nor actions that are not connected with + state changes. In a later section, more detail is offered with + respect to the reaction of the TCP implementation to events. Some + state names are abbreviated or hyphenated differently in the diagram + from how they appear elsewhere in the document. + + NOTA BENE: This diagram is only a summary and must not be taken as + the total specification. Many details are not included. + + +---------+ ---------\ active OPEN + | CLOSED | \ ----------- + +---------+<---------\ \ create TCB + | ^ \ \ snd SYN + passive OPEN | | CLOSE \ \ + ------------ | | ---------- \ \ + create TCB | | delete TCB \ \ + V | \ \ + rcv RST (note 1) +---------+ CLOSE | \ + -------------------->| LISTEN | ---------- | | + / +---------+ delete TCB | | + / rcv SYN | | SEND | | + / ----------- | | ------- | V + +--------+ snd SYN,ACK / \ snd SYN +--------+ + | |<----------------- ------------------>| | + | SYN | rcv SYN | SYN | + | RCVD |<-----------------------------------------------| SENT | + | | snd SYN,ACK | | + | |------------------ -------------------| | + +--------+ rcv ACK of SYN \ / rcv SYN,ACK +--------+ + | -------------- | | ----------- + | x | | snd ACK + | V V + | CLOSE +---------+ + | ------- | ESTAB | + | snd FIN +---------+ + | CLOSE | | rcv FIN + V ------- | | ------- + +---------+ snd FIN / \ snd ACK +---------+ + | FIN |<---------------- ------------------>| CLOSE | + | WAIT-1 |------------------ | WAIT | + +---------+ rcv FIN \ +---------+ + | rcv ACK of FIN ------- | CLOSE | + | -------------- snd ACK | ------- | + V x V snd FIN V + +---------+ +---------+ +---------+ + |FINWAIT-2| | CLOSING | | LAST-ACK| + +---------+ +---------+ +---------+ + | rcv ACK of FIN | rcv ACK of FIN | + | rcv FIN -------------- | Timeout=2MSL -------------- | + | ------- x V ------------ x V + \ snd ACK +---------+delete TCB +---------+ + -------------------->|TIME-WAIT|------------------->| CLOSED | + +---------+ +---------+ + + Figure 5: TCP Connection State Diagram + + The following notes apply to Figure 5: + + Note 1: The transition from SYN-RECEIVED to LISTEN on receiving a + RST is conditional on having reached SYN-RECEIVED after a passive + OPEN. + + Note 2: The figure omits a transition from FIN-WAIT-1 to TIME-WAIT + if a FIN is received and the local FIN is also acknowledged. + + Note 3: A RST can be sent from any state with a corresponding + transition to TIME-WAIT (see [70] for rationale). These + transitions are not explicitly shown; otherwise, the diagram would + become very difficult to read. Similarly, receipt of a RST from + any state results in a transition to LISTEN or CLOSED, though this + is also omitted from the diagram for legibility. + +3.4. Sequence Numbers + + A fundamental notion in the design is that every octet of data sent + over a TCP connection has a sequence number. Since every octet is + sequenced, each of them can be acknowledged. The acknowledgment + mechanism employed is cumulative so that an acknowledgment of + sequence number X indicates that all octets up to but not including X + have been received. This mechanism allows for straightforward + duplicate detection in the presence of retransmission. The numbering + scheme of octets within a segment is as follows: the first data octet + immediately following the header is the lowest numbered, and the + following octets are numbered consecutively. + + It is essential to remember that the actual sequence number space is + finite, though large. This space ranges from 0 to 2^32 - 1. Since + the space is finite, all arithmetic dealing with sequence numbers + must be performed modulo 2^32. This unsigned arithmetic preserves + the relationship of sequence numbers as they cycle from 2^32 - 1 to 0 + again. There are some subtleties to computer modulo arithmetic, so + great care should be taken in programming the comparison of such + values. The symbol "=<" means "less than or equal" (modulo 2^32). + + The typical kinds of sequence number comparisons that the TCP + implementation must perform include: + + (a) Determining that an acknowledgment refers to some sequence + number sent but not yet acknowledged. + + (b) Determining that all sequence numbers occupied by a segment have + been acknowledged (e.g., to remove the segment from a + retransmission queue). + + (c) Determining that an incoming segment contains sequence numbers + that are expected (i.e., that the segment "overlaps" the receive + window). + + In response to sending data, the TCP endpoint will receive + acknowledgments. The following comparisons are needed to process the + acknowledgments: + + SND.UNA = oldest unacknowledged sequence number + + SND.NXT = next sequence number to be sent + + SEG.ACK = acknowledgment from the receiving TCP peer (next + sequence number expected by the receiving TCP peer) + + SEG.SEQ = first sequence number of a segment + + SEG.LEN = the number of octets occupied by the data in the segment + (counting SYN and FIN) + + SEG.SEQ+SEG.LEN-1 = last sequence number of a segment + + A new acknowledgment (called an "acceptable ack") is one for which + the inequality below holds: + + SND.UNA < SEG.ACK =< SND.NXT + + A segment on the retransmission queue is fully acknowledged if the + sum of its sequence number and length is less than or equal to the + acknowledgment value in the incoming segment. + + When data is received, the following comparisons are needed: + + RCV.NXT = next sequence number expected on an incoming segment, + and is the left or lower edge of the receive window + + RCV.NXT+RCV.WND-1 = last sequence number expected on an incoming + segment, and is the right or upper edge of the receive window + + SEG.SEQ = first sequence number occupied by the incoming segment + + SEG.SEQ+SEG.LEN-1 = last sequence number occupied by the incoming + segment + + A segment is judged to occupy a portion of valid receive sequence + space if + + RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND + + or + + RCV.NXT =< SEG.SEQ+SEG.LEN-1 < RCV.NXT+RCV.WND + + The first part of this test checks to see if the beginning of the + segment falls in the window, the second part of the test checks to + see if the end of the segment falls in the window; if the segment + passes either part of the test, it contains data in the window. + + Actually, it is a little more complicated than this. Due to zero + windows and zero-length segments, we have four cases for the + acceptability of an incoming segment: + + +=========+=========+======================================+ + | Segment | Receive | Test | + | Length | Window | | + +=========+=========+======================================+ + | 0 | 0 | SEG.SEQ = RCV.NXT | + +---------+---------+--------------------------------------+ + | 0 | >0 | RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND | + +---------+---------+--------------------------------------+ + | >0 | 0 | not acceptable | + +---------+---------+--------------------------------------+ + | >0 | >0 | RCV.NXT =< SEG.SEQ < RCV.NXT+RCV.WND | + | | | | + | | | or | + | | | | + | | | RCV.NXT =< SEG.SEQ+SEG.LEN-1 < | + | | | RCV.NXT+RCV.WND | + +---------+---------+--------------------------------------+ + + Table 5: Segment Acceptability Tests + + Note that when the receive window is zero no segments should be + acceptable except ACK segments. Thus, it is possible for a TCP + implementation to maintain a zero receive window while transmitting + data and receiving ACKs. A TCP receiver MUST process the RST and URG + fields of all incoming segments, even when the receive window is zero + (MUST-66). + + We have taken advantage of the numbering scheme to protect certain + control information as well. This is achieved by implicitly + including some control flags in the sequence space so they can be + retransmitted and acknowledged without confusion (i.e., one and only + one copy of the control will be acted upon). Control information is + not physically carried in the segment data space. Consequently, we + must adopt rules for implicitly assigning sequence numbers to + control. The SYN and FIN are the only controls requiring this + protection, and these controls are used only at connection opening + and closing. For sequence number purposes, the SYN is considered to + occur before the first actual data octet of the segment in which it + occurs, while the FIN is considered to occur after the last actual + data octet in a segment in which it occurs. The segment length + (SEG.LEN) includes both data and sequence space-occupying controls. + When a SYN is present, then SEG.SEQ is the sequence number of the + SYN. + +3.4.1. Initial Sequence Number Selection + + A connection is defined by a pair of sockets. Connections can be + reused. New instances of a connection will be referred to as + incarnations of the connection. The problem that arises from this is + -- "how does the TCP implementation identify duplicate segments from + previous incarnations of the connection?" This problem becomes + apparent if the connection is being opened and closed in quick + succession, or if the connection breaks with loss of memory and is + then reestablished. To support this, the TIME-WAIT state limits the + rate of connection reuse, while the initial sequence number selection + described below further protects against ambiguity about which + incarnation of a connection an incoming packet corresponds to. + + To avoid confusion, we must prevent segments from one incarnation of + a connection from being used while the same sequence numbers may + still be present in the network from an earlier incarnation. We want + to assure this even if a TCP endpoint loses all knowledge of the + sequence numbers it has been using. When new connections are + created, an initial sequence number (ISN) generator is employed that + selects a new 32-bit ISN. There are security issues that result if + an off-path attacker is able to predict or guess ISN values [42]. + + TCP initial sequence numbers are generated from a number sequence + that monotonically increases until it wraps, known loosely as a + "clock". This clock is a 32-bit counter that typically increments at + least once every roughly 4 microseconds, although it is neither + assumed to be realtime nor precise, and need not persist across + reboots. The clock component is intended to ensure that with a + Maximum Segment Lifetime (MSL), generated ISNs will be unique since + it cycles approximately every 4.55 hours, which is much longer than + the MSL. Please note that for modern networks that support high data + rates where the connection might start and quickly advance sequence + numbers to overlap within the MSL, it is recommended to implement the + Timestamp Option as mentioned later in Section 3.4.3. + + A TCP implementation MUST use the above type of "clock" for clock- + driven selection of initial sequence numbers (MUST-8), and SHOULD + generate its initial sequence numbers with the expression: + + ISN = M + F(localip, localport, remoteip, remoteport, secretkey) + + where M is the 4 microsecond timer, and F() is a pseudorandom + function (PRF) of the connection's identifying parameters ("localip, + localport, remoteip, remoteport") and a secret key ("secretkey") + (SHLD-1). F() MUST NOT be computable from the outside (MUST-9), or + an attacker could still guess at sequence numbers from the ISN used + for some other connection. The PRF could be implemented as a + cryptographic hash of the concatenation of the TCP connection + parameters and some secret data. For discussion of the selection of + a specific hash algorithm and management of the secret key data, + please see Section 3 of [42]. + + For each connection there is a send sequence number and a receive + sequence number. The initial send sequence number (ISS) is chosen by + the data sending TCP peer, and the initial receive sequence number + (IRS) is learned during the connection-establishing procedure. + + For a connection to be established or initialized, the two TCP peers + must synchronize on each other's initial sequence numbers. This is + done in an exchange of connection-establishing segments carrying a + control bit called "SYN" (for synchronize) and the initial sequence + numbers. As a shorthand, segments carrying the SYN bit are also + called "SYNs". Hence, the solution requires a suitable mechanism for + picking an initial sequence number and a slightly involved handshake + to exchange the ISNs. + + The synchronization requires each side to send its own initial + sequence number and to receive a confirmation of it in acknowledgment + from the remote TCP peer. Each side must also receive the remote + peer's initial sequence number and send a confirming acknowledgment. + + 1) A --> B SYN my sequence number is X + 2) A <-- B ACK your sequence number is X + 3) A <-- B SYN my sequence number is Y + 4) A --> B ACK your sequence number is Y + + Because steps 2 and 3 can be combined in a single message this is + called the three-way (or three message) handshake (3WHS). + + A 3WHS is necessary because sequence numbers are not tied to a global + clock in the network, and TCP implementations may have different + mechanisms for picking the ISNs. The receiver of the first SYN has + no way of knowing whether the segment was an old one or not, unless + it remembers the last sequence number used on the connection (which + is not always possible), and so it must ask the sender to verify this + SYN. The three-way handshake and the advantages of a clock-driven + scheme for ISN selection are discussed in [69]. + +3.4.2. Knowing When to Keep Quiet + + A theoretical problem exists where data could be corrupted due to + confusion between old segments in the network and new ones after a + host reboots if the same port numbers and sequence space are reused. + The "quiet time" concept discussed below addresses this, and the + discussion of it is included for situations where it might be + relevant, although it is not felt to be necessary in most current + implementations. The problem was more relevant earlier in the + history of TCP. In practical use on the Internet today, the error- + prone conditions are sufficiently unlikely that it is safe to ignore. + Reasons why it is now negligible include: (a) ISS and ephemeral port + randomization have reduced likelihood of reuse of port numbers and + sequence numbers after reboots, (b) the effective MSL of the Internet + has declined as links have become faster, and (c) reboots often + taking longer than an MSL anyways. + + To be sure that a TCP implementation does not create a segment + carrying a sequence number that may be duplicated by an old segment + remaining in the network, the TCP endpoint must keep quiet for an MSL + before assigning any sequence numbers upon starting up or recovering + from a situation where memory of sequence numbers in use was lost. + For this specification the MSL is taken to be 2 minutes. This is an + engineering choice, and may be changed if experience indicates it is + desirable to do so. Note that if a TCP endpoint is reinitialized in + some sense, yet retains its memory of sequence numbers in use, then + it need not wait at all; it must only be sure to use sequence numbers + larger than those recently used. + +3.4.3. The TCP Quiet Time Concept + + Hosts that for any reason lose knowledge of the last sequence numbers + transmitted on each active (i.e., not closed) connection shall delay + emitting any TCP segments for at least the agreed MSL in the internet + system that the host is a part of. In the paragraphs below, an + explanation for this specification is given. TCP implementers may + violate the "quiet time" restriction, but only at the risk of causing + some old data to be accepted as new or new data rejected as old + duplicated data by some receivers in the internet system. + + TCP endpoints consume sequence number space each time a segment is + formed and entered into the network output queue at a source host. + The duplicate detection and sequencing algorithm in TCP relies on the + unique binding of segment data to sequence space to the extent that + sequence numbers will not cycle through all 2^32 values before the + segment data bound to those sequence numbers has been delivered and + acknowledged by the receiver and all duplicate copies of the segments + have "drained" from the internet. Without such an assumption, two + distinct TCP segments could conceivably be assigned the same or + overlapping sequence numbers, causing confusion at the receiver as to + which data is new and which is old. Remember that each segment is + bound to as many consecutive sequence numbers as there are octets of + data and SYN or FIN flags in the segment. + + Under normal conditions, TCP implementations keep track of the next + sequence number to emit and the oldest awaiting acknowledgment so as + to avoid mistakenly reusing a sequence number before its first use + has been acknowledged. This alone does not guarantee that old + duplicate data is drained from the net, so the sequence space has + been made large to reduce the probability that a wandering duplicate + will cause trouble upon arrival. At 2 megabits/sec., it takes 4.5 + hours to use up 2^32 octets of sequence space. Since the maximum + segment lifetime in the net is not likely to exceed a few tens of + seconds, this is deemed ample protection for foreseeable nets, even + if data rates escalate to 10s of megabits/sec. At 100 megabits/sec., + the cycle time is 5.4 minutes, which may be a little short but still + within reason. Much higher data rates are possible today, with + implications described in the final paragraph of this subsection. + + The basic duplicate detection and sequencing algorithm in TCP can be + defeated, however, if a source TCP endpoint does not have any memory + of the sequence numbers it last used on a given connection. For + example, if the TCP implementation were to start all connections with + sequence number 0, then upon the host rebooting, a TCP peer might re- + form an earlier connection (possibly after half-open connection + resolution) and emit packets with sequence numbers identical to or + overlapping with packets still in the network, which were emitted on + an earlier incarnation of the same connection. In the absence of + knowledge about the sequence numbers used on a particular connection, + the TCP specification recommends that the source delay for MSL + seconds before emitting segments on the connection, to allow time for + segments from the earlier connection incarnation to drain from the + system. + + Even hosts that can remember the time of day and use it to select + initial sequence number values are not immune from this problem + (i.e., even if time of day is used to select an initial sequence + number for each new connection incarnation). + + Suppose, for example, that a connection is opened starting with + sequence number S. Suppose that this connection is not used much and + that eventually the initial sequence number function (ISN(t)) takes + on a value equal to the sequence number, say S1, of the last segment + sent by this TCP endpoint on a particular connection. Now suppose, + at this instant, the host reboots and establishes a new incarnation + of the connection. The initial sequence number chosen is S1 = ISN(t) + -- last used sequence number on old incarnation of connection! If + the recovery occurs quickly enough, any old duplicates in the net + bearing sequence numbers in the neighborhood of S1 may arrive and be + treated as new packets by the receiver of the new incarnation of the + connection. + + The problem is that the recovering host may not know for how long it + was down between rebooting nor does it know whether there are still + old duplicates in the system from earlier connection incarnations. + + One way to deal with this problem is to deliberately delay emitting + segments for one MSL after recovery from a reboot -- this is the + "quiet time" specification. Hosts that prefer to avoid waiting and + are willing to risk possible confusion of old and new packets at a + given destination may choose not to wait for the "quiet time". + Implementers may provide TCP users with the ability to select on a + connection-by-connection basis whether to wait after a reboot, or may + informally implement the "quiet time" for all connections. + Obviously, even where a user selects to "wait", this is not necessary + after the host has been "up" for at least MSL seconds. + + To summarize: every segment emitted occupies one or more sequence + numbers in the sequence space, and the numbers occupied by a segment + are "busy" or "in use" until MSL seconds have passed. Upon + rebooting, a block of space-time is occupied by the octets and SYN or + FIN flags of any potentially still in-flight segments. If a new + connection is started too soon and uses any of the sequence numbers + in the space-time footprint of those potentially still in-flight + segments of the previous connection incarnation, there is a potential + sequence number overlap area that could cause confusion at the + receiver. + + High-performance cases will have shorter cycle times than those in + the megabits per second that the base TCP design described above + considers. At 1 Gbps, the cycle time is 34 seconds, only 3 seconds + at 10 Gbps, and around a third of a second at 100 Gbps. In these + higher-performance cases, TCP Timestamp Options and Protection + Against Wrapped Sequences (PAWS) [47] provide the needed capability + to detect and discard old duplicates. + +3.5. Establishing a Connection + + The "three-way handshake" is the procedure used to establish a + connection. This procedure normally is initiated by one TCP peer and + responded to by another TCP peer. The procedure also works if two + TCP peers simultaneously initiate the procedure. When simultaneous + open occurs, each TCP peer receives a SYN segment that carries no + acknowledgment after it has sent a SYN. Of course, the arrival of an + old duplicate SYN segment can potentially make it appear, to the + recipient, that a simultaneous connection initiation is in progress. + Proper use of "reset" segments can disambiguate these cases. + + Several examples of connection initiation follow. Although these + examples do not show connection synchronization using data-carrying + segments, this is perfectly legitimate, so long as the receiving TCP + endpoint doesn't deliver the data to the user until it is clear the + data is valid (e.g., the data is buffered at the receiver until the + connection reaches the ESTABLISHED state, given that the three-way + handshake reduces the possibility of false connections). It is a + trade-off between memory and messages to provide information for this + checking. + + The simplest 3WHS is shown in Figure 6. The figures should be + interpreted in the following way. Each line is numbered for + reference purposes. Right arrows (-->) indicate departure of a TCP + segment from TCP Peer A to TCP Peer B or arrival of a segment at B + from A. Left arrows (<--) indicate the reverse. Ellipses (...) + indicate a segment that is still in the network (delayed). Comments + appear in parentheses. TCP connection states represent the state + AFTER the departure or arrival of the segment (whose contents are + shown in the center of each line). Segment contents are shown in + abbreviated form, with sequence number, control flags, and ACK field. + Other fields such as window, addresses, lengths, and text have been + left out in the interest of clarity. + + TCP Peer A TCP Peer B + + 1. CLOSED LISTEN + + 2. SYN-SENT --> --> SYN-RECEIVED + + 3. ESTABLISHED <-- <-- SYN-RECEIVED + + 4. ESTABLISHED --> --> ESTABLISHED + + 5. ESTABLISHED --> --> ESTABLISHED + + Figure 6: Basic Three-Way Handshake for Connection Synchronization + + In line 2 of Figure 6, TCP Peer A begins by sending a SYN segment + indicating that it will use sequence numbers starting with sequence + number 100. In line 3, TCP Peer B sends a SYN and acknowledges the + SYN it received from TCP Peer A. Note that the acknowledgment field + indicates TCP Peer B is now expecting to hear sequence 101, + acknowledging the SYN that occupied sequence 100. + + At line 4, TCP Peer A responds with an empty segment containing an + ACK for TCP Peer B's SYN; and in line 5, TCP Peer A sends some data. + Note that the sequence number of the segment in line 5 is the same as + in line 4 because the ACK does not occupy sequence number space (if + it did, we would wind up ACKing ACKs!). + + Simultaneous initiation is only slightly more complex, as is shown in + Figure 7. Each TCP peer's connection state cycles from CLOSED to + SYN-SENT to SYN-RECEIVED to ESTABLISHED. + + TCP Peer A TCP Peer B + + 1. CLOSED CLOSED + + 2. SYN-SENT --> ... + + 3. SYN-RECEIVED <-- <-- SYN-SENT + + 4. ... --> SYN-RECEIVED + + 5. SYN-RECEIVED --> ... + + 6. ESTABLISHED <-- <-- SYN-RECEIVED + + 7. ... --> ESTABLISHED + + Figure 7: Simultaneous Connection Synchronization + + A TCP implementation MUST support simultaneous open attempts (MUST- + 10). + + Note that a TCP implementation MUST keep track of whether a + connection has reached SYN-RECEIVED state as the result of a passive + OPEN or an active OPEN (MUST-11). + + The principal reason for the three-way handshake is to prevent old + duplicate connection initiations from causing confusion. To deal + with this, a special control message, reset, is specified. If the + receiving TCP peer is in a non-synchronized state (i.e., SYN-SENT, + SYN-RECEIVED), it returns to LISTEN on receiving an acceptable reset. + If the TCP peer is in one of the synchronized states (ESTABLISHED, + FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT), it + aborts the connection and informs its user. We discuss this latter + case under "half-open" connections below. + + TCP Peer A TCP Peer B + + 1. CLOSED LISTEN + + 2. SYN-SENT --> ... + + 3. (duplicate) ... --> SYN-RECEIVED + + 4. SYN-SENT <-- <-- SYN-RECEIVED + + 5. SYN-SENT --> --> LISTEN + + 6. ... --> SYN-RECEIVED + + 7. ESTABLISHED <-- <-- SYN-RECEIVED + + 8. ESTABLISHED --> --> ESTABLISHED + + Figure 8: Recovery from Old Duplicate SYN + + As a simple example of recovery from old duplicates, consider + Figure 8. At line 3, an old duplicate SYN arrives at TCP Peer B. + TCP Peer B cannot tell that this is an old duplicate, so it responds + normally (line 4). TCP Peer A detects that the ACK field is + incorrect and returns a RST (reset) with its SEQ field selected to + make the segment believable. TCP Peer B, on receiving the RST, + returns to the LISTEN state. When the original SYN finally arrives + at line 6, the synchronization proceeds normally. If the SYN at line + 6 had arrived before the RST, a more complex exchange might have + occurred with RSTs sent in both directions. + +3.5.1. Half-Open Connections and Other Anomalies + + An established connection is said to be "half-open" if one of the TCP + peers has closed or aborted the connection at its end without the + knowledge of the other, or if the two ends of the connection have + become desynchronized owing to a failure or reboot that resulted in + loss of memory. Such connections will automatically become reset if + an attempt is made to send data in either direction. However, half- + open connections are expected to be unusual. + + If at site A the connection no longer exists, then an attempt by the + user at site B to send any data on it will result in the site B TCP + endpoint receiving a reset control message. Such a message indicates + to the site B TCP endpoint that something is wrong, and it is + expected to abort the connection. + + Assume that two user processes A and B are communicating with one + another when a failure or reboot occurs causing loss of memory to A's + TCP implementation. Depending on the operating system supporting A's + TCP implementation, it is likely that some error recovery mechanism + exists. When the TCP endpoint is up again, A is likely to start + again from the beginning or from a recovery point. As a result, A + will probably try to OPEN the connection again or try to SEND on the + connection it believes open. In the latter case, it receives the + error message "connection not open" from the local (A's) TCP + implementation. In an attempt to establish the connection, A's TCP + implementation will send a segment containing SYN. This scenario + leads to the example shown in Figure 9. After TCP Peer A reboots, + the user attempts to reopen the connection. TCP Peer B, in the + meantime, thinks the connection is open. + + TCP Peer A TCP Peer B + + 1. (REBOOT) (send 300,receive 100) + + 2. CLOSED ESTABLISHED + + 3. SYN-SENT --> --> (??) + + 4. (!!) <-- <-- ESTABLISHED + + 5. SYN-SENT --> --> (Abort!!) + + 6. SYN-SENT CLOSED + + 7. SYN-SENT --> --> + + Figure 9: Half-Open Connection Discovery + + When the SYN arrives at line 3, TCP Peer B, being in a synchronized + state, and the incoming segment outside the window, responds with an + acknowledgment indicating what sequence it next expects to hear (ACK + 100). TCP Peer A sees that this segment does not acknowledge + anything it sent and, being unsynchronized, sends a reset (RST) + because it has detected a half-open connection. TCP Peer B aborts at + line 5. TCP Peer A will continue to try to establish the connection; + the problem is now reduced to the basic three-way handshake of + Figure 6. + + An interesting alternative case occurs when TCP Peer A reboots and + TCP Peer B tries to send data on what it thinks is a synchronized + connection. This is illustrated in Figure 10. In this case, the + data arriving at TCP Peer A from TCP Peer B (line 2) is unacceptable + because no such connection exists, so TCP Peer A sends a RST. The + RST is acceptable so TCP Peer B processes it and aborts the + connection. + + TCP Peer A TCP Peer B + + 1. (REBOOT) (send 300,receive 100) + + 2. (??) <-- <-- ESTABLISHED + + 3. --> --> (ABORT!!) + + Figure 10: Active Side Causes Half-Open Connection Discovery + + In Figure 11, two TCP Peers A and B with passive connections waiting + for SYN are depicted. An old duplicate arriving at TCP Peer B (line + 2) stirs B into action. A SYN-ACK is returned (line 3) and causes + TCP A to generate a RST (the ACK in line 3 is not acceptable). TCP + Peer B accepts the reset and returns to its passive LISTEN state. + + TCP Peer A TCP Peer B + + 1. LISTEN LISTEN + + 2. ... --> SYN-RECEIVED + + 3. (??) <-- <-- SYN-RECEIVED + + 4. --> --> (return to LISTEN!) + + 5. LISTEN LISTEN + + Figure 11: Old Duplicate SYN Initiates a Reset on Two Passive Sockets + + A variety of other cases are possible, all of which are accounted for + by the following rules for RST generation and processing. + +3.5.2. Reset Generation + + A TCP user or application can issue a reset on a connection at any + time, though reset events are also generated by the protocol itself + when various error conditions occur, as described below. The side of + a connection issuing a reset should enter the TIME-WAIT state, as + this generally helps to reduce the load on busy servers for reasons + described in [70]. + + As a general rule, reset (RST) is sent whenever a segment arrives + that apparently is not intended for the current connection. A reset + must not be sent if it is not clear that this is the case. + + There are three groups of states: + + 1. If the connection does not exist (CLOSED), then a reset is sent + in response to any incoming segment except another reset. A SYN + segment that does not match an existing connection is rejected by + this means. + + If the incoming segment has the ACK bit set, the reset takes its + sequence number from the ACK field of the segment; otherwise, the + reset has sequence number zero and the ACK field is set to the + sum of the sequence number and segment length of the incoming + segment. The connection remains in the CLOSED state. + + 2. If the connection is in any non-synchronized state (LISTEN, SYN- + SENT, SYN-RECEIVED), and the incoming segment acknowledges + something not yet sent (the segment carries an unacceptable ACK), + or if an incoming segment has a security level or compartment + (Appendix A.1) that does not exactly match the level and + compartment requested for the connection, a reset is sent. + + If the incoming segment has an ACK field, the reset takes its + sequence number from the ACK field of the segment; otherwise, the + reset has sequence number zero and the ACK field is set to the + sum of the sequence number and segment length of the incoming + segment. The connection remains in the same state. + + 3. If the connection is in a synchronized state (ESTABLISHED, FIN- + WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT), + any unacceptable segment (out-of-window sequence number or + unacceptable acknowledgment number) must be responded to with an + empty acknowledgment segment (without any user data) containing + the current send sequence number and an acknowledgment indicating + the next sequence number expected to be received, and the + connection remains in the same state. + + If an incoming segment has a security level or compartment that + does not exactly match the level and compartment requested for + the connection, a reset is sent and the connection goes to the + CLOSED state. The reset takes its sequence number from the ACK + field of the incoming segment. + +3.5.3. Reset Processing + + In all states except SYN-SENT, all reset (RST) segments are validated + by checking their SEQ fields. A reset is valid if its sequence + number is in the window. In the SYN-SENT state (a RST received in + response to an initial SYN), the RST is acceptable if the ACK field + acknowledges the SYN. + + The receiver of a RST first validates it, then changes state. If the + receiver was in the LISTEN state, it ignores it. If the receiver was + in SYN-RECEIVED state and had previously been in the LISTEN state, + then the receiver returns to the LISTEN state; otherwise, the + receiver aborts the connection and goes to the CLOSED state. If the + receiver was in any other state, it aborts the connection and advises + the user and goes to the CLOSED state. + + TCP implementations SHOULD allow a received RST segment to include + data (SHLD-2). It has been suggested that a RST segment could + contain diagnostic data that explains the cause of the RST. No + standard has yet been established for such data. + +3.6. Closing a Connection + + CLOSE is an operation meaning "I have no more data to send." The + notion of closing a full-duplex connection is subject to ambiguous + interpretation, of course, since it may not be obvious how to treat + the receiving side of the connection. We have chosen to treat CLOSE + in a simplex fashion. The user who CLOSEs may continue to RECEIVE + until the TCP receiver is told that the remote peer has CLOSED also. + Thus, a program could initiate several SENDs followed by a CLOSE, and + then continue to RECEIVE until signaled that a RECEIVE failed because + the remote peer has CLOSED. The TCP implementation will signal a + user, even if no RECEIVEs are outstanding, that the remote peer has + closed, so the user can terminate their side gracefully. A TCP + implementation will reliably deliver all buffers SENT before the + connection was CLOSED so a user who expects no data in return need + only wait to hear the connection was CLOSED successfully to know that + all their data was received at the destination TCP endpoint. Users + must keep reading connections they close for sending until the TCP + implementation indicates there is no more data. + + There are essentially three cases: + + 1) The user initiates by telling the TCP implementation to CLOSE the + connection (TCP Peer A in Figure 12). + + 2) The remote TCP endpoint initiates by sending a FIN control signal + (TCP Peer B in Figure 12). + + 3) Both users CLOSE simultaneously (Figure 13). + + Case 1: Local user initiates the close + + In this case, a FIN segment can be constructed and placed on the + outgoing segment queue. No further SENDs from the user will be + accepted by the TCP implementation, and it enters the FIN-WAIT-1 + state. RECEIVEs are allowed in this state. All segments + preceding and including FIN will be retransmitted until + acknowledged. When the other TCP peer has both acknowledged the + FIN and sent a FIN of its own, the first TCP peer can ACK this + FIN. Note that a TCP endpoint receiving a FIN will ACK but not + send its own FIN until its user has CLOSED the connection also. + + Case 2: TCP endpoint receives a FIN from the network + + If an unsolicited FIN arrives from the network, the receiving TCP + endpoint can ACK it and tell the user that the connection is + closing. The user will respond with a CLOSE, upon which the TCP + endpoint can send a FIN to the other TCP peer after sending any + remaining data. The TCP endpoint then waits until its own FIN is + acknowledged whereupon it deletes the connection. If an ACK is + not forthcoming, after the user timeout the connection is aborted + and the user is told. + + Case 3: Both users close simultaneously + + A simultaneous CLOSE by users at both ends of a connection causes + FIN segments to be exchanged (Figure 13). When all segments + preceding the FINs have been processed and acknowledged, each TCP + peer can ACK the FIN it has received. Both will, upon receiving + these ACKs, delete the connection. + + TCP Peer A TCP Peer B + + 1. ESTABLISHED ESTABLISHED + + 2. (Close) + FIN-WAIT-1 --> --> CLOSE-WAIT + + 3. FIN-WAIT-2 <-- <-- CLOSE-WAIT + + 4. (Close) + TIME-WAIT <-- <-- LAST-ACK + + 5. TIME-WAIT --> --> CLOSED + + 6. (2 MSL) + CLOSED + + Figure 12: Normal Close Sequence + + TCP Peer A TCP Peer B + + 1. ESTABLISHED ESTABLISHED + + 2. (Close) (Close) + FIN-WAIT-1 --> ... FIN-WAIT-1 + <-- <-- + ... --> + + 3. CLOSING --> ... CLOSING + <-- <-- + ... --> + + 4. TIME-WAIT TIME-WAIT + (2 MSL) (2 MSL) + CLOSED CLOSED + + Figure 13: Simultaneous Close Sequence + + A TCP connection may terminate in two ways: (1) the normal TCP close + sequence using a FIN handshake (Figure 12), and (2) an "abort" in + which one or more RST segments are sent and the connection state is + immediately discarded. If the local TCP connection is closed by the + remote side due to a FIN or RST received from the remote side, then + the local application MUST be informed whether it closed normally or + was aborted (MUST-12). + + +3.6.1. Half-Closed Connections + + The normal TCP close sequence delivers buffered data reliably in both + directions. Since the two directions of a TCP connection are closed + independently, it is possible for a connection to be "half closed", + i.e., closed in only one direction, and a host is permitted to + continue sending data in the open direction on a half-closed + connection. + + A host MAY implement a "half-duplex" TCP close sequence, so that an + application that has called CLOSE cannot continue to read data from + the connection (MAY-1). If such a host issues a CLOSE call while + received data is still pending in the TCP connection, or if new data + is received after CLOSE is called, its TCP implementation SHOULD send + a RST to show that data was lost (SHLD-3). See [23], Section 2.17 + for discussion. + + When a connection is closed actively, it MUST linger in the TIME-WAIT + state for a time 2xMSL (Maximum Segment Lifetime) (MUST-13). + However, it MAY accept a new SYN from the remote TCP endpoint to + reopen the connection directly from TIME-WAIT state (MAY-2), if it: + + (1) assigns its initial sequence number for the new connection to be + larger than the largest sequence number it used on the previous + connection incarnation, and + + (2) returns to TIME-WAIT state if the SYN turns out to be an old + duplicate. + + When the TCP Timestamp Options are available, an improved algorithm + is described in [40] in order to support higher connection + establishment rates. This algorithm for reducing TIME-WAIT is a Best + Current Practice that SHOULD be implemented since Timestamp Options + are commonly used, and using them to reduce TIME-WAIT provides + benefits for busy Internet servers (SHLD-4). + +3.7. Segmentation + + The term "segmentation" refers to the activity TCP performs when + ingesting a stream of bytes from a sending application and + packetizing that stream of bytes into TCP segments. Individual TCP + segments often do not correspond one-for-one to individual send (or + socket write) calls from the application. Applications may perform + writes at the granularity of messages in the upper-layer protocol, + but TCP guarantees no correlation between the boundaries of TCP + segments sent and received and the boundaries of the read or write + buffers of user application data. In some specific protocols, such + as Remote Direct Memory Access (RDMA) using Direct Data Placement + (DDP) and Marker PDU Aligned Framing (MPA) [34], there are + performance optimizations possible when the relation between TCP + segments and application data units can be controlled, and MPA + includes a specific mechanism for detecting and verifying this + relationship between TCP segments and application message data + structures, but this is specific to applications like RDMA. In + general, multiple goals influence the sizing of TCP segments created + by a TCP implementation. + + Goals driving the sending of larger segments include: + + * Reducing the number of packets in flight within the network. + + * Increasing processing efficiency and potential performance by + enabling a smaller number of interrupts and inter-layer + interactions. + + * Limiting the overhead of TCP headers. + + Note that the performance benefits of sending larger segments may + decrease as the size increases, and there may be boundaries where + advantages are reversed. For instance, on some implementation + architectures, 1025 bytes within a segment could lead to worse + performance than 1024 bytes, due purely to data alignment on copy + operations. + + Goals driving the sending of smaller segments include: + + * Avoiding sending a TCP segment that would result in an IP datagram + larger than the smallest MTU along an IP network path because this + results in either packet loss or packet fragmentation. Making + matters worse, some firewalls or middleboxes may drop fragmented + packets or ICMP messages related to fragmentation. + + * Preventing delays to the application data stream, especially when + TCP is waiting on the application to generate more data, or when + the application is waiting on an event or input from its peer in + order to generate more data. + + * Enabling "fate sharing" between TCP segments and lower-layer data + units (e.g., below IP, for links with cell or frame sizes smaller + than the IP MTU). + + Towards meeting these competing sets of goals, TCP includes several + mechanisms, including the Maximum Segment Size Option, Path MTU + Discovery, the Nagle algorithm, and support for IPv6 Jumbograms, as + discussed in the following subsections. + +3.7.1. Maximum Segment Size Option + + TCP endpoints MUST implement both sending and receiving the MSS + Option (MUST-14). + + TCP implementations SHOULD send an MSS Option in every SYN segment + when its receive MSS differs from the default 536 for IPv4 or 1220 + for IPv6 (SHLD-5), and MAY send it always (MAY-3). + + If an MSS Option is not received at connection setup, TCP + implementations MUST assume a default send MSS of 536 (576 - 40) for + IPv4 or 1220 (1280 - 60) for IPv6 (MUST-15). + + The maximum size of a segment that a TCP endpoint really sends, the + "effective send MSS", MUST be the smaller (MUST-16) of the send MSS + (that reflects the available reassembly buffer size at the remote + host, the EMTU_R [19]) and the largest transmission size permitted by + the IP layer (EMTU_S [19]): + + Eff.snd.MSS = min(SendMSS+20, MMS_S) - TCPhdrsize - IPoptionsize + + where: + + * SendMSS is the MSS value received from the remote host, or the + default 536 for IPv4 or 1220 for IPv6, if no MSS Option is + received. + + * MMS_S is the maximum size for a transport-layer message that TCP + may send. + + * TCPhdrsize is the size of the fixed TCP header and any options. + This is 20 in the (rare) case that no options are present but may + be larger if TCP Options are to be sent. Note that some options + might not be included on all segments, but that for each segment + sent, the sender should adjust the data length accordingly, within + the Eff.snd.MSS. + + * IPoptionsize is the size of any IPv4 options or IPv6 extension + headers associated with a TCP connection. Note that some options + or extension headers might not be included on all packets, but + that for each segment sent, the sender should adjust the data + length accordingly, within the Eff.snd.MSS. + + The MSS value to be sent in an MSS Option should be equal to the + effective MTU minus the fixed IP and TCP headers. By ignoring both + IP and TCP Options when calculating the value for the MSS Option, if + there are any IP or TCP Options to be sent in a packet, then the + sender must decrease the size of the TCP data accordingly. RFC 6691 + [43] discusses this in greater detail. + + The MSS value to be sent in an MSS Option must be less than or equal + to: + + MMS_R - 20 + + where MMS_R is the maximum size for a transport-layer message that + can be received (and reassembled at the IP layer) (MUST-67). TCP + obtains MMS_R and MMS_S from the IP layer; see the generic call + GET_MAXSIZES in Section 3.4 of RFC 1122. These are defined in terms + of their IP MTU equivalents, EMTU_R and EMTU_S [19]. + + When TCP is used in a situation where either the IP or TCP headers + are not fixed, the sender must reduce the amount of TCP data in any + given packet by the number of octets used by the IP and TCP options. + This has been a point of confusion historically, as explained in RFC + 6691, Section 3.1. + +3.7.2. Path MTU Discovery + + A TCP implementation may be aware of the MTU on directly connected + links, but will rarely have insight about MTUs across an entire + network path. For IPv4, RFC 1122 recommends an IP-layer default + effective MTU of less than or equal to 576 for destinations not + directly connected, and for IPv6 this would be 1280. Using these + fixed values limits TCP connection performance and efficiency. + Instead, implementation of Path MTU Discovery (PMTUD) and + Packetization Layer Path MTU Discovery (PLPMTUD) is strongly + recommended in order for TCP to improve segmentation decisions. Both + PMTUD and PLPMTUD help TCP choose segment sizes that avoid both on- + path (for IPv4) and source fragmentation (IPv4 and IPv6). + + PMTUD for IPv4 [2] or IPv6 [14] is implemented in conjunction between + TCP, IP, and ICMP. It relies both on avoiding source fragmentation + and setting the IPv4 DF (don't fragment) flag, the latter to inhibit + on-path fragmentation. It relies on ICMP errors from routers along + the path whenever a segment is too large to traverse a link. Several + adjustments to a TCP implementation with PMTUD are described in RFC + 2923 in order to deal with problems experienced in practice [27]. + PLPMTUD [31] is a Standards Track improvement to PMTUD that relaxes + the requirement for ICMP support across a path, and improves + performance in cases where ICMP is not consistently conveyed, but + still tries to avoid source fragmentation. The mechanisms in all + four of these RFCs are recommended to be included in TCP + implementations. + + The TCP MSS Option specifies an upper bound for the size of packets + that can be received (see [43]). Hence, setting the value in the MSS + Option too small can impact the ability for PMTUD or PLPMTUD to find + a larger path MTU. RFC 1191 discusses this implication of many older + TCP implementations setting the TCP MSS to 536 (corresponding to the + IPv4 576 byte default MTU) for non-local destinations, rather than + deriving it from the MTUs of connected interfaces as recommended. + +3.7.3. Interfaces with Variable MTU Values + + The effective MTU can sometimes vary, as when used with variable + compression, e.g., RObust Header Compression (ROHC) [37]. It is + tempting for a TCP implementation to advertise the largest possible + MSS, to support the most efficient use of compressed payloads. + Unfortunately, some compression schemes occasionally need to transmit + full headers (and thus smaller payloads) to resynchronize state at + their endpoint compressors/decompressors. If the largest MTU is used + to calculate the value to advertise in the MSS Option, TCP + retransmission may interfere with compressor resynchronization. + + As a result, when the effective MTU of an interface varies packet-to- + packet, TCP implementations SHOULD use the smallest effective MTU of + the interface to calculate the value to advertise in the MSS Option + (SHLD-6). + +3.7.4. Nagle Algorithm + + The "Nagle algorithm" was described in RFC 896 [17] and was + recommended in RFC 1122 [19] for mitigation of an early problem of + too many small packets being generated. It has been implemented in + most current TCP code bases, sometimes with minor variations (see + Appendix A.3). + + If there is unacknowledged data (i.e., SND.NXT > SND.UNA), then the + sending TCP endpoint buffers all user data (regardless of the PSH + bit) until the outstanding data has been acknowledged or until the + TCP endpoint can send a full-sized segment (Eff.snd.MSS bytes). + + A TCP implementation SHOULD implement the Nagle algorithm to coalesce + short segments (SHLD-7). However, there MUST be a way for an + application to disable the Nagle algorithm on an individual + connection (MUST-17). In all cases, sending data is also subject to + the limitation imposed by the slow start algorithm [8]. + + Since there can be problematic interactions between the Nagle + algorithm and delayed acknowledgments, some implementations use minor + variations of the Nagle algorithm, such as the one described in + Appendix A.3. + +3.7.5. IPv6 Jumbograms + + In order to support TCP over IPv6 Jumbograms, implementations need to + be able to send TCP segments larger than the 64-KB limit that the MSS + Option can convey. RFC 2675 [24] defines that an MSS value of 65,535 + bytes is to be treated as infinity, and Path MTU Discovery [14] is + used to determine the actual MSS. + + The Jumbo Payload Option need not be implemented or understood by + IPv6 nodes that do not support attachment to links with an MTU + greater than 65,575 [24], and the present IPv6 Node Requirements does + not include support for Jumbograms [55]. + +3.8. Data Communication + + Once the connection is established, data is communicated by the + exchange of segments. Because segments may be lost due to errors + (checksum test failure) or network congestion, TCP uses + retransmission to ensure delivery of every segment. Duplicate + segments may arrive due to network or TCP retransmission. As + discussed in the section on sequence numbers (Section 3.4), the TCP + implementation performs certain tests on the sequence and + acknowledgment numbers in the segments to verify their acceptability. + + The sender of data keeps track of the next sequence number to use in + the variable SND.NXT. The receiver of data keeps track of the next + sequence number to expect in the variable RCV.NXT. The sender of + data keeps track of the oldest unacknowledged sequence number in the + variable SND.UNA. If the data flow is momentarily idle and all data + sent has been acknowledged, then the three variables will be equal. + + When the sender creates a segment and transmits it, the sender + advances SND.NXT. When the receiver accepts a segment, it advances + RCV.NXT and sends an acknowledgment. When the data sender receives + an acknowledgment, it advances SND.UNA. The extent to which the + values of these variables differ is a measure of the delay in the + communication. The amount by which the variables are advanced is the + length of the data and SYN or FIN flags in the segment. Note that, + once in the ESTABLISHED state, all segments must carry current + acknowledgment information. + + The CLOSE user call implies a push function (see Section 3.9.1), as + does the FIN control flag in an incoming segment. + +3.8.1. Retransmission Timeout + + Because of the variability of the networks that compose an + internetwork system and the wide range of uses of TCP connections, + the retransmission timeout (RTO) must be dynamically determined. + + The RTO MUST be computed according to the algorithm in [10], + including Karn's algorithm for taking RTT samples (MUST-18). + + RFC 793 contains an early example procedure for computing the RTO, + based on work mentioned in IEN 177 [71]. This was then replaced by + the algorithm described in RFC 1122, which was subsequently updated + in RFC 2988 and then again in RFC 6298. + + RFC 1122 allows that if a retransmitted packet is identical to the + original packet (which implies not only that the data boundaries have + not changed, but also that none of the headers have changed), then + the same IPv4 Identification field MAY be used (see Section 3.2.1.5 + of RFC 1122) (MAY-4). The same IP Identification field may be reused + anyways since it is only meaningful when a datagram is fragmented + [44]. TCP implementations should not rely on or typically interact + with this IPv4 header field in any way. It is not a reasonable way + to indicate duplicate sent segments nor to identify duplicate + received segments. + +3.8.2. TCP Congestion Control + + RFC 2914 [5] explains the importance of congestion control for the + Internet. + + RFC 1122 required implementation of Van Jacobson's congestion control + algorithms slow start and congestion avoidance together with + exponential backoff for successive RTO values for the same segment. + RFC 2581 provided IETF Standards Track description of slow start and + congestion avoidance, along with fast retransmit and fast recovery. + RFC 5681 is the current description of these algorithms and is the + current Standards Track specification providing guidelines for TCP + congestion control. RFC 6298 describes exponential backoff of RTO + values, including keeping the backed-off value until a subsequent + segment with new data has been sent and acknowledged without + retransmission. + + A TCP endpoint MUST implement the basic congestion control algorithms + slow start, congestion avoidance, and exponential backoff of RTO to + avoid creating congestion collapse conditions (MUST-19). RFC 5681 + and RFC 6298 describe the basic algorithms on the IETF Standards + Track that are broadly applicable. Multiple other suitable + algorithms exist and have been widely used. Many TCP implementations + support a set of alternative algorithms that can be configured for + use on the endpoint. An endpoint MAY implement such alternative + algorithms provided that the algorithms are conformant with the TCP + specifications from the IETF Standards Track as described in RFC + 2914, RFC 5033 [7], and RFC 8961 [15] (MAY-18). + + Explicit Congestion Notification (ECN) was defined in RFC 3168 and is + an IETF Standards Track enhancement that has many benefits [51]. + + A TCP endpoint SHOULD implement ECN as described in RFC 3168 (SHLD- + 8). + +3.8.3. TCP Connection Failures + + Excessive retransmission of the same segment by a TCP endpoint + indicates some failure of the remote host or the internetwork path. + This failure may be of short or long duration. The following + procedure MUST be used to handle excessive retransmissions of data + segments (MUST-20): + + (a) There are two thresholds R1 and R2 measuring the amount of + retransmission that has occurred for the same segment. R1 and + R2 might be measured in time units or as a count of + retransmissions (with the current RTO and corresponding backoffs + as a conversion factor, if needed). + + (b) When the number of transmissions of the same segment reaches or + exceeds threshold R1, pass negative advice (see Section 3.3.1.4 + of [19]) to the IP layer, to trigger dead-gateway diagnosis. + + (c) When the number of transmissions of the same segment reaches a + threshold R2 greater than R1, close the connection. + + (d) An application MUST (MUST-21) be able to set the value for R2 + for a particular connection. For example, an interactive + application might set R2 to "infinity", giving the user control + over when to disconnect. + + (e) TCP implementations SHOULD inform the application of the + delivery problem (unless such information has been disabled by + the application; see the "Asynchronous Reports" section + (Section 3.9.1.8)), when R1 is reached and before R2 (SHLD-9). + This will allow a remote login application program to inform the + user, for example. + + The value of R1 SHOULD correspond to at least 3 retransmissions, at + the current RTO (SHLD-10). The value of R2 SHOULD correspond to at + least 100 seconds (SHLD-11). + + An attempt to open a TCP connection could fail with excessive + retransmissions of the SYN segment or by receipt of a RST segment or + an ICMP Port Unreachable. SYN retransmissions MUST be handled in the + general way just described for data retransmissions, including + notification of the application layer. + + However, the values of R1 and R2 may be different for SYN and data + segments. In particular, R2 for a SYN segment MUST be set large + enough to provide retransmission of the segment for at least 3 + minutes (MUST-23). The application can close the connection (i.e., + give up on the open attempt) sooner, of course. + +3.8.4. TCP Keep-Alives + + A TCP connection is said to be "idle" if for some long amount of time + there have been no incoming segments received and there is no new or + unacknowledged data to be sent. + + Implementers MAY include "keep-alives" in their TCP implementations + (MAY-5), although this practice is not universally accepted. Some + TCP implementations, however, have included a keep-alive mechanism. + To confirm that an idle connection is still active, these + implementations send a probe segment designed to elicit a response + from the TCP peer. Such a segment generally contains SEG.SEQ = + SND.NXT-1 and may or may not contain one garbage octet of data. If + keep-alives are included, the application MUST be able to turn them + on or off for each TCP connection (MUST-24), and they MUST default to + off (MUST-25). + + Keep-alive packets MUST only be sent when no sent data is + outstanding, and no data or acknowledgment packets have been received + for the connection within an interval (MUST-26). This interval MUST + be configurable (MUST-27) and MUST default to no less than two hours + (MUST-28). + + It is extremely important to remember that ACK segments that contain + no data are not reliably transmitted by TCP. Consequently, if a + keep-alive mechanism is implemented it MUST NOT interpret failure to + respond to any specific probe as a dead connection (MUST-29). + + An implementation SHOULD send a keep-alive segment with no data + (SHLD-12); however, it MAY be configurable to send a keep-alive + segment containing one garbage octet (MAY-6), for compatibility with + erroneous TCP implementations. + +3.8.5. The Communication of Urgent Information + + As a result of implementation differences and middlebox interactions, + new applications SHOULD NOT employ the TCP urgent mechanism (SHLD- + 13). However, TCP implementations MUST still include support for the + urgent mechanism (MUST-30). Information on how some TCP + implementations interpret the urgent pointer can be found in RFC 6093 + [39]. + + The objective of the TCP urgent mechanism is to allow the sending + user to stimulate the receiving user to accept some urgent data and + to permit the receiving TCP endpoint to indicate to the receiving + user when all the currently known urgent data has been received by + the user. + + This mechanism permits a point in the data stream to be designated as + the end of urgent information. Whenever this point is in advance of + the receive sequence number (RCV.NXT) at the receiving TCP endpoint, + then the TCP implementation must tell the user to go into "urgent + mode"; when the receive sequence number catches up to the urgent + pointer, the TCP implementation must tell user to go into "normal + mode". If the urgent pointer is updated while the user is in "urgent + mode", the update will be invisible to the user. + + The method employs an urgent field that is carried in all segments + transmitted. The URG control flag indicates that the urgent field is + meaningful and must be added to the segment sequence number to yield + the urgent pointer. The absence of this flag indicates that there is + no urgent data outstanding. + + To send an urgent indication, the user must also send at least one + data octet. If the sending user also indicates a push, timely + delivery of the urgent information to the destination process is + enhanced. Note that because changes in the urgent pointer correspond + to data being written by a sending application, the urgent pointer + cannot "recede" in the sequence space, but a TCP receiver should be + robust to invalid urgent pointer values. + + A TCP implementation MUST support a sequence of urgent data of any + length (MUST-31) [19]. + + The urgent pointer MUST point to the sequence number of the octet + following the urgent data (MUST-62). + + A TCP implementation MUST (MUST-32) inform the application layer + asynchronously whenever it receives an urgent pointer and there was + previously no pending urgent data, or whenever the urgent pointer + advances in the data stream. The TCP implementation MUST (MUST-33) + provide a way for the application to learn how much urgent data + remains to be read from the connection, or at least to determine + whether more urgent data remains to be read [19]. + +3.8.6. Managing the Window + + The window sent in each segment indicates the range of sequence + numbers the sender of the window (the data receiver) is currently + prepared to accept. There is an assumption that this is related to + the data buffer space currently available for this connection. + + The sending TCP endpoint packages the data to be transmitted into + segments that fit the current window, and may repackage segments on + the retransmission queue. Such repackaging is not required but may + be helpful. + + In a connection with a one-way data flow, the window information will + be carried in acknowledgment segments that all have the same sequence + number, so there will be no way to reorder them if they arrive out of + order. This is not a serious problem, but it will allow the window + information to be on occasion temporarily based on old reports from + the data receiver. A refinement to avoid this problem is to act on + the window information from segments that carry the highest + acknowledgment number (that is, segments with an acknowledgment + number equal to or greater than the highest previously received). + + Indicating a large window encourages transmissions. If more data + arrives than can be accepted, it will be discarded. This will result + in excessive retransmissions, adding unnecessarily to the load on the + network and the TCP endpoints. Indicating a small window may + restrict the transmission of data to the point of introducing a + round-trip delay between each new segment transmitted. + + The mechanisms provided allow a TCP endpoint to advertise a large + window and to subsequently advertise a much smaller window without + having accepted that much data. This so-called "shrinking the + window" is strongly discouraged. The robustness principle [19] + dictates that TCP peers will not shrink the window themselves, but + will be prepared for such behavior on the part of other TCP peers. + + A TCP receiver SHOULD NOT shrink the window, i.e., move the right + window edge to the left (SHLD-14). However, a sending TCP peer MUST + be robust against window shrinking, which may cause the "usable + window" (see Section 3.8.6.2.1) to become negative (MUST-34). + + If this happens, the sender SHOULD NOT send new data (SHLD-15), but + SHOULD retransmit normally the old unacknowledged data between + SND.UNA and SND.UNA+SND.WND (SHLD-16). The sender MAY also + retransmit old data beyond SND.UNA+SND.WND (MAY-7), but SHOULD NOT + time out the connection if data beyond the right window edge is not + acknowledged (SHLD-17). If the window shrinks to zero, the TCP + implementation MUST probe it in the standard way (described below) + (MUST-35). + +3.8.6.1. Zero-Window Probing + + The sending TCP peer must regularly transmit at least one octet of + new data (if available), or retransmit to the receiving TCP peer even + if the send window is zero, in order to "probe" the window. This + retransmission is essential to guarantee that when either TCP peer + has a zero window the reopening of the window will be reliably + reported to the other. This is referred to as Zero-Window Probing + (ZWP) in other documents. + + Probing of zero (offered) windows MUST be supported (MUST-36). + + A TCP implementation MAY keep its offered receive window closed + indefinitely (MAY-8). As long as the receiving TCP peer continues to + send acknowledgments in response to the probe segments, the sending + TCP peer MUST allow the connection to stay open (MUST-37). This + enables TCP to function in scenarios such as the "printer ran out of + paper" situation described in Section 4.2.2.17 of [19]. The behavior + is subject to the implementation's resource management concerns, as + noted in [41]. + + When the receiving TCP peer has a zero window and a segment arrives, + it must still send an acknowledgment showing its next expected + sequence number and current window (zero). + + The transmitting host SHOULD send the first zero-window probe when a + zero window has existed for the retransmission timeout period (SHLD- + 29) (Section 3.8.1), and SHOULD increase exponentially the interval + between successive probes (SHLD-30). + +3.8.6.2. Silly Window Syndrome Avoidance + + The "Silly Window Syndrome" (SWS) is a stable pattern of small + incremental window movements resulting in extremely poor TCP + performance. Algorithms to avoid SWS are described below for both + the sending side and the receiving side. RFC 1122 contains more + detailed discussion of the SWS problem. Note that the Nagle + algorithm and the sender SWS avoidance algorithm play complementary + roles in improving performance. The Nagle algorithm discourages + sending tiny segments when the data to be sent increases in small + increments, while the SWS avoidance algorithm discourages small + segments resulting from the right window edge advancing in small + increments. + +3.8.6.2.1. Sender's Algorithm -- When to Send Data + + A TCP implementation MUST include a SWS avoidance algorithm in the + sender (MUST-38). + + The Nagle algorithm from Section 3.7.4 additionally describes how to + coalesce short segments. + + The sender's SWS avoidance algorithm is more difficult than the + receiver's because the sender does not know (directly) the receiver's + total buffer space (RCV.BUFF). An approach that has been found to + work well is for the sender to calculate Max(SND.WND), which is the + maximum send window it has seen so far on the connection, and to use + this value as an estimate of RCV.BUFF. Unfortunately, this can only + be an estimate; the receiver may at any time reduce the size of + RCV.BUFF. To avoid a resulting deadlock, it is necessary to have a + timeout to force transmission of data, overriding the SWS avoidance + algorithm. In practice, this timeout should seldom occur. + + The "usable window" is: + + U = SND.UNA + SND.WND - SND.NXT + + i.e., the offered window less the amount of data sent but not + acknowledged. If D is the amount of data queued in the sending TCP + endpoint but not yet sent, then the following set of rules is + recommended. + + Send data: + + (1) if a maximum-sized segment can be sent, i.e., if: + + min(D,U) >= Eff.snd.MSS; + + (2) or if the data is pushed and all queued data can be sent now, + i.e., if: + + [SND.NXT = SND.UNA and] PUSHed and D <= U + + (the bracketed condition is imposed by the Nagle algorithm); + + (3) or if at least a fraction Fs of the maximum window can be sent, + i.e., if: + + [SND.NXT = SND.UNA and] + + min(D,U) >= Fs * Max(SND.WND); + + (4) or if the override timeout occurs. + + Here Fs is a fraction whose recommended value is 1/2. The override + timeout should be in the range 0.1 - 1.0 seconds. It may be + convenient to combine this timer with the timer used to probe zero + windows (Section 3.8.6.1). + +3.8.6.2.2. Receiver's Algorithm -- When to Send a Window Update + + A TCP implementation MUST include a SWS avoidance algorithm in the + receiver (MUST-39). + + The receiver's SWS avoidance algorithm determines when the right + window edge may be advanced; this is customarily known as "updating + the window". This algorithm combines with the delayed ACK algorithm + (Section 3.8.6.3) to determine when an ACK segment containing the + current window will really be sent to the receiver. + + The solution to receiver SWS is to avoid advancing the right window + edge RCV.NXT+RCV.WND in small increments, even if data is received + from the network in small segments. + + Suppose the total receive buffer space is RCV.BUFF. At any given + moment, RCV.USER octets of this total may be tied up with data that + has been received and acknowledged but that the user process has not + yet consumed. When the connection is quiescent, RCV.WND = RCV.BUFF + and RCV.USER = 0. + + Keeping the right window edge fixed as data arrives and is + acknowledged requires that the receiver offer less than its full + buffer space, i.e., the receiver must specify a RCV.WND that keeps + RCV.NXT+RCV.WND constant as RCV.NXT increases. Thus, the total + buffer space RCV.BUFF is generally divided into three parts: + + |<------- RCV.BUFF ---------------->| + 1 2 3 + ----|---------|------------------|------|---- + RCV.NXT ^ + (Fixed) + + 1 - RCV.USER = data received but not yet consumed; + 2 - RCV.WND = space advertised to sender; + 3 - Reduction = space available but not yet + advertised. + + The suggested SWS avoidance algorithm for the receiver is to keep + RCV.NXT+RCV.WND fixed until the reduction satisfies: + + RCV.BUFF - RCV.USER - RCV.WND >= + + min( Fr * RCV.BUFF, Eff.snd.MSS ) + + where Fr is a fraction whose recommended value is 1/2, and + Eff.snd.MSS is the effective send MSS for the connection (see + Section 3.7.1). When the inequality is satisfied, RCV.WND is set to + RCV.BUFF-RCV.USER. + + Note that the general effect of this algorithm is to advance RCV.WND + in increments of Eff.snd.MSS (for realistic receive buffers: + Eff.snd.MSS < RCV.BUFF/2). Note also that the receiver must use its + own Eff.snd.MSS, making the assumption that it is the same as the + sender's. + +3.8.6.3. Delayed Acknowledgments -- When to Send an ACK Segment + + A host that is receiving a stream of TCP data segments can increase + efficiency in both the network and the hosts by sending fewer than + one ACK (acknowledgment) segment per data segment received; this is + known as a "delayed ACK". + + A TCP endpoint SHOULD implement a delayed ACK (SHLD-18), but an ACK + should not be excessively delayed; in particular, the delay MUST be + less than 0.5 seconds (MUST-40). An ACK SHOULD be generated for at + least every second full-sized segment or 2*RMSS bytes of new data + (where RMSS is the MSS specified by the TCP endpoint receiving the + segments to be acknowledged, or the default value if not specified) + (SHLD-19). Excessive delays on ACKs can disturb the round-trip + timing and packet "clocking" algorithms. More complete discussion of + delayed ACK behavior is in Section 4.2 of RFC 5681 [8], including + recommendations to immediately acknowledge out-of-order segments, + segments above a gap in sequence space, or segments that fill all or + part of a gap, in order to accelerate loss recovery. + + Note that there are several current practices that further lead to a + reduced number of ACKs, including generic receive offload (GRO) [72], + ACK compression, and ACK decimation [28]. + +3.9. Interfaces + + There are of course two interfaces of concern: the user/TCP interface + and the TCP/lower-level interface. We have a fairly elaborate model + of the user/TCP interface, but the interface to the lower-level + protocol module is left unspecified here since it will be specified + in detail by the specification of the lower-level protocol. For the + case that the lower level is IP, we note some of the parameter values + that TCP implementations might use. + +3.9.1. User/TCP Interface + + The following functional description of user commands to the TCP + implementation is, at best, fictional, since every operating system + will have different facilities. Consequently, we must warn readers + that different TCP implementations may have different user + interfaces. However, all TCP implementations must provide a certain + minimum set of services to guarantee that all TCP implementations can + support the same protocol hierarchy. This section specifies the + functional interfaces required of all TCP implementations. + + Section 3.1 of [53] also identifies primitives provided by TCP and + could be used as an additional reference for implementers. + + The following sections functionally characterize a user/TCP + interface. The notation used is similar to most procedure or + function calls in high-level languages, but this usage is not meant + to rule out trap-type service calls. + + The user commands described below specify the basic functions the TCP + implementation must perform to support interprocess communication. + Individual implementations must define their own exact format and may + provide combinations or subsets of the basic functions in single + calls. In particular, some implementations may wish to automatically + OPEN a connection on the first SEND or RECEIVE issued by the user for + a given connection. + + In providing interprocess communication facilities, the TCP + implementation must not only accept commands, but must also return + information to the processes it serves. The latter consists of: + + (a) general information about a connection (e.g., interrupts, remote + close, binding of unspecified remote socket). + + (b) replies to specific user commands indicating success or various + types of failure. + +3.9.1.1. Open + + Format: OPEN (local port, remote socket, active/passive [, timeout] + [, Diffserv field] [, security/compartment] [, local IP address] [, + options]) -> local connection name + + If the active/passive flag is set to passive, then this is a call to + LISTEN for an incoming connection. A passive OPEN may have either a + fully specified remote socket to wait for a particular connection or + an unspecified remote socket to wait for any call. A fully specified + passive call can be made active by the subsequent execution of a + SEND. + + A transmission control block (TCB) is created and partially filled in + with data from the OPEN command parameters. + + Every passive OPEN call either creates a new connection record in + LISTEN state, or it returns an error; it MUST NOT affect any + previously created connection record (MUST-41). + + A TCP implementation that supports multiple concurrent connections + MUST provide an OPEN call that will functionally allow an application + to LISTEN on a port while a connection block with the same local port + is in SYN-SENT or SYN-RECEIVED state (MUST-42). + + On an active OPEN command, the TCP endpoint will begin the procedure + to synchronize (i.e., establish) the connection at once. + + The timeout, if present, permits the caller to set up a timeout for + all data submitted to TCP. If data is not successfully delivered to + the destination within the timeout period, the TCP endpoint will + abort the connection. The present global default is five minutes. + + The TCP implementation or some component of the operating system will + verify the user's authority to open a connection with the specified + Diffserv field value or security/compartment. The absence of a + Diffserv field value or security/compartment specification in the + OPEN call indicates the default values must be used. + + TCP will accept incoming requests as matching only if the security/ + compartment information is exactly the same as that requested in the + OPEN call. + + The Diffserv field value indicated by the user only impacts outgoing + packets, may be altered en route through the network, and has no + direct bearing or relation to received packets. + + A local connection name will be returned to the user by the TCP + implementation. The local connection name can then be used as a + shorthand term for the connection defined by the pair. + + The optional "local IP address" parameter MUST be supported to allow + the specification of the local IP address (MUST-43). This enables + applications that need to select the local IP address used when + multihoming is present. + + A passive OPEN call with a specified "local IP address" parameter + will await an incoming connection request to that address. If the + parameter is unspecified, a passive OPEN will await an incoming + connection request to any local IP address and then bind the local IP + address of the connection to the particular address that is used. + + For an active OPEN call, a specified "local IP address" parameter + will be used for opening the connection. If the parameter is + unspecified, the host will choose an appropriate local IP address + (see RFC 1122, Section 3.3.4.2). + + If an application on a multihomed host does not specify the local IP + address when actively opening a TCP connection, then the TCP + implementation MUST ask the IP layer to select a local IP address + before sending the (first) SYN (MUST-44). See the function + GET_SRCADDR() in Section 3.4 of RFC 1122. + + At all other times, a previous segment has either been sent or + received on this connection, and TCP implementations MUST use the + same local address that was used in those previous segments (MUST- + 45). + + A TCP implementation MUST reject as an error a local OPEN call for an + invalid remote IP address (e.g., a broadcast or multicast address) + (MUST-46). + +3.9.1.2. Send + + Format: SEND (local connection name, buffer address, byte count, + URGENT flag [, PUSH flag] [, timeout]) + + This call causes the data contained in the indicated user buffer to + be sent on the indicated connection. If the connection has not been + opened, the SEND is considered an error. Some implementations may + allow users to SEND first; in which case, an automatic OPEN would be + done. For example, this might be one way for application data to be + included in SYN segments. If the calling process is not authorized + to use this connection, an error is returned. + + A TCP endpoint MAY implement PUSH flags on SEND calls (MAY-15). If + PUSH flags are not implemented, then the sending TCP peer: (1) MUST + NOT buffer data indefinitely (MUST-60), and (2) MUST set the PSH bit + in the last buffered segment (i.e., when there is no more queued data + to be sent) (MUST-61). The remaining description below assumes the + PUSH flag is supported on SEND calls. + + If the PUSH flag is set, the application intends the data to be + transmitted promptly to the receiver, and the PSH bit will be set in + the last TCP segment created from the buffer. + + The PSH bit is not a record marker and is independent of segment + boundaries. The transmitter SHOULD collapse successive bits when it + packetizes data, to send the largest possible segment (SHLD-27). + + If the PUSH flag is not set, the data may be combined with data from + subsequent SENDs for transmission efficiency. When an application + issues a series of SEND calls without setting the PUSH flag, the TCP + implementation MAY aggregate the data internally without sending it + (MAY-16). Note that when the Nagle algorithm is in use, TCP + implementations may buffer the data before sending, without regard to + the PUSH flag (see Section 3.7.4). + + An application program is logically required to set the PUSH flag in + a SEND call whenever it needs to force delivery of the data to avoid + a communication deadlock. However, a TCP implementation SHOULD send + a maximum-sized segment whenever possible (SHLD-28) to improve + performance (see Section 3.8.6.2.1). + + New applications SHOULD NOT set the URGENT flag [39] due to + implementation differences and middlebox issues (SHLD-13). + + If the URGENT flag is set, segments sent to the destination TCP peer + will have the urgent pointer set. The receiving TCP peer will signal + the urgent condition to the receiving process if the urgent pointer + indicates that data preceding the urgent pointer has not been + consumed by the receiving process. The purpose of the URGENT flag is + to stimulate the receiver to process the urgent data and to indicate + to the receiver when all the currently known urgent data has been + received. The number of times the sending user's TCP implementation + signals urgent will not necessarily be equal to the number of times + the receiving user will be notified of the presence of urgent data. + + If no remote socket was specified in the OPEN, but the connection is + established (e.g., because a LISTENing connection has become specific + due to a remote segment arriving for the local socket), then the + designated buffer is sent to the implied remote socket. Users who + make use of OPEN with an unspecified remote socket can make use of + SEND without ever explicitly knowing the remote socket address. + + However, if a SEND is attempted before the remote socket becomes + specified, an error will be returned. Users can use the STATUS call + to determine the status of the connection. Some TCP implementations + may notify the user when an unspecified socket is bound. + + If a timeout is specified, the current user timeout for this + connection is changed to the new one. + + In the simplest implementation, SEND would not return control to the + sending process until either the transmission was complete or the + timeout had been exceeded. However, this simple method is both + subject to deadlocks (for example, both sides of the connection might + try to do SENDs before doing any RECEIVEs) and offers poor + performance, so it is not recommended. A more sophisticated + implementation would return immediately to allow the process to run + concurrently with network I/O, and, furthermore, to allow multiple + SENDs to be in progress. Multiple SENDs are served in first come, + first served order, so the TCP endpoint will queue those it cannot + service immediately. + + We have implicitly assumed an asynchronous user interface in which a + SEND later elicits some kind of SIGNAL or pseudo-interrupt from the + serving TCP endpoint. An alternative is to return a response + immediately. For instance, SENDs might return immediate local + acknowledgment, even if the segment sent had not been acknowledged by + the distant TCP endpoint. We could optimistically assume eventual + success. If we are wrong, the connection will close anyway due to + the timeout. In implementations of this kind (synchronous), there + will still be some asynchronous signals, but these will deal with the + connection itself, and not with specific segments or buffers. + + In order for the process to distinguish among error or success + indications for different SENDs, it might be appropriate for the + buffer address to be returned along with the coded response to the + SEND request. TCP-to-user signals are discussed below, indicating + the information that should be returned to the calling process. + +3.9.1.3. Receive + + Format: RECEIVE (local connection name, buffer address, byte count) + -> byte count, URGENT flag [, PUSH flag] + + This command allocates a receiving buffer associated with the + specified connection. If no OPEN precedes this command or the + calling process is not authorized to use this connection, an error is + returned. + + In the simplest implementation, control would not return to the + calling program until either the buffer was filled or some error + occurred, but this scheme is highly subject to deadlocks. A more + sophisticated implementation would permit several RECEIVEs to be + outstanding at once. These would be filled as segments arrive. This + strategy permits increased throughput at the cost of a more elaborate + scheme (possibly asynchronous) to notify the calling program that a + PUSH has been seen or a buffer filled. + + A TCP receiver MAY pass a received PSH bit to the application layer + via the PUSH flag in the interface (MAY-17), but it is not required + (this was clarified in RFC 1122, Section 4.2.2.2). The remainder of + text describing the RECEIVE call below assumes that passing the PUSH + indication is supported. + + If enough data arrive to fill the buffer before a PUSH is seen, the + PUSH flag will not be set in the response to the RECEIVE. The buffer + will be filled with as much data as it can hold. If a PUSH is seen + before the buffer is filled, the buffer will be returned partially + filled and PUSH indicated. + + If there is urgent data, the user will have been informed as soon as + it arrived via a TCP-to-user signal. The receiving user should thus + be in "urgent mode". If the URGENT flag is on, additional urgent + data remains. If the URGENT flag is off, this call to RECEIVE has + returned all the urgent data, and the user may now leave "urgent + mode". Note that data following the urgent pointer (non-urgent data) + cannot be delivered to the user in the same buffer with preceding + urgent data unless the boundary is clearly marked for the user. + + To distinguish among several outstanding RECEIVEs and to take care of + the case that a buffer is not completely filled, the return code is + accompanied by both a buffer pointer and a byte count indicating the + actual length of the data received. + + Alternative implementations of RECEIVE might have the TCP endpoint + allocate buffer storage, or the TCP endpoint might share a ring + buffer with the user. + +3.9.1.4. Close + + Format: CLOSE (local connection name) + + This command causes the connection specified to be closed. If the + connection is not open or the calling process is not authorized to + use this connection, an error is returned. Closing connections is + intended to be a graceful operation in the sense that outstanding + SENDs will be transmitted (and retransmitted), as flow control + permits, until all have been serviced. Thus, it should be acceptable + to make several SEND calls, followed by a CLOSE, and expect all the + data to be sent to the destination. It should also be clear that + users should continue to RECEIVE on CLOSING connections since the + remote peer may be trying to transmit the last of its data. Thus, + CLOSE means "I have no more to send" but does not mean "I will not + receive any more." It may happen (if the user-level protocol is not + well thought out) that the closing side is unable to get rid of all + its data before timing out. In this event, CLOSE turns into ABORT, + and the closing TCP peer gives up. + + The user may CLOSE the connection at any time on their own + initiative, or in response to various prompts from the TCP + implementation (e.g., remote close executed, transmission timeout + exceeded, destination inaccessible). + + Because closing a connection requires communication with the remote + TCP peer, connections may remain in the closing state for a short + time. Attempts to reopen the connection before the TCP peer replies + to the CLOSE command will result in error responses. + + Close also implies push function. + +3.9.1.5. Status + + Format: STATUS (local connection name) -> status data + + This is an implementation-dependent user command and could be + excluded without adverse effect. Information returned would + typically come from the TCB associated with the connection. + + This command returns a data block containing the following + information: + + local socket, + + remote socket, + + local connection name, + + receive window, + + send window, + + connection state, + + number of buffers awaiting acknowledgment, + + number of buffers pending receipt, + + urgent state, + + Diffserv field value, + + security/compartment, and + + transmission timeout. + + Depending on the state of the connection, or on the implementation + itself, some of this information may not be available or meaningful. + If the calling process is not authorized to use this connection, an + error is returned. This prevents unauthorized processes from gaining + information about a connection. + +3.9.1.6. Abort + + Format: ABORT (local connection name) + + This command causes all pending SENDs and RECEIVES to be aborted, the + TCB to be removed, and a special RST message to be sent to the remote + TCP peer of the connection. Depending on the implementation, users + may receive abort indications for each outstanding SEND or RECEIVE, + or may simply receive an ABORT-acknowledgment. + +3.9.1.7. Flush + + Some TCP implementations have included a FLUSH call, which will empty + the TCP send queue of any data that the user has issued SEND calls + for but is still to the right of the current send window. That is, + it flushes as much queued send data as possible without losing + sequence number synchronization. The FLUSH call MAY be implemented + (MAY-14). + +3.9.1.8. Asynchronous Reports + + There MUST be a mechanism for reporting soft TCP error conditions to + the application (MUST-47). Generically, we assume this takes the + form of an application-supplied ERROR_REPORT routine that may be + upcalled asynchronously from the transport layer: + + ERROR_REPORT(local connection name, reason, subreason) + + The precise encoding of the reason and subreason parameters is not + specified here. However, the conditions that are reported + asynchronously to the application MUST include: + + * ICMP error message arrived (see Section 3.9.2.2 for description of + handling each ICMP message type since some message types need to + be suppressed from generating reports to the application) + + * Excessive retransmissions (see Section 3.8.3) + + * Urgent pointer advance (see Section 3.8.5) + + However, an application program that does not want to receive such + ERROR_REPORT calls SHOULD be able to effectively disable these calls + (SHLD-20). + +3.9.1.9. Set Differentiated Services Field (IPv4 TOS or IPv6 Traffic + Class) + + The application layer MUST be able to specify the Differentiated + Services field for segments that are sent on a connection (MUST-48). + The Differentiated Services field includes the 6-bit Differentiated + Services Codepoint (DSCP) value. It is not required, but the + application SHOULD be able to change the Differentiated Services + field during the connection lifetime (SHLD-21). TCP implementations + SHOULD pass the current Differentiated Services field value without + change to the IP layer, when it sends segments on the connection + (SHLD-22). + + The Differentiated Services field will be specified independently in + each direction on the connection, so that the receiver application + will specify the Differentiated Services field used for ACK segments. + + TCP implementations MAY pass the most recently received + Differentiated Services field up to the application (MAY-9). + +3.9.2. TCP/Lower-Level Interface + + The TCP endpoint calls on a lower-level protocol module to actually + send and receive information over a network. The two current + standard Internet Protocol (IP) versions layered below TCP are IPv4 + [1] and IPv6 [13]. + + If the lower-level protocol is IPv4, it provides arguments for a type + of service (used within the Differentiated Services field) and for a + time to live. TCP uses the following settings for these parameters: + + Diffserv field: The IP header value for the Diffserv field is given + by the user. This includes the bits of the Diffserv Codepoint + (DSCP). + + Time to Live (TTL): The TTL value used to send TCP segments MUST be + configurable (MUST-49). + + * Note that RFC 793 specified one minute (60 seconds) as a + constant for the TTL because the assumed maximum segment + lifetime was two minutes. This was intended to explicitly ask + that a segment be destroyed if it could not be delivered by the + internet system within one minute. RFC 1122 updated RFC 793 to + require that the TTL be configurable. + + * Note that the Diffserv field is permitted to change during a + connection (Section 4.2.4.2 of RFC 1122). However, the + application interface might not support this ability, and the + application does not have knowledge about individual TCP + segments, so this can only be done on a coarse granularity, at + best. This limitation is further discussed in RFC 7657 + (Sections 5.1, 5.3, and 6) [50]. Generally, an application + SHOULD NOT change the Diffserv field value during the course of + a connection (SHLD-23). + + Any lower-level protocol will have to provide the source address, + destination address, and protocol fields, and some way to determine + the "TCP length", both to provide the functional equivalent service + of IP and to be used in the TCP checksum. + + When received options are passed up to TCP from the IP layer, a TCP + implementation MUST ignore options that it does not understand (MUST- + 50). + + A TCP implementation MAY support the Timestamp (MAY-10) and Record + Route (MAY-11) Options. + +3.9.2.1. Source Routing + + If the lower level is IP (or other protocol that provides this + feature) and source routing is used, the interface must allow the + route information to be communicated. This is especially important + so that the source and destination addresses used in the TCP checksum + be the originating source and ultimate destination. It is also + important to preserve the return route to answer connection requests. + + An application MUST be able to specify a source route when it + actively opens a TCP connection (MUST-51), and this MUST take + precedence over a source route received in a datagram (MUST-52). + + When a TCP connection is OPENed passively and a packet arrives with a + completed IP Source Route Option (containing a return route), TCP + implementations MUST save the return route and use it for all + segments sent on this connection (MUST-53). If a different source + route arrives in a later segment, the later definition SHOULD + override the earlier one (SHLD-24). + +3.9.2.2. ICMP Messages + + TCP implementations MUST act on an ICMP error message passed up from + the IP layer, directing it to the connection that created the error + (MUST-54). The necessary demultiplexing information can be found in + the IP header contained within the ICMP message. + + This applies to ICMPv6 in addition to IPv4 ICMP. + + [35] contains discussion of specific ICMP and ICMPv6 messages + classified as either "soft" or "hard" errors that may bear different + responses. Treatment for classes of ICMP messages is described + below: + + Source Quench + TCP implementations MUST silently discard any received ICMP Source + Quench messages (MUST-55). See [11] for discussion. + + Soft Errors + For IPv4 ICMP, these include: Destination Unreachable -- codes 0, + 1, 5; Time Exceeded -- codes 0, 1; and Parameter Problem. + + For ICMPv6, these include: Destination Unreachable -- codes 0, 3; + Time Exceeded -- codes 0, 1; and Parameter Problem -- codes 0, 1, + 2. + + Since these Unreachable messages indicate soft error conditions, a + TCP implementation MUST NOT abort the connection (MUST-56), and it + SHOULD make the information available to the application (SHLD-25). + + Hard Errors + For ICMP these include Destination Unreachable -- codes 2-4. + + These are hard error conditions, so TCP implementations SHOULD + abort the connection (SHLD-26). [35] notes that some + implementations do not abort connections when an ICMP hard error is + received for a connection that is in any of the synchronized + states. + + Note that [35], Section 4 describes widespread implementation + behavior that treats soft errors as hard errors during connection + establishment. + +3.9.2.3. Source Address Validation + + RFC 1122 requires addresses to be validated in incoming SYN packets: + + | An incoming SYN with an invalid source address MUST be ignored + | either by TCP or by the IP layer [(MUST-63)] (see + | Section 3.2.1.3). + | + | A TCP implementation MUST silently discard an incoming SYN segment + | that is addressed to a broadcast or multicast address [(MUST-57)]. + + This prevents connection state and replies from being erroneously + generated, and implementers should note that this guidance is + applicable to all incoming segments, not just SYNs, as specifically + indicated in RFC 1122. + +3.10. Event Processing + + The processing depicted in this section is an example of one possible + implementation. Other implementations may have slightly different + processing sequences, but they should differ from those in this + section only in detail, not in substance. + + The activity of the TCP endpoint can be characterized as responding + to events. The events that occur can be cast into three categories: + user calls, arriving segments, and timeouts. This section describes + the processing the TCP endpoint does in response to each of the + events. In many cases, the processing required depends on the state + of the connection. + + Events that occur: + + User Calls + + OPEN + + SEND + + RECEIVE + + CLOSE + + ABORT + + STATUS + + Arriving Segments + + SEGMENT ARRIVES + + Timeouts + + USER TIMEOUT + + RETRANSMISSION TIMEOUT + + TIME-WAIT TIMEOUT + + The model of the TCP/user interface is that user commands receive an + immediate return and possibly a delayed response via an event or + pseudo-interrupt. In the following descriptions, the term "signal" + means cause a delayed response. + + Error responses in this document are identified by character strings. + For example, user commands referencing connections that do not exist + receive "error: connection not open". + + Please note in the following that all arithmetic on sequence numbers, + acknowledgment numbers, windows, et cetera, is modulo 2^32 (the size + of the sequence number space). Also note that "=<" means less than + or equal to (modulo 2^32). + + A natural way to think about processing incoming segments is to + imagine that they are first tested for proper sequence number (i.e., + that their contents lie in the range of the expected "receive window" + in the sequence number space) and then that they are generally queued + and processed in sequence number order. + + When a segment overlaps other already received segments, we + reconstruct the segment to contain just the new data and adjust the + header fields to be consistent. + + Note that if no state change is mentioned, the TCP connection stays + in the same state. + +3.10.1. OPEN Call + + CLOSED STATE (i.e., TCB does not exist) + + * Create a new transmission control block (TCB) to hold connection + state information. Fill in local socket identifier, remote + socket, Diffserv field, security/compartment, and user timeout + information. Note that some parts of the remote socket may be + unspecified in a passive OPEN and are to be filled in by the + parameters of the incoming SYN segment. Verify the security and + Diffserv value requested are allowed for this user, if not, return + "error: Diffserv value not allowed" or "error: security/ + compartment not allowed". If passive, enter the LISTEN state and + return. If active and the remote socket is unspecified, return + "error: remote socket unspecified"; if active and the remote + socket is specified, issue a SYN segment. An initial send + sequence number (ISS) is selected. A SYN segment of the form + is sent. Set SND.UNA to ISS, SND.NXT to ISS+1, + enter SYN-SENT state, and return. + + * If the caller does not have access to the local socket specified, + return "error: connection illegal for this process". If there is + no room to create a new connection, return "error: insufficient + resources". + + LISTEN STATE + + * If the OPEN call is active and the remote socket is specified, + then change the connection from passive to active, select an ISS. + Send a SYN segment, set SND.UNA to ISS, SND.NXT to ISS+1. Enter + SYN-SENT state. Data associated with SEND may be sent with SYN + segment or queued for transmission after entering ESTABLISHED + state. The urgent bit if requested in the command must be sent + with the data segments sent as a result of this command. If there + is no room to queue the request, respond with "error: insufficient + resources". If the remote socket was not specified, then return + "error: remote socket unspecified". + + SYN-SENT STATE + + SYN-RECEIVED STATE + + ESTABLISHED STATE + + FIN-WAIT-1 STATE + + FIN-WAIT-2 STATE + + CLOSE-WAIT STATE + + CLOSING STATE + + LAST-ACK STATE + + TIME-WAIT STATE + + * Return "error: connection already exists". + +3.10.2. SEND Call + + CLOSED STATE (i.e., TCB does not exist) + + * If the user does not have access to such a connection, then return + "error: connection illegal for this process". + + * Otherwise, return "error: connection does not exist". + + LISTEN STATE + + * If the remote socket is specified, then change the connection from + passive to active, select an ISS. Send a SYN segment, set SND.UNA + to ISS, SND.NXT to ISS+1. Enter SYN-SENT state. Data associated + with SEND may be sent with SYN segment or queued for transmission + after entering ESTABLISHED state. The urgent bit if requested in + the command must be sent with the data segments sent as a result + of this command. If there is no room to queue the request, + respond with "error: insufficient resources". If the remote + socket was not specified, then return "error: remote socket + unspecified". + + SYN-SENT STATE + + SYN-RECEIVED STATE + + * Queue the data for transmission after entering ESTABLISHED state. + If no space to queue, respond with "error: insufficient + resources". + + ESTABLISHED STATE + + CLOSE-WAIT STATE + + * Segmentize the buffer and send it with a piggybacked + acknowledgment (acknowledgment value = RCV.NXT). If there is + insufficient space to remember this buffer, simply return "error: + insufficient resources". + + * If the URGENT flag is set, then SND.UP <- SND.NXT and set the + urgent pointer in the outgoing segments. + + FIN-WAIT-1 STATE + + FIN-WAIT-2 STATE + + CLOSING STATE + + LAST-ACK STATE + + TIME-WAIT STATE + + * Return "error: connection closing" and do not service request. + +3.10.3. RECEIVE Call + + CLOSED STATE (i.e., TCB does not exist) + + * If the user does not have access to such a connection, return + "error: connection illegal for this process". + + * Otherwise, return "error: connection does not exist". + + LISTEN STATE + + SYN-SENT STATE + + SYN-RECEIVED STATE + + * Queue for processing after entering ESTABLISHED state. If there + is no room to queue this request, respond with "error: + insufficient resources". + + ESTABLISHED STATE + + FIN-WAIT-1 STATE + + FIN-WAIT-2 STATE + + * If insufficient incoming segments are queued to satisfy the + request, queue the request. If there is no queue space to + remember the RECEIVE, respond with "error: insufficient + resources". + + * Reassemble queued incoming segments into receive buffer and return + to user. Mark "push seen" (PUSH) if this is the case. + + * If RCV.UP is in advance of the data currently being passed to the + user, notify the user of the presence of urgent data. + + * When the TCP endpoint takes responsibility for delivering data to + the user, that fact must be communicated to the sender via an + acknowledgment. The formation of such an acknowledgment is + described below in the discussion of processing an incoming + segment. + + CLOSE-WAIT STATE + + * Since the remote side has already sent FIN, RECEIVEs must be + satisfied by data already on hand, but not yet delivered to the + user. If no text is awaiting delivery, the RECEIVE will get an + "error: connection closing" response. Otherwise, any remaining + data can be used to satisfy the RECEIVE. + + CLOSING STATE + + LAST-ACK STATE + + TIME-WAIT STATE + + * Return "error: connection closing". + +3.10.4. CLOSE Call + + CLOSED STATE (i.e., TCB does not exist) + + * If the user does not have access to such a connection, return + "error: connection illegal for this process". + + * Otherwise, return "error: connection does not exist". + + LISTEN STATE + + * Any outstanding RECEIVEs are returned with "error: closing" + responses. Delete TCB, enter CLOSED state, and return. + + SYN-SENT STATE + + * Delete the TCB and return "error: closing" responses to any queued + SENDs, or RECEIVEs. + + SYN-RECEIVED STATE + + * If no SENDs have been issued and there is no pending data to send, + then form a FIN segment and send it, and enter FIN-WAIT-1 state; + otherwise, queue for processing after entering ESTABLISHED state. + + ESTABLISHED STATE + + * Queue this until all preceding SENDs have been segmentized, then + form a FIN segment and send it. In any case, enter FIN-WAIT-1 + state. + + FIN-WAIT-1 STATE + + FIN-WAIT-2 STATE + + * Strictly speaking, this is an error and should receive an "error: + connection closing" response. An "ok" response would be + acceptable, too, as long as a second FIN is not emitted (the first + FIN may be retransmitted, though). + + CLOSE-WAIT STATE + + * Queue this request until all preceding SENDs have been + segmentized; then send a FIN segment, enter LAST-ACK state. + + CLOSING STATE + + LAST-ACK STATE + + TIME-WAIT STATE + + * Respond with "error: connection closing". + +3.10.5. ABORT Call + + CLOSED STATE (i.e., TCB does not exist) + + * If the user should not have access to such a connection, return + "error: connection illegal for this process". + + * Otherwise, return "error: connection does not exist". + + LISTEN STATE + + * Any outstanding RECEIVEs should be returned with "error: + connection reset" responses. Delete TCB, enter CLOSED state, and + return. + + SYN-SENT STATE + + * All queued SENDs and RECEIVEs should be given "connection reset" + notification. Delete the TCB, enter CLOSED state, and return. + + SYN-RECEIVED STATE + + ESTABLISHED STATE + + FIN-WAIT-1 STATE + + FIN-WAIT-2 STATE + + CLOSE-WAIT STATE + + * Send a reset segment: + + + + * All queued SENDs and RECEIVEs should be given "connection reset" + notification; all segments queued for transmission (except for the + RST formed above) or retransmission should be flushed. Delete the + TCB, enter CLOSED state, and return. + + CLOSING STATE + + LAST-ACK STATE + + TIME-WAIT STATE + + * Respond with "ok" and delete the TCB, enter CLOSED state, and + return. + +3.10.6. STATUS Call + + CLOSED STATE (i.e., TCB does not exist) + + * If the user should not have access to such a connection, return + "error: connection illegal for this process". + + * Otherwise, return "error: connection does not exist". + + LISTEN STATE + + * Return "state = LISTEN" and the TCB pointer. + + SYN-SENT STATE + + * Return "state = SYN-SENT" and the TCB pointer. + + SYN-RECEIVED STATE + + * Return "state = SYN-RECEIVED" and the TCB pointer. + + ESTABLISHED STATE + + * Return "state = ESTABLISHED" and the TCB pointer. + + FIN-WAIT-1 STATE + + * Return "state = FIN-WAIT-1" and the TCB pointer. + + FIN-WAIT-2 STATE + + * Return "state = FIN-WAIT-2" and the TCB pointer. + + CLOSE-WAIT STATE + + * Return "state = CLOSE-WAIT" and the TCB pointer. + + CLOSING STATE + + * Return "state = CLOSING" and the TCB pointer. + + LAST-ACK STATE + + * Return "state = LAST-ACK" and the TCB pointer. + + TIME-WAIT STATE + + * Return "state = TIME-WAIT" and the TCB pointer. + +3.10.7. SEGMENT ARRIVES + +3.10.7.1. CLOSED STATE + + If the state is CLOSED (i.e., TCB does not exist), then + + all data in the incoming segment is discarded. An incoming + segment containing a RST is discarded. An incoming segment not + containing a RST causes a RST to be sent in response. The + acknowledgment and sequence field values are selected to make the + reset sequence acceptable to the TCP endpoint that sent the + offending segment. + + If the ACK bit is off, sequence number zero is used, + + + + If the ACK bit is on, + + + + Return. + +3.10.7.2. LISTEN STATE + + If the state is LISTEN, then + + First, check for a RST: + + - An incoming RST segment could not be valid since it could not + have been sent in response to anything sent by this incarnation + of the connection. An incoming RST should be ignored. Return. + + Second, check for an ACK: + + - Any acknowledgment is bad if it arrives on a connection still + in the LISTEN state. An acceptable reset segment should be + formed for any arriving ACK-bearing segment. The RST should be + formatted as follows: + + + + - Return. + + Third, check for a SYN: + + - If the SYN bit is set, check the security. If the security/ + compartment on the incoming segment does not exactly match the + security/compartment in the TCB, then send a reset and return. + + + + - Set RCV.NXT to SEG.SEQ+1, IRS is set to SEG.SEQ, and any other + control or text should be queued for processing later. ISS + should be selected and a SYN segment sent of the form: + + + + - SND.NXT is set to ISS+1 and SND.UNA to ISS. The connection + state should be changed to SYN-RECEIVED. Note that any other + incoming control or data (combined with SYN) will be processed + in the SYN-RECEIVED state, but processing of SYN and ACK should + not be repeated. If the listen was not fully specified (i.e., + the remote socket was not fully specified), then the + unspecified fields should be filled in now. + + Fourth, other data or control: + + - This should not be reached. Drop the segment and return. Any + other control or data-bearing segment (not containing SYN) must + have an ACK and thus would have been discarded by the ACK + processing in the second step, unless it was first discarded by + RST checking in the first step. + +3.10.7.3. SYN-SENT STATE + + If the state is SYN-SENT, then + + First, check the ACK bit: + + - If the ACK bit is set, + + o If SEG.ACK =< ISS or SEG.ACK > SND.NXT, send a reset (unless + the RST bit is set, if so drop the segment and return) + + + + o and discard the segment. Return. + + o If SND.UNA < SEG.ACK =< SND.NXT, then the ACK is acceptable. + Some deployed TCP code has used the check SEG.ACK == SND.NXT + (using "==" rather than "=<"), but this is not appropriate + when the stack is capable of sending data on the SYN because + the TCP peer may not accept and acknowledge all of the data + on the SYN. + + Second, check the RST bit: + + - If the RST bit is set, + + o A potential blind reset attack is described in RFC 5961 [9]. + The mitigation described in that document has specific + applicability explained therein, and is not a substitute for + cryptographic protection (e.g., IPsec or TCP-AO). A TCP + implementation that supports the mitigation described in RFC + 5961 SHOULD first check that the sequence number exactly + matches RCV.NXT prior to executing the action in the next + paragraph. + + o If the ACK was acceptable, then signal to the user "error: + connection reset", drop the segment, enter CLOSED state, + delete TCB, and return. Otherwise (no ACK), drop the + segment and return. + + Third, check the security: + + - If the security/compartment in the segment does not exactly + match the security/compartment in the TCB, send a reset: + + o If there is an ACK, + + + + o Otherwise, + + + + - If a reset was sent, discard the segment and return. + + Fourth, check the SYN bit: + + - This step should be reached only if the ACK is ok, or there is + no ACK, and the segment did not contain a RST. + + - If the SYN bit is on and the security/compartment is + acceptable, then RCV.NXT is set to SEG.SEQ+1, IRS is set to + SEG.SEQ. SND.UNA should be advanced to equal SEG.ACK (if there + is an ACK), and any segments on the retransmission queue that + are thereby acknowledged should be removed. + + - If SND.UNA > ISS (our SYN has been ACKed), change the + connection state to ESTABLISHED, form an ACK segment + + + + - and send it. Data or controls that were queued for + transmission MAY be included. Some TCP implementations + suppress sending this segment when the received segment + contains data that will anyways generate an acknowledgment in + the later processing steps, saving this extra acknowledgment of + the SYN from being sent. If there are other controls or text + in the segment, then continue processing at the sixth step + under Section 3.10.7.4 where the URG bit is checked; otherwise, + return. + + - Otherwise, enter SYN-RECEIVED, form a SYN,ACK segment + + + + - and send it. Set the variables: + + SND.WND <- SEG.WND + + SND.WL1 <- SEG.SEQ + + SND.WL2 <- SEG.ACK + + If there are other controls or text in the segment, queue them + for processing after the ESTABLISHED state has been reached, + return. + + - Note that it is legal to send and receive application data on + SYN segments (this is the "text in the segment" mentioned + above). There has been significant misinformation and + misunderstanding of this topic historically. Some firewalls + and security devices consider this suspicious. However, the + capability was used in T/TCP [21] and is used in TCP Fast Open + (TFO) [48], so is important for implementations and network + devices to permit. + + Fifth, if neither of the SYN or RST bits is set, then drop the + segment and return. + +3.10.7.4. Other States + + Otherwise, + + First, check sequence number: + + - SYN-RECEIVED STATE + + - ESTABLISHED STATE + + - FIN-WAIT-1 STATE + + - FIN-WAIT-2 STATE + + - CLOSE-WAIT STATE + + - CLOSING STATE + + - LAST-ACK STATE + + - TIME-WAIT STATE + + o Segments are processed in sequence. Initial tests on + arrival are used to discard old duplicates, but further + processing is done in SEG.SEQ order. If a segment's + contents straddle the boundary between old and new, only the + new parts are processed. + + o In general, the processing of received segments MUST be + implemented to aggregate ACK segments whenever possible + (MUST-58). For example, if the TCP endpoint is processing a + series of queued segments, it MUST process them all before + sending any ACK segments (MUST-59). + + o There are four cases for the acceptability test for an + incoming segment: + + +=========+=========+======================================+ + | Segment | Receive | Test | + | Length | Window | | + +=========+=========+======================================+ + | 0 | 0 | SEG.SEQ = RCV.NXT | + +---------+---------+--------------------------------------+ + | 0 | >0 | RCV.NXT =< SEG.SEQ < | + | | | RCV.NXT+RCV.WND | + +---------+---------+--------------------------------------+ + | >0 | 0 | not acceptable | + +---------+---------+--------------------------------------+ + | >0 | >0 | RCV.NXT =< SEG.SEQ < | + | | | RCV.NXT+RCV.WND | + | | | | + | | | or | + | | | | + | | | RCV.NXT =< SEG.SEQ+SEG.LEN-1 | + | | | < RCV.NXT+RCV.WND | + +---------+---------+--------------------------------------+ + + Table 6: Segment Acceptability Tests + + o In implementing sequence number validation as described + here, please note Appendix A.2. + + o If the RCV.WND is zero, no segments will be acceptable, but + special allowance should be made to accept valid ACKs, URGs, + and RSTs. + + o If an incoming segment is not acceptable, an acknowledgment + should be sent in reply (unless the RST bit is set, if so + drop the segment and return): + + + + o After sending the acknowledgment, drop the unacceptable + segment and return. + + o Note that for the TIME-WAIT state, there is an improved + algorithm described in [40] for handling incoming SYN + segments that utilizes timestamps rather than relying on the + sequence number check described here. When the improved + algorithm is implemented, the logic above is not applicable + for incoming SYN segments with Timestamp Options, received + on a connection in the TIME-WAIT state. + + o In the following it is assumed that the segment is the + idealized segment that begins at RCV.NXT and does not exceed + the window. One could tailor actual segments to fit this + assumption by trimming off any portions that lie outside the + window (including SYN and FIN) and only processing further + if the segment then begins at RCV.NXT. Segments with higher + beginning sequence numbers SHOULD be held for later + processing (SHLD-31). + + Second, check the RST bit: + + - RFC 5961 [9], Section 3 describes a potential blind reset + attack and optional mitigation approach. This does not provide + a cryptographic protection (e.g., as in IPsec or TCP-AO) but + can be applicable in situations described in RFC 5961. For + stacks implementing the protection described in RFC 5961, the + three checks below apply; otherwise, processing for these + states is indicated further below. + + 1) If the RST bit is set and the sequence number is outside + the current receive window, silently drop the segment. + + 2) If the RST bit is set and the sequence number exactly + matches the next expected sequence number (RCV.NXT), then + TCP endpoints MUST reset the connection in the manner + prescribed below according to the connection state. + + 3) If the RST bit is set and the sequence number does not + exactly match the next expected sequence value, yet is + within the current receive window, TCP endpoints MUST send + an acknowledgment (challenge ACK): + + + + After sending the challenge ACK, TCP endpoints MUST drop + the unacceptable segment and stop processing the incoming + packet further. Note that RFC 5961 and Errata ID 4772 [99] + contain additional considerations for ACK throttling in an + implementation. + + - SYN-RECEIVED STATE + + o If the RST bit is set, + + + If this connection was initiated with a passive OPEN + (i.e., came from the LISTEN state), then return this + connection to LISTEN state and return. The user need not + be informed. If this connection was initiated with an + active OPEN (i.e., came from SYN-SENT state), then the + connection was refused; signal the user "connection + refused". In either case, the retransmission queue + should be flushed. And in the active OPEN case, enter + the CLOSED state and delete the TCB, and return. + + - ESTABLISHED STATE + + - FIN-WAIT-1 STATE + + - FIN-WAIT-2 STATE + + - CLOSE-WAIT STATE + + o If the RST bit is set, then any outstanding RECEIVEs and + SEND should receive "reset" responses. All segment queues + should be flushed. Users should also receive an unsolicited + general "connection reset" signal. Enter the CLOSED state, + delete the TCB, and return. + + - CLOSING STATE + + - LAST-ACK STATE + + - TIME-WAIT STATE + + o If the RST bit is set, then enter the CLOSED state, delete + the TCB, and return. + + Third, check security: + + - SYN-RECEIVED STATE + + o If the security/compartment in the segment does not exactly + match the security/compartment in the TCB, then send a reset + and return. + + - ESTABLISHED STATE + + - FIN-WAIT-1 STATE + + - FIN-WAIT-2 STATE + + - CLOSE-WAIT STATE + + - CLOSING STATE + + - LAST-ACK STATE + + - TIME-WAIT STATE + + o If the security/compartment in the segment does not exactly + match the security/compartment in the TCB, then send a + reset; any outstanding RECEIVEs and SEND should receive + "reset" responses. All segment queues should be flushed. + Users should also receive an unsolicited general "connection + reset" signal. Enter the CLOSED state, delete the TCB, and + return. + + - Note this check is placed following the sequence check to + prevent a segment from an old connection between these port + numbers with a different security from causing an abort of the + current connection. + + Fourth, check the SYN bit: + + - SYN-RECEIVED STATE + + o If the connection was initiated with a passive OPEN, then + return this connection to the LISTEN state and return. + Otherwise, handle per the directions for synchronized states + below. + + - ESTABLISHED STATE + + - FIN-WAIT-1 STATE + + - FIN-WAIT-2 STATE + + - CLOSE-WAIT STATE + + - CLOSING STATE + + - LAST-ACK STATE + + - TIME-WAIT STATE + + o If the SYN bit is set in these synchronized states, it may + be either a legitimate new connection attempt (e.g., in the + case of TIME-WAIT), an error where the connection should be + reset, or the result of an attack attempt, as described in + RFC 5961 [9]. For the TIME-WAIT state, new connections can + be accepted if the Timestamp Option is used and meets + expectations (per [40]). For all other cases, RFC 5961 + provides a mitigation with applicability to some situations, + though there are also alternatives that offer cryptographic + protection (see Section 7). RFC 5961 recommends that in + these synchronized states, if the SYN bit is set, + irrespective of the sequence number, TCP endpoints MUST send + a "challenge ACK" to the remote peer: + + + + o After sending the acknowledgment, TCP implementations MUST + drop the unacceptable segment and stop processing further. + Note that RFC 5961 and Errata ID 4772 [99] contain + additional ACK throttling notes for an implementation. + + o For implementations that do not follow RFC 5961, the + original behavior described in RFC 793 follows in this + paragraph. If the SYN is in the window it is an error: send + a reset, any outstanding RECEIVEs and SEND should receive + "reset" responses, all segment queues should be flushed, the + user should also receive an unsolicited general "connection + reset" signal, enter the CLOSED state, delete the TCB, and + return. + + o If the SYN is not in the window, this step would not be + reached and an ACK would have been sent in the first step + (sequence number check). + + Fifth, check the ACK field: + + - if the ACK bit is off, drop the segment and return + + - if the ACK bit is on, + + o RFC 5961 [9], Section 5 describes a potential blind data + injection attack, and mitigation that implementations MAY + choose to include (MAY-12). TCP stacks that implement RFC + 5961 MUST add an input check that the ACK value is + acceptable only if it is in the range of ((SND.UNA - + MAX.SND.WND) =< SEG.ACK =< SND.NXT). All incoming segments + whose ACK value doesn't satisfy the above condition MUST be + discarded and an ACK sent back. The new state variable + MAX.SND.WND is defined as the largest window that the local + sender has ever received from its peer (subject to window + scaling) or may be hard-coded to a maximum permissible + window value. When the ACK value is acceptable, the per- + state processing below applies: + + o SYN-RECEIVED STATE + + + If SND.UNA < SEG.ACK =< SND.NXT, then enter ESTABLISHED + state and continue processing with the variables below + set to: + + SND.WND <- SEG.WND + + SND.WL1 <- SEG.SEQ + + SND.WL2 <- SEG.ACK + + + If the segment acknowledgment is not acceptable, form a + reset segment + + + + + and send it. + + o ESTABLISHED STATE + + + If SND.UNA < SEG.ACK =< SND.NXT, then set SND.UNA <- + SEG.ACK. Any segments on the retransmission queue that + are thereby entirely acknowledged are removed. Users + should receive positive acknowledgments for buffers that + have been SENT and fully acknowledged (i.e., SEND buffer + should be returned with "ok" response). If the ACK is a + duplicate (SEG.ACK =< SND.UNA), it can be ignored. If + the ACK acks something not yet sent (SEG.ACK > SND.NXT), + then send an ACK, drop the segment, and return. + + + If SND.UNA =< SEG.ACK =< SND.NXT, the send window should + be updated. If (SND.WL1 < SEG.SEQ or (SND.WL1 = SEG.SEQ + and SND.WL2 =< SEG.ACK)), set SND.WND <- SEG.WND, set + SND.WL1 <- SEG.SEQ, and set SND.WL2 <- SEG.ACK. + + + Note that SND.WND is an offset from SND.UNA, that SND.WL1 + records the sequence number of the last segment used to + update SND.WND, and that SND.WL2 records the + acknowledgment number of the last segment used to update + SND.WND. The check here prevents using old segments to + update the window. + + o FIN-WAIT-1 STATE + + + In addition to the processing for the ESTABLISHED state, + if the FIN segment is now acknowledged, then enter FIN- + WAIT-2 and continue processing in that state. + + o FIN-WAIT-2 STATE + + + In addition to the processing for the ESTABLISHED state, + if the retransmission queue is empty, the user's CLOSE + can be acknowledged ("ok") but do not delete the TCB. + + o CLOSE-WAIT STATE + + + Do the same processing as for the ESTABLISHED state. + + o CLOSING STATE + + + In addition to the processing for the ESTABLISHED state, + if the ACK acknowledges our FIN, then enter the TIME-WAIT + state; otherwise, ignore the segment. + + o LAST-ACK STATE + + + The only thing that can arrive in this state is an + acknowledgment of our FIN. If our FIN is now + acknowledged, delete the TCB, enter the CLOSED state, and + return. + + o TIME-WAIT STATE + + + The only thing that can arrive in this state is a + retransmission of the remote FIN. Acknowledge it, and + restart the 2 MSL timeout. + + Sixth, check the URG bit: + + - ESTABLISHED STATE + + - FIN-WAIT-1 STATE + + - FIN-WAIT-2 STATE + + o If the URG bit is set, RCV.UP <- max(RCV.UP,SEG.UP), and + signal the user that the remote side has urgent data if the + urgent pointer (RCV.UP) is in advance of the data consumed. + If the user has already been signaled (or is still in the + "urgent mode") for this continuous sequence of urgent data, + do not signal the user again. + + - CLOSE-WAIT STATE + + - CLOSING STATE + + - LAST-ACK STATE + + - TIME-WAIT STATE + + o This should not occur since a FIN has been received from the + remote side. Ignore the URG. + + Seventh, process the segment text: + + - ESTABLISHED STATE + + - FIN-WAIT-1 STATE + + - FIN-WAIT-2 STATE + + o Once in the ESTABLISHED state, it is possible to deliver + segment data to user RECEIVE buffers. Data from segments + can be moved into buffers until either the buffer is full or + the segment is empty. If the segment empties and carries a + PUSH flag, then the user is informed, when the buffer is + returned, that a PUSH has been received. + + o When the TCP endpoint takes responsibility for delivering + the data to the user, it must also acknowledge the receipt + of the data. + + o Once the TCP endpoint takes responsibility for the data, it + advances RCV.NXT over the data accepted, and adjusts RCV.WND + as appropriate to the current buffer availability. The + total of RCV.NXT and RCV.WND should not be reduced. + + o A TCP implementation MAY send an ACK segment acknowledging + RCV.NXT when a valid segment arrives that is in the window + but not at the left window edge (MAY-13). + + o Please note the window management suggestions in + Section 3.8. + + o Send an acknowledgment of the form: + + + + o This acknowledgment should be piggybacked on a segment being + transmitted if possible without incurring undue delay. + + - CLOSE-WAIT STATE + + - CLOSING STATE + + - LAST-ACK STATE + + - TIME-WAIT STATE + + o This should not occur since a FIN has been received from the + remote side. Ignore the segment text. + + Eighth, check the FIN bit: + + - Do not process the FIN if the state is CLOSED, LISTEN, or SYN- + SENT since the SEG.SEQ cannot be validated; drop the segment + and return. + + - If the FIN bit is set, signal the user "connection closing" and + return any pending RECEIVEs with same message, advance RCV.NXT + over the FIN, and send an acknowledgment for the FIN. Note + that FIN implies PUSH for any segment text not yet delivered to + the user. + + o SYN-RECEIVED STATE + + o ESTABLISHED STATE + + + Enter the CLOSE-WAIT state. + + o FIN-WAIT-1 STATE + + + If our FIN has been ACKed (perhaps in this segment), then + enter TIME-WAIT, start the time-wait timer, turn off the + other timers; otherwise, enter the CLOSING state. + + o FIN-WAIT-2 STATE + + + Enter the TIME-WAIT state. Start the time-wait timer, + turn off the other timers. + + o CLOSE-WAIT STATE + + + Remain in the CLOSE-WAIT state. + + o CLOSING STATE + + + Remain in the CLOSING state. + + o LAST-ACK STATE + + + Remain in the LAST-ACK state. + + o TIME-WAIT STATE + + + Remain in the TIME-WAIT state. Restart the 2 MSL time- + wait timeout. + + and return. + +3.10.8. Timeouts + + USER TIMEOUT + + * For any state if the user timeout expires, flush all queues, + signal the user "error: connection aborted due to user timeout" in + general and for any outstanding calls, delete the TCB, enter the + CLOSED state, and return. + + RETRANSMISSION TIMEOUT + + * For any state if the retransmission timeout expires on a segment + in the retransmission queue, send the segment at the front of the + retransmission queue again, reinitialize the retransmission timer, + and return. + + TIME-WAIT TIMEOUT + + * If the time-wait timeout expires on a connection, delete the TCB, + enter the CLOSED state, and return. + +4. Glossary + + ACK + A control bit (acknowledge) occupying no sequence space, + which indicates that the acknowledgment field of this segment + specifies the next sequence number the sender of this segment + is expecting to receive, hence acknowledging receipt of all + previous sequence numbers. + + connection + A logical communication path identified by a pair of sockets. + + datagram + A message sent in a packet-switched computer communications + network. + + Destination Address + The network-layer address of the endpoint intended to receive + a segment. + + FIN + A control bit (finis) occupying one sequence number, which + indicates that the sender will send no more data or control + occupying sequence space. + + flush + To remove all of the contents (data or segments) from a store + (buffer or queue). + + fragment + A portion of a logical unit of data. In particular, an + internet fragment is a portion of an internet datagram. + + header + Control information at the beginning of a message, segment, + fragment, packet, or block of data. + + host + A computer. In particular, a source or destination of + messages from the point of view of the communication network. + + Identification + An Internet Protocol field. This identifying value assigned + by the sender aids in assembling the fragments of a datagram. + + internet address + A network-layer address. + + internet datagram + A unit of data exchanged between internet hosts, together + with the internet header that allows the datagram to be + routed from source to destination. + + internet fragment + A portion of the data of an internet datagram with an + internet header. + + IP + Internet Protocol. See [1] and [13]. + + IRS + The Initial Receive Sequence number. The first sequence + number used by the sender on a connection. + + ISN + The Initial Sequence Number. The first sequence number used + on a connection (either ISS or IRS). Selected in a way that + is unique within a given period of time and is unpredictable + to attackers. + + ISS + The Initial Send Sequence number. The first sequence number + used by the sender on a connection. + + left sequence + This is the next sequence number to be acknowledged by the + data-receiving TCP endpoint (or the lowest currently + unacknowledged sequence number) and is sometimes referred to + as the left edge of the send window. + + module + An implementation, usually in software, of a protocol or + other procedure. + + MSL + Maximum Segment Lifetime, the time a TCP segment can exist in + the internetwork system. Arbitrarily defined to be 2 + minutes. + + octet + An eight-bit byte. + + Options + An Option field may contain several options, and each option + may be several octets in length. + + packet + A package of data with a header that may or may not be + logically complete. More often a physical packaging than a + logical packaging of data. + + port + The portion of a connection identifier used for + demultiplexing connections at an endpoint. + + process + A program in execution. A source or destination of data from + the point of view of the TCP endpoint or other host-to-host + protocol. + + PUSH + A control bit occupying no sequence space, indicating that + this segment contains data that must be pushed through to the + receiving user. + + RCV.NXT + receive next sequence number + + RCV.UP + receive urgent pointer + + RCV.WND + receive window + + receive next sequence number + This is the next sequence number the local TCP endpoint is + expecting to receive. + + receive window + This represents the sequence numbers the local (receiving) + TCP endpoint is willing to receive. Thus, the local TCP + endpoint considers that segments overlapping the range + RCV.NXT to RCV.NXT + RCV.WND - 1 carry acceptable data or + control. Segments containing sequence numbers entirely + outside this range are considered duplicates or injection + attacks and discarded. + + RST + A control bit (reset), occupying no sequence space, + indicating that the receiver should delete the connection + without further interaction. The receiver can determine, + based on the sequence number and acknowledgment fields of the + incoming segment, whether it should honor the reset command + or ignore it. In no case does receipt of a segment + containing RST give rise to a RST in response. + + SEG.ACK + segment acknowledgment + + SEG.LEN + segment length + + SEG.SEQ + segment sequence + + SEG.UP + segment urgent pointer field + + SEG.WND + segment window field + + segment + A logical unit of data. In particular, a TCP segment is the + unit of data transferred between a pair of TCP modules. + + segment acknowledgment + The sequence number in the acknowledgment field of the + arriving segment. + + segment length + The amount of sequence number space occupied by a segment, + including any controls that occupy sequence space. + + segment sequence + The number in the sequence field of the arriving segment. + + send sequence + This is the next sequence number the local (sending) TCP + endpoint will use on the connection. It is initially + selected from an initial sequence number curve (ISN) and is + incremented for each octet of data or sequenced control + transmitted. + + send window + This represents the sequence numbers that the remote + (receiving) TCP endpoint is willing to receive. It is the + value of the window field specified in segments from the + remote (data-receiving) TCP endpoint. The range of new + sequence numbers that may be emitted by a TCP implementation + lies between SND.NXT and SND.UNA + SND.WND - 1. + (Retransmissions of sequence numbers between SND.UNA and + SND.NXT are expected, of course.) + + SND.NXT + send sequence + + SND.UNA + left sequence + + SND.UP + send urgent pointer + + SND.WL1 + segment sequence number at last window update + + SND.WL2 + segment acknowledgment number at last window update + + SND.WND + send window + + socket (or socket number, or socket address, or socket identifier) + An address that specifically includes a port identifier, that + is, the concatenation of an Internet Address with a TCP port. + + Source Address + The network-layer address of the sending endpoint. + + SYN + A control bit in the incoming segment, occupying one sequence + number, used at the initiation of a connection to indicate + where the sequence numbering will start. + + TCB + Transmission control block, the data structure that records + the state of a connection. + + TCP + Transmission Control Protocol: a host-to-host protocol for + reliable communication in internetwork environments. + + TOS + Type of Service, an obsoleted IPv4 field. The same header + bits currently are used for the Differentiated Services field + [4] containing the Differentiated Services Codepoint (DSCP) + value and the 2-bit ECN codepoint [6]. + + Type of Service + See "TOS". + + URG + A control bit (urgent), occupying no sequence space, used to + indicate that the receiving user should be notified to do + urgent processing as long as there is data to be consumed + with sequence numbers less than the value indicated by the + urgent pointer. + + urgent pointer + A control field meaningful only when the URG bit is on. This + field communicates the value of the urgent pointer that + indicates the data octet associated with the sending user's + urgent call. + +5. Changes from RFC 793 + + This document obsoletes RFC 793 as well as RFCs 6093 and 6528, which + updated 793. In all cases, only the normative protocol specification + and requirements have been incorporated into this document, and some + informational text with background and rationale may not have been + carried in. The informational content of those documents is still + valuable in learning about and understanding TCP, and they are valid + Informational references, even though their normative content has + been incorporated into this document. + + The main body of this document was adapted from RFC 793's Section 3, + titled "FUNCTIONAL SPECIFICATION", with an attempt to keep formatting + and layout as close as possible. + + The collection of applicable RFC errata that have been reported and + either accepted or held for an update to RFC 793 were incorporated + (Errata IDs: 573 [73], 574 [74], 700 [75], 701 [76], 1283 [77], 1561 + [78], 1562 [79], 1564 [80], 1571 [81], 1572 [82], 2297 [83], 2298 + [84], 2748 [85], 2749 [86], 2934 [87], 3213 [88], 3300 [89], 3301 + [90], 6222 [91]). Some errata were not applicable due to other + changes (Errata IDs: 572 [92], 575 [93], 1565 [94], 1569 [95], 2296 + [96], 3305 [97], 3602 [98]). + + Changes to the specification of the urgent pointer described in RFCs + 1011, 1122, and 6093 were incorporated. See RFC 6093 for detailed + discussion of why these changes were necessary. + + The discussion of the RTO from RFC 793 was updated to refer to RFC + 6298. The text on the RTO in RFC 1122 originally replaced the text + in RFC 793; however, RFC 2988 should have updated RFC 1122 and has + subsequently been obsoleted by RFC 6298. + + RFC 1011 [18] contains a number of comments about RFC 793, including + some needed changes to the TCP specification. These are expanded in + RFC 1122, which contains a collection of other changes and + clarifications to RFC 793. The normative items impacting the + protocol have been incorporated here, though some historically useful + implementation advice and informative discussion from RFC 1122 is not + included here. The present document, which is now the TCP + specification rather than RFC 793, updates RFC 1011, and the comments + noted in RFC 1011 have been incorporated. + + RFC 1122 contains more than just TCP requirements, so this document + can't obsolete RFC 1122 entirely. It is only marked as "updating" + RFC 1122; however, it should be understood to effectively obsolete + all of the material on TCP found in RFC 1122. + + The more secure initial sequence number generation algorithm from RFC + 6528 was incorporated. See RFC 6528 for discussion of the attacks + that this mitigates, as well as advice on selecting PRF algorithms + and managing secret key data. + + A note based on RFC 6429 was added to explicitly clarify that system + resource management concerns allow connection resources to be + reclaimed. RFC 6429 is obsoleted in the sense that the clarification + it describes has been reflected within this base TCP specification. + + The description of congestion control implementation was added based + on the set of documents that are IETF BCP or Standards Track on the + topic and the current state of common implementations. + +6. IANA Considerations + + In the "Transmission Control Protocol (TCP) Header Flags" registry, + IANA has made several changes as described in this section. + + RFC 3168 originally created this registry but only populated it with + the new bits defined in RFC 3168, neglecting the other bits that had + previously been described in RFC 793 and other documents. Bit 7 has + since also been updated by RFC 8311 [54]. + + The "Bit" column has been renamed below as the "Bit Offset" column + because it references each header flag's offset within the 16-bit + aligned view of the TCP header in Figure 1. The bits in offsets 0 + through 3 are the TCP segment Data Offset field, and not header + flags. + + IANA has added a column for "Assignment Notes". + + IANA has assigned values as indicated below. + + +========+===================+===========+====================+ + | Bit | Name | Reference | Assignment Notes | + | Offset | | | | + +========+===================+===========+====================+ + | 4 | Reserved for | RFC 9293 | | + | | future use | | | + +--------+-------------------+-----------+--------------------+ + | 5 | Reserved for | RFC 9293 | | + | | future use | | | + +--------+-------------------+-----------+--------------------+ + | 6 | Reserved for | RFC 9293 | | + | | future use | | | + +--------+-------------------+-----------+--------------------+ + | 7 | Reserved for | RFC 8311 | Previously used by | + | | future use | | Historic RFC 3540 | + | | | | as NS (Nonce Sum). | + +--------+-------------------+-----------+--------------------+ + | 8 | CWR (Congestion | RFC 3168 | | + | | Window Reduced) | | | + +--------+-------------------+-----------+--------------------+ + | 9 | ECE (ECN-Echo) | RFC 3168 | | + +--------+-------------------+-----------+--------------------+ + | 10 | Urgent pointer | RFC 9293 | | + | | field is | | | + | | significant (URG) | | | + +--------+-------------------+-----------+--------------------+ + | 11 | Acknowledgment | RFC 9293 | | + | | field is | | | + | | significant (ACK) | | | + +--------+-------------------+-----------+--------------------+ + | 12 | Push function | RFC 9293 | | + | | (PSH) | | | + +--------+-------------------+-----------+--------------------+ + | 13 | Reset the | RFC 9293 | | + | | connection (RST) | | | + +--------+-------------------+-----------+--------------------+ + | 14 | Synchronize | RFC 9293 | | + | | sequence numbers | | | + | | (SYN) | | | + +--------+-------------------+-----------+--------------------+ + | 15 | No more data from | RFC 9293 | | + | | sender (FIN) | | | + +--------+-------------------+-----------+--------------------+ + + Table 7: TCP Header Flags + + The "TCP Header Flags" registry has also been moved to a subregistry + under the global "Transmission Control Protocol (TCP) Parameters" + registry . + + The registry's Registration Procedure remains Standards Action, but + the Reference has been updated to this document, and the Note has + been removed. + +7. Security and Privacy Considerations + + The TCP design includes only rudimentary security features that + improve the robustness and reliability of connections and application + data transfer, but there are no built-in cryptographic capabilities + to support any form of confidentiality, authentication, or other + typical security functions. Non-cryptographic enhancements (e.g., + [9]) have been developed to improve robustness of TCP connections to + particular types of attacks, but the applicability and protections of + non-cryptographic enhancements are limited (e.g., see Section 1.1 of + [9]). Applications typically utilize lower-layer (e.g., IPsec) and + upper-layer (e.g., TLS) protocols to provide security and privacy for + TCP connections and application data carried in TCP. Methods based + on TCP Options have been developed as well, to support some security + capabilities. + + In order to fully provide confidentiality, integrity protection, and + authentication for TCP connections (including their control flags), + IPsec is the only current effective method. For integrity protection + and authentication, the TCP Authentication Option (TCP-AO) [38] is + available, with a proposed extension to also provide confidentiality + for the segment payload. Other methods discussed in this section may + provide confidentiality or integrity protection for the payload, but + for the TCP header only cover either a subset of the fields (e.g., + tcpcrypt [57]) or none at all (e.g., TLS). Other security features + that have been added to TCP (e.g., ISN generation, sequence number + checks, and others) are only capable of partially hindering attacks. + + Applications using long-lived TCP flows have been vulnerable to + attacks that exploit the processing of control flags described in + earlier TCP specifications [33]. TCP-MD5 was a commonly implemented + TCP Option to support authentication for some of these connections, + but had flaws and is now deprecated. TCP-AO provides a capability to + protect long-lived TCP connections from attacks and has superior + properties to TCP-MD5. It does not provide any privacy for + application data or for the TCP headers. + + The "tcpcrypt" [57] experimental extension to TCP provides the + ability to cryptographically protect connection data. Metadata + aspects of the TCP flow are still visible, but the application stream + is well protected. Within the TCP header, only the urgent pointer + and FIN flag are protected through tcpcrypt. + + The TCP Roadmap [49] includes notes about several RFCs related to TCP + security. Many of the enhancements provided by these RFCs have been + integrated into the present document, including ISN generation, + mitigating blind in-window attacks, and improving handling of soft + errors and ICMP packets. These are all discussed in greater detail + in the referenced RFCs that originally described the changes needed + to earlier TCP specifications. Additionally, see RFC 6093 [39] for + discussion of security considerations related to the urgent pointer + field, which also discourages new applications from using the urgent + pointer. + + Since TCP is often used for bulk transfer flows, some attacks are + possible that abuse the TCP congestion control logic. An example is + "ACK-division" attacks. Updates that have been made to the TCP + congestion control specifications include mechanisms like Appropriate + Byte Counting (ABC) [29] that act as mitigations to these attacks. + + Other attacks are focused on exhausting the resources of a TCP + server. Examples include SYN flooding [32] or wasting resources on + non-progressing connections [41]. Operating systems commonly + implement mitigations for these attacks. Some common defenses also + utilize proxies, stateful firewalls, and other technologies outside + the end-host TCP implementation. + + The concept of a protocol's "wire image" is described in RFC 8546 + [56], which describes how TCP's cleartext headers expose more + metadata to nodes on the path than is strictly required to route the + packets to their destination. On-path adversaries may be able to + leverage this metadata. Lessons learned in this respect from TCP + have been applied in the design of newer transports like QUIC [60]. + Additionally, based partly on experiences with TCP and its + extensions, there are considerations that might be applicable for + future TCP extensions and other transports that the IETF has + documented in RFC 9065 [61], along with IAB recommendations in RFC + 8558 [58] and [67]. + + There are also methods of "fingerprinting" that can be used to infer + the host TCP implementation (operating system) version or platform + information. These collect observations of several aspects, such as + the options present in segments, the ordering of options, the + specific behaviors in the case of various conditions, packet timing, + packet sizing, and other aspects of the protocol that are left to be + determined by an implementer, and can use those observations to + identify information about the host and implementation. + + Since ICMP message processing also can interact with TCP connections, + there is potential for ICMP-based attacks against TCP connections. + These are discussed in RFC 5927 [100], along with mitigations that + have been implemented. + +8. References + +8.1. Normative References + + [1] Postel, J., "Internet Protocol", STD 5, RFC 791, + DOI 10.17487/RFC0791, September 1981, + . + + [2] Mogul, J. and S. Deering, "Path MTU discovery", RFC 1191, + DOI 10.17487/RFC1191, November 1990, + . + + [3] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, + DOI 10.17487/RFC2119, March 1997, + . + + [4] Nichols, K., Blake, S., Baker, F., and D. Black, + "Definition of the Differentiated Services Field (DS + Field) in the IPv4 and IPv6 Headers", RFC 2474, + DOI 10.17487/RFC2474, December 1998, + . + + [5] Floyd, S., "Congestion Control Principles", BCP 41, + RFC 2914, DOI 10.17487/RFC2914, September 2000, + . + + [6] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition + of Explicit Congestion Notification (ECN) to IP", + RFC 3168, DOI 10.17487/RFC3168, September 2001, + . + + [7] Floyd, S. and M. Allman, "Specifying New Congestion + Control Algorithms", BCP 133, RFC 5033, + DOI 10.17487/RFC5033, August 2007, + . + + [8] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion + Control", RFC 5681, DOI 10.17487/RFC5681, September 2009, + . + + [9] Ramaiah, A., Stewart, R., and M. Dalal, "Improving TCP's + Robustness to Blind In-Window Attacks", RFC 5961, + DOI 10.17487/RFC5961, August 2010, + . + + [10] Paxson, V., Allman, M., Chu, J., and M. Sargent, + "Computing TCP's Retransmission Timer", RFC 6298, + DOI 10.17487/RFC6298, June 2011, + . + + [11] Gont, F., "Deprecation of ICMP Source Quench Messages", + RFC 6633, DOI 10.17487/RFC6633, May 2012, + . + + [12] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC + 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, + May 2017, . + + [13] Deering, S. and R. Hinden, "Internet Protocol, Version 6 + (IPv6) Specification", STD 86, RFC 8200, + DOI 10.17487/RFC8200, July 2017, + . + + [14] McCann, J., Deering, S., Mogul, J., and R. Hinden, Ed., + "Path MTU Discovery for IP version 6", STD 87, RFC 8201, + DOI 10.17487/RFC8201, July 2017, + . + + [15] Allman, M., "Requirements for Time-Based Loss Detection", + BCP 233, RFC 8961, DOI 10.17487/RFC8961, November 2020, + . + +8.2. Informative References + + [16] Postel, J., "Transmission Control Protocol", STD 7, + RFC 793, DOI 10.17487/RFC0793, September 1981, + . + + [17] Nagle, J., "Congestion Control in IP/TCP Internetworks", + RFC 896, DOI 10.17487/RFC0896, January 1984, + . + + [18] Reynolds, J. and J. Postel, "Official Internet protocols", + RFC 1011, DOI 10.17487/RFC1011, May 1987, + . + + [19] Braden, R., Ed., "Requirements for Internet Hosts - + Communication Layers", STD 3, RFC 1122, + DOI 10.17487/RFC1122, October 1989, + . + + [20] Almquist, P., "Type of Service in the Internet Protocol + Suite", RFC 1349, DOI 10.17487/RFC1349, July 1992, + . + + [21] Braden, R., "T/TCP -- TCP Extensions for Transactions + Functional Specification", RFC 1644, DOI 10.17487/RFC1644, + July 1994, . + + [22] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP + Selective Acknowledgment Options", RFC 2018, + DOI 10.17487/RFC2018, October 1996, + . + + [23] Paxson, V., Allman, M., Dawson, S., Fenner, W., Griner, + J., Heavens, I., Lahey, K., Semke, J., and B. Volz, "Known + TCP Implementation Problems", RFC 2525, + DOI 10.17487/RFC2525, March 1999, + . + + [24] Borman, D., Deering, S., and R. Hinden, "IPv6 Jumbograms", + RFC 2675, DOI 10.17487/RFC2675, August 1999, + . + + [25] Xiao, X., Hannan, A., Paxson, V., and E. Crabbe, "TCP + Processing of the IPv4 Precedence Field", RFC 2873, + DOI 10.17487/RFC2873, June 2000, + . + + [26] Floyd, S., Mahdavi, J., Mathis, M., and M. Podolsky, "An + Extension to the Selective Acknowledgement (SACK) Option + for TCP", RFC 2883, DOI 10.17487/RFC2883, July 2000, + . + + [27] Lahey, K., "TCP Problems with Path MTU Discovery", + RFC 2923, DOI 10.17487/RFC2923, September 2000, + . + + [28] Balakrishnan, H., Padmanabhan, V., Fairhurst, G., and M. + Sooriyabandara, "TCP Performance Implications of Network + Path Asymmetry", BCP 69, RFC 3449, DOI 10.17487/RFC3449, + December 2002, . + + [29] Allman, M., "TCP Congestion Control with Appropriate Byte + Counting (ABC)", RFC 3465, DOI 10.17487/RFC3465, February + 2003, . + + [30] Fenner, B., "Experimental Values In IPv4, IPv6, ICMPv4, + ICMPv6, UDP, and TCP Headers", RFC 4727, + DOI 10.17487/RFC4727, November 2006, + . + + [31] Mathis, M. and J. Heffner, "Packetization Layer Path MTU + Discovery", RFC 4821, DOI 10.17487/RFC4821, March 2007, + . + + [32] Eddy, W., "TCP SYN Flooding Attacks and Common + Mitigations", RFC 4987, DOI 10.17487/RFC4987, August 2007, + . + + [33] Touch, J., "Defending TCP Against Spoofing Attacks", + RFC 4953, DOI 10.17487/RFC4953, July 2007, + . + + [34] Culley, P., Elzur, U., Recio, R., Bailey, S., and J. + Carrier, "Marker PDU Aligned Framing for TCP + Specification", RFC 5044, DOI 10.17487/RFC5044, October + 2007, . + + [35] Gont, F., "TCP's Reaction to Soft Errors", RFC 5461, + DOI 10.17487/RFC5461, February 2009, + . + + [36] StJohns, M., Atkinson, R., and G. Thomas, "Common + Architecture Label IPv6 Security Option (CALIPSO)", + RFC 5570, DOI 10.17487/RFC5570, July 2009, + . + + [37] Sandlund, K., Pelletier, G., and L-E. Jonsson, "The RObust + Header Compression (ROHC) Framework", RFC 5795, + DOI 10.17487/RFC5795, March 2010, + . + + [38] Touch, J., Mankin, A., and R. Bonica, "The TCP + Authentication Option", RFC 5925, DOI 10.17487/RFC5925, + June 2010, . + + [39] Gont, F. and A. Yourtchenko, "On the Implementation of the + TCP Urgent Mechanism", RFC 6093, DOI 10.17487/RFC6093, + January 2011, . + + [40] Gont, F., "Reducing the TIME-WAIT State Using TCP + Timestamps", BCP 159, RFC 6191, DOI 10.17487/RFC6191, + April 2011, . + + [41] Bashyam, M., Jethanandani, M., and A. Ramaiah, "TCP Sender + Clarification for Persist Condition", RFC 6429, + DOI 10.17487/RFC6429, December 2011, + . + + [42] Gont, F. and S. Bellovin, "Defending against Sequence + Number Attacks", RFC 6528, DOI 10.17487/RFC6528, February + 2012, . + + [43] Borman, D., "TCP Options and Maximum Segment Size (MSS)", + RFC 6691, DOI 10.17487/RFC6691, July 2012, + . + + [44] Touch, J., "Updated Specification of the IPv4 ID Field", + RFC 6864, DOI 10.17487/RFC6864, February 2013, + . + + [45] Touch, J., "Shared Use of Experimental TCP Options", + RFC 6994, DOI 10.17487/RFC6994, August 2013, + . + + [46] McPherson, D., Oran, D., Thaler, D., and E. Osterweil, + "Architectural Considerations of IP Anycast", RFC 7094, + DOI 10.17487/RFC7094, January 2014, + . + + [47] Borman, D., Braden, B., Jacobson, V., and R. + Scheffenegger, Ed., "TCP Extensions for High Performance", + RFC 7323, DOI 10.17487/RFC7323, September 2014, + . + + [48] Cheng, Y., Chu, J., Radhakrishnan, S., and A. Jain, "TCP + Fast Open", RFC 7413, DOI 10.17487/RFC7413, December 2014, + . + + [49] Duke, M., Braden, R., Eddy, W., Blanton, E., and A. + Zimmermann, "A Roadmap for Transmission Control Protocol + (TCP) Specification Documents", RFC 7414, + DOI 10.17487/RFC7414, February 2015, + . + + [50] Black, D., Ed. and P. Jones, "Differentiated Services + (Diffserv) and Real-Time Communication", RFC 7657, + DOI 10.17487/RFC7657, November 2015, + . + + [51] Fairhurst, G. and M. Welzl, "The Benefits of Using + Explicit Congestion Notification (ECN)", RFC 8087, + DOI 10.17487/RFC8087, March 2017, + . + + [52] Fairhurst, G., Ed., Trammell, B., Ed., and M. Kuehlewind, + Ed., "Services Provided by IETF Transport Protocols and + Congestion Control Mechanisms", RFC 8095, + DOI 10.17487/RFC8095, March 2017, + . + + [53] Welzl, M., Tuexen, M., and N. Khademi, "On the Usage of + Transport Features Provided by IETF Transport Protocols", + RFC 8303, DOI 10.17487/RFC8303, February 2018, + . + + [54] Black, D., "Relaxing Restrictions on Explicit Congestion + Notification (ECN) Experimentation", RFC 8311, + DOI 10.17487/RFC8311, January 2018, + . + + [55] Chown, T., Loughney, J., and T. Winters, "IPv6 Node + Requirements", BCP 220, RFC 8504, DOI 10.17487/RFC8504, + January 2019, . + + [56] Trammell, B. and M. Kuehlewind, "The Wire Image of a + Network Protocol", RFC 8546, DOI 10.17487/RFC8546, April + 2019, . + + [57] Bittau, A., Giffin, D., Handley, M., Mazieres, D., Slack, + Q., and E. Smith, "Cryptographic Protection of TCP Streams + (tcpcrypt)", RFC 8548, DOI 10.17487/RFC8548, May 2019, + . + + [58] Hardie, T., Ed., "Transport Protocol Path Signals", + RFC 8558, DOI 10.17487/RFC8558, April 2019, + . + + [59] Ford, A., Raiciu, C., Handley, M., Bonaventure, O., and C. + Paasch, "TCP Extensions for Multipath Operation with + Multiple Addresses", RFC 8684, DOI 10.17487/RFC8684, March + 2020, . + + [60] Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based + Multiplexed and Secure Transport", RFC 9000, + DOI 10.17487/RFC9000, May 2021, + . + + [61] Fairhurst, G. and C. Perkins, "Considerations around + Transport Header Confidentiality, Network Operations, and + the Evolution of Internet Transport Protocols", RFC 9065, + DOI 10.17487/RFC9065, July 2021, + . + + [62] IANA, "Transmission Control Protocol (TCP) Parameters", + . + + [63] Gont, F., "Processing of IP Security/Compartment and + Precedence Information by TCP", Work in Progress, + Internet-Draft, draft-gont-tcpm-tcp-seccomp-prec-00, 29 + March 2012, . + + [64] Gont, F. and D. Borman, "On the Validation of TCP Sequence + Numbers", Work in Progress, Internet-Draft, draft-gont- + tcpm-tcp-seq-validation-04, 11 March 2019, + . + + [65] Touch, J. and W. M. Eddy, "TCP Extended Data Offset + Option", Work in Progress, Internet-Draft, draft-ietf- + tcpm-tcp-edo-12, 15 April 2022, + . + + [66] McQuistin, S., Band, V., Jacob, D., and C. Perkins, + "Describing Protocol Data Units with Augmented Packet + Header Diagrams", Work in Progress, Internet-Draft, draft- + mcquistin-augmented-ascii-diagrams-10, 7 March 2022, + . + + [67] Thomson, M. and T. Pauly, "Long-Term Viability of Protocol + Extension Mechanisms", RFC 9170, DOI 10.17487/RFC9170, + December 2021, . + + [68] Minshall, G., "A Suggested Modification to Nagle's + Algorithm", Work in Progress, Internet-Draft, draft- + minshall-nagle-01, 18 June 1999, + . + + [69] Dalal, Y. and C. Sunshine, "Connection Management in + Transport Protocols", Computer Networks, Vol. 2, No. 6, + pp. 454-473, DOI 10.1016/0376-5075(78)90053-3, December + 1978, . + + [70] Faber, T., Touch, J., and W. Yui, "The TIME-WAIT state in + TCP and Its Effect on Busy Servers", Proceedings of IEEE + INFOCOM, pp. 1573-1583, DOI 10.1109/INFCOM.1999.752180, + March 1999, . + + [71] Postel, J., "Comments on Action Items from the January + Meeting", IEN 177, March 1981, + . + + [72] "Segmentation Offloads", The Linux Kernel Documentation, + . + + [73] RFC Errata, Erratum ID 573, RFC 793, + . + + [74] RFC Errata, Erratum ID 574, RFC 793, + . + + [75] RFC Errata, Erratum ID 700, RFC 793, + . + + [76] RFC Errata, Erratum ID 701, RFC 793, + . + + [77] RFC Errata, Erratum ID 1283, RFC 793, + . + + [78] RFC Errata, Erratum ID 1561, RFC 793, + . + + [79] RFC Errata, Erratum ID 1562, RFC 793, + . + + [80] RFC Errata, Erratum ID 1564, RFC 793, + . + + [81] RFC Errata, Erratum ID 1571, RFC 793, + . + + [82] RFC Errata, Erratum ID 1572, RFC 793, + . + + [83] RFC Errata, Erratum ID 2297, RFC 793, + . + + [84] RFC Errata, Erratum ID 2298, RFC 793, + . + + [85] RFC Errata, Erratum ID 2748, RFC 793, + . + + [86] RFC Errata, Erratum ID 2749, RFC 793, + . + + [87] RFC Errata, Erratum ID 2934, RFC 793, + . + + [88] RFC Errata, Erratum ID 3213, RFC 793, + . + + [89] RFC Errata, Erratum ID 3300, RFC 793, + . + + [90] RFC Errata, Erratum ID 3301, RFC 793, + . + + [91] RFC Errata, Erratum ID 6222, RFC 793, + . + + [92] RFC Errata, Erratum ID 572, RFC 793, + . + + [93] RFC Errata, Erratum ID 575, RFC 793, + . + + [94] RFC Errata, Erratum ID 1565, RFC 793, + . + + [95] RFC Errata, Erratum ID 1569, RFC 793, + . + + [96] RFC Errata, Erratum ID 2296, RFC 793, + . + + [97] RFC Errata, Erratum ID 3305, RFC 793, + . + + [98] RFC Errata, Erratum ID 3602, RFC 793, + . + + [99] RFC Errata, Erratum ID 4772, RFC 5961, + . + + [100] Gont, F., "ICMP Attacks against TCP", RFC 5927, + DOI 10.17487/RFC5927, July 2010, + . + +Appendix A. Other Implementation Notes + + This section includes additional notes and references on TCP + implementation decisions that are currently not a part of the RFC + series or included within the TCP standard. These items can be + considered by implementers, but there was not yet a consensus to + include them in the standard. + +A.1. IP Security Compartment and Precedence + + The IPv4 specification [1] includes a precedence value in the (now + obsoleted) Type of Service (TOS) field. It was modified in [20] and + then obsoleted by the definition of Differentiated Services + (Diffserv) [4]. Setting and conveying TOS between the network layer, + TCP implementation, and applications is obsolete and is replaced by + Diffserv in the current TCP specification. + + RFC 793 required checking the IP security compartment and precedence + on incoming TCP segments for consistency within a connection and with + application requests. Each of these aspects of IP have become + outdated, without specific updates to RFC 793. The issues with + precedence were fixed by [25], which is Standards Track, and so this + present TCP specification includes those changes. However, the state + of IP security options that may be used by Multi-Level Secure (MLS) + systems is not as apparent in the IETF currently. + + Resetting connections when incoming packets do not meet expected + security compartment or precedence expectations has been recognized + as a possible attack vector [63], and there has been discussion about + amending the TCP specification to prevent connections from being + aborted due to nonmatching IP security compartment and Diffserv + codepoint values. + +A.1.1. Precedence + + In Diffserv, the former precedence values are treated as Class + Selector codepoints, and methods for compatible treatment are + described in the Diffserv architecture. The RFC TCP specification + defined by RFCs 793 and 1122 included logic intending to have + connections use the highest precedence requested by either endpoint + application, and to keep the precedence consistent throughout a + connection. This logic from the obsolete TOS is not applicable to + Diffserv and should not be included in TCP implementations, though + changes to Diffserv values within a connection are discouraged. For + discussion of this, see RFC 7657 (Sections 5.1, 5.3, and 6) [50]. + + The obsoleted TOS processing rules in TCP assumed bidirectional (or + symmetric) precedence values used on a connection, but the Diffserv + architecture is asymmetric. Problems with the old TCP logic in this + regard were described in [25], and the solution described is to + ignore IP precedence in TCP. Since RFC 2873 is a Standards Track + document (although not marked as updating RFC 793), current + implementations are expected to be robust in these conditions. Note + that the Diffserv field value used in each direction is a part of the + interface between TCP and the network layer, and values in use can be + indicated both ways between TCP and the application. + +A.1.2. MLS Systems + + The IP Security Option (IPSO) and compartment defined in [1] was + refined in RFC 1038, which was later obsoleted by RFC 1108. The + Commercial IP Security Option (CIPSO) is defined in FIPS-188 + (withdrawn by NIST in 2015) and is supported by some vendors and + operating systems. RFC 1108 is now Historic, though RFC 791 itself + has not been updated to remove the IP Security Option. For IPv6, a + similar option (Common Architecture Label IPv6 Security Option + (CALIPSO)) has been defined [36]. RFC 793 includes logic that + includes the IP security/compartment information in treatment of TCP + segments. References to the IP "security/compartment" in this + document may be relevant for Multi-Level Secure (MLS) system + implementers but can be ignored for non-MLS implementations, + consistent with running code on the Internet. See Appendix A.1 for + further discussion. Note that RFC 5570 describes some MLS networking + scenarios where IPSO, CIPSO, or CALIPSO may be used. In these + special cases, TCP implementers should see Section 7.3.1 of RFC 5570 + and follow the guidance in that document. + +A.2. Sequence Number Validation + + There are cases where the TCP sequence number validation rules can + prevent ACK fields from being processed. This can result in + connection issues, as described in [64], which includes descriptions + of potential problems in conditions of simultaneous open, self- + connects, simultaneous close, and simultaneous window probes. The + document also describes potential changes to the TCP specification to + mitigate the issue by expanding the acceptable sequence numbers. + + In Internet usage of TCP, these conditions rarely occur. Common + operating systems include different alternative mitigations, and the + standard has not been updated yet to codify one of them, but + implementers should consider the problems described in [64]. + +A.3. Nagle Modification + + In common operating systems, both the Nagle algorithm and delayed + acknowledgments are implemented and enabled by default. TCP is used + by many applications that have a request-response style of + communication, where the combination of the Nagle algorithm and + delayed acknowledgments can result in poor application performance. + A modification to the Nagle algorithm is described in [68] that + improves the situation for these applications. + + This modification is implemented in some common operating systems and + does not impact TCP interoperability. Additionally, many + applications simply disable Nagle since this is generally supported + by a socket option. The TCP standard has not been updated to include + this Nagle modification, but implementers may find it beneficial to + consider. + +A.4. Low Watermark Settings + + Some operating system kernel TCP implementations include socket + options that allow specifying the number of bytes in the buffer until + the socket layer will pass sent data to TCP (SO_SNDLOWAT) or to the + application on receiving (SO_RCVLOWAT). + + In addition, another socket option (TCP_NOTSENT_LOWAT) can be used to + control the amount of unsent bytes in the write queue. This can help + a sending TCP application to avoid creating large amounts of buffered + data (and corresponding latency). As an example, this may be useful + for applications that are multiplexing data from multiple upper-level + streams onto a connection, especially when streams may be a mix of + interactive/real-time and bulk data transfer. + +Appendix B. TCP Requirement Summary + + This section is adapted from RFC 1122. + + Note that there is no requirement related to PLPMTUD in this list, + but that PLPMTUD is recommended. + + +=================+=========+======+========+=====+========+======+ + | Feature | ReqID | MUST | SHOULD | MAY | SHOULD | MUST | + | | | | | | NOT | NOT | + +=================+=========+======+========+=====+========+======+ + | PUSH flag | + +=================+=========+======+========+=====+========+======+ + | Aggregate or | MAY-16 | | | X | | | + | queue un-pushed | | | | | | | + | data | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Sender collapse | SHLD-27 | | X | | | | + | successive PSH | | | | | | | + | bits | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | SEND call can | MAY-15 | | | X | | | + | specify PUSH | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * If cannot: | MUST-60 | | | | | X | + | sender | | | | | | | + | buffer | | | | | | | + | indefinitely | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * If cannot: | MUST-61 | X | | | | | + | PSH last | | | | | | | + | segment | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Notify | MAY-17 | | | X | | | + | receiving ALP^1 | | | | | | | + | of PSH | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Send max size | SHLD-28 | | X | | | | + | segment when | | | | | | | + | possible | | | | | | | + +=================+=========+======+========+=====+========+======+ + | Window | + +=================+=========+======+========+=====+========+======+ + | Treat as | MUST-1 | X | | | | | + | unsigned number | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Handle as | REC-1 | | X | | | | + | 32-bit number | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Shrink window | SHLD-14 | | | | X | | + | from right | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Send new | SHLD-15 | | | | X | | + | data when | | | | | | | + | window | | | | | | | + | shrinks | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Retransmit | SHLD-16 | | X | | | | + | old unacked | | | | | | | + | data within | | | | | | | + | window | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Time out | SHLD-17 | | | | X | | + | conn for | | | | | | | + | data past | | | | | | | + | right edge | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Robust against | MUST-34 | X | | | | | + | shrinking | | | | | | | + | window | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Receiver's | MAY-8 | | | X | | | + | window closed | | | | | | | + | indefinitely | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Use standard | MUST-35 | X | | | | | + | probing logic | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Sender probe | MUST-36 | X | | | | | + | zero window | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * First probe | SHLD-29 | | X | | | | + | after RTO | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Exponential | SHLD-30 | | X | | | | + | backoff | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Allow window | MUST-37 | X | | | | | + | stay zero | | | | | | | + | indefinitely | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Retransmit old | MAY-7 | | | X | | | + | data beyond | | | | | | | + | SND.UNA+SND.WND | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Process RST and | MUST-66 | X | | | | | + | URG even with | | | | | | | + | zero window | | | | | | | + +=================+=========+======+========+=====+========+======+ + | Urgent Data | + +=================+=========+======+========+=====+========+======+ + | Include support | MUST-30 | X | | | | | + | for urgent | | | | | | | + | pointer | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Pointer | MUST-62 | X | | | | | + | indicates first | | | | | | | + | non-urgent | | | | | | | + | octet | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Arbitrary | MUST-31 | X | | | | | + | length urgent | | | | | | | + | data sequence | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Inform ALP^1 | MUST-32 | X | | | | | + | asynchronously | | | | | | | + | of urgent data | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | ALP^1 can learn | MUST-33 | X | | | | | + | if/how much | | | | | | | + | urgent data Q'd | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | ALP employ the | SHLD-13 | | | | X | | + | urgent | | | | | | | + | mechanism | | | | | | | + +=================+=========+======+========+=====+========+======+ + | TCP Options | + +=================+=========+======+========+=====+========+======+ + | Support the | MUST-4 | X | | | | | + | mandatory | | | | | | | + | option set | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Receive TCP | MUST-5 | X | | | | | + | Option in any | | | | | | | + | segment | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Ignore | MUST-6 | X | | | | | + | unsupported | | | | | | | + | options | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Include length | MUST-68 | X | | | | | + | for all options | | | | | | | + | except EOL+NOP | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Cope with | MUST-7 | X | | | | | + | illegal option | | | | | | | + | length | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Process options | MUST-64 | X | | | | | + | regardless of | | | | | | | + | word alignment | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Implement | MUST-14 | X | | | | | + | sending & | | | | | | | + | receiving MSS | | | | | | | + | Option | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | IPv4 Send MSS | SHLD-5 | | X | | | | + | Option unless | | | | | | | + | 536 | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | IPv6 Send MSS | SHLD-5 | | X | | | | + | Option unless | | | | | | | + | 1220 | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Send MSS Option | MAY-3 | | | X | | | + | always | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | IPv4 Send-MSS | MUST-15 | X | | | | | + | default is 536 | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | IPv6 Send-MSS | MUST-15 | X | | | | | + | default is 1220 | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Calculate | MUST-16 | X | | | | | + | effective send | | | | | | | + | seg size | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | MSS accounts | SHLD-6 | | X | | | | + | for varying MTU | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | MSS not sent on | MUST-65 | | | | | X | + | non-SYN | | | | | | | + | segments | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | MSS value based | MUST-67 | X | | | | | + | on MMS_R | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Pad with zero | MUST-69 | X | | | | | + +=================+=========+======+========+=====+========+======+ + | TCP Checksums | + +=================+=========+======+========+=====+========+======+ + | Sender compute | MUST-2 | X | | | | | + | checksum | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Receiver check | MUST-3 | X | | | | | + | checksum | | | | | | | + +=================+=========+======+========+=====+========+======+ + | ISN Selection | + +=================+=========+======+========+=====+========+======+ + | Include a | MUST-8 | X | | | | | + | clock-driven | | | | | | | + | ISN generator | | | | | | | + | component | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Secure ISN | SHLD-1 | | X | | | | + | generator with | | | | | | | + | a PRF component | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | PRF computable | MUST-9 | | | | | X | + | from outside | | | | | | | + | the host | | | | | | | + +=================+=========+======+========+=====+========+======+ + | Opening Connections | + +=================+=========+======+========+=====+========+======+ + | Support | MUST-10 | X | | | | | + | simultaneous | | | | | | | + | open attempts | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | SYN-RECEIVED | MUST-11 | X | | | | | + | remembers last | | | | | | | + | state | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Passive OPEN | MUST-41 | | | | | X | + | call interfere | | | | | | | + | with others | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Function: | MUST-42 | X | | | | | + | simultaneously | | | | | | | + | LISTENs for | | | | | | | + | same port | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Ask IP for src | MUST-44 | X | | | | | + | address for SYN | | | | | | | + | if necessary | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Otherwise, | MUST-45 | X | | | | | + | use local | | | | | | | + | addr of | | | | | | | + | connection | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | OPEN to | MUST-46 | | | | | X | + | broadcast/ | | | | | | | + | multicast IP | | | | | | | + | address | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Silently | MUST-57 | X | | | | | + | discard seg to | | | | | | | + | bcast/mcast | | | | | | | + | addr | | | | | | | + +=================+=========+======+========+=====+========+======+ + | Closing Connections | + +=================+=========+======+========+=====+========+======+ + | RST can contain | SHLD-2 | | X | | | | + | data | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Inform | MUST-12 | X | | | | | + | application of | | | | | | | + | aborted conn | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Half-duplex | MAY-1 | | | X | | | + | close | | | | | | | + | connections | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Send RST to | SHLD-3 | | X | | | | + | indicate | | | | | | | + | data lost | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | In TIME-WAIT | MUST-13 | X | | | | | + | state for 2MSL | | | | | | | + | seconds | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Accept SYN | MAY-2 | | | X | | | + | from TIME- | | | | | | | + | WAIT state | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Use | SHLD-4 | | X | | | | + | Timestamps | | | | | | | + | to reduce | | | | | | | + | TIME-WAIT | | | | | | | + +=================+=========+======+========+=====+========+======+ + | Retransmissions | + +=================+=========+======+========+=====+========+======+ + | Implement | MUST-19 | X | | | | | + | exponential | | | | | | | + | backoff, slow | | | | | | | + | start, and | | | | | | | + | congestion | | | | | | | + | avoidance | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Retransmit with | MAY-4 | | | X | | | + | same IP | | | | | | | + | identity | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Karn's | MUST-18 | X | | | | | + | algorithm | | | | | | | + +=================+=========+======+========+=====+========+======+ + | Generating ACKs | + +=================+=========+======+========+=====+========+======+ + | Aggregate | MUST-58 | X | | | | | + | whenever | | | | | | | + | possible | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Queue out-of- | SHLD-31 | | X | | | | + | order segments | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Process all Q'd | MUST-59 | X | | | | | + | before send ACK | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Send ACK for | MAY-13 | | | X | | | + | out-of-order | | | | | | | + | segment | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Delayed ACKs | SHLD-18 | | X | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Delay < 0.5 | MUST-40 | X | | | | | + | seconds | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Every 2nd | SHLD-19 | | X | | | | + | full-sized | | | | | | | + | segment or | | | | | | | + | 2*RMSS ACK'd | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Receiver SWS- | MUST-39 | X | | | | | + | Avoidance | | | | | | | + | Algorithm | | | | | | | + +=================+=========+======+========+=====+========+======+ + | Sending Data | + +=================+=========+======+========+=====+========+======+ + | Configurable | MUST-49 | X | | | | | + | TTL | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Sender SWS- | MUST-38 | X | | | | | + | Avoidance | | | | | | | + | Algorithm | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Nagle algorithm | SHLD-7 | | X | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Application | MUST-17 | X | | | | | + | can disable | | | | | | | + | Nagle | | | | | | | + | algorithm | | | | | | | + +=================+=========+======+========+=====+========+======+ + | Connection Failures | + +=================+=========+======+========+=====+========+======+ + | Negative advice | MUST-20 | X | | | | | + | to IP on R1 | | | | | | | + | retransmissions | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Close | MUST-20 | X | | | | | + | connection on | | | | | | | + | R2 | | | | | | | + | retransmissions | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | ALP^1 can set | MUST-21 | X | | | | | + | R2 | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Inform ALP of | SHLD-9 | | X | | | | + | R1<=retxs | | | | | | | + | inform ALP | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Abort on | MUST-56 | | | | | X | + | Dest Unreach | | | | | | | + | (0,1,5) | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Dest Unreach | SHLD-26 | | X | | | | + | (2-4) => | | | | | | | + | abort conn | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Source | MUST-55 | X | | | | | + | Quench => | | | | | | | + | silent | | | | | | | + | discard | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Abort on | MUST-56 | | | | | X | + | Time | | | | | | | + | Exceeded | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Abort on | MUST-56 | | | | | X | + | Param | | | | | | | + | Problem | | | | | | | + +=================+=========+======+========+=====+========+======+ + | Address Validation | + +=================+=========+======+========+=====+========+======+ + | Reject OPEN | MUST-46 | X | | | | | + | call to invalid | | | | | | | + | IP address | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Reject SYN from | MUST-63 | X | | | | | + | invalid IP | | | | | | | + | address | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Silently | MUST-57 | X | | | | | + | discard SYN to | | | | | | | + | bcast/mcast | | | | | | | + | addr | | | | | | | + +=================+=========+======+========+=====+========+======+ + | TCP/ALP Interface Services | + +=================+=========+======+========+=====+========+======+ + | Error Report | MUST-47 | X | | | | | + | mechanism | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | ALP can disable | SHLD-20 | | X | | | | + | Error Report | | | | | | | + | Routine | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | ALP can specify | MUST-48 | X | | | | | + | Diffserv field | | | | | | | + | for sending | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | * Passed | SHLD-22 | | X | | | | + | unchanged to | | | | | | | + | IP | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | ALP can change | SHLD-21 | | X | | | | + | Diffserv field | | | | | | | + | during | | | | | | | + | connection | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | ALP generally | SHLD-23 | | | | X | | + | changing | | | | | | | + | Diffserv during | | | | | | | + | conn. | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Pass received | MAY-9 | | | X | | | + | Diffserv field | | | | | | | + | up to ALP | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + | FLUSH call | MAY-14 | | | X | | | + +-----------------+---------+------+--------+-----+--------+------+ + | Optional local | MUST-43 | X | | | | | + | IP addr param | | | | | | | + | in OPEN | | | | | | | + +=================+=========+======+========+=====+========+======+ + | RFC 5961 Support | + +=================+=========+======+========+=====+========+======+ + | Implement data | MAY-12 | | | X | | | + | injection | | | | | | | + | protection | | | | | | | + +=================+=========+======+========+=====+========+======+ + | Explicit Congestion Notification | + +=================+=========+======+========+=====+========+======+ + | Support ECN | SHLD-8 | | X | | | | + +=================+=========+======+========+=====+========+======+ + | Alternative Congestion Control | + +=================+=========+======+========+=====+========+======+ + | Implement | MAY-18 | | | X | | | + | alternative | | | | | | | + | conformant | | | | | | | + | algorithm(s) | | | | | | | + +-----------------+---------+------+--------+-----+--------+------+ + + Table 8: TCP Requirements Summary + + FOOTNOTES: (1) "ALP" means Application-Layer Program. + +Acknowledgments + + This document is largely a revision of RFC 793, of which Jon Postel + was the editor. Due to his excellent work, it was able to last for + three decades before we felt the need to revise it. + + Andre Oppermann was a contributor and helped to edit the first + revision of this document. + + We are thankful for the assistance of the IETF TCPM working group + chairs over the course of work on this document: + + Michael Scharf + + + Yoshifumi Nishida + + + Pasi Sarolahti + + + Michael Tüxen + + + During the discussions of this work on the TCPM mailing list, in + working group meetings, and via area reviews, helpful comments, + critiques, and reviews were received from (listed alphabetically by + last name): Praveen Balasubramanian, David Borman, Mohamed Boucadair, + Bob Briscoe, Neal Cardwell, Yuchung Cheng, Martin Duke, Francis + Dupont, Ted Faber, Gorry Fairhurst, Fernando Gont, Rodney Grimes, Yi + Huang, Rahul Jadhav, Markku Kojo, Mike Kosek, Juhamatti Kuusisaari, + Kevin Lahey, Kevin Mason, Matt Mathis, Stephen McQuistin, Jonathan + Morton, Matt Olson, Tommy Pauly, Tom Petch, Hagen Paul Pfeifer, Kyle + Rose, Anthony Sabatini, Michael Scharf, Greg Skinner, Joe Touch, + Michael Tüxen, Reji Varghese, Bernie Volz, Tim Wicinski, Lloyd Wood, + and Alex Zimmermann. + + Joe Touch provided additional help in clarifying the description of + segment size parameters and PMTUD/PLPMTUD recommendations. Markku + Kojo helped put together the text in the section on TCP Congestion + Control. + + This document includes content from errata that were reported by + (listed chronologically): Yin Shuming, Bob Braden, Morris M. Keesan, + Pei-chun Cheng, Constantin Hagemeier, Vishwas Manral, Mykyta + Yevstifeyev, EungJun Yi, Botong Huang, Charles Deng, Merlin Buge. + +Author's Address + + Wesley M. Eddy (editor) + MTI Systems + United States of America + Email: wes@mti-systems.com diff --git a/decisiongrounding/scenarios_candidates/rfc_tcp_supersession/provenance.json b/decisiongrounding/scenarios_candidates/rfc_tcp_supersession/provenance.json new file mode 100644 index 0000000..48e76b2 --- /dev/null +++ b/decisiongrounding/scenarios_candidates/rfc_tcp_supersession/provenance.json @@ -0,0 +1,44 @@ +{ + "source": "rfc-editor.org", + "immutable": true, + "rfcs": [ + { + "id": "RFC-0793", + "number": 793, + "file": "corpus/RFC-0793.md", + "url": "https://www.rfc-editor.org/rfc/rfc793.txt", + "source_sha256": "e55b1faa35edbeecceb2400233a16ae81897bdf9c31d5b3031e91139e5ea8142", + "category": "", + "title": "TRANSMISSION CONTROL PROTOCOL", + "obsoletes": [], + "replaces": [] + }, + { + "id": "RFC-9293", + "number": 9293, + "file": "corpus/RFC-9293.md", + "url": "https://www.rfc-editor.org/rfc/rfc9293.txt", + "source_sha256": "6d9ac8be4b0286f8c3d337addf442b2eb6a9b14e1366594ea7fbc273f93dc2d9", + "category": "Standards Track", + "title": "6691", + "obsoletes": [ + "RFC-0793", + "RFC-0879", + "RFC-2873", + "RFC-6093", + "RFC-6429", + "RFC-6528" + ], + "replaces": [ + "RFC-0793" + ] + } + ], + "supersedes_edges": [ + { + "source": "RFC-9293", + "type": "supersedes", + "target": "RFC-0793" + } + ] +} diff --git a/decisiongrounding/scenarios_candidates/rfc_utf8_supersession/corpus/RFC-2279.md b/decisiongrounding/scenarios_candidates/rfc_utf8_supersession/corpus/RFC-2279.md new file mode 100644 index 0000000..c75957d --- /dev/null +++ b/decisiongrounding/scenarios_candidates/rfc_utf8_supersession/corpus/RFC-2279.md @@ -0,0 +1,599 @@ +--- +schema_version: 1 +id: RFC-2279 +type: decision +tags: [rfc] +--- + +# RFC-2279 — UTF-8, a transformation format of ISO 10646 + +## Status + +Superseded + +## Context + +Public decision ingested verbatim from the RFC Editor (RFC 2279); +source https://www.rfc-editor.org/rfc/rfc2279.txt. RFCs are immutable, so the RFC number is the +pin and the upstream sha256 is recorded in provenance.json. Upstream +category: Standards Track. The RAC envelope sections are derived from +the RFC's own header fields; the authoritative content is the unedited RFC +reproduced under Source Text. + +## Decision + +Defined by the verbatim RFC text under Source Text below. + +## Consequences + +As stated by the upstream RFC; see Source Text. + +## Superseded By + +- RFC-3629 + +## Source Text + + + + + + + +Network Working Group F. Yergeau +Request for Comments: 2279 Alis Technologies +Obsoletes: 2044 January 1998 +Category: Standards Track + + + UTF-8, a transformation format of ISO 10646 + +Status of this Memo + + This document specifies an Internet standards track protocol for the + Internet community, and requests discussion and suggestions for + improvements. Please refer to the current edition of the "Internet + Official Protocol Standards" (STD 1) for the standardization state + and status of this protocol. Distribution of this memo is unlimited. + +Copyright Notice + + Copyright (C) The Internet Society (1998). All Rights Reserved. + +Abstract + + ISO/IEC 10646-1 defines a multi-octet character set called the + Universal Character Set (UCS) which encompasses most of the world's + writing systems. Multi-octet characters, however, are not compatible + with many current applications and protocols, and this has led to the + development of a few so-called UCS transformation formats (UTF), each + with different characteristics. UTF-8, the object of this memo, has + the characteristic of preserving the full US-ASCII range, providing + compatibility with file systems, parsers and other software that rely + on US-ASCII values but are transparent to other values. This memo + updates and replaces RFC 2044, in particular addressing the question + of versions of the relevant standards. + +1. Introduction + + ISO/IEC 10646-1 [ISO-10646] defines a multi-octet character set + called the Universal Character Set (UCS), which encompasses most of + the world's writing systems. Two multi-octet encodings are defined, + a four-octet per character encoding called UCS-4 and a two-octet per + character encoding called UCS-2, able to address only the first 64K + characters of the UCS (the Basic Multilingual Plane, BMP), outside of + which there are currently no assignments. + + It is noteworthy that the same set of characters is defined by the + Unicode standard [UNICODE], which further defines additional + character properties and other application details of great interest + to implementors, but does not have the UCS-4 encoding. Up to the + + + +Yergeau Standards Track [Page 1] + +RFC 2279 UTF-8 January 1998 + + + present time, changes in Unicode and amendments to ISO/IEC 10646 have + tracked each other, so that the character repertoires and code point + assignments have remained in sync. The relevant standardization + committees have committed to maintain this very useful synchronism. + + The UCS-2 and UCS-4 encodings, however, are hard to use in many + current applications and protocols that assume 8 or even 7 bit + characters. Even newer systems able to deal with 16 bit characters + cannot process UCS-4 data. This situation has led to the development + of so-called UCS transformation formats (UTF), each with different + characteristics. + + UTF-1 has only historical interest, having been removed from ISO/IEC + 10646. UTF-7 has the quality of encoding the full BMP repertoire + using only octets with the high-order bit clear (7 bit US-ASCII + values, [US-ASCII]), and is thus deemed a mail-safe encoding + ([RFC2152]). UTF-8, the object of this memo, uses all bits of an + octet, but has the quality of preserving the full US-ASCII range: + US-ASCII characters are encoded in one octet having the normal US- + ASCII value, and any octet with such a value can only stand for an + US-ASCII character, and nothing else. + + UTF-16 is a scheme for transforming a subset of the UCS-4 repertoire + into pairs of UCS-2 values from a reserved range. UTF-16 impacts + UTF-8 in that UCS-2 values from the reserved range must be treated + specially in the UTF-8 transformation. + + UTF-8 encodes UCS-2 or UCS-4 characters as a varying number of + octets, where the number of octets, and the value of each, depend on + the integer value assigned to the character in ISO/IEC 10646. This + transformation format has the following characteristics (all values + are in hexadecimal): + + - Character values from 0000 0000 to 0000 007F (US-ASCII repertoire) + correspond to octets 00 to 7F (7 bit US-ASCII values). A direct + consequence is that a plain ASCII string is also a valid UTF-8 + string. + + - US-ASCII values do not appear otherwise in a UTF-8 encoded + character stream. This provides compatibility with file systems + or other software (e.g. the printf() function in C libraries) that + parse based on US-ASCII values but are transparent to other + values. + + - Round-trip conversion is easy between UTF-8 and either of UCS-4, + UCS-2. + + + + + +Yergeau Standards Track [Page 2] + +RFC 2279 UTF-8 January 1998 + + + - The first octet of a multi-octet sequence indicates the number of + octets in the sequence. + + - The octet values FE and FF never appear. + + - Character boundaries are easily found from anywhere in an octet + stream. + + - The lexicographic sorting order of UCS-4 strings is preserved. Of + course this is of limited interest since the sort order is not + culturally valid in either case. + + - The Boyer-Moore fast search algorithm can be used with UTF-8 data. + + - UTF-8 strings can be fairly reliably recognized as such by a + simple algorithm, i.e. the probability that a string of characters + in any other encoding appears as valid UTF-8 is low, diminishing + with increasing string length. + + UTF-8 was originally a project of the X/Open Joint + Internationalization Group XOJIG with the objective to specify a File + System Safe UCS Transformation Format [FSS-UTF] that is compatible + with UNIX systems, supporting multilingual text in a single encoding. + The original authors were Gary Miller, Greger Leijonhufvud and John + Entenmann. Later, Ken Thompson and Rob Pike did significant work for + the formal UTF-8. + + A description can also be found in Unicode Technical Report #4 and in + the Unicode Standard, version 2.0 [UNICODE]. The definitive + reference, including provisions for UTF-16 data within UTF-8, is + Annex R of ISO/IEC 10646-1 [ISO-10646]. + +2. UTF-8 definition + + In UTF-8, characters are encoded using sequences of 1 to 6 octets. + The only octet of a "sequence" of one has the higher-order bit set to + 0, the remaining 7 bits being used to encode the character value. In + a sequence of n octets, n>1, the initial octet has the n higher-order + bits set to 1, followed by a bit set to 0. The remaining bit(s) of + that octet contain bits from the value of the character to be + encoded. The following octet(s) all have the higher-order bit set to + 1 and the following bit set to 0, leaving 6 bits in each to contain + bits from the character to be encoded. + + The table below summarizes the format of these different octet types. + The letter x indicates bits available for encoding bits of the UCS-4 + character value. + + + + +Yergeau Standards Track [Page 3] + +RFC 2279 UTF-8 January 1998 + + + UCS-4 range (hex.) UTF-8 octet sequence (binary) + 0000 0000-0000 007F 0xxxxxxx + 0000 0080-0000 07FF 110xxxxx 10xxxxxx + 0000 0800-0000 FFFF 1110xxxx 10xxxxxx 10xxxxxx + + 0001 0000-001F FFFF 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx + 0020 0000-03FF FFFF 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx + 0400 0000-7FFF FFFF 1111110x 10xxxxxx ... 10xxxxxx + + Encoding from UCS-4 to UTF-8 proceeds as follows: + + 1) Determine the number of octets required from the character value + and the first column of the table above. It is important to note + that the rows of the table are mutually exclusive, i.e. there is + only one valid way to encode a given UCS-4 character. + + 2) Prepare the high-order bits of the octets as per the second column + of the table. + + 3) Fill in the bits marked x from the bits of the character value, + starting from the lower-order bits of the character value and + putting them first in the last octet of the sequence, then the + next to last, etc. until all x bits are filled in. + + The algorithm for encoding UCS-2 (or Unicode) to UTF-8 can be + obtained from the above, in principle, by simply extending each + UCS-2 character with two zero-valued octets. However, pairs of + UCS-2 values between D800 and DFFF (surrogate pairs in Unicode + parlance), being actually UCS-4 characters transformed through + UTF-16, need special treatment: the UTF-16 transformation must be + undone, yielding a UCS-4 character that is then transformed as + above. + + Decoding from UTF-8 to UCS-4 proceeds as follows: + + 1) Initialize the 4 octets of the UCS-4 character with all bits set + to 0. + + 2) Determine which bits encode the character value from the number of + octets in the sequence and the second column of the table above + (the bits marked x). + + 3) Distribute the bits from the sequence to the UCS-4 character, + first the lower-order bits from the last octet of the sequence and + proceeding to the left until no x bits are left. + + If the UTF-8 sequence is no more than three octets long, decoding + can proceed directly to UCS-2. + + + +Yergeau Standards Track [Page 4] + +RFC 2279 UTF-8 January 1998 + + + NOTE -- actual implementations of the decoding algorithm above + should protect against decoding invalid sequences. For + instance, a naive implementation may (wrongly) decode the + invalid UTF-8 sequence C0 80 into the character U+0000, which + may have security consequences and/or cause other problems. See + the Security Considerations section below. + + A more detailed algorithm and formulae can be found in [FSS_UTF], + [UNICODE] or Annex R to [ISO-10646]. + +3. Versions of the standards + + ISO/IEC 10646 is updated from time to time by published amendments; + similarly, different versions of the Unicode standard exist: 1.0, 1.1 + and 2.0 as of this writing. Each new version obsoletes and replaces + the previous one, but implementations, and more significantly data, + are not updated instantly. + + In general, the changes amount to adding new characters, which does + not pose particular problems with old data. Amendment 5 to ISO/IEC + 10646, however, has moved and expanded the Korean Hangul block, + thereby making any previous data containing Hangul characters invalid + under the new version. Unicode 2.0 has the same difference from + Unicode 1.1. The official justification for allowing such an + incompatible change was that no implementations and no data + containing Hangul existed, a statement that is likely to be true but + remains unprovable. The incident has been dubbed the "Korean mess", + and the relevant committees have pledged to never, ever again make + such an incompatible change. + + New versions, and in particular any incompatible changes, have q + conseuences regarding MIME character encoding labels, to be discussed + in section 5. + +4. Examples + + The UCS-2 sequence "A." (0041, 2262, 0391, + 002E) may be encoded in UTF-8 as follows: + + 41 E2 89 A2 CE 91 2E + + The UCS-2 sequence representing the Hangul characters for the Korean + word "hangugo" (D55C, AD6D, C5B4) may be encoded as follows: + + ED 95 9C EA B5 AD EC 96 B4 + + + + + + +Yergeau Standards Track [Page 5] + +RFC 2279 UTF-8 January 1998 + + + The UCS-2 sequence representing the Han characters for the Japanese + word "nihongo" (65E5, 672C, 8A9E) may be encoded as follows: + + E6 97 A5 E6 9C AC E8 AA 9E + +5. MIME registration + + This memo is meant to serve as the basis for registration of a MIME + character set parameter (charset) [CHARSET-REG]. The proposed + charset parameter value is "UTF-8". This string labels media types + containing text consisting of characters from the repertoire of + ISO/IEC 10646 including all amendments at least up to amendment 5 + (Korean block), encoded to a sequence of octets using the encoding + scheme outlined above. UTF-8 is suitable for use in MIME content + types under the "text" top-level type. + + It is noteworthy that the label "UTF-8" does not contain a version + identification, referring generically to ISO/IEC 10646. This is + intentional, the rationale being as follows: + + A MIME charset label is designed to give just the information needed + to interpret a sequence of bytes received on the wire into a sequence + of characters, nothing more (see RFC 2045, section 2.2, in [MIME]). + As long as a character set standard does not change incompatibly, + version numbers serve no purpose, because one gains nothing by + learning from the tag that newly assigned characters may be received + that one doesn't know about. The tag itself doesn't teach anything + about the new characters, which are going to be received anyway. + + Hence, as long as the standards evolve compatibly, the apparent + advantage of having labels that identify the versions is only that, + apparent. But there is a disadvantage to such version-dependent + labels: when an older application receives data accompanied by a + newer, unknown label, it may fail to recognize the label and be + completely unable to deal with the data, whereas a generic, known + label would have triggered mostly correct processing of the data, + which may well not contain any new characters. + + Now the "Korean mess" (ISO/IEC 10646 amendment 5) is an incompatible + change, in principle contradicting the appropriateness of a version + independent MIME charset label as described above. But the + compatibility problem can only appear with data containing Korean + Hangul characters encoded according to Unicode 1.1 (or equivalently + ISO/IEC 10646 before amendment 5), and there is arguably no such data + to worry about, this being the very reason the incompatible change + was deemed acceptable. + + + + + +Yergeau Standards Track [Page 6] + +RFC 2279 UTF-8 January 1998 + + + In practice, then, a version-independent label is warranted, provided + the label is understood to refer to all versions after Amendment 5, + and provided no incompatible change actually occurs. Should + incompatible changes occur in a later version of ISO/IEC 10646, the + MIME charset label defined here will stay aligned with the previous + version until and unless the IETF specifically decides otherwise. + + It is also proposed to register the charset parameter value + "UNICODE-1-1-UTF-8", for the exclusive purpose of labelling text data + containing Hangul syllables encoded to UTF-8 without taking into + account Amendment 5 of ISO/IEC 10646 (i.e. using the pre-amendment 5 + code point assignments). Any other UTF-8 data SHOULD NOT use this + label, in particular data not containing any Hangul syllables, and it + is felt important to strongly recommend against creating any new + Hangul-containing data without taking Amendment 5 of ISO/IEC 10646 + into account. + +6. Security Considerations + + Implementors of UTF-8 need to consider the security aspects of how + they handle illegal UTF-8 sequences. It is conceivable that in some + circumstances an attacker would be able to exploit an incautious + UTF-8 parser by sending it an octet sequence that is not permitted by + the UTF-8 syntax. + + A particularly subtle form of this attack could be carried out + against a parser which performs security-critical validity checks + against the UTF-8 encoded form of its input, but interprets certain + illegal octet sequences as characters. For example, a parser might + prohibit the NUL character when encoded as the single-octet sequence + 00, but allow the illegal two-octet sequence C0 80 and interpret it + as a NUL character. Another example might be a parser which + prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits the + illegal octet sequence 2F C0 AE 2E 2F. + +Acknowledgments + + The following have participated in the drafting and discussion of + this memo: + + James E. Agenbroad Andries Brouwer + Martin J. D|rst Ned Freed + David Goldsmith Edwin F. Hart + Kent Karlsson Markus Kuhn + Michael Kung Alain LaBonte + John Gardiner Myers Murray Sargent + Keld Simonsen Arnold Winkler + + + + +Yergeau Standards Track [Page 7] + +RFC 2279 UTF-8 January 1998 + + +Bibliography + + [CHARSET-REG] Freed, N., and J. Postel, "IANA Charset Registration + Procedures", BCP 19, RFC 2278, January 1998. + + [FSS_UTF] X/Open CAE Specification C501 ISBN 1-85912-082-2 28cm. + 22p. pbk. 172g. 4/95, X/Open Company Ltd., "File + System Safe UCS Transformation Format (FSS_UTF)", + X/Open Preleminary Specification, Document Number + P316. Also published in Unicode Technical Report #4. + + [ISO-10646] ISO/IEC 10646-1:1993. International Standard -- + Information technology -- Universal Multiple-Octet + Coded Character Set (UCS) -- Part 1: Architecture and + Basic Multilingual Plane. Five amendments and a + technical corrigendum have been published up to now. + UTF-8 is described in Annex R, published as Amendment + 2. UTF-16 is described in Annex Q, published as + Amendment 1. 17 other amendments are currently at + various stages of standardization. + + [MIME] Freed, N., and N. Borenstein, "Multipurpose Internet + Mail Extensions (MIME) Part One: Format of Internet + Message Bodies", RFC 2045. N. Freed, N. Borenstein, + "Multipurpose Internet Mail Extensions (MIME) Part + Two: Media Types", RFC 2046. K. Moore, "MIME + (Multipurpose Internet Mail Extensions) Part Three: + Message Header Extensions for Non-ASCII Text", RFC + 2047. N. Freed, J. Klensin, J. Postel, "Multipurpose + Internet Mail Extensions (MIME) Part Four: + Registration Procedures", RFC 2048. N. Freed, N. + Borenstein, " Multipurpose Internet Mail Extensions + (MIME) Part Five: Conformance Criteria and Examples", + RFC 2049. All November 1996. + + [RFC2152] Goldsmith, D., and M. Davis, "UTF-7: A Mail-safe + Transformation Format of Unicode", RFC 1642, Taligent + inc., May 1997. (Obsoletes RFC1642) + + [UNICODE] The Unicode Consortium, "The Unicode Standard -- + Version 2.0", Addison-Wesley, 1996. + + [US-ASCII] Coded Character Set--7-bit American Standard Code for + Information Interchange, ANSI X3.4-1986. + + + + + + + +Yergeau Standards Track [Page 8] + +RFC 2279 UTF-8 January 1998 + + +Author's Address + + Francois Yergeau + Alis Technologies + 100, boul. Alexis-Nihon + Suite 600 + Montreal QC H4M 2P2 + Canada + + Phone: +1 (514) 747-2547 + Fax: +1 (514) 747-2561 + EMail: fyergeau@alis.com + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Yergeau Standards Track [Page 9] + +RFC 2279 UTF-8 January 1998 + + +Full Copyright Statement + + Copyright (C) The Internet Society (1998). All Rights Reserved. + + This document and translations of it may be copied and furnished to + others, and derivative works that comment on or otherwise explain it + or assist in its implementation may be prepared, copied, published + and distributed, in whole or in part, without restriction of any + kind, provided that the above copyright notice and this paragraph are + included on all such copies and derivative works. However, this + document itself may not be modified in any way, such as by removing + the copyright notice or references to the Internet Society or other + Internet organizations, except as needed for the purpose of + developing Internet standards in which case the procedures for + copyrights defined in the Internet Standards process must be + followed, or as required to translate it into languages other than + English. + + The limited permissions granted above are perpetual and will not be + revoked by the Internet Society or its successors or assigns. + + This document and the information contained herein is provided on an + "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING + TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING + BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION + HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF + MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + + + + + + + + + + + + + + + + + + + + + + + + +Yergeau Standards Track [Page 10] + diff --git a/decisiongrounding/scenarios_candidates/rfc_utf8_supersession/corpus/RFC-3629.md b/decisiongrounding/scenarios_candidates/rfc_utf8_supersession/corpus/RFC-3629.md new file mode 100644 index 0000000..0a8e58d --- /dev/null +++ b/decisiongrounding/scenarios_candidates/rfc_utf8_supersession/corpus/RFC-3629.md @@ -0,0 +1,823 @@ +--- +schema_version: 1 +id: RFC-3629 +type: decision +tags: [rfc] +--- + +# RFC-3629 — UTF-8, a transformation format of ISO 10646 + +## Status + +Final + +## Context + +Public decision ingested verbatim from the RFC Editor (RFC 3629); +source https://www.rfc-editor.org/rfc/rfc3629.txt. RFCs are immutable, so the RFC number is the +pin and the upstream sha256 is recorded in provenance.json. Upstream +category: Standards Track. The RAC envelope sections are derived from +the RFC's own header fields; the authoritative content is the unedited RFC +reproduced under Source Text. + +## Decision + +Defined by the verbatim RFC text under Source Text below. + +## Consequences + +As stated by the upstream RFC; see Source Text. + +## Supersedes + +- RFC-2279 + +## Source Text + + + + + + + +Network Working Group F. Yergeau +Request for Comments: 3629 Alis Technologies +STD: 63 November 2003 +Obsoletes: 2279 +Category: Standards Track + + + UTF-8, a transformation format of ISO 10646 + +Status of this Memo + + This document specifies an Internet standards track protocol for the + Internet community, and requests discussion and suggestions for + improvements. Please refer to the current edition of the "Internet + Official Protocol Standards" (STD 1) for the standardization state + and status of this protocol. Distribution of this memo is unlimited. + +Copyright Notice + + Copyright (C) The Internet Society (2003). All Rights Reserved. + +Abstract + + ISO/IEC 10646-1 defines a large character set called the Universal + Character Set (UCS) which encompasses most of the world's writing + systems. The originally proposed encodings of the UCS, however, were + not compatible with many current applications and protocols, and this + has led to the development of UTF-8, the object of this memo. UTF-8 + has the characteristic of preserving the full US-ASCII range, + providing compatibility with file systems, parsers and other software + that rely on US-ASCII values but are transparent to other values. + This memo obsoletes and replaces RFC 2279. + +Table of Contents + + 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 2 + 2. Notational conventions . . . . . . . . . . . . . . . . . . . . 3 + 3. UTF-8 definition . . . . . . . . . . . . . . . . . . . . . . . 4 + 4. Syntax of UTF-8 Byte Sequences . . . . . . . . . . . . . . . . 5 + 5. Versions of the standards . . . . . . . . . . . . . . . . . . 6 + 6. Byte order mark (BOM) . . . . . . . . . . . . . . . . . . . . 6 + 7. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 + 8. MIME registration . . . . . . . . . . . . . . . . . . . . . . 9 + 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 + 10. Security Considerations . . . . . . . . . . . . . . . . . . . 10 + 11. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 11 + 12. Changes from RFC 2279 . . . . . . . . . . . . . . . . . . . . 11 + 13. Normative References . . . . . . . . . . . . . . . . . . . . . 12 + + + +Yergeau Standards Track [Page 1] + +RFC 3629 UTF-8 November 2003 + + + 14. Informative References . . . . . . . . . . . . . . . . . . . . 12 + 15. URI's . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 + 16. Intellectual Property Statement . . . . . . . . . . . . . . . 13 + 17. Author's Address . . . . . . . . . . . . . . . . . . . . . . . 13 + 18. Full Copyright Statement . . . . . . . . . . . . . . . . . . . 14 + +1. Introduction + + ISO/IEC 10646 [ISO.10646] defines a large character set called the + Universal Character Set (UCS), which encompasses most of the world's + writing systems. The same set of characters is defined by the + Unicode standard [UNICODE], which further defines additional + character properties and other application details of great interest + to implementers. Up to the present time, changes in Unicode and + amendments and additions to ISO/IEC 10646 have tracked each other, so + that the character repertoires and code point assignments have + remained in sync. The relevant standardization committees have + committed to maintain this very useful synchronism. + + ISO/IEC 10646 and Unicode define several encoding forms of their + common repertoire: UTF-8, UCS-2, UTF-16, UCS-4 and UTF-32. In an + encoding form, each character is represented as one or more encoding + units. All standard UCS encoding forms except UTF-8 have an encoding + unit larger than one octet, making them hard to use in many current + applications and protocols that assume 8 or even 7 bit characters. + + UTF-8, the object of this memo, has a one-octet encoding unit. It + uses all bits of an octet, but has the quality of preserving the full + US-ASCII [US-ASCII] range: US-ASCII characters are encoded in one + octet having the normal US-ASCII value, and any octet with such a + value can only stand for a US-ASCII character, and nothing else. + + UTF-8 encodes UCS characters as a varying number of octets, where the + number of octets, and the value of each, depend on the integer value + assigned to the character in ISO/IEC 10646 (the character number, + a.k.a. code position, code point or Unicode scalar value). This + encoding form has the following characteristics (all values are in + hexadecimal): + + o Character numbers from U+0000 to U+007F (US-ASCII repertoire) + correspond to octets 00 to 7F (7 bit US-ASCII values). A direct + consequence is that a plain ASCII string is also a valid UTF-8 + string. + + + + + + + + +Yergeau Standards Track [Page 2] + +RFC 3629 UTF-8 November 2003 + + + o US-ASCII octet values do not appear otherwise in a UTF-8 encoded + character stream. This provides compatibility with file systems + or other software (e.g., the printf() function in C libraries) + that parse based on US-ASCII values but are transparent to other + values. + + o Round-trip conversion is easy between UTF-8 and other encoding + forms. + + o The first octet of a multi-octet sequence indicates the number of + octets in the sequence. + + o The octet values C0, C1, F5 to FF never appear. + + o Character boundaries are easily found from anywhere in an octet + stream. + + o The byte-value lexicographic sorting order of UTF-8 strings is the + same as if ordered by character numbers. Of course this is of + limited interest since a sort order based on character numbers is + almost never culturally valid. + + o The Boyer-Moore fast search algorithm can be used with UTF-8 data. + + o UTF-8 strings can be fairly reliably recognized as such by a + simple algorithm, i.e., the probability that a string of + characters in any other encoding appears as valid UTF-8 is low, + diminishing with increasing string length. + + UTF-8 was devised in September 1992 by Ken Thompson, guided by design + criteria specified by Rob Pike, with the objective of defining a UCS + transformation format usable in the Plan9 operating system in a non- + disruptive manner. Thompson's design was stewarded through + standardization by the X/Open Joint Internationalization Group XOJIG + (see [FSS_UTF]), bearing the names FSS-UTF (variant FSS/UTF), UTF-2 + and finally UTF-8 along the way. + +2. Notational conventions + + The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", + "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this + document are to be interpreted as described in [RFC2119]. + + UCS characters are designated by the U+HHHH notation, where HHHH is a + string of from 4 to 6 hexadecimal digits representing the character + number in ISO/IEC 10646. + + + + + +Yergeau Standards Track [Page 3] + +RFC 3629 UTF-8 November 2003 + + +3. UTF-8 definition + + UTF-8 is defined by the Unicode Standard [UNICODE]. Descriptions and + formulae can also be found in Annex D of ISO/IEC 10646-1 [ISO.10646] + + In UTF-8, characters from the U+0000..U+10FFFF range (the UTF-16 + accessible range) are encoded using sequences of 1 to 4 octets. The + only octet of a "sequence" of one has the higher-order bit set to 0, + the remaining 7 bits being used to encode the character number. In a + sequence of n octets, n>1, the initial octet has the n higher-order + bits set to 1, followed by a bit set to 0. The remaining bit(s) of + that octet contain bits from the number of the character to be + encoded. The following octet(s) all have the higher-order bit set to + 1 and the following bit set to 0, leaving 6 bits in each to contain + bits from the character to be encoded. + + The table below summarizes the format of these different octet types. + The letter x indicates bits available for encoding bits of the + character number. + + Char. number range | UTF-8 octet sequence + (hexadecimal) | (binary) + --------------------+--------------------------------------------- + 0000 0000-0000 007F | 0xxxxxxx + 0000 0080-0000 07FF | 110xxxxx 10xxxxxx + 0000 0800-0000 FFFF | 1110xxxx 10xxxxxx 10xxxxxx + 0001 0000-0010 FFFF | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx + + Encoding a character to UTF-8 proceeds as follows: + + 1. Determine the number of octets required from the character number + and the first column of the table above. It is important to note + that the rows of the table are mutually exclusive, i.e., there is + only one valid way to encode a given character. + + 2. Prepare the high-order bits of the octets as per the second + column of the table. + + 3. Fill in the bits marked x from the bits of the character number, + expressed in binary. Start by putting the lowest-order bit of + the character number in the lowest-order position of the last + octet of the sequence, then put the next higher-order bit of the + character number in the next higher-order position of that octet, + etc. When the x bits of the last octet are filled in, move on to + the next to last octet, then to the preceding one, etc. until all + x bits are filled in. + + + + + +Yergeau Standards Track [Page 4] + +RFC 3629 UTF-8 November 2003 + + + The definition of UTF-8 prohibits encoding character numbers between + U+D800 and U+DFFF, which are reserved for use with the UTF-16 + encoding form (as surrogate pairs) and do not directly represent + characters. When encoding in UTF-8 from UTF-16 data, it is necessary + to first decode the UTF-16 data to obtain character numbers, which + are then encoded in UTF-8 as described above. This contrasts with + CESU-8 [CESU-8], which is a UTF-8-like encoding that is not meant for + use on the Internet. CESU-8 operates similarly to UTF-8 but encodes + the UTF-16 code values (16-bit quantities) instead of the character + number (code point). This leads to different results for character + numbers above 0xFFFF; the CESU-8 encoding of those characters is NOT + valid UTF-8. + + Decoding a UTF-8 character proceeds as follows: + + 1. Initialize a binary number with all bits set to 0. Up to 21 bits + may be needed. + + 2. Determine which bits encode the character number from the number + of octets in the sequence and the second column of the table + above (the bits marked x). + + 3. Distribute the bits from the sequence to the binary number, first + the lower-order bits from the last octet of the sequence and + proceeding to the left until no x bits are left. The binary + number is now equal to the character number. + + Implementations of the decoding algorithm above MUST protect against + decoding invalid sequences. For instance, a naive implementation may + decode the overlong UTF-8 sequence C0 80 into the character U+0000, + or the surrogate pair ED A1 8C ED BE B4 into U+233B4. Decoding + invalid sequences may have security consequences or cause other + problems. See Security Considerations (Section 10) below. + +4. Syntax of UTF-8 Byte Sequences + + For the convenience of implementors using ABNF, a definition of UTF-8 + in ABNF syntax is given here. + + A UTF-8 string is a sequence of octets representing a sequence of UCS + characters. An octet sequence is valid UTF-8 only if it matches the + following syntax, which is derived from the rules for encoding UTF-8 + and is expressed in the ABNF of [RFC2234]. + + UTF8-octets = *( UTF8-char ) + UTF8-char = UTF8-1 / UTF8-2 / UTF8-3 / UTF8-4 + UTF8-1 = %x00-7F + UTF8-2 = %xC2-DF UTF8-tail + + + +Yergeau Standards Track [Page 5] + +RFC 3629 UTF-8 November 2003 + + + UTF8-3 = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2( UTF8-tail ) / + %xED %x80-9F UTF8-tail / %xEE-EF 2( UTF8-tail ) + UTF8-4 = %xF0 %x90-BF 2( UTF8-tail ) / %xF1-F3 3( UTF8-tail ) / + %xF4 %x80-8F 2( UTF8-tail ) + UTF8-tail = %x80-BF + + NOTE -- The authoritative definition of UTF-8 is in [UNICODE]. This + grammar is believed to describe the same thing Unicode describes, but + does not claim to be authoritative. Implementors are urged to rely + on the authoritative source, rather than on this ABNF. + +5. Versions of the standards + + ISO/IEC 10646 is updated from time to time by publication of + amendments and additional parts; similarly, new versions of the + Unicode standard are published over time. Each new version obsoletes + and replaces the previous one, but implementations, and more + significantly data, are not updated instantly. + + In general, the changes amount to adding new characters, which does + not pose particular problems with old data. In 1996, Amendment 5 to + the 1993 edition of ISO/IEC 10646 and Unicode 2.0 moved and expanded + the Korean Hangul block, thereby making any previous data containing + Hangul characters invalid under the new version. Unicode 2.0 has the + same difference from Unicode 1.1. The justification for allowing + such an incompatible change was that there were no major + implementations and no significant amounts of data containing Hangul. + The incident has been dubbed the "Korean mess", and the relevant + committees have pledged to never, ever again make such an + incompatible change (see Unicode Consortium Policies [1]). + + New versions, and in particular any incompatible changes, have + consequences regarding MIME charset labels, to be discussed in MIME + registration (Section 8). + +6. Byte order mark (BOM) + + The UCS character U+FEFF "ZERO WIDTH NO-BREAK SPACE" is also known + informally as "BYTE ORDER MARK" (abbreviated "BOM"). This character + can be used as a genuine "ZERO WIDTH NO-BREAK SPACE" within text, but + the BOM name hints at a second possible usage of the character: to + prepend a U+FEFF character to a stream of UCS characters as a + "signature". A receiver of such a serialized stream may then use the + initial character as a hint that the stream consists of UCS + characters and also to recognize which UCS encoding is involved and, + with encodings having a multi-octet encoding unit, as a way to + + + + + +Yergeau Standards Track [Page 6] + +RFC 3629 UTF-8 November 2003 + + + recognize the serialization order of the octets. UTF-8 having a + single-octet encoding unit, this last function is useless and the BOM + will always appear as the octet sequence EF BB BF. + + It is important to understand that the character U+FEFF appearing at + any position other than the beginning of a stream MUST be interpreted + with the semantics for the zero-width non-breaking space, and MUST + NOT be interpreted as a signature. When interpreted as a signature, + the Unicode standard suggests than an initial U+FEFF character may be + stripped before processing the text. Such stripping is necessary in + some cases (e.g., when concatenating two strings, because otherwise + the resulting string may contain an unintended "ZERO WIDTH NO-BREAK + SPACE" at the connection point), but might affect an external process + at a different layer (such as a digital signature or a count of the + characters) that is relying on the presence of all characters in the + stream. It is therefore RECOMMENDED to avoid stripping an initial + U+FEFF interpreted as a signature without a good reason, to ignore it + instead of stripping it when appropriate (such as for display) and to + strip it only when really necessary. + + U+FEFF in the first position of a stream MAY be interpreted as a + zero-width non-breaking space, and is not always a signature. In an + attempt at diminishing this uncertainty, Unicode 3.2 adds a new + character, U+2060 "WORD JOINER", with exactly the same semantics and + usage as U+FEFF except for the signature function, and strongly + recommends its exclusive use for expressing word-joining semantics. + Eventually, following this recommendation will make it all but + certain that any initial U+FEFF is a signature, not an intended "ZERO + WIDTH NO-BREAK SPACE". + + In the meantime, the uncertainty unfortunately remains and may affect + Internet protocols. Protocol specifications MAY restrict usage of + U+FEFF as a signature in order to reduce or eliminate the potential + ill effects of this uncertainty. In the interest of striking a + balance between the advantages (reduction of uncertainty) and + drawbacks (loss of the signature function) of such restrictions, it + is useful to distinguish a few cases: + + o A protocol SHOULD forbid use of U+FEFF as a signature for those + textual protocol elements that the protocol mandates to be always + UTF-8, the signature function being totally useless in those + cases. + + o A protocol SHOULD also forbid use of U+FEFF as a signature for + those textual protocol elements for which the protocol provides + character encoding identification mechanisms, when it is expected + that implementations of the protocol will be in a position to + always use the mechanisms properly. This will be the case when + + + +Yergeau Standards Track [Page 7] + +RFC 3629 UTF-8 November 2003 + + + the protocol elements are maintained tightly under the control of + the implementation from the time of their creation to the time of + their (properly labeled) transmission. + + o A protocol SHOULD NOT forbid use of U+FEFF as a signature for + those textual protocol elements for which the protocol does not + provide character encoding identification mechanisms, when a ban + would be unenforceable, or when it is expected that + implementations of the protocol will not be in a position to + always use the mechanisms properly. The latter two cases are + likely to occur with larger protocol elements such as MIME + entities, especially when implementations of the protocol will + obtain such entities from file systems, from protocols that do not + have encoding identification mechanisms for payloads (such as FTP) + or from other protocols that do not guarantee proper + identification of character encoding (such as HTTP). + + When a protocol forbids use of U+FEFF as a signature for a certain + protocol element, then any initial U+FEFF in that protocol element + MUST be interpreted as a "ZERO WIDTH NO-BREAK SPACE". When a + protocol does NOT forbid use of U+FEFF as a signature for a certain + protocol element, then implementations SHOULD be prepared to handle a + signature in that element and react appropriately: using the + signature to identify the character encoding as necessary and + stripping or ignoring the signature as appropriate. + +7. Examples + + The character sequence U+0041 U+2262 U+0391 U+002E "A." is encoded in UTF-8 as follows: + + --+--------+-----+-- + 41 E2 89 A2 CE 91 2E + --+--------+-----+-- + + The character sequence U+D55C U+AD6D U+C5B4 (Korean "hangugeo", + meaning "the Korean language") is encoded in UTF-8 as follows: + + --------+--------+-------- + ED 95 9C EA B5 AD EC 96 B4 + --------+--------+-------- + + The character sequence U+65E5 U+672C U+8A9E (Japanese "nihongo", + meaning "the Japanese language") is encoded in UTF-8 as follows: + + --------+--------+-------- + E6 97 A5 E6 9C AC E8 AA 9E + --------+--------+-------- + + + +Yergeau Standards Track [Page 8] + +RFC 3629 UTF-8 November 2003 + + + The character U+233B4 (a Chinese character meaning 'stump of tree'), + prepended with a UTF-8 BOM, is encoded in UTF-8 as follows: + + --------+----------- + EF BB BF F0 A3 8E B4 + --------+----------- + +8. MIME registration + + This memo serves as the basis for registration of the MIME charset + parameter for UTF-8, according to [RFC2978]. The charset parameter + value is "UTF-8". This string labels media types containing text + consisting of characters from the repertoire of ISO/IEC 10646 + including all amendments at least up to amendment 5 of the 1993 + edition (Korean block), encoded to a sequence of octets using the + encoding scheme outlined above. UTF-8 is suitable for use in MIME + content types under the "text" top-level type. + + It is noteworthy that the label "UTF-8" does not contain a version + identification, referring generically to ISO/IEC 10646. This is + intentional, the rationale being as follows: + + A MIME charset label is designed to give just the information needed + to interpret a sequence of bytes received on the wire into a sequence + of characters, nothing more (see [RFC2045], section 2.2). As long as + a character set standard does not change incompatibly, version + numbers serve no purpose, because one gains nothing by learning from + the tag that newly assigned characters may be received that one + doesn't know about. The tag itself doesn't teach anything about the + new characters, which are going to be received anyway. + + Hence, as long as the standards evolve compatibly, the apparent + advantage of having labels that identify the versions is only that, + apparent. But there is a disadvantage to such version-dependent + labels: when an older application receives data accompanied by a + newer, unknown label, it may fail to recognize the label and be + completely unable to deal with the data, whereas a generic, known + label would have triggered mostly correct processing of the data, + which may well not contain any new characters. + + Now the "Korean mess" (ISO/IEC 10646 amendment 5) is an incompatible + change, in principle contradicting the appropriateness of a version + independent MIME charset label as described above. But the + compatibility problem can only appear with data containing Korean + Hangul characters encoded according to Unicode 1.1 (or equivalently + ISO/IEC 10646 before amendment 5), and there is arguably no such data + to worry about, this being the very reason the incompatible change + was deemed acceptable. + + + +Yergeau Standards Track [Page 9] + +RFC 3629 UTF-8 November 2003 + + + In practice, then, a version-independent label is warranted, provided + the label is understood to refer to all versions after Amendment 5, + and provided no incompatible change actually occurs. Should + incompatible changes occur in a later version of ISO/IEC 10646, the + MIME charset label defined here will stay aligned with the previous + version until and unless the IETF specifically decides otherwise. + +9. IANA Considerations + + The entry for UTF-8 in the IANA charset registry has been updated to + point to this memo. + +10. Security Considerations + + Implementers of UTF-8 need to consider the security aspects of how + they handle illegal UTF-8 sequences. It is conceivable that in some + circumstances an attacker would be able to exploit an incautious + UTF-8 parser by sending it an octet sequence that is not permitted by + the UTF-8 syntax. + + A particularly subtle form of this attack can be carried out against + a parser which performs security-critical validity checks against the + UTF-8 encoded form of its input, but interprets certain illegal octet + sequences as characters. For example, a parser might prohibit the + NUL character when encoded as the single-octet sequence 00, but + erroneously allow the illegal two-octet sequence C0 80 and interpret + it as a NUL character. Another example might be a parser which + prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits the + illegal octet sequence 2F C0 AE 2E 2F. This last exploit has + actually been used in a widespread virus attacking Web servers in + 2001; thus, the security threat is very real. + + Another security issue occurs when encoding to UTF-8: the ISO/IEC + 10646 description of UTF-8 allows encoding character numbers up to + U+7FFFFFFF, yielding sequences of up to 6 bytes. There is therefore + a risk of buffer overflow if the range of character numbers is not + explicitly limited to U+10FFFF or if buffer sizing doesn't take into + account the possibility of 5- and 6-byte sequences. + + Security may also be impacted by a characteristic of several + character encodings, including UTF-8: the "same thing" (as far as a + user can tell) can be represented by several distinct character + sequences. For instance, an e with acute accent can be represented + by the precomposed U+00E9 E ACUTE character or by the canonically + equivalent sequence U+0065 U+0301 (E + COMBINING ACUTE). Even though + UTF-8 provides a single byte sequence for each character sequence, + the existence of multiple character sequences for "the same thing" + may have security consequences whenever string matching, indexing, + + + +Yergeau Standards Track [Page 10] + +RFC 3629 UTF-8 November 2003 + + + searching, sorting, regular expression matching and selection are + involved. An example would be string matching of an identifier + appearing in a credential and in access control list entries. This + issue is amenable to solutions based on Unicode Normalization Forms, + see [UAX15]. + +11. Acknowledgements + + The following have participated in the drafting and discussion of + this memo: James E. Agenbroad, Harald Alvestrand, Andries Brouwer, + Mark Davis, Martin J. Duerst, Patrick Faltstrom, Ned Freed, David + Goldsmith, Tony Hansen, Edwin F. Hart, Paul Hoffman, David Hopwood, + Simon Josefsson, Kent Karlsson, Dan Kohn, Markus Kuhn, Michael Kung, + Alain LaBonte, Ira McDonald, Alexey Melnikov, MURATA Makoto, John + Gardiner Myers, Chris Newman, Dan Oscarsson, Roozbeh Pournader, + Murray Sargent, Markus Scherer, Keld Simonsen, Arnold Winkler, + Kenneth Whistler and Misha Wolf. + +12. Changes from RFC 2279 + + o Restricted the range of characters to 0000-10FFFF (the UTF-16 + accessible range). + + o Made Unicode the source of the normative definition of UTF-8, + keeping ISO/IEC 10646 as the reference for characters. + + o Straightened out terminology. UTF-8 now described in terms of an + encoding form of the character number. UCS-2 and UCS-4 almost + disappeared. + + o Turned the note warning against decoding of invalid sequences into + a normative MUST NOT. + + o Added a new section about the UTF-8 BOM, with advice for + protocols. + + o Removed suggested UNICODE-1-1-UTF-8 MIME charset registration. + + o Added an ABNF syntax for valid UTF-8 octet sequences + + o Expanded Security Considerations section, in particular impact of + Unicode normalization + + + + + + + + + +Yergeau Standards Track [Page 11] + +RFC 3629 UTF-8 November 2003 + + +13. Normative References + + [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate + Requirement Levels", BCP 14, RFC 2119, March 1997. + + [ISO.10646] International Organization for Standardization, + "Information Technology - Universal Multiple-octet coded + Character Set (UCS)", ISO/IEC Standard 10646, comprised + of ISO/IEC 10646-1:2000, "Information technology -- + Universal Multiple-Octet Coded Character Set (UCS) -- + Part 1: Architecture and Basic Multilingual Plane", + ISO/IEC 10646-2:2001, "Information technology -- + Universal Multiple-Octet Coded Character Set (UCS) -- + Part 2: Supplementary Planes" and ISO/IEC 10646- + 1:2000/Amd 1:2002, "Mathematical symbols and other + characters". + + [UNICODE] The Unicode Consortium, "The Unicode Standard -- Version + 4.0", defined by The Unicode Standard, Version 4.0 + (Boston, MA, Addison-Wesley, 2003. ISBN 0-321-18578-1), + April 2003, . + +14. Informative References + + [CESU-8] Phipps, T., "Unicode Technical Report #26: Compatibility + Encoding Scheme for UTF-16: 8-Bit (CESU-8)", UTR 26, + April 2002, + . + + [FSS_UTF] X/Open Company Ltd., "X/Open Preliminary Specification -- + File System Safe UCS Transformation Format (FSS-UTF)", + May 1993, . + + [RFC2045] Freed, N. and N. Borenstein, "Multipurpose Internet Mail + Extensions (MIME) Part One: Format of Internet Message + Bodies", RFC 2045, November 1996. + + [RFC2234] Crocker, D. and P. Overell, "Augmented BNF for Syntax + Specifications: ABNF", RFC 2234, November 1997. + + [RFC2978] Freed, N. and J. Postel, "IANA Charset Registration + Procedures", BCP 19, RFC 2978, October 2000. + + + + + + + +Yergeau Standards Track [Page 12] + +RFC 3629 UTF-8 November 2003 + + + [UAX15] Davis, M. and M. Duerst, "Unicode Standard Annex #15: + Unicode Normalization Forms", An integral part of The + Unicode Standard, Version 4.0.0, April 2003, . + + [US-ASCII] American National Standards Institute, "Coded Character + Set - 7-bit American Standard Code for Information + Interchange", ANSI X3.4, 1986. + +15. URIs + + [1] + +16. Intellectual Property Statement + + The IETF takes no position regarding the validity or scope of any + intellectual property or other rights that might be claimed to + pertain to the implementation or use of the technology described in + this document or the extent to which any license under such rights + might or might not be available; neither does it represent that it + has made any effort to identify any such rights. Information on the + IETF's procedures with respect to rights in standards-track and + standards-related documentation can be found in BCP-11. Copies of + claims of rights made available for publication and any assurances of + licenses to be made available, or the result of an attempt made to + obtain a general license or permission for the use of such + proprietary rights by implementors or users of this specification can + be obtained from the IETF Secretariat. + + The IETF invites any interested party to bring to its attention any + copyrights, patents or patent applications, or other proprietary + rights which may cover technology that may be required to practice + this standard. Please address the information to the IETF Executive + Director. + +17. Author's Address + + Francois Yergeau + Alis Technologies + 100, boul. Alexis-Nihon, bureau 600 + Montreal, QC H4M 2P2 + Canada + + Phone: +1 514 747 2547 + Fax: +1 514 747 2561 + EMail: fyergeau@alis.com + + + + + +Yergeau Standards Track [Page 13] + +RFC 3629 UTF-8 November 2003 + + +18. Full Copyright Statement + + Copyright (C) The Internet Society (2003). All Rights Reserved. + + This document and translations of it may be copied and furnished to + others, and derivative works that comment on or otherwise explain it + or assist in its implementation may be prepared, copied, published + and distributed, in whole or in part, without restriction of any + kind, provided that the above copyright notice and this paragraph are + included on all such copies and derivative works. However, this + document itself may not be modified in any way, such as by removing + the copyright notice or references to the Internet Society or other + Internet organizations, except as needed for the purpose of + developing Internet standards in which case the procedures for + copyrights defined in the Internet Standards process must be + followed, or as required to translate it into languages other than + English. + + The limited permissions granted above are perpetual and will not be + revoked by the Internet Society or its successors or assignees. + + This document and the information contained herein is provided on an + "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING + TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING + BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION + HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF + MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. + +Acknowledgement + + Funding for the RFC Editor function is currently provided by the + Internet Society. + + + + + + + + + + + + + + + + + + + +Yergeau Standards Track [Page 14] + diff --git a/decisiongrounding/scenarios_candidates/rfc_utf8_supersession/provenance.json b/decisiongrounding/scenarios_candidates/rfc_utf8_supersession/provenance.json new file mode 100644 index 0000000..629b43e --- /dev/null +++ b/decisiongrounding/scenarios_candidates/rfc_utf8_supersession/provenance.json @@ -0,0 +1,41 @@ +{ + "source": "rfc-editor.org", + "immutable": true, + "rfcs": [ + { + "id": "RFC-2279", + "number": 2279, + "file": "corpus/RFC-2279.md", + "url": "https://www.rfc-editor.org/rfc/rfc2279.txt", + "source_sha256": "e7873c93473ef2e21db92a4702f8e743fe5b70ff9ebef1096896aafe1ab0144d", + "category": "Standards Track", + "title": "UTF-8, a transformation format of ISO 10646", + "obsoletes": [ + "RFC-2044" + ], + "replaces": [] + }, + { + "id": "RFC-3629", + "number": 3629, + "file": "corpus/RFC-3629.md", + "url": "https://www.rfc-editor.org/rfc/rfc3629.txt", + "source_sha256": "a2a3a39457d30420c812f87a38cc2b9194832f6982ef17e619de7d816879d6a6", + "category": "Standards Track", + "title": "UTF-8, a transformation format of ISO 10646", + "obsoletes": [ + "RFC-2279" + ], + "replaces": [ + "RFC-2279" + ] + } + ], + "supersedes_edges": [ + { + "source": "RFC-3629", + "type": "supersedes", + "target": "RFC-2279" + } + ] +}