From 5635909af42eaf83ac817224648c23ebaf62b188 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mike=20Gro=C3=9Fmann?= Date: Tue, 25 Aug 2026 10:14:40 +0200 Subject: [PATCH 1/4] Restore the JES2 DDs the STC procedure allocates for its modules c35ab59 removed HASPCKPT and HASPACE1 from samplib/httpd because HTTPJES2 was gone and "nothing else in the server opens them". That check was run against httpd/src/ alone, and the question it answered is not the question that matters: a CGI module is dispatched by the LINK SVC into HTTPD's own task, so it has no allocations of its own and opens every ddname against the STC's. mvsMF's jobs API opens both data sets by name through libc370's jesopen() (src/jes/jesopen.c:36,46 -- checkpoint_open("DD:HASPCKPT"), spool_open("DD:HASPACE1")), from three call sites in jobsapi.c: jobListHandler, do_print_sysout and find_job_by_name_and_id. Without the DDs each answers 500 and writes two console lines per request, libc370's "Unable to open checkpoint dataset DD:HASPCKPT" and MVSMF201E. Job submit survives; jesiropn() dynallocs its own INTRDR. Measured on the built artifacts rather than argued: the 4.0.0 samplib XMIT decodes to zero "//HASP* DD" statements, the rebuilt one to both. The two lines come back verbatim as 3.3.x had them, not as symbolic parameters. Symbolics would add an installation parameter that can be got wrong and change the PROC interface in a patch release, and an operator diffing against 3.3.x should find nothing new to get right. The hardcoded VOL=SER is a genuine portability problem, but its fix is libc370 dynallocating the checkpoint and spool the way jesiropn() already dynallocs the INTRDR -- a library change, filed on its own. The comment says to check UNIT/VOL=SER against the site's JES2 procedure. migration.md is the more severe half and it is reversed here: it told a working 3.3.x system to delete the two DDs. installation.md gains the requirement in the section that already explains why a module needs the STC's STEPLIB -- same mechanism, one paragraph further. CLAUDE.md records the failure class: a grep that stops at this repo's edge cannot answer "does anything still use this", because four consumer repos run inside this address space. Re-ran that grep for the neighbouring removals in c35ab59 -- stck2tv, httpds_, HTTPJES2, HTTPDSL across mvsmf, httplua and httprexx: no hits. Those were sound. httplua dynallocs its own DDs, httprexx opens none, and the procedure's remaining DDs are complete -- SYSPRINT/SYSTERM/SYSIN are absent deliberately, httpstrt.c refuses to start when they are allocated. docs/release-notes/ is new: the v4.0.0 GitHub Release body, captured before the release is replaced. GitHub is not a source of record for a body that gets deleted with its release, and the 4.0.1 notes are built from this file so the changelog survives the correction. Fixes #256 --- CLAUDE.md | 11 ++++ docs/installation.md | 32 +++++++++ docs/migration.md | 6 +- docs/release-notes/README.md | 14 ++++ docs/release-notes/v4.0.0.md | 124 +++++++++++++++++++++++++++++++++++ samplib/httpd | 20 +++++- 6 files changed, 203 insertions(+), 4 deletions(-) create mode 100644 docs/release-notes/README.md create mode 100644 docs/release-notes/v4.0.0.md diff --git a/CLAUDE.md b/CLAUDE.md index c338d99..18c8d58 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -181,6 +181,17 @@ httpx->http_printf(httpc, "Content-Type: text/html\r\n\r\n"); In HTTPD 4.0.0, CGI registration moves from Lua defaults to Parmlib-only. No CGIs are active unless explicitly configured. +**A module runs in HTTPD's task, so `samplib/httpd` is part of the modules' +contract, not just the server's.** The LINK SVC dispatches into this task; a +module has no allocations of its own and opens any ddname against the STC's. +`HASPCKPT` / `HASPACE1` are there for exactly that reason — HTTPD never touches +them, mvsMF's jobs API opens both through libc370's `jesopen()`. They were +dropped in 4.0.0 on the reasoning that "nothing else in the server opens them", +which was measured against `httpd/src/` alone and missed the four consumer +repos; issue #256. **Before removing anything from the PROC, grep `../mvsmf`, +`../httplua` and `../httprexx` too** — a `grep` that stops at this repo's edge +cannot answer the question. + ### HTTP/1.1 Design (implemented) **Response body framing — decision logic:** diff --git a/docs/installation.md b/docs/installation.md index 0aa971a..b1dfb61 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -176,6 +176,38 @@ Comment the three lines out, or install mvsMF: STC's STEPLIB concatenation — HTTPD dispatches a module through the MVS LINK SVC, which searches the task's own libraries. +#### mvsMF's jobs API needs the JES2 DDs + +The same mechanism decides more than where the module is found. A module runs +**in HTTPD's task**, so every ddname it opens has to be allocated by the STC +procedure — a CGI has no allocations of its own. + +mvsMF's jobs API reads the JES2 checkpoint and spool directly, by DD name, and +the shipped procedure allocates them: + +``` +//HASPCKPT DD DISP=SHR,DSN=SYS1.HASPCKPT,UNIT=3350,VOL=SER=MVS000 +//HASPACE1 DD DISP=SHR,DSN=SYS1.HASPACE,UNIT=3350,VOL=SER=SPOOL1 +``` + +**Check `UNIT` and `VOL=SER` against your own JES2 procedure** before the first +start — they are site values, not constants, and the ones above are only what +the pattern has always carried. + +Without the two DDs the server starts normally and everything else works. The +jobs API does not: job list, single-job lookup and spool retrieval each answer +500 and write two console lines per request: + +``` +Unable to open checkpoint dataset DD:HASPCKPT +MVSMF201E UNABLE TO OPEN THE JES2 CHECKPOINT AND SPOOL DATA SETS +``` + +Job submit is unaffected; it dynallocs its own INTRDR. + +If you serve no JES2 traffic at all, the DDs can go. Nothing else in HTTPD opens +them. + ### RAKF is required for logins HTTPD has no user database of its own. Every login — the HTML form, HTTP Basic, diff --git a/docs/migration.md b/docs/migration.md index cb577bd..ed787ee 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -95,7 +95,11 @@ The 4-tier time-series statistics (`httpstat.c`, `httprepo.c`) with dataset pers The built-in web interfaces for JES2 job browsing (`HTTPJES2`) and dataset listing (`HTTPDSL`) are no longer built or shipped. [mvsMF](https://github.com/mvslovers/mvsmf)'s dataset and jobs REST APIs replace them. -Two things to check when migrating a 3.3.x configuration: drop any `CGI=`/`MOD=` line naming `HTTPDSL` or `HTTPJES2` — the route is accepted but the program load fails on the first matching request — and drop the `HASPCKPT` and `HASPACE1` DDs from the STC procedure, which existed only so `HTTPJES2` could read the JES2 spool. +When migrating a 3.3.x configuration, drop any `CGI=`/`MOD=` line naming `HTTPDSL` or `HTTPJES2` — the route is accepted but the program load fails on the first matching request. + +**Keep the `HASPCKPT` and `HASPACE1` DDs in the STC procedure.** `HTTPJES2` was not their only reader. A CGI module is dispatched by the MVS LINK SVC into HTTPD's own task, so it sees HTTPD's allocations — and mvsMF's jobs API opens both data sets by DD name, through libc370's `jesopen()`. Without them `GET /zosmf/restjobs/jobs`, single-job lookup and spool retrieval each answer 500 and write two console lines per request. Job submit still works; it dynallocs its own INTRDR. + +Remove them only if no module in your server reaches JES2. 4.0.0 shipped a procedure without them — see [issue #256](https://github.com/mvslovers/httpd/issues/256). The sources are kept under `tbd/` for reference and are outside the build. diff --git a/docs/release-notes/README.md b/docs/release-notes/README.md new file mode 100644 index 0000000..cfc6535 --- /dev/null +++ b/docs/release-notes/README.md @@ -0,0 +1,14 @@ +# Release notes + +The published GitHub Release body for each tag, kept here as the source of +record. GitHub is not one: a release can be deleted or replaced, and the notes +go with it. + +That is not hypothetical. `v4.0.0` shipped an STC procedure without the JES2 +DDs its own documented CGI needed ([#256](https://github.com/mvslovers/httpd/issues/256)), +so the release was replaced rather than amended. `v4.0.0.md` is that body, +captured before the deletion — the 4.0.1 notes are built from it so the +changelog survives the correction. + +One file per tag, named for it. Write the file when the release is published, +not later. diff --git a/docs/release-notes/v4.0.0.md b/docs/release-notes/v4.0.0.md new file mode 100644 index 0000000..122dae9 --- /dev/null +++ b/docs/release-notes/v4.0.0.md @@ -0,0 +1,124 @@ +HTTPD 4.0.0 is the first release of the web server since Mike Rayborn's 3.3.x, +and the first that could be installed through **SMP 4** — the SMP that +ships with MVS 3.8j, not SMP/E — so the system keeps a record of what is +installed, and there is a defined way back out again. + +It is also a smaller server than 3.3.x. Everything that was not an HTTP server +has left the code base, and nothing is reachable that the configuration member +does not name. + +## Installing + +Download **`httpd-4.0.0-dist.zip`** (or `.tar.gz`). It holds everything needed +and a step-by-step guide: + +| | | +|---|---| +| `README.md` | the installation guide | +| `httpd-4.0.0-load.xmit` | the five load modules | +| `httpd-4.0.0-samplib.xmit` | the STC procedure, the configuration pattern, and a job for the webroot | +| `httpd-4.0.0-alloc.jcl` | allocate the datasets — run once | +| `httpd-4.0.0-inst.jcl` | receive, apply, accept | +| `httpd-webroot.img` | the document root as a UFS370 disk image | + +Upload the two XMITs in **binary**, submit the allocation job once, then the +install job — the only lines to edit are the two dataset names you uploaded to. +The SYSMOD travels inline in the job, so there is no third file to transfer. + +Everything lands under `HTTPD.V4R0M0.*`. Copying the procedure and the +configuration pattern into your PROCLIB and PARMLIB stays a manual step, by +design: the product owns the patterns, your system owns the copies. The guide +says which member goes where. + +FMID is **`THTP400`**. + +## What is new since 3.3.x + +**HTTP/1.1.** Persistent connections and chunked transfer encoding, both on by +default for 1.1 clients; 1.0 clients always get `Connection: close`. A response +whose length is known goes out with `Content-Length`, everything else is +chunked, and a body-less status never gets either. + +**Configuration is a Parmlib member, not a Lua script.** `DD:HTTPPRM`, FB-80, +`KEYWORD=VALUE`. The Lua config engine is gone, and with it the interpreter. +`F HTTPD,D CONFIG` reports what the server actually parsed, which is the one +place a typo that fell back to a default becomes visible. + +**Authentication is per route, and only per route.** Each `MOD=` and `LOC=` +line carries its own `AUTH=NONE|FORM|BASIC|TOKEN`, optionally with a +`RES=class:resource` check against RAKF. There is no global policy to fall back +on any more — **a route without `AUTH=` is public**, which the shipped pattern +now says in as many words. `TOKEN` answers a bare 401 without a +`WWW-Authenticate`, for API clients that handle the 401 themselves. + +**Nothing is registered unless you ask for it.** No CGI, no display module, no +static prefix is reachable unless a line in the member names it. A fresh +install serves nothing at all until you say what it should serve. + +**Static files come from UFSD and nowhere else.** The DD-based document root of +3.3.x is gone. `DOCROOT` names a path in the UFSD filesystem — and this release +ships one: `httpd-webroot.img`, a formatted UFS370 disk holding the welcome +page. It is uploaded with **IND$FILE** into a `RECFM=U BLKSIZE=4096` dataset +and mounted by UFSD at `/www`; the sample library carries the allocation job +(`HTTPWEBR`), and section 9 of the guide walks the whole path. Neither SMP nor +TSO RECEIVE can carry a filesystem image, which is why it travels on its own. + +**SMF recording.** Type 243 records plus counters, at four levels from `NONE` +to `ALL`, replacing the ad-hoc statistics of 3.3.x. + +**An abending CGI costs the request, not the address space.** Every module runs +with its heap in a request subpool that is released in one `FREEMAIN` when the +request ends, however it ends. + +**Removed, and where it went:** +the embedded FTP daemon → [mvslovers/ftpd](https://github.com/mvslovers/ftpd); +the Lua and REXX handlers → [mvslovers/httplua](https://github.com/mvslovers/httplua) and + [mvslovers/httprexx](https://github.com/mvslovers/httprexx); (both not released, yet) +the MQTT telemetry, with no replacement; +the dataset lister and the JES2 spool browser, both superseded by [mvsMF](https://github.com/mvslovers/mvsmf)'s APIs. + +Upgrading from 3.3.x: the +[migration guide](https://github.com/mvslovers/httpd/blob/main/docs/migration.md) +covers the configuration rewrite and every removal. +The `v3.3.x` branch stays maintained for Mike Rayborn's version. + +## Before you start it + +- **RAKF is required.** HTTPD authorizes itself through SVC 244 and refuses to + start without either RAKF or an APF list entry. No option turns that off, and + every login is verified by RAKF — on a system without it, nobody can log in. +- **mvsMF is what the shipped configuration expects.** `HTTPPRM0` routes + `/zosmf/*` to it. Install + [mvsMF](https://github.com/mvslovers/mvsmf/releases) and put its library in + the STC's STEPLIB, or comment the three lines out. +- **UFSD is optional** and takes static files with it. Without the started task + the server writes `HTTPD044W` and runs on; every `MOD=` route works normally. + A CGI-only HTTPD is an ordinary deployment. +- **On TK5, `SYS2.PARMLIB` does not exist.** Put `HTTPPRM0` in `SYS1.PARMLIB` + and start with `/S HTTPD,D='SYS1.PARMLIB'`. + +## The four modules beside the server + +`HTTPDSRV`, `HTTPDM`, `HTTPDMTT` and `ABEND0C1` ship in the load library and are +inert: `/.dsrv` reads server control blocks, `/.dm` arbitrary storage, `/.dmtt` +the console log, and `ABEND0C1` abends on purpose. They exist so a problem on a +live system can be looked at without a private build. + +They are diagnostic, debugging and development tools, **and the next release +will not deliver them**. Nothing depends on them, and if you do register one, +give it an `AUTH=` — `/.dm` without one hands arbitrary storage reads to anyone +who can reach the port. + +## Removing it again + +`RESTORE` followed by `REJECT` does not work, and that is not a defect in the +package: the install accepts the FMID in the same run as the APPLY, and an +accepted function SYSMOD refuses `RESTORE` because it was accepted, then +refuses `REJECT` because accepting removed the control statements `REJECT` +works from. Section 12 of the shipped guide has the `UCLIN` job that does work +— the same one that was run in the rehearsal above. + +--- + +**Full changelog**: https://github.com/mvslovers/httpd/compare/3.3.0...v4.0.0 + diff --git a/samplib/httpd b/samplib/httpd index a61d55e..5339994 100644 --- a/samplib/httpd +++ b/samplib/httpd @@ -46,6 +46,20 @@ //HTTPSTAT DD SYSOUT=* //HTTPPRM DD DSN=&D(&M),DISP=SHR,FREE=CLOSE //* -//* HASPCKPT / HASPACE1 were the JES2 spool datasets read by HTTPJES2. -//* That module is no longer shipped (use mvsMF's jobs API), and nothing -//* else in the server opens them, so the DDs are gone. +//* HASPCKPT / HASPACE1 are the JES2 checkpoint and spool data sets. +//* HTTPD itself never opens them. They are allocated here for the CGI +//* modules that run in this address space: mvsMF's jobs API calls +//* jesopen(), which opens them by DD name, and a module is dispatched +//* by the LINK SVC into this task - so the allocation has to be the +//* server's. Without them every job list, job lookup and spool read +//* answers 500 and writes two console lines. Job submit is unaffected, +//* it dynallocs its own INTRDR. +//* +//* HTTPJES2 read the same two data sets and is no longer shipped. That +//* is why they were briefly dropped, which broke mvsMF (issue #256). +//* Remove them only if no module in this server reaches JES2. +//* +//* UNIT and VOL=SER must match your own JES2 procedure - check them +//* there before the first start. DISP=SHR: JES2 owns these data sets. +//HASPCKPT DD DISP=SHR,DSN=SYS1.HASPCKPT,UNIT=3350,VOL=SER=MVS000 +//HASPACE1 DD DISP=SHR,DSN=SYS1.HASPACE,UNIT=3350,VOL=SER=SPOOL1 From 9fea98c0c5bb577cee768884598f8119b3256f3a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mike=20Gro=C3=9Fmann?= Date: Tue, 25 Aug 2026 10:17:19 +0200 Subject: [PATCH 2/4] TODO: rank #256 first and record the decision it leaves open The file was reconciled against a tracker with three issues open and 4.0.0 untagged. Both moved: 4.0.0 shipped on 2026-08-24, #254 and #256 are open, and #256 outranks everything because it is the only item that is wrong on a system someone can install today. What the tracker cannot hold, and what this file is for: the code half of #256 is written, the release half is a decision. THTP400 is ACCEPTed on any system that installed 4.0.0, so a 4.0.1 SYSMOD under the same FMID cannot be received over it -- and does not need to be, because the corrected member is in the sample library and the sample library is not an SMP element. Such a system needs one PROCLIB member replaced and nothing from SMP. Also records why the SMP rehearsal in smp-todo.md did not catch this: it verified that the install works, never that the procedure it installs still allocates what the CGI it routes to opens. The two issues filed out of #256 go under Cross-repo, both explicitly not blocking 4.0.1: mvsmf#357 (state the requirement where a mvsMF installer looks) and libc370#142 (dynalloc the checkpoint and spool, removing the class). --- TODO.md | 104 +++++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 84 insertions(+), 20 deletions(-) diff --git a/TODO.md b/TODO.md index 4a3c21b..bcb0612 100644 --- a/TODO.md +++ b/TODO.md @@ -11,9 +11,10 @@ stops. `CLAUDE.md` forbids a task list in itself because a copy of a tracker is wrong the first time someone closes something, and the only defence that works is to hold nothing worth going stale. -*Last reconciled against the tracker: 2026-08-24, three issues open (#252 filed -and closed the same day by PR #253; before it #250, #237 by PR #249, #245 by -PR #248, #233 by PR #244, #242 by PR #246 and #243 by PR #247).* +*Last reconciled against the tracker: 2026-08-25, five issues open (#256 filed +that day with PR #257; #254; before them #252 filed and closed 2026-08-24 by +PR #253, #250, #237 by PR #249, #245 by PR #248, #233 by PR #244, #242 by +PR #246 and #243 by PR #247).* --- @@ -21,35 +22,90 @@ PR #248, #233 by PR #244, #242 by PR #246 and #243 by PR #247).* | | Issue | Kind | Waiting on | |---|---|---|---| -| 1 | #250 | `type:research` — the `type:docs` half landed | **MVS time**, two `/.dm` calls | +| 1 | #256 | **regression in a published release** | PR #257 merged, then a **decision** — below | +| 2 | #254 | `type:docs` — six Parmlib keywords undocumented | nothing | +| 3 | #250 | `type:research` — the `type:docs` half landed | **MVS time**, two `/.dm` calls | | — | #198 | hygiene, explicitly not a bug | **#250(b)**, then milestone 4.1.0 | | — | #176 | security, the heaviest by a wide margin | **RAKF** — see *Deferred* | -**Nothing open waits on a decision any more,** and with #237 merged nothing -open is a code bug either. Nothing open blocks 4.0.0: both #250 and #198 carry -the 4.1.0 milestone, and #176 is parked on another organisation. - -#250 is still ranked first even though it is milestoned out, and the two are not -in conflict. The milestone says it is not a 4.0.0 deliverable; the rank says it -is the next thing to do, because it is read-only, costs minutes, and #198's -second step cannot be estimated until it is answered. +#256 outranks everything because it is the only item that is wrong on a system +someone can install today, and because it is the only one whose fix is already +written. #176 stays last for the reason it always did, not because it is small. + +#250 is still ranked above the two parked items even though it is milestoned +out, and the two are not in conflict. The milestone says it is not a 4.0.0 +deliverable; the rank says it is the next thing to do once #256 is off the +table, because it is read-only, costs minutes, and #198's second step cannot be +estimated until it is answered. + +**One decision is open, and it is #256's second half.** The code fix is written +(PR #257); what needs the maintainer is how 4.0.1 reaches a system that already +installed 4.0.0. `THTP400` is ACCEPTed there, so a 4.0.1 full-function SYSMOD +under the same FMID cannot simply be received over it — and it does not need to +be. The corrected member is in the **sample library, which is not an SMP +element** (the generated `inst.jcl` says so itself), so such a system needs one +PROCLIB member replaced and nothing from SMP at all. That makes 4.0.1 a fresh +install's package plus a one-line instruction for everyone else, but whether it +ships as a re-tag over a deleted `v4.0.0` or beside it is not a call this file +makes. **The return-code work is finished.** #226 and #245 between them settled every exit that could end a refused start `CC 0000`; nothing in that thread is open, and `docs/messages.md` §*A refused start* is where the resulting contract is written down. -**Nothing MVS-side blocks the 4.0.0 tag any more.** The release work was never -in the tracker — it is in [`smp-todo.md`](smp-todo.md) — and as of 2026-08-24 it -is done: `THTP400` is free in the CDS *and* the ACDS on both stands (O1), the -full install was rehearsed under the throwaway FMID `TTST400` and cleaned back -off with UCLIN, every step CC 0000 (O2), and the document root ships as a UFS -image (O3, #252). What is left there is a checksum for the release assets and -O4, a convenience sample nobody is blocked on. +**4.0.0 tagged and shipped on 2026-08-24, and is being replaced.** The release +work was never in the tracker — it is in [`smp-todo.md`](smp-todo.md) — and it +was done: `THTP400` free in the CDS *and* the ACDS on both stands (O1), the full +install rehearsed under the throwaway FMID `TTST400` and cleaned back off with +UCLIN, every step CC 0000 (O2), the document root shipping as a UFS image (O3, +#252). What is left there is a checksum for the release assets and O4, a +convenience sample nobody is blocked on. + +What none of that rehearsal caught is #256: the install was verified to *work*, +never that the procedure it installed still allocated what the CGI it routes to +needs. The published body is preserved at +[`docs/release-notes/v4.0.0.md`](release-notes/v4.0.0.md) so the changelog +survives the replacement — the 4.0.1 notes are built from it. --- -### 1 · #250 — does a LINKed module survive in the Job Pack Area? +### 1 · #256 — the STC procedure stopped allocating what its modules open + +*fix written (PR #257); what is left is a release decision, not code* + +`c35ab59` took `HASPCKPT` and `HASPACE1` out of `samplib/httpd` because +`HTTPJES2` was gone and nothing in `httpd/src/` opened them. Both halves of that +were true and the conclusion was still wrong: a CGI module is dispatched by the +LINK SVC **into HTTPD's task**, so it opens every ddname against the STC's +allocations, and mvsMF's jobs API opens both data sets through libc370's +`jesopen()`. 4.0.0 therefore shipped a procedure that breaks the API its own +default configuration routes to. + +The rank is not about size — the diff is two lines and some prose. It is that +this is the only open item that is wrong on a system someone can install today. + +Two things worth keeping out of the issue thread: + +- **`docs/migration.md` was the more severe half.** A missing DD breaks a *new* + install; that file told a *working* 3.3.x system to delete one it needs. + Damage that spreads by being read is worse than damage that sits in a package. +- **The lesson generalises past this bug.** Every "nothing uses this any more" + check in the 4.0.0 removals was scoped to this repo, and four consumer repos + run inside this address space. Re-running it for the neighbouring removals in + `c35ab59` (`stck2tv`, `httpds_`, `HTTPJES2`, `HTTPDSL` across mvsmf, httplua, + httprexx) found nothing — so this was the one instance, not the pattern. The + rule is now in `CLAUDE.md`. + +### 2 · #254 — six Parmlib keywords the parser accepts are undocumented + +*nothing blocks it; it is small and nobody has picked it up* + +Ranked above #250 only because it costs no MVS time. `httpprm.c` is the +authority for what the parser accepts; `docs/configuration.md` is what an +operator reads, and the two disagree by six keywords. + +### 3 · #250 — does a LINKed module survive in the Job Pack Area? *what is left of it needs a live server and nothing else* @@ -216,6 +272,14 @@ Unblocked and outside this repo: `mvslovers/ftpd#97` (now the last Phase 1 item), `mvslovers/mvsmf#329`, `mvslovers/mvsmf#345`. Also `blocked:rakf`: `mvslovers/ftpd#64`. +Filed out of #256 on 2026-08-25, neither blocking 4.0.1: +`mvslovers/mvsmf#357` states the DD requirement where a mvsMF installer will +look for it — httpd documents it from one side of the boundary, nothing +documented it from the other. `mvslovers/libc370#142` is the fix that removes +the class: `jesopen()` should dynalloc the checkpoint and spool the way +`jesiropn()` already dynallocs the INTRDR, which also gets the site-specific +`VOL=SER` out of every procedure in the ecosystem. + Closed *not planned* on 2026-08-23: `mvslovers/ufsd#65` — reasoning in its closing comment and in `docs/identity-redesign.md` §3.2. The per-client permission question that replaces it is `mvslovers/ufsd#67`. From d12e81eed70175ef8a6ba5a82b2c447344ea0e4d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mike=20Gro=C3=9Fmann?= Date: Tue, 25 Aug 2026 10:20:11 +0200 Subject: [PATCH 3/4] Correct three claims in the #256 documentation All three were written from one measurement and generalised, which is the same move that produced the bug being fixed. The jobs API does not fail uniformly. Two of the three jesopen() call sites answer 500, as documented -- jobListHandler and do_print_sysout both send REASON_INCORRECT_JES_VSAM_HANDLE. The third does not: find_job_by_name_and_id() returns NULL on an unreachable JES2, and all four of its callers report that as 404 job not found, the same answer they give for a job that genuinely is not there. So a client resolving one job by name and id is told it does not exist while it does. That is worth more than the sentence it replaces: the status code misreports the cause, and the console pair is the only reliable signal. docs/release-notes/README.md described a convention the release machinery does not implement. mbt's shared release.yml deletes every existing release for the tag and recreates it with `gh release create --generate-notes`, so a body is built from commit subjects and a hand-written one does not survive a re-push. The README now says what actually applies a body (`gh release edit --notes-file`, after the workflow, not before) and warns that reusing v4.0.0's means substituting its six artifact names and every HTTPD.V4R0M0.* qualifier. "Four consumer repos" was three: mvsmf, httplua and httprexx ship modules that run in this address space. ftpd is a separate server, not a module. The wrong count sat in the two files that now carry the rule. --- CLAUDE.md | 4 ++-- TODO.md | 4 ++-- docs/installation.md | 14 ++++++++++-- docs/migration.md | 2 +- docs/release-notes/README.md | 43 ++++++++++++++++++++++++++++-------- 5 files changed, 51 insertions(+), 16 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 18c8d58..bca3619 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -187,8 +187,8 @@ module has no allocations of its own and opens any ddname against the STC's. `HASPCKPT` / `HASPACE1` are there for exactly that reason — HTTPD never touches them, mvsMF's jobs API opens both through libc370's `jesopen()`. They were dropped in 4.0.0 on the reasoning that "nothing else in the server opens them", -which was measured against `httpd/src/` alone and missed the four consumer -repos; issue #256. **Before removing anything from the PROC, grep `../mvsmf`, +which was measured against `httpd/src/` alone and missed the three repos whose +modules run in this address space; issue #256. **Before removing anything from the PROC, grep `../mvsmf`, `../httplua` and `../httprexx` too** — a `grep` that stops at this repo's edge cannot answer the question. diff --git a/TODO.md b/TODO.md index bcb0612..52d4edd 100644 --- a/TODO.md +++ b/TODO.md @@ -91,8 +91,8 @@ Two things worth keeping out of the issue thread: install; that file told a *working* 3.3.x system to delete one it needs. Damage that spreads by being read is worse than damage that sits in a package. - **The lesson generalises past this bug.** Every "nothing uses this any more" - check in the 4.0.0 removals was scoped to this repo, and four consumer repos - run inside this address space. Re-running it for the neighbouring removals in + check in the 4.0.0 removals was scoped to this repo, and three other repos + ship modules that run inside this address space. Re-running it for the neighbouring removals in `c35ab59` (`stck2tv`, `httpds_`, `HTTPJES2`, `HTTPDSL` across mvsmf, httplua, httprexx) found nothing — so this was the one instance, not the pattern. The rule is now in `CLAUDE.md`. diff --git a/docs/installation.md b/docs/installation.md index b1dfb61..87eba85 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -195,8 +195,18 @@ start — they are site values, not constants, and the ones above are only what the pattern has always carried. Without the two DDs the server starts normally and everything else works. The -jobs API does not: job list, single-job lookup and spool retrieval each answer -500 and write two console lines per request: +jobs API does not — and it does not fail uniformly: + +| request | answer | +|---|---| +| job list (`GET /zosmf/restjobs/jobs`) | `500`, `REASON_INCORRECT_JES_VSAM_HANDLE` | +| spool retrieval | `500`, same reason | +| anything resolving one job by name and id | **`404` job not found** | + +The 404 is the one to know about. `find_job_by_name_and_id()` returns NULL +whether JES2 is unreachable or the job genuinely is not there, and its callers +report the second. A client is told the job does not exist while it does — so +read the console, not the status code: ``` Unable to open checkpoint dataset DD:HASPCKPT diff --git a/docs/migration.md b/docs/migration.md index ed787ee..df04d36 100644 --- a/docs/migration.md +++ b/docs/migration.md @@ -97,7 +97,7 @@ The built-in web interfaces for JES2 job browsing (`HTTPJES2`) and dataset listi When migrating a 3.3.x configuration, drop any `CGI=`/`MOD=` line naming `HTTPDSL` or `HTTPJES2` — the route is accepted but the program load fails on the first matching request. -**Keep the `HASPCKPT` and `HASPACE1` DDs in the STC procedure.** `HTTPJES2` was not their only reader. A CGI module is dispatched by the MVS LINK SVC into HTTPD's own task, so it sees HTTPD's allocations — and mvsMF's jobs API opens both data sets by DD name, through libc370's `jesopen()`. Without them `GET /zosmf/restjobs/jobs`, single-job lookup and spool retrieval each answer 500 and write two console lines per request. Job submit still works; it dynallocs its own INTRDR. +**Keep the `HASPCKPT` and `HASPACE1` DDs in the STC procedure.** `HTTPJES2` was not their only reader. A CGI module is dispatched by the MVS LINK SVC into HTTPD's own task, so it sees HTTPD's allocations — and mvsMF's jobs API opens both data sets by DD name, through libc370's `jesopen()`. Without them the job list and spool retrieval answer 500, and anything resolving a single job by name and id answers **404 job not found** — `find_job_by_name_and_id()` returns NULL for an unreachable JES2 exactly as it does for an absent job, so the status code misreports the cause. Every one of them writes two console lines per request; that is the reliable signal. Job submit still works — it dynallocs its own INTRDR. Remove them only if no module in your server reaches JES2. 4.0.0 shipped a procedure without them — see [issue #256](https://github.com/mvslovers/httpd/issues/256). diff --git a/docs/release-notes/README.md b/docs/release-notes/README.md index cfc6535..6267258 100644 --- a/docs/release-notes/README.md +++ b/docs/release-notes/README.md @@ -1,14 +1,39 @@ # Release notes The published GitHub Release body for each tag, kept here as the source of -record. GitHub is not one: a release can be deleted or replaced, and the notes -go with it. +record. GitHub is not one, for two separate reasons. -That is not hypothetical. `v4.0.0` shipped an STC procedure without the JES2 -DDs its own documented CGI needed ([#256](https://github.com/mvslovers/httpd/issues/256)), -so the release was replaced rather than amended. `v4.0.0.md` is that body, -captured before the deletion — the 4.0.1 notes are built from it so the -changelog survives the correction. +**The workflow overwrites the body.** `make release` pushes a tag, which runs +mbt's shared `release.yml`. That job **deletes every existing release for the +tag** and recreates it with `gh release create --generate-notes` — a body built +from commit subjects. A hand-written body is not an input to that process and +does not survive a re-push of the same tag. -One file per tag, named for it. Write the file when the release is published, -not later. +**A deleted release takes its body with it.** `v4.0.0` shipped an STC procedure +without the JES2 DDs its own documented CGI needed +([#256](https://github.com/mvslovers/httpd/issues/256)), so it was replaced +rather than amended. [`v4.0.0.md`](v4.0.0.md) is that body, captured before the +deletion — the 4.0.1 notes are built from it so the changelog survives the +correction. + +## How a body here reaches a release + +It does not, on its own. After the release workflow finishes: + +```sh +gh release edit v4.0.1 --repo mvslovers/httpd \ + --notes-file docs/release-notes/v4.0.1.md +``` + +Order matters — run it after the workflow, or the recreate discards it. + +## Writing the next one from the last + +Reusing a body means substituting every version string in it, not just the +heading. `v4.0.0.md` carries `httpd-4.0.0-dist.zip` and five more artifact +names, the `HTTPD.V4R0M0.*` dataset qualifiers, and `V4R0M0` inside the sample +JCL it quotes. Diff the rendered file against the real `dist/` listing before +publishing. + +One file per tag, named for it. Write it when the release is published, not +later. From 14e54ef41d6d24c748b8cb8ba17aaa09a33fb71a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mike=20Gro=C3=9Fmann?= Date: Tue, 25 Aug 2026 10:30:26 +0200 Subject: [PATCH 4/4] Release notes for 4.0.1, built from the withdrawn 4.0.0 body Every version string substituted -- six artifact names, the HTTPD.V4R0M0.* qualifiers, and the compare link. FMID stays THTP400: no code changed between the two releases, so there is no functional level to cut. The new section leads because it is what an existing installation needs, and what it needs is small: add the two DD statements to the PROCLIB member and restart. SMP has nothing to do -- the corrected member is in the sample library, which is not an SMP element, and the load modules are identical. It also says what not to do, which is the trap this package sets. The shipped procedure's STEPLIB names HTTPD.V4R0M1.LINKLIB, the library this package installs into. Copying the new member wholesale onto a system running 4.0.0 points the started task at a dataset that does not exist there. Take the two lines, not the member. --- docs/release-notes/v4.0.1.md | 171 +++++++++++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 docs/release-notes/v4.0.1.md diff --git a/docs/release-notes/v4.0.1.md b/docs/release-notes/v4.0.1.md new file mode 100644 index 0000000..e610d89 --- /dev/null +++ b/docs/release-notes/v4.0.1.md @@ -0,0 +1,171 @@ +HTTPD 4.0.1 **replaces 4.0.0**, which was withdrawn. Everything below is the +4.0.0 release text, corrected — read *What 4.0.1 fixes* first if you already +installed 4.0.0. + +It is the first release of the web server since Mike Rayborn's 3.3.x, and the +first that could be installed through **SMP 4** — the SMP that ships with +MVS 3.8j, not SMP/E — so the system keeps a record of what is installed, and +there is a defined way back out again. + +It is also a smaller server than 3.3.x. Everything that was not an HTTP server +has left the code base, and nothing is reachable that the configuration member +does not name. + +## What 4.0.1 fixes + +4.0.0 shipped an STC procedure that had stopped allocating the JES2 checkpoint +and spool: + +``` +//HASPCKPT DD DISP=SHR,DSN=SYS1.HASPCKPT,UNIT=3350,VOL=SER=MVS000 +//HASPACE1 DD DISP=SHR,DSN=SYS1.HASPACE,UNIT=3350,VOL=SER=SPOOL1 +``` + +They were dropped with `HTTPJES2`, the module that used to read them, on the +reasoning that nothing else in the server opened them. Nothing else in the +*server* does — but a CGI module is dispatched by the LINK SVC **into HTTPD's +own task**, so it has no allocations of its own and opens every ddname against +the STC's. mvsMF's jobs API opens both data sets by name, and the shipped +configuration routes `/zosmf/*` straight to it. + +Without the two DDs the server starts normally and everything else works. The +jobs API does not, and it does not fail uniformly: the job list and spool +retrieval answer `500`, while anything resolving a single job by name and id +answers **`404` job not found** for a job that exists. Two console lines are +written per attempt — that pair is the reliable signal, not the status code. +Job submit is unaffected. + +`docs/migration.md` carried the same error the other way round, telling anyone +migrating a working 3.3.x configuration to delete the two DDs. Both are fixed, +and the installation guide now states the requirement where it explains why a +module needs the STC's STEPLIB — it is the same mechanism. + +Details in [#256](https://github.com/mvslovers/httpd/issues/256). + +### If you already installed 4.0.0 + +**No code changed between 4.0.0 and 4.0.1.** The load modules are functionally +identical, the FMID is unchanged, and SMP has nothing to do. Add the two DD +statements above to your existing `PROCLIB` member, adjusting `UNIT` and +`VOL=SER` to match your own JES2 procedure, and restart the started task. + +**Do not copy the new procedure over your own.** Its `STEPLIB` names +`HTTPD.V4R0M1.LINKLIB`, the library *this* package installs into — on a system +running 4.0.0 that dataset does not exist and the task will not start. Take the +two lines, not the member. + +## Installing + +Download **`httpd-4.0.1-dist.zip`** (or `.tar.gz`). It holds everything needed +and a step-by-step guide: + +| | | +|---|---| +| `README.md` | the installation guide | +| `httpd-4.0.1-load.xmit` | the five load modules | +| `httpd-4.0.1-samplib.xmit` | the STC procedure, the configuration pattern, and a job for the webroot | +| `httpd-4.0.1-alloc.jcl` | allocate the datasets — run once | +| `httpd-4.0.1-inst.jcl` | receive, apply, accept | +| `httpd-webroot.img` | the document root as a UFS370 disk image | + +Upload the two XMITs in **binary**, submit the allocation job once, then the +install job — the only lines to edit are the two dataset names you uploaded to. +The SYSMOD travels inline in the job, so there is no third file to transfer. + +Everything lands under `HTTPD.V4R0M1.*`. Copying the procedure and the +configuration pattern into your PROCLIB and PARMLIB stays a manual step, by +design: the product owns the patterns, your system owns the copies. The guide +says which member goes where. + +FMID is **`THTP400`**. + +## What is new since 3.3.x + +**HTTP/1.1.** Persistent connections and chunked transfer encoding, both on by +default for 1.1 clients; 1.0 clients always get `Connection: close`. A response +whose length is known goes out with `Content-Length`, everything else is +chunked, and a body-less status never gets either. + +**Configuration is a Parmlib member, not a Lua script.** `DD:HTTPPRM`, FB-80, +`KEYWORD=VALUE`. The Lua config engine is gone, and with it the interpreter. +`F HTTPD,D CONFIG` reports what the server actually parsed, which is the one +place a typo that fell back to a default becomes visible. + +**Authentication is per route, and only per route.** Each `MOD=` and `LOC=` +line carries its own `AUTH=NONE|FORM|BASIC|TOKEN`, optionally with a +`RES=class:resource` check against RAKF. There is no global policy to fall back +on any more — **a route without `AUTH=` is public**, which the shipped pattern +now says in as many words. `TOKEN` answers a bare 401 without a +`WWW-Authenticate`, for API clients that handle the 401 themselves. + +**Nothing is registered unless you ask for it.** No CGI, no display module, no +static prefix is reachable unless a line in the member names it. A fresh +install serves nothing at all until you say what it should serve. + +**Static files come from UFSD and nowhere else.** The DD-based document root of +3.3.x is gone. `DOCROOT` names a path in the UFSD filesystem — and this release +ships one: `httpd-webroot.img`, a formatted UFS370 disk holding the welcome +page. It is uploaded with **IND$FILE** into a `RECFM=U BLKSIZE=4096` dataset +and mounted by UFSD at `/www`; the sample library carries the allocation job +(`HTTPWEBR`), and section 9 of the guide walks the whole path. Neither SMP nor +TSO RECEIVE can carry a filesystem image, which is why it travels on its own. + +**SMF recording.** Type 243 records plus counters, at four levels from `NONE` +to `ALL`, replacing the ad-hoc statistics of 3.3.x. + +**An abending CGI costs the request, not the address space.** Every module runs +with its heap in a request subpool that is released in one `FREEMAIN` when the +request ends, however it ends. + +**Removed, and where it went:** +the embedded FTP daemon → [mvslovers/ftpd](https://github.com/mvslovers/ftpd); +the Lua and REXX handlers → [mvslovers/httplua](https://github.com/mvslovers/httplua) and + [mvslovers/httprexx](https://github.com/mvslovers/httprexx); (both not released, yet) +the MQTT telemetry, with no replacement; +the dataset lister and the JES2 spool browser, both superseded by [mvsMF](https://github.com/mvslovers/mvsmf)'s APIs. + +Upgrading from 3.3.x: the +[migration guide](https://github.com/mvslovers/httpd/blob/main/docs/migration.md) +covers the configuration rewrite and every removal. +The `v3.3.x` branch stays maintained for Mike Rayborn's version. + +## Before you start it + +- **RAKF is required.** HTTPD authorizes itself through SVC 244 and refuses to + start without either RAKF or an APF list entry. No option turns that off, and + every login is verified by RAKF — on a system without it, nobody can log in. +- **mvsMF is what the shipped configuration expects.** `HTTPPRM0` routes + `/zosmf/*` to it. Install + [mvsMF](https://github.com/mvslovers/mvsmf/releases) and put its library in + the STC's STEPLIB, or comment the three lines out. +- **UFSD is optional** and takes static files with it. Without the started task + the server writes `HTTPD044W` and runs on; every `MOD=` route works normally. + A CGI-only HTTPD is an ordinary deployment. +- **On TK5, `SYS2.PARMLIB` does not exist.** Put `HTTPPRM0` in `SYS1.PARMLIB` + and start with `/S HTTPD,D='SYS1.PARMLIB'`. + +## The four modules beside the server + +`HTTPDSRV`, `HTTPDM`, `HTTPDMTT` and `ABEND0C1` ship in the load library and are +inert: `/.dsrv` reads server control blocks, `/.dm` arbitrary storage, `/.dmtt` +the console log, and `ABEND0C1` abends on purpose. They exist so a problem on a +live system can be looked at without a private build. + +They are diagnostic, debugging and development tools, **and the next release +will not deliver them**. Nothing depends on them, and if you do register one, +give it an `AUTH=` — `/.dm` without one hands arbitrary storage reads to anyone +who can reach the port. + +## Removing it again + +`RESTORE` followed by `REJECT` does not work, and that is not a defect in the +package: the install accepts the FMID in the same run as the APPLY, and an +accepted function SYSMOD refuses `RESTORE` because it was accepted, then +refuses `REJECT` because accepting removed the control statements `REJECT` +works from. Section 12 of the shipped guide has the `UCLIN` job that does work +— the same one that was run in the rehearsal above. + +--- + +**Full changelog**: https://github.com/mvslovers/httpd/compare/3.3.0...v4.0.1 +