Skip to content

packages.json page order is not deterministic for auto-resolved file attachments #172

Description

@simontaurus

Problem

create_page_package writes the pages array of packages.json in a different order on every run, even when the wiki content has not changed. A package release that adds one page therefore produces a diff of several hundred lines, which hides the actual change.

Measured on world.opensemantic.meta.docs (91 pages), two consecutive builds against an unchanged wiki:

sha run 1: c18e9061914911172481
sha run 2: 6508a05386d07192a55c

Semantic comparison of a release that added exactly one page: 1 page added, 0 removed, 0 content-changed, no metadata change, but 743 changed lines.

Cause

In src/osw/wtsite.py#L992-L1002 the bundle is assembled by walking added_titles and appending each page, followed by that page's discovered file attachments:

for page_title in added_titles:
    if page_title in page_dumps:
        bundle.packages[config.name].pages.append(page_dumps[page_title])
    if page_title in page_files:
        for file_title in page_files[page_title]:
            if file_title in file_dumps:
                bundle.packages[config.name].pages.append(file_dumps[file_title])

# bundle.packages[config.name].pages.sort(key=lambda x: x.urlPath)

added_titles follows the configured page_titles, so pages listed explicitly keep a stable order. The attachments in page_files are discovered rather than listed, and the pages they come from are fetched in parallel (wtsite.py#L446, auto-enabled above 5 titles, appending in completion order), so their order varies per run.

Confirmed against the data: NS_ITEM order is stable across builds, NS_FILE order is not.

Scope

Only packages that pull in auto-resolved attachments are affected. world.opensemantic.meta.docs has 66 such File: pages and churns on every build; world.opensemantic.core and world.opensemantic.base list every page explicitly, package no discovered attachments, and show a one-line diff for the same kind of release.

Note on the existing sort

Line 1002 already contains the fix, commented out. Enabling it as written would also reorder the NS_ITEM entries by urlPath and lose the current correspondence with page_titles, which is stable and readable today. Sorting only the attachments appended per page would remove the churn without changing the order of listed pages:

for file_title in sorted(page_files[page_title]):

Affects v2.3.2 and earlier.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions