feat(activesync): persist folder UID map across FolderSync resets

doc/todo.md

@@ -127,51 +127,33 @@ when a migration plan is written.
   to stop converting between them at runtime.
 - Consolidate folder UID ↔ backend ID mapping (today split across
   `Horde_ActiveSync_Collections` and the driver).
-- **Folder UID map vs volatile folder cache (long-term)**
-
-  When the folder hierarchy changes (e.g. multiplexed Turba address books
-  added/removed from ActiveSync), clients may issue concurrent EAS sessions.
-  `FolderSync` with `synckey=0` clears the per-device folder cache
-  (`State/Base.php` → `_resetDeviceState()` → `SyncCache::clearFolders()`).
-  The backend-id → EAS-folder-UID map lives in that cache
-  (`getFolderUidToBackendIdMap()`), so concurrent rebuilds can race on an
-  empty map while `FolderSync` is still exporting.
-
-  #81 (2026-07-09) mitigates this with deterministic UID generation when the
-  map is empty; see **Recently completed**. The structural issue remains:
-  folder identity should not depend on a cache that is wiped mid-rebuild.
-
-  **Long-term direction:**
-
-  1. **Persist UID assignments separately** — e.g. `(device_id, user,
-     backend_serverid)` → `eas_folder_uid`, updated on first assignment and
-     rename (`old_id` in `_getFolderUidForBackendId()`), not cleared by
-     `clearFolders()`. Hierarchy diff drives FolderSync Add/Update/Remove;
-     drop map entries only when the backend folder is gone.
-
-  2. **Or: atomic folder-cache rebuild** — build the new hierarchy in a
-     staging structure, swap in one `SyncCache::save()`; no empty-map window
-     visible to parallel SYNC/PING/FolderSync requests.
-
-  3. **Fold into Horde 6 storage refactor** — overlaps “Unify serverid vs
-     backend folder names” and “Fold SyncCache into device object” above.
-
-  **Key code paths:** `Driver/Base.php` (`_getFolderUidForBackendId`,
-  `_tempMap`), `State/Base.php` (`getFolderUidToBackendIdMap`, `loadState`
-  synckey `0`), `SyncCache.php` (`clearFolders`, `updateFolder`),
-  `Collections.php` (`getBackendIdForFolderUid`, `initHierarchySync`, `save`),
-  `Request/FolderSync.php`.
-
-  **Done when:** concurrent `FolderSync` `synckey=0` after a hierarchy
-  change yields one consistent saved UID map; devices converge without account
-  removal; renames preserve UID. Do not drop #81 deterministic generation
-  until (1) or (2) is in place.
-
-  **After structural fix:** Prefer switching back to opaque (e.g. random)
-  UIDs for *new* map entries. Deterministic derivation was only needed so
-  parallel rebuilds agreed while the map was empty; a persisted map assigns
-  each backend folder once and removes that race. Opaque IDs also avoid
-  leaking backend folder identity via a predictable `crc32(prefix:id)` scheme.
+- **Folder UID map vs volatile folder cache**
+
+  **Done:** Persistent SyncCache `foldermap` (`backend_serverid` →
+  `eas_folder_uid`) survives `clearFolders()`. FolderSync `synckey=0` no longer
+  clears+saves an empty folders map mid-rebuild (Sql/Mongo `_resetDeviceState`);
+  the live hierarchy is reconciled via `reconcileFolders()` after export.
+  `getFolderUidToBackendIdMap()` prefers `foldermap`; Collections resolve UIDs
+  via foldermap fallback when the folders cache entry is missing. Renames update
+  foldermap in place; deletes prune map entries.
+
+  Deterministic UID generation (#81) remains as a safety net when the map is
+  empty (first sync / upgrade seed from `folders`). Opaque random UIDs for *new*
+  map entries can replace deterministic derivation in a follow-up once foldermap
+  has baked in production.
+
+  **Optional follow-ups:**
+
+  1. Switch new map entries to opaque (random) UIDs once foldermap is ubiquitous.
+  2. Stronger SyncCache CAS / partial-field save to reduce last-write-wins under
+     concurrent non-FolderSync writers.
+  3. Fold into Horde 6 storage refactor — overlaps “Unify serverid vs backend
+     folder names” and “Fold SyncCache into device object”.
+
+  **Key code paths:** `SyncCache.php` (`foldermap`, `reconcileFolders`,
+  `ensureFolderMap`), `State/{Base,Sql,Mongo}.php` (`getFolderUidToBackendIdMap`,
+  `_resetDeviceState`), `Collections.php`, `Request/FolderSync.php`.
+
 - Pass `FILTERTYPE_*` to the driver by constant, not precomputed cutoff
   timestamps (supersedes the near-term `INCOMPLETETASKS` fix style).
 - Split `getMessage()` into per-class methods with shared base logic.
@@ -282,8 +264,9 @@ active backlog; kept here so this file does not resurrect settled work.
 
 ### Stability (3.0.0-RC1 and related)
 
-- Deterministic EAS folder UIDs on cache rebuild (#81, 2026-07-09); see
-  “Folder UID map vs volatile folder cache” for the remaining structural work.
+- Deterministic EAS folder UIDs on cache rebuild (#81, 2026-07-09).
+- Persistent SyncCache `foldermap` + FolderSync reconcile (see
+  “Folder UID map vs volatile folder cache”).
 - `FILTERTYPE_INCOMPLETETASKS`: pass FilterType to the driver; incomplete-only
   task sync in `horde/core` / `horde/nag`; no longer encode filter `8` as a
   Unix cutoff (avoids mail/calendar misconfiguration).

lib/Horde/ActiveSync/Collections.php

@@ -374,10 +374,24 @@
         $folder = $this->_cache->getFolder($folderid);
         if ($folder) {
             return $folder['serverid'];
-        } else {
-            $this->_logger->err('COLLECTIONS: Horde_ActiveSync_Collections::getBackendIdForFolderUid failed because folder was not found in cache.');
-            throw new Horde_ActiveSync_Exception_FolderGone('Folder not found in cache.');
         }
+
+        // Fall back to the persistent foldermap. FolderSync synckey=0 used to
+        // clear+save an empty folders cache, leaving concurrent SYNC/PING
+        // unable to resolve known UIDs. foldermap survives that window.
+        $map = $this->_as->state->getFolderUidToBackendIdMap();
+        $backendId = array_search($folderid, $map, true);
+        if ($backendId !== false) {
+            $this->_logger->meta(sprintf(
+                'COLLECTIONS: Resolved folder uid %s via persistent foldermap to %s',
+                $folderid,
+                $backendId
+            ));
+            return $backendId;
+        }
+
+        $this->_logger->err('COLLECTIONS: Horde_ActiveSync_Collections::getBackendIdForFolderUid failed because folder was not found in cache.');
+        throw new Horde_ActiveSync_Exception_FolderGone('Folder not found in cache.');
     }
 
     /**
@@ -737,6 +751,9 @@
         $update = false
     ) {
         $this->_cache->updateFolder($folder);
+        if (!empty($folder->_serverid) && !empty($folder->serverid)) {
+            $this->_as->state->rememberFolderUid($folder->_serverid, $folder->serverid);
+        }
         $cols = $this->_cache->getCollections(false);
         $cols[$folder->serverid]['serverid'] = $folder->_serverid;
         $this->_cache->updateCollection($cols[$folder->serverid]);
@@ -753,12 +770,29 @@
     public function deleteFolderFromHierarchy($uid)
     {
         $this->_cache->deleteFolder($uid);
+        $this->_as->state->resetFolderUidMap();
         $this->_as->state->removeState([
             'id' => $uid,
             'devId' => $this->_as->device->id,
             'user' => $this->_as->device->user]);
     }
 
+    /**
+     * After FolderSync synckey=0, drop folders that are no longer live.
+     *
+     * Folders are intentionally kept during the reset window; reconcile once
+     * the new hierarchy UIDs are known.
+     *
+     * @param array $liveFolderUids  EAS folder uids present after export.
+     *
+     * @since 3.0.3
+     */
+    public function reconcileHierarchyFolders(array $liveFolderUids)
+    {
+        $this->_cache->reconcileFolders($liveFolderUids);
+        $this->_as->state->resetFolderUidMap();
+    }
+
     /**
      * Return all know hierarchy changes.
      *
@@ -788,9 +822,9 @@
             if (isset($folder->serverid)
                 && $syncFolder = $this->_cache->getFolder($folder->serverid)
                 && in_array($folder->serverid, $seenFolders)
                 && $syncFolder['parentid'] == $folder->parentid
                 && $syncFolder['displayname'] == $folder->displayname
                 && $syncFolder['type'] == $folder->type) {
 
                 $this->_logger->meta(
                     sprintf(
@@ -910,8 +944,8 @@
             $this->save();
             return false;
         }
         $this->shortSyncRequest = true;
         $this->hangingSync = true;
         $this->save(true);
 
         return true;
@@ -1273,7 +1307,7 @@
 
         // Need to update AND SAVE the timestamp for race conditions to be
         // detected.
         $this->lasthbsyncstarted = $started;
         $this->save();
 
         // We only check for remote wipe request once every 5 iterations to
@@ -1537,7 +1571,7 @@
             sprintf(
                 'COLLECTIONS: Looping Sync complete: DataAvailable: %s, DataImported: %s',
                 $dataavailable,
                 $this->importedChanges
             )
         );
 

lib/Horde/ActiveSync/Request/FolderSync.php

@@ -88,6 +88,7 @@ protected function _handle()
             $this->_handleError();
             return true;
         }
+        $initialHierarchySync = empty($synckey) || $synckey === '0';
 
         // Prepare the collections handler.
         $collections = $this->_activeSync->getCollectionsObject();
@@ -241,6 +242,13 @@ protected function _handle()
         }
         $this->_cleanUpAfterPairing();
 
+        // Folders were kept across synckey=0 reset to avoid an empty-map race;
+        // drop any that are no longer in the live hierarchy. Skip when the live
+        // set is empty so a transient empty getFolderList() cannot wipe the map.
+        if ($initialHierarchySync && !empty($seenfolders)) {
+            $collections->reconcileHierarchyFolders($seenfolders);
+        }
+
         $collections->save();
 
         return true;

lib/Horde/ActiveSync/State/Base.php

@@ -485,6 +485,10 @@ public function getKnownFolders()
     /**
      * Return the mapping of folder uids to backend folderids.
      *
+     * Prefers the persistent SyncCache foldermap (survives clearFolders /
+     * FolderSync synckey=0). Falls back to deriving the map from the folders
+     * cache for caches that predate foldermap.
+     *
      * @return array  An array of backend folderids -> uids.
      * @since 2.9.0
      */
@@ -495,16 +499,60 @@ public function getFolderUidToBackendIdMap()
             $cache = $this->getSyncCache(
                 $this->_deviceInfo->id,
                 $this->_deviceInfo->user,
-                ['folders']
+                ['foldermap', 'folders']
             );
-            foreach ($cache['folders'] as $id => $folder) {
-                $this->_folderUidMap[$folder['serverid']] = $id;
+            if (!empty($cache['foldermap']) && is_array($cache['foldermap'])) {
+                $this->_folderUidMap = $cache['foldermap'];
+            } elseif (!empty($cache['folders']) && is_array($cache['folders'])) {
+                foreach ($cache['folders'] as $id => $folder) {
+                    if (!empty($folder['serverid'])) {
+                        $this->_folderUidMap[$folder['serverid']] = $id;
+                    }
+                }
             }
         }
 
         return $this->_folderUidMap;
     }
 
+    /**
+     * Remember a backend-id → EAS-uid assignment for this request.
+     *
+     * Keeps the in-memory map coherent while FolderSync rebuilds folders.
+     * Persistence happens via SyncCache::updateFolder() / setFolderMapEntry().
+     *
+     * @param string $backendId  Backend folder id.
+     * @param string $uid        EAS folder uid.
+     *
+     * @since 3.0.3
+     */
+    public function rememberFolderUid($backendId, $uid)
+    {
+        if ($backendId === '' || $backendId === null || $uid === '' || $uid === null) {
+            return;
+        }
+        if (!isset($this->_folderUidMap)) {
+            $this->getFolderUidToBackendIdMap();
+        }
+        foreach ($this->_folderUidMap as $existingBackendId => $existingUid) {
+            if ($existingUid === $uid && $existingBackendId !== $backendId) {
+                unset($this->_folderUidMap[$existingBackendId]);
+            }
+        }
+        $this->_folderUidMap[$backendId] = $uid;
+    }
+
+    /**
+     * Drop the request-local folder UID map so the next lookup reloads
+     * from SyncCache storage.
+     *
+     * @since 3.0.3
+     */
+    public function resetFolderUidMap()
+    {
+        unset($this->_folderUidMap);
+    }
+
     /**
      * Get a EAS Folder Uid for the given backend server id.
      *

lib/Horde/ActiveSync/State/Mongo.php

@@ -1474,12 +1474,21 @@ protected function _resetDeviceState($id)
         if ($id != Horde_ActiveSync::REQUEST_TYPE_FOLDERSYNC) {
             $cache->removeCollection($id, false);
         } else {
-            $this->_logger->notice(' Clearing foldersync state from synccache.');
-            $cache->clearFolders();
+            // Do not clearFolders() here. Clearing and saving an empty folders
+            // map opens a race where concurrent SYNC/PING see FolderGone for
+            // every collection until this FolderSync finishes rebuilding.
+            // Hierarchy diff uses State::_folder (reset separately); SyncCache
+            // folders + foldermap stay available and are reconciled to the live
+            // set at the end of FolderSync (synckey=0).
+            //
+            // @author Torben Dannhauer <torben@dannhauer.de>
+            $this->_logger->notice(' Clearing foldersync collections/hierarchy from synccache (keeping folders + foldermap).');
+            $cache->ensureFolderMap();
             $cache->clearCollections();
             $cache->hierarchy = '0';
         }
         $cache->save();
+        $this->resetFolderUidMap();
     }
 
     /**
@@ -1772,6 +1781,7 @@ public function getSyncCache($devid, $user, ?array $fields = null)
                 'wait' => false,
                 'hbinterval' => false,
                 'folders' => [],
+                'foldermap' => [],
                 'hierarchy' => false,
                 'collections' => [],
                 'pingheartbeat' => false];

lib/Horde/ActiveSync/State/Sql.php

@@ -1757,6 +1757,7 @@ public function getSyncCache($devid, $user, ?array $fields = null)
                 'wait' => false,
                 'hbinterval' => false,
                 'folders' => [],
+                'foldermap' => [],
                 'hierarchy' => false,
                 'collections' => [],
                 'pingheartbeat' => false,
@@ -2172,12 +2173,21 @@ protected function _resetDeviceState($id)
         if ($id != Horde_ActiveSync::REQUEST_TYPE_FOLDERSYNC) {
             $cache->removeCollection($id, false);
         } else {
-            $this->_logger->notice('STATE: Clearing foldersync state from synccache.');
-            $cache->clearFolders();
+            // Do not clearFolders() here. Clearing and saving an empty folders
+            // map opens a race where concurrent SYNC/PING see FolderGone for
+            // every collection until this FolderSync finishes rebuilding.
+            // Hierarchy diff uses State::_folder (reset separately); SyncCache
+            // folders + foldermap stay available and are reconciled to the live
+            // set at the end of FolderSync (synckey=0).
+            //
+            // @author Torben Dannhauer <torben@dannhauer.de>
+            $this->_logger->notice('STATE: Clearing foldersync collections/hierarchy from synccache (keeping folders + foldermap).');
+            $cache->ensureFolderMap();
             $cache->clearCollections();
             $cache->hierarchy = '0';
         }
         $cache->save();
+        $this->resetFolderUidMap();
     }
 
     /**