At a configurable interval, perform the following tasks:
- Remove the oldest posts of each feed, retaining either
n most recent or all up to d duration old. This is to limit the volume of data stored in the DB at any given time and account for the fact that most use cases will only have feeds' most recent content ever viewed, so one-off retrospectives shouldn't permanently increase the cache size.
- Refresh the data of all posts by directly invoking the provider for content until the oldest piece of content should have been reached. If any piece of content exists in the cache but is no longer provided by the API, consider it deleted and remove from the cache. If any piece of content has its fields updated, update it in the cache. If any new content appears in the middle of the cache, add it (e.g. what would happen if an old post is archived, then unarchived).
Old post cleanup is doable in most databases natively, most likely, but ideally the database instances can just be spun up and OneFeed will handle all the extra stuff for the sake of consistency across database providers.
At a configurable interval, perform the following tasks:
nmost recent or all up todduration old. This is to limit the volume of data stored in the DB at any given time and account for the fact that most use cases will only have feeds' most recent content ever viewed, so one-off retrospectives shouldn't permanently increase the cache size.Old post cleanup is doable in most databases natively, most likely, but ideally the database instances can just be spun up and OneFeed will handle all the extra stuff for the sake of consistency across database providers.