Skip to content

Improvement: Persistent Wikipedia Caching for faster startup - #14

Open
ssuppe wants to merge 3 commits into
stefan-kp:mainfrom
ssuppe:feature/wikipedia-cache-persistence
Open

Improvement: Persistent Wikipedia Caching for faster startup#14
ssuppe wants to merge 3 commits into
stefan-kp:mainfrom
ssuppe:feature/wikipedia-cache-persistence

Conversation

@ssuppe

@ssuppe ssuppe commented May 9, 2026

Copy link
Copy Markdown

Hello!

My name is Steve. I've been spending a lot of time with Chess Tutor lately and I really love what you've built. It’s a wonderful piece of software and it’s been a joy to use.

I’ve come across a few quality-of-life improvements that I found useful in my own setup and I would love to contribute them back to the project. Please know that it’s not my intent to make the project more complicated—I just wanted to share some things that made the experience a bit smoother for me.

I have a few different sets of changes ready (including some model migration updates and UI tweaks), but I thought I'd start with the Wikipedia caching. I'm happy to submit these all together as one large PR or keep them broken up into smaller, focused ones like this one—whichever is easiest for you to review and manage!

Feature: Wikipedia Cache Persistence

The Problem:
While the application has a massive database of over 12,000 openings, it fetches Wikipedia summaries for the major opening families (~70+) to provide educational context. Previously, these were being refetched frequently without long-term persistence, which occasionally triggered "Too Many Requests" (429) errors from Wikipedia and added unnecessary overhead to the startup process.

The Solution:
I've implemented a persistent caching system so these summaries are fetched once and stored locally.

Key Improvements:

  • Persistence: Wikipedia data is now saved to a local directory (public/wikipedia). I've also updated the Docker configuration so this folder is preserved across container restarts via a volume.
  • Efficiency: The system now checks for existing files before downloading. If the data is already there, it skips the network request entirely.
  • Robustness: Added exponential backoff and rate-limiting to the fetcher to ensure we remain respectful of Wikipedia’s API.
  • Manual Control: Added "Clear Cache" and "Rebuild Cache" options in the Settings menu, along with a loading indicator to show progress.
  • Safety: Implemented a file-based lock system to prevent multiple rebuild processes from running simultaneously.

I've also included a suite of tests to ensure the cache management remains stable.

Thank you again for creating such a great tool! Let me know if you have any questions or if you'd like me to adjust the way I'm submitting these.

Best,
Steve

@ssuppe

ssuppe commented May 9, 2026

Copy link
Copy Markdown
Author

Just a quick note to help with the review: although there are 12 files, a significant portion of the changes are unit tests and translations to ensure the new caching logic is stable and fully localized!

@ssuppe ssuppe closed this May 10, 2026
@ssuppe
ssuppe deleted the feature/wikipedia-cache-persistence branch May 10, 2026 12:06
@ssuppe
ssuppe restored the feature/wikipedia-cache-persistence branch May 10, 2026 12:12
@ssuppe

ssuppe commented May 10, 2026

Copy link
Copy Markdown
Author

Apologies, I tried to rename the branch but this broke the PR. I've restored the name and reopened.

@ssuppe ssuppe reopened this May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant