Automatically generating the screenshots and/or a combination of OCR would improve the setup for different HUD's, screens, graphics cards, brightness settings etc.
This will be a bit of a dev blog of my attempts to make this work.
Having played with pyteseract this weekend, OCR is not going to all the way. It struggles with some text written on a slant - the left/right HUD windows, and oddly it struggles with some of the dark text on light background images. For example, on my setup I can't get it to read "CARRIER SERVICES" when selected, or "INVENTORY" when selected, but it is able to read "TRITIUM DEPOT" and "CARRIER MANAGEMENT" when selected.
Inverting the text doesn't help. Select edges doesn't help. Normally it can recognise "INVENTORY" to 90% confidence, but mist rising behind the HUD when docked can reduce that. The text for "TRANSFER" it seems to only get with about 65% confidence.
All of these tests are on grayscale images. I've not had better luck with thresholding the image, either using OTSU or Gausian. Gausian should be better but with the default settings at least turns the text into outlines. Maybe because of the width. There is some promise there. OpenCV Thresholding
Overall I think OCR may have a place in helping to build screenshots, but the screenshots will be better for generally navigating the UI - unless I can come up with a better way of detecting the selected buttons text.
But, step 1:
- I can add a teseract windows build as a portable binary to the release
- Using a mixture of text recognition and known key sequences, slowing down the keypress timing as well, I can screenshot "CARRIER SERVICES", "CARRIER MANAGEMENT", "TRITIUM DEPOT", "INVENTORY" (selected and unselected) and "TRANSFER" images.
For the ICON images, hopefully either they work or they are in known positions on the screen.
This is in test code, I will look at add a UI function for these, which saves the images as "image/name99.png" so they wind up as part of the image search and continue refining additional UI elements.
Automatically generating the screenshots and/or a combination of OCR would improve the setup for different HUD's, screens, graphics cards, brightness settings etc.
This will be a bit of a dev blog of my attempts to make this work.
Having played with pyteseract this weekend, OCR is not going to all the way. It struggles with some text written on a slant - the left/right HUD windows, and oddly it struggles with some of the dark text on light background images. For example, on my setup I can't get it to read "CARRIER SERVICES" when selected, or "INVENTORY" when selected, but it is able to read "TRITIUM DEPOT" and "CARRIER MANAGEMENT" when selected.
Inverting the text doesn't help. Select edges doesn't help. Normally it can recognise "INVENTORY" to 90% confidence, but mist rising behind the HUD when docked can reduce that. The text for "TRANSFER" it seems to only get with about 65% confidence.
All of these tests are on grayscale images. I've not had better luck with thresholding the image, either using OTSU or Gausian. Gausian should be better but with the default settings at least turns the text into outlines. Maybe because of the width. There is some promise there. OpenCV Thresholding
Overall I think OCR may have a place in helping to build screenshots, but the screenshots will be better for generally navigating the UI - unless I can come up with a better way of detecting the selected buttons text.
But, step 1:
For the ICON images, hopefully either they work or they are in known positions on the screen.
This is in test code, I will look at add a UI function for these, which saves the images as "image/name99.png" so they wind up as part of the image search and continue refining additional UI elements.