Add HST bundle preview generation scripts and configuration files - #250
Add HST bundle preview generation scripts and configuration files#250juzen2003 wants to merge 1 commit into
Conversation
* Introduced hst_bundle_previews.sh for creating preview images from PDS4 HST bundle browse collections. * Added hst_bundle_browse_map.txt and hst_bundle_fits_map.txt to define browse and FITS fallback collection policies. * Included hst_bundle_fits_previews.txt for FITS image processing options. * Established structure for handling browse and FITS data collections in the preview generation process.
WalkthroughAdds a new HST preview-generation script (hst_bundle_previews.sh) along with supporting text configuration files: browse and FITS fallback mapping files defining per-collection include/skip policies and strip suffixes, and picmaker option files specifying output sizes for thumbnail, small, medium, and full previews. ChangesHST Bundle Previews
Estimated code review effort: 3 (Moderate) | ~25 minutes Related PRs: None identified. Suggested labels: enhancement, previews, HST Suggested reviewers: None identified. Poem 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@previews/HST/hst_bundle_previews.sh`:
- Around line 18-19: The script currently overwrites HOME with the current
directory, which changes the environment for picmaker and any subprocesses it
starts. Update hst_bundle_previews.sh to keep HOME unchanged and store the
caller working directory in a separate variable instead, using the shell
variable assignment near the top of the script.
- Around line 156-164: The picmaker call sites in hst_bundle_previews.sh only
stop on exit status 2, so other non-zero failures are being ignored. Update both
invocations around the preview generation blocks to capture picmaker’s exit code
and exit immediately for any non-zero status except the intended 2-case, using
the existing picmaker command and its surrounding if checks as the place to fix
this.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 874a5764-730c-41c6-9ac8-a918c93fb3b1
📒 Files selected for processing (5)
previews/HST/hst_bundle_browse_map.txtpreviews/HST/hst_bundle_fits_map.txtpreviews/HST/hst_bundle_fits_previews.txtpreviews/HST/hst_bundle_previews.shpreviews/HST/hst_bundle_previews.txt
| HOME=`pwd` | ||
|
|
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
sed -n '1,120p' previews/HST/hst_bundle_previews.shRepository: SETI/rms-data-projects
Length of output: 3203
🏁 Script executed:
rg -n '\bHOME\b|resolve_path|picmaker|export HOME' previews/HST/hst_bundle_previews.shRepository: SETI/rms-data-projects
Length of output: 412
🏁 Script executed:
bash -lc '
set -e
printf "parent: initial exported HOME? "; export -p | grep -q "declare -x HOME=" && echo yes || echo no
HOME="/tmp/fake-home"
printf "parent: after assignment exported HOME? "; export -p | grep -q "declare -x HOME=" && echo yes || echo no
printf "child sees HOME: "
bash -lc "printf %s \"\$HOME\""
printf "\n"
'Repository: SETI/rms-data-projects
Length of output: 469
Avoid overwriting HOME previews/HST/hst_bundle_previews.sh:18
HOME=\pwd`changes the exported home directory forpicmakerand anything it spawns. Use a separate variable for the caller cwd and leaveHOME` untouched.
🧰 Tools
🪛 Shellcheck (0.11.0)
[style] 18-18: Use $(...) notation instead of legacy backticks ....
(SC2006)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@previews/HST/hst_bundle_previews.sh` around lines 18 - 19, The script
currently overwrites HOME with the current directory, which changes the
environment for picmaker and any subprocesses it starts. Update
hst_bundle_previews.sh to keep HOME unchanged and store the caller working
directory in a separate variable instead, using the shell variable assignment
near the top of the script.
| picmaker "$data_dir" \ | ||
| --directory "$outpath/$data_collection" \ | ||
| --recursive --pattern \*.fits \ | ||
| --strip "$strip" --verbose=2 --proceed \ | ||
| --versions hst_bundle_fits_previews.txt | ||
|
|
||
| if [ "$?" = "2" ]; then | ||
| exit 2 | ||
| fi |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the relevant script and search for picmaker usage/definition/docs.
git ls-files | rg '^(previews/HST/hst_bundle_previews\.sh|.*picmaker.*|.*Picmaker.*|.*README.*|.*docs?.*)$' || true
echo "---- hst_bundle_previews.sh (around the cited lines) ----"
sed -n '130,210p' previews/HST/hst_bundle_previews.sh
echo "---- search for picmaker references ----"
rg -n --hidden --no-messages '\bpicmaker\b|exit code 2|proceed' .
echo "---- locate picmaker file(s) if present ----"
fd -a -t f -H 'picmaker*' .Repository: SETI/rms-data-projects
Length of output: 9358
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "---- HST preview script headers / shell options ----"
sed -n '1,80p' previews/HST/hst_bundle_previews.sh
echo "---- nearby HST preview scripts for exit handling patterns ----"
sed -n '1,90p' previews/HST/HSTO_previews.sh
printf '\n----\n'
sed -n '1,90p' previews/HST/HSTJ_previews.sh
printf '\n----\n'
sed -n '1,90p' previews/HST/HSTI_previews.sh
printf '\n----\n'
sed -n '1,90p' previews/HST/HSTN_previews.sh
printf '\n----\n'
sed -n '1,90p' previews/HST/HSTU_previews.sh
echo "---- repo docs mentioning picmaker or exit codes ----"
rg -n --hidden --no-messages 'picmaker|exit code 2|status 2|return 2|proceed' README.md previews vicar .github . || trueRepository: SETI/rms-data-projects
Length of output: 23647
🌐 Web query:
picmaker exit code 2 documentation preview generation
💡 Result:
There is no official documentation for a "Picmaker exit code 2" related to preview generation. Picmaker is a web-based graphic design and AI social media tool [1][2], and users typically interact with it through a browser interface rather than a command-line environment that would return specific exit codes [3][4]. If you are encountering an "exit code 2" error in the context of "preview generation," it is highly likely that this is related to a different software system, as this is a common generic error code across various platforms [5]. The following systems are often associated with preview generation issues and exit codes: 1. Nextcloud: The Nextcloud "previewgenerator" app frequently encounters errors during background preview generation tasks [6][7]. Users often debug these issues using the occ command-line tool [7][8]. 2. Ruby on Rails (ActiveStorage): Modern versions of Rails (7+) introduced a PreviewError when previewer child processes (such as those using the Poppler library) fail to generate a preview, often resulting in non-zero exit statuses [9][10]. 3. Compilation/Build Tools: Exit code 2 in many development environments (such as VEXcode or standard build systems) typically indicates a generic compilation or build failure [5]. If you are using a tool other than Picmaker, please verify the software name or check your system's error logs for more specific information. If you are experiencing issues with Picmaker, it is recommended to contact their official support team through their help center [11][4].
Citations:
- 1: https://www.picmaker.com/ai-social-media-post-generator
- 2: https://www.picmaker.com/
- 3: https://www.picmaker.com/tutorials/get-started-with-picmaker/
- 4: https://web.picmaker.com/features
- 5: https://kb.vex.com/hc/en-us/articles/360039405272-Troubleshooting-Exit-Code-2-on-VEXcode-Pro-V5
- 6: Preview is not generating for .jpg and .mp4 nextcloud/previewgenerator#593
- 7: preview:generate-all crashses without an error nextcloud/previewgenerator#641
- 8: preview generation fails nextcloud/previewgenerator#408
- 9: https://blog.saeloun.com/2021/08/17/active-storage-preview-error/
- 10: https://www.mintbit.com/blog/previewerror-raised-rails7/
- 11: https://support.picmaker.com/en/articles/4732330-picmaker-1-0-to-2-0-what-should-i-be-aware-of
Propagate non-2 picmaker failures previews/HST/hst_bundle_previews.sh:156-164,188-197 — both call sites only special-case exit 2; every other non-zero picmaker status is ignored, so preview generation can still succeed after renderer or input failures.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@previews/HST/hst_bundle_previews.sh` around lines 156 - 164, The picmaker
call sites in hst_bundle_previews.sh only stop on exit status 2, so other
non-zero failures are being ignored. Update both invocations around the preview
generation blocks to capture picmaker’s exit code and exit immediately for any
non-zero status except the intended 2-case, using the existing picmaker command
and its surrounding if checks as the place to fix this.
1 participant
Here is the current logic flow and some questions:
- Question:
- for the bundle (like 16167) that has multiple browse directories (browse_wfc3_{raw,flt,flc,drz,drc}), do we apply picmaker to all of them? Or do we apply only to drz like PDS3 HSTI (raw tiff or drz jpeg, and since we only download jpg, jpeg, and png from MAST for browse, we won't have tiff file, we apply drz in this case)?
- for the case like 13736, it has browse_stis_x1d, but it also has data_stis_x2d, should we apply picmaker on fits in data_stis_x2d instead of applying picmaker to resize files in browse_stis_x1d (becasue PDS3 HSTO apply picmaker on x2d jpeg)?
Question: