CWL: Fix step input valueFrom with loadContents: true

[CGuichard]

This PR implement the resolution of the issue #5568.

Changelog Entry

CWL

• Fix failing read of toildir in ValueFrom.eval_prep when loadContents: true (CWL step input errors with valueFrom and loadContents: true #5568)
• Load only the concerned step input contents when loadContents: true (CWL step input errors with valueFrom and loadContents: true #5568)
• Support step input loadContents: true for File array (CWL step input errors with valueFrom and loadContents: true #5568)

Reviewer Checklist

• Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
  • If it is coming from an external repo, make sure to pull it in for CI with:

    contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    

  • If there is no associated issue, create one.
• Read through the code changes. Make sure that it doesn't have:
  • Addition of trailing whitespace.
  • New variable or member names in camelCase that want to be in snake_case.
  • New functions without type hints.
  • New functions or classes without informative docstrings.
  • Changes to semantics not reflected in the relevant docstrings.
  • New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
  • New features without tests.
• Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
• Finish the review with an overall description of your opinion.

Merger Checklist

• Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.
• Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.
• Merge with the Github "Squash and merge" feature.
  • If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
• Copy its recommended changelog entry to the Draft Changelog.
• Append the issue number in parentheses to the changelog entry.

[adamnovak]

This looks like it should work.

It seems weird that we only support a file or an array of files, and not, say, an array of arrays of files, but that's indeed what the spec says about loadContents in 1.2:

Only valid when type: File or is an array of items: File.

[adamnovak]

This isn't actually true, as far as I can tell. It looks like this always gets called, and is itself responsible for checking if loadContents is set.

Suggested change

	Called when loadContents is specified on the input associated with this
	ValueFrom.
	Responsible for checking loadContents.

[CGuichard]

I updated the docstring

[adamnovak]

The lint tests failed because something is wrong with the type annotations.

In a Python virtual environment, after make prepare && make develop extras='[all]', I can replicate the problem:

(venv) [anovak@swords toil]% make mypy
MYPYPATH=/Users/anovak/workspace/toil/contrib/mypy-stubs mypy --strict /Users/anovak/workspace/toil/src/toil/{cwl/cwltoil.py,test/cwl/cwlTest.py}
src/toil/cwl/cwltoil.py: note: In member "eval_prep" of class "ValueFrom":
src/toil/cwl/cwltoil.py:654:19: error: Redundant cast to "MutableMapping[str, CWLOutputType | None]"  [redundant-cast]
                val = cast(CWLObjectType, v)
                      ^
src/toil/cwl/cwltoil.py: note: In function "resolve_dict_w_promises":
src/toil/cwl/cwltoil.py:775:29: error: Argument 1 to "eval_prep" of "ValueFrom" has incompatible type "CWLOutputType | None"; expected "MutableMapping[str, CWLOutputType | None]"  [arg-type]
                    v.eval_prep(first_pass_results[k], file_store)
                                ^~~~~~~~~~~~~~~~~~~~~
Found 2 errors in 1 file (checked 2 source files)
make: *** [mypy] Error 1

I think the problem is we no longer need to cast v to val (so we can just use v), and also that now eval_prep should take a CWLOutputType | None and deal with the case where the input value it received is None.

[CGuichard]

The lint tests failed because something is wrong with the type annotations.

In a Python virtual environment, after make prepare && make develop extras='[all]', I can replicate the problem:

(venv) [anovak@swords toil]% make mypy
MYPYPATH=/Users/anovak/workspace/toil/contrib/mypy-stubs mypy --strict /Users/anovak/workspace/toil/src/toil/{cwl/cwltoil.py,test/cwl/cwlTest.py}
src/toil/cwl/cwltoil.py: note: In member "eval_prep" of class "ValueFrom":
src/toil/cwl/cwltoil.py:654:19: error: Redundant cast to "MutableMapping[str, CWLOutputType | None]"  [redundant-cast]
                val = cast(CWLObjectType, v)
                      ^
src/toil/cwl/cwltoil.py: note: In function "resolve_dict_w_promises":
src/toil/cwl/cwltoil.py:775:29: error: Argument 1 to "eval_prep" of "ValueFrom" has incompatible type "CWLOutputType | None"; expected "MutableMapping[str, CWLOutputType | None]"  [arg-type]
                    v.eval_prep(first_pass_results[k], file_store)
                                ^~~~~~~~~~~~~~~~~~~~~
Found 2 errors in 1 file (checked 2 source files)
make: *** [mypy] Error 1

I think the problem is we no longer need to cast v to val (so we can just use v), and also that now eval_prep should take a CWLOutputType | None and deal with the case where the input value it received is None.

Thanks for your review, these mypy issues were fixed.

I think instructions could be added to the docs contributing section for running lint tools like mypy, currently the contributing section linked by the "CONTRIBUTING.md" talks about running tests, naming conventions for commit/branch, but not lint unless I missed it.