task-based execution: add option to increase max_tries

[atravitz]

Description

resolves #2214

LLM / AI generated code disclosure

LLMs or other AI-powered tools (beyond simple IDE use cases) were used in this contribution: no

Checklist

• All new code is appropriately documented (user-facing code must have complete docstrings).
• Added a news entry, or the changes are not user-facing.
• Ran pre-commit: you can run pre-commit locally or comment on this PR with pre-commit.ci autofix.
• Adhered to the contribution guidelines
• Filled in the AI generated code disclosure.

Manual Tests: these are slow so don't need to be run every commit, only before merging and when relevant changes are made (generally at reviewer-discretion).

• GPU integration tests
• example notebook testing
• packaging tests: run this for any large feature PRs or PRs that add test data.

Developers certificate of origin

• I certify that this contribution is covered by the MIT License here and the Developer Certificate of Origin at https://developercertificate.org/.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.42%. Comparing base (b917eda) to head (11b77b4).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2227      +/-   ##
==========================================
- Coverage   95.13%   90.42%   -4.71%     
==========================================
  Files         224      226       +2     
  Lines       22009    22198     +189     
==========================================
- Hits        20938    20073     -865     
- Misses       1071     2125    +1054     

Flag	Coverage Δ
fast-tests	90.42% <100.00%> (?)
slow-tests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[IAlibay]

Overall looks good - maybe a quick line or two in the userguide on how to use this would be useful.

[atravitz]

Overall looks good - maybe a quick line or two in the userguide on how to use this would be useful.

will add here once this PR gets merged: OpenFreeEnergy/ExampleNotebooks#322

[IAlibay]

Couple things, please feel free to ignore the nits.