[abhalsod-sonata]

Description

Very large assessment units, for example a 90-question mock exam where every question comes from a randomized question bank, can take longer to render in a single response than the request timeout allows, so the unit never loads.

This PR adds incremental loading for those units, behind a new course waffle flag, courseware.incremental_assessment_load (opt-in per course). With the flag on, a unit with more than INCREMENTAL_LOAD_PROBLEM_THRESHOLD (20) problems:

1. Renders its first INCREMENTAL_LOAD_EAGER_COUNT (5) children inline and leaves placeholders for the rest.
2. Fills the placeholders from the new GET /api/courseware/xblock_children/{usage_key}?child_usage_keys=... endpoint, in sequential batches of at most XBLOCK_CHILDREN_BATCH_MAX (10). The page shows "Loading question X of Y...", retries transient failures with backoff, halves a batch that times out, and offers a Retry button for any child that fails.

Units at or below the threshold, and requests from the mobile apps, always render in full.

The PR is four commits:

1. feat: incrementally load large assessment units: the flag, settings, endpoint, vertical shell and page script.
2. fix: hydrate learner state for incrementally loaded children: lazily loaded problems now show the learner's saved answer, score and attempts. Without this, an answered problem rendered as a fresh attempt and resubmitting it failed with "The state of this problem has changed since you loaded this page". One FieldDataCache is populated per batch (new FieldDataCache.add_block_descendents_batch), so a batch costs a single learner-state lookup.
3. fix: hydrate item bank problems and staff debug in lazily loaded units: the same fix for problems inside randomized banks (item_bank / library_content), which are loaded through the shared runtime the parent bound. It also makes Staff Debug Info and Submission History work for lazily loaded children, by inserting them with edx.HtmlUtils.setHtml so their inline setup scripts run, the same as an inline render.
4. docs: annotate incremental assessment load settings.

User roles impacted:

• Learner: large exams in courses with the flag on load reliably; the rest of the questions appear a moment after the page opens.
• Operator: a new opt-in course waffle flag and three settings, all annotated (lms/djangoapps/courseware/toggles.py, lms/envs/common.py).
• Developer: a new courseware API endpoint, xblock_children.

Before vs after

Measured on a local devstack with a 90-question unit, each question an item bank holding one problem, warm cache. The absolute numbers depend on the machine; the comparison is what matters.

	Before: flag off (current master behavior)	After: flag on
First response	6.0 to 6.3 s server time, 2.9 MB HTML, all 90 problems	2.6 to 2.7 s server time, 241 KB HTML, 5 problems and 85 placeholders
Follow-up requests	None	9 batch requests of up to 10 children, each 1.1 s or less
All 90 questions on screen	About 8 s	About 10 s

The total time to show every question stays about the same. The difference is that no single request has to render the whole unit, and the learner can start right after the first response.

1. Before (flag off): the whole 90-question unit arrives in a single 2.9 MB response.

2. After (flag on): the page arrives with the first 5 questions; questions 6 to 90 are placeholders.

3. After: the rest load in batches while the status line reports progress. The batch responses were slowed down in the browser to capture this state.

4. After: Staff Debug Info on question 7, a lazily loaded problem inside an item bank, opens.

5. After: question 7 answered and the page reloaded: the saved answer, the score (1/1 point) and the attempts (1 of 1) are kept, and Submit stays disabled.

Supporting information

• feat: incrementally load large assessment xblocks edx/edx-platform#445: the original incremental load.
• feat: hydrate learner state for paginated assessment children edx/edx-platform#465: learner-state hydration for lazily loaded children.
• fix: LP-1397 hydrate item bank problems and staff debug in lazily loaded units edx/edx-platform#529: hydration for problems inside randomized banks, plus Staff Debug Info and Submission History.

The original feature and the first fix were written by Santhosh Kumar; this PR ports them, with the follow-up fix, onto master.

Testing instructions

1. Create a course with a unit that has more than 20 problems. To cover the bank case, make each question a randomized content block (item bank) that picks one problem.
2. With the flag off, open the unit as a learner: it renders in full, as before.
3. In Django admin, add a waffle flag course override for courseware.incremental_assessment_load, forced on, for that course.
4. Reload the unit:
   • The first 5 questions arrive with the page.
   • The rest show "Loading question X of Y..." and fill in.
   • The network tab shows GET /api/courseware/xblock_children/<unit key>/?child_usage_keys=... requests with at most 10 keys each.
5. Answer a question after Q5 and refresh the page: the answer, score and attempts are still shown, and submitting again does not show "The state of this problem has changed...".
6. As course staff, click Staff Debug Info and Submission history under a question after Q5: both open.
7. Units with 20 or fewer problems, and mobile app requests, still render in full with the flag on.

Automated tests:

• lms/djangoapps/courseware/tests/test_incremental_render.py: the threshold and flag decision, the shell split, the endpoint helpers.
• openedx/core/djangoapps/courseware_api/tests/test_views.py (XBlockChildrenApiTestViews):
  • batch rendering, auth, 400 cases and partial failure;
  • the security boundary: only the parent's own, learner-visible children are served;
  • persisted state for plain problems and for problems inside an item bank.
• lms/djangoapps/courseware/tests/test_model_data.py: add_block_descendents_batch populates the cache once per batch.

Deadline

None.

Other information

• No migrations. The new endpoint uses JWT or session authentication and requires an enrolled learner with access to the parent unit. It only serves the parent's own children that are visible to that learner. For a randomized bank that means only the learner's assigned problems, never the whole pool.
• Security: child HTML comes from this LMS's own render of the same blocks the eager path inlines. The page script inserts it with edx.HtmlUtils.setHtml(el, edx.HtmlUtils.HTML(html)), the same idiom capa's display.js uses for server-rendered problem HTML.
• Not included: the downstream prefetch narrowing for randomized banks (edx/edx-platform@04e7ed024f). Without it, a batch prefetches each bank's whole pool, which is what master does today. It is independent of this feature and can be proposed separately.
• Locally verified with Python 3.12 and the uv.lock dependencies:
  • Tests: test_incremental_render.py, courseware_api/tests/test_views.py, test_model_data.py, xmodule/tests/test_vertical.py,
    lms/djangoapps/courseware/tests/test_block_render.py and lms/djangoapps/courseware/tests/test_views.py (496 passed).
  • Checks: ruff check ., make xsslint, make pii_check, make check_keywords, make lint-imports, make check-types, and pylint on the changed files.