Skip to content

fix(ops): make IdxMax and IdxMin skip NaN inside the window - #2353

Open
Ludwig J. Marx (LudwigJMarx) wants to merge 2 commits into
microsoft:mainfrom
LudwigJMarx:fix/idxmax-idxmin-nan
Open

Ludwig J. Marx (LudwigJMarx) wants to merge 2 commits into
microsoft:mainfrom
LudwigJMarx:fix/idxmax-idxmin-nan

Conversation

@LudwigJMarx

Copy link
Copy Markdown

Description

IdxMax and IdxMin used ndarray.argmax / ndarray.argmin, which return
the position of a NaN rather than skipping it. Replaced with np.nanargmax /
np.nanargmin, four lines.

Motivation and Context

Refs #2352.

Max and Min skip NaN through pandas, IdxMax and IdxMin did not, so the
two disagreed on the same series. Mad and Rank in the same file already
filter NaN explicitly.

Alpha158 builds IMAX{d}, IMIN{d} and IMXD{d} on these operators over
$high and $low, which are NaN while an instrument is suspended.

No guard against ValueError: All-NaN slice encountered is needed, and that is
measured rather than assumed: with min_periods=1 pandas does not call apply
at all for a window that holds only NaN. Checked for both the rolling and the
expanding path. The min_periods=1 sits on the same line as the apply
call, so the assumption is local.

Rejected: filtering NaN out first, the way Mad and Rank do. The returned
position would then be the one inside the filtered array rather than inside the
window, which is exactly the quantity the field means (days since the high).

How Has This Been Tested?

  • Pass the test by running: pytest qlib/tests/test_all_pipeline.py under upper directory of qlib.
  • If you are adding a new feature, test on your own test scripts.

tests/test_all_pipeline.py is the one that matters here: it runs
DATASET_ALPHA158_CLASS over csi300 from 2008 to 2020, so it exercises the
fields this change touches, and test_1_backtest checks a real threshold.
It carries pytest.mark.slow, so CI skips it. Ran both sides:

result
before the fix, on 7bd57c18 3 passed in 357 s
after the fix, on 4f5941e0 3 passed in 258 s

The runtimes differ because GBDT_MODEL sets no seed and uses
num_threads: 20. The test checks a threshold, not equality.

New file tests/ops/test_rolling_operator.py. It asserts no invented number
but the invariant against the real sibling operators: IdxMax has to point at
the value Max reports. The two commits carry the evidence: the test is red on
7bd57c18 and green on 4f5941e0.

Before the fix:

FAILED ops/test_rolling_operator.py::TestIdxRollingOperator::test_idxmax_points_at_the_maximum
FAILED ops/test_rolling_operator.py::TestIdxRollingOperator::test_idxmin_points_at_the_minimum
E   AssertionError: np.float64(nan) != np.float64(5.0) : IdxMax points at nan but Max reports 5.0
E   AssertionError: np.float64(nan) != np.float64(1.0) : IdxMin points at nan but Min reports 1.0
2 failed in 2.07s

After the fix:

2 passed in 2.21s

Rest of the suite, same command as CI (cd tests && pytest . -m "not slow"):

result
before, untouched tree 71 passed, 1 skipped, 10 deselected, 79 subtests, 818 s
after 73 passed, 1 skipped, 10 deselected, 79 subtests, 718 s

The two extra are the new tests.

Counter-check that nothing else moves: 200000 random windows without NaN, ties
forced, argmax against nanargmax and argmin against nanargmin, zero
differences.

black -l 120 --check clean, flake8 clean, pylint 10.00/10 on
qlib/data/ops.py.

Environment: Python 3.12.13, numpy 2.5.3, pandas 2.3.3, MacOS 26.6.2 arm64,
commit be72549.

Types of changes

  • Fix bugs
  • Add new feature
  • Update documentation

IdxMax and IdxMin return the position of the extreme value inside the
rolling window, but nothing held them to the value Max and Min report for
the same window. The new test states that invariant instead of an invented
number, so it stays meaningful if the implementation changes.

The leaf feature only replaces the data access; the operators under test
run their real code.

It fails on this commit: with [1.0, 5.0, nan, 2.0, 3.0] and window 3, every
window holding a NaN makes IdxMax point at the NaN while Max reports the
real maximum.

Refs microsoft#2352
ndarray.argmax and ndarray.argmin do not skip NaN, they return its
position. Max and Min go through pandas and do skip it, so the two
contradicted each other on the same series, and Mad and Rank in the same
file already filter NaN explicitly.

Alpha158 builds IMAX, IMIN and IMXD on these two operators over $high and
$low, which are NaN while an instrument is suspended. On csi300 over 2019,
30 instruments, every one of them holds at least one NaN in $high, and
IMAX5 differs in 3.06 percent of its values.

No guard against ValueError: All-NaN slice encountered is needed, and that
is measured rather than assumed: with min_periods=1 pandas does not call
apply at all for a window holding only NaN, on the rolling and on the
expanding path. The min_periods=1 sits on the same line as the apply call.

Filtering NaN out first, the way Mad and Rank do, was rejected: the
returned position would be the one inside the filtered array rather than
inside the window, which is exactly what the field means.

Refs microsoft#2352
@LudwigJMarx

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant