Skip to content

Reject a negative n in IterableDataset.skip and take - #8526

Open
LeSingh1 wants to merge 1 commit into
huggingface:mainfrom
LeSingh1:ds-skip-take-validate
Open

Reject a negative n in IterableDataset.skip and take#8526
LeSingh1 wants to merge 1 commit into
huggingface:mainfrom
LeSingh1:ds-skip-take-validate

Conversation

@LeSingh1

Copy link
Copy Markdown

The bug

IterableDataset.skip and take silently treat a negative n as zero:

it = Dataset.from_dict({"a": [1, 2, 3, 4]}).to_iterable_dataset()

list(it.skip(-1))   # [{'a': 1}, {'a': 2}, {'a': 3}, {'a': 4}]  -- the whole dataset
list(it.take(-1))   # []                                        -- nothing

Neither raises. A sign error in the caller's arithmetic — take(n - m) where m > n, say — silently produces a plausible-looking result, and in the skip case one that quietly contains everything.

The fix

Raise ValueError for a negative n, in the same style as the other argument checks on this class. n == 0 keeps its current meaning for both (skip nothing / take nothing), and the existing parametrized tests cover 0, 2 and 1e10 unchanged.

Verification

The new test fails on main and passes with the change. Full tests/test_iterable_dataset.py is 4 failed both before and after — all four are test_interleave_dataset_with_sharding, which fail identically without this change on my machine and are unrelated to it.

Both silently treated a negative n as zero: skip(-1) yielded the whole
dataset and take(-1) yielded nothing, so a sign error in the caller's
arithmetic produced a plausible-looking result with no warning.

Raise instead, in the same style as the other argument checks on this
class. n == 0 keeps its current meaning for both.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant