GH-51301: Fix deprecation warnings in the tests with numpy/pandas nightly by AlenkaF · Pull Request #51404 · apache/arrow

AlenkaF · 2026-09-21T10:24:44Z

Rationale for this change

Test warnings accumulated in our CI.

What changes are included in this PR?

The warnings are fixed or filtered if the deprecated functionality is still being tested.

Are these changes tested?

Yes.

Are there any user-facing changes?

No. Fixing test warnings in our CI.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

Human
AI

Reviewed before submission by:

Human
AI
Not reviewed

GitHub Issue: [Python] Fix deprecation warnings in the tests with numpy/pandas nightly #51301

github-actions · 2026-09-21T10:25:13Z

⚠️ GitHub issue #51301 has been automatically assigned in GitHub to PR creator.

AlenkaF · 2026-09-21T10:28:48Z

Opened a PR though we have one already opened: #51309.
Might be wrong, but the process seems very similar to other reviews where quite a lot of time is needed to communicate with the contributor (or the contributor's agent) what would be good to change. Decided to fix this quickly by myself.

AlenkaF · 2026-09-21T14:04:32Z

Warnings gone:
https://github.com/apache/arrow/actions/runs/35588604946/job/106297664397?pr=51404#step:6:4899

vs:
https://github.com/apache/arrow/actions/runs/35589580522/job/106300673910#step:6:4897

from the same AMD64 Conda Python 3.14 Pandas latest job.

@jorisvandenbossche if you have time for review.

Copilot

Copilot review overview

🟡 Changes recommended

Warning failures can leak allocated DLPack tensors in both export paths.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2

Open (2)

Prevent tensor leak when warning emission raises · New
Prevent exported tensor leak when warning emission raises · New

What changed in this PR

This PR updates tests and DLPack handling to address NumPy/Pandas deprecation warnings while preserving legacy behavior coverage.

Changes:

Updated deprecated NumPy and pandas test usage.
Filtered intentional deprecation warnings.
Adjusted legacy DLPack export warning handling and tests.

File	Summary
`python/pyarrow/tests/test_pandas.py`	Uses explicit timedelta units.
`python/pyarrow/tests/test_dlpack.py`	Updates warning filters and versioned DLPack coverage.
`python/pyarrow/tests/test_compute.py`	Updates timestamp construction and warning filters.
`python/pyarrow/tests/test_array.py`	Handles NumPy generic-unit deprecations.
`python/pyarrow/tensor.pxi`	Adjusts legacy DLPack export warning handling.
`python/pyarrow/array.pxi`	Adjusts legacy DLPack export warning handling.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

AlenkaF · 2026-09-22T08:34:46Z

Converting to draft as I need to think about Copilot's comment, looks right from a quick read.
I also need to rebase as I included some changes that have been fixed by #51307

This reverts commit 180c881.

AlenkaF · 2026-09-22T13:01:24Z

I decided to remove changes connected to the depr warning in __dlpack__:
The solution would need or check if warning is turned into an error and calling a deleter, or constructing a PyCapsule before emitting a warning. Both seem to be things that need some discussion and a fix in a separate issue I think.

Currently the CI is not showing any deprecation warnings in out tests: https://github.com/apache/arrow/actions/runs/35728469810/job/106747835234?pr=51404#step:6:4240

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved NumPy compatibility issues affect DLPack tests and nightly generic-unit warning coverage.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

Versioned DLPack tests fail on supported NumPy 2.0 · New

Resolved since last review (2)

Prevent exported tensor leak when warning emission raises
Prevent tensor leak when warning emission raises

jorisvandenbossche · 2026-09-22T13:25:35Z

@github-actions crossbow submit test-conda-python-3.14-pandas-nightly-numpy-nightly

github-actions · 2026-09-22T13:28:07Z

Revision: 84a8518

Submitted crossbow builds: ursacomputing/crossbow @ actions-0a1f3fcf86

Task	Status
test-conda-python-3.14-pandas-nightly-numpy-nightly

jorisvandenbossche · 2026-09-22T13:29:44Z

    assert np.dtype(np.timedelta64) == expected

-    df = pd.DataFrame({"a": [np.timedelta64()]})
+    df = pd.DataFrame({"a": [np.timedelta64(0, "s")]})


While this makes the test run without the warning, I am not sure if then the surrounding asserts still make sense.
One thing to do is also to change expected = np.dtype('m8') to use "m8[s]", but no idea if that would then still have produced the initial bug (the reason this test was added, #13553)

Ah, OK, sorry. Didn't try locally and assumed np.dtype(np.timedelta64) produces same result for any unit. I am guessing we want to keep testing this even for higher numpy versions. Will look first into changing expected value and what that means for the initial bug.

From testing locally it seems the initial bug is not produced even if using a time unit everywhere. Will push a commit.

jorisvandenbossche · 2026-09-22T13:38:39Z

    with pytest.raises(TypeError, match="Can only use DLPack "
                       "on arrays with no nulls."):
-        np.from_dlpack(arr)
+        np.from_dlpack(DLPackForwarder(arr, max_version=(1, 0)))


If leaving the DLPack depr question for a separate issue, I would maybe also leave out any of the dlpack-related changes here.

Reverted the rest dlpack changes: a148c13
and created a new issue: #51481

jorisvandenbossche · 2026-09-22T13:39:45Z

@AlenkaF thanks for the PR! I triggered a nightly pandas/numpy crossbow build, because it is there that I checked for the warnings (it might be that some of those were also showing up in the main CI builds, though)

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved moderate findings affect deprecation guards and regression-test coverage.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2

Open (2)

Preserve strict frequency checks in shared round-trip helper · New
Use 2.5.0.dev0 threshold for NumPy nightly deprecation guard

Previously missed (1)

In code that hasn't changed since last review

Handle NumPy nightly prerelease versions in deprecated-path guards

python/pyarrow/tests/test_array.py:2684

A PEP 440 nightly such as 2.5.0.devN compares less than Version("2.5.0"), so these guards still run the deprecated generic-unit paths in NumPy nightly—the exact environments this change targets. Compare the release/base version (and apply the same correction to the inverse guard in test_array_from_numpy_timedelta_incorrect_unit) so prereleases are handled consistently.

jorisvandenbossche · 2026-09-30T07:16:44Z

                            expected_schema=None,
-                            check_dtype=True, schema=None,
-                            preserve_index=False,
+                            check_dtype=True, check_freq=False,


Should this one be addressed?

I was not sure this needs to be addressed, but you asking means it should be ;)
My thought process was: check_freq will be tested by default and as we do not keep freq information from pandas (AFAIU) we might default to keeping the current behavior which is not checking the frequency in any of our roundtrips. Have no problem to change this as suggested though.

AlenkaF · 2026-09-25T08:15:56Z

The warnings should now only include the ones tracked in separate issues (#51302 and #51481):
https://github.com/ursacomputing/crossbow/actions/runs/35987607163/job/107593839234#step:7:4418

Copilot

Copilot review overview

🔵 Needs a closer look

Moderate issues remain with prerelease guards, DLPack warnings, frequency validation, and generic timedelta coverage.

Review effort: Lite
Findings: None

Resolved since last review (2)

Preserve strict frequency checks in shared round-trip helper
Use 2.5.0.dev0 threshold for NumPy nightly deprecation guard

Copilot

Copilot review overview

🔵 Needs a closer look

Unresolved moderate findings remain in the pandas and NumPy test updates.

Review effort: Lite
Findings: None

Previously missed (2)

In code that hasn't changed since last review

Filter deprecation warnings instead of skipping coverage

python/pyarrow/tests/test_array.py:2684

These conditionals skip the generic-unit tests on NumPy 2.5+, even though this deprecation is still a warning and the constructors continue to run. That removes coverage of Arrow's rejection of generic datetime/timedelta units on the nightly versions this PR targets (the same pattern is used in the timedelta test below); please filter the expected warning around the deprecated constructor instead of skipping the behavior test.

Keep generic timedelta dtype regression assertions

python/pyarrow/tests/test_pandas.py:5008

This changes the regression test from checking the generic np.timedelta64 dtype before and after conversion to checking only a concrete seconds dtype. The generic dtype is the state this test is intended to detect Arrow mutating, so a regression there would now pass unnoticed; keep the generic assertions and only make the DataFrame value explicit.

rok

Thanks for working on this @AlenkaF !

jorisvandenbossche · 2026-09-30T07:16:44Z

                            expected_schema=None,
-                            check_dtype=True, schema=None,
-                            preserve_index=False,
+                            check_dtype=True, check_freq=False,


Should this one be addressed?

Co-authored-by: Rok Mihevc <rok@mihevc.org>

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

Copilot

Copilot review overview

🔵 Needs a closer look

Two moderate issues remain in version guarding and shared frequency validation behavior.

Review effort: Lite
Findings: 1

Open (1)

Preserve strict frequency checks in shared round-trip helper

Copilot

Copilot review overview

🟡 Changes recommended

A critical syntax error and a moderate test-coverage regression remain unresolved.

Review effort: Lite
Findings: 1 · 1

Open (2)

Fix invalid groupby syntax in dataset test module · New
Preserve strict frequency checks in shared round-trip helper

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Copilot review overview

🔵 Needs a closer look

Address the incomplete warning filtering, prerelease-aware NumPy thresholds, and narrowly scoped frequency-check change.

Review effort: Lite
Findings: 1

Open (1)

Preserve strict frequency checks in shared round-trip helper

Resolved since last review (1)

Fix invalid groupby syntax in dataset test module

github-actions Bot added awaiting review Awaiting review Component: Python labels Sep 21, 2026

AlenkaF marked this pull request as ready for review September 21, 2026 14:04

Copilot AI lite review requested due to automatic review settings September 21, 2026 14:04

AlenkaF requested review from raulcd and rok as code owners September 21, 2026 14:04

Copilot started reviewing on behalf of AlenkaF September 21, 2026 14:05 View session

Copilot AI reviewed Sep 21, 2026

View reviewed changes

Comment thread python/pyarrow/array.pxi Outdated

Comment thread python/pyarrow/tensor.pxi Outdated

AlenkaF marked this pull request as draft September 22, 2026 08:33

AlenkaF added 6 commits September 22, 2026 10:35

Fix for deprecated generic units

1ed3c25

Fix dlpack warnings

b2d348a

Fix null_placement warnings

180c881

Fix unit keyward warning in pd.timestamp

43de55d

Revert "Fix null_placement warnings"

0baa242

This reverts commit 180c881.

Revert the change on __dlpack__

84a8518

AlenkaF force-pushed the gh-51301-fix-deprecation-warnings branch from b51c0a4 to 84a8518 Compare September 22, 2026 12:39

AlenkaF marked this pull request as ready for review September 22, 2026 13:01

Copilot AI review requested due to automatic review settings September 22, 2026 13:01

Copilot started reviewing on behalf of AlenkaF September 22, 2026 13:02 View session

Copilot AI reviewed Sep 22, 2026

View reviewed changes

Comment thread python/pyarrow/tests/test_dlpack.py Outdated

jorisvandenbossche reviewed Sep 22, 2026

View reviewed changes

github-actions Bot removed the awaiting review Awaiting review label Sep 22, 2026

Copilot AI reviewed Sep 24, 2026

View reviewed changes

Run linter

1bac8c5

Copilot AI review requested due to automatic review settings September 25, 2026 08:13

Copilot started reviewing on behalf of AlenkaF September 25, 2026 08:14 View session

Copilot AI reviewed Sep 25, 2026

View reviewed changes

Line too long

692fd5c

Copilot AI review requested due to automatic review settings September 25, 2026 08:35

Copilot started reviewing on behalf of AlenkaF September 25, 2026 08:36 View session

Copilot AI reviewed Sep 25, 2026

View reviewed changes

rok approved these changes Sep 29, 2026

View reviewed changes

Comment thread python/pyarrow/tests/test_compute.py

github-actions Bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Sep 29, 2026

jorisvandenbossche reviewed Sep 30, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Sep 30, 2026

Apply suggestion from @rok

fc384ff

Co-authored-by: Rok Mihevc <rok@mihevc.org>

Copilot AI lite review requested due to automatic review settings September 30, 2026 08:48

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 30, 2026

Copilot started reviewing on behalf of AlenkaF September 30, 2026 08:48 View session

Apply batched suggestions from code review

ed577f9

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

Copilot AI reviewed Sep 30, 2026

View reviewed changes

Copilot AI lite review requested due to automatic review settings September 30, 2026 08:50

Copilot started reviewing on behalf of AlenkaF September 30, 2026 08:51 View session

Copilot AI reviewed Sep 30, 2026

View reviewed changes

Comment thread python/pyarrow/tests/test_dataset.py Outdated

Fix groupby syntax for partitioning by weekday and color

8e0342f

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI lite review requested due to automatic review settings September 30, 2026 09:01

Copilot started reviewing on behalf of AlenkaF September 30, 2026 09:02 View session

Copilot AI reviewed Sep 30, 2026

View reviewed changes

Conversation

AlenkaF commented Sep 21, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Was AI used for this PR?

Uh oh!

github-actions Bot commented Sep 21, 2026

Uh oh!

AlenkaF commented Sep 21, 2026

Uh oh!

AlenkaF commented Sep 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

AlenkaF commented Sep 22, 2026

Uh oh!

AlenkaF commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

jorisvandenbossche commented Sep 22, 2026

Uh oh!

github-actions Bot commented Sep 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorisvandenbossche commented Sep 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlenkaF commented Sep 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

rok left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

AlenkaF commented Sep 21, 2026 •

edited by github-actions Bot

Loading