[C++] Add a "list_contains" kernel

[asfimport]

Assume you have a list array:

arr = pa.array([["a", "b"], ["a", "c"], ["b", "c", "d"]])

And you want to know for each list if it contains a certain value (of the same type as the list's values). A "list_contains" function (or other name) would be useful for that:

pc.list_contains(arr, "a")
# -> True, True False

The current workaround that I found was flattening, checking equality, and then reducing again with groupby, but this is quite tedious:

>>> temp = pa.table({'index': pc.list_parent_indices(arr), 'contains_value': pc.equal(pc.list_flatten(arr), "a")})
>>> temp.group_by('index').aggregate([('contains_value', 'any')])['contains_value_any'].chunk(0)
<pyarrow.lib.BooleanArray object at 0x7ffaf3f8de20>
[
  true,
  true,
  false
]

But this also only works if there are no empty or missing list values.

Reporter: Joris Van den Bossche / @jorisvandenbossche

Related issues:

• [C++] Add compute functions for list array containment #32045 (duplicates)

_{Note: This issue was originally created as ARROW-18097. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Then there probably should be a "list_index" function as well, similar to "is_in" vs. "index_in" ?

pc.list_index(arr, "b")
# -> 1, None, 0

[asfimport]

Jacek Pliszka / @JacekPliszka:
Maybe no need for new name but is_in can be reused:

pc.is_in("a", arr)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~jacek.pliszka] That would give a different signature; "is_in" is currently a scalar kernel that returns a result of the same shape as the first input argument. So that generally wouldn't work here (for lists, it would the second argument that we want to check element-by-element in your example).
While it might be technically possible in C++ to have kernels with different input types, I am not sure it is a good idea to reuse the same name for this.

[thisisnic]

This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.

[alexshpilkin]

Mere passage of time does not cause a feature to appear. Not stale.

[thisisnic]

Thanks @alexshpilkin, will mark as not stale. Are you interested in submitting a PR?

[alexshpilkin]

@thisisnic This was one of the issue that pushed me to Polars (not the issue, by any means, just one of the things that contributed to the decision). So it’s less than immediately relevant to me at the moment. Still, I’m not against giving it a shot in principle, if you or someone else could point me in the right direction. I just don’t promise the kernel will be particularly good :)

[thisisnic]

If you make a start and draft a PR, folks around here are pretty friendly and happy to help, just pop some comments in explaining where you got to and where you aren't sure :) Perfection is not a requirement, for sure!

[jonasdedden]

@pitrou @jorisvandenbossche @thisisnic please have a look at #51611, where I implement a list.contains compute kernel.