[R] Update the docs to show how to avoid situations like data loss with leading zero in partition column

[arthurgailes]

Describe the bug, including details regarding any error messages, version, and platform.

Arrow does not correctly interpret some partition values as character columns, resulting in data loss.

library(arrow)
library(dplyr)

# character col with leading zero
mtcars2 <- mutate(mtcars, test = paste0(0, mtcars$carb)) |> as_tibble()

tmp <- tempfile()

write_dataset(mtcars2, tmp, partitioning = "test")

mtcars3 <- open_dataset(tmp) |> collect()

unique(mtcars2$test)
#> [1] "04" "01" "02" "03" "06" "08"
unique(mtcars3$test)
#> [1] 1 2 3 4 6 8

^{Created on 2024-01-17 with reprex v2.0.2}

Component(s)

R

[assignUser]

Looking at the generated dataset it correctly writes the leading zero: 'test=08' but the parquet reader implicitly converts to int. So the fix probably needs to happen in the C++ lib, right @paleolimbot

[paleolimbot]

I think it may work to explicitly set the partitioning argument in open_dataset()...something like open_dataset(..., partitioning = hive_partitioning(test = string()).

[assignUser]

Thanks @paleolimbot, that does work:

r$> mtcars4 = open_dataset(tmp, partitioning = hive_partition(test = string()))

r$> mtcars4
FileSystemDataset with 6 Parquet files
mpg: double
cyl: double
disp: double
hp: double
drat: double
wt: double
qsec: double
vs: double
am: double
gear: double
carb: double
test: string

r$> mtcars4 |> as_tibble()
# A tibble: 32 × 12
     mpg   cyl  disp    hp  drat    wt  qsec    vs    am  gear  carb test
   <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <chr>
 1  22.8     4 108      93  3.85  2.32  18.6     1     1     4     1 01
 2  21.4     6 258     110  3.08  3.22  19.4     1     0     3     1 01
 3  18.1     6 225     105  2.76  3.46  20.2     1     0     3     1 01
 4  32.4     4  78.7    66  4.08  2.2   19.5     1     1     4     1 01
 5  33.9     4  71.1    65  4.22  1.84  19.9     1     1     4     1 01
 6  21.5     4 120.     97  3.7   2.46  20.0     1     0     3     1 01
 7  27.3     4  79      66  4.08  1.94  18.9     1     1     4     1 01
 8  18.7     8 360     175  3.15  3.44  17.0     0     0     3     2 02
 9  24.4     4 147.     62  3.69  3.19  20       1     0     4     2 02
10  22.8     4 141.     95  3.92  3.15  22.9     1     0     4     2 02

[thisisnic]

This is great stuff! We should add this to the docs as it's not immediately obvious (i.e. I had no idea this was the case)

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

[nil68657]

Anyone working on this?