Thanos Store : strange permissions in created directory

[JohnGrime]

Docker image:tag: quay.io/thanos/thanos:v0.41.0

Object Storage Provider: Containerized instance of MinIO (image: mini/minio - slightly old, but functional)

What happened:

I'm trying to run the Thanos Store gateway as part of a Docker Compose ensemble.

The relevant compose.yaml service entries:

  minio:
    image: ${MINIO_CONTAINER_IMAGE}
    container_name: minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ADMIN_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ADMIN_PASSWORD}
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 10s
      timeout: 5s
      retries: 12

  minio-init:
    image: ${MINIO_CONTAINER_IMAGE}
    container_name: minio-init
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      mc alias set local http://minio:9000 ${MINIO_ADMIN_USER} ${MINIO_ADMIN_PASSWORD};
      mc mb --ignore-existing local/${THANOS_BUCKET};
      mc anonymous set none local/${THANOS_BUCKET};
      exit 0;
      "
    restart: "no"

  thanos-store:
    image: ${THANOS_CONTAINER_IMAGE}
    container_name: thanos-store
    command:
      - store
      - --data-dir=/var/thanos/store
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
      - --log.level=info
    depends_on:
      - minio-init
    volumes:
      - thanos-store-data:/var/thanos/store
      - ./configs/objstore.yml:/etc/thanos/objstore.yml:ro
    ports:
      - "31901:10901"
      - "31902:10902"

Relevant compose.yaml volume entries:

  minio-data:
  thanos-store-data:

After running docker compose up I get the following messages from the Thanos Store gateway container:

thanos-store        | ts=2026-06-08T20:58:30.55548007Z caller=factory.go:39 level=info msg="loading bucket configuration"
thanos-store        | ts=2026-06-08T20:58:30.556369637Z caller=inmemory.go:185 level=info msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
thanos-store        | ts=2026-06-08T20:58:30.556634112Z caller=main.go:151 level=error err="mkdir /var/thanos/store/meta-syncer: permission denied\nmeta fetcher\nmain.runStore\n\t/app/cmd/thanos/store.go:407\nmain.registerStore.func1\n\t/app/cmd/thanos/store.go:264\nmain.main\n\t/app/cmd/thanos/main.go:149\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693\npreparing store command failed\nmain.main\n\t/app/cmd/thanos/main.go:151\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1693"
thanos-store exited with code 1

As a check, I ran chmod -R 777 /path/to/thanos-store-data/volume (where the path was the directory indicated by docker volume inspect) after calling docker compose down.

Running the ensemble again, the thanos-store container apparently succeeded with creating the /var/thanos/store/meta-syncer directory. The output was now:

thanos-store        | ts=2026-06-08T21:04:56.97709054Z caller=factory.go:39 level=info msg="loading bucket configuration"
thanos-store        | ts=2026-06-08T21:04:56.977835518Z caller=inmemory.go:185 level=info msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
thanos-store        | ts=2026-06-08T21:04:56.978712244Z caller=options.go:29 level=info protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
thanos-store        | ts=2026-06-08T21:04:56.981789936Z caller=store.go:594 level=info msg="starting store node"
thanos-store        | ts=2026-06-08T21:04:56.983188446Z caller=store.go:492 level=info msg="initializing bucket store"
thanos-store        | ts=2026-06-08T21:04:56.992159987Z caller=intrumentation.go:75 level=info msg="changing probe status" status=healthy
thanos-store        | ts=2026-06-08T21:04:56.993787363Z caller=http.go:72 level=info service=http/server component=store msg="listening for requests and metrics" address=0.0.0.0:10902
thanos-store        | ts=2026-06-08T21:04:56.994398074Z caller=handler.go:87 level=info service=http/server component=store caller=tls_config.go:354 time=2026-06-08T21:04:56.994369978Z msg="Listening on" address=[::]:10902
thanos-store        | ts=2026-06-08T21:04:56.994441392Z caller=handler.go:87 level=info service=http/server component=store caller=tls_config.go:357 time=2026-06-08T21:04:56.99443017Z msg="TLS is disabled." http2=false address=[::]:10902
thanos-store        | ts=2026-06-08T21:04:57.000466789Z caller=fetcher.go:691 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=8.115256ms duration_ms=8 cached=0 returned=0 partial=0
thanos-store        | ts=2026-06-08T21:04:57.000619669Z caller=store.go:509 level=info msg="bucket store ready" init_duration=8.360706ms
thanos-store        | ts=2026-06-08T21:04:57.000974921Z caller=intrumentation.go:56 level=info msg="changing probe status" status=ready
thanos-store        | ts=2026-06-08T21:04:57.001055518Z caller=grpc.go:158 level=info service=gRPC/server component=store msg="listening for serving gRPC" address=0.0.0.0:10901
thanos-store        | ts=2026-06-08T21:04:57.003424599Z caller=fetcher.go:691 level=info component=block.BaseFetcher msg="successfully synchronized block metadata" duration=2.752827ms duration_ms=2 cached=0 returned=0 partial=0

I looked into the local directory containing the thanos-store-data volume while the containers were running to examine the meta-syncer directory:

[me]$ stat /path/to/volumes/blah_thanos-store-data/_data/meta-syncer/
  File: /path/to/volumes/blah_thanos-store-data/_data/meta-syncer/
  Size: 6               Blocks: 0          IO Block: 4096   directory
Device: fd04h/64772d    Inode: 190747      Links: 2
Access: (0755/drwxr-xr-x)  Uid: (101000/ UNKNOWN)   Gid: (101000/pubarch-borr-ro)
Access: 2026-06-08 17:04:56.998694408 -0400
Modify: 2026-06-08 17:04:56.976694410 -0400
Change: 2026-06-08 17:04:56.976694410 -0400
 Birth: 2026-06-08 17:04:56.976694410 -0400

Hmmm. Who is the pubarch-borr-ro user? That's not the user thanos-store is running as in the container:

[me]$ docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED         STATUS                   PORTS                                                                                              NAMES
...
d9647e93d295   quay.io/thanos/thanos:v0.41.0   "/bin/thanos store -…"   2 minutes ago   Up 2 minutes             0.0.0.0:31901->10901/tcp, [::]:31901->10901/tcp, 0.0.0.0:31902->10902/tcp, [::]:31902->10902/tcp   thanos-store
...
0.0.0.0:9000-9001->9000-9001/tcp, [::]:9000-9001->9000-9001/tcp                                    minio
[me]$ docker exec -it d9647e93d295 sh
/ $ ps
PID   USER     TIME  COMMAND
    1 thanos    0:00 /bin/thanos store --data-dir=/var/thanos/store --objstore.config-file=/etc/thanos/objstore.yml --grpc-address=0.0.0.0:10901 --http-address=0.0.0.0:10902 --log.level=info
   17 thanos    0:00 sh
   23 thanos    0:00 ps
/ $ 

Why is the meta-syncer directory created by the user pubarch-borr-ro? Who is that user? A quick Google turned up nothing useful, and nor did I find anything useful by searching the Thanos repo ...

What you expected to happen: I'd expect:

1. The Thanos Store gateway to be able to create the meta-syncer directory in the /var/thanos/store directory; that directory is actually mounted in from a Docker volume defined in compose.yaml. The same volume creation/mapping approach does not break Thanos commands in other containers run from the same source image (for example, Sidecar instances).
2. The Thanos Store gateway might also create the meta-suncer directory under e.g. the thanos user and not the mysterious pub arch-borr-ro user.

How to reproduce it (as minimally and precisely as possible):

Contents of .env file :

MINIO_ADMIN_USER=whatever
MINIO_ADMIN_PASSWORD=whatever

THANOS_CONTAINER_IMAGE=quay.io/thanos/thanos:v0.41.0
MINIO_CONTAINER_IMAGE=minio/minio

THANOS_BUCKET=thanos

compose.yaml:

services:

  minio:
    image: ${MINIO_CONTAINER_IMAGE}
    container_name: minio
    command: server /data --console-address ":9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ADMIN_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ADMIN_PASSWORD}
    volumes:
      - minio-data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 10s
      timeout: 5s
      retries: 12

  minio-init:
    image: ${MINIO_CONTAINER_IMAGE}
    container_name: minio-init
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      mc alias set local http://minio:9000/ ${MINIO_ADMIN_USER} ${MINIO_ADMIN_PASSWORD};
      mc mb --ignore-existing local/${THANOS_BUCKET};
      mc anonymous set none local/${THANOS_BUCKET};
      exit 0;
      "
    restart: "no"

  thanos-store:
    image: ${THANOS_CONTAINER_IMAGE}
    container_name: thanos-store
      #    user: jgrimeadmin # <- avoid permissions issues where Prometheus/Sidecar container share a volume
    command:
      - store
      - --data-dir=/var/thanos/store
      - --objstore.config-file=/etc/thanos/objstore.yml
      - --grpc-address=0.0.0.0:10901
      - --http-address=0.0.0.0:10902
      - --log.level=info
    depends_on:
      - minio-init
    volumes:
      - thanos-store-data:/var/thanos/store:rw
      - ./configs/objstore.yml:/etc/thanos/objstore.yml:ro
    ports:
      - "31901:10901"
      - "31902:10902"

volumes:
  minio-data:
  thanos-store-data:

Then docker compose up

Full logs to relevant components:

See log messages, previous.

Anything else we need to know:

Red Hat Enterprise Linux 9.7, Docker version 29.5.3, build d1c06ef

[s3onghyun]

The quay.io/thanos/thanos image runs as a non-root user (UID 1001, set in its Dockerfile with adduser -u 1001 and USER 1001). A freshly created Docker named volume is owned by root:root, so the process cannot create meta-syncer inside its data dir. That is the permission error.

chmod -R 777 works but is a blunt, insecure fix. The correct one is to make the volume owned by the user the container actually runs as:

• Add an init step (an init container, or a one-shot service in Compose) that runs chown -R 1001:1001 /var/thanos/store before thanos store starts, or
• Set user: / an fsGroup so the mount is owned by 1001.

Once the data dir is owned by 1001, thanos store starts cleanly without loosening permissions on the volume.

[Yash121l]

The mode is not the problem. Store creates that directory in pkg/block/fetcher.go with os.MkdirAll(cacheDir, os.ModePerm), which is 0777 before umask, and a default 022 umask is exactly how it ends up 0755 in stat. Thanos never sets an owner; the image runs as uid 1001 (adduser -u 1001 and USER 1001 in the Dockerfile), so the odd name you see is only your host's passwd or LDAP resolving whatever that uid maps to locally. The original mkdir ... permission denied is about the parent: a fresh named volume is created root-owned, and uid 1001 cannot create meta-syncer inside it. Instead of chmod -R 777, hand the volume to the uid the container runs as:

  thanos-store-init:
    image: ${THANOS_CONTAINER_IMAGE}
    user: "0"
    entrypoint: ["/bin/sh", "-c", "chown -R 1001:1001 /var/thanos/store"]
    volumes:
      - thanos-store-data:/var/thanos/store
    restart: "no"

Does that match what you see on the volume?