Description
Describe the bug
When running the Loki 1.2.0 Docker image, Loki is reporting that it can't write chunks to disk because there is "no space left on device", although there appears to be plenty of space.
level=error ts=2020-01-11T19:13:11.822567024Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="open /tmp/loki/chunks/ZmFrZS84NDBiODY0MTMwOWFkOTZlOjE2Zjk1ZWNjNmU1OjE2Zjk1ZWNkM2JjOmRkMWUwMjUx: no space left on device"
level=error ts=2020-01-11T19:13:11.851323284Z caller=flush.go:178 org_id=fake msg="failed to flush user" err="open /tmp/loki/chunks/ZmFrZS82ZDNlZmFhODk1OWZiYjQxOjE2Zjk1ZTgzOTI4OjE2Zjk1ZmMyNzRiOjg3MTQ1OTkw: no space left on device"
Plenty of space and inodes available on disk where /tmp/loki volume lives:
$ df -h /dev/sda1
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 915G 223G 646G 26% /
$ df -i /dev/sda1
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 60981248 5473071 55508177 9% /
/tmp/loki named volume mount from docker inspect
"Mounts": [
{
"Type": "volume",
"Name": "loki",
"Source": "/var/lib/docker/volumes/loki/_data",
"Destination": "/tmp/loki",
"Driver": "local",
"Mode": "rw",
"RW": true,
"Propagation": ""
}
$ docker volume inspect loki
[
{
"CreatedAt": "2020-01-11T10:37:39-08:00",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/loki/_data",
"Name": "loki",
"Options": null,
"Scope": "local"
}
]
Execing into the loki container and doing manual write tests to verify that files can be written
$ docker exec -it loki sh
/ # cd /tmp/loki
/tmp/loki # ls -l
total 596644
drwxr-xr-x 2 root root 610926592 Jan 11 19:24 chunks
drwxr-xr-x 2 root root 4096 Jan 9 00:01 index
/tmp/loki # cd chunks/
/tmp/loki/chunks # ls -l | wc -l
5286025
/tmp/loki/chunks # dd if=/dev/zero of=write_test count=1024 bs=1048576
1024+0 records in
1024+0 records out
/tmp/loki/chunks # ls -l write_test
-rw-r--r-- 1 root root 1073741824 Jan 11 19:27 write_test
/tmp/loki/chunks # rm write_test
/tmp/loki/chunks # dd if=/dev/urandom of=write_test count=1024 bs=1048576
1024+0 records in
1024+0 records out
/tmp/loki/chunks # ls -l write_test
-rw-r--r-- 1 root root 1073741824 Jan 11 19:28 write_test
/tmp/loki/chunks # rm write_test
I haven't been able to find any disk limitation in the Docker container, and the fact that I can still manually write files to the volume inside the container makes me suspect the bug is in the loki code, but I could definitely be wrong!
To Reproduce
Steps to reproduce the behavior:
- Run Loki (1.2.0, commit ccef3da) Docker image with Docker 18.09.6
- ???
Expected behavior
Loki continues to successfully write chunks to /tmp/loki while disk space and inodes are available.
Environment:
- Infrastructure: Docker 18.09.6, Debian 9.9, kernel 4.9.0-9-amd64
- Deployment tool: Ansible (using default Loki config file in Docker image at
/etc/loki/local-config.yaml)