The df command shows the disk is full but it is not

by Daniel Pham
Published: Updated:

In this article, I will share with you a case where the df command shows the disk is full but it is not.

Case description

I am using a MariaDB cluster. On the master server number 1, I received a warning that the folder /var/log is full.

I login to the server, checking with the df command, the result that the folder using 99-100%.

[root@db-01 danie]# df -Th
Filesystem                       Type      Size  Used Avail Use% Mounted on
/dev/mapper/centos-root          ext4       30G  2,6G   26G  10% /
devtmpfs                         devtmpfs  3,9G     0  3,9G   0% /dev
tmpfs                            tmpfs     3,9G     0  3,9G   0% /dev/shm
tmpfs                            tmpfs     3,9G  425M  3,5G  11% /run
tmpfs                            tmpfs     3,9G     0  3,9G   0% /sys/fs/cgroup
/dev/sda1                        xfs       497M  162M  336M  33% /boot
/dev/mapper/centos-tmp           ext4       22G   45M   20G   1% /tmp
/dev/mapper/centos-var_log       ext4       30G   28G  494M  99% /var/log
/dev/mapper/centos-home          ext4      9,8G   37M  9,2G   1% /home
/dev/mapper/centos-var_lib_mysql ext4      197G   24G  164G  13% /var/lib/mysql
tmpfs                            tmpfs     783M     0  783M   0% /run/user/1000

You can see that this folder is 30GB in size and it is using 28GB.

Next, I use the du command to display the current size of the /var/log directory. What surprised me was that it was only using 1.1GB.

[root@db-01 danie]# du -sh /var/log
1,1G   /var/log

So what’s going on?

Find the cause of the df command shows the full is disk

the df command shows the disk is full but it is not
The df command shows the disk is full but it is not.

With my experience, I have encountered the case of files that have been deleted but it is not actually deleted. These files are labeled deleted but it is still on the disk.

This causes the df command to shows an incorrect result.

I use the lsof command to find deleted files. And I found out that there is a slow log file of about 27-28GB in size labeled deleted.

[root@db-01 danie]# lsof +L1
systemd-j  523  root  txt    REG  253,0      274752     0  660140 /usr/lib/systemd/systemd-journald;5bad8321 (deleted)
systemd-l  776  root  txt    REG  253,0      572320     0  660142 /usr/lib/systemd/systemd-logind;5bad8321 (deleted)
dbus-daem  784  dbus  txt    REG  253,0      442080     0  660381 /usr/bin/dbus-daemon;5bad8321 (deleted)
agetty     799  root  txt    REG  253,0       36864     0  659369 /usr/sbin/agetty;5bad8321 (deleted)
NetworkMa  811  root  txt    REG  253,0     2618600     0  660555 /usr/sbin/NetworkManager;5bad8321 (deleted)
mysqld    9700 mysql   20u   REG  253,5        9083     0      12 /tmp/ibWC769x (deleted)
mysqld    9700 mysql   21u   REG  253,5           0     0      13 /tmp/ibQKzxPO (deleted)
mysqld    9700 mysql   22u   REG  253,5        8551     0      14 /tmp/ibgmSBbm (deleted)
mysqld    9700 mysql   26u   REG  253,5           0     0      15 /tmp/ibqJ15SC (deleted)
mysqld    9700 mysql   30w   REG  253,4 28315029916     0 1048578 /var/log/mariadb/slow.log-20181223 (deleted)

Immediately at this step, I knew why the notification disk was full but not really.

My job now is to delete this file. I have written a script to delete these files.

Recommended Reading: Script free disk space from deleted files

Specifically in this case. I will do the following steps.

Step 1: Find files with deleted labels.

[root@db-01 danie]# find /proc/*/fd -ls | grep  '(deleted)'
815211450    0 lrwx------   1 mysql    mysql          64 Mar 20 13:52 /proc/9700/fd/20 -> /tmp/ibWC769x\ (deleted)
815211451    0 lrwx------   1 mysql    mysql          64 Mar 20 13:52 /proc/9700/fd/21 -> /tmp/ibQKzxPO\ (deleted)
815211452    0 lrwx------   1 mysql    mysql          64 Mar 20 13:52 /proc/9700/fd/22 -> /tmp/ibgmSBbm\ (deleted)
815211456    0 lrwx------   1 mysql    mysql          64 Mar 20 13:52 /proc/9700/fd/26 -> /tmp/ibqJ15SC\ (deleted)
724035709    0 l-wx------   1 mysql    mysql          64 Mar  5 18:08 /proc/9700/fd/30 -> /var/log/mariadb/slow.log-20181223\ (deleted)

Step 2: Get the file name to delete.

[root@db-01 danie]# find /proc/*/fd -ls | grep  '(deleted)' | awk '{print $11}'

In this case, I need to delete the slow log file with the deleted label. Therefore, , I will only need to delete the file /proc/9700/fd/30.

Step 3: delete files.

rm -f /proc/9700/fd/30

Alternatively, you can use the df command with the -i option to display the disk capacity more accurately.

[root@db-01 danie]# df -i
Filesystem                         Inodes IUsed    IFree IUse% Mounted on
/dev/mapper/centos-root           1966080 43052  1923028    3% /
devtmpfs                           998417   402   998015    1% /dev
tmpfs                             1001256     1  1001255    1% /dev/shm
tmpfs                             1001256   542  1000714    1% /run
tmpfs                             1001256    16  1001240    1% /sys/fs/cgroup
/dev/sda1                          256000   334   255666    1% /boot
/dev/mapper/centos-tmp            1411680    24  1411656    1% /tmp
/dev/mapper/centos-var_log        1966080   108  1965972    1% /var/log
/dev/mapper/centos-home            655360    29   655331    1% /home
/dev/mapper/centos-var_lib_mysql 13107200   449 13106751    1% /var/lib/mysql
tmpfs                             1001256     1  1001255    1% /run/user/1000

You see the above result is completely different, it has displayed the correct capacity of the disk.

What does this parameter -i mean?

-i, --inodes
      list inode information instead of block usage


This is one of the experiences I have experienced. If in the process of working with Linux servers, you encounter it. Hope this article will be useful to you, don’t be too worried if you don’t understand why the disk is always full.

(This is an article from my old blog that has been inactive for a long time, I don’t want to throw it away so I will keep it and hope it helps someone).

0 0 votes
Article Rating

You may also like

Notify of
Inline Feedbacks
View all comments

DevOps Lite is a personal blog specializing in technology with main topics about DevOps, DevSecOps, SRE and System Administrator. Articles are shared for free and contributed to the community.



Subscribe my Newsletter for new blog posts. Stay updated from your inbox!

© 2021-2024 – All rights reserved.

Please write sources “” when using articles from this website.

This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Accept Read More

Would love your thoughts, please comment.x

Adblock Detected

Please support us by disabling your AdBlocker extension from your browsers for our website.