(This question deals with a similar issue, but it talks about a rotated log file.)
Today I got a system message regarding very low /var space.
As usual I executed the commands in the line of sudo apt-get clean which improved the scenario only slightly. Then I
deleted the rotated log files which again provided very little improvement.
Upon examination I find that some log files in the /var/log has grown up to be very huge ones. To be specific, ls -lSh /var/log gives,
total 28G
-rw-r----- 1 syslog adm 14G Aug 23 21:56 kern.log
-rw-r----- 1 syslog adm 14G Aug 23 21:56 syslog
-rw-rw-r-- 1 root utmp 390K Aug 23 21:47 wtmp
-rw-r--r-- 1 root root 287K Aug 23 21:42 dpkg.log
-rw-rw-r-- 1 root utmp 287K Aug 23 20:43 lastlog
As we can see, the first two are the offending ones. I am mildly surprised why such large files have not been rotated.
So, what should I do? Simply delete these files and then reboot? Or go for some more prudent steps?
I am using Ubuntu 14.04.
UPDATE 1
To begin with, the system is only several months old. I had to install the system from scratch couple of months back after a hard disk crash.
Now, as advised in this answer,
I first checked the offending log files using tail, no surprise there. Then, for deeper inspection, I executed this script from the same answer.
for log in /var/log/{syslog,kern.log}; do
echo "${log} :"
sed -e 's/[[^]]+]//' -e 's/.*[0-9]{2}:[0-9]{2}:[0-9]{2}//' ${log}
| sort | uniq -c | sort -hr | head -10
done
The process took several hours.
The output was in the line of,
/var/log/syslog :
71209229 Rafid-Hamiz-Dell kernel: sda3: rw=1, want=7638104968240336200, limit=1681522688
53929977 Rafid-Hamiz-Dell kernel: attempt to access beyond end of device
17280298 Rafid-Hamiz-Dell kernel: attempt to access beyond end of device
1639 Rafid-Hamiz-Dell kernel: EXT4-fs warning (device sda3): ext4_end_bio:317: I/O error -5 writing to inode 6819258 (offset 0 size 4096 starting block 54763121030042024)
<snipped>
/var/log/kern.log.1 :
71210257 Rafid-Hamiz-Dell kernel: attempt to access beyond end of device
71209212 Rafid-Hamiz-Dell kernel: sda3: rw=1, want=7638104968240336200, limit=1681522688
1639 Rafid-Hamiz-Dell kernel: EXT4-fs warning (device sda3): ext4_end_bio:317: I/O error -5 writing to inode 6819258 (offset 0 size 4096 starting block 954763121030042024)
(/dev/sda3 is my home directory. As we can find,
lsblk /dev/sda
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 931.5G 0 disk
├─sda1 8:1 0 122.1G 0 part /
├─sda2 8:2 0 7.6G 0 part [SWAP]
└─sda3 8:3 0 801.8G 0 part /home
Why a process will want to write beyond the limit is actually outside the scope of my comprehension. Perhaps I will want to ask a different question in this forum if this
continues even after a system update.)
Then, from this answer (you may want to check this for a deeper understanding), I
executed,
sudo su -
> kern.log
> syslog
Now, these files have zero sizes. The system is running fine before and after a reboot.
I will watch these files (along with others) in the next few days and report back should
they behave out-of-line.
As a final note, both the offending files (kern.log and syslog), are set to be rotated, as inspection of the files (grep helped) inside
/etc/logrotate.d/ shows.
UPDATE 2
The log files are actually rotated. Looks like the large sizes were attained on a single day.
