You've got some out-of-control logs. Instead of deleting like crazy everyday, find the fast growing file or files, and look inside to investigate what may be causing this. Maybe some program is spinning in a loop logging some condition. Either disable that program, disable its logging or try to fix the condition that it's complaining about.

If a file is growing before your eyes, and you have no idea which program is writing to it, you may be able to find that out easily. Here is an example. Who has /var/log/syslog open? We use the fuser command:

# fuser /var/log/syslog

/var/log/syslog:      602

Only one process has /var/log/syslog open. It is process 602. What is that? Let us not bother with ps and grep, but look at the /proc filesystem directly:

# ls -l /proc/602/exe

lrwxrwxrwx 1 root root 0 Mar 29 17:45 /proc/602/exe -> /usr/sbin/rsyslogd

Aha, it is rsyslogd. We are not surprised that rsyslogd has /var/log/syslog/ open.

This method is not guaranteed to work. The reason is that programs do not have to keep files open ino rder to write to them. Suppose you have a process which opens a file, appends to it, and then closes it. You will have a somewhat more difficult investigation. You could run fuser many times until by chance you catch the process "red handed". That process itself could be going into and out of existence quickly. Another problem is that multiple processes could have the file open, but only one is making it larger. In that case, you can trace their system calls.

# fuser /var/log/huge-annoying-file

/var/log/huge-annoying-file:   1234 23459

Oops! Two processes have it open: 1234 and 23459. Let's see what they are doing:

# strace -p 1234

Process 1234 attached - interrupt to quit

select(1, NULL, NULL, NULL, {9, 922666}

It's not doing anything, just blocking in a select call. Ctrl-C to break the trace:

select(1, NULL, NULL, NULL, {9, 922666}^C <unfinished ...>

Check the next one:

# strace -p 23459

write(5, "Useless garbage ..."..., 512) = 512

write(5, "More useless garbage ..."..., 512) = 512

write(5, "More useless garbage ..."..., 512) = 512

write(5, "More useless garbage ..."..., 512) = 512

write(5, "More useless garbage ..."..., 512) = 512

write(5, "More useless garbage ..."..., 512) = 512

write(5, "More useless garbage ..."..., 512) = 512

^C

Oops, that one is writing constantly. It must be the bad one. We can even check that the file descriptor 5 which the process is writing to is in fact the large file:

# ls -l /proc/23459/fd/5

lr-x------ 1 root root 64 Apr  3 23:39 /proc/23459/fd/5 -> /var/log/huge-annoying-file

I don't suspect you have a corrupt filesystem, but to force a full check, you don't have to boot a DVD.

Firstly, review your filesystem's maximum mount count setting. Identify your partition using the df command. Example on an Ubuntu system I have here:

# df

Filesystem     1K-blocks    Used Available Use% Mounted on

/dev/sda1       18062108 5499320  11645284  33% /

udev              392152       4    392148   1% /dev

tmpfs             159768     768    159000   1% /run

none                5120       0      5120   0% /run/lock

none              399416     200    399216   1% /run/shm

/dev/sr0           43668   43668         0 100% /media/VBOXADDITIONS_4.1.4_74291

You can see that the / filesystem is mounted on /dev/sda1. So /dev/sda1 is the storage device of the root partition (and the only partition in this particular system).

Let's look at some attributes of that filesystem. This is safe to do even though it is mounted. The command spews a lot of output. Here is an excerpt:

$ dumpe2fs /dev/sda1

dumpe2fs 1.42 (29-Nov-2011)

Filesystem volume name:   <none>

Last mounted on:          /

[ ... SNIP ... ]

Last mount time:          Fri Mar 29 17:45:18 2013

Last write time:          Tue Mar  5 09:08:03 2013

Mount count:              22

Maximum mount count:      22

[ ... SNIP ... ]

Hey look, the mount count is equal to the maximum mount count. Next time I reboot, there will be a filesystem check. The important thing is that the mount count is a positive value. If yours is zero, change it to some positive value like 22 using tune2fs -c 22 /dev/whatever. Zero means that a check is never forced regardless of how many times the partition is mounted. Rarely rebooted systems should have low values here. A server that goes down once a year could probably use a fsck each time it reboots. You can set date-based check intervals also.

Now to force a check, you can override the actual count to be greater than or equal to the maximum, and then reboot. That's done with capital C: tune2fs -C 1234 /dev/whatever. Now the partition looks like it has been mounted 1234 times without a check, which is greater than the one- or two-digit maximum.