Automatically divide your files into a (recursive) directory with an arbitrary number of files per (sub) folder / folders per superior folder

The easiest and IMHO most efficient way is to have a script reorganize the files into folders, if necessary even different layers of directory levels. This will make your files browsable without choking nautilus.

The script below will do that for you. It will create folders with an arbitrary number of files. These folders will be organized into sub folders if they exceed an (the same) arbitrary number, etc. In other words; each (sub-) level will have the same maximum number of files / sub directories, making browsing easily possible.

Each of the created folders shows the folder number + the number of created sub levels (where e.g. 22_1 only contains files):

The test

I tested in on a directory of 300.000 files, to be reorganized in chunks of (max) 100 files, to be organized into superior directories of (max) 100 folders etc.

It took less then a minute on my system. A test of 100.000 files into smaller chunks was a matter of seconds.

The script

#!/usr/bin/env python3

import subprocess

import os

import shutil



#--- set the directory to reorganize below

dr = "/path/to/directory"

#--- set the number of files/folders per level

size = 100



level = 0

def move(fn, drn, level):

    folder = dr+"/"+str(drn)+"_"+str(level)

    if not os.path.exists(folder):

        os.mkdir(folder)

    shutil.move(dr+"/"+f, folder+"/"+f)



while len(os.listdir(dr)) > size:

    level += 1

    fn = 0; drn = 1

    for f in os.listdir(dr):

        if fn < size:

            move(fn, drn, level)

        else:

            fn = 0

            drn += 1

            move(fn, drn, level)

        fn += 1

How to use

1. Copy the script into an empty folder, save it as reorganize.py


2. In the head section, set the path to your directory, the desired number of files per subdirectory (= equal to folders per containing directory).



3. Run it by the command:

   
   
   

   python3 /path/to/reorganize.py
   
   

Note

The script (as it is) just creates a directory structure where each level has a defined number of files/folders. It does not take into account any kind of organisation by name, date or whatever.

EDIT

As requested in a comment, a script to move the files back into one flat directory after having processed the files.

The usage is pretty much the same. As directory, set the same directory as the first script, but that seems obvious.

#!/usr/bin/env python3

import shutil

import os



#--- set the directory, the same as the first script

dr = "/path/to/directory"

#---



# move the files back

for root, dirs, files in os.walk(dr):

    for file in files:

        shutil.move(root+"/"+file, dr+"/"+file)

# remove the (now empty) subdirectories

for d in os.listdir(dr):

    folder = dr+"/"+d

    if os.path.isdir(folder):

        shutil.rmtree(folder)

Note

As mentioned in a comment, the script assumes there is not risk of name clashes, since all files initially came from the same (flat) directory.