Question

25

Placing many (10 million) files in one folder

rated 0 times [ 25] [ 0] / answers: 1 / hits: 12758 / 2 Years ago, wed, june 22, 2022, 8:48:34

I've just added a predictive search (see example below) feature to my site that runs on a Ubuntu Server. This runs direct from a database. I want to cache the result for each search and use that if it exists, else create it.

Would there be any problem with me saving the potential cira 10 million results in separate files in one directory? Or is it advisable to split them down into folders?

Example:

Answers

Only authorized users can answer the question. Please sign in first, or register a free account.

uxuriousrnal

Add To Favorites

Follow

Total Points: 279

Total Questions: 106

Total Answers: 96

Location: Fiji

Member since Wed, Mar 29, 2023

1 Year ago

uxuriousrnal questions

1 libgl1-mesa-dri:i386 libglx-mesa0:i386 errors

Wed, Mar 9, 22, 09:04, 2 Years ago

1 Cannot Remove Preinstalled Apps in 22.04

Mon, Jul 18, 22, 01:48, 2 Years ago

1 How to break string into N 80 character lines?

Wed, Apr 13, 22, 01:15, 2 Years ago

1 How can I create a shortcut to open a selected file with a particular Application?

Wed, Apr 20, 22, 13:18, 2 Years ago

1 Synaptics 6cb:00df fingerprint sensor not working

Thu, Aug 26, 21, 22:01, 3 Years ago

View All

answered 2 Years ago tusmuumu · Accepted Answer

Would there be any problem with me saving the potential circa 10 million results in separate files in one directory?

Yes. There probably are more reasons but these I can post off the top of my head:

tune2fs has an option called dir_index that tends to be turned on by default (on Ubuntu it is) that lets you store roughly 100k files in a directory before you see a performance hit. That is not even close to the 10m files you are thinking about.

ext filesystems have a fixed maximum number of inodes. Every file and directory uses 1 inode. Use df -i for a view of your partitions and inodes free. When you run out of inodes you can not make new files or folders.

commands like rm and ls when using wildcards expand the command and will end up with a "argument list too long". You will have to use find to delete or list files. And find tends to be slow.

Or is it advisable to split them down into folders?

Yes. Most definitely. Basically you can not even store 10m files in 1 directory.

I would use the database. If you want to cache it for a website have a look at "solr" ("providing distributed indexing, replication and load-balanced querying").