managing [many] uploaded files

gc — Thu, 11 Dec 2008 22:44:10 +0000

most web applications have to deal with user provided files. Being them photos, word files, compressed archives, the big problem is what can happen when they become to be so many that become to reach the system limits of your filesystem.
Just think about sites like myspace, hacebook, flikr: they must provide upload features for millions of files/day and every each node of their webfarm cluser should be able to get it in milliseconds.
The solution I usually provide in such case, creates a n-levels of directoryes generating a unique 32-chars wide filename foreach file uploaded and then storing it. this method can fulfill the usual requirements for a medium sized high traffic application, and can store millions of files with no fear for performances.
Each file can be retrieved with a simple numeric ID or a supersecret 32bit key adding security or applications requiring privacy protection.
Also ther’s thumbnail generating method with caching mechanism if eeded.

here you can find it: too much code to explain, but if you don’t understand something, let me know

Working Beta » thumbnail generation

managing [many] uploaded files