Long Weekend of Compression

I was brought into a meeting on Friday with a basic idea of how to test file uploads and downloads with files of varying sizes and find out where the limitation of our servers is. The gist of my task is to upload files of varying sizes that are uncompressible because there are certain algorithms that are used for data transmission that use compression and speed it up. There is apparently a lot of importance on this issue and I am being pulled off of application development, and you guys should know how I feel about testing when I could be writing code.

So the charge fell on my shoulders to find the files to use for this. Usually I would just get on google and do a search like “index of” “50M” but I figured that it would be in my interest to code this one out, and besides I need randomly generated files pretty often. I wrote a script in per1 that will create a file and fill it with randomly generated characters, thereby breaking the encryption/compression algorithm. Objective number one is complete. Creating 50 megabyte files takes between 20 and 30 seconds so that also speeds up my testing because it has a goal of being able to transfer files of 4 gigs, and the act of downloading a 4gig file is pure craziness, but I could create it in a couple hours.

Having the basics of the script taken care of I decided to add into the app some other functionality that may come to someone’s aide in a future test. File check to make sure you aren’t overwriting another file as well as a few flags controlling how the size input is interpreted. Instead of saying I want 1024 bytes in a file I say I want 1 -kb. I added calculations to be able to create files up to and above a terabyte, but obviously I haven’t tested that. It was pretty neat, and works fine.

For anyone who is in need of it without having the compression check you can simply give it a random character and have it write the info to the file, which will in turn speed up the app more than 80 fold, it does a lot of its work in the chunk generation.

File Creator

Side note.

Why the fuck can’t I use the word Perl` in my posts on this site without a character after it? Wiki, forum or blog pages are all fucked and won’t let me post the word Perl` in it without something after it.

I have to track some sort of answer down before it makes me pull my hair out.

Has anyone ever seen this before?