Performance: Reduce the time complexity of Faker::Crypto generators

[alextaujenis]

Motivation / Background

This Pull Request has been created because the Faker::Crypto methods are generating extraordinarily long random strings to run through each hashing algorithm. The performance of each method within Faker::Crypto can be improved by up to 80% by balancing the complexity of the random strings for each hash algorithm.

Additional Information

The MD5 hash algorithm returns a string of 32 characters within the range of "a - f" and "0 - 9", which is a range of 16 characters. The total number of possible combinations with this setup is 16^32.

The method Faker::Crypto.md5 computes the MD5 hash of a random string like this:

OpenSSL::Digest::MD5.hexdigest(Lorem.characters)

Digging a little deeper we see that Lorem.characters returns a default number of 255 Alphanumeric characters:

def characters(number: 255, min_alpha: 0, min_numeric: 0)
  Alphanumeric.alphanumeric(number: number, min_alpha: min_alpha, min_numeric: min_numeric)
end

Going further we find out that Faker::Alphanumeric.alphanumeric returns random characters within the range of "a - z" and "0 -9", which is a range of 36 characters. The total number of possible combinations with this setup is 36^255, which is mind-bogglingly larger than 16^32 (the total number of possible combinations for the MD5 hash algorithm).

Optimization

The total number of random characters passed through the MD5 hashing algorithm within Faker::Crypto can be safely reduced from 255 down to a specific number. You can find that number by solving for the minimum value of x within: 36^x > 16^32. This allows MD5 collisions to occur before (non-unique) collisions within Faker::Lorem.characters.

We can find that perfect number for each hash algorithm by running this ruby solver:

algorithms = {MD5: 32, SHA1: 40, SHA256: 64, SHA512: 128}
algorithms.each do |algorithm, hash_length|
  puts "Algorithm: #{algorithm} - Hash Length: #{hash_length}"
  i = 0
  number = nil
  while(number == nil) do
    i+=1
    greater_than = ((36 ** i) > (16 ** hash_length))
    difference = (36 ** i) - (16 ** hash_length)
    number = i if greater_than
    puts "36^#{i} #{greater_than ? ">" : "<"} 16^#{hash_length} by #{difference}"
  end
  output = "OpenSSL::Digest::#{algorithm}.hexdigest(Lorem.characters(number: #{number})) SUGGESTED"
  puts output, "*" * output.length
end

Which provides this output:

Algorithm: MD5 - Hash Length: 32
36^1 < 16^32 by -340282366920938463463374607431768211420
36^2 < 16^32 by -340282366920938463463374607431768210160
36^3 < 16^32 by -340282366920938463463374607431768164800
36^4 < 16^32 by -340282366920938463463374607431766531840
36^5 < 16^32 by -340282366920938463463374607431707745280
36^6 < 16^32 by -340282366920938463463374607429591429120
36^7 < 16^32 by -340282366920938463463374607353404047360
36^8 < 16^32 by -340282366920938463463374604610658304000
...
36^24 < 16^32 by -317830109213583906223287396307975536640
36^25 > 16^32 by 467998910543825597179764993024768081920
OpenSSL::Digest::MD5.hexdigest(Lorem.characters(number: 25)) SUGGESTED
...

From this output we can see the point at which 36^x > 16^32:

36^24 < 16^32 by -317830109213583906223287396307975536640
36^25 > 16^32 by 467998910543825597179764993024768081920

This shows that Faker::Lorem.characters(number: 24) has 317830109213583906223287396307975536640 less possible combinations than the MD5 hash algorithm, while Faker::Lorem.characters(number: 25) has 467998910543825597179764993024768081920 more possible combinations than the MD5 hash algorithm.

We can safely reduce the number of random characters to 25 for the Faker::Crypto.md5 algorithm while still returning deterministically unique hashes. Here are the other optimal values for each algorithm:

OpenSSL::Digest::MD5.hexdigest(Lorem.characters(number: 25))
OpenSSL::Digest::SHA1.hexdigest(Lorem.characters(number: 31))
OpenSSL::Digest::SHA256.hexdigest(Lorem.characters(number: 50))
OpenSSL::Digest::SHA512.hexdigest(Lorem.characters(number: 100))

Performance Benchmark

You can see from the benchmark below that after reducing library complexity the Faker::Crypto methods are up to 80% faster, while the Faker::Omniauth methods enjoy performance gains up to 50% depending upon how heavily they rely upon the Faker::Crypto methods.

Checklist

Before submitting the PR make sure the following are checked:

• This Pull Request is related to one change. Changes that are unrelated should be opened in separate PRs.
• Commit message has a detailed description of what changed and why. If this PR fixes a related issue include it in the commit message. Ex: [Fix #issue-number]
• Tests are added or updated if you fix a bug, refactor something, or add a feature.
• Tests and Rubocop are passing before submitting your proposed changes.
• Double-check the existing generators documentation to make sure the new generator you want to add doesn't already exist.
• You've reviewed and followed the Contributing guidelines.

[alextaujenis]

@stefannibrasil

[stefannibrasil]

I think this is great, thank you so much for working on this! Just left one small suggestion.

[thdaraujo]

excellent, thanks for improving these generators @alextaujenis ! 👏