Skip to content

Latest commit

 

History

History
253 lines (196 loc) · 10.7 KB

README.md

File metadata and controls

253 lines (196 loc) · 10.7 KB
Raw

mmh3

Documentation Status GitHub Super-Linter Build PyPi Version Python Versions License: MIT Total Downloads Recent Downloads

mmh3 is a Python extension for MurmurHash (MurmurHash3), a set of fast and robust non-cryptographic hash functions invented by Austin Appleby.

By combining mmh3 with probabilistic techniques like Bloom filter, MinHash, and feature hashing, you can develop high-performance systems in fields such as data mining, machine learning, and natural language processing.

Another popular use of mmh3 is to calculate favicon hashes, which are utilized by Shodan, the world's first IoT search engine.

This page provides a quick start guide. For more comprehensive information, please refer to the documentation.

Installation

pip install mmh3

Usage

Basic usage

>>> import mmh3
>>> mmh3.hash(b"foo") # returns a 32-bit signed int
-156908512
>>> mmh3.hash("foo") # accepts str (UTF-8 encoded)
-156908512
>>> mmh3.hash(b"foo", 42) # uses 42 as the seed
-1322301282
>>> mmh3.hash(b"foo", 0, False) # returns a 32-bit unsigned int
4138058784

mmh3.mmh3_x64_128_digest(), introduced in version 5.0.0, efficienlty hashes buffer objects that implement the buffer protocol (PEP 688) without internal memory copying. The function returns a bytes object of 16 bytes (128 bits). It is particularly suited for hashing large memory views, such as bytearray, memoryview, and numpy.ndarray, and performs faster than the 32-bit variants like hash() on 64-bit machines.

>>> mmh3.mmh3_x64_128_digest(numpy.random.rand(100))
b'\x8c\xee\xc6z\xa9\xfeR\xe8o\x9a\x9b\x17u\xbe\xdc\xee'

Various alternatives are available, offering different return types (e.g., signed integers, tuples of unsigned integers) and optimized for different architectures. For a comprehensive list of functions, refer to the API Reference.

hashlib-style hashers

mmh3 implements hasher objects with interfaces similar to those in hashlib from the standard library, although they are still experimental. See Hasher Classes in the API Reference for more information.

Changelog

See Changelog for the complete changelog.

5.0.1 - 2024-09-22

Fixed

  • Fix the issue that the package cannot be built from the source distribution (#90).

5.0.0 - 2024-09-18

Added

  • Add support for Python 3.13.
  • Improve the performance of the hash() function with METH_FASTCALL, reducing the overhead of function calls. For data sizes between 1–2 KB (e.g., 48x48 favicons), performance is 10%–20% faster. For smaller data (~500 bytes, like 16x16 favicons), performance increases by approximately 30% (#87).
  • Add digest functions that support the new buffer protocol (PEP 688) as input (#75). These functions are implemented with METH_FASTCALL too, offering improved performance (#84).
  • Slightly improve the performance of the hash_bytes() function (#88)
  • Add Read the Docs documentation (#54).
  • Document benchmark results (#53).

Changed

  • Backward-incompatible: The seed argument is now strictly validated to ensure it falls within the range [0, 0xFFFFFFFF]. A ValueError is raised if the seed is out of range (#84).
  • Backward-incompatible: Change the constructors of hasher classes to accept a buffer as the first argument (#83).
  • The type of flag argumens has been changed from bool to Any (#84).
  • Change the format of CHANGELOG.md to conform to the Keep a Changelog standard (#63).

Deprecated

  • Deprecate the hash_from_buffer() function. Use mmh3_32_sintdigest() or mmh3_32_uintdigest() as alternatives (#84).

Fixed

  • Fix a reference leak in the hash_from_buffer() function (#75).
  • Fix type hints (#76, #77, #84).

4.1.0 - 2024-01-09

Added

  • Add support for Python 3.12.

Fixed

  • Fix issues with Bazel by changing the directory structure of the project (#50).
  • Fix incorrect type hints (#51).
  • Fix invalid results on s390x when the arg x64arch of hash64 or hash_bytes() is set to False (#52).

License

MIT, unless otherwise noted within a file.

Known Issues

Different results from other MurmurHash3-based libraries

By default, mmh3 returns signed values for the 32-bit and 64-bit versions and unsigned values for hash128 due to historical reasons. To get the desired result, use the signed keyword argument.

Starting from version 4.0.0, mmh3 is endian-neutral, meaning that its hash functions return the same values on big-endian platforms as they do on little-endian ones. In contrast, the original C++ library by Appleby is endian-sensitive. If you need results that comply with the original library on big-endian systems, please use version 3.*.

For compatibility with Google Guava (Java), see https://stackoverflow.com/questions/29932956/murmur3-hash-different-result-between-python-and-java-implementation.

For compatibility with murmur3 (Go), see #46.

Contributing Guidelines

See Contributing.

Authors

MurmurHash3 was originally developed by Austin Appleby and distributed under public domain https://github.com/aappleby/smhasher.

Ported and modified for Python by Hajime Senuma.

External Tutorials

High-performance computing

The following textbooks and tutorials are great resources for learning how to use mmh3 (and other hash algorithms in general) for high-performance computing.

Internet of things

Shodan, the world's first IoT search engine, uses MurmurHash3 hash values for favicons (icons associated with web pages). ZoomEye follows Shodan's convention. Calculating these values with mmh3 is useful for OSINT and cybersecurity activities.

Related Libraries