Mom¶

Mother of all our Python projects. Batteries for Python.

Getting the library¶

$ pip install mom

or

$ git clone git://github.com/gorakhargosh/mom.git

or

$ git clone http://code.google.com/p/python-mom/

$ cd mom
$ python setup.py install

User Guides¶

Contributing¶

Welcome hackeratti! So you have got something you would like to see in mom? Whee. This document will help you get started.

Important URLs¶

mom uses git to track code history and hosts its code repository at github. The issue tracker is where you can file bug reports and request features or enhancements to mom.

Before you start¶

Ensure your system has the following programs and libraries installed before beginning to hack:

Python
git
ssh

Setting up the Work Environment¶

mom makes extensive use of zc.buildout to set up its work environment. You should get familiar with it.

Steps to setting up a clean environment:

Fork the code repository into your github account. Let us call you hackeratti. That is your name innit? Replace hackeratti with your own username below if it isn’t.

Clone your fork and setup your environment:

$ git clone --recursive git@github.com:hackeratti/mom.git
$ cd mom
$ python bootstrap.py --distribute
$ bin/buildout

Important

Re-run bin/buildout every time you make a change to the buildout.cfg file.

That’s it with the setup. Now you’re ready to hack on mom.

Enabling Continuous Integration¶

The repository checkout contains a script called autobuild.sh which you should run prior to making changes. It will detect changes to Python source code or restructuredText documentation files anywhere in the directory tree and rebuild sphinx documentation, run all tests using unittest2, and generate coverage reports.

Start it by issuing this command in the mom directory checked out earlier:

$ tools/autobuild.sh
...

Happy hacking!

API Documentation¶

mom¶

synopsis:	Mother of all our Python projects.
module:	mom

How many times have you noticed a `utils` subpackage or module?¶

Yeah. There is a lot of code duplication that occurs throughout our Python-based projects and results in code that is harder to maintain in the long term. Not to mention all the duplicate test code and more maintenance.

Therefore, we have decided to move all those util modules and subpackages to a central library, which we use throughout our projects. If you have a utils module, chances are you’re duplicating and wasting effort whereas instead you could use tested code provided by this library. If there’s something not included in this library and think it should, speak up.

synopsis:	Deals with a lot of cross-version issues.
module:	mom.builtins

bytes, str, unicode, and basestring mean different things to Python 2.5, 2.6, and 3.x.

These are the original meanings of the types.

Python 2.5

bytes is not available.
str is a byte string.
unicode converts to unicode string.
basestring exists.

Python 2.6

bytes is available and maps to str
str is a byte string.
unicode converts to unicode string
basestring exists.

Python 3.x

bytes is available and does not map to str.
str maps to the earlier unicode, but unicode has been removed.
basestring has been removed.
unicode has been removed

This module introduces the “bytes” type for Python 2.5 and adds a few utility functions that will continue to keep working as they should even when Python versions change.

Rules to follow: * Use bytes where you want byte strings (binary data).

The meanings of these types have been changed to suit Python 3.

Encodings¶

mom.builtins.bin(number, prefix='0b')¶

Converts a long value to its binary representation.

Parameters:	number – Long value. prefix – The prefix to use for the bitstring. Default “0b” to mimic Python builtin `bin()`.
Returns:	Bit string.

mom.builtins.hex(number, prefix='0x')¶

Converts a integer value to its hexadecimal representation.

Parameters:	number – Integer value. prefix – The prefix to use for the hexadecimal string. Default “0x” to mimic `hex()`.
Returns:	Hexadecimal string.

mom.builtins.byte(number)¶

Converts a number between 0 and 255 (both inclusive) to a base-256 (byte) representation.

Use it as a replacement for chr where you are expecting a byte because this will work on all versions of Python.

Raises :class:struct.error on overflow.

Parameters:	number – An unsigned integer between 0 and 255 (both inclusive).
Returns:	A single byte.

mom.builtins.byte_ord(byte_)¶

Returns the ordinal value of the given byte.

Parameters:	byte – The byte.
Returns:	Integer representing ordinal value of the byte.

Bits and bytes size counting¶

mom.builtins.bytes_leading(raw_bytes, needle='\x00')¶

Finds the number of prefixed byte occurrences in the haystack.

Useful when you want to deal with padding.

Parameters:	raw_bytes – Raw bytes. needle – The byte to count. Default .
Returns:	The number of leading needle bytes.

mom.builtins.bytes_trailing(raw_bytes, needle='\x00')¶

Finds the number of suffixed byte occurrences in the haystack.

Useful when you want to deal with padding.

Parameters:	raw_bytes – Raw bytes. needle – The byte to count. Default .
Returns:	The number of trailing needle bytes.

mom.builtins.integer_bit_length(number)¶

Number of bits needed to represent a integer excluding any prefix 0 bits.

Parameters:	number – Integer value. If num is 0, returns 0. Only the absolute value of the number is considered. Therefore, signed integers will be abs(num) before the number’s bit length is determined.
Returns:	Returns the number of bits in the integer.

mom.builtins.integer_bit_size(number)¶

Number of bits needed to represent a integer excluding any prefix 0 bits.

Parameters:	number – Integer value. If num is 0, returns 1. Only the absolute value of the number is considered. Therefore, signed integers will be abs(num) before the number’s bit length is determined.
Returns:	Returns the number of bits in the integer.

mom.builtins.integer_byte_length(number)¶

Number of bytes needed to represent a integer excluding any prefix 0 bytes.

Parameters:	number – Integer value. If num is 0, returns 0.
Returns:	The number of bytes in the integer.

mom.builtins.integer_byte_size(number)¶

Size in bytes of an integer.

Parameters:	number – Integer value. If num is 0, returns 1.
Returns:	Size in bytes of an integer.

Type detection predicates¶

mom.builtins.is_bytes(obj)¶

Determines whether the given value is a bytes instance.

Parameters:	obj – The value to test.
Returns:	`True` if value is a bytes instance; `False` otherwise.

mom.builtins.is_bytes_or_unicode(obj)¶

Determines whether the given value is an instance of a string irrespective of whether it is a byte string or a Unicode string.

Parameters:	obj – The value to test.
Returns:	`True` if value is any type of string; `False` otherwise.

mom.builtins.is_integer(obj)¶

Determines whether the object value is actually an integer and not a bool.

Parameters:	obj – The value to test.
Returns:	`True` if yes; `False` otherwise.

mom.builtins.is_sequence(obj)¶

Determines whether the given value is a sequence.

Sets, lists, tuples, bytes, dicts, and strings are treated as sequence.

Parameters:	obj – The value to test.
Returns:	`True` if value is a sequence; `False` otherwise.

mom.builtins.is_unicode(obj)¶

Determines whether the given value is a Unicode string.

Parameters:	obj – The value to test.
Returns:	`True` if value is a Unicode string; `False` otherwise.

Number predicates¶

People screw these up too. Useful in functional programming.

mom.builtins.is_even(num)¶

Determines whether a number is even.

Parameters:	num – Integer
Returns:	`True` if even; `False` otherwise.

mom.builtins.is_negative(num)¶

Determines whether a number is negative.

Parameters:	num – Number
Returns:	`True` if positive; `False` otherwise.

mom.builtins.is_odd(num)¶

Determines whether a number is odd.

Parameters:	num – Integer
Returns:	`True` if odd; `False` otherwise.

mom.builtins.is_positive(num)¶

Determines whether a number is positive.

Parameters:	num – Number
Returns:	`True` if positive; `False` otherwise.

synopsis:	Common collections.
module:	mom.

Queues¶

class mom.collections.SetQueue(maxsize=0)¶

Thread-safe implementation of an ordered set queue, which coalesces duplicate items into a single item if the older occurrence has not yet been read and maintains the order of items in the queue.

Ordered set queues are useful when implementing data structures like event buses or event queues where duplicate events need to be coalesced into a single event. An example use case is the inotify API in the Linux kernel which shares the same behavior.

Queued items must be immutable and hashable so that they can be used as dictionary keys or added to sets. Items must have only read-only properties and must implement the __hash__(), __eq__(), and __ne__() methods to be hashable.

Author:	Yesudeep Manglapilly <yesudeep@gmail.com>
Author:	Lukáš Lalinský <lalinsky@gmail.com>

An example item class implementation follows:

class QueuedItem(object):
    def __init__(self, a, b):
        self._a = a
        self._b = b

    @property
    def a(self):
        return self._a

    @property
    def b(self):
        return self._b

    def _key(self):
        return (self._a, self._b)

    def __eq__(self, item):
        return self._key() == item._key()

    def __ne__(self, item):
        return self._key() != item._key()

    def __hash__(self):
        return hash(self._key())

class mom.collections.AttributeDict(*args, **kw)¶

A dictionary with attribute-style access. It maps attribute access to the real dictionary.

Subclass properties will override dictionary keys.

Author:	Alice Zoë Bevan–McGregor
License:	MIT License.

mom.collections.attrdict¶: alias of AttributeDict

synopsis:	Decorators used throughout the library.
module:	mom.decorators

mom.decorators.deprecated(func)¶

Marks functions as deprecated.

This is a decorator which can be used to mark functions as deprecated. It will result in a warning being emitted when the function is used.

Usage:

@deprecated
def my_func():
    pass

@other_decorators_must_be_upper
@deprecated
def my_func():
    pass

Parameters:	func – The function to deprecate.
Returns:	Deprecated function object.

synopsis:	Functional programming primitives.
module:	mom.functional

Higher-order functions¶

These functions accept other functions as arguments and apply them over specific types of data structures. Here’s an example of how to find the youngest person and the oldest person from among people. Place it into a Python module and run it:

import pprint

from mom import functional

PEOPLE = [
    {"name": "Harry",    "age": 100},
    {"name": "Hermione", "age": 16},
    {"name": "Rob",      "age": 200},
]

def youngest(person1, person2):
    '''Comparator that returns the youngest of two people.'''
    return person1 if person1["age"] <= person2["age"] else person2

def oldest(person1, person2):
    '''Comparator that returns the oldest of two people.'''
    return person1 if person1["age"] >= person2["age"] else person2

who_youngest = functional.reduce(youngest, PEOPLE)
who_oldest = functional.reduce(oldest, PEOPLE)

pprint.print(who_youngest)
# -> {"age" : 16, "name" : "Hermione"}
pprint.print(who_oldest)
# -> {"age" : 200, "name" : "Rob"}

# More examples.
# Now let's list all the names of the people.
names_of_people = functional.pluck(PEOPLE, "name")
pprint.print(names_of_people)
# -> ("Harry", "Hermione", "Rob")

# Let's weed out all people who don't have an "H" in their names.
pprint.print(functional.reject(lambda name: "H" not in name,
                               names_of_people))
# -> ("Harry", "Hermione")

# Or let's partition them into two groups
pprint.print(functional.partition(lambda name: "H" in name,
                                  names_of_people))
# -> (["Harry", "Hermione"], ["Rob"])

# Let's find all the members of a module that are not exported to wildcard
# imports by its ``__all__`` member.
pprint.print(functional.difference(dir(functional), functional.__all__))
# -> ["__all__",
#     "__author__",
#     "__builtins__",
#     "__doc__",
#     "__file__",
#     "__name__",
#     "__package__",
#     "_compose",
#     "_contains_fallback",
#     "_get_iter_next",
#     "_ifilter",
#     "_ifilterfalse",
#     "_leading",
#     "_some1",
#     "_some2",
#     "absolute_import",
#     "builtins",
#     "chain",
#     "collections",
#     "functools",
#     "itertools",
#     "map",
#     "starmap"]

Higher-order functions are extremely useful where you want to express yourself succinctly instead of writing a ton of for and while loops.

Warning

About consuming iterators multiple times

Now before you go all guns blazing with this set of functions, please note that Python generators/iterators are for single use only. Attempting to use the same iterator multiple times will cause unexpected behavior in your code.

Be careful.

Terminology¶

A predicate is a function that returns the truth value of its argument.
A complement is a predicate function that returns the negated truth value of its argument.
A walker is a function that consumes one or more items from a sequence at a time.
A transform is a function that transforms its arguments to produce a result.
Lazy evaluation is evaluation delayed until the last possible instant.
Materialized iterables are iterables that take up memory equal to their size.
Dematerialized iterables are iterables (usually iterators/generators) that are evaluated lazily.

Iteration and aggregation¶

mom.functional.each(walker, iterable)¶

Iterates over iterable yielding each item in turn to the walker function.

Parameters:	walker – The method signature is as follows: f(x, y) where `x, y` is a `key, value` pair if iterable is a dictionary, otherwise `x, y` is an `index, item` pair. iterable – Iterable sequence or dictionary.

mom.functional.reduce(transform, iterable, *args)¶

Aggregate a sequence of items into a single item. Python equivalent of Haskell’s left fold.

Please see Python documentation for reduce. There is no change in behavior. This is simply a wrapper function.

If you need reduce_right (right fold):

reduce_right = foldr = lambda f, i: lambda s: reduce(f, s, i)

Parameters:	transform – Function with signature: f(x, y) iterable – Iterable sequence. args – Initial value.
Returns:	Aggregated item.

Logic and search¶

mom.functional.every(predicate, iterable)¶

Determines whether the predicate is true for all elements in the iterable.

Parameters:	predicate – Predicate function of the form: f(x) -> bool iterable – Iterable sequence.
Returns:	`True` if the predicate is true for all elements in the iterable.

mom.functional.find(predicate, iterable, start=0)¶

Determines the first index where the predicate is true for an element in the iterable.

Parameters:	predicate – Predicate function of the form: f(x) -> bool iterable – Iterable sequence. start – Start index.
Returns:	-1 if not found; index (>= start) if found.

mom.functional.none(predicate, iterable)¶

Determines whether the predicate is false for all elements in in iterable.

Parameters:	predicate – Predicate function of the form: f(x) -> bool iterable – Iterable sequence.
Returns:	`True` if the predicate is false for all elements in the iterable.

mom.functional.some(predicate, iterable)¶

Determines whether the predicate applied to any element of the iterable is true.

Parameters:	predicate – Predicate function of the form: f(x) -> bool iterable – Iterable sequence.
Returns:	`True` if the predicate applied to any element of the iterable is true; `False` otherwise.

Filtering¶

mom.functional.ireject(predicate, iterable)¶

Reject all items from the sequence for which the predicate is true.

ireject(function or None, sequence) –> iterator

Parameters:	predicate – Predicate function. If `None`, reject all truthy items. iterable – Iterable to filter through.
Yields:	A sequence of all items for which the predicate is false.

mom.functional.iselect(predicate, iterable)¶

Select all items from the sequence for which the predicate is true.

iselect(function or None, sequence) –> iterator

Parameters:	predicate – Predicate function. If `None`, select all truthy items. iterable – Iterable.
Yields:	A iterable of all items for which the predicate is true.

mom.functional.partition(predicate, iterable)¶

Partitions an iterable into two iterables where for the elements of one iterable the predicate is true and for those of the other it is false.

Parameters:	predicate – Function of the format: f(x) -> bool iterable – Iterable sequence.
Returns:	Tuple (selected, rejected)

mom.functional.reject(predicate, iterable)¶

Reject all items from the sequence for which the predicate is true.

reject(function or None, sequence) -> list

Parameters:	predicate – Predicate function. If `None`, reject all truthy items. iterable – The iterable to filter through.
Returns:	A list of all items for which the predicate is false.

mom.functional.select(predicate, iterable)¶

Select all items from the sequence for which the predicate is true.

select(function or None, sequence) -> list

Parameters:	predicate – Predicate function. If `None`, select all truthy items. iterable – Iterable.
Returns:	A list of all items for which the predicate is true.

Counting¶

mom.functional.leading(predicate, iterable, start=0)¶

Returns the number of leading elements in the iterable for which the predicate is true.

Parameters:	predicate – Predicate function of the form: f(x) -> bool iterable – Iterable sequence. start – Start index. (Number of items to skip before starting counting.)

mom.functional.tally(predicate, iterable)¶

Count how many times the predicate is true.

Taken from the Python documentation. Under the PSF license.

Parameters:	predicate – Predicate function. iterable – Iterable sequence.
Returns:	The number of times a predicate is true.

mom.functional.trailing(predicate, iterable, start=-1)¶

Returns the number of trailing elements in the iterable for which the predicate is true.

Parameters:	predicate – Predicate function of the form: f(x) -> bool iterable – Iterable sequence. start – If start is negative, -1 indicates starting from the last item. Therefore, -2 would mean start counting from the second last item. If start is 0 or positive, it indicates the number of items to skip before beginning to count.

Function-generators¶

mom.functional.complement(predicate)¶

Generates a complementary predicate function for the given predicate function.

Parameters:	predicate – Predicate function.
Returns:	Complementary predicate function.

mom.functional.compose(function, *functions)¶

Composes a sequence of functions such that:

compose(g, f, s) -> g(f(s()))

Parameters:	functions – An iterable of functions.
Returns:	A composition function.

Iterators¶

These functions take iterators as arguments.

mom.functional.eat(iterator, amount)¶

Advance an iterator n-steps ahead. If n is None, eat entirely.

Taken from the Python documentation. Under the PSF license.

Parameters:	iterator – An iterator. amount – The number of steps to advance.
Yields:	An iterator.

Iterable sequences¶

These functions allow you to filter, manipulate, slice, index, etc. iterable sequences.

Indexing and slicing¶

mom.functional.chunks(iterable, size, *args, **kwargs)¶

Splits an iterable into materialized chunks each of specified size.

Parameters:

iterable – The iterable to split. Must be an ordered sequence to guarantee order.
size – Chunk size.
padding –
This must be an iterable or None. So if you want a True filler, use [True] or (True, ) depending on whether the iterable is a list or a tuple. Essentially, it must be the same type as the iterable.

If a pad value is specified appropriate multiples of it will be concatenated at the end of the iterable if the size is not an integral multiple of the length of the iterable:

tuple(chunks(“aaabccd”, 3, “-”)) -> (“aaa”, “bcc”, “d–”)
tuple(chunks((1, 1, 1, 2, 2), 3, (None,))) -> ((1, 1, 1, ), (2, 2, None))

If no padding is specified, nothing will be appended if the chunk size is not an integral multiple of the length of the iterable. That is, the last chunk will have chunk size less than the specified chunk size. :yields: Generator of materialized chunks.

mom.functional.head(iterable)¶

Returns the first element out of an iterable.

Parameters:	iterable – Iterable sequence.
Returns:	First element of the iterable sequence.

mom.functional.ichunks(iterable, size, *args, **kwargs)¶

Splits an iterable into iterators for chunks each of specified size.

Parameters:

iterable – The iterable to split. Must be an ordered sequence to guarantee order.
size – Chunk size.
padding –
If a pad value is specified appropriate multiples of it will be appended to the end of the iterator if the size is not an integral multiple of the length of the iterable:

map(tuple, ichunks(“aaabccd”, 3, “-”)) -> [(“a”, “a”, “a”), (“b”, “c”, “c”), (“d”, “-”, “-”)]
map(tuple, ichunks(“aaabccd”, 3, None)) -> [(“a”, “a”, “a”), (“b”, “c”, “c”), (“d”, None, None)]

If no padding is specified, nothing will be appended if the chunk size is not an integral multiple of the length of the iterable. That is, the last chunk will have chunk size less than the specified chunk size. :yields: Generator of chunk iterators.

mom.functional.ipeel(iterable, count=1)¶

Returns an iterator for the meat of an iterable by peeling off the specified number of elements from both ends.

Parameters:	iterable – Iterable sequence. count – The number of elements to remove from each end.
Yields:	Peel iterator.

mom.functional.itail(iterable)¶

Returns an iterator for all elements excluding the first out of an iterable.

Parameters:	iterable – Iterable sequence.
Yields:	Iterator for all elements of the iterable sequence excluding the first.

mom.functional.last(iterable)¶

Returns the last element out of an iterable.

Parameters:	iterable – Iterable sequence.
Returns:	Last element of the iterable sequence.

mom.functional.nth(iterable, index, default=None)¶

Returns the nth element out of an iterable.

Parameters:	iterable – Iterable sequence. index – Index default – If not found, this or `None` will be returned.
Returns:	nth element of the iterable sequence.

mom.functional.peel(iterable, count=1)¶

Returns the meat of an iterable by peeling off the specified number of elements from both ends.

Parameters:	iterable – Iterable sequence. count – The number of elements to remove from each end.
Returns:	Peeled sequence.

mom.functional.tail(iterable)¶

Returns all elements excluding the first out of an iterable.

Parameters:	iterable – Iterable sequence.
Returns:	All elements of the iterable sequence excluding the first.

mom.functional.round_robin(*iterables)¶

Returns items from the iterables in a round-robin fashion.

Taken from the Python documentation. Under the PSF license. Recipe credited to George Sakkis

Example:

round_robin("ABC", "D", "EF") --> A D E B F C"

Parameters:	iterables – Variable number of inputs for iterable sequences.
Yields:	Items from the iterable sequences in a round-robin fashion.

mom.functional.take(iterable, amount)¶

Return first n items of the iterable as a tuple.

Taken from the Python documentation. Under the PSF license.

Parameters:	amount – The number of items to obtain. iterable – Iterable sequence.
Returns:	First n items of the iterable as a tuple.

mom.functional.ncycles(iterable, times)¶

Yields the sequence elements n times.

Taken from the Python documentation. Under the PSF license.

Parameters:	iterable – Iterable sequence. times – The number of times to yield the sequence.
Yields:	Iterator.

mom.functional.occurrences(iterable)¶

Returns a dictionary of counts (multiset) of each element in the iterable.

Taken from the Python documentation under PSF license.

Parameters:	iterable – Iterable sequence with hashable elements.
Returns:	A dictionary of counts of each element in the iterable.

Manipulation, filtering¶

mom.functional.contains(iterable, item)¶

Determines whether the iterable contains the value specified.

Parameters:	iterable – Iterable sequence. item – The value to find.
Returns:	`True` if the iterable sequence contains the value; `False` otherwise.

mom.functional.omits(iterable, item)¶

Determines whether the iterable omits the value specified.

Parameters:	iterable – Iterable sequence. item – The value to find.
Returns:	`True` if the iterable sequence omits the value; `False` otherwise.

mom.functional.falsy(iterable)¶

Returns a iterable with only the falsy values.

Example:

falsy((0, 1, 2, False, None, True)) -> (0, False, None)

Parameters:	iterable – Iterable sequence.
Returns:	Iterable with falsy values.

mom.functional.ifalsy(iterable)¶

Returns a iterator for an iterable with only the falsy values.

Example:

tuple(ifalsy((0, 1, 2, False, None, True))) -> (0, False, None)

Parameters:	iterable – Iterable sequence.
Yields:	Iterator for an iterable with falsy values.

mom.functional.itruthy(iterable)¶

Returns an iterator to for an iterable with only the truthy values.

Example:

tuple(itruthy((0, 1, 2, False, None, True))) -> (1, 2, True)

Parameters:	iterable – Iterable sequence.
Yields:	Iterator for an iterable with truthy values.

mom.functional.truthy(iterable)¶

Returns a iterable with only the truthy values.

Example:

truthy((0, 1, 2, False, None, True)) -> (1, 2, True)

Parameters:	iterable – Iterable sequence.
Returns:	Iterable with truthy values.

mom.functional.without(iterable, *values)¶

Returns the iterable without the values specified.

Parameters:	iterable – Iterable sequence. values – Variable number of input values.
Returns:	Iterable sequence without the values specified.

Flattening, grouping, unions, differences, and intersections¶

mom.functional.flatten(iterable)¶

Flattens nested iterables into a single iterable.

Example:

flatten((1, (0, 5, ("a", "b")), (3, 4))) -> [1, 0, 5, "a", "b", 3, 4]

Parameters:	iterable – Iterable sequence of iterables.
Returns:	Iterable sequence of items.

mom.functional.flatten1(iterable)¶

Flattens nested iterables into a single iterable only one level deep.

Example:

flatten1((1, (0, 5, ("a", "b")), (3, 4))) -> [1, 0, 5, ("a", "b"), 3, 4]

Parameters:	iterable – Iterable sequence of iterables.
Returns:	Iterable sequence of items.

mom.functional.group_consecutive(predicate, iterable)¶

Groups consecutive elements into subsequences:

things = [("phone", "android"),
          ("phone", "iphone"),
          ("tablet", "ipad"),
          ("laptop", "dell studio"),
          ("phone", "nokia"),
          ("laptop", "macbook pro")]

list(group_consecutive(lambda w: w[0], things))
-> [(("phone", "android"), ("phone", "iphone")),
    (("tablet", "ipad"),),
    (("laptop", "dell studio"),),
    (("phone", "nokia"),),
    (("laptop", "macbook pro"),)]

list(group_consecutive(lambda w: w[0], "mississippi"))
-> [("m",), ("i",),
    ("s", "s"), ("i",),
    ("s", "s"), ("i",),
    ("p", "p"), ("i",)]

Parameters:	predicate – Predicate function that returns `True` or `False` for each element of the iterable. iterable – An iterable sequence of elements.
Returns:	An iterator of lists.

mom.functional.flock(predicate, iterable)¶

Groups elements into subsequences after sorting:

things = [("phone", "android"),
          ("phone", "iphone"),
          ("tablet", "ipad"),
          ("laptop", "dell studio"),
          ("phone", "nokia"),
          ("laptop", "macbook pro")]

list(flock(lambda w: w[0], things))
-> [(("laptop", "dell studio"), ("laptop", "macbook pro")),
    (("phone", "android"), ("phone", "iphone"), ("phone", "nokia")),
    (("tablet", "ipad"),)]

list(flock(lambda w: w[0], "mississippi"))
-> [("i", "i", "i", "i"), ("m",), ("p", "p"), ("s", "s", "s", "s")]

Parameters:	predicate – Predicate function that returns `True` or `False` for each element of the iterable. iterable – An iterable sequence of elements.
Returns:	An iterator of lists.

mom.functional.intersection(iterable, *iterables)¶

Returns the intersection of given iterable sequences.

Parameters:	iterables – Variable number of input iterable sequences.
Returns:	Intersection of the iterable sequences in the order of appearance in the first sequence.

mom.functional.idifference(iterable1, iterable2)¶

Difference between one iterable and another. Items from the first iterable are included in the difference.

iterable1 - iterable2 = difference

Parameters:	iterable1 – Iterable sequence. iterable2 – Iterable sequence.
Yields:	Generator for the difference between the two given iterables.

mom.functional.difference(iterable1, iterable2)¶

Difference between one iterable and another. Items from the first iterable are included in the difference.

iterable1 - iterable2 = difference

For example, here is how to find out what your Python module exports to other modules using wildcard imports:

>> difference(dir(mom.functional), mom.functional.__all__)
["__all__",
 # Elided...
 "range",
 "takewhile"]

Parameters:	iterable1 – Iterable sequence. iterable2 – Iterable sequence.
Returns:	Iterable sequence containing the difference between the two given iterables.

mom.functional.union(iterable, *iterables)¶

Returns the union of given iterable sequences.

Parameters:	iterables – Variable number of input iterable sequences.
Returns:	Union of the iterable sequences.

mom.functional.unique(iterable, is_sorted=False)¶

Returns an iterable sequence of unique values from the given iterable.

Parameters:	iterable – Iterable sequence. is_sorted – Whether the iterable has already been sorted. Works faster if it is.
Returns:	Iterable sequence of unique values.

Dictionaries and dictionary sequences¶

mom.functional.invert_dict(dictionary)¶

Inverts a dictionary.

Parameters:	dictionary – Dictionary to invert.
Returns:	New dictionary with the keys and values switched.

mom.functional.ipluck(dicts, key, *args, **kwargs)¶

Plucks values for a given key from a series of dictionaries as an iterator.

Parameters:	dicts – Iterable sequence of dictionaries. key – The key to fetch. default – The default value to use when a key is not found. If this value is not specified, a KeyError will be raised when a key is not found.
Yields:	Iterator of values for the key.

mom.functional.map_dict(transform, dictionary)¶

Maps over a dictionary of key, value pairs.

Parameters:	transform – Function that accepts two arguments `key, value` and returns a `(new key, new value)` pair.
Returns:	New dictionary of `new key=new value` pairs.

mom.functional.pluck(dicts, key, *args, **kwargs)¶

Plucks values for a given key from a series of dictionaries.

Parameters:	dicts – Iterable sequence of dictionaries. key – The key to fetch. default – The default value to use when a key is not found. If this value is not specified, a KeyError will be raised when a key is not found.
Returns:	Tuple of values for the key.

mom.functional.reject_dict(predicate, dictionary)¶

Reject from a dictionary.

Parameters:	predicate – Predicate function that accepts two arguments `key, value` and returns `True` for rejected elements.
Returns:	New dictionary of selected `key=value` pairs.

mom.functional.select_dict(predicate, dictionary)¶

Select from a dictionary.

Parameters:	predicate – Predicate function that accepts two arguments `key, value` and returns `True` for selectable elements.
Returns:	New dictionary of selected `key=value` pairs.

mom.functional.partition_dict(predicate, dictionary)¶

Partitions a dictionary into two dictionaries where for the elements of one dictionary the predicate is true and for those of the other it is false.

Parameters:	predicate – Function of the format: f(key, value) -> bool dictionary – Dictionary.
Returns:	Tuple (selected_dict, rejected_dict)

Predicates, transforms, and walkers¶

mom.functional.identity(arg)¶

Identity function. Produces what it consumes.

Parameters:	arg – Argument
Returns:	Argument.

mom.functional.loob(arg)¶

Complement of bool.

Parameters:	arg – Python value.
Returns:	Complementary boolean value.

mom.functional.always(_)¶

Predicate function that returns True always.

Parameters:	_ – Argument
Returns:	`True`.

mom.functional.never(_)¶

Predicate function that returns False always.

Parameters:	_ – Argument
Returns:	`False`.

synopsis:	Implements `itertools` for older versions of Python.
module:	mom.itertools
copyright:	2010-2011 by Daniel Neuhäuser
license:	BSD, PSF

Borrowed from brownie.itools.

synopsis:	Math routines.
module:	mom.math

Math¶

mom.math.gcd(num_a, num_b)¶

Calculates the greatest common divisor.

Non-recursive fast implementation.

Parameters:	num_a – Long value. num_b – Long value.
Returns:	Greatest common divisor.

mom.math.inverse_mod(num_a, num_b)¶

Returns inverse of a mod b, zero if none

Uses Extended Euclidean Algorithm

Parameters:	num_a – Long value num_b – Long value
Returns:	Inverse of a mod b, zero if none.

mom.math.lcm(num_a, num_b)¶

Least common multiple.

Parameters:	num_a – Integer value. num_b – Integer value.
Returns:	Least common multiple.

mom.math.pow_mod(base, power, modulus)¶

Calculates:: base**pow mod modulus

Uses multi bit scanning with nBitScan bits at a time. From Bryan G. Olson’s post to comp.lang.python

Does left-to-right instead of pow()’s right-to-left, thus about 30% faster than the python built-in with small bases

Parameters:	base – Base power – Power modulus – Modulus
Returns:	base**pow mod modulus

Primes¶

mom.math.generate_random_prime(bits)¶

Generates a random prime number.

Parameters:	bits – Number of bits.
Returns:	Prime number long value.

mom.math.generate_random_safe_prime(bits)¶

Unused at the moment.

Generates a random prime number.

Parameters:	bits – Number of bits.
Returns:	Prime number long value.

mom.math.is_prime(num, iterations=5, sieve=sieve)¶

Determines whether a number is prime.

Parameters:	num – Number iterations – Number of iterations.
Returns:	`True` if prime; `False` otherwise.

synopsis:	MIME-Type Parser.
module:	mom.mimeparse

This module provides basic functions for handling mime-types. It can handle matching mime-types against a list of media-ranges. See section 14.1 of the HTTP specification [RFC 2616] for a complete explanation.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1

Contents¶

mom.mimeparse.parse_mime_type(mime_type)¶

Parses a mime-type into its component parts.

Parameters:	mime_type – Mime type as a byte string.
Returns:	A tuple of the (type, subtype, params) where ‘params’ is a dictionary of all the parameters for the media range. For example, the media range b’application/xhtml;q=0.5’ would get parsed into: (b’application’, b’xhtml’, {‘q’: b‘0.5’})

mom.mimeparse.parse_media_range(media_range)¶

Parse a media-range into its component parts.

Parameters: media_range – Media range as a byte string.

Returns:

A tuple of the (type, subtype, params) where ‘params’ is a dictionary of all the parameters for the media range. For example, the media range b’application/xhtml;q=0.5’ would get parsed into:

(b’application’, b’xhtml’, {‘q’: b‘0.5’})

In addition this function also guarantees that there is a value for ‘q’ in the params dictionary, filling it in with a proper default if necessary.

mom.mimeparse.quality(mime_type, ranges)¶

Return the quality (‘q’) of a mime-type against a list of media-ranges.

For example:

>>> Quality(b'text/html',b'text/*;q=0.3, text/html;q=0.7,
              text/html;level=1, text/html;level=2;q=0.4, */*;q=0.5')
0.7

Parameters:	mime_type – The mime-type to compare. ranges – A list of media ranges.
Returns:	Returns the quality ‘q’ of a mime-type when compared against the media-ranges in ranges.

mom.mimeparse.quality_parsed(mime_type, parsed_ranges)¶

Find the best match for a mime-type amongst parsed media-ranges.

Parameters:	mime_type – The mime-type to compare. parsed_ranges – A list of media ranges that have already been parsed by parsed_media_range().
Returns:	The ‘q’ quality parameter of the best match, 0 if no match was found.

mom.mimeparse.best_match(supported, header)¶

Return mime-type with the highest quality (‘q’) from list of candidates.

Takes a list of supported mime-types and finds the best match for all the media-ranges listed in header.

>>> BestMatch([b'application/xbel+xml', b'text/xml'],
               b'text/*;q=0.5,*/*; q=0.1')
b'text/xml'

Parameters:	supported – A list of supported mime-types. The list of supported mime-types should be sorted in order of increasing desirability, in case of a situation where there is a tie. header – Atring that conforms to the format of the HTTP Accept: header.
Returns:	Mime-type with the highest quality (‘q’) from the list of candidates.

synopsis:	string module compatibility.
module:	mom.string

Codecs¶

mom.codec¶

synopsis:	Many different types of common encode/decode function.
module:	mom.codec

This module contains codecs for converting between hex, base64, base85, base58, base62, base36, decimal, and binary representations of bytes.

Understand that bytes are simply base-256 representation. A PNG file:

\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00
\x05\x00\x00\x00\x05\x08\x06\x00\x00\x00\x8do&
\xe5\x00\x00\x00\x1cIDAT\x08\xd7c\xf8\xff\xff?
\xc3\x7f\x06 \x05\xc3 \x12\x84\xd01\xf1\x82X\xcd
\x04\x00\x0e\xf55\xcb\xd1\x8e\x0e\x1f\x00\x00\x00
\x00IEND\xaeB`\x82

That is what an example PNG file looks like as a stream of bytes (base-256) in Python (with line-breaks added for visual-clarity).

If we wanted to send this PNG within an email message, which is restricted to ASCII characters, we cannot simply add these bytes in and hope they go through unchanged. The receiver at the other end expects to get a copy of exactly the same bytes that you send. Because we are limited to using ASCII characters, we need to “encode” this binary data into a subset of ASCII characters before transmitting, and the receiver needs to “decode” those ASCII characters back into binary data before attempting to display it.

Base-encoding raw bytes into ASCII characters is used to safely transmit binary data through any medium that does not inherently support non-ASCII data.

Therefore, we need to convert the above PNG binary data into something that looks like (again, line-breaks have been added for visual clarity only):

iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI
12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJg
gg==

The base-encoding method that we can use is limited by these criteria:

The number of ASCII characters, a subset of ASCII, that we can use to represent binary data (case-sensitivity, ambiguity, base, deviation from standard characters, etc.)
Whether human beings are involved in the transmission of data. Ergo, visual clarity, legibility, readability, human-inputability, and even double-click-to-select-ability! (Hint: try double-clicking the encoded data above to see whether it selects all of it–it won’t). This is a corollary for point 1.
Whether we want the process to be more time-efficient or space-efficient. That is, whether we can process binary data in chunks or whether we need to convert it into an arbitrarily large integer before encoding, respectively.

Terminology¶

Answer this question:

How many times should I multiply 2 by itself to obtain 8?

You say:

That’s a dumb question. 3 times!

Well, congratulations! You have just re-discovered logarithms. In a system of equations, we may have unknowns. Given an equation with 3 parts, 2 of which are known, we often need to find the 3rd. Logarithms are used when you know the base (radix) and the number, but not the exponent.

Logarithms help you find exponents.

Take for example:

2**0 = 1
2**1 = 2
2**2 = 4
2**3 = 8
2**4 = 16
2**5 = 32
2**6 = 64

Alternatively, logarithms can be thought of as answering the question:

Raising 2 to which exponent gets me 64?

This is the same as doing:

import math
math.log(64, 2)    # 6.0; number 64, base 2.
6.0

read as “logarithm to the base 2 of 64” which gives 6. That is, if we raise 2 to the power 6, we get 64.

The concept of roots or radicals is also related. Roots help you find the base (radix) given the exponent and the number. So:

root(8, 3)   # 2.0; cube root. exponent 3, number 8.

Roots help you find the base.

Hopefully, that brings you up to speed and enables you to clearly see the relationship between powers, logarithms, and roots.

We will often refer to the term byte and mean it to be an octet (8) of bits. The number of bits in a byte is dependent on the processor architecture. Therefore, we can have a 9-bit byte or even a 36-bit byte.

For our purposes, however, a byte means a chunk of 8 bits–that is, an octet.

By the term “encoding,” throughout this discussion, we mean a way of representing a sequence of bytes in terms of a subset of US-ASCII characters, each of which uses 7 bits. This ensures that in communication and messaging that involves the transmission of binary data, at a small increase in encoded size, we can safely transmit this data encoded as ASCII text. We could be pedantic and use the phrase “ASCII-subset-based encoding” everywhere, but we’ll simply refer to it as “encoding” instead.

How it applies to encodings¶

Byte, or base-256, representation allows each byte to be represented using one of 256 values (0-255 inclusive). Modern processors can process data in chunks of 32 bits (4 bytes), 64 bits (8 bytes), and so on. Notice that these are powers of 2 given that our processors are binary machines.

We could feed a 64-bit processor with 8 bits of data at a time, but that would guarantee that the codec will be only 1/8th as time-efficient as it can be. That is, if you feed the same 64-bit processor with 64 bits of data at a time instead, the encoding process will be 8 times as fast. Whoa!

Therefore, in order to ensure that our codecs are fast, we need to feed our processors data in chunks to be more time-efficient. The two types of encoding we discuss here are:

big-integer-based polynomial-time base-conversions
chunked linear-time base-conversions.

These two types of encoding are not always compatible with each other.

Big-integer based encoding¶

This method of encoding is generally costlier because the raw bytes (base-256 representation) are first converted into a big integer, which is then subsequently repeatedly divided to obtain an encoded sequence of bytes. Bases 58, 60, and 62 are not powers of 2, and therefore cannot be reliably or efficiently encoded in chunks of powers of 2 (used by microprocessors) so as to produce the same encoded representations as their big integer encoded representations. Therefore, using these encodings for a large amount of binary data is not advised. The base-58 and base-62 modules in this library are meant to be used with small amounts of binary data.

Chunked encoding¶

Base encoding a chunk of 4 bytes at a time (32 bits at a time) means we would need a way to represent each of the 256**4 (4294967296) values with our encoding:

256**4 # 4294967296
2**32  # 4294967296

Given an encoding alphabet of 85 ASCII characters, for example, we need to find an exponent (logarithm) that allows us to represent each one of these 4294967296 values:

85**4 # 52200625
85**5 # 4437053125

>>> 85**5 >= 2**32
True

Done using logarithms:

import math
math.log(2**32, 85)   # 4.9926740807111996

Therefore, we would need 5 characters from this encoding alphabet to represent 4 bytes. Since 85 is not a power of 2, there is going to be a little wastage of space and the codec will need to deal with padding and de-padding bytes to ensure the resulting size to be a multiple of the chunk size, but the byte sequence will be more compact than its base-16 (hexadecimal) representation, for example:

import math
math.log(2**32, 16)   # 8.0

As you can see, if we used hexadecimal representation instead, each 4-byte chunk would be represented using 8 characters from the encoding alphabet. This is clearly less space-efficient than using 5 characters per 4 bytes of binary data.

Base-64 as another example¶

Base-64 allows us to represent 256**4 (4294967296) values using 64 ASCII characters.

Bytes base-encoding¶

These codecs preserve bytes “as is” when decoding back to bytes. In a more mathematical sense,

g(f(x)) is an identity function

where g is the decoder and f is the encoder.

Why have we reproduced base64 encoding/decoding functions here when the standard library has them? Well, those functions behave differently in Python 2.x and Python 3.x. The Python 3.x equivalents do not accept Unicode strings as their arguments, whereas the Python 2.x versions would happily encode your Unicode strings without warning you-you know that you are supposed to encode them to UTF-8 or another byte encoding before you base64-encode them right? These wrappers are re-implemented so that you do not make these mistakes. Use them. They will help prevent unexpected bugs.

mom.codec.base85_encode(raw_bytes, charset='ASCII85')¶

Encodes raw bytes into ASCII85 representation.

Encode your Unicode strings to a byte encoding before base85-encoding them.

Parameters:	raw_bytes – Bytes to encode. charset – “ASCII85” (default) or “RFC1924”.
Returns:	ASCII85 encoded string.

mom.codec.base85_decode(encoded, charset='ASCII85')¶

Decodes ASCII85-encoded bytes into raw bytes.

Parameters:	encoded – ASCII85 encoded representation. charset – “ASCII85” (default) or “RFC1924”.
Returns:	Raw bytes.

mom.codec.base64_encode(raw_bytes)¶

Encodes raw bytes into base64 representation without appending a trailing newline character. Not URL-safe.

Encode your Unicode strings to a byte encoding before base64-encoding them.

Parameters:	raw_bytes – Bytes to encode.
Returns:	Base64 encoded bytes without newline characters.

mom.codec.base64_decode(encoded)¶

Decodes base64-encoded bytes into raw bytes. Not URL-safe.

Parameters:	encoded – Base-64 encoded representation.
Returns:	Raw bytes.

mom.codec.base64_urlsafe_encode(raw_bytes)¶

Encodes raw bytes into URL-safe base64 bytes.

Encode your Unicode strings to a byte encoding before base64-encoding them.

Parameters:	raw_bytes – Bytes to encode.
Returns:	Base64 encoded string without newline characters.

mom.codec.base64_urlsafe_decode(encoded)¶

Decodes URL-safe base64-encoded bytes into raw bytes.

Parameters:	encoded – Base-64 encoded representation.
Returns:	Raw bytes.

mom.codec.base62_encode(raw_bytes)¶

Encodes raw bytes into base-62 representation. URL-safe and human safe.

Encode your Unicode strings to a byte encoding before base-62-encoding them.