Macroator

Sun, 05 Aug 2018 07:00:00 -0000

this post was (mostly) written around the same time as the code, then sat unpublished for most of a year.

Macroator (a terrible combination of "macro" and "decorator") is a new library ~~I've written to push the boundaries of~~ there's no way to justify this horror. It provides AST-level macros in Python by abusing decorator syntax and import hooks.

You can find the source code here

Let's look at a simple example:

Here's a simple, somewhat bad implementation of factorial as a tail-recursive function. For fun, it also includes identity, which is simply the identity function. This is not necessary, but does make the later demonstration (slightly) more interesting.

# factorial.py
def identity(x):
  return x

@magic_trampoline
def factorial(n, m=1):
  if n < 2:
    return identity(m)
  else:
    return factorial(n-1, m*n)

We'll just import this and call it:

# combined.py
# this is where the magic happens
import macroator

from factorial import factorial

# let's lower the recursion limit a bit.
import sys
sys.setrecursionlimit(10)

# now, our tail recursive function should go to a stack depth equal to `n`.
# But thanks to our _magic_, this works.
print(factorial(50))

# The answer is right, too.
import math
assert factorial(50) == math.factorial(50)

Now of course, that shouldn't work. Our factorial will recursively call itself 50 times, but our call stack isn't allowed to get any larger than 10 functions deep.

However...

$ python combined.py 
30414093201713378043612608166064768844377641568960512000000000000

What?! How?!

import macroator installs an import hook that catches all later imports and, if the imported module is a python source file, parses it, then looks through the abstract syntax tree for magic decorators, such as @magic_trampoline.

When it finds one, it passes the AST of the function to an AST-rewriter. After the AST has been rewritten, it compiles it and turns it into a module, then returns that as the result of the import process.

The @magic_trampoline decorator gets mapped to TrampolineMacro, which searches the function for all return statements whose return value is a function call expression. It replaces those calls with dictionary literals containing all the information required to call the function. It also adds a new, regular decorator to the decorator list for the function. That decorator, ★trampoline_decorator¹, has a loop which checks the return value of the function for those special dictionaries. When it sees one, it performs the function call described by the dictionary, then repeats the process for its return value. If and when the function stops returning special dictionaries, it returns the final return value.

I'm so sorry.

Which you may note is not a valid Python identifer, guaranteeing no "real" code will have a name collision with it. ↩

LZJB

Thu, 06 Jul 2017 07:00:00 -0000

LZJB is a lossless compression algorithm created by Jeff Bonwick. Outside of the original source code, I only know of one description of the algorithm, in a short piece on Wikibooks.

From the description and a hundred or so sample input/output pairs, here is a clean description (with code!) of the decompressor.¹

Wikibooks says:

The LZJB decoder fetches a "literal/copy" bit from the compressed file.

This is only mostly true. The first byte of an LZJB stream is a control byte. Each bit in that byte is a "literal/copy" byte as described. Once all those bits have been handled (each one advancing the input stream by 1 or 2 bytes), another control byte is read and the process repeats.

A 0 bit means: read 1 byte from the input and append it to the output.

A 1 bit means: read 2 bytes from the input, the high 6 bits of the first byte are length, and the remaining 10 bits are distance. The 2 bits left from the length byte are the high bits of distance. Because it takes 2 bytes to encode a copy instruction, LZJB does not encode repeated sequences of length 1 or 2. Instead, length is incremented by 3, allowing sequences of length [3,66) to be expressed. Then:

Call the current end of the output end.
Seek distance bytes back from the current end of the output.
Copy length bytes starting from that position to the end of the output.
If length is greater than distance (as happens commonly), the bytes from distance to end are repeatedly appended until length is smaller than distance, then a final partial copy is performed.

def decompress(bs, size):
    bs = bytearray(bs)
    out = bytearray()
    blen = len(bs)
    pos = 0
    while pos < blen and len(out) < size:
        control = bs[pos]
        pos += 1
        for i in xrange(8):
            b = control & (1 << i) > 0
            if not pos < blen:
                break
            if not b:
                out.append(bs[pos])
                pos += 1
            else:
                length = (bs[pos] >> 2) + 3 
                distance = (bs[pos] & 0b11) << 8 | bs[pos+1]
                pos += 2
                backref = out[-distance:]
                lookup = backref * (length / distance) + backref[:(length % distance)]
                out += lookup
    return out[:size]

Implementing a compressor is left as an exercise to the interested reader. ↩

blog

Macroator

LZJB