pygen - Python-style generators library

The library is a set of macros tries to emulate python's generators functionality in lisp.

One difference from the python's generators it is impossible to save a generator to a variable and call 'next' method manually. Only 'foreach' syntax is supported.

How it works

The reason of missed 'next' method is that pygen does not use real continuations to implement generators (as python does). It uses continuations idea only. Library's macros perform CPS transformation for FOREACH and YIELD forms.

Let see the code:

(defgenerator gen1 ()
  (yield 1)
  (yield 2)
  (yield 3))

(defun use-generator ()
  (foreach x in (gen1)
    (print x)))

Normal thinking of the code is: OK, call gen1(), save the return value into a variable 'x' and call print(x); call gen1() again and do the same; cal gen1() again, again, again... In short, 'foreach' is a master and 'generator' is a slave - master calls slave while slave has a job to do.

But we can thing a little bit different: OK, we know what we are going to do with the generator's value - run the body of FOREACH form ((print x) in our case). So, it is possible to pass that code to a generator and the generator will call it when needed. The generator becomes a master and foreach becomes a slave. We can think about a generator as a normal function that receives additional parameter - what to do with the result value aka continuation. By pygen's macros the code above is transformed into something like this:

(defun gen1 (*cont*)
  (funcall *cont* 1)
  (funcall *cont* 2)
  (funcall *cont* 3))

(defun use-generator ()
  (gen1 #'(lambda (x) (print x))))

When to use

If a generator function is simple that means it is relatively easy to save and and restore the function's state in a closure it might be more appropriate to use series package or some other lazy-lists package. For someone it could be more convenient to pass a functional parameter represented a continuation to a generator directly. pygen is doing well when a generator function is a quite complex, e.g. there are several inner loops or when it is needed to work with 'unwind-protect' resources such as files. For example, SAX and pull XML parsers could be implemented using generators.

There's another very efficient approach that generates tagbody and go statements in the generator function and local macros in order to change control structures like if, progn and loops to flat forms whose bodies may be reached by the go statements: (for Emacs-Lisp, but it could easily be translated to Common Lisp). This approach even works for mutually recursive generator functions.


pygen can be downloaded directly from here


Generators are an abstraction. If it is suitable for you - pygen is here :)


pygen is created by Vladimir Buzuev (vbuzuev[R2D2]


yield and break

(defgenerator prime-numbers ()
  (do ((i 2 (1+ i)) (primes nil)) (nil)
    (if (not (some #'(lambda (p) (= 0 (rem i p))) primes))
	(progn (push i primes) (yield i)))))

(test primes
  (let ((result nil))
    (foreach p in (prime-numbers)
      (push p result)
      (if (> p 20)
    (is (equal '(2 3 5 7 11 13 17 19 23) (nreverse result)))))

protected resources

(defvar *gen-calls* nil)

(defgenerator resource-protect-gen (throw-error)
         (yield 1)
         (yield 2)
         (if throw-error
             (error "error"))
	 (yield 3)
         (yield 4))
    (push 'finally *gen-calls*)))

(test resource-protect-1
  (let ((result nil))
    (setf *gen-calls* '(start))
    (foreach x in (resource-protect-gen nil)
      (push x result))
    (is (equal '(1 2 3 4) (nreverse result)))
    (is (equal '(start finally) (nreverse *gen-calls*)))))

(test resource-protect-2
  (let ((result nil))
    (setf *gen-calls* '(start))
	(foreach x in (resource-protect-gen t)
	  (push x result))
      (simple-error () (push 'error *gen-calls*)))
    (is (equal '(1 2) (nreverse result)))
    (is (equal '(start finally error) (nreverse *gen-calls*)))))

License: ?

See Also: cl-itertools, snakes

Topics: language extension, continuations, python