Good-Py: Known Good Versions Manager for Python and zc.buildout

Good-Py is a web application hosted on Google App Engine that aims to help manage known-good version sets of Python packages.

Why do we need this?

In the Zope and Plone worlds, we depend on large set of eggs (in the setuptools sense) published on PyPI (pypi.python.org) and other indices. Because setuptools (and zc.buildout) will typically prefer the latest release for a given package, it is easy to end up in a situation where an application ends up pulling in a too-recent release that does not work well with the other packages in the system.

This problem can be solved with pinning all releases to a "known good set". With zc.buildout, this is typically achieved using a [versions] block in buildout.cfg like:

[versions]
zope.interface = 3.4.0
zope.component = 3.4.0

Such a versions block can be set in a separate file that is included from the buildout:

[buildout]
...
extends =
    http://dist.plone.org/release/3.3rc2/versions.cfg
    http://www.example.org/release/some-system/1.0/versions.cfg
versions = versions

[versions]
some.package = 1.2

This example shows some important aspects of the version pinning system:

• We can extend multiple configurations containing version information.
• The extended files can be loaded over the network.
• Each of the .cfg files contains a [versions] block. These are merged so that if a given package is listed in multiple places, the last-loaded file takes precedence.
• We can make 'local' overrides in the master buildout.cfg file.

In the example above, some-system is a fictional plugin to Plone that contains additional dependencies, including some newer versions than what are included in Plone.

What does Good-Py do?

The Good-Py application aims to simplify the management of known-good sets, by allowing a [versions] block to be calculated automatically based on various predicates. If 'some-system' were using Good-Py, the 'extends' block could look like this:

[buildout]
...
extends =
    http://dist.plone.org/release/3.3rc2/versions.cfg
    http://good-py.appspot.com/release/some-system:1.0?plone=3.0&archetypes=1.5.1
versions = versions

The web application at http://good-py.appspot.com/release calculates the versions block for the given version of the system (1.0). It is also possible to specify a target platform (Plone 3), to indicate that you need a version of the dependencies that will work with the given platform.

The versions block is calculated from a number of predicates of the form:

known-good: foo.bar = 1.2.3

This package is a known good version for the given version of the system. An unlimited number of such predicates may be stored against each system version. The most recent package version will be included.

known-bad: foo.bar <= 1.2

A version specifier for packages that definitely will not work. Again, any number of such predicates may be used. The various setuptools version qualifiers are allowed: e.g. <=, >=, <, >, and ==, and multiple versions can be combined using commas, e.g. foo.bar <= 1.2,>3.2.

extends: other-system = 1.0

Imports all predicates from another system at a given version.

The generated [versions] block will include:

• the latest version of each package listed in one or more 'known-good' predicates, including those imported via an 'extends' predicate
• excluding any versions that violate any 'known-bad' predicates from the target system/version, any extended system/version, or any listed platforms, including system/versions extended by those platforms.

Note that platforms specified on the query string are only considered as constraints. That is, the version set returned will not violate any constraints from the platform any system/version specified, but the known-good predicates from the platforms are not included either. You can always load the platform version explicitly with another 'extends' line, however.

How is it managed?

The Good-Py application includes an administrative interface with the following features:

• Create or edit a system
  • name
  • version
  • extended system
• Create or edit a version of a system
• Update known-good and known-bad predicates
• Paste a [versions] block to import known good predicates (such a file can be generated with the buildout.dumppickedversions extension to zc.buildout)
• Flush the cache (by default, each unique versions list is cached for up to an hour)

Security is managed with Google Accounts. A list of accounts with edit rights is maintained against each system. The application administrator can change any system or version, and may also flush the cache globally.

Who's responsible?

This application was created by Martin Aspeli (optilude at gmail dot com)