Stop abusing namespace packages

For about the past two years, Plone has been in a nosedive toward a development complexity asymptote that, if unchecked, will put a halt to new developer interest. I hate this. Every time I see a developer give up and start over, I mourn his or her inevitable reinvention of the tens of years of real-world-tested knowledge embodied in Plone's codebase.

To be sure, complexity is sometimes the price for solving a genuine problem, but today I'm going to hammer on one instance that's been almost purely a reckless running after shiny new toys. In particular, we have acquired an unhealthy habit of abusing namespace packages. How and why? Here's a primer on when we should use them and when they just make things more complicated.

What's a namespace package?

Namespace packages are those things like collective in collective.tagcloud or plone and app in plone.app.portlets. They are prefixes on the pathname of a Python module, implemented by wrapping an empty folder (called say, "collective") around it and plopping a short, magical __init__.py inside. The result is that you can distribute collective.foo and collective.bar as separate packages but still import them via import collective.foo and import collective.bar.

A much hairier and more comprehensive explanation is available in the documentation of setuptools, out of which namespace packages spring.

Good namespace packages: breaking up is easy to do

Namespace packages are great for breaking enormous legacy packages into smaller parts that can be maintained and distributed separately, without breaking people's imports. They were likely motivated by the gigantic PEAK framework which is by the same author as setuptools, Phillip Eby. As PEAK grew, Eby wanted a way to update subpackages like peak.config or peak.storage without having to ship an entire new 50MB tarball. However, he couldn't just change over to separately distributed modules called peak_config and peak_storage without breaking all the client code that said things like…

import peak.config

So, he made peak a namespace package and had the best of both worlds, the only cost being some occasional weird lines in an __init__.py.

Zope has a similar story. I'm guessing at the archeology, but it should be safe to say that zope started out manageably small and only later grew into a 68MB behemoth that was inconvenient to distribute as one gigantic tarball. Of course, by that point, enough people depended on it that they couldn't just change all the import paths around, so the zope package became a namespace package, too, giving us today's zope.index, zope.event, and so on. All well and good.

Bad namespace packages. Bad. Go sit in the corner.

However, for a variety of reasons, the Plone community has gone absolutely bonkers with namespace packages. We use them for branding, like the Simplon company did with simplon.plone.ldap. We use them for categorization, like plonetheme.nautica05. We even use them for no apparent reason at all, like the 315 products called collective.something. We've caused a lot of noise for no technical or usability benefit.

Filesystem paths: ouch!

Out of the box, if I want to get to the source code of Plone's redirection machinery, I have to go into plone.app.redirector-1.0.10-py2.4.egg, then into plone, then into app, then, finally, into the redirector folder, where all the code lives. The extra traversals are a waste of time. David Glick even got mad enough to cook up the omelette buildout extension to patch around the problem. Am I missing some reason we didn't simply go with plone_app_redirector and save 3 levels of folders and weird __init__.pys? It's not like we had legacy imports to placate.

Subversion hide and seek: ouch and a half!

In svn, the problem repeats itself with an additional twist. People eggifying classic products (our own FacultyStaffDirectory is guilty of this) often move their source to a new folder with "Products" prepended: Products.FacultyStaffDirectory. I can understand the use of a Products namespace package for compatibility with Zope's old import magic, but is it really necessary for 50% of the stuff in a repository to start with "Products"? This kills typeahead, and, more importantly, makes developers look in two places for the code: "Hmm, let's look in FacultyStaffDirectory. No trunk folder! Ah, a README-trunk.txt: 'Your princess is in another castle.' Okay, I'll back out of here and try Products.FacultyStaffDirectory. Those are 2 minutes I'll never get back." Sure, use the Products namespace if you must, but let's keep our svn directory names unsullied.

Branding: double ouch!

When namespace packages are used for branding, everything has to change when the company divests itself of the product. When Simplon donated simplon.plone.ldap to the community at large, it had to rename it to plone.app.ldap, thus breaking everybody's buildouts and imports. Let's keep company names out of package names.

Description: ouch a third time!

Namespaces are often misused to describe functionality, like in plonetheme.*. This is just as wrong as when I named AutoMemberMakerPasPlugin (which I've remedied in WebServerAuth) and is nothing more than a repetition of the horrors of file extensions. Actually, it's worse: at least extensions are postfix and don't impede type-to-select. We have perfectly good egg metadata where we can declare that a package is a Plone theme. In fact, the plone3_theme ZopeSkel/paster template spits out the keywords "web zope plone theme" by default. The plonetheme prefix is redundant right out of the box!

Let's fix this

A lot of the excess namespaces come from some bad ZopeSkel/paster templates which make it really hard to create an egg-based product without a namespace package. I certainly don't blame newbies for getting led down the wrong road, but a lot of us should know better. Let's clean up our act:

• Reserve namespace packages for their intended use: breaking up enormous legacy packages. I see a world market for maybe five namespace packages.
• Stop using the collective namespace. It conveys no useful information. It's just noise.
• When you use the Products namespace to eggify an old-style product, don't move your trunk. You're going to break people's checkouts when you delete the old trunk anyway, so you might as well just change it in place.
• Scribble your company name all over your readme and egg metadata, not in your package name. Learn from Apple, who is now stuck with an API full of NSThis and NSThat.
• Fix the ZopeSkel templates that spit out things like plonetheme.*. We can't expect themers to be Python import machinery wizards and fight it themselves.

Those of us with an understanding of namespace packages have a responsibility to use them properly—everyone else will follow, for better or worse. This is a battle in the complexity war that's technically easy to win and one that new users and developers see early in their Plone experiences. Let's swat this piece of low-hanging fruit down.

After you read the excellent comments below, check out my summary and response in Namespace packages: replies to replies.