Posted Jun 16, 2009
Namespace packages: replies to replies
Clarifications, your comments addressed, and why I still think we should lay off the namespace packages
Many thanks to all who posted such thoughtful replies to my apparently incendiary post on namespace packages. Since I was off-network last week (Grand Canyon—woo!) and not able to respond in realtime, I’ve taken the liberty of summarizing and addressing some salient points en masse below.
“Namespaces are useful for avoiding name collisions.”
Of course they are. I should been clearer in my main point, as a lot of folks ended up banging on an open door here. Namespaces—as an abstract concept—are lovely. What’s wrong is our use of setuptools’ implementation: as we use them, namespace packages are awfully noisy and obtrusive from a package consumer’s point of view—they make new users walk straight through their implementation details, both in svn and on disk, before they can get to any code.
But a lot of this is due to funny conventions. We use namespace packages in an unusual way in the Plone world: we never put anything but a single package inside them, making the outer folder mostly empty and mostly noise. We don’t take advantage of setuptools’ support for putting N submodules and packages inside a namespace package, which would make the signal-to-noise ratio more acceptable. So I wonder, can we come up with a shorter, fewer-empty-folders way of namespacing?
Indeed we can. One of my proposed solutions, using names like plone_app_redirector instead of plone.app.redirector, provoked an implication that I was simply inventing a different spelling. Quite the contrary, plone_app_redirector is simpler in 2 substantial ways:
- No 2 extra folders to traverse on disk and in svn
- No __init__.py files with unusual import magic to understand or ignore
“Products.* is no longer uniquely magic.”
Thanks to Martin Aspeli for pointing this out; I meant to mention it. As Martin says, any package can have the auto-loading magic of Products.* in Plone 3.3 if it declares the right egg entrypoints. No more ZCML slugs—hooray!
However, Ian Hood and Chris McDonough also raise a good point: as funny-looking as the Products namespace is, it at least signifies “this package is a Plone plugin”. (There aren’t many third-party Plone plugins that are also useful in a general Python context, so the distinction remains relevant.) We probably should include this I-am-for-Plone-ness in the package name so as to not crash into comparable plugins for other frameworks. A few possible naming conventions:
- PloneFormGen. Start everything with a “Plone” prefix, as most Trac plugins start with “Trac”. But, if applied inconsistently (as it would be during any transition), this would land us in the same situation as KDE’s start menu, where things randomly start with K and new users wanting a spreadsheet have to look for both Spreadsheet and KSpreadsheet.
- PonyPloneTheme. Put the Plone identifier at the end. With the entropy up front, this is better for visually scanning alphabetized lists and for type-to-select. (In places like the Collective svn, similar types of products, like themes, could be grouped into folders as with the current pasplugins folder.)
- NerfHerder. Keep Plone signifiers out of the name, and use egg metadata like trove classifiers. This has the possible disadvantage of colliding with non-Plone packages, but, then again, namespace is pretty big. Regardless, we should get some trove classifiers for common types of Plone add-ons. If there can be a Topic :: Desktop Environment :: Window Managers :: Oroborus :: Themes, there can certainly be one for Plone themes, one for Plone plugins in general, and perhaps a few other subslicings.
- Products.NerfHerder, perhaps the most reasonable compromise of evils: we’re already a decade into that particular hole, and it’s only one level deep. It doesn’t invent any more conventions to memorize or transition to like collective.* or plone.*, it doesn’t smash into general-Python namespace, and, if you don’t move your trunk when you eggify, it doesn’t defeat type-to-select either. I still dislike the namespace package due to the previously discussed implementation ugliness, but that may be fixable by technical means (watch this space). If I could erase the world and start over, I might go with a different scheme. But, since a transition would take forever and be confusing all the while, I’m willing to settle for Products.*. In fact, I’m fairly persuaded by Chris McDonough’s statement that “It’s completely, totally useless to have Products outside the Products namespace.”
“Namespaces should communicate licensing.”
Martin writes:
Namespaces also communicate ownership. For example, we’ve been able to formulate useful policy around the licensing of packages in the plone.* (soon to allow BSD licensing) and plone.app.* (to stay GPL licensed) namespaces.
Cordially disagree on this one: this is what the license metadata field is for. (Incidentally, I suspect no one outside an intimate group of core developers has any idea the difference between plone.app and plone.*.)
“The trunk should move when you eggify.”
Plone is one of the few projects where egg names typically reflect their import paths. Joel Burton, who interfaces with lots of new users in his training classes, finds this aids comprehension. So let’s keep it. In fact, I never suggested otherwise: either way, you’ll be checking out a folder called "trunk", not a folder named after the egg. My beef is with the practice of uprooting an existing svn trunk and moving it to a new location when eggifying an old product. There’s just no point. WebLion trains new developers all day long, and this practice is a rich source of confusion. It’s also a waste of time to have to look in two places when browsing a repository, and, if carried through, it’ll result in hundreds of abandoned top-level folders in places like the Collective.
Joel further writes, of keeping trunks still:
[T]he intended benefit, doesn’t exist. Your buildout wouldn’t still work if it expected a classic product at /svn/JoelsBlog--there isn’t a classic product there, but a nested egg namespace. So, your buildout would -still- break, and in a much less newbie-understandable way.
The intended benefit is not to keep buildout from breaking. It’s going to break no matter what—which is another failure that should be separately addressed, perhaps with better checking for common mistakes. My intent, rather, is to surprise people less when browsing repositories and, secondarily, to reduce top-level pollution in repositories.
“You’re an idiot.”
Probably. Here are my recommendations nonetheless, now a little better shaped thanks to your good-natured hammer blows:
- Reserve namespace packages for breaking up enormous legacy packages, as intended. Namespacing for any other purpose can be done more simply with plain prefixes, and many of our existing uses of namespacing (collective.*, plonetheme.*) would be better served by egg metadata. I'll be using egg metadata only and hanging out in the Products namespace to avoid colliding with non-Plone projects.
- Don’t move the trunk when eggifying. You’re going to break people’s checkouts either way, so why double the number of folders in the repo, make everyone search twice when browsing, and defeat type-to-select?
- Keep company names, license names, and other metadata out of package names—since changing them breaks people’s imports and buildouts—and nest them cozily in egg metadata. Plone products should probably all be in the Products.* namespace. (You talked me into it. :-))
- Fix those ZopeSkel templates. In answer to jjmojojjmojo’s question about how best to do that, I’d have the plone3_theme template spit out products in the Products.* namespace but give an example like “PonyTheme" when prompting for a name. And continue to put “web zope plone theme” in the egg keywords.
Again, I really appreciate all your responses. If any of my reasoning is wrong, please point it out and save me pain!
Some clarification
As far as I can tell, your main criticisms are:
- setuptools forces you to spell the namespaces inside your packages as directories, so you get usually two levels of nesting that's boilerplate
- changing namespaces is bad (d'uh)
I can't find any other non-emotional rationale to ditch them, but then it's late.
The first point I think is a smaller issue than you suggest. The latter is completely valid, but is not really anything to do with namespaces. It could be rephrased as "don't change your import locations if people are relying on your code", which is just good practice.
"""
But a lot of this is due to funny conventions. We use namespace packages in an unusual way in the Plone world: we never put anything but a single package inside them, making the outer folder mostly empty and mostly noise. We don’t take advantage of setuptools’ support for putting N submodules and packages inside a namespace package, which would make the signal-to-noise ratio more acceptable. So I wonder, can we come up with a shorter, fewer-empty-folders way of namespacing?
"""
I think you're confusing namespaces packages with another setuptools concept, namely having eggs that contain many packages. The plone.* namespace has *tons* of packages underneath it.
As far as I know, we don't use namespaces any differently to anyone else. We do have a convention that the egg name is the same as the dotted name to the 'main' package, including its namespace. As Joel says, that's just to make the eggs more discoverable, and I think it serves us well.
"""
Indeed we can. One of my proposed solutions, using names like plone_app_redirector instead of plone.app.redirector, provoked an implication that I was simply inventing a different spelling. Quite the contrary, plone_app_redirector is simpler in 2 substantial ways:
No 2 extra folders to traverse on disk and in svn
No __init__.py files with unusual import magic to understand or ignore"
"""
You are inventing a different way of spelling, except now we've lost the ability to manage those namespaces using Python import semantics. This namespace spelling is not consistent with how namespacing works in Python in general, and so people are more likely to get it wrong. It's all just one flat namespace, but you're hoping people will follow the convention to avoid collisions.
I'd be great if setuptools allowed me to say "the src/ folder contains the plone.foobar package root, where plone is a namespace package". But it doesn't, for various good and not so good reasons. I don't quite see how it's such a big problem. It feels like you're throwing the baby out with the proverbial bath water here.
"""
However, Ian Hood and Chris McDonough also raise a good point: as funny-looking as the Products namespace is, it at least signifies “this package is a Plone plugin”. (There aren’t many third-party Plone plugins that are also useful in a general Python context, so the distinction remains relevant.) We probably should include this I-am-for-Plone-ness in the package name so as to not crash into comparable plugins for other frameworks.
"""
It's not just about other frameworks. It's also about other packages in the Plone world, which is actually the largest community on PyPI, I think.
Products.* has the same problem as a flat top level namespace in that sense. And it should die. ;)
I've got some real, if anecdotal, evidence that namespacing in general and small packages in particular has had a positive impact. Before Plone 3, there were way fewer small-and-re-usable packages, and we were terrible at producing packages that the outside world could use. By moving to a toolset, conventions and patterns that other Python people use, we have been able to encourage developers to think about re-usability, split out re-usable parts into separate packages, and so on. Namespaces also give us a way to refactor, as you identified in your first post.
"""
A few possible naming conventions:
"""
These all suffer from the same problem. They rely on vague conventions that are easy to forget and don't sit well with Python's general approach to code structuring. By trying to invent a new set of conventions and a new form of namespacing (which is what it is you're trying to do), you've just made a more error-prone and less discoverable alternative. All to save you going into two folders? Mmmm....
"""
Cordially disagree on this one: this is what the license metadata field is for. (Incidentally, I suspect no one outside an intimate group of core developers has any idea the difference between plone.app and plone.*.)
"""
Well, true. The license metadata is the ultimate source of truth. But we can stipulate useful policy and general guidelines around the use of a namespace that the Foundation controls, and relate it to the Plone trademark. This has already happened, so you can't really disagree with it. ;-)
The plone.* and plone.app.* licensing restriction doesn't exist yet. When it will, people will know.
But ultimately, plone.* and plone.app.* are namespaces that "we the Plone core developers" own and manage. It's not really so important that others know about the standards we have for managing what code goes where. To users of those packages, it's all just code and package metadata. To us, the core developers, they are an important way to structure our code and manage our releases. As a case in point, moving from Products.CMFPlone as one uber-package to lots of small packages has allowed us to make releases more rapidly and incrementally, and allowed people to get upgrades to parts of the system (e.g. I need a new plone.portlets release to fix a bug in my current project) without having to track svn trunk of a huge package that is released rarely.
Of course, we could've achieved the same with plone_portlets and plone_app_portlets or whatever, but that'd be more painful, not less.
So, to your recommendations:
"""
Reserve namespace packages for breaking up enormous legacy packages, as intended.
"""
This is disingenuous. It's not the main or only intended use case for namespaces or namespace packages.
"""
Namespacing for any other purpose can be done more simply with plain prefixes, and many of our existing uses of namespacing (collective.*, plonetheme.*) would be better served by egg metadata. I'll be using egg metadata only and hanging out in the Products namespace to avoid colliding with non-Plone projects.
"""
I sincerely, sincerely hope that people don't follow this advice. I'd be disinclined to accept into Plone core any package that does not use sensible namespacing, for instance. For the sake of not having to navigate into two directories (which you can and should fix with collective.recipe.omelette anyway), we'd be creating confusion by moving away from a convention that's well established now (and which I still think is very valuable).
"""
Don’t move the trunk when eggifying. You’re going to break people’s checkouts either way, so why double the number of folders in the repo, make everyone search twice when browsing, and defeat type-to-select?
"""
I don't think this is very important one way or the other, so I won't reply.
"""
Keep company names, license names, and other metadata out of package names—since changing them breaks people’s imports and buildouts—and nest them cozily in egg metadata. Plone products should probably all be in the Products.* namespace. (You talked me into it. :-))
"""
You shouldn't change them. Your namespace needs to be something you own and control to guarantee uniqueness. I have no problem with company names there. But don't change them.
And I think the Products.* namespace should die. I disagree with Chris M's suggestion that there's no reason for Zope 2 products outside the Products.* namespace, but I think he's decided to disagree with me. ;-) The fact that something is a Zope 2 product or a Plone product or a Plone non-product package or a Trac plugin is implementation detail and completely orthogonal to namespacing in general. We try to refactor our packages to no longer require Zope 2 product-ness (i.e. an initialize() function), for example.
"""
Fix those ZopeSkel templates. In answer to jjmojojjmojo’s question about how best to do that, I’d have the plone3_theme template spit out products in the Products.* namespace but give an example like “PonyTheme" when prompting for a name. And continue to put “web zope plone theme” in the egg keywords.
"""
Better defaults would be nice. So long as they are actually better. ;-)
Martin