2008-07-02

Pure Python Plugins

After some searching and asking around I didn't find any good explanation of the simplest way to implement plugins in Python. So, for posterity's sake, here's my solution. I'm sure there's a better way: I'd love to hear your suggestions.

First, the requirements. The code cannot have knowledge of how the plugins are named (.so files, .py, package dirs, etc.). I don't want to hard-code the list of plugins, as this defeats its dynamic nature altogether. I have to be able to iterate across all plugins. Any user should be able to use the module without knowing that it's got plugins. Finally, it's got to be as simple as possible.

First up, we have whatever/__init__.py:


import pkgutil
import imp
import os

plugin_path = [os.path.join(__path__[0], "plugins/")]

for loader, name, ispkg in pkgutil.iter_modules(plugin_path):
file, pathname, desc = imp.find_module(name, plugin_path)
imp.load_module(name, file, pathname, desc)


This basically uses Python's module search to find all contents of the plugins sub-directory and load them. Now we have some more scaffolding in the same directory, as whatever/whatever.py:


plugins = []

class plugin(object):
"""Abstract plugin base class."""
...

def register_plugin(plugin)
global plugins
plugins += [ plugin ]

# utility functions to iterate across and use plugins...


Finally, each plugin looks something like this, in plugins/foo.py:


from whatever/whatever import *

class fooplugin(plugin):
"""Concrete class for foo plugin."""
...

register_plugin(fooplugin)


Simple enough, but it took a while to work it out. Unfortunately, it doesn't seem possible to merge whatever.py into __init__.py as we have a recursive import problem.

11 comments:

WadeMealing said...

I've attempted to follow the idea from http://gulopine.gamemusic.org/2008/jan/10/simple-plugin-framework/

which seems to cover some of the same basic ground, but doesn't mention the plugin loading.

Meanwhile, one of the legends over at poocoo also covered a similar topic ( http://lucumr.pocoo.org/blogarchive/python-plugin-system )

John Levon said...

The latter fails on (at least) two of my requirements.

Anonymous said...

I recently wrote an image manipulation program in python and needed to load filters, first is listed all the files in the directory:

allfiles = os.listdir('filters')
allfiles = [f for f in allfiles if os.path.isfile('filters/'+f)]

Then i filtered the list to only include those with a '_ilf.py' extension:

filters = [f for f in allfiles if f[-7:] == '_ilf.py']

Then i imported the modules and added them to global namespace and a new list:

filternames = []
for f in filters:
  name = f[:-3]
  globals()[name] = __import__[name]
  filternames.append(name)

worked well in my program...

John Levon said...

That fails this requirement:

"The code cannot have knowledge of how the plugins are named"

WadeMealing said...

John,

Its a one line remove of code to get make it "pass" that requirement.

Why ask questions when just rebuff them right off the bat.

John Levon said...

I'm not sure which piece of code you're referring to Wade. Is it the one posted by "anonymous" ? If so, then I'm afraid it's not a one-liner at all to meet the requirements.

Whilst I appreciate your comments, I do have something working that meets the requirements: I was just wondering if there was a better more seasoned Python hackers might know of.

WadeMealing said...

Apologies if I come across a little aggressive, was never the intention.

The post I was referring to was the post by anonymous.

I guess it all comes down to what your needs are for the application.

My take on the situation.

Gulopine brings a uniform "view" to the plugin architecture. Although compile C code "imported" wont be able to be a plugin and will require a wrapper (small price to pay)

The poocoo method can give you an interface into a plugin system, but you must abstract this away (probably good practice anyway).

Anonymous method, while simple has limited reusable.

In regards to my common on anonymous's post were f you could adapt the program to not use the 'filters =' line, they can be imported (but it does have no mention of re-use).

This would then overcome "The code cannot have knowledge of how the plugins are named" As it could load whatever is in that directory, be it .so .dll or .py.

I'll assume you dropped the word method from your last paragraph.

John Levon said...

Right, it sounds like setuptools is the "official" way to do plugins, but it seems overkill slightly for my needs.

WRT anon's code: you'd also have to fix the assumption that plugins are files instead of directories. And then, you have the concern that it will attempt to
import stuff that isn't Python at all.
My version doesn't have these (admittedly minor) drawbacks.

Anonymous said...

http://twistedmatrix.com/projects/core/documentation/howto/plugin.html

John Levon said...

Interesting. Twisted is fine, but almost certainly overkill for my needs. I don't want to assume Twisted...

WadeMealing said...

John,

Not caring what it loads.. one may consider that an advantage, (as you may load .so, dll, or py

Wade