Over the past several days, I've been heavily into Zope development.  I have been working with both the Zope Management Interface to build small web applications right inside the web browser, to then learning how to create full-on Zope Products.  It has been really fun and interesting to learn something that had alluded me for years, since I did originally try Zope back in the mid-2000s, but since I didn't know much Python back then, I couldn't really do much.  Zope has a really neat web interface which allows you to browse it's special Zope Object Database file system.  This is probably one of the most interesting and unique aspects about Zope, is this object database.

Zope Object Database

So, let's start this blog post with talking about that.  This was developed way before the concept of an object database even hit the mainstream with now common NoSQL servers like MongoDB and CouchDB.  The modern ZODB even allows it's database to be stored on a compatible relational database, namely PostgreSQL's RelStorage.  The object database in Zope is a first class citizen within the web framework, rather than a second thought like in other web frameworks.  As a result, you get some really powerful APIs and tools with Zope right out of the box.  Zope even has support for hyper-scaling through it's ZEO Server.  It is although a bit unfortunate that it's overall documentation isn't always fully updated, as the ZEO documentation on the main Zope website for example references Zope 2 along the top with a bit of a warning.  This warning on their documentation can be mostly ignored, as a lot of these features are still in modern Zope 6, and still actively developed.  In fact, for production scaling, many documentations on deployment still speak of using ZEO very proactively.  There is even an alternative, and compatible which is distributed called NEO, showing just how much this object database can be massively scaled if needed.

What makes ZODB better?

With the Zope Object Database driving the base of your entire website, you have a few unique advantages that you don't usually get with traditional object databases, or even relational databases.  Here is a list provided by the ZODB team themselves, then I'll discuss some of them below.

• Transactions are atomic
• Transactions provide isolation
• Transactions affect multiple objects
• Database caching with invalidation
• Pluggable layered storage
• Easy testing
• Garbage collection
• Binary large objects, Blobs
• Time travel
• Objects can have full permissions, roles, and ownership

The most interesting ones that I want to focus on here are the transactions and the time travel, which work together in a way, and here how that works.  When you create a new object in ZODB, it is encapsulated within a transaction.  In fact, the entire request is in this transaction layer, and if any part of this request fails, then the transaction is immediately rolled back.  This provides a fully atomic database, which is much less prone to errors.  For example, if you have a method which is going to be creating a container, and placing a bunch of objects inside it.  If any of these fails to be created, either through permissions, or another error, the entire operation is rolled back, and nothing is created.  The time travel part comes into play here because the entire database is only written sequentially, if you can even believe that.  That means, that it is actually possible to back-up an entire website's object database while the site is actively taking requests, and this entire ZODB backup will be a working backup that can be easily restored later.  This also means that each object in the object database has full history and revisions attached to them, allowing users with the appropriate permissions to go back to a previous version of an object.  This is all built into ZODB, or rather Zope.  As I'm not sure how the history works on a raw ZODB.  This can mean that the object database can grow large over time, however that is why there is a pack feature which can remove much older history to keep everything running smoothly.

One other feature that the  ZODB website does not mention, as it's more specific to the Zope Application Server, is the permissions model.  This allows every object in the ZODB to have an individual permission or role attached to it.  This permission system works with the web publishing portion to ensure that only users authorized to view or edit specific objects in the database can.

Leveraging Relational Storage

Having a web framework that has an object database as it's first class citizen doesn't mean you are locked out of using relational databases.  In fact, Zope offers many database connectors for all the popular database servers out there.  On top of that, you can also store the actual object database on a PostgreSQL database server, taking advantage of how PostgreSQL scaling works.

Using relational data from within Zope is rather interesting, as from my current experience, you need to add the specific relational adapters into the ZODB as objects themselves, and then configure them from the Zope Management Interface.  There then exists several other objects which can interact with this data.  In the future, I will be putting together a demo site which will run on my website here to showcase how access to a typical relational database works.

Zope Product Development

In Zope, a custom component is called a Product.  A Zope Product is installed into the server's Products directory, and is a standard Python package, in that it contains a __init__.py file.  Other than containing this file to signify that it is a Python package, the directory is otherwise freeform.  There are some recommendations, such as having a dtml/ pt/ and www/ directory for those specific type of assets.  Most Zope Products consist of one or more classes, which are then registered with Zope upon start-up.  These classes will normally add a new object type which can then be added to the various folders within the ZODB through the Zope Management Interface.  You can see an example of a Zope Product I put together on my GitHub in the TestProduct repository.  I build this repository as both a way to learn more about Zope Product development, and to also have a basic set of Python classes that I can just copy and paste into a new Product to start development on something new much quicker.

Through-the-Web Development

While it is possible to development full web applications in Zope that are built entirely within the ZODB, this isn't recommended by the Zope and Plone communities.  One of the main reasons cited is the lack of proper version control, as you would need to manually export or import your work into files on your local file system for the purpose of keeping track in a Git repository.  In most development workflows, this is unacceptable.  I do believe that it could actually be possible to say develop a custom container type within Zope which could connect to an external Git repository, which could solve this problem.  I do see some advantages to building parts or all of some smaller web applications within the browser, and here are these advantages:

• Python Script objects are highly restricted
• Each object created through the web is bound to that user's permissions
• File system modules like Zope Products and Plone add-ons have full unrestricted access

If you develop a web application outside of the Zope Management Interface on the host file system, you need to ensure that you manage all the required permissions checks and such yourself.  With code objects developed in the browser, they are automatically restricted by whichever permissions of the user whom either created that object, or the user viewing that object.  Here are some interesting examples of this:

• A Python script created in the browser tries to create an object in the ZODB, what happens?
  • If the user is authenticated, the Python Script will actually run with their Zope permissions automatically, making the objects that are created by this Python script owned by them.
  • If an anonymous web user accesses the same Python script, unless a proper proxy is specified, the object will not be created, and an authorization error will be triggered.
• Zope objects that can run code have the concept of a Proxy, which can be set by that object's creator so that the object is essentially setuid when executed.
  • The user cannot set it to a higher role than they currently have, but this can allow scripts that need to say add a new file to contain an anonymous user's comment for example, possible.
• Can be set to only allow users with certain roles to perform tasks like database queries as well.
  • This permission model is heavily baked into Zope and it's ZODB, permissions can be really fine-grained.

Through-the-Web Customization

One really neat aspect of the Zope Management Interface, is that if you do develop a highly specialized Zope Product, your product can easily create what are called management screens when your custom object is opened.  This can be as simple as showing all the various properties on the object itself, allowing a user to modify them, or even make some or all of them read-only.  This can be great during development and debugging, as you can see what your object contains in a live view of that object right from an easy to use Web Interface.  These management screens can be as deeply customized as you want, or just display a simple property sheet.  These properties can then control how the object you developed functions.

This feature can enable regular users of the website that aren't particularly developers to customize how the various objects the developer has created function.  These custom properties can also be exported, and then imported onto a different server that has the same Zope Products for example.  This can be used to have a staging environment within the corporation which has various employees working with these custom objects in the Zope Management Interface to test out, and stage potential objects which can then be placed into production.  These exports could also be backed up, or even be a part of a deployment pipeline of sorts.  One thing to note about these exports, is that they are binary, and should only be imported on Zope servers using the exact same Python version as the exported server.  The reason being is that these exports from my understanding are Python pickles, Python's specific serialization implementation.  Zope used to offer an XML export, but there was never an XML import.  Perhaps in the future, Zope can offer a JSON Export/Import feature.  This could allow some object instances to be programmatically generated for example, or analyzed by a human, or even a program.

How do Zope Products work?

A Zope Product as mentioned is a Python class, or a set of Python classes which represent a model which will have it's instance created in Zope as an object within the ZODB.  As a result of this, a Zope Product doesn't generally conform to modern programming mentalities of Model-View-Controller, or even Django's Model-View-Template.  Within a Zope Product class, both the Model and the View are essentially combined into one.  This is the class within the Zope Product that turns into an instance when added into the ZODB as a new object.  Each property on this object is persisted into the object database with no additional logic required, just change a property in a method call, and it will automatically be taken care, no need to worry about manually tell Zope to load or save anything.  This might sound a bit confusing at first, but once you understand how programming with Zope Objects works, as I guess I'm calling them here, it's actually really easy.  The template portion of this, is that, your model can reference external template files, these can either be in the older DTML format, or the newer and more modern Page Template format.  The more modern Page Template is actually using the chameleon template system, which I believe was actually originally a fork for the Zope Template Pages.  So, this project I guess circles back into itself, as now Zope is using the fork of it's own Page Templates.  Regardless, this template engine is really powerful, and in native Zope, it's actually better than when using this template engine alone.

The most interesting part about a Zope Product class is how it is exposed through web publishing, and one of Zope's strengths in my opinion.  It is possible to make any property or method entirely private as well by using the well established Python standard of prefixing it with an underscore.  If a method is public, it can be accessed through a URL from the object in the ZODB, or through a template attached onto the class itself.  Permissions can also be easily attached to any external method, leveraging the permissions in Zope itself.  Each permissions added onto the class will then be fully access within the Zope Management Interface to give to specific roles.  This makes it possible to really fine-tune which externally available methods can be accessed and by whom.

As mentioned, each method is exposed from that object to the web.  As example, say we created a new instance of our Poll class from our Zope Product into the root of our website at say /poll.  If we create a property on this class in our Product, and call it index_html, the default index page in Zope for all objects, then visiting just /poll will then run this method.  In a lot of cases, this would be a main page to display the object to a visitor, such as index_html = PageTemplateFile('pt/poll_view', globals()), which would render the Page Template in pt/poll_view.zpt.  It's really just as simple as that to get a stand view working.  Inside this page template, it would then have access to the various properties and methods of the Poll instance in the object database.  So, if you had a property with a list of poll items, you can easily iterate through this list with no additional code or logic, as the list is already loaded and ready to go.  Of course, you could also add various method calls, which can take parameters if needed from the Page Template as well.  When submitting a poll response, the form or JavaScript request in the browser would target an appropriate method, which can also have permissions attached to it, for example, /poll/submitResponse could increase the variable for one of the properties on the poll instance, and then redirect the user to the poll results at /poll/results.  The actual method code for this would also be incredibly simple, as it doesn't need to do any explicit loading or even saving of any data, all it has to do, is modify the appropriate properties on the poll instance object, and it's done.  The results can then also be viewed from the Zope Management Interface by merely navigating to the poll object instance there and viewing it's properties.

Developing Zope Products has a very different workflow than developing for other web frameworks as a result, but once you understand these things, it becomes incredibly easy and satisfying to create Zope Products.

What about Plone?

While Plone runs on the Zope Application Server, and it is still possible to develop stand alone Zope Products which can indeed run alongside a Plone site.  Plone handles the development of it's additional features and extending it properly, it's own unique way called add-ons, which I won't be diving into in this blog post.  As a result, I'll keep this section rather brief.  I wrote another blog post about using Zope Application in Plone that you can read if you are curious on how such integration could work.