Sysadmin by day, developer by night

First, since this is a blog post where I will discuss Appengine with some negativity, I want to say that I think that the work Google has done with it has been great, and it is a suitable environment for a lot of projects. I in fact still use it to demonstrate some mockups and such for the freelance business I’m trying to get myself into and find it great for that. I also developed the proof of concept of my project on Appengine using appengine-patch for Django with great speed, and enjoyed the process.

The first barrier for me on Appengine was I didn’t know Python. Java wasn’t supported at the time I started on Appengine, and I don’t care for using Java to build websites anyway. Fortunately Python isn’t that hard to learn, Django made it easier, and the guys who made appengine-patch made that even easier. It also helps I support several developers at work who primarily work with Python/Zope, so when I had questions I had someone to ask.

The next barrier was sessions. I didn’t want to use just Google accounts for my project. I’m using Oauth, OpenID, and Facebook Connect for authentication. I didn’t care for a lot of the session libraries for Python I found, and most wouldn’t work natively with the Datastore anyway. However, this is no excuse to drop an application environment, it’s an opportunity to write code to help others. Thus, gaeutilities was born. I created an opensource project that got some help from a couple other contributors that provides sessions and cache to Appengine, that integrates with both the Datastore and Memcache functionality. I still fondly remember doing the rewrite to include Memcache when it was released. ‘

After getting gaeutilities stable, I started working on my project. Eventually, I got my project to where it is now, a functionaly proof of concept. So, I started using it on a daily basis. This is where I ran into the final barrier that is causing me to leave Appengine. Datastore requests are unreliable. You will get timeout on write and read operations from time to time, and they happen a lot more than what I can consider acceptable.

Now, I tried my best to work around this. The latest version of gaeutilities in svn has been rewritten from the last release to write to both the memcache, and datastore. It writes to memcache first, then the datastore. If the datastore write fails, the memcache is flagged as “dirty”, so that on the next read/write it will try again to write to the datastore with the version of the data it has. Cache does something similar, however it doesn’t try to write to the datastore every again, as the general principle for cache us is attempt to read from cache, if it’s not there, create the item and cache it. Cache is used for optimization so repeated attempts to write is actually worse for performance.

The catch is, this is only useful for write. What about timeouts on read? You can try again in the same request, however my experience has shows that when you get a datastore timeout on a request, repeated attempts will timeout also. Actually for write (and possibly read, I never got an answer on my question asking for clarification on that), the Datastore internally retries up to 3 times before returning the Datastore.timeout exception.

When I’m the only user of my application, and I’m seeing these timeouts, especially on read, several times a day, I have to reconsider using the application environment. Perhaps I’m a victim of my application not being busy enough to warrant ample resources. If that’s the case, then how do I launch when my early adopters will have to deal with these errors?

Appengine is in a preview release. However, they are charging. Google has stated in a several threads that the Datastore timeout issue won’t go away. That being the case, I can’t justify continuing development on appengine when the data repository used for both data storage, and state management, is unreliable. This is why I’m ceasing development on Appengine for my primary project, and why I’m offering gaeutilities to anyone who wishes to take over the project. As I’ll be going to back to a VPS or some other similar solution, I won’t be putting the time on gaeutilities any longer.

If you’re working on a small project that doesn’t need to keep state using the datastore, such as cookie based session data being acceptable, or using the Google Accounts interface, Appengine is a great solution. If you’re working on something where you absolutely need your reads from the datastore to work for things like state, I’d suggest looking for another solution.

  1. joerussbowman posted this
blog comments powered by Disqus
Technorati Profile