On Distributed Consistency Part 1

On Distributed Consistency Part 2

To start off the author mentions three companies as examples of “cloud hosting”: Amazon, GoGrid and CrackerHost. Cracker who? Yea that was my reaction. I happend to scroll down and saw the authors bio… You guessed it – Rudhir is the founder of CrackerHost. The over aggresive, and frankly out of context promotion of his service is already off-putting.

In the opening sections of the article Rudhir states the following definition of cloud computing: “Cloud hosting has been broadly defined as an on-demand, pay-as-you-go, high-availability service. … In simple words, cloud hosting allows you to use a web server for an hour, pay for an hour and be done with it”. A quick glance at the CrackerHost billing reveals monthly, and not hourly pricing.

Rudhir then continues to use an interesting definition of VPS  in a comparison of “cloud computing” and VPS providers. According to him provisioning a VPS instance is manual, and not on-demand/realtime, and does not afford the ability to resize instances. So… where do Rackspace Cloud, Slicehost and Linode fit into this picture? And how do they differ from Amazon… I’m pretty sure SliceHost and Linode self identify as VPS providers. Though other VPS providers like arpnetworks and prgmr are closer do his definition of a “VPS” provider.

And finally in his last sentance he says:

You should. One. In case of a disaster, a cloud hosting-based provider will be able to recover sooner. This means that your web site will have a lower downtime.

Cloud hosting will allow your web hosting provider to ensure that backup recovery is fail-safe. So, this will mean better data protection for you.

Oh boy – where do I start. Perhaps he hasn’t had time to read the post-mortem of the recent GAE outages, or is forgetting about the various AWS outages. At best, and only in advanced setups (like GAE) will there be any cross data-center failover, and being a “cloud” provider has nothing to do with this. What facts or evidence support these claims? None as far as I can tell.

If you are interested read the article here

The SAN like storage mounts are (only) available in a given “Availability Zone”. Nothing specific is mentioned about distributed file systems like GFS, so that is something I’ll be looking into ASAP. Like a regular volume you can create several EBS volumes and do your own software raid on top of that.

There are also some interesting scenarios around the snapshot facility such as having one “master” EBS volume in the designated “write” availability and replicating out to several “slave” availability zones in geographically disparate regions.

Performance seems inline with high end RAID arrays:

To recap:

Price: $0.1 per GB
Performance:
Features: Reliability, Snapshots, Traditional Posix filesystem

The S3 manage allows you to create US or EU buckets and drag and drop stuff from your desktop. The EC2 component comes with an AMI browser and built in SSH client.

Immediately I found a few bugs. The S3 component was’t displaying any buckets (that I knew I had) and instead displayed and error “Invalid Argument”. So I guessed that perhaps my credentials had a space in them so I deleted the account and created a new one. However after entered the Access and Secret key, it now says “License invalid”.

I hope I can figure out the “License invalid” issue so I can actually try the product.

Jollat is a free download, but only for 30 days.

movie = Movie()
movie.title = self.request.get(‘title’, None)
movie.director = self.request.get(‘director’, None)
movie.put()

Possible Errors:
Out of Memory
Deadline Exceeded
OverQuotaError
Server Crash
DataStore Crash
Indentical Entity Already Exists

Use logging module: logging.error, logging.debug — which shows up in Admin Console

This is called whenever the Request Handler encounters an exception

Email upon exception:
def handle_exception(self, exception, debug_mode):
lines = “.join(traceback.format_exception(*sys.exc.info()))
logging.error(lines)
mail.send_mail_to_admins(sender=’myapp-noreply@gmail.com’, subject=’Caught Exceptoin’, body=lines)
template_values = {}
if users.is_current_user_admin():
template_values = lines (FIX LATER)
self.response.out.write(template.render(‘error.html’, template_values))

Python Profiler

def profile_main():
logging.info(“Profiler data: %s, profile_data)
if random.randint(0,100) == 4:
profile_main()
else:
non_profile_main()

Load Tests
Reveal which part of webapp are slow, which resources are scare (CPU, Bandwidth, etc)

Remember, the dev_appserver is not designed for performance

Performance has been optimized for organic load increases (whatever that means) versus synthetic benchmarks.

Under load the App Engine front end will create extra python interpreters, and keeps the recently spawned interpreters warm. Talking about load test characteristics , ramp up slowly, and test sustained load over many minutes (10, 15) vs 1min.

Httperf is being recomened as good tool for driving http requests.

Talking about realistic values
50,000 users per day
2 page views per user
100,000 pageviews per day
5 requests per page view
500,000 requests per day

Leads to 5.8 requests per second

Qualities of good load test
Use production system, not dev_appserver
Gradual ramp up to allow BigTable warm up
Sustained Load, so any hiccups will be averaged out
Realistic Load

Safe Deployment
Upload code and don’t worry about pushing bug live to users — Versions — On every upload App Engine increments revision and points to latest revision — However, you can have multiple versions running in parallel (0.1, 0.2) can be accessed via http://2.1.appname.appspot.com — However, both will point to same BigTable data-store

http://...appspot.com

You control the version via app.yaml — Suggestion: Create a build script to populate version from SCM revision number

Test deployment with Selenium, use the log viewer to look for errors

Feature requests:
Easy A/B testing
Integration with SVN
Edit Running Code (Google not really in favor of this)

Summary:
Write Code to handle errors, use logging and catch exceptions
Do realistic load tests
Use versions for safe deployment

However, the Administration interface doesn’t allow you to configure any redirects, which could lead to major search engines associating your domains with duplicate content.

I’m going to cover how I enabled canonical redirects in Google App Engine as I would have normally done with Apache and mod_rewrite.

This will allow you to redirect the www sub-domain to the non-www domain, or the non-www domain to the www sub-domain — whichever one you’ve standardized on.

I solved this by creating two application instances, and then calling wsgiref.handlers.CGIHandler().run(application) on one or the other if the request came in for a domain that contained www or not.

The WSGIApplication method takes two parameters and returns an application instance. The first a list of regular expressions and the Request Handlers they map to, and the second a boolean to set the debug status to on or off. I’m going to exploit this and create one application instance that maps everything to a redirect request handler, and the other “production” application instance which contains the various production mappings such as “/”, and “/about”…etc

In your main.py, the main() method should contain the following. First import required modules, then pluck out the host name from the environment, then do a simple regex on it.

import os
import re
host_name = os.environ['HTTP_HOST']
m = re.match('www', hostname)
if m:
  application = webapp.WSGIApplication(
                         [('(.*)', CanonicalRedirectHandler),],
                           False)
else:
  application = webapp.WSGIApplication(_PRODUCTION_URLS, False)

wsgiref.handlers.CGIHandler().run(application)

The redirect method (webapp.RequestHandler.redirect() ) takes two parameters, the URI to redirect to, and a boolean to indicate whether or not the redirect is permanent (HTTP 301) or temporary (HTTP 302).

class CanonicalRedirectHandler(webapp.RequestHandler):
  def get(self,path):
    self.redirect("http://yourdomain.com%s" % (path), True)

That’s that. A fairly workable solution. Something is nagging at me, and I wonder if there is a more elegant solution to this within the App Engine framework. If so please let me know in the comments section.

It will be interesting to see how this plays out on one hand you have heroku the all in one IDE/Enviroment for Ruby On Rails that is backed by Amazon AWS and auto-deploys for you (to the Amazon cloud) and on the other hand you have Python, Django and Google App engine with a similar but different model.

Revisiting this thought process I think Heroku is probably in a rather secure position. The App engine team has mentioned numerous times that they will support multiple languages. In fact the App Engine configuration file (app.yaml) already contains a runtime directive which currently can only be set to “python”.

runtime: python

However, in my opinion it is highly unlikely that Ruby will be supported in the near term future, giving Heroku enough of a first mover advantage.

There have been some rumors that the next language implementation will be Java: (Java the Next App Engine Language). This makes a fair amount of sense. As the author notes Google permits only a select group of programming languages: C++, Java, Python, and JavaScript.There is also another reason that Java might be the next App Engine Language: The Dalvik VM.

Google has Guido van Rossum to help with the customization of the Python interpreter/Environment for App Engine, and Google has the Dalvik VM team for a custom Java VM for App Engine code, probably a slightly modified version of the Dalvik VM used in Google Andriod.

Heroku is also part IDE (web based). There is a difference in providing a hosting solution and developer tools. Google has never really expressed any serious interest in developer tools.

The analog here is AWS to Heroku is Google to App Engine (except that App Engine is obviously a Google team).

I think the only front Heroku and App Engine will compete on is developer mind share. This isn’t to say Heroku is safe, after all Google can drop serious money on developer programs like the I/O conference. However, I just don’t think App Engine will implement Rails and defeat the purpose of Heroku (any time soon at least).

However, Heroku is well positioned to disrupt Engine Yard — and I hope they do.

It will be interesting to watch the number of open source app-engine centric applications blossom — a blogging tool is a good first step.

http://picasaweb.google.com/appengine.hackathon/SanFranciscoHackAThon

You can download Cloud Studio or watch an introductory flash video at service-cloud

I think it’s great that people are jumping in and creating solutions to these pain points — but I have some reservations: 1) I would imagine that if Amazon released a new feature their own tools would support it sooner 2) I like the idea of standardizing around a common set of tools versus everyone implementing their own favorite AWS manager

Also featured at the Google Apps Hackathon in San Francisco

More updates soon, things are starting to roll.

10:03: Google docs presentation up on screens / projectors

10:15: Brief announcement — people are still trickling in so we’re waiting another 10-15 minutes.

10:20: I just uploaded the hello world for my pet application “Engine Snippets” it was as simple as editing app.yaml, cd’ing up one directory and running appcfg.py update enginesnippets (which prompts you for your email, and password) — I’m going to be doing this a lot so I’ve created a bash alias to make my life easy:

alias gaeup=”appcfg.py update ../enginesnippets”

10:30: room filling — slight modification of agenda — All Googlers are introducing themselves — we’re being told to keep our eyes on people with Google or Google App Engine tee shirts. Also, app engine accounts are being handed out.

10:30 Agenda being presented:

• (10:30 -> 11:10) Getting Started
• (11:10 -> 11:20) Intro to Googlers
• (12:30) -> Lunch + debugging (rumors of pizza)

Talking about Mac Engine Launcher (covered last night) — very slick Mac GUI for deploying App Engine applications and integrates with Mac text editors, IDEs

We’ll be hacking on a basic Wiki today, code is available for all here

Talking about DataStore DataTypes, all the usual suspects with a few implentation choices: IntegerProperty, FloatProperty, BooleanProperty, StringProperty, TextProperty, DateTimeProperty, ListProperty, ReferenceProperty, SelfReferenceProperty, UserProperty, BlobProperty, LinkProperty, EmailProperty…etc
You define Models using python class (Similar to Django models) call put() to add to data store (BigTable)

Guido Van Rossum, inventor of Python and Googler is in the room.

object = Wikipage(title=foo)

object.put()

object.delete()

GQL is like a Simple SQL, similar semantics. Google App API handles ORM mapping to defined models. GQL can query with named parameters.

app.yaml is where everything is configured, this is where name and version of application are configured. Handlers (URL Routing) is also defined in app.yaml.

application: appengine-app
version: 1
runtime: python
api_version: 1

handlers:
- url: .*
script: main.py

WSGI uses request handlers to serve pages (very similar to CGI … CGI-NG might have been a better name).

Talking about views, and Http Redirects.

GAE Doesn’t have any required flush, but buffers output and doesn’t stream response (details abstracted by GAE API)

Looking at Django templates — see django documentation, same stuff.

Speaker going fast, going through wiki-example, hard to type at same speed — might have to revisit later.

The DataStore API can auto update DateTimeProperty via auto_now_add=True

11:18:
(Names typed phonetically, many apologies for butchered names)

Dick Wall introduces himself (speaker) — Google Developer Evangelist, currently working

Ryan Berret, DATA STORE

Bret SDK, Python

Alohn Navi Admin Console

Kevin Gibbs TechLead

John UI Designer

Joshn Rebowski Launcher Mac

Dan O’Brian Dev Support (General Questions)

Guido Van Rossum: PYTHON

11:26: Pointing out differences in DataStore vs traditional DB — no schema, only code. Inheritance should work as expected, you define the models via python (See Django), and GAE Api handles ORM mapping to BigTable

11:44: Someone asked about pricing — Googler said it was undecided after limits (see current limits). No time line for announcing pricing — but they say will be competitive with Amazon AWS. This is kind of a big deal, and I’m a little annoyed that there hasn’t been any movement on pricing since the preview launch.

11:52: Awaiting rumored pizza — I’m starting to play around with the local dev server from my project working directory I simply typed the following (see below) and was greeted with pretty errors informing me of all things broken.

dev_appserver.py .

12:00: Just finished Mango Madeness snapple , my fact: Fish Can Drown. My Take: So can products without pricing.

12:05: The pizza is a lie! I feel betrayed, but that will come out in therapy a few years from now. I had to settle for a superb tuna sandwich and delicious salad with walnuts and a balsamic dressing. Overheard: Guido lamenting an Asus Eee Keyboard “I can’t, I just can’t use this…” hands back to owner.

12:25: Playing around with the local dev server I keep on getting the following warnings on STDOUT

WARNING 2008-05-16 19:24:58,045 datastore_file_stub.py] Could not read datastore data from /tmp/dev_appserver.datastore
WARNING 2008-05-16 19:24:58,045 datastore_file_stub.py] Could not read datastore data from /tmp/dev_appserver.datastore.history

Will revisit and fix this later.

12:35: Lunch time talk, covering tools (IDEs…etc) — We’ve already looked at the Mac only App Engine launcher. Mac is obviously well supported for Python, TextMate is the defacto editor. There is a Django-bundle for TextMate. Komodo Edit (ActiveState) not as complete (in terms of code completion) as TextMate, but getting props, and cross platform. Debugging is a bit of a pain point — Apparently Eclipse and PyDEV have good python debugging support. Covering Admin interface on local dev server, browse to: http://localhost:8080/_ah/admin/datastore — Also has interactive console for python, and interface to Data Store.

1:08: Things are picking up again, talking about the user authentication API, and authentication sources (GMail, Google Accounts for your domain…etc)

1:25: Tip on how to generate images from binary data in db.BlobProperty types http://code.google.com/appengine/articles/images.html

1:54: More hacking, here is a thread on Google App Engine best practices

2:40 looking at fetch and mail API as part of wiki example. Google App Engine instances can’t write to files ystem, nor can they open sockets to make network requests, must go through Fetcher api. URLFetch Responses has properties headers, status_code, content, content_was_truncated, and does not follow redirects.

Mail API — must specificy sender (admin of GAE application) can set other properties

2:57: Running into this error when trying to query DataStore API: SystemError: Parent module ‘google.appengine.ext.db’ not loaded — speaking with Bret , this seems to be an issue in the API or possibly (gasp) the interpreter getting into a funky state — restarting the development server cleared it up. Another dude ran into the same issue here

4:00pm: Talking about Google Data APIs — code.google.com/apis/gdata

Python API — usable on Google App Engine — gdata-python-client (via URLFetch API)

Article on using GDATA from Google App Engine

I’m looking at pygments for syntax highlighting — might not be app engine friendly — will have to investigate later.

Looking at GeoDataStore code here