Google App Engine Campfire One Transcript

Kevin Gibbs, App Engine Tech Lead
April 2008

The following is text from a presentation on Google App Engine, delivered by Kevin Gibbs at Campfire One, April 7, 2008. You can watch the video of this presentation by clicking the play button below.

Introduction

Hi everyone, thanks for coming. I'd like to start by emphasizing that Google App Engine is different from other systems you may heard about for running web applications. From the ground up, Google App Engine is an attempt at making creating and running web apps easy. In the same way Blogger made it much easier to create a blog, we're looking to make it easier to create and deploy a web application — and make it free to get started.

So, what is Google App Engine? Well, as Tom mentioned, Google App Engine is a system that exposes various pieces of Google's scalable infrastructure so that you can write server-side applications on top of them.

Why are we doing this? What's the problem?

But let's step back a bit. Why is Google doing this? Well, we built App Engine because we want more web apps to get created. What we noticed is that, today, it's pretty hard to create one: there are significant upfront challenges to deploying even the simplest of web applications. You've got a lot of tasks to do. First, you have to write the code for your app, of course.

But then, you also have to write your Apache web server configs and startup scripts, set up your SQL database, create all of its tables and hook up the passwords, set up monitoring so you can tell what's going on with your traffic and logs, decide how you'll push new versions of your code, and on, and on.

That's the technical setup challenge that we noticed. And then, once you've done all that sysadmin work, you have another challenge: you have to actually go find machines you can use somewhere, physically or from a virtual provider, to run your app somewhere. Right now, that costs money: even for the smallest app, which you use a few times a week, you have to pay a pretty big upfront fee to run that app with a traditional hosting provider.

So that's the financial or physical challenge. And then, once you've got the whole thing set up and working, and found and paid for a place to test it out, you've got another challenge: you've got to maintain it all as your app grows. Your machines crash, your configs have errors, your hard disks break, your traffic starts to grow, you have to re-shard your databases, set up more machines and on. Keeping everything going as your app grows is a hassle.

All of these hassles are what we're trying to abstract away with App Engine. They are the problems that we're trying to fix.

Design goals

Thus, it follows that our design goals for the system are: Make the system easy to use. Make it easy to scale. And make it free to get started.

OK, I've mentioned our design goals, and talked about what we're trying to fix. Let's get a little more detail: what is App Engine?

What is App Engine?

There are three key facets to the App Engine system that are important to understand.

First, Google App Engine provides an infrastructure for running web apps. By that, I mean that we're focused, specifically on web applications: making them easy to run, easy to deploy, and easy to scale. App Engine is different than a lot of other things out there: App Engine is not a grid computing solution-- we don't run arbitrary compute jobs. We also don't give you a raw virtual machine. Instead, we provide a way for you to package up your code, specify how you want it to run in response to requests, and then we run and serve it for you. You don't reserve resources, or machines, or RAM or a number of CPUs, or anything like that. It's a fluid system, that runs your code in response to load and demand.

What you define are the URLs you use, what code they map to, and upload an application. We do everything else.

This leads me to my next point: Another big part of App Engine is that we provide for the entire life cycle of a web app. By that, I mean that App Engine is a complete system. We provide ways to run your code, serve your static content, a database, request and application logs, methods to push new releases of your code, and more. Ultimately, we are trying to provide an simpler alternative to the traditional LAMP stack. This is a big leap for you to take, and this is a big challenge for us. We've got a lot of tools that we need to provide good alternatives for. What we're trying to do, though, is to integrate all of these tools together into one system. We do this because we want to make the whole task of writing serious web applications easier.

Finally, the other key part of App Engine is that we're providing you access to Google's infrastructure. The APIs and systems we are providing to you are built off of the same distributed, scalable infrastructure we use to power Google's other applications, like Google Accounts, GFS, and Bigtable. We're giving you access to those powerful building blocks, and giving you the ability to write real code and real apps that make use of them.

What's our stack?

Those three facets explain how the App Engine system works. Now, what's the App Engine stack that we provide? To answer that, I'd like to give a quick overview of the key components of the system Then, I want to link you to a demo, so you can see how it all works.

Our first major component is our web serving infrastructure, which connects HTTP requests from the user down to your app and your code. The serving infrastructure is automatic and transparent, regardless of how large your app gets. Second, we are providing a Python language runtime, which is our first supported language for server-side code. You can include and run nearly any Python module in your app. Next there is our SDK. The SDK completely emulates our server environment, and lets you develop all your code locally, in real time, without uploading it. The fourth component is a web based admin interface, which integrates all the tools you need to manage and debug your app. Finally, our fifth component is the Datastore, our scalable persistence layer.

[Demo]

What did you see? What development is like

What did you just see in that demo?

Well, step 1, as I've mentioned, is that you use your local development environment to write your code and develop your app locally. You can test all aspects of serving your app, setting up a framework, and populating our APIs with sample data.

Step 2, as part of that local development, you write a deployment file, called app.yaml. This file is simple, and it encompasses bits of your program configuration and your web server configuration, all in one file. Here, I've given my app a specific identifier, and a version number, so that I can deploy multiple versions of my app, one for testing, one for users, etc. Then, I've indicated that I'm using the Python runtime, which is the only language we offer for right now. Finally, you set up a simple mapping of URLs to code files, which tells us how to execute your code in response to requests.

Finally, step 3, once you've happy with everything and have it working, you run our appcfg command-line utility to deploy a copy of your app to Google. In this case, I'd just type in http://helloworld.appspot.com into my browser, and I can start testing my app online. If I find some bugs, I hit update and deploy a new copy.

And that's it. Whenever you're ready, just send your app out to the world. You've launched.

Now, what didn't you see? Well, you didn't see any setup. We didn't have to set up a database, or go create a machine image and do a bunch of config files. We just wrote our code, and we were ready to go.

Components

Now that you've seen the system in action, I'd like to cover our five major components in a little more detail.

One of the biggest things that we offer is a scalable web serving infrastructure. To be more technical, Google App Engine is a distributed system for running your web applications with a very low overhead. When you submit your app, your code is automatically pushed to a number of fault tolerant servers, and Google's web serving and datacenter infrastructure handles all the details of connecting requests from the user down to a running instance of your code. You're never tied to just one machine, or just one failure. Further, the system is designed to make it easy to scale your application. We watch your app, and fluidly devote more resources to your app when it needs it, and move your app around as its needs change.

Next, right now what we're offering in this preview release is a Python language runtime for your code. You write your server side code in Python, and you get to do the same things you're already used to: you program in real Python code, you can make use of a wealth of Python modules and libraries out there, and you can use any web framework you'd like. You can use any Python code you like, serve up your requests any way you like, and use any sort of HTML output you can think of. We don't want to tie you down to any specific framework or one way of doing things.

On that note, I'd also like to mention that Python is the just first language that we're offering with App Engine. We're working on more, but for now, we'd like to get feedback from you on the system as a whole, to see what you like.

Now, as you saw in the demo, another pretty big part of App Engine is our SDK. Our SDK lets you build, debug, and test your applications locally, on your own machine, using emulated versions of all of our APIs. It's a snap to set up, and it means that you can do all of your tough development and debugging work locally, with an incredibly short code-compile-test cycle: just change a line of your code, and hit reload in your browser.

I told you earlier that we're providing an alternative to the standard LAMP stack with Google App Engine. What we provide instead is our web-based Admin Console. The admin console allows you to check the status of your application, see what versions you have deployed, control who can administer your app, and use various other tools. It also integrates a number of tasks: it gives you access to your logs, both request logs and app logs. It provides a Datastore explorer to view and edit your data. It lets you hook up a domain name. It gives a stats dashboard with real-time counts on your application's requests, any errors, and your resource usage. It integrates all of this information into a single tool for administering your app.

Datastore: In Depth

One of the most significant components in the system is our Datastore. It is the most important API in App Engine, and our key persistence layer. As I mentioned earlier, the Datastore is something new: it's a flexible, schema-less object database, or an entity store. It supports huge datasets with millions of entities, but under the covers, it does not run on a set of clustered SQL servers, which is the usual way you do something like this. Instead, it is based on Bigtable, the same system Google uses for its massive scalable applications.

For most of you, working with the Datastore will probably take a little getting used to: as I've said, it's not SQL. That's a big difference. However, we think that after a while, the Datastore may actually grow on you, because it makes some things easier. For one thing, our datastore is schema-less, meaning it can support arbitrary new properties or columns, which you can create as you code, without having to design everything up front and create a schema. This comes back to our goal of making writing a web app as easy as possible: just start coding. Your data model can evolve along with your app.

Even though the Datastore is a departure from SQL, we still support a lot of powerful functionality that you usually expect from a traditional database. The Datastore supports efficient queries on any single property or set of properties you provide. It supports provides sort orderings on your query results, including sort orders on multiple properties. It supports transactions for writes, with transactional groupings that you control. It supports batch operations for fetching or creating a large number of entities. It optionally allows you to control the primary key of your entities, for more efficient queries and shorter URLs.

And, even though the Datastore is not SQL, we're providing you with a SQL-like query language, called GQL, to make it easier to formulate queries. GQL is in the spirit of jQuery and FBQL: the underlying store is not SQL, but nearly all of the queries that you'd like to do can still be accomplished.

Here's an example of the sort of complex queries you can do with GQL:

SELECT * FROM Story
         WHERE title = 'App Engine Launch'
           AND author = :current_user
           AND rating >= 10
         ORDER BY rating, created DESC

One big feature that you may have noticed that our Datastore doesn't have, though, is joins. The reason for this is that joins are usually a source of performance problems in a distributed system, when you go beyond a single machine: it's much harder to efficiently support a join on a distributed system that spans many computers and many hard disks.

That actually underscores the strength of the Datastore, and why what we are doing is exciting. The Datastore is built on Google infrastructure like GFS, the Google Distributed Filesystem, and Bigtable, our horizontally scalable distributed storage layer. You can read papers about both GFS and Bigtable online, if you'd like-- they're fascinating. The long and the short of it, though, is that Bigtable is a storage system that can be used to support queries like ours, but which doesn't run on a single computer, or even a sharded set of computers, like a SQL database does. Rather, it is a fault-tolerant, distributed system that can span tens of thousands of hard disks on thousands of machines, making all of them appear to be a single storage table. It moves your data around and restructures the system automatically to account for hotspots and increased storage.

What it all means, is that if you use the Datastore correctly, making efficient queries and thinking ahead a little bit about how your app will grow, the datastore will make it easy to scale your application as it grows, from a few entities to millions. No need to shard your databases, or to restructure your schema. This is part of how App Engine makes it easy to scale your web application from the ground up.

More to know

Now, there are a lot of other things about the system that I haven't covered. We've got other APIs beyond the datastore.

One is Email sending. Your app can send emails to users, and we make it easy: no SMTP setup, no other fancy footwork required. Just create a Mail object and send it.

Another API we provide the URL fetch API, which allows you to make outbound HTTP requests from the system, to contact other servers. You can fetch other web pages and make web service calls. This is really helpful for integrating your code with other existing systems.

For authentication, we provide you with standard Google Accounts authentication, and we handle maintaining login, logout, and user accounts for you. You don't have to maintain your own account infrastructure, and we help you cut down on spam by ensuring accounts have passed a CAPTCHA, along with other features.

We also support much larger web frameworks, like Django. These frameworks are really helpful from when you go from writing small apps to big, complex applications.

There's tons more about the system that I won't get to right now, but I hope that I've given you a quick overview of what App Engine is, the components we provide, and how App Engine provides a simpler alternative for writing web apps.