Tuesday, March 30, 2010

OAuth access to IMAP/SMTP in Gmail

Google has long believed that users should be able to export their data and use it with whichever service they choose. For years, the Gmail service has supported standard API protocols like POP and IMAP at no extra cost to our users. These efforts are consistent with our broader data liberation efforts.

In addition to making it easier for users to export their data, we also enable them to authorize third party (non-Google developed) applications and websites to access their data at Google. One of the more common examples is allowing a social network to access your address book in order to send invitations to your friends.

While it is possible for a user to authorize this access by disclosing their Google Account password to the third party app, it is more secure for the app developer to use the industry standard protocol called OAuth which enables the user to give their consent for specific access without sharing their password. Most Google APIs support this OAuth standard, and starting today it is also available for the IMAP/SMTP feature of Gmail.

The feature is available in Google Code Labs and we have provided a site with documentation and sample code. In addition, Google has begun working with other companies like Yahoo and Mozilla on a formal Internet standard for using OAuth with IMAP/SMTP (learn more at the OAuth for IMAP mailing list).

One of the first companies using this feature is Syphir, in their SmartPush application for the iPhone, as shown in the screenshots below. Unlike other push apps, Sypher's SmartPush application never sees or stores the user’s Gmail password thanks to this new OAuth support.



We look forward to finalizing an Internet standard for using OAuth with IMAP/SMTP, and working with IMAP/SMTP mail clients to add that support.

Monday, March 29, 2010

Student Applications Open for Google Summer of Code 2010

Want to work on a cool open source project, hone your development skills with the help of a dedicated mentor, and get paid? Look no further - student applications are now open for Google Summer of Code™ 2010.

Since its inception in 2005, the Google Summer of Code program has brought together nearly 3,400 students and more than 3,000 mentors from nearly 100 countries worldwide - all for the love of code. Through the program, accepted student applicants are paired with a mentor or mentors from participating projects, thus gaining exposure to real-world software development scenarios. They also receive an opportunity for employment in areas related to their academic pursuits. And best of all, more source code is created and released for the benefit of users and developers everywhere.

Full details, including pointers on how to apply, are available on the Google Open Source Blog.

Monday, March 22, 2010

Making APIs Faster: Introducing Partial Response and Partial Update

At Google, we strive to make the web faster. Today, we’re proud to take our first big step in making APIs faster by introducing two experimental features in the Google Data Protocol, partial response and partial update. Together, partial response and partial update can drastically reduce the network, memory, and CPU resources needed to work with Google APIs.

It’s easy to understand the benefit of partial response and partial update. Imagine that you are writing a new Android calendar widget, and you want to display the time and title of the recently changed events on your Google Calendar. With the old Calendar Data API, you would request your calendar’s events feed and receive a large amount of information in response -- including lots of extra data like the attendee list and the event description.

With the addition of partial response, however, you can now use the fields query parameter to request only relevant information -- in this case, event titles and times. Constructing such a request using the fields query parameter is simple:

GET http://www.google.com/calendar/feeds/zachpm@google.com/private/full?fields=entry(title,gd:when)

By including the entry argument and specifying title and gd:when, this request ensures that the partial response contains only the title and time for each event, along with a small amount of wrapping metadata.

But say you want to also enable the widget to change the time of calendar events. With partial update, you can easily accomplish this: simply edit the data you received in the partial response and use the HTTP PATCH verb to send the modified data back to the server. The server then intelligently interprets your PATCH, updating only the fields you chose to send. Throughout this entire read-modify-write cycle, the unneeded data remains server-side and untouched.

Now for a quick demo. If you’re currently logged into a Google account, compare the size of your full calendar feed and your partial calendar feed. When we ran this test, our full calendar feed contained 160 kB of data while the partial feed only contained 8 kB -- the partial response successfully reduced total data transfer by 95%! Performance enhancements like this are especially apparent on a mobile device, where every byte of memory and every CPU cycle count. In nearly all clients, partial response and partial update make it more efficient to send, store, parse, cache, and modify only the data that you need.

As of today, partial response and partial update are supported in four Google APIs:
... and we’re planning on adding support for most of the APIs that are built on the Google Data Protocol soon. Stay tuned for more information, and if you can’t wait, feel free to lobby for partial update and partial response in your favorite API’s public support group. And for those of you who’ll be at Google I/O this year, be sure to check out the Google API sessions that are in store.

Thanks for joining us in our effort to make APIs on the web as fast and as efficient as possible!

Friday, March 19, 2010

Faster Subversion Hosting for Project Hosting on Google Code

When we launched our first Subversion-on-Bigtable service in 2006 our goal was to scale to support hundreds of thousands of projects, with the idea that we could continue to improve the service over time. A year ago, however, we realized that we would have to rebuild our Subversion service to make dramatic improvements in performance. So, we did what we had to do: we rebuilt our service from the ground up, focusing on speed and reliability.

We are now happy to announce that we have rolled out our new service to all our Subversion users. As a result, most common Subversion operations are about 3 times faster than they used to be.

One of the features of Subversion's HTTP-based protocol is that anyone can browse repositories through a normal web browser. Many open source projects hosted on Google Code use this feature to host websites for their project or post the latest versions of their software. We didn't anticipate how popular this would be when we designed our first Subversion service, but our new system has special optimizations for browser access. Latency for these pages are much lower and international users will see a dramatic improvement. We also set the appropriate caching headers, which can be manually controlled with the google-cache-control Subversion property.

To improve our reliability, our new service now has a custom replication system based on the Paxos algorithm. Whenever you make a change to your repository, the new data is now copied to several different data centers before our service reports that the commit has succeeded --- so you can code in peace knowing that your data is stored safely in multiple locations.

If you haven’t already, we encourage you to try out our new Subversion service and let us know what you think.

Tuesday, March 16, 2010

Ignite at Google I/O 2010!

Ignite will be at this year’s Google I/O! Last year, we had talks on big data, cartography and DIY devices -- a "typical" Ignite line-up. This year, our line-up includes folks like Cheezburger CEO, Ben Huh, and digital artist, Aaron Koblin. However, we also want you! We want to hear your cool ideas, hacks, how-to's, and war stories.

Each Ignite talk is 5 minutes long -- with 20 slides and only 15 seconds a slide (they auto-advance) -- and I'll be hosting the talk at I/O. If you’re not sure what to talk about, watch Scott Berkun's excellent How and Why to Give an Ignite Talk.

Submit your talk by March 31st, and we’ll announce the selected speakers on April 3rd. Those who are chosen to give an Ignite talk will receive a free ticket to Google I/O.

If you need further inspiration, you can watch any of the hundreds of Ignite videos at Ignite Show.

By Brady Forrest, O'Reilly/Ignite

@brady

Happy 5th Birthday Google Code!

How time flies! It was about five years ago that we launched Google Code to the world. When we launched Google Code, we wanted to make code.google.com a great resource where developers could learn about our vision for open source and the open web. We started in 2005 with a handful of our own open source projects, links to just eight APIs and an announcement of the first ever Google Summer of Code, our now-annual program that introduces university students to open source development. By 2006 our API list had grown to 21, in 2007 there were 37, and today our collection of more than 60 APIs receive over four billion hits per day. Check out the changing face of Google Code below -- from 2005, 2007 to the present.



With the meaning of open in mind, Google Code set out to foster best practices in developer documentation, build a community around web and open source development and demonstrate the power of Google technologies. Over the past five years code.google.com has come a long way from eight APIs, maturing into a destination for developers to explore our growing family of APIs and developer products, whether they speed up the web, alleviate cross-browser issues, make hosting web applications easy and scalable or make the web a more social place.

Google Code has also become an interactive place to share ideas. Not only can developers prototype their work in a Code Playground, they can also use Project Hosting on Google Code, a fast, reliable and easy way for developers to host all kinds of open source projects. Today, there are more than 240,000 projects registered, with commits coming in at about 17,000 per day...about 1 every 5 seconds. We also host 800 open source projects of our own, including four projects (Android, Chrome, Chrome OS and GWT) with over a million lines of code each.

It’s been an amazing five years, but there’s still a lot of work ahead. We’re dedicated to helping the developer and open source communities thrive in as many ways as we can. To celebrate our birthday and thank everyone for supporting code.google.com over the years we’re rolling out a new, faster Subversion server, which will double the source code storage for Project Hosting on Google Code from 1GB to 2GB. Happy coding!

Monday, March 15, 2010

Fireside chats at Google I/O

At Google I/O (just 2 months away!), we're excited to bring back a series of sessions called fireside chats. Fireside chats are smaller, intimate sessions where Google teams will talk about major developments with their product, what's in store for developers, and spend most of the time answering burning questions from the audience.

I/O will feature the following fireside chats:
  • Fireside chat with the Android team
    Speakers: The Android team with Chris DiBona moderating

    Pull up a chair and join the Android team at Google for a fireside chat. It's your opportunity to ask us about the platform and to tell us where you'd like to see it go in the future.

  • Fireside chat with Android handset manufacturers
    Come join us for a fireside chat with the top Android handset manufacturers. Hear about the types of devices being planned for 2010 and get your device-specific questions answered.

  • Fireside chat with the App Engine team
    Speakers: Brett Slatkin, Guido van Rossum, Matt Blain, Max Ross, Don Schwarz, Alfred Fuller, Kevin Gibbs, Sean Lynch
    It's been an busy year for the App Engine team with lots of new features and lots of new developers. Come tell us about what you've loved and what still bugs you. With several members of the App Engine team on deck, you'll get the answers to your questions straight from the source.

  • Fireside chat with the Google Chrome and Chrome OS teams
    Speakers: Ian Fette, Brian Rakowski, Linus Upson, Caesar Sengupta, Matt Papakipos

    Curious about what's new in Google Chrome, or what makes Google Chrome OS so exciting? We'll talk briefly about the major developments over the past year, and then field questions from the audience. If you're dying to know something, this is the place to find an answer.

  • Fireside chat with the Enterprise team
    Speakers: Chris Vander Mey, Scott McMullan, Ryan Boyd, David Glazer, Ken Lin

    With the launch of the Google Apps Marketplace, we've introduced a new way to expose your software to businesses - and a new way to extend Google Apps. If you're interested in building apps, what we're thinking about, or if you have other questions about the Marketplace, pull up a chair.

  • Fireside chat with the Geo team
    Speakers: Thor Mitchell, Peter Birch, Matt Holden, Ben Appleton, Bart Locanthi, Thatcher Ulrich

    Here's your opportunity to pick the brains of the people behind the Maps, Earth, and Maps Data APIs! We'll take a quick walk through the milestones of the last year, and then open it up to your questions. Don't miss your opportunity to get the straight scoop on all that's new in the world of Google Geo APIs.

  • Fireside chat with the GWT team
    Speakers: Bruce Johnson, Joel Webber, Ray Ryan, Amit Manjhi, Jaime Yap, Kathrin Probst, Eric Ayers

    If you're interested in what the GWT team has been up to since 2.0, here's your chance. We'll have several of the core engineers available to discuss the new features and frameworks in GWT, as well as to answer any questions that you might have.

  • Fireside chat with the Social Web team
    Speakers: David Glazer, DeWitt Clinton, John Panzer, Joseph Smarr, Sami Shalabi, Todd Jackson

    Social is quickly becoming an integral part of how we experience the web, and this is your chance to pick the brains of the people who are working on Buzz, the Buzz API and the underlying open protocols such as Activity Streams and OAuth which are an essential component of a truly open & social web.
You can check out the fireside chats and other sessions on code.google.com/io. The teams are looking forward to your questions!


Posted by Christine Tsai, Google I/O team

Friday, March 12, 2010

Coming soon: Gmail contextual gadgets available for trusted testers

At Campfire One this week we announced that we will soon open Gmail contextual gadgets as a new extension point for developers. These gadgets can smartly draw information from the web and let users perform relevant actions based on the content of an email message, all without leaving the Gmail inbox. For instance, contextual gadgets currently available in Gmail can detect links in emails to show previews of documents, videos, photos, and more, right inside the messages.

For businesses, Gmail contextual gadgets can boost employee productivity by complementing email in a context-specific and actionable way. Appirio, a cloud solution provider, provided a demonstration of the potential of Gmail contextual gadgets and other experimental features
with their new product PS Connect:



Soon we’ll be opening Gmail contextual gadgets as an extension for trusted testing by developers. If you have a good idea for this type of gadget today, please fill out this form. And for those of you who will be attending Google I/O in May, be sure to check out our session on building Gmail contextual gadgets.

Tuesday, March 09, 2010

Integrate, Publish, Sell - The Google Apps Marketplace

The Google Apps Marketplace, announced this evening at Campfire One, allows you to publish applications which integrate with Google Apps and sell them to more than 2 million businesses. Listing your integrated cloud app on the Google Apps Marketplace enables it to have seamless OpenID-based single sign-on with Google Apps, OAuth-authorized access to Google Apps data and makes it easy for customers to access your application from Google Apps' universal navigation bar.

There are three simple steps for a Google Apps Marketplace application:
1) Have or create a cloud application and host it on the platform of your choice
2) Integrate your cloud app with Google Apps using available APIs
3) Create a manifest file describing your application and list it for sale on the Marketplace

The Google Apps Marketplace launched with more than 50 applications from companies like Intuit and Atlassian, with more coming soon from companies like NetSuite and SuccessFactors. We hope you'll join them by integrating with Google Apps and selling your applications on the Google Apps Marketplace to our more than 25 million users and 2 million businesses.

For more information on the Google Apps Marketplace, please see our post on the new Google Apps Developer blog. You can also dive into the details of how to sell your application in the Google Apps Marketplace by visiting our Developer Programs site and technical documentation. And if you'll be at Google I/O this year, you can learn first-hand from the team in the Google Apps sessions on May 19-20 in San Francisco.

Check out the Campfire One announcement on YouTube:


Monday, March 08, 2010

Open Source Projects - Apply for Google Summer of Code

Google Summer of CodeTM, our flagship program to introduce college students to open source development, opens today. Over the past five years, we've seen more than 3,400 successful students "graduate" from the program, and we're looking forward to welcoming another group of students for our sixth year. We're now accepting applications from open source projects who wish to act as mentoring organizations and will begin accepting applications from students on March 29th. For more details, check out the Google Open Source Blog.

The Business of Code, The Code of Business

This post is part of the Who's @ Google I/O, a series of blog posts that gives a closer look at developers who'll be speaking or demoing at Google I/O. This post is written by Albert Wenger, partner at Union Square Ventures (and still enjoys writing code!). Albert will be speaking alongside others in venture capital on a panel at Google I/O.

Reading the Google Code blog, it is hard not to marvel at the fundamental transformation that is taking place in the business of code. By the business of code, I mean the economics of developing and selling software. My first exposure to the software business was as a teenager in Germany some twenty five years ago. Driver's education there is quite expensive because one has to take many mandatory lessons. After all, once you have passed you get to drive on the Autobahn, which, to this date, has long stretches without a speed limit! I thought I was being clever by agreeing to write software for a driving school in exchange for free lessons. It turns out the clever one was the owner of the driving school who turned around and sold the software to several other schools.

So what would it have taken for me then to become an ISV (independent software vendor), other than actually having the idea? These were the early days of PCs. I would have had to spend a lot of money on marketing and a lot of time on in-person sales and on-premise / telephone support. Most ISVs at the time grew locally for that reason and it was not uncommon to have highly fragmented markets with literally dozens of different vendors. As the software would have grown past the very simple initial functionality that I had created, I would have had to write pretty much everything I needed myself. Need billing? Write a billing module. Need asset tracking? Write an asset tracking module. The marketplace for components evolved only much later and was almost as fragmented as the ISV market.

This situation persisted until quite recently. In fact, in 2003 I spent a year getting to know the market for software for trucking companies in the US (Why? That's a long story). That market still had essentially the same characteristics: highly fragmented, regional customer bases, and almost 100% monolithic custom systems.

Since then, the situation has changed dramatically. Today, creating new software means focusing on what the unique contribution is that one wants to offer and figuring out how to integrate with everything else. Need a spreadsheet? Use Google Apps. Need telephony? Use Twilio (disclosure: Twilio is a USV portfolio company). Marketing can happen on the web through keyword advertising and, better yet, SEO and customer sharing. For many solutions, even sales can be entirely web-based (enter a credit card!). Support can happen over the web and often users can support each other through community. Add cloud computing to the mix and you eliminate the fixed cost that was such a high barrier to entry in the early days of the web (I remember the extraordinary bills for servers and bandwidth in 1999!).

All of this has made it possible for small teams to create big successes. It is amazing what a few great coders can do, leveraging all the services that are now available. It is,however, not just the cost side of the business of code that has changed dramatically. Competition has gone global. Someone in a faraway place can create a system, and it is instantly available everywhere. The days of the profitable, regional ISV business are over. The source of competitive advantage has also shifted. In the past, if your solution had better features, you could land a sale even against a competitor that had more customers. Now, better features don't mean that much if the larger competitor has built a network effect into their business. Imagine trying to start a LinkedIn or Salesforce competitor with better features. So the very same forces that are making it much easier to get started are making it much harder to build a successful and sustainable business.

Does that mean that there will be fewer opportunities going forward? Maybe. But there is a strong countervailing force that is creating important new opportunities: the code of business is also changing. By the code of business, I mean how companies and industries (and even societies) are organized. At Union Square Ventures, we are convinced that over time, the Internet will transform most, if not all, industries as much as it is changing the software business. This has started with the media industry, where after years of prediction of change we are now seeing massive shifts.

The reason that the code of business will change is that much of it is based on historic constraints on the bandwidth and latency of information flows. For instance, a command-and-control type hierarchy is still at the heart of (almost) all large corporations. Information flows up the hierarchy with middle management in charge of aggregating information flows. Commands then flow down with middle management translating into finer grained actions. This basic structure dates back to a time of messengers and telegraphs. Corporations are slowly shifting away towards more of an Internet architecture of "small pieces, loosely joined" -- but in many cases the end state may mean that the "pieces" are independent, small companies instead of units of a large company.

These changes will take a great deal of time (decades) because existing structures have a ton of inertia. Far more people tend to be interested in preserving the status quo than in making radical changes. Also, when a whole system needs changing, it is often difficult to get there one piece-at-a-time because all the components need to fit together. But as they start to occur in other industries, the opportunities will be massive. To give just one example, consider education: the size of the textbook industry alone in the US is estimated at $7 billion annually. This is a pure content business ripe for disruption.

Enough reading -- time for everyone with a transformative idea to start coding!

Thursday, March 04, 2010

Registration for Google I/O 2010 is now closed

This year's conference is now sold out, which means we'll be seeing over 4,000 of you on May 19-20 at Moscone West! For those of you who can't join us in person, video recordings of all sessions and keynotes will be available on YouTube following the conference.

Continue to follow us on Twitter for updates on sessions, speakers and the Sandbox. We'll also continue posting updates and Google I/O-relevant content on this blog.

Wednesday, March 03, 2010

SCVNGR and QR codes in location-based mobile gaming

This post is part of the Who's @ Google I/O, a series of blog posts that give a closer look at developers who'll be speaking or demoing at Google I/O. This guest post is written by Seth Priebatsch, Chief Ninja of SCVNGR, who's creating a mobile game for the conference.


SCVNGR is a platform for quickly and easily building location-based mobile games. Each game is all about doing challenges at places. Go here and take a photo, go there and solve this riddle. You happen to be at this coffee shop? Awesome! Try this challenge and earn a couple points! SCVNGR powers games for all sorts of institutions ranging from Princeton to Harvard to the Smithsonian Institutes to SIGGRAPH and even the U.S Navy.


If you're attending Google I/O this year, you'll get to try out our mobile game at the conference! (Don't forget to bring your Android phone, if you've got one!) I'm not going to give it all away here, but I do want to talk about one of the especially cool features that we're rolling out using some neat Google APIs.


One of the biggest challenges that our game-builders face is how to build location-based challenges that truly verify the user has actually made it to the right location. There are some non-technical solutions to this problem, such as creating riddles that require the user to be there to solve them (i.e. what is the third word on the fourth line of the plaque on the back wall) or taking photos which are then verified manually by the community or the game developer themselves.


We've also looked at a number of more technical solutions. The most obvious being to take the geo-tagged coordinates of each of the locations with a game and then use GPS to ensure that the player is within a certain radius of the location. Unfortunately, GPS verification has issues when the locations are indoors (as many are) and can vary greatly in accuracy across difference devices.


A new option, one that we're launching in a couple of weeks, uses QR codes planted at locations within the game-board. Players must scan these QR codes to verify that they've made it to the right spot. We're using the Google Charts API as an easy way to programmatically generate QR codes. Of course, generating and planting the QR codes is only half of the equation. You've also got to be able to decode them from the phones during the game. We experimented with a couple of options as to how to best achieve this.


Our first thought was to simply have the players snap pictures of the QR code which we'd then post back to our server, decode and respond with whether or not that was in fact the right QR code (or a QR code at all). The benefit of this solution was only having to utilize one QR code processing library for both our iPhone and Android applications. But we ran into a couple issues right up front:


  1. The time cost incurred by having to transmit a reasonably high-resolution image to our servers.
  2. Most players aren't very good at taking pictures of QR codes. They move the lens of the camera very close and snap the picture. (It's actually best to take the picture from 12-24 inches away to enable the camera to focus sharply on the QR code.) This led to a high-number of failed submissions before we were actually able to recognize a valid QR code.

Add all this up and it created a pretty poor game-play experience.


So we turned to the ZXing project (pronounced Zebra Crossing) which is an open source barcode processing library written in Java and highly suited to the Android environment. Running the ZXing code right on the device rid us of the time-delay introduced by transmitting images to the server. But we still had the issue of the high-fail rate of the user snapping unsuitable images. Rather than trying to implement any form of "auto-scanning", we've chosen to simply grab images from the camera every 1/8 of a second or so, scan them and stop the process once a QR code is recognized.


As for the iPhone, ZXing has an objective-c port for the iPhone, but in order to grab images in real-time from the camera, you'll have to use a private API call for iPhone OS 3.1. Luckily, Apple has officially authorized its public usage until it's made public.


We're hard at work integrating QR codes into the great game that we're building for Google I/O and we hope you'll get a chance to play. The I/O team will let you know when the game is ready! The SCVNGR team will also be there in person, so please come by and say hello! We'd love to get your thoughts on the game or just chat about some of the Google APIs that we've used within SCVNGR.


Google PowerMeter API introduced for device manufacturers

Today we're excited to introduce the Google PowerMeter API on code.google.com, for developers interested in integrating with Google PowerMeter. This API will allow device manufacturers to build home energy monitoring devices that work with Google PowerMeter. We're launching this API in order to help build the ecosystem of innovative developers working towards making energy information more widely available to consumers.

In today's launch of the API on code.google.com we are highlighting the core design principles towards integrating with Google PowerMeter. In particular we outline the underlying data model and the accompanying protocols to ensure that Google PowerMeter provides consumers access to their energy consumption with utmost care in maintaining the user's privacy and control on access to the information. We also highlight, with code samples and client implementations, how to easily start building your PowerMeter-compatible device.

Tune into our blog and subscribe to our notification list for announcements on upcoming developments. We are thrilled to bring together a rich framework to help more developers integrate with Google PowerMeter with our open, standards-based API.  We are looking to expose expanded features of this framework to the developer community in the coming months.

Finally, we want your feedback! Ask questions, suggest topics, and share your stories. You can do this at the Developer Lounge section of the Google PowerMeter forum.

We hope you join us for the ride ahead.