OpenStack cloud users share pros and cons

IntroCompanies that use OpenStack in production spoke out this week at the OpenStack Summit about the advantages and challenges of deploying the cloud management platform in the real world. Presenters here found OpenStack to be the most flexible choice for building their own cloud computing environments, but said the platform still requires a "do it yourself" attitude to fill in feature gaps. New York-based media giant Bloomberg deployed OpenStack cloud on one of the biggest private networks in the world, with some 200 points of presence globally. But it encountered problems "below and above the stack," according to Pravir Chandra, chief technology officer and security architect for the company, who spoke in a keynote presentation here. Below the stack, there was the matter of setting up highly available databases, application messaging in the form of RabbitMQ, and Inktank's Ceph storage. Above the stack, there were log aggregation and metrics for monitoring hardware, for which the company used the Graphite open source utility. "We had to solve a lot of these problems ourselves," Chandra said. The company puts its Chef cookbooks on GitHub for others to learn from. Samsung SDS, one of the largest information and communication technology providers in Asia, has tested OpenStack for its private cloud to deliver mobile applications to customers. It expects to have an OpenStack cloud in production by July. The company began with the Diablo release two years ago and found it very unstable, according to Kirk Kim, director at Samsung SDS in Korea. Subsequent releases have improved, but they required such customizations as the ability to auto-boot volumes from a storage area network, a Nova backup service, and network metering using IP tables. Samsung SDS also found it necessary to bring in multi-cloud management software from RightScale to span its private and public cloud environments. "OpenStack is highly flexible and configurable but also, especially in the early days, complex and error-prone," Kim said. Kim also touched on the difficulty of upgrading between releases, which in some cases have necessitated re-architecting and rebuilding components of the OpenStack cloud platform. The BestBuy.com online retail website is now entirely based on OpenStack after a yearlong process of evaluation and development. The goal of this project was to reduce the amount of JavaScript rendered at the client, and instead serve pages centrally from the cloud to improve performance. It was also meant to cut costs, according to Steve Eastham, director of Web architecture for the company. An internal study of IT costs revealed an average cost of $20,000 per managed virtual machine (VM), and the company's own quality assurance process was costing it $500,000 per major release, he said. Now the Web property has about 500 instances active in its OpenStack cloud on average, for a cost of $91,000 per rack of hardware. But as other OpenStack users have found, customization was required. One of the chief examples of this for BestBuy.com was a switch away from Gluster, an open source distributed file system now owned by Red Hat Inc., to Ceph storage. Recoveries of VM images, some as large as 200 GB each, took too long with the previous storage system. BestBuy.com also developed a custom Chef management tool called Ginsu, a plug-in that replaces the knife-OpenStack utility, and is more tightly integrated with the Nova compute application programming interface. It uses a collection of monitoring tools, including Sensu, Collectd, Graphite, and a custom dashboard based on Ruby Sinatra. This last utility caused a problem when all of BestBuy.com's 40 development teams tried to log in at once. "It was kind of a DDOS-it-yourself situation," Eastham said. The company still runs the Essex release of OpenStack, but plans to upgrade to Grizzly, the latest release, in the coming months. In general, automation is a requirement for OpenStack cloud environments rather than a nice-to-have, according to Jim O'Neill, chief information officer at Cambridge, Mass.-based HubSpot Inc., a marketing Software-as-a-Service firm. This was something the company did for itself using both Chef and Puppet. O'Neill likes the DIY nature of OpenStack. "If something doesn't work the way we like, we open it up, fix what's wrong and contribute it back to the community," he said.