We currently use 10 web dynos and 6 background workers costing us about $500 for just the application layer. Considering high performance of Heroku dynos, its quite good. However, Heroku’s dedicated DB plans are not that sweet of a story. Minimum offering is $200 Ronin which comes with 1 compute unit and only 1.7 GB of RAM. For any Database of considerable size, that’s nothing. Our DB holds about 20 Million records and approximate 6/7 queries hitting it at any second. We started to see wait times on Ronin and our end user were not happy.

To alleviate the pain for our users, we updated the DB plan to Fugu ($400) considering the fact that it has 5x more compute units. That did us NO good. Investigation revealed that if your queries are less in number but high in arithmetic (lots of count, max, min etc) than this plan suits you but for high query frequency you need IKA plan for a decent start. And guess what; this plans cost $800!

Heroku while being expensive is not all bad. The ease of use via web console and Heroku command line is great. Do note that you can scale the application layer in seconds but for DB, you have to take an outage and export your DB to a bigger plan – so no quick scaling for DB layer.

Customer service was OK. Response time is quick but we only got superficial response times for the most part while investigating scaling issues. Finally we signed up New Relic Pro to our app (additionally costing us $30 a day)

My advice is carefully choose Heroku if you got the buck!

1) Bugs are nasty – keep an eye on them

Like any piece of software mobile apps whether iOS or Android, are prone to software bugs. Mobile apps are usually thick clients with a few megabytes of code running on a client handset and throwing exceptions daily. Its important to catch them over the network and compare the severity and quantity of exceptions over builds. A very useful service is BugSense who hooks into your iOS compiler and Android JVM, and records all handled and unhandled (fatal/crashes) exceptions over the network. To save network data, you can optimize recording of bugs by putting a delay and bulk recording of exceptions. BugSense provides detail information along exception trace like application number, hardware and OS etc. A great internal test cycle and unit test habits are important but I have found non-technical user possessing magical powers to produce scenarios and errors that techies miss out while testing.

2) Versioning

iTunes and Android MarketPlace make it pretty simple to version mobile apps. You can release a new version via iTunes or market and user gets the notification however what you can not do via iTunes or market is ‘force’ an update to client device. For example, on a Halloween, you accidentally published an app that messed up user data or had some other critical bug. Your day just got scarier! You can release a fixed version again but there is a propagation delay (sometimes 4-5 hours on Android market). We worked around the problem by building a simple yet robust versioning system within our apps. Its all about keeping a version number within build plus creating a web service on server side to check the version. With a simple boolean flag you can block an application from further usage only to prevent further destruction of user data and love for your app.

3) Testing of Apps

This is the most exciting part. As discussed above, once a bad build is shipped then its impossible to retrieve. So you really have to test the apps thoroughly before shipping. Unfortunately, automated testing of mobile apps is not as simple as automated web site testing. There aren’t any tools available – if there are then they are expensive. Also you need a testing team which can write pretty good code to ‘auto-test’ the app.

Another challenge is diversity of Android handsets which will drive you crazy. Almost every manufacturer manipulates the Android Kernel and as a result some feature of your app stops working. Its impossible to buy all handsets. Possible options are limiting the handsets to 8-10 famous sets or buy a slightly expensive Device AnyWhere account (costs around $3000 as a starter). Its quite slow though but they have most sets available for remote testing.

4) Network Response (3G and Wifi)

Application experience changes drastically because of network and data speed. If you only test on Wifi, you will never know how your application behaves on cellular data which is slower. Testing on a slower network also implies you to really think about use of network in your app and optimizing it. It is also important to build in your app handy messages to educate user when network slowness is frustrating user. ideally if there isn’t a response in 7-10 sec, user should know why and in a  simple language (not HTTP error code language)

e) User Feedback

We have found it handy to build a quick and simple Feedback feature with in your app. While user can leave comments on iTunes and Market but they take time. User will log in to market and leave a message. While a quick button on your app that populates the feedback screen can let user encourage or scare you with in seconds. With a simple integration within your server you can put a dashboard to list the feedback. Ont tip: always log users hardware, app version and OS to stay on top of issues.

thanks

Android marketplace does not provide a mechanism to ship a beta or test version of Android apps. So alternatively you can email the apk to users which is not most efficient. A better solution is to setup a webpage or a hockeykit instance.

In case you decide to build your own webpage, the simple anchor <a> link will not allow the download of app to Android handsets. You need to alter the anchor tag:

<a href=”downloads/my_apk.apk” type=”application/vnd.android.package-archive”>Download App</a>

Also you need to check ‘Allow untrusted sources’ under Settings->Manage Applications on your Android handset to install a non-market apk

happy holidays!

We recently did an extended research on available cloud platforms for a client. The requirements were however no straight forward. Client’s top priorities were ability of platform to scale automatically, ease of maintenance, and performance of infrastructure. Then, to make the task challenging and exciting, the client wanted a geographical fail-over ability in case of a disaster. Amazon alone had 2 major outages in year 2011 so users are quite concern about business continuity in time of a disaster.

We evaluated many options starting from Amzon EC2, RDS, Heroku, Bluebox, RackSpace, High Velocity, Xeround etc and concluded that only one provider will not be able to answer all questions from our client. So we divided the architecture into App layer and DB layer and choose different hosts for each layer. In such a division the biggest concern is network latency but interesting fact is most cloud platforms are built over EC2 with additional wrapping so within EC2 network latency is not huge at all. We ran some test with Heroku (app) and Xeround (DB) and found network latency to be less than half a sec per transaction.

You can find below a presentation we put together after our analysis of different cloud hosts. There is a slide that links to a Google spreadsheet with calculation of estimated cost for our proposed infrastructure. Feel free to drop a comment to correct me or ask me

Presentation:

An mentioned in part 1 of the post, this information is by no means exhaustive but to server as a guide line for performance tuning.

Load Test Simulator

We used a very simply and handy tool, JMeter, to simulate load on our server. It gives nice flexibility in terms of setting up request load (multiple threads) and we can also configure data pool to diversify payload. The load tests can be scheduled or setup for fixed time. The results are real time and quantitative (no charts or graphics though). Attached a couple of sample screen shots from JMeter that demonstrate thread settings and real time results of a sample load test

Some Cheeky Moves that Enhance Performance

In Memory Processing is a Blessing: Processing (calculation, comparison, searching is much much faster in memory than on disk. Though MySQL comes with native SQLCache (read here). Other options such as Memcache are also available that give user more control on caching policies

Group Process Database Requests: A common programming practice is to programatically loop DB inserts. For large data sets this becomes quite a performance bottleneck because of a separate DB transaction being initiated every time (new connection, new statement object, new commit every iteration). This bottleneck can we avoided by using a group insert e.g.: insert into _table  (col1, col 2) values (‘val1′ , ‘val2′), (‘val3′,’val4′) , (‘val5′,’val6′) …..

Move Independent Processing to Background: In our experience quite some processing (like firing emails, PUSH, or updating OLAP schemas) can be moved to background to run independently using a good background processing solution. The one we used was Recue. It uses Redis to create a namespace and is quite efficient

:include is faster than JOIN: take a look at http://stackoverflow.com/questions/1208636/rails-include-vs-joins

Calibrating App Server and DB Threads

A normal processing sequence of a request landing on a server is : HTTP Server -> Interpreter/Compiler -> DB

Modern day Web, application an DB servers come with configurations to increase or decrease thread counts. More threads mean more parallel processing BUT more system resource requirement as well so its very important to:

a) Set the number of threads for each server layer that does not burn out your CPU and Memory, OR does not use CPU and memory at all

b) Keep a balanced ratio between server layers. If Application server threads are too many and DB threads are too low then most of the time application server threads will be hitting DB asking for a connection and producing load that does not help user

Apache Server Threads

Apacher server config can be calibrated to set an optimal value for MaxClients which represents the simultaneous requests the apache server will handle. At any time apache threads can be viewed by hitting http://your-server/server-status. A sample snapshot follows:

MySQL Threads

MySQL uses a thread based client connection model. The more the connections, higher will be load on MySQL and system hardware resources. Read more

Monitor Your Load Tests

Load tests are as good as the results deduced from them. The results should at the minimum record the following:

a) System Throughput (No of requests going in per sec VS number of  requests completed per sec) – we used JMeter for this purpose

Screenshot:

b) System Response Time (Time to entertain single request. There is no point of taking a load too high and keep user waiting for a response) – we used JMeter to record System response time

Screenshot:

c) Server Hardware Status (CPU, memory, Disk-Swap are the most important in my opinion) – we used monit for this purpose

Screenshot:

Have fun optimizing!

Clearly one nor want too small a server muscle, neither too big and expensive of a server. To look eye-to-eye with the load creep, there is need of a strategy. Here is part I of my kung-fu attempt to chop it down. I would like to keep it high level and at a strategic level. In part II I will discuss specific details of

our recent optimization exercise on Rails. I must disclaim that server tuning or optimization is a big topic with tons of material available and what follows is my experience working with load testing and optimization of a Ruby on Rails server for a messaging engine project.

Before You Start Optimizing and Scaling:

a) Size the Beast: Estimate your target load on server. I prefer a number like transactions per min rather than transactions per second. Per seconds is much smaller a number when you have heavy processing and a normal transaction spans over a few seconds. It is also important to clearly define what a transaction means in your system, does a single DB hit counts for a transaction or entertaining a user request end-end counts for a transaction?

b) Benchmark Response Times: The goal of optimization and scaling should not be solely handling more load on the server, but also serving requests within a decent response time. A server handling tons of loads but keeping the user waiting for longer period will soon put the CEO out of business

c) Choose an appropriate load generation mechanism: This could be a free tool like JMeter or SOAPUI who can can create massive HTTP hits on the server. The flexibility these tools provide is quite nice ranging from configuring exact load to put on server using multi-threading and ability to attach a data pool to vary request data. You can also write your own code to generate a load if the request structure is complex. In our case, we used both.

Places to look for Optimization:

a) Starting with code optimizations. Hotspots are DB calls, third party web service calls and parsing large JSONs, XMLs etc. I have experienced that using an async approach of DB writing and JSON/XML parsing (wherever possible) greatly improves system performance and user experience. We optimized one of our routines by 800% using asynchronous DB writing

b) Application server threads: Application server request threads should always maintain appropriate ratio with hardware muscle. You don’t want to do too much or too less parallel request handling on application server. Too much will lead to CPU or Memory starvation and too less means you have getting an oversize server invoice month end. With out pretty standard request size, we have enabled 50 maxClients for Apache on standard EC2 XLarge instance and hitting about 50% of CPU capacity

c) Caching: Caching saves us from disk and notwork latency by reusing already fetched data. Caching is also available at multiple levels starting from Web serer caching, SQL caching provided by standard RDBMS and third party caching such as Memcached

d) DB Indexing: This is not something super latest or cutting-edge and has been in use for a while, but, there is a catch. Normally we create DB indexes on tables whom we hit the most in searching etc. However, if there are massive CUD operation (Create, Update and Delete) on the table as well then indexes will really slow them down because it updates the B-trees every time

Guide Yourself in Load Testing:

a) A cyclic approach is what works. Run more than one tests while recording them. I have found it useful to create a simple spreadsheet that records details and results of every test run. I suggest to record basic information like hardware profile, change in settings/hardware from previous test, load put on server, throughput of the server, exceptions/crashed, and duration of test

b) Its important to bring one change at a time to the system – let it be DB index, memcached, or more memory attached to the system If we bring more than one changes to the system for test run then it will be hard to determine the adverse or positive affect of a change independently

c) Profile your system: We recorded following information during the tests. CPU, memory and disk usage using Munin, system throughput using NewRelic and system response times using JMeter

c) Do not forget longevity tests: While we run many short duration tests it is important to run 10 hour or a day long tests as well to figure out if there are any dormant memory leaks that might crash the system in a few days time

Below is my attempt to picture the optimization process in a simple flow chart:

In part II I will discuss specifics of our recent load testing exercise on Rails

We were recently challenged at work to design and develop a next generation multi-media messaging system for a fortune corporate. The requirements were very interesting. Starting with cross platform support i.e. iPhone, Andriod, BB, WP7 and Web, the system was also required to record live audio and video and send as real-time attachments. The diaspora of media support in different mobile platforms is quite big thus the need of inter-format media encoding. Additional features were PUSH notifications upon message receive, and coupling with client’s proprietary LDAP. We planned a stage-wise release that is starting from a group of hundred users to acceptance-test the system, we targetted a 2nd release to 10K users and then eventually to full scale i.e. 30K users. the anticipated load on the system was 30 TPS resulting from 4-5 new message check calls from every user per second.

During the design phase of the system, we faced the question of choosing between non-SOA approach (do-it-yourself) versus a services based architecture for features like media encoding, PUSH messages and file storage. Though the decision of not doing-it-yourself means lesser control, lesser customization, dependency on a third company but thanks to SOA buzz created many years ago, plenty of smart people have founded SAAS companies that cater to most users needs of media encoding, storage and notifications. So our engineering team and client decided smartly and chose the SOA approach. We evaluated different options and settled with Encoding.com for media encoding, UrbanAship for notifications and S3 for media storage. One of the reason of choosing Encoding.com was its capability of solid integration with S3 storage.

The decision of following an SOA approach helped us in many ways. First and far-most we saved so much time. Rather setting up our own server for solutions like ffmpeg and then handling scaling and reliability, we shipped all the media encoding burden on encoding.com’s shoulders. Although they do have some scalability and video rotation issues but overall they have solid services to offer. PUSH notifications are generally not fully relied upon, but UrbanAship has been pretty solid delivering notifications. As I think of the time and effort we have saved in building our next generation messaging system by choosing an SOA architecture, its quite amazing. Not tp mention the savings in cost of development for the client!

As we move forward and add new interesting features to our system, we want to build upon the already ‘handy’ SOA architecture. Our next targets are addition of great Analytics for system usage. We have boiled down the list of available services to Google Analytics and Flurry. The range of analysis provided by both are quite interesting and we are inclined to use one of these SAAS providers than building our own analytics solution.

To implement this in a graceful and backward compatible fashion, we added ‘-s<SOCKET>’ and ‘–socket <SOCKET>’ option to the Starling server by overriding some of its function documented in simobo_extensions.rb.

To run this modified Starling server, we need to created a ruby script along the same lines as the original starling script to run the server but use the overridden functionality before creating the Starling server object.

#!/usr/bin/env ruby

require 'rubygems'#  unless ENV['NO_RUBYGEMS']
require 'starling'

curr_dir = File.dirname(__FILE__)
require File.join(curr_dir, '../lib/extensions/starling_extension.rb')

StarlingServer::Runner.run

The above script assume that the new starling script is in ‘scripts’ folder and the starling extension is in ‘lib/extensions’ folder, relative to the Rails application root. Now to start the starling server, all that is required is to call the script passing in the socket file to use:

ruby script/starling -s /tmp/starling.sock

Source

Memcache server supports unix sockets, but the existing memcache clients available for Ruby only support TCP sockets. Our aim was to integrate it with existing memcache client implementation in such a way that both flavors can be used with little change in existing memcache configurations. For that purpose, we simply introduced an ‘is_socket’ flag and used the existing ‘servers’ parameters to specify the unix socket location or tcp server address.

First, we install the memcache-client gem and download memcache_extension.rb file containing the overridden functions of the client to support Unix sockets. The memcache_extension.rb file should be placed in an appropriate directory (lib perhaps), in your application, so that anytime we need to use the modified memcache client code, we insert the statement

require "memcache_extension.rb"

before creating a memcache client.

Also by default, the Memcache client uses tcp socket, and any overrides are picked up from memcache.yml file. So in order to activate the unix socket, we will need to add the following lines to the memcache.yml file:

servers: /tmp/memcached.sock
is_socket: true

Finally, ensure that memcached server is listening on the unix socket:

memcached -s /tmp/memcached.sock

Note that Rails contains a memcache client by default. To use the desired memcache-client in Rails < 2.3, we need to perform some brain-surgery originally described here, but modified slightly to fit our needs.

Source

To enable scalability of WiceGrid, which offers some great functionality, we went about integrating it with a search engine. This avoids database table locking during read queries and results in a significant performance improvement by using indexes optimized for fetching data summaries and associations. Our choice for search engine is Sphinx, although others too can be used to achieve similar results.The integration with Sphinx was carried out such that existing applications that use WiceGrid are not broken. If performance limitations using WiceGrid is observed due to massive data volume, then this is just the thing for you!

The details of the integration are below:

First step is to configure sphinx is to specify if you want search results or facets and any preference in which your results are to be sorted.

def initialize(klass, controller, opts = {})  #:nodoc:

.

.

.

.

@options = {

:after                => nil,

:conditions           => nil,

:csv_file_name        => nil,

:custom_order         => {},

:enable_export_to_csv => Defaults::ENABLE_EXPORT_TO_CSV,

:include              => nil,

:joins                => nil,

:name                 => Defaults::GRID_NAME,

  rder                => nil,

  rder_direction      => Defaults::ORDER_DIRECTION,

:page                 => 1,

:per_page             => Defaults::PER_PAGE,

:saved_query          => nil,

:total_entries        => nil,

:with                 => nil,

:without              => nil,

:sort_mode            => nil,

:is_sphinx            => false,

:is_facets             => false

}

@klass.define_indexes if @status[:is_sphinx] or @status[:is_facets]

@status[:indexes] = collect_indexes if @status[:is_sphinx] or @status[:is_facets]

.

.

.

.

.

end

A non-trivial task is is to distinguish Sphinx fields from Sphinx attributes.Fields are content for your search queries and attributes are used for filtering, sorting grouping ..etc. Sphinx server requires that searching on fields and attributes is separately passed by the client. In order to implement this, w

we first call the define_index method on the model and then collect the indexes manually.

def collect_indexes

all_indexes =[]

for index in @klass.sphinx_indexes.first.fields

all_indexes << index.unique_name

end

all_indexes

end

Now, to convert the filter options received from wicegrid, the standard sql implementation generates the ‘conditions’ clause based on the filter. For Sphinx, the ‘conditions’ clause is used for fields, whereas a separate ‘with’ clause is used for attributes. This is where the collected indexes come into play and we use them for distinguishing the two.

def form_ar_options_sphinx(opts = {})

.

.

.

add_filter_to_conditions

add_filter_to_with

.

.

.

end

In standard WiceGrid plugin, a separate query for getting the number of records is used for pagination purposes. Sphinx provides a separate count query for the data it has indexed, which is more optimized and can be used where sphinx was enabled.

alias_method :size, :count

def count  #:nodoc:

if @status[:is_sphinx] or @status[:is_facets]

form_ar_options_with_sphinx(:skip_ordering => true, :forget_generated_options => true)

@klass.search_count('',:conditions => @ar_options[:conditions],

:joins => @ar_options[:joins], :include => @ar_options[:include],

:with => @ar_options[:with], :without => @ar_options[:without])

else

form_ar_options(:skip_ordering => true, :forget_generated_options => true)

@klass.count(:conditions => @ar_options[:conditions], :joins => @ar_options[:joins], :include => @ar_options[:include])

end

end

Another feature of Sphinx is to that it provides facets which are count summaries for the sphinx indexes defined on the records. This data is very useful for analysis purposes and we aimed to incorporate this functionality into WiceGrid. Since facet results are not derived from ActiveRecord, but is basically a hash table of counts, we need to override the pagination functionality of WiceGrid to ignore pagination in case of facets.

module GridViewHelper

alias_method  riginal_pagination_panel_content ,:pagination_panel_content

def pagination_panel_content(grid, extra_request_parameters, allow_showing_all_records) #:nodoc:

return if grid.is_facets

original_pagination_panel_content(grid, extra_request_parameters, allow_showing_all_records)

end

end

Source Code