Android marketplace does not provide a mechanism to ship a beta or test version of Android apps. So alternatively you can email the apk to users which is not most efficient. A better solution is to setup a webpage or a hockeykit instance.

In case you decide to build your own webpage, the simple anchor <a> link will not allow the download of app to Android handsets. You need to alter the anchor tag:

<a href=”downloads/my_apk.apk” type=”application/vnd.android.package-archive”>Download App</a>

Also you need to check ‘Allow untrusted sources’ under Settings->Manage Applications on your Android handset to install a non-market apk

happy holidays!

We recently did an extended research on available cloud platforms for a client. The requirements were however no straight forward. Client’s top priorities were ability of platform to scale automatically, ease of maintenance, and performance of infrastructure. Then, to make the task challenging and exciting, the client wanted a geographical fail-over ability in case of a disaster. Amazon alone had 2 major outages in year 2011 so users are quite concern about business continuity in time of a disaster.

We evaluated many options starting from Amzon EC2, RDS, Heroku, Bluebox, RackSpace, High Velocity, Xeround etc and concluded that only one provider will not be able to answer all questions from our client. So we divided the architecture into App layer and DB layer and choose different hosts for each layer. In such a division the biggest concern is network latency but interesting fact is most cloud platforms are built over EC2 with additional wrapping so within EC2 network latency is not huge at all. We ran some test with Heroku (app) and Xeround (DB) and found network latency to be less than half a sec per transaction.

You can find below a presentation we put together after our analysis of different cloud hosts. There is a slide that links to a Google spreadsheet with calculation of estimated cost for our proposed infrastructure. Feel free to drop a comment to correct me or ask me

Presentation:

An mentioned in part 1 of the post, this information is by no means exhaustive but to server as a guide line for performance tuning.

Load Test Simulator

We used a very simply and handy tool, JMeter, to simulate load on our server. It gives nice flexibility in terms of setting up request load (multiple threads) and we can also configure data pool to diversify payload. The load tests can be scheduled or setup for fixed time. The results are real time and quantitative (no charts or graphics though). Attached a couple of sample screen shots from JMeter that demonstrate thread settings and real time results of a sample load test

Some Cheeky Moves that Enhance Performance

In Memory Processing is a Blessing: Processing (calculation, comparison, searching is much much faster in memory than on disk. Though MySQL comes with native SQLCache (read here). Other options such as Memcache are also available that give user more control on caching policies

Group Process Database Requests: A common programming practice is to programatically loop DB inserts. For large data sets this becomes quite a performance bottleneck because of a separate DB transaction being initiated every time (new connection, new statement object, new commit every iteration). This bottleneck can we avoided by using a group insert e.g.: insert into _table  (col1, col 2) values (‘val1′ , ‘val2′), (‘val3′,’val4′) , (‘val5′,’val6′) …..

Move Independent Processing to Background: In our experience quite some processing (like firing emails, PUSH, or updating OLAP schemas) can be moved to background to run independently using a good background processing solution. The one we used was Recue. It uses Redis to create a namespace and is quite efficient

:include is faster than JOIN: take a look at http://stackoverflow.com/questions/1208636/rails-include-vs-joins

Calibrating App Server and DB Threads

A normal processing sequence of a request landing on a server is : HTTP Server -> Interpreter/Compiler -> DB

Modern day Web, application an DB servers come with configurations to increase or decrease thread counts. More threads mean more parallel processing BUT more system resource requirement as well so its very important to:

a) Set the number of threads for each server layer that does not burn out your CPU and Memory, OR does not use CPU and memory at all

b) Keep a balanced ratio between server layers. If Application server threads are too many and DB threads are too low then most of the time application server threads will be hitting DB asking for a connection and producing load that does not help user

Apache Server Threads

Apacher server config can be calibrated to set an optimal value for MaxClients which represents the simultaneous requests the apache server will handle. At any time apache threads can be viewed by hitting http://your-server/server-status. A sample snapshot follows:

MySQL Threads

MySQL uses a thread based client connection model. The more the connections, higher will be load on MySQL and system hardware resources. Read more

Monitor Your Load Tests

Load tests are as good as the results deduced from them. The results should at the minimum record the following:

a) System Throughput (No of requests going in per sec VS number of  requests completed per sec) – we used JMeter for this purpose

Screenshot:

b) System Response Time (Time to entertain single request. There is no point of taking a load too high and keep user waiting for a response) – we used JMeter to record System response time

Screenshot:

c) Server Hardware Status (CPU, memory, Disk-Swap are the most important in my opinion) – we used monit for this purpose

Screenshot:

Have fun optimizing!

Clearly one nor want too small a server muscle, neither too big and expensive of a server. To look eye-to-eye with the load creep, there is need of a strategy. Here is part I of my kung-fu attempt to chop it down. I would like to keep it high level and at a strategic level. In part II I will discuss specific details of

our recent optimization exercise on Rails. I must disclaim that server tuning or optimization is a big topic with tons of material available and what follows is my experience working with load testing and optimization of a Ruby on Rails server for a messaging engine project.

Before You Start Optimizing and Scaling:

a) Size the Beast: Estimate your target load on server. I prefer a number like transactions per min rather than transactions per second. Per seconds is much smaller a number when you have heavy processing and a normal transaction spans over a few seconds. It is also important to clearly define what a transaction means in your system, does a single DB hit counts for a transaction or entertaining a user request end-end counts for a transaction?

b) Benchmark Response Times: The goal of optimization and scaling should not be solely handling more load on the server, but also serving requests within a decent response time. A server handling tons of loads but keeping the user waiting for longer period will soon put the CEO out of business

c) Choose an appropriate load generation mechanism: This could be a free tool like JMeter or SOAPUI who can can create massive HTTP hits on the server. The flexibility these tools provide is quite nice ranging from configuring exact load to put on server using multi-threading and ability to attach a data pool to vary request data. You can also write your own code to generate a load if the request structure is complex. In our case, we used both.

Places to look for Optimization:

a) Starting with code optimizations. Hotspots are DB calls, third party web service calls and parsing large JSONs, XMLs etc. I have experienced that using an async approach of DB writing and JSON/XML parsing (wherever possible) greatly improves system performance and user experience. We optimized one of our routines by 800% using asynchronous DB writing

b) Application server threads: Application server request threads should always maintain appropriate ratio with hardware muscle. You don’t want to do too much or too less parallel request handling on application server. Too much will lead to CPU or Memory starvation and too less means you have getting an oversize server invoice month end. With out pretty standard request size, we have enabled 50 maxClients for Apache on standard EC2 XLarge instance and hitting about 50% of CPU capacity

c) Caching: Caching saves us from disk and notwork latency by reusing already fetched data. Caching is also available at multiple levels starting from Web serer caching, SQL caching provided by standard RDBMS and third party caching such as Memcached

d) DB Indexing: This is not something super latest or cutting-edge and has been in use for a while, but, there is a catch. Normally we create DB indexes on tables whom we hit the most in searching etc. However, if there are massive CUD operation (Create, Update and Delete) on the table as well then indexes will really slow them down because it updates the B-trees every time

Guide Yourself in Load Testing:

a) A cyclic approach is what works. Run more than one tests while recording them. I have found it useful to create a simple spreadsheet that records details and results of every test run. I suggest to record basic information like hardware profile, change in settings/hardware from previous test, load put on server, throughput of the server, exceptions/crashed, and duration of test

b) Its important to bring one change at a time to the system – let it be DB index, memcached, or more memory attached to the system If we bring more than one changes to the system for test run then it will be hard to determine the adverse or positive affect of a change independently

c) Profile your system: We recorded following information during the tests. CPU, memory and disk usage using Munin, system throughput using NewRelic and system response times using JMeter

c) Do not forget longevity tests: While we run many short duration tests it is important to run 10 hour or a day long tests as well to figure out if there are any dormant memory leaks that might crash the system in a few days time

Below is my attempt to picture the optimization process in a simple flow chart:

In part II I will discuss specifics of our recent load testing exercise on Rails

We were recently challenged at work to design and develop a next generation multi-media messaging system for a fortune corporate. The requirements were very interesting. Starting with cross platform support i.e. iPhone, Andriod, BB, WP7 and Web, the system was also required to record live audio and video and send as real-time attachments. The diaspora of media support in different mobile platforms is quite big thus the need of inter-format media encoding. Additional features were PUSH notifications upon message receive, and coupling with client’s proprietary LDAP. We planned a stage-wise release that is starting from a group of hundred users to acceptance-test the system, we targetted a 2nd release to 10K users and then eventually to full scale i.e. 30K users. the anticipated load on the system was 30 TPS resulting from 4-5 new message check calls from every user per second.

During the design phase of the system, we faced the question of choosing between non-SOA approach (do-it-yourself) versus a services based architecture for features like media encoding, PUSH messages and file storage. Though the decision of not doing-it-yourself means lesser control, lesser customization, dependency on a third company but thanks to SOA buzz created many years ago, plenty of smart people have founded SAAS companies that cater to most users needs of media encoding, storage and notifications. So our engineering team and client decided smartly and chose the SOA approach. We evaluated different options and settled with Encoding.com for media encoding, UrbanAship for notifications and S3 for media storage. One of the reason of choosing Encoding.com was its capability of solid integration with S3 storage.

The decision of following an SOA approach helped us in many ways. First and far-most we saved so much time. Rather setting up our own server for solutions like ffmpeg and then handling scaling and reliability, we shipped all the media encoding burden on encoding.com’s shoulders. Although they do have some scalability and video rotation issues but overall they have solid services to offer. PUSH notifications are generally not fully relied upon, but UrbanAship has been pretty solid delivering notifications. As I think of the time and effort we have saved in building our next generation messaging system by choosing an SOA architecture, its quite amazing. Not tp mention the savings in cost of development for the client!

As we move forward and add new interesting features to our system, we want to build upon the already ‘handy’ SOA architecture. Our next targets are addition of great Analytics for system usage. We have boiled down the list of available services to Google Analytics and Flurry. The range of analysis provided by both are quite interesting and we are inclined to use one of these SAAS providers than building our own analytics solution.

To implement this in a graceful and backward compatible fashion, we added ‘-s<SOCKET>’ and ‘–socket <SOCKET>’ option to the Starling server by overriding some of its function documented in simobo_extensions.rb.

To run this modified Starling server, we need to created a ruby script along the same lines as the original starling script to run the server but use the overridden functionality before creating the Starling server object.

#!/usr/bin/env ruby

require 'rubygems'#  unless ENV['NO_RUBYGEMS']
require 'starling'

curr_dir = File.dirname(__FILE__)
require File.join(curr_dir, '../lib/extensions/starling_extension.rb')

StarlingServer::Runner.run

The above script assume that the new starling script is in ‘scripts’ folder and the starling extension is in ‘lib/extensions’ folder, relative to the Rails application root. Now to start the starling server, all that is required is to call the script passing in the socket file to use:

ruby script/starling -s /tmp/starling.sock

Source

Memcache server supports unix sockets, but the existing memcache clients available for Ruby only support TCP sockets. Our aim was to integrate it with existing memcache client implementation in such a way that both flavors can be used with little change in existing memcache configurations. For that purpose, we simply introduced an ‘is_socket’ flag and used the existing ‘servers’ parameters to specify the unix socket location or tcp server address.

First, we install the memcache-client gem and download memcache_extension.rb file containing the overridden functions of the client to support Unix sockets. The memcache_extension.rb file should be placed in an appropriate directory (lib perhaps), in your application, so that anytime we need to use the modified memcache client code, we insert the statement

require "memcache_extension.rb"

before creating a memcache client.

Also by default, the Memcache client uses tcp socket, and any overrides are picked up from memcache.yml file. So in order to activate the unix socket, we will need to add the following lines to the memcache.yml file:

servers: /tmp/memcached.sock
is_socket: true

Finally, ensure that memcached server is listening on the unix socket:

memcached -s /tmp/memcached.sock

Note that Rails contains a memcache client by default. To use the desired memcache-client in Rails < 2.3, we need to perform some brain-surgery originally described here, but modified slightly to fit our needs.

Source

To enable scalability of WiceGrid, which offers some great functionality, we went about integrating it with a search engine. This avoids database table locking during read queries and results in a significant performance improvement by using indexes optimized for fetching data summaries and associations. Our choice for search engine is Sphinx, although others too can be used to achieve similar results.The integration with Sphinx was carried out such that existing applications that use WiceGrid are not broken. If performance limitations using WiceGrid is observed due to massive data volume, then this is just the thing for you!

The details of the integration are below:

First step is to configure sphinx is to specify if you want search results or facets and any preference in which your results are to be sorted.

def initialize(klass, controller, opts = {})  #:nodoc:

.

.

.

.

@options = {

:after                => nil,

:conditions           => nil,

:csv_file_name        => nil,

:custom_order         => {},

:enable_export_to_csv => Defaults::ENABLE_EXPORT_TO_CSV,

:include              => nil,

:joins                => nil,

:name                 => Defaults::GRID_NAME,

  rder                => nil,

  rder_direction      => Defaults::ORDER_DIRECTION,

:page                 => 1,

:per_page             => Defaults::PER_PAGE,

:saved_query          => nil,

:total_entries        => nil,

:with                 => nil,

:without              => nil,

:sort_mode            => nil,

:is_sphinx            => false,

:is_facets             => false

}

@klass.define_indexes if @status[:is_sphinx] or @status[:is_facets]

@status[:indexes] = collect_indexes if @status[:is_sphinx] or @status[:is_facets]

.

.

.

.

.

end

A non-trivial task is is to distinguish Sphinx fields from Sphinx attributes.Fields are content for your search queries and attributes are used for filtering, sorting grouping ..etc. Sphinx server requires that searching on fields and attributes is separately passed by the client. In order to implement this, w

we first call the define_index method on the model and then collect the indexes manually.

def collect_indexes

all_indexes =[]

for index in @klass.sphinx_indexes.first.fields

all_indexes << index.unique_name

end

all_indexes

end

Now, to convert the filter options received from wicegrid, the standard sql implementation generates the ‘conditions’ clause based on the filter. For Sphinx, the ‘conditions’ clause is used for fields, whereas a separate ‘with’ clause is used for attributes. This is where the collected indexes come into play and we use them for distinguishing the two.

def form_ar_options_sphinx(opts = {})

.

.

.

add_filter_to_conditions

add_filter_to_with

.

.

.

end

In standard WiceGrid plugin, a separate query for getting the number of records is used for pagination purposes. Sphinx provides a separate count query for the data it has indexed, which is more optimized and can be used where sphinx was enabled.

alias_method :size, :count

def count  #:nodoc:

if @status[:is_sphinx] or @status[:is_facets]

form_ar_options_with_sphinx(:skip_ordering => true, :forget_generated_options => true)

@klass.search_count('',:conditions => @ar_options[:conditions],

:joins => @ar_options[:joins], :include => @ar_options[:include],

:with => @ar_options[:with], :without => @ar_options[:without])

else

form_ar_options(:skip_ordering => true, :forget_generated_options => true)

@klass.count(:conditions => @ar_options[:conditions], :joins => @ar_options[:joins], :include => @ar_options[:include])

end

end

Another feature of Sphinx is to that it provides facets which are count summaries for the sphinx indexes defined on the records. This data is very useful for analysis purposes and we aimed to incorporate this functionality into WiceGrid. Since facet results are not derived from ActiveRecord, but is basically a hash table of counts, we need to override the pagination functionality of WiceGrid to ignore pagination in case of facets.

module GridViewHelper

alias_method  riginal_pagination_panel_content ,:pagination_panel_content

def pagination_panel_content(grid, extra_request_parameters, allow_showing_all_records) #:nodoc:

return if grid.is_facets

original_pagination_panel_content(grid, extra_request_parameters, allow_showing_all_records)

end

end

Source Code

Presentation covers topics from effective page titles, meta descriptions, URL structures, custom 404 pages, effective Content & keyword analysis, optimizing images seo, Sitemaps, usage of Robots.txt, PageRank and Beyond.

Through Pakwheels, users can search classifieds, research about brand new cars, discuss car related issues with other members, share information, pictures, videos, etc. Confiz Solutions is the technology partner for Pakwheels.com

Ruby on Rails/MySQL for core development.

Migration

The original solution designed for Pakwheels was based on ASP and SQL Server 2000 technologies. The solution designed was not based on high performance modules and a scalable architecture.

Images to Amazon services

Amazon Simple Storage Service (S3) is an online storage web service offered by Amazon Web Services. Using this, we ensure efficient use of bandwidth and scalable file storage system.

Data Migration

This being the most significant step to completing the migration required a smooth and stable solution. Data had to be migrated from MS SQL Server 2000 platform to MySQL.

Recovery mechanism design

To make sure that there is no data loss in an extremely unlikely case of system downtime, we worked out a recovery mechanism that makes the solution even more reliable.

User experience design – Community feel

More community features, Enhance traffic, Better users retention.

The basic idea was to build feature requirements and interaction plans based upon the user’s goals. Pakwheels interactive system is meant to positively impact the overall experience of it’s users.

• Wire framing and Story boarding based on actual user input.
• Flows and navigation maps designed based on user stories and scenarios.
• A/B testing was performed to ensure that the newer version performs better on business goals.
• Forums were audited by select social groups. This helped us facilitate notable user behaviors.
• Optimization: Better placement of widgets, page components, and search elements resulted in better experience.

The interaction design makes the community features more effective. The users of the forums actually add value to the community which makes the portal community-centric.

One stop shop solution

Single Sign-On integration within Pak eVentures umbrella products.

Pak eVentures umbrella products include,

• PakWheels – #1 automobile website in Pakistan dealing in classifieds and a wealth of auto-related information
• PakBoard – a classifieds platform for everything you can think of: laptops, mobiles, real estate, services etc. The craigslist of Pakistan.
• PakShops – prop up your own shop online. Give your business presence on the web by setting up a shop or…. simply take your garage sale online

These are all high-traffic content-rich portals that are designed to form and engage communities. Due to single sign-on integration the user can now access all solutions while maintaining a single profile.

Feature enhancement and Change management

Aggressive roadmap.

Rapid development makes change management effective.

PakWheels has been developed using agile methodologies. This makes it easier for us to incorporate new features or modify existing features at any time we want.

Community features

PakWheels envisions an aggressive and ambitious roadmap. We bank on our scalable and reliable solution to grow. Being an off-the-shelf solution, vBulletin has made the forums extremely extensible. With traffic consistently on the up, we can already see users adding value to the whole automobile community.