ABSTRACT

In this paper we look at different push messaging alternatives available for Android. Push messaging provides an important aspect of server to device communication, and we specifically focus on the integration of cloud computing with mobile devices through the use of push-based technologies. By conducting a benchmarking test, we investigate the performance of four relevant push technologies for the Android platform, namely C2DM, XMPP, Xtify and Urban Airship. The comparison focuses on three aspects of the libraries: 1) The stability; 2) Response times; and 3) Energy consumption. The test is conducted on both WLAN and 3G, and includes several mobile device types. Additionally, we also integrate with the Google App Engine to provide the cloud integration server that is responsible for sending push messages to the mobile devices.

1. Introduction

The goal of cloud computing is to provide the appearance of unlimited scalability and storage for less money than the in-house data centers [1]. Its success is based on an economy of scale and the relative ease of administration of services, as an entire cloud-based data center could be configured through, for example, a series of web pages. Many businesses are today looking at cloud computing as a viable and cost-effective alternative to hosting their own data centers internally, with large IT companies like Microsoft, Google, and IBM all having initiatives relating to cloud computing [2]. Popular cloud-based platforms include Microsoft Azure, Amazon EC2 and the Google App Engine.

In 2011 Gartner reported that cloud computing, and mobile applications and media tablets are on the top 10-list of strategic technologies [3]. In this paper we will investigate how these technologies can cooperate through push messaging, where a content provider publishes information to a subscriber. We have focused on combining cloud computing with mobile applications through the use of push technology on the Android platform.

Before the platform support for push messaging was added to Android it was common to use a polling mechanism. This worked by making the application constantly poll the server for updates. There are several drawbacks with this alternative, especially the challenge of configuring the frequency of poll-requests sent. Another possibility is to push messages using SMS (Short Message Service). Android is able to receive and intercept SMS messages, but they come with their own limitations like availability, cost and message size.

In this paper we present our experience from working closely with push messaging technologies. Specifically, we compare different technologies available for the Android platform, from the standard library provided by Google to commercial options. In total we will look at four alternatives that all provide similar push messaging features. We believe this gives an in-depth look at the state of the art in integration between cloud computing and the Android platform not found in existing research. On the server side we have used the Google App Engine as the cloud-based platform.

We focused on push messaging on Android because it is an important aspect for application developers. Push messaging is in many situations a vital aspect of the usability and functionality of an application. Additionally, as stated by Gartner [4], the popularity of the Android platform means that this area of research highlights challenges that affect a considerable amount of software developers.

Accordingly, the main contribution of our research is to investigate the following topic: Compare four pushmessaging technologies for Android, which are integrated with a cloud-computing environment, in regards to the stability of responses, response times and energy consumption.

The paper is organised as follows: We begin with a look at related work before presenting a short introduction of the investigated technologies. A description and review of the benchmarking test is shown, and finally the conclusion is presented towards the end.

2. Related Work

Cloud computing is becoming increasingly popular. Features like elasticity, scalability and a new cost-model are providing new and interesting opportunities for many companies. It has proven particularly useful for small and medium enterprises that have a large variation in their computing needs [5]. However, not all businesses will benefit from moving their data centres to the cloud, e.g. when there are government regulations not allowing sensitive data to be stored with an external cloud provider [6].

In an attempt to help decision makers identify their concerns with moving all or parts of their computing needs to the cloud, Khajeh-Hosseini et al. [7] has created a Cloud Adoption Toolkit. With this toolkit, they argue that one can identify the potential benefit or drawback from moving the IT infrastructure and applications to a cloud provider.

When selecting a cloud platform, there are three main service models to select from [8]:

1) Software as a Service (SaaS), the consumer uses the cloud provider’s applications running on a cloud infrastructure;

2) Platform as a Service (PaaS), the consumer is able to deploy (either customer created or acquired) applications onto the cloud infrastructure. The consumer does not manage or control the cloud platform/infrastructure;

3) Infrastructure as a Service (IaaS), the consumer can provision processing, storage, networks, and other computing resources that can be utilised to deploy and run the applications.

In our work we focus on the Google App Engine, which is a PaaS service model making it possible for developers to run their own applications on Google’s infrastructure [9]. We used the Google App Engine because it provides good support for the technologies we wanted to test, such as a close integration with C2DM and XMPP, and also a simple and easy administration feature.

The Google App Engine is a cloud-based PaaS service, the platform is pre-configured by Google and provides a much higher abstraction than an IaaS (Infrastructure as a Service) platform like Amazon EC2, where almost the entire stack from kernel and upwards can be controlled [10]. The Google App Engine is also well integrated with other Google services like e-mail and authentication. The platform imposes certain limitations on the developers, for example that threads cannot outlive the request that creates it and a limit of 10 concurrent request threads [11]. However, by enforcing these Google is able to provide very high scalability [9].

In work more closely related to ours, Minstrel [12] has been developed to provide a push-based messaging system. The system utilises the publish/subscribe paradigm and has been extended to support mobile devices. Minstrel and the standard push messaging library for Android, which is called Cloud to Device Messaging (C2DM), have several similarities in how they are built, including the publish/subscribe model, where subscribers must register at the content publisher to receive the messages.

In similar research, the Bakabs application, created for Android and iOS, Paniagua et al. [13] use C2DM and the Apple Push Notification Service (APNS) to implement push messaging. The application aims to provide a management tool that allows information to be retrieved about the web applications’ traffic and then launch or stop cloud instances based on the current load. The use of push messaging was included to allow for the cloudbased services to send messages asynchronously back to the handset, thus eliminating the need for the client to wait for a response.

One of the topics investigated, as part of the benchmarking test we conducted, is energy consumption. Research in this area includes the work done by Flinn and Satyanarayanan [14]. They specifically look at energyaware systems, where they show that a solution for dynamic balancing of energy consumption and application quality is an essential part of comprehensive energy management solutions. Similarly, Rivoire et al. [15] present a research effort in the area of energy efficiency. They propose JouleSort, an external sort benchmark for evaluating the energy efficiency of various devices, from laptops, desktops and servers.

The research described effort by Rivoire et al. [15] is quite different from ours. Firstly, we focus on one specific element, namely push messaging technologies running on an Android device. We chose to focus on the Android platform because it is well integrated with other Google technologies and, more importantly, it is an open platform making it ideal for experimentation. Secondly, we are not trying to identify the performance of disk I/O, CPU capacity and so on, but we compare the stability, response times and energy consumption of the important push messaging technologies. Generally, our test is much more specific, targeting specific mobile operating system and push-messaging technologies, in contrast to the general-purpose benchmark proposed by Rivoire et al. [15].

In the context of performance testing, Calheiros et al. [5] have examined the performance of cloud computing alternatives. They present a simulation toolkit, called CloudSim, making it possible to model and simulate cloud computing systems and applications. They argue that it is impossible to perform benchmarking experiments in a repeatable, dependable and scalable environment using real-world cloud-based platforms. With CloudSim, they have created a tool making it fast and easy to configure and run these tests.

Nonetheless, getting highly reliable and repeatable results from a real-world environment is difficult [5]. This is especially so when using cloud-based resources, where there are many components, hardware and software, working together and it can be difficult to isolate the specific parts of the overall system one wants to test. However, we still believe there is value in doing a benchmarking test of different push messaging technologies integrated with cloud computing. There is value in providing performance results for the different technologies because it gives the developers information on the strengths and weaknesses of the popular push-messaging alternatives on the platform.

Also, it is important to note that we are integrating external components, the mobile devices, into the tests and not just using a cloud-based system, making it even more challenging to use a simulation tool that will provide realistic results.

We have focused our work on combining mobile devices with cloud computing. In this context, Binnig et al. [1] argue that a benchmarking test in a cloud computing environment should address the following issues:

1) Adaptability of the system, the ability to adapt to changing load in terms of scalability and cost;

2) Conduct the benchmarking tests from different locations;

3) Access more dynamic “Web 2.0”-like applications including multimedia content.

These pointers provide valuable insight into what a cloud computing benchmark test should include. We want to use some of the general ideas from this list, but we also want to stress the differences between what we are benchmarking, where a mobile client and cloud integration is involved, with the pure cloud-based benchmarking test that was described by Binnig et al. [1]. Particularly interesting for our test is to include the stability, response times and energy consumption of the system and also results from different networks.

The next section will introduce the different push messaging technologies.

3. Push Messaging on Android

The technologies used in our experiment all deal with the integration of cloud computing and mobile applications. Not all push messaging technologies investigated are directly related to cloud computing, such as XMPP (Extensible Messaging Presence Protocol). However, it is well integrated with the Google App Engine and is therefore included in our test.

In our benchmarking test we considered a total of six alternative technologies that offer push messaging for Android. These libraries are currently the main competitors in the market. SMS was not considered as part of the push messaging libraries, this was because of limitations such as availability and cost. Specifically, we conducted the test on a tablet device, Samsung Galaxy Tab 10.1, which does not support SMS.

The libraries we found particularly interesting are: C2DM, Urban Airship, Xtify, XMPP, MQTT (Message Queue Telemetry Transport) and Deacon. Although we believe these technologies present the most promising and useful push messaging libraries on Android, we cannot completely rule out the possibility of other interesting options we were not able to find.

Of these alternatives, we did not go into detail for two specific libraries, namely MQTT and Deacon. MQTT was not included because we wanted to investigate pushmessaging technologies that can be easily integrated into the cloud, and specifically on the Google App Engine. MQTT is useful for connections that require a small code footprint and where network bandwidth is limited [16]. It does require a message broker hosted on a separate server. We did not find an easy way to integrate this service with the Google App Engine.

The second technology, The Deacon Project [17], is an open source project providing push notifications to Java and Android applications. We felt that this project was the least mature technology of the options we considered, as it is currently in beta release. The project also states that it is created for users wanting to run push notifications on their own server and support Android versions lower than 2.2, whereas C2DM requires at least Android 2.2. None of these requirements matched what we wanted to investigate, which included a close integration with a cloud-based server application and devices running on at least the 2.2 version of Android.

3.1. XMPP

The first push-messaging technology we wanted to include in our benchmarking test was the XMPP protocol. It is created for real-time communication [18] and for streaming XML [19]. The technology behind XMPP was created in 1998 and then refined in the Jabber open source community in 1999 and 2000, before it was formalised by IETF (The Internet Engineering Task Force) in 2002 and 2003 [20]. It is commonly used in Instant Messaging (IM) and has been used by Google Talk, Jabber and other IM networks.

XMPP is offered as a service on the Google App Engine, making it possible to write cloud-based applications on the Google infrastructure that is able to communicate with users or applications. Accordingly, we integrated with XMPP through the Google App Engine infrastructure. Our Android client used an XMPP library called asmack [20], which is a patched version of smack created for Android. Smack offers an XMPP library and is a pure Java implementation [21].

XMPP on the Google App Engine has a daily limit of 1 GB data sent and 100,000 invitations with the free default limit [22]. More resources can be purchased, with paid applications incurring a minimum spend of $2.10 per week.

3.2. Cloud to Device Messaging

Cloud to Device Messaging (C2DM) was made available from Android 2.2, where the goal was to make it easier for mobile applications to sync data with servers [23]. The technology is used in several standard Google applications including Gmail, Contacts and Calendar. When messages are received on the Android client, the system will wake up the application via an Intent broadcast, and pass the message data [13]. The message limit is set to 1024 bytes and developers are encouraged to send short messages, essentially notifying the mobile application that updated information can be retrieved from the server. C2DM is a free service, and the maximum number of messages that can be sent is approximately 200,000 per day [24]; however this can be increased if there is a need for more resources.

Google offers standard libraries for Android that makes it possible to use C2DM directly. Figure 1 shows a basic overview of C2DM. We believe the C2DM library provides a good basis for a standard push messaging technology. However, we identified certain limitations with C2DM that we wanted to simplify and provide more features than the standard solution. This will be presented in more detail in a later section.

There will also be a market for different solutions, offering more comprehensive services and novel features not covered by the C2DM technology, such as a webpage for administration and multi-platform support. These features are targeted by other commercial technologies, such as Urban Airship and Xtify.

3.3. Urban Airship

Urban Airship provides a commercial option for sending push notifications on Android, Blackberry and the iOS platform [25]. It makes it easier for developers to create applications for multiple device types since it provides a single API for all the supported platforms. It consists of a library that is added to the project to hide all the low level complexity related to push messaging. In addition to push notifications, Urban Airship provides features such as rich push, push composer, reports, in-app purchase and subscriptions.

Urban Airship also offers a proprietary push-messaging platform called Helium that supports Android 1.6 and newer. With newer phones (minimum Android 2.2) it supports the use of C2DM. The pro priceplan costs $199 per month, and includes support for up to 10,000 users and unlimited push messages, with an additional $0.01 per user over this limit [26].

3.4. Xtify

Similar to Urban Airship, Xtify is a commercial option that provides push messaging for Android. Xitfy also supports the Android, Blackberry and iOS platforms. For Android it uses the C2DM technology offered by Google and adds features like registration management, notification handling, notification inbox, rich notification support and the ability to send messages based on user location.

The Just Push-package from Xtify costs $199 per month, and includes support for 30,000 devices and unlimited notifications. Each additional device over this limit costs $0.01 [27].

Both Urban Airship and Xtify support C2DM, and in addition they have their own proprietary Android push notification service [28]. These proprietary alternatives are used in the benchmarking test to provide a way of comparing C2DM-based applications with other push messaging technologies. For Xtify this is implemented on their infrastructure with the XMPP protocol [28]. It is important to note that even though two technologies in our benchmarking test are based on XMPP, we use XMPP directly when integrating with the Google App Engine. When testing Xtify we use their API and infrastructure. Xtify recommends using XMPP in cases where one for example needs to communicate frequently with the mobile devices over a short period of time. In other cases it recommends using the C2DM alternative it provides.

In our research reported here, we have included all of these technologies and performed a benchmarking test to see how they perform.

4. Benchmark Test

We created a benchmarking test to compare different push-messaging technologies on Android. The system consisted of a mobile client that in sequence invoked all the different push messaging technologies and record the time used. On the server side we have a Google App Engine server application that sends messages when requested to do so from the client. This application is also responsible for storing all the data received from the mobile application.

In our research we wanted to compare the performance of C2DM with other push-based technologies integrated in a cloud environment. We defined three main characteristics that are important for push-based technologies:

• Response times, what are the response times for the different push messaging technologies?

• Stability, are the response times providing stable results over the time we run the test?

• Energy consumption, how efficient, in regards to battery power, is the various push-messaging technologies?

The test was conducted in two main iterations. In iteration 1 we started by looking at the response times and stability for each push-based alternative. Moreover, the test was performed on two network types, namely WIFI and 3G. The message size sent was 450 bytes on all technologies and the tests were run on and off over several days with messages sent every 5 minutes.

For iteration 1 we included the following devices in the benchmarking test: Samsung Galaxy Tab 10.1 (SG), HTC Evo (HE), and HTC Nexus One (HN).

For iteration 2, we wanted to compare the energy consumption of the various push-messaging technologies. This test included the same message size (450 bytes), but we only used one device, which was SG, and we also increased the message frequency to 10 minutes. This device was selected to get a more comprehensive test because the SG device has a significantly larger battery (7000 mAh) when compared to for example the HE (1500 mAh).

This benchmark test lasted about 6 days for C2DM and Urban Airship, while the XMPP test only lasted about 2 days because of certain limitations in the platform, which we will describe in more detail later in this paper.

When doing our pilot-tests we noticed that the screen would consume a considerable amount of battery power and made it difficult to find any differences between the push-messaging technologies. For this reason we turned the screen on the device off when testing the energy consumption. Additionally, we also disabled the autosync feature to prevent applications using network communication resources on the device. These steps were taken to try to eliminate other factors that might impact the energy consumption on the device.

4.1. Test Procedure

Both iterations, as explained above, followed these main steps:

1) The Android client registers with the server. This is done differently for each technology, for example C2DM will send a registration id to the device;

2) A timer is started on the client, followed by a message being sent to the server requesting a new push message;

3) The server application receives the message and immediately sends out a message consisting of 450 bytes to the mobile device. This will happen for each technology type;

4) When the message is received, the Android client stops the timer and registers the result. This result is then sent back to a result-servlet that is part of the server application, which will permanently store the information;

5) Finally, the process waits 5/10 minutes before continuing with the sending the next message.

In our tests we compare the following technologies: 1) XMPP; 2) Urban Airship Helium; 3) Xtify (proprietary push messaging infrastructure) and 4) C2DM. All push messaging alternatives are integrated with a cloud-based Google App Engine server application. It is important to note that we did not include other technologies built on top of C2DM, like Urban Airship or Xtify with C2DM enabled, because they send messages in the same way as standard C2DM.

4.2. Results

Table 1 presents the overall results from the benchmarking test. We start by looking at the numbers for both the SG and HE. Both devices ran on the same WIFI network, and we were also able to provide a fairly equal number of messages for each technology providing a good basis for the comparison.

As can be seen in the table below, the results are relatively consistent, even though the differences between the technologies (see the standard deviation) were biggest on the SG. The average response time for XMPP was the shortest, with C2DM on second and finally Urban Airship. There is a difference of 276.12 ms (SG) and 156.98 (HE) between XMPP, which had the shortest response times, and Urban Airship with the longest response times. As seen in both Figure 2 (results from SG) and 3 (results from HE), the difference is mostly due to spikes in the response from Urban Airship. The max time used for Urban Airship was 5337 ms (SG) and 3601 ms (HE), whereas both XMPP and C2DM provided considerably more stable results in the benchmarking test. These spikes were more frequent on the results gathered from SG than with HE, however, this trend was evident on all the devices included in the test, which is confirmed by looking at the standard deviation.

Overall the C2DM results were stable and the performance results recorded showed an average response time of 466.82 ms (SG) over 281 messages and 401.89 ms (HE) over 174 messages. Comparing the response time for C2DM on SG and HE, there is only a difference of 23.01 ms in the average response times. The HE had fewer messages received, with 174 compared to 281 for the tablet.

XMPP had the most stable results in our test, with a standard deviation of 172.91 (SG) and 67.84 (HE). Urban Airship did appear to have more stability issues than the rest and these issues surfaced several times during the test. When comparing the results from different WIFI networks and 3G, the same pattern emerges. The 3G response times are higher, but this is to be expected since they will have less bandwidth than the WIFI connection.

The final technology we tested was Xtify, and it comes very close to the overall performance of C2DM. It provides more stable results than Urban Airship and with slightly better average response time than C2DM. We were unable to conduct the benchmarking test with Xitfy on other devices than HE, because of limitations in the account we used. Additionally, since we were only able to send a limited number of messages this technology was not included in iteration 2.

The results for SG and HE are presented in Figures 2 and 3 on the next page, and it is easy to the spikes in the response times for Urban Airship as previously mentioned.

For the final test, iteration 2, we wanted to investigate the energy consumption of the different push-messaging technologies. In this part of the benchmark test, we ran the same application as before, but we increased the time

between messages to 10 minutes. Another difference was that instead of running each technology in sequence, we only recorded one technology at a time. This was done because we wanted to provide results based on the battery level for each technology, and also to expand the test over a longer period of time. As previously described, the client ran the test by sending requests to the cloud-based server, but as part of this iteration we also added a feature that triggered a new request from the mobile device for each change in battery level. By doing this, we were able to record the messages and also the corresponding battery level.

In this test we included C2DM, Urban Airship, and XMPP. Xtify was not included due to limitations with the developer account we had created.

With both C2DM and Urban Airship we were able to provide fairly equal number of messages, with 862 and 858 messages sent respectively. However, with XMPP we were unable to send more than 295 messages because of quotas and limits in the Google App Engine [29]. The results are presented in Figure 4, where we have added trend lines for each result to make it easier to see the differences between the technologies.

As can be seen in Figure 4, both C2DM and Urban Airship provided the best results, using less energy than XMPP. We did expect this because it is recommended to use XMPP in scenarios where one needs to communicate frequently over a short period of time, as stated by Xtify

[30]. This aspect is also verified by the normal use of XMPP, for instance Instant Messaging applications such as Google Talk.

The two other technologies, Urban Airship and C2DM, provided fairly similar results. However, from the last 50% and towards 0%, Urban Airship did provide slightly better results. It is difficult to draw any final conclusions of the difference between these technologies because the change is fairly small, and further testing is needed. However, we still find these results interesting, and especially the big difference between C2DM/Urban Airship and XMPP. We would certainly recommend using either C2DM or Urban Airship over XMPP for applications not dependent on frequent messages being pushed from the server to the devices.

Table 2 presents a short summary of the advantages and disadvantages of the libraries we tested. The overall results show that the C2DM, which is the standard Android library offered by Google, provided good performance compared to the alternatives. Moreover, it also performed well in the energy consumption test, especially compared to XMPP. We found that C2DM provided the best overall results in the three categories investigated, namely response time, stability and energy consumption.