Enterprise Service Bus: a Performance Evaluation

The flexibility offered by an Enterprise Service Bus (ESB) in enabling various applications to exchange data makes it a very important middleware layer that is responsible for transporting data in a Service-Oriented Architecture (SOA). The popularity of the ESB has given rise to a number of commercial off the shelf (COTS) products as well as open source ESBs. In this study, we evaluated three open source ESBs and compared them both qualitatively and quantitatively. The empirical results were statistically tested to determine the statistical significance of the results.


Introduction
The complexity of application integration for a point to point model (Figure 1) rises substantially with every new application that needs to communicate and share data with it.Every new application needs to have custom code written to 'glue' it to the existing network, and thus, increasing maintenance costs.
This inefficient model gave rise to a new 'spoke and wheel' paradigm (Figure 2) called the Enterprise Application Integration (EAI).In this architecture, all communication is facilitated by the message broker.The message broker was designed not just for routing, but often used for data transformation as well.However, this architecture has scalability issues and introduces a single point of failure in the network.The Enterprise Service Bus (Figure 3) is an improvement over these two architectures and plays a critical role in connecting heterogeneous applications and services in a Service-Oriented Architecture [1,2].This middleware layer is responsible for not only transporting data, but also serves as a 'transformation' layer.This 'transformation' of data allows legacy systems to communicate and share data with newer applications.The ESB takes on the responsibility and the burden of ensuring the data sent by the service consumers match the format requirements of the service providers.This core functionality of an ESB is a very important feature for any organization developing applications with an eye towards scalability.
Routing of consumer requests and messages is another important role of the ESB.This functionality of the ESB helps simplify the integration efforts of disparate appli- that need to communicate with each other.The routing decisions made by the ESB can be based upon a number of factors, such as message content, message header, and transport type.Thus, the ESB takes on the role of transporting the data, transforming it, and routing it to the appropriate service providers [1,3].The ESB simplifies the task of both service consumers and service providers by adding a layer of abstraction that shields the consumers and the providers from having to worry about the specifics of message format or message transport.
Virtualization, or proxying, is another role that an ESB can play.In this role, the ESB acts as a proxy for the service provider and handles the requests of the service consumer.The ESB can handle authentication, authorization, and auditing, so the service provider can focus solely on the business logic.
Contrary to common belief, an ESB is not based solely on Web Services.Based on the Enterprise Application Integration (EAI) pattern, an ESB is a flexible and standards based architecture that supports a wide array of transport mediums.Thus, an ESB is a standards-based integration platform that combines: 1) Messaging 2) Web services 3) Data transformation 4) Intelligent routing Each ESB selected for study for this paper offers a wide array of features and enhancements like support for 'Java Business Integration' and 'Business Process Execution Language'.However, documenting and comprehensively evaluating every feature of the ESB was beyond the scope of this study.The intention was to evaluate the core features of the ESB, according to the metrics listed in Chapter 5.The open source ESBs selected for study and evaluation were Mule, WSO2 ESB, and Ser-viceMix.

ESB Core Functionality
Virtualization, Content Based Routing and Mediation are often referred to as the three core features of an ESB.For our study, we captured metrics to evaluate the ESBs on each of these core features.To test load-handling and scalability for each scenario, we ran two sets of tests.They included: 1) Multiple clients sending a fixed payload of 100 KB.The number of clients tested was 1, 20, 40, 80, and 160.

Virtualization
Virtualization, or proxying, is one of the core capabilities of an ESB.In this test scenario, the ESB received an incoming request and forwarded it the real WebService, as shown in Figure 4.There was no other processing done by the ESB in this scenario.
The ESB received the request from the client and forwarded it to the WebService.The web service received the payload, appended a string to the payload, and sent it back to the ESB.The ESB, in turn, returned this payload to the client.

Content Based Routing
The ESB has the capability to route the incoming requests on a single endpoint to the appropriate service.The ESB can look at a wide array of things like the message content or the message header to determine where the request should be routed.
In this scenario, the ESB received the request from the client and inspected the payload for a keyword to determine the WebService to which the request should be sent.The WebService received the payload, appended a string to the payload, and sent it back to the ESB.The ESB, in turn, returned the payload to the client as shown in Figure 5.

Mediation
Mediation, or message transformation, is another core feature of an ESB.The ESB has the capability to take an incoming request and transform the message payload before sending it to the end WebService [4,5].
In this scenario, the ESB got a request from the client and transformed the message payload using XSLT.It then forwarded the message to the WebService as shown in Figure 6.

Evaluation Metrics
Different factors were considered when comparing the   open source ESBs in this project.The metrics are used to determine performance and efficiency [6,7].This section explains these metrics.

Mean Response Time
We calculated the Mean Response Time as the amount of time elapsed from the moment the request was sent to the time a reply was received.

Throughput
We calculated Throughput, as measured in transactions per second.A transaction was counted as successful, if it matched the expected response for the given request.

Statistical Analysis Methods
After retrieving the test data to compare the perfor mances, we need a method to analyze the results.Simply calculating the throughput or the mean response times and generating graphs is not sufficient for the analysis.

Student's Paired T-Test
For our tests, we used Student's Paired T-Test to validate our null hypothesis that all three ESBs would have a similar performance.Thus, the mean difference between any two ESBs compared over a set of matched pairs of data points would be zero [8].
The P-Value threshold chosen for statistical signifycance for our tests was 0.05.Thus, if the calculated P-Value was below 0.05, the null hypothesis was rejected.
For each test, a P-Value was calculated for the metric collected: transactions per second, mean response time, ESB CPU usage, and the CPU usage of the WebServices machine.The P-Value was calculated when comparing the performance was: 1) Mule [9][10][11] vs. WSO2 2) Mule vs. ServiceMix [12][13][14] 3) WSO2 [15] vs. ServiceMix This P-Value helped us analyze the test results by allowing us to focus on P-Values below the set threshold, and thus, of significance.This also helped us prove whether or not the performance of an ESB was equal to its peer, for a particular testing scenario.

Results and Analysis
In order to obtain the best results, each test was run a minimum of 10 times each.Testing was done using Grinder, an open source stress testing tool.
To test scalability of the 3 core scenarios for each ESB, we configured the Grinder script file to simulate 1, 20, 40, 80 and 160 Clients.For this test, we had a fixed payload of 100 KB.
To test load-handling of the 3 core scenarios for each ESB, we configured the Grinder script file to simulate a single client sending varying payloads of 1 KB, 50 KB, 100 KB, 500 KB and 1MB.

Direct Proxy Scenario
The ESBs were configured to act as a proxy and forward all incoming requests to the end WebService.There was no other processing done by the ESB in this scenario.

Scalability Test
Each ESB had the best throughput when the number of clients was 40, as shown in Figure 7.For 40 clients, ServiceMix had the best throughput of 15.78TPS; while Mule had a throughput of 14.69TPS.
Mean Response times of all three ESBs increased dramatically when the number of clients exceeded 80, as shown in Figure 8.

Load Handling Test
The throughputs for all three ESBs (Figure 9) were similar, with a higher throughput when the payload was in the range of 10 KB to 100 KB.There was a drop in throughput when the payload was 1 MB.The throughput for WSO2 and ServiceMix was higher than that of Mule, regardless of the payload.
The mean response time (Figure 10) was similar for all three ESBs for payloads up to 100 KB.The response time increased as the payloads increased.There was a rise in response times for all three ESBs once the payloads exceeded 100 KB.ServiceMix had a better response time than WSO2 for payloads ranging from 10 K to 1 M. Mule had the highest response time when the payload exceeded 100 KB.

Content-Based Routing Proxy Scenario
The ESBs were configured to act as a proxy and forward all incoming client requests to the appropriate end web service.The ESBs looked for a keyword in the message payload that determined the appropriate web service for the given client request.

Scalability Test
The throughput for all three ESBs (Figure 11) was similar in this test.The highest throughput achieved was by ServiceMix at 40 clients.Mule had the lowest throughput at 160 clients.The trend of the mean response time (Figure 12) was similar for all three ESBs.The response time increased as the number of clients increased.There was an increase in response times for all three ESBs once the number of clients exceeded 80.

Load Handling Test
The throughputs for all three ESBs (Figure 13) were similar with a higher throughput when the payload was in the range of 1 KB to 100 KB.There was a drop in throughput when the payload was 1 MB.ServiceMix had a better throughput than Mule, regardless of the payload size.
The mean response time (Figure 14) was similar for all three ESBs for payloads up to 100 KB.The response time increased as the payloads increased.There was an increase in response times for all three ESBs once the number of payloads exceeded 100 KB.

Transformation Routing Proxy Scenario
The ESBs were configured to act as a proxy and forward all incoming client requests to the appropriate end Web-Service.Before the request is forwarded to the WebService, the ESBs transform the message payload using XSLT.XSLT is a very powerful tool that can be used to transform the layout and the content of the message payload to suit the requirements of the end application.

Scalability Test
The throughput for all three ESBs (Figure 15) was similar in this test.The highest throughput achieved was by ServiceMix at 40 clients.WSO2 had the lowest throughput at 160 clients.
The trend of the mean response time (Figure 16) was similar for all three ESBs.The response time increased as the number of clients increased.There was a dramatic rise in response times for all three once the number of clients exceeded 80.

Load Handling Test
The throughputs for all three ESBs (Figure 17) were similar with a higher throughput when the payload was in the range of 10 KB to 100 KB.There was a drop in throughput when the payload was 1 MB.The throughput for WSO2 and ServiceMix was similar for payloads of 25 KB and higher.The mean response time (Figure 18) was similar for all three ESBs for payloads up to 100 KB.The response time increased as the payloads increased.There was a rise in response times for all three ESBs once the payload exceeded 100 KB.ServiceMix had a better response time than WSO2 for payloads between 10 K and 1 M. Mule had the highest response time when the payload exceeded 100 KB.

Subjective Observations
We established a set of criteria for our subjective assessment of the ESBs.We created a three point scale, with 'A' being the best and 'C' being the worst.The ESBs were assigned a score based on this scale for each criterion and the results were recorded, as illustrated in Table 1.

Conclusions
Although the graphs give us visual representation how the ESB performs for a given metric, a statistical analysis is needed to give meaning to the data collected.As stated earlier, we ran 'Student's T-Test' on each metric collected and looked for P-Values that were below the average of the mean response times for WSO2 was 821.12 ms, whereas the computed average response times for ServiceMix was 730.20 ms.Thus, looking at the computed average of the mean response times for the statistically significant data, we can conclude ServiceMix handled scalability better than WSO2.