ABSTRACT

To ensure the quality of Web applications, Web testing is one of the effective methods. The testing is a process of revealing errors that is used to give confidence that the implementation of a Web application meets its original specification. This work proposes a Web testing framework based on Stream X-Machines (SXMs), which provides a way to derive test cases for a Web application. It starts from constructing the SXM model, from which a test translator is employed to extract the test paths and then translates them into an XML-style test specification, which is the input of test engine. The test engine generates test cases and then executes them, and finally produces test report. This testing method is a significant contribution to informed research.

1. Introduction

A wide range of important activities are supported by Web applications. Given the importance of such activities, bad Web applications can have far-ranging consequences on businesses, economies, scientific progress, health, and so on. Web testing is an effective technique to ensure the quality of Web applications. Traditional testing approaches are no longer adequate for Web applications. Web applications typically undergo maintenance at a faster rate than other software systems and this maintenance often consists of small incremental changes [1]. To accommodate such changes, Web testing approaches must be automatable and test sets must be adaptable. However, Web applications raise important and challenging test issues that cannot be solved directly by existing test techniques for conventional programs [2,3].

The problem of test effectiveness is best addressed if the test set can be guaranteed to find all faults of the implementation. One approach is to consider two algebraic objects (the specification and the implementation), each of them characterized by an input/output behavior, and to prove that, if the behaviors of these objects coincide for any input in the test set, they will coincide for any input in the domain. Thus, the specification and the implementation will be guaranteed to have identical behavior provided that they behave identically when supplied with the inputs in the test set. A considerable amount of work by this approach has been employed in the area of test generation for software modelled by Finite State Machines (FSMs). Here, the assumption is that the control aspects of the software can be somehow separated from the system data and can be modelled as an FSM. However, for non-trivial systems, it is usually impossible to describe the system control independent of its data processing. Furthermore, it may be very difficult, or even impossible, to derive the control structure from a system specification, unless appropriate specification languages are used. For example, an FSM can be derived from a VDM specification only under certain conditions and, even in this case, the resulting machine may be of an unmanageable size. Therefore, a more complex specification model that integrates these two aspects is needed. Such a model is the X-machine, a blend of FSMs, data structures and processing functions.

Essencially, an X-machine is like an FSM, but with one important difference: the labels of the transitions are (partial) functions instead of abstract symbols. Of particular practical importance are those X-machines, called Stream X-Machines (SXMs), where these functions process input and output symbols while changing the value of an internal memory or data set M. M is often an array consisting of fields such as registers, stacks, database filestores, etc., so it is possible to model very general systems in a transparent way. SXMs are, however, best suited for specifying interactive systems such as Web applicatiions.

We discuss, in this paper, the testing problems for Web applications based on SXMs. The SXM-based testing method does not rely on the finiteness of the memory set, for there is no need to construct the equivalent FSM and furthermore, it avoids the state explosion problem.

2. Stream X-Machines

SXMs are special instances of the X-machines introduced in 1974 by S. Eilenberg [4]. They employ a diagrammatic method of modelling control flow by extending the expressive power of FSMs. Compared to an FSM, instead of using abstract symbols, the labels of the transitions in a SXM are relations (often partial functions) that operate on a basic data set X. The set of these relations, Φ, is called the type of the SXM and represents the elementary operations that the SXM is capable of performing. In SXMs, all data are triples consisting of a stream of input symbols, a stream of output symbols and an internal memory value.

The basic idea is that the SXM has some internal memory, M, and the stream of inputs determine, depending on the current state of control and the current state of the memory, the next control state, the next memory state and the output value. SXMs [5,6] are a computational model capable of representing both the data and the control of a system. The computation of the SXM starts in a given initial state (control state) and a given state of the system’s underlying data set X (the data state).

In Figure 1, for example, there are a number of paths that can be traced out from the initial state q₁ and each edge is labelled by a relation: φ₁, φ₂, φ₃, etc. Sequences of relations are thus derived from each path in the state space and these may be composed to produce a relation that may be defined on the data state. This is then applied to the value x () providing that the composed relation is defined on x. This then gives a new value, x (if there is more than one such x, then one will be picked in a nondeterministic way) for the data state and a new control state.

Definition 2.1 A Stream X-Machine (SXM) is a nonuple, Where:

• and are finite sets called the input alphabet and output alphabet respectively.

• Q is the finite set of states.

• M is a (possibly) infinite set called memory.

• Φ is the type, a finite set of distinct non-empty processing relations of the form; Φ is often a set of (partial) functions.

• F is the (partial) next state function (or state transition function), F: Q × Φ → 2^Q; as for finite automata, F is usually described by a State Transition Diagram (STD).

• I and T are the sets of initial and terminal states respectively,.

• m₀ is the initial memory value,.

Thus, SXMs are X-machines, of which each processing relation will read an input symbol, then discard it and produce one or more output symbols while (possibly) changing the value of the memory.

Definition 2.2 Given a SXM Z = (, , Q, M, Φ, F, I, T, m₀), the finite automaton FA_Z = (Φ, Q, F, I, T) over the alphabet Φ is called the associated FA of Z.

It is sometimes helpful to regard an X-machine as a finite automaton with the arcs labelled by relations from the type Φ.

Definition 2.3 A SXM Z is called completely defined if, , , such that (m, σ) dom (φ) and (q, φ) dom (F) hold.

A completely defined SXM is one, in which there is at least one possible transition for any triplet, ,. A SXM that is not complete is assumed to ignore an input, which does not cause any transition, by remaining in the same state with an unchanged memory value.

In contrast to FSMs, SXMs are capable of modelling non-trivial data structures by employing a memory attached to the state machine. Moreover, transitions between states are not labelled with simple input symbols but with processing functions. Processing functions receive input symbols and read memory values, and produce output symbols while modifying memory values. The benefit of adding the memory construct is that the state explosion is avoided and the number of states is reduced to those states which are considered critical for the correct modelling of the system’s abstract control structure. A divide-and-conquer approach to designing allows the model to hide some of the complexity in the transition functions, which are later exposed as simpler SXMs at the next level.

Apart from being formal and proven to possess the computational power of Turing machines [5], SXMs offer a highly effective testing method for verifying the conformance of a system’s implementation against a specification. SXM models can be represented in XMDL (X-Machine Definition Language), a special-purpose markup language introduced by E. Kapeti and P. Kefalas [7]. XMDL has served as a common language for the development of numerous tools supporting SXMs [8].

3. Testing Generation Based-On Stream X-Machines

A SXM-based testing method [5,9] that extends the Wmethod [10], enables us to derive a complete finite set of test cases that is proven to find all faults in the implementation. The outcome of the test generation algorithm is a finite set of input and expected output sequences. The inputs and expected outputs need to be mapped to concrete executable test cases that can be processed by a testing engine, in order to interact with the Web component under test and provide the results. The main point about SXM testing, the type of test generation technique discussed here, however, is to establish, not only efficient test sets from a specification but also to relate the test process to a hierarchical decomposition of the implementation in order to make the management of the test process more convenient.

In this work, it is assumed that a tester applies a test sequence as a whole and subsequently checks an output sequence produced by an implementation, rather than applying an input sequence symbol-by-symbol and observing output symbols as they come out.

Definition 3.1 In a SXM Z, if q, , , and, we say that φ is an arc from q to q’ of Z and denote as.

In a SXM Z, for, if any other state is reachable directly from q, then q is called a to-all state; if any other state can not be reached from q, then q is called a to-none state; if q is reachable directly from any other state, then q is called an all-to state; if q can not be reached from any other state, then q is called a none-to state; if q is a to-none state, also a none-to state, then q is called an isolative (or starved) state. An isolative state is not allowed in a Web testing model, for users of a Web application can never stay in an isolative state.

Definition 3.2 In a SXM Z, if q, are such that with q₁ = q and so that φ₁: q₁ → q₂, φ₂: q₂ → q₃, , φ_n: q_n → q_n+1, we say that we have a (test) path p = φ₁φ_n from q to of Z and denote as p:. Each path p = φ₁φ_n gives rise to a (partial) function (or the path function) [p]: defined by

[p] if, , , with m₁ = m, , and so that ,. The functioon corresponding to the empty path is defined by [].

The sequence of transitions (path) caused by the stream of input symbols is called a computation. The computation halts when all input symbols are consumed. The result of a computation is the sequence of outputs produced by this path.

Example 1. In Figure 1, there are four test paths from q₁ to q₆. These test paths are φ₁φ₂φ₇φ₁₀, φ₁φ₃φ₈φ₁₀, φ₁φ₅φ₉ and φ₁φ₅φ₆φ₈φ₁₀. In each of these test paths, it is usual to generate more than one test cases from q₁ to q₆.

Definition 3.3 Two deterministic SXMs (DSXMs [4]) Z and Z' are called weak testing compatible if they have identical input alphabets, output alphabets, memory sets and initial memory values.

Definition 3.4 Two weak testing compatible DSXMs are called testing compatible if they have identical types.

The basic idea of the method is to translate test sets of the associated FA into test sets of the DSXM specification. SXMs are a subclass of X-machines [4] that extend FSMs and are supported by a test generation method that is guaranteed to reveal all faults in an implementation under test, given that certain realistic conditions hold. This works if the DSXM specification satisfies two conditions: input-completeness (i.e., all processing functions can be exercised from any memory value using appropriate inputs) and output-distinguishability (i.e., any two different processing functions will produce different outputs if applied on the same memory/input pair).

Definition 3.5 Φ is called input-complete if, , such that.

This condition ensures that any processing function can be exercised from any memory value using appropriate input symbols.

Definition 3.6 A SXM is output-distinguishable if, , if and then.

The output-distinguishability condition can also be descirbed in the following.

Definition 3.7 Φ is called output-distinguishable if, (,with).

This says that we must be able to distinguish between any two different processing functions by examining outputs. If we cannot then we will not always be able to tell them apart.

These two conditions (output-distinguishability and input-completeness) are generally known as design for test conditions [9]. The output-distinguishability condition ensures that any processing function can be identified from the machine computation by examining the outputs produced. The input-completeness condition ensures that all sequences of processing functions in the associated FA can be exercised using appropriate inputs, so they can be tested against the implementation. Without the described conditions, it would be extremely difficult to test a Web application properly. For instance, if a method generates a particular sequence of functions to execute and there is no corresponding sequence of inputs to attempt this sequence, such a sequence cannot be executed. As a result, some faults of the implementation may go undetected.

There are often many test cases in testing a Web application, for representing a test set in a reduced way, we introduce the following several notations:

• , which says that either tc₁ or tc₂ is selected to execute each time.

• , which says that both tc₁ and tc₂ are executed in any order.

• , which says that both tc₁ and tc₂ are executed and tc₁ is executed before tc₂.

For example, a test set may be written as, which is the reduced representation of these four test sets, , and.

The above-mentioned testing method makes it possible to find a test input for every memory value. Instead, it could be possible only to use a subset of inputs and memory values for testing which could make attempting elements of Φ easier in practice. For a relation φ, inputs used for testing are denoted Uφ; memory values traversed during testing are assumed to be contained in. Usage of a subset of inputs is only possible if

• Memory of an X-machine under test will stay within V.

• It is possible to attempt every relation using a subset of inputs.

• Outputs from these relations make it possible to distinguish between them.

The above three items are formalized using the idea of a testing context (V,U) [11]. The purpose of Uφ is to contain inputs to attempt φ ().

Definition 3.8 Let (V, U) be such that and. The set Φ is called closed w.r.t (V, U) if the following two conditions are satisfied• .

• , , , if there is, such that, then.

Definition 3.9 Let Φ be closed w.r.t. some (V, U) where and. Then Φ is input-complete w.r.t (V, U) if, , such that.

Definition 3.10 Let Φ be closed w.r.t some (V, U) where and. Then Φ is output-distinguishable w.r.t (V, U) if, , , ,. if and, then φ₁ = φ₂.

For V = M and a DSXM, the above two definitions reduce to input-completeness and output-distinguishability; the set Φ of any SXM is closed w.r.t (M,) by definition of a SXM. For these reasons, Definitions 3.9 and 3.10 are more general than Definitions 3.5 and 3.6 (or 3.7).

Definition 3.11 Let be a non-empty subset of memory values and n ≥ 1. The SXM is called n-complete w.r.t. V if there exist, V₀ = V, such that for any

• if then there exists such that.

• if with and then, 1 ≤ i ≤ n − 1.

The above condition guarantees that any path of length at most n of the associated automaton can be exercised by an input test sequence from the initial state and an initial memory value in V.

4. A Feasible SXM-Based Web Testing Framework

It is widely accepted that models constitute important mechanism in the process of a Web application development and help us understand the system by omitting some details.

To test a Web application, test cases must be generated. Our test paths produced can be easily employed to construct test cases. A test case defined by our Web testing framework (cf. Figure 2) is one test path with user input values. So, a test path may be used to construct multiple test cases if only the tester provides different user input values. Our test specification is an extended version of the one described in [12], where the specification is depicted using XML. The test specification is based on request specification, response specification, and predicate definition. Request specification specifies a pattern of HTTP requests, while response specification specifies the assertions on the HTTP response generated from the HTTP request in the same test step, and finally, the predicate definition specifies the assertions on the results of testing.

The Web testing framework extends the testing approach proposed by X. Jia, et al. [12]. At the beginning, the SXM is constructed through the commonly-used requiring and analyzing methods of Web applications. The test translator of the testing framework extracts component relations from the SXM. It then translates the test paths into test script frame (of test specification). This frame defines the test set, containing test cases. Every test case is a sequence of test steps. Each test step is for one component to be verified. Every test step defines HTTP requests, expected response and predicates with condition definitions (such as , , , , and ). The test script frame does not contain values for input variables, since they will be added later. Some inputs have to be created by hand, including user ids and passwords. Other inputs are either automatically extracted

from the HTML files or randomly generated. Moreover, the test script frame does not contain hard-coded expected output, that is to say, the Web testing framework defines frames with empty response, since they will be filled manually by the tester.

After editing the necessary information of the test script frame, the tester gets a formal XML-style specification, which is the input to test engine. The specification contains templates of test cases. Test engine includes a test case generation engine (test generator) that is able to determine the test paths from the test specification, and to generate test cases from it, provided that a test criterion is specified. Generated test cases are sequences of links which, once executed, grant the coverage of the selected criterion. Then the test engine executes the test cases and validates its result against the test oracles specified as expected result. That is to say, test engine’s test executor can now provide the link sequences of each test case to the Web server, attaching proper inputs to each form. After the execution, test engine produces a test report summarizing the results of all test cases. For such evaluation, the tester opens the output pages on a Web browser and checks whether the output is correct for each given input. By the way, the test engine behaves as a Web client accessing the Web application.

5. Related Work

Many Web testing challenges are discussed in [2,13], and a number of Web testing techniques for Web applications have been already proposed [14-21], each of which has different origins and pursues different test goals for dealing with the unique characteristics of Web applications.

Andrews, et al. [15] illustrated an approach to modeling and testing Web applications based on FSMs after analyzing eight kinds of connections among Web pages and software components of Web applications. They partitioned a Web application into several functional clusters and logical pages, and tried to use hierarchical constrained FSMs to represent the logical pages and their navigations. However, the interactions and composition of components are not considered further.

Elbaum, et al. [17] proposed a method to use what they called user session data to generate test cases for Web applications. Instead of looking at the data kept in J2EE servlet session, their user session data is the input data collected and remembered from previous user sessions. The user session data is captured from HTML forms and includes name-value pairs. Our approach is flexible, and the user input data can be produced by various methods presented by existing research work.

Ricca and Tonella [18] suggested a UML model of Web applications and proposed that all paths that satisfy selected criteria should be tested. They also presented an analysis model and corresponding testing strategy. Their strategy is mainly based on static Web page analysis and some preliminary dynamic analysis. Liu, et al. [19] extended traditional data flow testing techniques to support Web application testing. A test model, WATM, which consists of an object model and a structure model, is presented to capture the data flow information of Web applications. These studies [18,19] consider only the underlying structure and semantics of Web applications towards a white-box testing approach. They focus on the internal structural aspect and involve in the details of a Web application.

Object driven performance testing was proposed by Subraya and Subrahmanya [20]. They illustrated a new testing process that employs the concept of decomposing the behavior of a Web application into testable components.

A survey of Web application testing was given by Lucca and Fasolino [3]. They presented the main differrences between Web applications and traditional ones, how these differences impact the testing of the former, and some relevant contributions in the field of Web application testing developed in recent years.

SXM-based testing has been developed in various directions. Moreover, numerous case studies have been carried out to establish if the approach is actually practical. Thus, a number of systems have been specified as SXMs and the tests produced by the approach used to validate the implementations.

Kourtesis, et al. [22] presented an approach towards effective service discovery and selection. They employed SXMs as a powerful modelling formalism for constructing the behavioural specification of a Web service, for performing verification through the generation of exhaustive test cases, and for performing validation through animation or model checking during service selection.

Ipate and Holcombe [23] provided a new variant of the SXM-based testing method that no longer depends on the size of a controllable model of the IUT. In data processingoriented applications, the new method can drastically reduce the size of the test suite produced at the expense of a (possibly) more complex generation process.

Merayo, et al. [24] presented a formal testing framework for systems where timeouts are critical. The model introduced for specifying the systems is a suitable extension of the classical concept of SXM. They introduced a notion of test that can delay the execution of the implementation and also proposed an algorithm to derive sound and complete test sets.

Bogdanov, et al. [25] described the X-machine testing method and its use for testing of different types of systems, both in terms of theory and practical outcomes. They surveyed the extensions of the X-machine testing method for testing of functions together with testing of a transition diagram, equivalence testing of a non-deterministic implementation against a non-deterministic specification, conformance testing of a deterministic implementation against a non-deterministic specification and equivalence testing of a system of concurrently executing and communicating X-machines, against a specification.

Ipate and Gheorghe [26] presented the complete nondeterministic SXM (NSXM) testing method to generalize the NSXM integration testing method. It no longer requires implementations of the processing relations to be proved correct before integration testing can take place. Instead, the testing of processing relations is performed along with the integration testing. The authors also showed how a SXM model of a P system can be obtained and how the NSXM testing approach can be applied to generate conformance test sets for the P system.

6. Conclusions and Future Work

Testing aims at finding errors in the tested object and giving confidence in its correct behavior by executing the tested object with selected input values. At present, there are no systematic method and tool that are employed to test Web applications efficiently. The improved traditional methods or a new method appropriate for Web application testing are desired urgently for all the characteristics of Web applications. Since the current testing methods depend primarily on the testers’ intuition and experience, the testing of Web applications is regarded as a time-consuming and expensive process. Therefore, a new methodology for Web testing is required imminently to automate the testing.

The features of Web applications make traditional coverage-based [27] or partition-based testing inappropriate (or inadequate) for Web applications. The proposed SXM model can capture information about control flow, data flow, transaction processing and associated usage as well as criticality information. Test cases can be produced by following the states and state transitions in SXM to select individual operations (states) and link them (transitions) together to form overall end-to-end operations.

There is the famous 80 - 20 rule declaring that typical software spends 80 percent of the time executing 20 percent of the code. That means different portions of software are executed with a higher frequency than others. Statistical usage testing, often originally referred to the work by Adams [28], aims to identify these portions and adjusts test suites subjecting more frequently executed parts to more thorough testing. It is often important, however, to ensure the exercise of specific operations of the software irrespective of their usage probabilities. Two examples are operations of high criticality due to potential impacts of a failure and those implemented by new software.

One of the challenging steps planed for further research is to combine SXM-based testing with statistical usage testing for Web application testing. Another critical problem of test generation is adequacy criteria. This problem should also be considered In Web testing. However, this work doesn’t cover it. It may be discussed in testing generation approach in the future research.

7. Acknowledgements

This work is financially supported by the National Natural Science Foundation of China (NSFC) under grant No. 61163007, the Jiangxi Provincial Natural Science Foundation of China under Grant No. 2010GQS0048, the Nuclear High Base Special Funds under Grant No. 2010 ZX01045-001-001 and the School Foundation of Jiangxi University of Finance & Economics of China under Grant No. 05072015.

REFERENCES