In this paper we describe how the test design can be done by using the Combinatorial Testing approach for internet of things operating systems. Contiki operating system is taken as a case study but we discuss what can be the approach for RIOT and Tiny OS operating systems. We discuss how the combinatorial coverage measurement can be gathered in addition to the traditional metrics code coverage. The test design generated by using Advanced Combinatorial Testing for Software is analyzed for Contiki operating system. We elaborate the code coverage gathering technique for Contiki simulator which happens to be in Java. We explain the usage of Combinatorial Coverage Measurement tool. Although we have explained the test design methodology for internet of things operating systems, the approach explained can be followed for other open source software.
Our previously published paper touches upon the test design using combinatorial testing approach for Contiki operating system [
While we want to have minimal overlap of this paper with the one published already [
National Institute for Standards and Technology (NIST) has been actively supported combinatorial testing. The research carried out by NIST has been well documented [
Combinatorial testing is not only adopted by researchers, and evidence shows the adoption by large organizations [
As mentioned earlier, this paper focuses on classic approaches of combinatorial testing and the same is being elaborated in the following sections.
If the gathered data indicates inadequate test suite, redesign the test suite using the combinatorial approach and gather the data. If the coverage data is less, it calls for re-visiting the test design. Base line the test suite once the adequate test criterion is met.
As can be seen from the diagram it can be iterative process. We document the process that was used for case study operating system in further sections. We describe the approach to be followed when the test suite already exists and when it does not. Section 3 is for the case when the base test suite already exists and Section 4 is for the case when the test suite does not exist.
Then we visited the existing test suite to know the reason for low coverage. Few areas of improvements were observed in the existing test suite.
・ No formal test design document existed.
・ It appears that the test cases were concentrated around few mote types (hardware or configurations in the context of combinatorial testing).
We went through the whole regression test suite to extract the configurations supported and input parameters being used. We came up with
Contiki is open source operating system widely used and accepted for Internet of Things. It has base-lined regression test suite for version 2.7. Contiki gives the user friendly operating system in the form of instant Contiki which has Ubuntu like the feel with the tool chains to make the iterative development easy. The developers can use the instant Contiki to test the patches and testers can use the same environment for ascertaining the reliability of the operating system without procuring the hardware for all the mote types.
Contiki gives the simulator which is called Cooja. The Cooja simulator talks to the Contiki using Java Native Interface (JNI). The test cases are called csc files which are understandable by Cooja. We found Eighty three test cases of this type in the regression folder. However, these test cases were concentrated around few mote types.
Parameters | Parameter Values |
---|---|
Platform | Exp5438, z1, wismote, micaz, sky, jcreate, sentilla-usb, esb, native, cooja |
base | Multithreading, coffee, check pointing |
Rime | collect, rucb, deluge, runicast, trickle, mesh |
Net Performance | Net Perf, Net Perf-lpp, Net Perf-cxmac |
collect | shell-collect, shell-collect-lossy |
ipv4 | telnet-ping, webserver |
ipv6 | ipv6-udp, udp-fragmentation, unicast-fragmentation, ipv6-rpl-collect |
RPL | up-root, root-reboot, large-network, up and down routes, temporary rootloss, random rearrngement, rpl-dao |
ipv6apps | servreg-hack, coap |
As already mentioned, Appendix A gives the test design generated using ACTS for
Now the task at hand is mapping these generated test cases to functional test cases (xml files called csc) which are understandable by Cooja and gathering the coverage data again. The coverage data should improve in principle. We are working on this.
In this section we document the changes that we did in the Contiki environment for the tasks at hand. Since we get implementation specific for case study operating system, this section can be conveniently skipped by the readers who are not interested in specific details for given operating system.
1. Log in as user in the instant Contiki environment.
2. Search for the .travis. yml
3. Add the build type you are interested in:
- BUILD_TYPE = “ipv6-apps”
- BUILD_TYPE = “CT”
-BUILD_TYPE = “compile-8051-ports”
4. Under the directory../contiki-2.7/regression-tests create a folder 02-CT
5. Under contiki-2.7/regression-tests/02-CT directory create *.csc files you are interested in viz. 01-custom.csc 02-custom.csc
6. The Make file should look like include../Makefile. simulation-test
7. Create a 01-custom.csc file in the Cooja tool. Use the test script editor to create a java script which will be essential while running the test case from command line using the makefile.
8. Modify the build.xml suitably as explained in Appendix C.
9. Run the regression test suite as usual.
10. Test run will create many *.clf files.
11. Create a script for analyze, merge and generate report.
In this paper we presented the approaches that could be employed for designing the regression test suite using combinatorial approach. We explained how the bench marking of the regression test suite could be done using the traditional approaches such as code coverage in addition to coverage gathered using combinatorial coverage measurement tools.
In this paper we investigated how the combinatorial approach could be applied for the cases when the regression test suite already existed viz. Contiki case. We were working on ascertaining the gain due to Combinatorial testing.
We were planning to explore the Combinatorial testing for the cases when the regression test suite did not exist viz. RIOT and TinyOS.
Further we were planning to model the System Under Test (SUT) and use New Symbolic Model Verifier (NuSMV) in conjunction with ACTs.
AbhinandanH. Patil,NeenaGoveas,KrishnanRangarajan, (2015) Test Suite Design Methodology Using Combinatorial Approach for Internet of Things Operating Systems. Journal of Software Engineering and Applications,08,303-312. doi: 10.4236/jsea.2015.87031
<!--?xml version=“1.0”?-->
<project name="“COOJA simulator” default="run" basedir=".">
<property name="java" location="java/">
.
.
<property name=“args” value=”/>
<property name=“codecoverDir” value=“/home/user/Desktop/CodeCover/codecover-batch-1.0/lib”/ >
<property name=“sourceDir” value=“/home/user/contiki-2.7/tools/cooja/java”/>
<property name=“instrumentedSourceDir” value=“instrumented”/>
<property name=“mainClassName” value=“se.sics.cooja.GUI”/>
<taskdef name=“codecover” classname=“org.codecover.ant.CodecoverTask” classpath=“${codecoverDir}/codecover-ant.jar”/ >
<target name=“clean”>
<delete>
<fileset dir=“.” includes=“*.clf”/>
</delete>
<delete file=“codecover.xml”/>
<delete file=“report.html”/>
<delete dir=“report.html-files”/>
</target>
<instrument containerId=“c” language=“java” destination=“${instrumentedSourceDir}” charset=“utf-8” copyUninstrumented=“yes” >
<source dir=“${sourceDir}”>
<include name=“**/*.java”/>
</source>
</instrument>
<save containerId=“c” filename=“codecover.xml”/>
</codecover>
</target>
<javac srcdir=“${instrumentedSourceDir}” destdir=“${instrumentedSourceDir}” encoding=“utf-8” target=“1.7” debug=“true” classpath=“${codecoverDir}/lib/codecover-instrumentationjava. jar:/home/user/contiki-2.7/tools/cooja/lib/log4j.jar:/home/user/contiki- 2.7/tools/cooja/lib/jdom.jar:/home/user/contiki-2.7/tools/cooja/lib/jsyntaxpane.jar” includeAntRuntime=“false” >
</target>
<java classpath=“${instrumentedSourceDir}:${codecoverDir}/lib/codecoverinstrumentation- java.jar:/home/user/contiki- 2.7/tools/cooja/lib/log4j.jar:/home/user/contiki- 2.7/tools/cooja/lib/jdom.jar:/home/user/contiki-2.7/tools/cooja/lib/jsyntaxpane.jar” fork=“true” failonerror=“true” classname=“${mainClassName}” >
<jvmarg value=“-Dorg.codecover.coverage-log-file=test.clf”/>
</java>
</target>
<target name=“create-report”>
<codecover>
<load containerId=“c” filename=“codecover.xml”/>
<analyze containerId=“c” coverageLog=“*.clf” name=“Test Session”/>
<save containerId=“c” filename=“codecover.xml”/>
<report containerId=“c” destination=“report.html” template=“/home/user/Desktop/CodeCover/codecover-batch-1.0/reporttemplates/ HTML_Report_hierarchic.xml” >
<testCases>
<testSession pattern=“.*”>
<testCase pattern=“.*”/>
</testSession>
</testCases>
</report>
</codecover>
</target>
<target name=“help”>
<echo>
.
.
<target name=“copy configs” depends=“init”>
<mkdir dir=“${build}”/>
<copy todir=“/home/user/contiki-2.7/tools/cooja/instrumented”>
<fileset dir=“${config}”/>
</copy>
.
.
.
<target name=“jar_cooja” depends=“init, compile, copy configs, compile instrumented “ >
<mkdir dir=“${dist}”/>
<jar destfile=“${dist}/cooja.jar” base dir=“/home/user/contiki- 2.7/tools/cooja/instrumented” >
<manifest>
<attribute name=“Main-Class” value=“se.sics.cooja.GUI”/>
<attribute name=“Class-Path” value=“. lib/log4j.jar lib/jdom.jar lib/jsyntaxpane.jar”/ >
</manifest>
</jar>
<mkdir dir=“${dist}/lib”/>
<copy todir=“${dist}/lib”>
<fileset dir=“${lib}”/>
</copy>
</target>
</project>