Visualization and Management Platform with Augmented Reality for Wireless Sensor Networks

Recently a ubiquitous sensor network which collects our environmental information gets increasingly popular, a visualization application is necessary for users to manage complicated wireless networks, however, these applications are developed individually for wireless communication standard or a type of wireless device. Therefore, users are forced to adopt and use the application individually according to the target of the wireless network. In this paper, we propose a visualization platform for wireless network environments using augmented reality technology, and evaluate the effectiveness of the platform. From the result of the evaluation, we have confirmed the proposed platform has availability for visualization and management of wireless networks.


Introduction
Recently, wireless networks of various standards such as Wi-Fi [1], Bluetooth [2], and ZigBee [3] are proliferating around us.Networks of small-scale wireless sensors, called the Internet of Things [4] or M2M [5], are also being built, and research is being done on technology to sense and automate our surroundings.The number of wireless devices using these wireless networks continues to increase, wireless networks are becoming increasingly complex, and new wireless communication standards are appearing to accommodate user applications.
On the other hand, little emphasis has been placed on overall visualization and management of wireless networks for the Internet of Things so far.There are visualization applications that can be used to visualize and manage complex wireless networks, but there are no such applications that handle various wireless communication standards or types of wireless devices, so applications must be selected and used depending on the conditions.However, as the number of wireless communication standards and types of wireless devices increases, it is becoming difficult for users who do not understand the differences among the wireless communication standards to select a visualization application and manage their wireless networks.Further, with general visualization techniques, which map wireless devices onto a virtual space to visualize the state of the wireless network, the location of the wireless device in virtual space is unrelated to the position of the device in real space, so it is difficult for users to associate the devices in the virtual space with those in real space, so it is difficult to identify them.
In this paper, we propose a platform that uses Augmented Reality (AR) technology [6] for visualization and management of wireless networks to resolve these problems.Visualizing wireless networks using AR technology allows us to build a user interface based on images of real space, so it is easy to identify wireless devices.The proposed platform also handles multiple wireless communication standards, and provides application programming interfaces (APIs) for visualization and management functions so that applications can be built using the platform functionality.This allows applications to handle multiple wireless communications standards and devices easily, and users can deal with faults and configure wireless networks by using a single visualization application.

Wireless Network Visualization and Management Issues
Generally, a visualization application is used to determine and manage the state of a wireless network.Visualization applications retrieve information from wireless devices in the wireless network, perform a mapping of each wireless device onto 2D space, and create a visualization showing the information from the wireless devices and the network.To manage the network, wireless devices are selected on the screen of the visualization application, and control operations are performed.
The three most common visualization functions in a visualization application are visualizing links, visualizing traffic, and visualizing wireless device information.In general, most visualization applications only implement a few of these functions, depending on their purpose.

Link Visualization
When two wireless devices are connected to each other, they are linked.Links are generally visualized by mapping wireless devices to a virtual space and showing the link status with a line joining the two devices.The network topology is shown by displaying all links in the wireless network.

Traffic Visualization
Traffic refers to the content of data (or packets) or the data volume flowing on a wireless network.Traffic is generally visualized by analyzing packet content and displaying it, by associating traffic volume with the thickness of the line used to display the link, or by using animation to display the packets.

Wireless Device Information Visualization
Wireless device information includes the device name, its network address, the type of wireless network, its role in the wireless network (base station, relay node, terminal node, etc.), and its hardware state.
The two most common control functions in a visualization application are link switching and power switching.
1) Link switching Link switching is the function of specifying wireless devices for a particular wireless device to connect to, and performing the required link connections and disconnections.For ordinary Wi-Fi networks, this refers to connecting and disconnecting wireless devices to a wireless router, and for Bluetooth it refers to connecting and disconnecting between wireless devices.
2) Power switching Power switching refers to selecting a particular wireless device and turning the power on or off, or re-booting the device.

Proposal Objectives
In this paper, we propose a platform that uses AR technology for visualization and management of wireless networks.The proposal has two objectives as described below.
The first is to abstract the various wireless specifications, such as Wi-Fi and Bluetooth, positioning the platform between the specifications and applications as shown in Figure 1.Wireless network visualization and management functions are implemented in the platform and provided through APIs, so visualization applications operating on the platform can handle multiple wireless specifications and wireless devices easily.Because of this, users no longer need to select an application for each wireless specification or wireless device.
The second is to integrate the virtual space, into which wireless devices are mapped, with real space using AR technology.To do this, it is necessary to integrate both position information, which is the basis for displaying supplementary information using AR technology, and the corresponding wireless-device-specific information, such as its ID, into the platform.With AR technology, a real-space image is used for the user interface, so applications can build a user interface by retrieving location information for wireless devices from the platform and using AR technology to overlay information from the virtual space onto the real-space image.This resolves any inconsistencies between the virtual space and real space with earlier visualization applications.

Functional Requirements
We now describe the functions required by the proposed platform in order to realize the objectives described in the previous section.

1) Abstraction of wireless communication specifications
The proposed platform supports multiple wireless communication standards, abstracts the wireless communication standard layer, and provides APIs for visualization and control functions.Control functions must be handled according to the type of each wireless device, but much of the wireless network information needed for the visualization functions, such as links and topology, does not depend on the wireless communication standard, so there is much in common that can be combined in the APIs, even for different wireless communication standards.Applications can retrieve wireless network information for multiple wireless communication standards by just changing parameters in the APIs.
2) Gathering wireless network and device information To implement a conventional visualization application operating on the platform, the platform must have functions to gather wireless network information such as link information, signal strength, and packet data, and also wireless device information such as network addresses and hardware information.3) Managing wireless network and device information Functions for the application to retrieve the gathered wireless network and wireless device information are needed.It is also essential to accumulate logs of wireless network information in cases when the past state of the wireless network is to be reproduced, or when comparing the current state to past state.

4) Wireless device control
To implement a conventional visualization application operating on the platform, the platform must have functions to control the wireless network upon receiving instructions from the application, such as for connecting and disconnecting links, and for controlling the wireless device hardware, such as for turning the power on or off.

5) Location data management
There are several ways to specify location information for objects being handled with AR technology, and each of them has advantages and disadvantages.For this reason, the proposed platform does not limit the method used for positioning objects, but only manages the information needed to implement each such method, and it leaves the object positioning method to be handled by the application.

Architecture and Functions
The architecture of the platform is shown in Figure 2, and the functionality of each element is described below.
1) Communication modules: These modules communicate with each wireless network, connect to the wireless network and devices, retrieve packets flowing on the networks and send packets on the networks.There is a communication module for each wireless communication standard.For example, there are Wi-Fi, ZigBee and Bluetooth modules for Wi-Fi, ZigBee, and Bluetooth communication respectively.Communication modules consist of two components: Packet sender and Packet capture.
Packet sender handles packet transmission within the communication module.When a packet needs to be sent (e.g. for an operation on a wireless device, or a packet has arrived for a terminal on the wireless network), this component generates and sends the packet.
Packet capture handles receiving packets within the communication module.It receives packets and sends them to the Network DB manager to register them in the database, according to commands from the Platform manager.2) Network DB manager: This is the component that manipulates the Network database, which stores information regarding the wireless networks.Wireless network information refers to packet data flowing on a wireless network.The Network DB manager stores, retrieves and deletes packet data in the Network database.The packet data stored in the Network database is obtained from the communication module, and the manager performs queries and sends the results to the Network analyzer when sending stored wireless network information to an application.
3) Network analyzer: This component analyzes and formats wireless network information.When a command to retrieve wireless network information is called using an API, it retrieves the wireless network information from the Network database through the Network DB manager according to the Platform manager's command, and performs operations such as filtering, protocol analysis, and link analysis.
4) Device DB manager: This component manipulates the Device database, which manages information regarding devices on the wireless network.The Device DB manager gets device information from the Communication module and stores it in the Device database.Device information includes device network address, name, type, state, and control messages.It retrieves device information from the Device database upon request from the Platform manager and sends it to the Platform manager.
5) ARDB manager: This component manipulates the AR database, which manages position data for wireless devices.The wireless device position data managed by the AR database is used as the basis for visualizing the wireless network using AR technology.There are two methods for storing and updating position information for wireless devices.The first is for wireless devices equipped with GPS, in which case, the ARDB manager retrieves the wireless device position information from the Communication module and stores or updates it in the AR database.For the second method, the application must call API methods to store or update the location information.
6) Platform manager: Platform is the central processing component and it receives data and passes to the other components.When an application calls an API, the content of the called API is passed to the Platform manager, where it is processed.When doing so, it gets information from each of the DB managers and directly from the wireless networks and devices through the Communication modules.
7) APIs: The set of commands used by an application to use the functionality of the proposed platform.Depending on the command for the API called, the Platform manager gets information from a database, executes a command directly on a Communication module, or reads a command from Device DB and executes it.There are two APIs: the Network API and the Device API.
The Network API is a set of commands to retrieve information from a wireless network and it handles information in the Network database.It does not depend on particular wireless communication standards.An application can call the Network API to retrieve link, topology, packet or statistical data for multiple wireless communication standards.
The Device API is a set of commands to retrieve wireless device information and to manipulate wireless devices.It handles information in the Device database and the AR database.There are differences in retrieving data from and operating different types of wireless device, so different data is managed in the Device database for each device.This data is used to support differences in data retrieval and operation for the different devices.

Implementation
Wi-Fi and ZigBee Communication modules were implemented based on the design.The implementation environment is shown in Table 1.Overall, the proposed platform was built using Java, and MySQL [7] was used for the databases.The APIs are provided to the application through messages using socket communications, and applications receive API results from the platform in XML format.
For communication with Wi-Fi networks, the device in the PC running the proposed platform was used.The Wi-Fi module uses WinPcap [8] for Packet capture, and a Native Wi-Fi API for the Control component.
For communication with ZigBee networks, a Sun SPOT [9] [10] base station connected to the PC was used.The ZigBee module uses a Packet Sniffer in the Sun Spot SDK for Packet capture, and the Sun SPOT API for the Control component.

Functional Definition
We implemented a visualization and management application using AR technology as an example application using the proposed platform.The visualization and management application defines the following three functions.

1) Display of wireless network information
This function displays the link state between two wireless devices that are communicating.When a wireless device is communicating with another one, a line is shown between the two devices using AR technology.The line disappears when communication stops.
2) Display of wireless device information This function displays the wireless device network address and hardware state.
3) Control of wireless devices This function connects and disconnects links between communicating wireless devices.Links shown with lines on the application screen can be manipulated using gestures, such as connecting and disconnecting links.
A wireless network is composed of two wireless devices.The program keeps one wireless device always communicating with another, and communication is started and stopped by connecting and disconnecting links.

Implementation Environment
The implementation environment and wireless devices used to build the wireless network when implementing the application are shown in Table 2.A Web camera is used for visualization using AR technology.A touch display is also used for gesture operations on the application screen.The NyARToolKit for Java [11] is used to display annotation information using AR technology.
Sun SPOTs are used for the wireless devices composing the wireless network.Sun SPOTs are programmable wireless devices, so network control is possible.The ZigBee stack can be used and various sensors are built in.An identifying 2D barcode marker is attached to each wireless device and registered in the platform database beforehand.

Implementation Results
A real-space image and the operating screen of the implemented application are shown in Figure 3 and Figure 4 respectively.On the real-space image, there are just two wireless devices.On the application screen, wireless network and device information is visualized over the real-space image using AR technology, resolving any inconsistencies between the virtual space and real space.For example, the link between wireless devices in the wireless network is shown as though they were connected by a cable, so it is easy to see the state of links and topology in real space.The network address and hardware state for wireless devices is also shown, so it is easy to know the state of wireless devices.The information for each wireless device can be shown or hidden by touching the device on the operating screen.
With the implemented application, wireless devices can be controlled directly using an interface based on   real-space video, resolving any problems with inconsistencies going from real space to virtual space.On the operating screen, a cable representing the communication between two devices is shown, and the link between the devices can be cut using a cutting (disconnecting) gesture on this cable.If a drag gesture on the operating screen is done between two wireless devices that are not connected by a link, a connection is made between them and communication resumes.

Config Free
Yoneyama et al. have proposed a wireless network visualization user interface called Config Free [12] [13].
Config Free has three functions: Network diagnostics, network device search, and connecting and disconnecting wireless devices.Config Free visualizes wireless devices arranged on a 2D virtual space.The user's PC is placed in the center of the 2D virtual space, and other wireless devices are arranged on concentric circles around it at distances that are closer as the strength of the signal from them increases.A drag-and-drop operation from the user's PC at the center to one of the devices on a concentric circle creates a connection between the wireless devices.When a connection is created, a line is shown between the PC and the other wireless device.The connection can be cut by clicking on the line when the pointer is shown as a pair of scissors.Config Free supports two types of network: Wi-Fi and Bluetooth.

3D-Tcpdump
Saito et al. have proposed 3D-tcpdump [14] [15], which is a visualization application that displays network communication state and topology graphically in 3D.3D-tcpdump has two functions: visualization of traffic on the network and visualization of topology.3D-tcpdump generates and arranges 3D objects in a virtual 3D space for the user's PC, the default router that is one hop away, and the group of destination PCs.When the network traffic occurs, it can be visualized as spherical objects according to the traffic volume from the 3D object to the destination one.

uMegane
Imaeda et al. have proposed the uMegane system [16], for visualizing sensor information using AR technology.uMegane uses AR technology to abstract and display sensor data from wireless sensor devices as 3D objects, using properties such as color, size, and motion of the objects so that ordinary users can understand them easily.This system can also be applied to detect abnormalities using user-definable filters.

Extate
Takimoto et al. have proposed the Extate system [17], which uses AR technology for visualization of wireless networks.Extate has functions to visualize link states in a wireless network and to visualize wireless device information.A 2D barcode is attached to 2D devices, lines are used to display link state, and text is used to display wireless device network addresses and sensor data.

Items Evaluated
Compared to earlier methods, in which visualization and management applications connected directly to wireless networks, with the proposed method, the platform connects to various wireless networks and provides the results of processing to the application.The platform is able to perform visualization of the wireless networks over real space, but if there is significant delay between retrieving wireless network data when performing visualization, it could cause confusion for users expecting to know the wireless network state in real time.
To implement real-time visualization and management of wireless networks using the proposed platform, any increase in the time taken to complete tasks through the platform is undesirable.Accordingly, we evaluated the performance of the proposed platform implementation.

Performance Evaluation
Four functions required in the proposed platform are retrieving link state, retrieving traffic information, retrieving device information, and controlling devices (connect or disconnect links).We measured the response time of each of these functions, namely the time from the instant the application calls the platform API until the application obtains the result.
The application can be used in two ways: either with the platform and the application running on the same host, or with the platform and application running on different hosts and the application using a platform on the same Local Area Network (LAN).Thus, performance evaluations were done in both an environment with the components on the same host, and with them on the same LAN.For the environment using the same host, platform API calls were made to the local host.For the same-LAN evaluation environment, the implemented platform was used, and the implemented application called the platform APIs.Response times in each case were measured using Wireshark.
Measurements are shown in Table 3. Response times for retrieving information are shorter than for device control functions because the proposed platform is asynchronous and gathers information from the wireless network, so it only needs to return results from its internal database.On the other hand, control commands are only sent to a wireless device after a control API is called, and the result is only sent to the application after the control operation has completed.This adds to the time required to control wireless devices, making response times long compared to information retrieval operations.

Discussion of Performance Evaluation
From the performance evaluation results, in both the same host and same LAN cases, the application was able to obtain wireless network state information from the proposed platform with approximately 0.1 to 0.5 seconds of delay.The proposed platform gathers information from the wireless network asynchronously, regardless of whether there is an API call from an application, so delays from the component of the platform gathering information from the wireless network are very small.For wireless networks other than ZigBee, which we used in the evaluation environment, the same asynchronous architecture is used to gather wireless network information so the results will be the same.
For the performance results when controlling wireless devices, the time required to control the wireless device is added, so response times were from 2 to 3 s, but this time depends on the type of device being controlled, so the amount of time added by passing through the proposed platform can be expected to be similar to that required to retrieve wireless network information.
From these results, the delay introduced by the proposed platform is approximately 0.1 to 0.5 s, indicating that the platform will not present problems with delay for real-time visualization of wireless networks using AR technology.

Comparison with Related Research
We compared the proposed method with related research in terms of functionality, visualization methods, and whether it resolves issues discussed above.The results of this comparison are shown in Table 4.
The comparison shows that the application built using the proposed platform is the most effective in terms of  functionality and resolving the issues we have discussed in this paper.

Conclusion
Recently, various devices are being used with wireless networks.Also, with research on the Internet of Things, the number of wireless devices being used by ordinary users is increasing, and we are surrounded by wireless networks based on various standards.With the increasing complexity of wireless networks, an application for visualizing them is essential for understanding and managing their state.However, existing visualization applications are only for particular wireless communication standards or device types, so users must select applications to suit their particular conditions.Also, these visualization applications generally map wireless devices into a virtual space, so there is no relation between the position of devices in the virtual space and their location in real space.Thus, in environments with increasing numbers of wireless devices, it is difficult to identify devices in real space based on information in the virtual space, and to match wireless devices in real space to the corresponding device in the virtual space.In this paper we resolve these issues by developing a wireless network visualization and control platform using AR technology.With the proposed platform, wireless network visualization and management functions are implemented in the platform, and are provided to applications through API functions.This allows applications to support multiple wireless communications standards easily, resolving the problem of having to choose among different applications.Further, visualization and management can be done using AR technology, resolving the difficulty with correspondence between the virtual and real spaces.The results of performance evaluations also demonstrate that the platform does not introduce performance issues.A comparison with related research confirmed that applications implemented using the proposed platform is effective in resolving the problems discussed here.

Figure 1 .
Figure 1.Abstraction of Wireless Network Interface for API.

Figure 3 .
Figure 3. Real image for sensor devices.

Figure 4 .
Figure 4. Virtual image (including annotation information) for sensor network.

Table 1 .
Implementation modules for proposed platform.

Table 3 .
Response time for each function.

Table 4 .
Comparison with related works.