Augmented Reality for Realistic Simulation Using Improved Snake and Picking Algorithm by Proportional Relational Expression

In realistic simulation of mobile Augmented Reality, essential point is how to best depict occluded area in such a way that the user can correctly infer the depth relationships between real and virtual objects. However, if the constructed 3D map of real world is not accurate or the density is not sufficient to estimate the object boundary, it is very difficult to determine the occluded area. In order to solve this problem, this paper proposes a new method for calculating the occlusion area using the improved snake algorithm and picking algo-rithm by the proportional relational expression. First, we generated the wireframe by the DEM in the experimental region and mapped to CCD real image using visual clues. And then, we calculated the 3D information at the point where occlusion problem for a moving virtual target by the proposed method. Experimental results show the validity of the proposed approach under the environment in which partial occlusions occur.


Introduction
This paper studied on the development of a realistic simulated training model through the display of virtual targets in the input images of CCD camera mounted in a vehicle and the determination of occlusion areas generated from the creation and movement of virtual objects through a movement path according to a scenario.Augmented reality has three general characteristics: image registration, interaction, and real time [1,2].Image registration refers to the matching of the locations of the real world object that user watch and the related virtual object.Interaction implies that the combination of virtual objects and the objects in real images must be harmonized with surrounding environment in a realistic manner, and refers to the determination of occlusion areas according to the changed location or line of sight of the observer or the re-rendering of virtual objects after detection of collisions.Real time refers to the real time image registration and interaction.A key problem in the AR field is how to best depict occluded objects in such a way that the viewer can correctly infer the depth relationships between different real and virtual objects.For occlusion processing such as the hiding of farther virtual objects by closer real objects and the covering of objects in real images by other virtual objects, the two worlds must be accurately coordinated and then the depth of the actual scene must be compared with the depth of virtual objects [3,4].But if the accuracy or density of the created map is insufficient to estimate the boundary of occlusion area, it is difficult to determine the occlusion area.To solve this problem, first, we created a 3D wireframe using the DEM of the experiment area and then coordinate this to CCD camera images using visual clues.Second, to solve the problem of occlusion by accurately estimating the boundary regardless of the density of map, this paper also proposed a method to obtain the reference 3D information of the occlusion points using the improved Snake algorithm and the Picking algorithm and then to infer the 3D information of other boundaries using the proportional relations between 2D and 3D DEM.Third, for improving processing speed, we suggest a method by comparing the MER (Minimum Enclosing Rectangle) area of the real object in the camera's angle of view and the MER of the virtual target.
We briefly review related work in Section 2. In Section 3, we outline the framework of our proposed algorithm.The methodology of Wireframe modeling, improved snake algorithm for extracting image boundary and picking algorithm for acquiring 3D information are explained in Section 4. Section 5, we show the experimental results and the validity of the proposed approach.Finally, in Section 6 we discuss drawbacks of the algorithm and propose possible future work.

Previous Work
A basic design decision in building an AR system is how to accomplish the combining of real and virtual.Toward this purpose, many researchers make efforts to minimize virtual objects registration error and to increase the realness of virtual objects [5].Drastic and Milgram list a number of cues that a user may use to interpret depth, including image resolution and clarity, contrast and luminance, occlusion, depth of field and accommodation [6].We can use one of two technologies to see the real world in AR, one is optical see-through and the other is video see-through.These technologies can present occluded objects, and each has a variety of challenges [7].Blurring also can help compensate for depth perception errors [8].Koch uses computer vision techniques to infer dense and accurate depth maps from image pairs, and uses this information to construct 3D graphical representations of the restricted static environment [9].Several authors observe that providing correct occlusion of real objects by virtual objects requires a scene model.Correct occlusion relationships do not necessarily need to be displayed at all pixels.The purpose of many applications is to see through real object.We introduce here mobile vehicle-mounted display system capable of resolving occlusion between real and virtual objects.We restricted the real environment to some area and constructed that area's scene Model using 3D information.The heart of our system is boundary extraction algorithm and 3D information inference algorithm of object boundary.Figure 1 shows our monitor-based training vehicle configuration.
In our experimental vehicle configurations, we send steering, acceleration, brake data to car driving controller through Bluetooth using remote car controller.Vehicle can be controlled by transmitted data and we can get feedback of present car location data by mounted sensor system.RS232 communicator is interface between vehicle driving controller and sensor fusion system.And it receives instructions from sensor system.CCD camera views the environment.The camera may be static or mobile.In mobile case, the camera might move around by being attached to a vehicle, with their locations tracked by GPS and INS.The image of real world and the virtual images generated by a scene generator are combined.

Formation of Wireframe Using DEM and Registration with Real Images Using Visual Clues
The topographical information DEM (Digital Elevation Model) is used to map the real world coordinates to each point of the 2D CCD image.DEM has information on the latitude and longitude coordinates expressed in X and Y and heights in fixed interval.The DEM used for this experiment is a grid-type DEM which had been produced to have the height information for 2D coordinates in 1M interval for the limited experiment area of 300 m x 300 m.The DEM data are read to create a mesh with the vertexes of each rectangle and a wireframe with 3D depth information [10,11] as Figure 3.This is overlaid on the sensor image to check the coordination, and visual clues are used to move the image to up, down, left or right as shown in Figure 4, thus reducing error.Based on this initial coordinated location, the location changes by movement of vehicles were frequently updated using GPS (Global Positioning System) and INS (Inertial Navigation System).

Edge Map Using Gradient Vector Flow
The Snake algorithm [12,13] is a method of finding the outline of an object by repeatedly moving to the direction of minimizing energy function from the snake vertex input by user.But existing snake algorithm cannot accurately extract the contour information when the object form is complex because as shown in Figure 5, the direction of the energy function appears as a composite vector of the current, previous, and the next snake points, and shrinks toward the center of these points.To solve this problem, this paper proposes a method to form an edge    map using the Gradient Vector Flow (GVF) algorithm [14,15,16], and add a new energy term that indicates the distance between the searched edge point and snake point so as to extract an accurate contour.
The GVF algorithm can measure the contour of complex objects using the gradient of edge, and move to the concave contour regardless of initialization.Further, the gradient vector of the edge map has a larger value as it is near edge, and approaches zero as it is farther.This paper uses the edge information of the gradient vector flow to search the proximal edge point, and when there is an edge, adds a new energy term(E edge-distance ) that shows the dis tance from the reference point to the searched edge as Equation (1).Here, α β, and γ are all set to 1 without exhaustive adjustment .

The Proximal Edge Search Method
Figure 6 shows a proposed proximal edge search method in this paper.First it searches edge points while rotating around the axis d which is the connection between current and previous snake points v i and v i-1 .In other words, if the angle formed by the three points v i , v i-1 , and v i-2 is  , to prevent the situation where the axis meets with or passes by v i-2 and meets v i again, it searches the edge point where the image strength ▽I is greater than the threshold while rotating only by direction for accurate search by assuming the following two facts: First, it was assumed that the initial snake points form a closed curve that encloses the object.Second, it was assumed that the points were arranged sequentially in one direction.The reason for this was because to search proximal edge, it must move inside the contour, but the direction may be wrong due to the diversity of object forms if simply the direction to the object center is set.Figure 7 is an example of setting the rotation direction of the snake points.

Calculation of E edge-distance
Figure 8 is an example of calculating the distance between an arbitrary snake point v i and the edge around it.If we surround the arbitrary point v i with a 9×9 window and assume that its distance with a new edge is , the height and width of the window are s, and the horizontal and vertical positions of the snake point in the window are m and n, the can be obtained with the Equation ( 2) by the Euclidean theorem, and the energy term to be added can be defined as the Equation ( 3) by applying the distance value instead of the brightness value of the image term.
Added new energy term E edge-distance is expressed together with continuity and curvature energy terms in Figure 9.When only the two terms of the existing algorithm were considered, the minimum point of energy was in line 3, column 5, but the location changed to line 4, column 6 when the energy value in consideration of the distance between proximal edges was included.In conclusion, the flow of the enhanced snake energy function to which the proximal edge energy function is added can extract the edge exactly in complex situations by approaching the edge more closely.
Table 1 shows the pseudo codes of the proposed algorithm using proximal edge search method.

Acquisition of 3D Information Using the Picking Algorithm
In order to acquire the 3D information of the extracted vertexes, this paper used the Picking algorithm which is a well-known 3D graphics technique [17].It finds the collision point with the 3D wireframe created by DEM that corresponds to the points in 2D image and provides the 3D information of the points.The picking search point is the lowest point of the vertexes of the objects /* following process determines where to allow corners */ For i=0 to n-1 extracted from the 2D image.The screen coordinate system that is a rectangular area indicating a figure that has been projection transformed in the 3D image rendering process must be converted to the viewport coordinate system in which the actual 3D topography exists to pick the coordinate system where the mouse is actually present.First, the conversion matrix to convert viewport to screen is used to obtain the conversion formula from 2D screen to 3D projection window, and then the ray of light is lengthened gradually from the projection window to the ground surface to obtain the collision point between the point to search and the ground surface.Figure 10 is an example of picking the collision point between the ray of light and DEM.The lowest point of the occlusion area indicated by an arrow is the reference point to search, and this becomes the actual position value of 2D image in a 3D space.

Creation of 3D Information Using Proportional Relational Expression
The collision point, or reference point, has 3D coordinates in DEM, but other vertexes of the snake indicated as object outline cannot obtain 3D coordinates because they don't have a collision point.Therefore, this paper suggested obtaining a proportional relation between 2D image and 3D DEM using the collision reference point and then obtaining the 3D coordinates of another vertex.Figure 11 shows the proportional relation between 2D and 3D vertexes.In Figure 11, m S is center of screen, B S is reference point of snake vertexes (lowest point), To get m p in 3D that passes the center of the screen using the coordinates of the reference point obtained above, must be obtained first.As the t value is given by the picking ray, the given t value and y B are used to get θ B and is obtained using thisθ B in Expression (4).

Creation of Virtual Target Path and Selection of Candidate Occlusion Objects Using MER (Minimum Enclosing Rectangle)
To test the proposed occlusion-resolving algorithm, we created the movement path of a virtual target, and determined the changes of the direction and shape of the target as well as the 3D position of the target.First, the beginning and end points of the target set by instructor were saved and the angle of these two points was calculated, and the direction and shape of the target were updated in accordance with the change of the angle.Further, the remaining distance was calculated using the speed and time of the target, and the 3D coordinates corresponding to the position after movements were determined.We also suggest a method of improving processing speed by comparing the MER (Minimum Enclosing Rectangle) area of the object in the camera's angle of vision and the MER of the virtual target because the relational operations between all objects extracted from the image for occlusion processing and the virtual target take much time.The MER (Minimum Enclosing Rectangle) of an object refers to the minimum rectangle that can enclose the object and determines the object that has an overlapping area by comparing the objects in the camera image and the MER of the virtual target.In addition, the distance between object and virtual target is obtained using the fact that the determined object and virtual target are placed more or less in a straight line from the camera, and this value was used to determine whether there exists an object between the virtual target and the camera.

Experimental Results
Figure 12(up) shows movement path of the virtual target which trainee sets.Also, (down) shows the various virtual targets created to display the targets changing with movement on the image.
Figure 13, Figure 14 compares the search results, accuracy    and speed for more complex leaf.As shown in the figures and graphs, we can see that the proposed algo rithm has much higher accuracy and less repetition counts, and the speed is equal to greedy algorithm.As shown in Figure 13, the proposed algorithm stopped search at the 80 th round, and the accuracy was 0.96 while the Kass and greedy algorithms showed the search count 96 and 150 and the accuracy 0.78 and 0.84, respectively.Therefore, we can conclude that the proposed algorithm has higher performance than existing algorithms.The search speed of the proposed algorithm was 1.65 seconds, which is equal level to the greedy algorithms.
Figure 15 shows the virtual images moving along the path by frame.We can see that as the frames increase, it is occluded between the tank and the object.
Table 3 compares between the case of using snake vertexes to select objects in the image to compare with virtual targets and the case of using the proposed MER.With the proposed method, the processing speed decreased by 1.671, which contributed to performance improvement.

Conclusions
To efficiently solve the problem of occlusion that occurs when virtual targets are moved along the specified path over an actual image, we created 3D virtual world using DEM and coordinated this using camera images and visual clues.Moreover, the enhanced Snake algorithm and the Picking algorithm were used to extract an object that is close to the original shape to determine the 3D information of the point to be occluded.To increase the occlusion processing speed, this paper also used the method of using the 3D information of the MER area of the object, and proved the validity of the proposed method through experiment.In the future, more research is required on a more accurate extracting method for occlusion area that is robust against illumination as well as on the improvement of operation speed.We also hope to study more in real time environment and to overcome complicated factors that were beyond our control, such as sensor error in the current settings, the brightness difference of same image.

Figure 2
Figure 2 outlines the framework of our proposed system.System has two stages.First stage is environment map construction stage.It consists of registration of two world using visual clues and object contour extraction and acquiring 3D information.Second stage is virtual object rendering stage.It has creation of virtual target path and selection of candidate occlusion object and occlusion processing.

Figure 4 .
Figure 4. Registration of two worlds using visual clues.

Figure 5 .
Figure 5.The direction of energy minimization search in snake algorithm.

' i v 2 
and adds a new energy term using the value of the distance d' between v i and ' to the existing algorithm.This paper determined the rotation i v Copyright © 2009 SciRes.IJCNS

Figure 8 .
Figure 8. Distance between a point of snake and edge.

Figure 10 .
Figure 10.3D information extraction using collision point of ray and DEM.(a)occlusion candidate (b)matching ref.point and DEM (c)3D information extraction.
of straight line of B P in 3D, which is through the center of screen.B P is a 3D correspondence point of , vector of t to xz plane.

Figure 11 .
Figure 11.Proportional relation of the vertexes in 2D and 3D.

Figure 12 .
Figure 12.Moving route creation(up) and appearance of virtual object as it moved(down).

Figure 15 .
Figure 15.Experimental results of moving and occlusion.