Motion Localization with Optic Flow for Autonomous Robot Teams and Swarms

The ability to localize moving objects within the environment is critical for autonomous robotic systems. This paper describes a moving object detection and localization system using multiple robots equipped with inexpensive optic flow sensors. We demonstrate an architecture capable of detecting motion along a plane by collecting three sets of one-dimensional optic flow data. The detected object is then localized with respect to each of the robots in the system.


Introduction
Sensor-equipped mobile swarm robots often have limited sensing and communication capabilities.In particular, sensors are sometimes constrained by the amount of data they can acquire or transmit to each other or to a base station.As a result, it is desirable to focus expensive sensors on only important targets and not waste sensing resources on empty space.We introduce a system for inexpensive moving object detection among swarm robots.Coarse optic flow information is computed from sensor data quickly on special hardware and is used to perform a rough localization on a plane.In this scheme, the addition of sensors scales well as optic flow is suited to computation on specialized hardware.The initial detection and localization given in this paper may be used for further investigation by more expensive sensors and computation.

Background Journal of Computer and Communications
served by a camera.This effect is caused by the motion of objects in a scene and relative to the camera [1].Although optic flow often refers to a dense vector field of motion of each pixel, averages over an entire scene can be very useful in many applications and is often much easier to compute.Optic flow has been used extensively in robotic applications such as egomotion estimation, navigation, and obstacle detection [2] [3].
Analogous to optic flow, the motion of points in three-dimensional space is sometimes called scene flow.There has been work in reconstructing the scene flow from optical flow data from multiple camera views.This problem is ill-posed in general because flow tangent to the direction of the image intensity gradient is indeterminate [1].

Image Interpolation Algorithm
There are a large number of popular algorithms for computing optic flow between two images.They exhibit varying computational complexities and accuracy.Most algorithms assume what is called the brightness constancy, which can be stated for an appropriate neighborhood around pixel coordinates ( ) where I denotes the image intensity at a given position and u x y t v x y t = is the optical flow [4].This assumption implies that an optical flow vector ( ) , u v exists whenever when the intensity of the image matches an image at a later time shifted by that flow vector.This works best when scenes are consistently illuminated and the reflectance of object surfaces is independent of viewing orientation.
Since Equation (1) imposes a single constraint with two unknowns, in general it is impossible to solve for the optic flow.For example, consider a pattern consisting of horizontal lines.While vertical motion is easily discerned, a horizontal displacement does not cause a change in the image.To solve this issue, called the aperture problem, algorithms impose other constraints such as smoothness of the optic flow vectors [4].
Many algorithms operate by matching features among a set of images, which can involve costly computation.The image interpolation algorithm (IIA) proposed by Srinivasan instead works directly on the image gradient in a single pass [5].IIA aims to estimate global optical flow and therefore arrives at a solution which best explains motion of an entire image instead of at each pixel.In particular it considers a number of versions of the image shifted by reference amounts and finds the ( ) , x y vector that produces the best interpolating image.Where f denotes an image intensity, we linearize Equation (1) about the first ) where ( ) ( ) IIA finds the ( ) x y which minimizes the mean square error between the second image f and its estimate f : ( ) We assume that the displacement consists of a vertical and horizontal component with no rotation.From substituting (2) into (3), ( ) Then the final equation can be solved as The IIA algorithm is simple to compute in a single pass and is robust to local failures of the assumption (1) as it averages over the entire image [5].

Approach
Optical flow is related to scene flow by the relative position of the object as well as the camera projection matrix.Assume an object in 3D space has a position represented in homogeneous coordinates by [ ] , , ,1 . Then i u is related to x by the projection matrix as follows: where ~ denotes equality up to a scaling factor.
We can state the constraints (Equations ( 5) and ( 6)) for each view Or where Equation ( 8) can be solved using the pseudo-inverse.This is called the direct linear transformation algorithm [6].
, we would like to recover 0 x and 1 x .The scene flow x  and optical flow u  over time Δt x and { } 0,1 i u are related by a scaling factor: Both 0 x and 1 x can be computed using Equation (8) for each set of points.
This gives us our estimate of the location of the object and its velocity.

Implementation
In this work we focus on coarse localization in sparse scenes.We assume that objects are in the foreground and move rigidly.Optical flow sensors are arranged on the robot agents as in Figure 1.We assume the positions and orientations of the robots are known.
The optical flow sensor we use is the Centeye Stonyman vision chip fitted with a cellphone-type lens.Among the advantages of this device are its low cost, low power consumption, and ease of interfacing with a micro-controller.The chip supports a resolution of up to 112 → 112 although data can be read asynchronously from any size pixel region.We determined the (horizontal) field of view of the sensor to be 40 by collecting images of a checkerboard pattern at various ranges and computing the angular extents.The projection matrix is assumed to be of the form We use the Mbed NXP LPC1768 microcontroller to interface with the Centeye sensor.This architecture is illustrated in Figure 2. Using a microcontroller allows the data to be processed with little expense and without introducing a burden to the CPU on board the robot.This device is fairly low power and inexpensive.This optic flow algorithm could also be implemented at even lower cost directly in hardware.From the robot's point of view, the microcontroller and Centeye configuration emulates a special purpose optical flow sensor.The robot communicates with the Mbed chip through a serial interface over USB.A simple message library was implemented to deal with initialization and collecting vertical and horizontal flow data from each region of the sensor.
Finally, optic flow data from each robot is relayed to a centralized processor which computes the target position and velocity estimate.It would also be easy to perform this calculation directly on one or all of the robots, given a communication link between them.As optic flow is computed for few large regions on each vehicle, there is little data to transmit compared to full images from the Centeye sensors.
We compute the optic flow using the image interpolation algorithm on blocks of size 24 × 24 pixels.Therefore, each window covers about 8.57˚ in the vertical and horizontal directions.Three such regions are used for each sensor: one in the center, and two 12.14˚ to the left and right.Optic flow is computed on each microcontroller at 6.67 Hz.A length 6 moving average filter is used to reduce sensor noise.This data is sent to the central computer for processing.Averaged optical flow data under a threshold is discarded to ignore sensor noise.For each region i , we assume 0 i u refers to the center of that cell.The final position 1 i u is then estimated from the computation of u  , performed on the microcontroller.From 0 i u and 1 i u we use the direct linear transform algorithm described above to compute 0 x and 1 x .

Results
The experiment was conducted in a Vicon arena approximately 10 meters by 12 meters in size.Three robots were situated in an approximate triangle.A human wearing a Vicon tracked hat walks into the scene between the robots.Each robot collected and averaged data, relaying it to a base station.The position was then estimated from the flow data and the positions and orientations of the robots.In our experiment, the human was alerted by an audible command (e.g."Stop right there") and the position estimate is relayed to a camera system which performs facial detection and finer localization.
The averaged, thresholded flow data is shown in Figure 3.The threshold was chosen to eliminate most false positives when there was no scene activity.It is clear that as the target walks onto the scene, the sensors register hits in either the positive or negative direction.
An overview of the scene is given in

Discussion
Optic flow was successful in detecting motion about a scene.Although individual sensors are very noisy, thresholding and combining data from multiple

Figure 2 .
Figure 2. Multi-robot optic flow architecture.Processor collects and processes data from multiple robots.

Figure 4 .
The path followed by the person is given in green, as tracked using a Vicon marker helmet.Black points are the estimated coordinates.Red vectors give the estimated scene flow in arbitrary units.The black points are generally on intersections of the yellow sensitive regions of the Centeye sensors.This is a limitation of the block-based optical flow computation.However, the red motion estimates also track the position change of the object, providing more data about its path.Estimates with extremely low velocities such as the extraneous point in Figure4can be discarded.

Figure 3 .
Figure 3. Optic flow scenario data collected on each robot.

Figure 4 .
Figure 4. Scenario view from above.Dimensions are in millimeters.The three points where yellow lines emanate from are the locations of the three robots.A spurious point near (0, 500) could be excluded due to its low scene flow value.
The coordinates of i u and i v