Automatic Body Feature Extraction from Front and Side Images

Human body feature extraction based on 2D images provides an efficient method for many applications, e.g. non-contact body size measurements, constructing 3D human model and recognizing human actions. In this paper a systematic approach is proposed to detect feature points of human body automatically from its front and side images. Firstly, an efficient approach for silhouette and contour detection is used to represent the contour curves of a human body shape with Freeman’s 8-connected chain codes. The contour curves are considered as a number of segments connected together. Then, a series of feature points on human body are extracted based on the specified rules by measuring the differences between the directions of the segments. In total, 101 feature points with clearly geometric properties (that rather accurately reflect the bump or turning of the contours) are extracted automatically, including 27 points corresponding to the definitions of the landmarks about garment measurements. Finally, the proposed approach was tested on ten human subjects and the entire 101 feature points with specific geography geometrical characteristics were correctly extracted, indicating an effective and robust performance.


Introduction
With the popularity of digital cameras, 2D images are captured conveniently with much lower cost, which are considered as the main information carriers for the human recording and understanding.In the past decades, how to efficiently and robustly extract more information from the 2D images, especially from the human images, has become a research hotspot.Recently, automatic body feature extraction from 2D human images is widely used in many fields, such as non-contact body size measurements [11], constructing 3D human models [9,10], recognizing human actions [1][2][3].
For anthropometric data collection, the traditional way is the manual measurement.Since many problems such as subjective judgments of landmarks may be involved in the manual measurement processing, a computerized image-based approach provides an alternative to the traditional method of manual measurement [5].The body scanner is developed for realizing non-contact 3D mea-surements that often need to segment the body parts and it is such a tedious process [4].Recently the 3D measurement based on 2D images has aroused extensive attention.Image-based systems are capable of providing anthropometric measurements that are quite comparable to traditional measurement methods, in terms of both accuracy and repeatability [5].Extracting human feature points automatically from the front and side images is the key part to the non-contact body measurement based on images for the garment industry.
Extraction of human body feature points accurately also plays a very important role in the establishment of a virtual human model.Lin and Wang [9,10] presented a systematic approach for constructing 3D human model using the front and side images of a person based on the feature points of the silhouettes that were identified.To recognize human actions, Ali and Aggarwal [13] identified the knee and hip by calculating the curvature of the silhouette as the feature points to recognize human actions.
In this paper, we propose an efficient, simple and ro-bust human body feature point detection algorithm based on the front and side images of a human body, whose key inspiration stems from a series of studies [9][10][11][12].Background subtraction is used to obtain the contours.Canny edge detection operator [8] and Freeman's 8-connected [6] chain codes are used to represent the silhouette curves.Then the curves are cut into a series of segments whose directions are denoted by the corresponding values of second chain codes in the segments.Therefore, specific rules for measuring the differences between the directions of segments are utilized to extract the feature points.The proposed method was tested on ten human subjects and all of the 101 pre-defined feature points were correctly extracted.Among these feature points, there are 27 points with geometrical properties that indicate concavity and convexity of the curves perfectly, corresponding to the definitions of landmarks related to garment measurements.
In [11,12], a total of 55 and 60 feature points can be extracted from front and side images, respectively.Comparing with [11,12], there are more feature points detected in our work.In addition, different methods of contour detection and different feature point extraction rules are used in our approach.Firstly, the used background subtraction is more applicable to complex background than the methods used in [11,12].Secondly, the feature points extracted in our work are independent with each other.However, in [11,12] the feature point always depend on the previous feature point, even both the adjacent feature points, so if one point is wrongly detected, the others may be unbelievable.Thirdly, in [11,12], the method to extract the secondary feature points is to use the primary feature points as the auxiliary points to help to locate the new feature points.Our method is more intuitive, since all feature points extracted in the same batch.Finally, the feather points detected in [11,12] had not compared with definitions in any standard, it is not enough to take it as left shoulder acromion or other landmarks, but it had been done in this paper.

Photographing and Pre-processing
Firstly, the camera is positioned at 5m distance from the subjects and 1m height from the ground and four images are captured, two of the background and two of the human subject.In experiments, the Canon SX130 camera was used to capture the images with a resolution of 4000×3000 pixels and with opening flash to reduce the effects of shadowing.In order to reduce the data processing time, the front and side images are resized into 1260×1854 pixels and 504×1854 ones which remain the parts of the human body and then are reduced to 211×309 pixels and 84×309 pixels, respectively.At the same time, the corresponding parts of the backgrounds are processed in the same way.Subjects are lightly clothed and also are required to wear a headgear to reduce the influences of hair on the test results.The subjects are requested to stand in front of a white wall with a standing posture.When shooting the human from the frontal view, Legs of the subject are apart from each other for shoulder width and arms open on both sides of the body, while the fingers face to the camera and are forced open.The results of a male model after photographing and pre-processing are shown in Figure 1 (The pictures in this paper all come from our experiment about a male model).

Silhouette Detection and Contour Representation
The used extraction algorithm of the body silhouettes is a standard background subtraction to isolate the subject from images, whose steps are described as follows: 1) Calculating the color difference values of the current image and the background as follows: where Ω denotes the fields of images, X ij and Y ij represent the RGB color value of the current image pixel (i,j), and the background pixel (i,j), respectively.
2) Differences between foreground and background pixels are measured as follows: where d 1 , d 2 and d 3 are the weighted values of Red, Green and Blue channels, respectively.All of our images were taken under the natural light.Since RGB are basically the same as the natural light, the weighted values are set to 1.
3) If D(i,j) is larger than a pre-defined threshold val-ue T ∆ , the pixel (i,j) is regarded as a former attractions, and vice versa for background.After that, two binary images of the body shapes can be obtained.In our experiments T ∆ was empirically set to 35.
where f 0 and s 0 are the starting points on the left-top of the head on each contour curves, f i and s i are the clockwise ( ) To identify these points automatically, the edge pixels of the body silhouettes are encoded by the Freeman's 8-connected chain codes [6,7].With the Freeman chain code, the numbers "0"-"7" are used to represent 45° increments in a counterclockwise directions starting from an initial horizontal vector "0".The body contours are descripted with a series of chain codes of pixels: =000000000000070...

Feature Extraction
The contour curves of the body are considered as connections of a series of line segments.A part of line segments on the head of the front curve are shown in Figure 3.The first segment a 1 connects the first point f 0 and the consecutive points with the same code value of "0" in sequence, so it starts from the point f 0 to the point f 12 .The next segments are the connections of the continuous points are in the same direction and the previous point in the different direction, such as a 2 starts from the point f 12 to the point f 13 .The freeman code of the second point of the line segment in addition to the starting point is taken as the inclination of the line segment or its direction.If the inclinations of the current line segment and the adjacent two segments satisfy one of the conditions that are shown in Figure 4, then the starting or last point (when it comes the left part of the front human silhouette curve) of the current segment line is considered as the feature point of human body.The current line segment direction is presented using red line with arrow and that of the two segments are adjacent to it are used blue lines with arrows to present.If one of the two adjacent segments is in the 45° clockwise direction and the other one must be in the 45° counterclockwise direction of the current segment line.So actually, there are 16 cases that can be used to judge a feature point.Another judgment principle is used while the current line segment is perpendicular to the previous line segment.The starting point of the current segment line is considered as the feature point of human body.All of the judgments principle can be expressed as: where  4)(f) correspond with the case shown in Figure 4(h).Moreover, another judgment principle that the absolute value of d 2 subtracts d 1 equals to 2 in formula (4)(g) corresponds with the condition of two segments are perpendicular.The testing processes of some points are shown in Figure 5 where the starting point on the left-top of the head is token as the first feature point H 1 .The directions of a 9 , a 10 and a 11 are 0, 7 and 6, respectively.Since the three segments conform to the case shown in Figure 4(h), the first point of a 10 is token as the second feature point F 1 .The remaining feature points are detected according to the rules.Detailed steps of the algorithm are described as follows (the front silhouette is taken as an example): 1) Calculating the inclinations of the segments.The contour curves are expressed with the code sequence as: p 0 , p 1 , …, p (n-1) , and then p k is taken as the code value of the point f k .At the first, f 0 is inserted into the sequence V.
In addi- tion to f 0 , every point in V happens to be the second point of each segment whose code value can be used to represent direction of the segment.
2) The detections of feature points.Z is taken as the sequence of feature points.V: v 0 , v 1 , …, v (u-1) represents the sequence of pixels.v 0 is inserted into Z as the first feature point.The code values of the current point v k and the adjacent points v (k-1) , v (k+1) , (0<k<u-1) just represent their line segment directions.Then the principles in formula ( 4) are used to judge the feature points.If v k , v (k-1) and v (k+1) conform one of the conditions in formula (4), the one of two endpoints of the segment line that includes the point v k is considered as a feature point to be inserted into the sequence Z.

Evaluation
The algorithm is demonstrated in Matlab 2009a, the 101 feature points are completely detected from ten objects.It proves that this approach is effective and robust.The feature points on front and side images are shown in Figure 6.The same positions of the feature points detected are also marked in the color images, as shown in Figure 7.It is demonstrated that the feature points is ra-ther accurate and reliable by visual inspection.The test results of ten subjects consisting of five males and five females are shown in Table 1.In total, there are 119 points detected from the front silhouette curve of the male models.However, only 71 points are taken as the feature points.The reasons are given as follows.Firstly, hands have large individual differences and severe changes of concavity and convexity.Only the 18 points representing fingertips and finger roots of the 60 points detected on the hands are taken as the feature points.Secondly, as shown in Figure 6, there are other 6 points marked in Green on the head and waist of the front silhouette curve, and 3 points on the side silhouette curve that are discarded, because they are not detected in the same parts of the other subjects.With these two reasons, the 101 points with reproducibility and stability are taken as the feature points.

Table 1. Test results of the ten subjects.
In [14], 35 landmarks were evaluated.It used "Lowest point", "Tip of process", "Uppermost point", "Most lateral point" and some other words to describe the landmarks related to body dimensions.Leong [15] launched geometry definitions of feature points according to basic definitions in ISO 8559(1989) [16].So the definitions of feature points in [14][15][16] and GB/T 16160-2008 [17] are combined to evaluate the effectiveness of feature point that be detected.
The geometrical properties of the feature points are analyzed.The results are shown in Table 2 and Table 3.  Experimental results show that the feature points can be extracted reliably and effectively by the proposed method.The 27 points that are in line with the definitions of measurement landmarks can be used in garment industry.In addition, all the points have clearly geometric properties what rather accurately reflect the bump, or turning of the contours, so they can be used for the establishment of a virtual human model and the recognition of human actions.In particular, the points of fingertips and the roots of fingers can be used for gesture recognition.

Conclusion
In this paper, an automatic approach is presented to extract body feature points by using front and side images.In total, 101 feature points can be extracted by the proposed approach which measuring the differences between the segment directions.Ten individuals, consisting of five males and five females, were tested in the experiments and 101 points are accurately detected.It shows that the approach has a good robustness.The analysis of the nature of these points shows that this approach is reliable and effective.All the points have clearly geometric properties that can be used in many fields.Furthermore, by comparing the proposed method with the previous researches, it demonstrates the improved performance in feature points extraction.In future work, we will focus on collecting the human dimensions, as well as the establishment of three-dimensional human models based on these extracted feature points.

Figure 1 .
Figure 1.The results of a male model after photographing & pre-processing: (a)-(b) The front image and its background; (c)-(d) The side image and its background.

Figure 2 .
Figure 2. Silhouettes and contours of a human body: (a) Silhouette extraction result of the front image; (b) Silhouette extraction result of the side image; (c) Body contour f C from the front image of the subject; (d) Body contour s C from the side image of the subject.As illustrated in Figures2(a)-(b), silhouettes of the front and side views of the subject are extracted.After extracting the body silhouettes from the 2D images, the Canny edge detector[8] is subsequently applied to locate the silhouettes of human body from the two binary images.The single-pixel and closed contour curves are extracted as shown in Figures2(c)-(d).The silhouette points on f C and s C are denoted as: i th points from the first points, and f C and s C have a total of n points and m ones respectively.

Figure 3 .
Figure 3.A part of line segments on the head.

Figure 4 .
Figure 4. Eight cases of the line segment directions.
1 , d 2 and d 3 denote the inclinations of the three adjacent segments.Formulas (4)(a) and (4)(b) correspond with conditions shown in Figures 4(b-g), formulas (c) and (4)(d) correspond with the case shown in Figure 4(a) and formulas (4)(e) and (

Figure 5 .
Figure 5. Feature points F 1 and F 2 extracted on the head.

Figure 6 .
Figure 6.Body feature points (marked with red circles) consisting of 71 front ones and 30 side ones.

Figure 7 .
Figure 7. Body feature points (marked with red circles) on a male model consisting of 71 front ones and 30 side ones.