Color and Texture Segmentation Using an Unified MRF Model

The color image segmentation problem has two main issues to be solved. The proper choice of a color model and the choice of an appropriate image model are the key issues in color image segmentation. In this work, Ohta ( is taken as the color model and different variants of Markov Random Field (MRF) models are proposed. In this regard, a Compound Markov Random Field (COMRF) model is porposed to take care of inter-color-plane and in-tra-color-plane interactions as well. In continuation to this model, a Constrained Compound Markov Random Field Model (CCOMRF) has been proposed to model the color images. The color image segmentation problem has been formulated in an unsupervised framework. The performance of the above proposed models has been compared with the standard MRF model and some of the state-of-the-art methods, and found to exhibit improved performance.


Introduction
The problem of image segmentation, is the subject of an active research topic over a period of quite a few years. It is connected with object labeling process, that is, assigning to each object a different label (all pixel of the object receive the same value). The problem is more compounded in the real world environment which is colored. Now-a-days, color imagery has become an integral part of human life, because of its tremendous use in internet as well as social media plat-forms. As color Image segmentation, convey much more information about the objects in scenes, this is a crucial issue while designing the front end of automated machine vision system. Even though a good number of image segmentation techniques and strategies has been reported in the literature [1], problem of color image segmentation is a challenging task. Color image segmentation poses two major challenges; 1) devise of appropriate representation or modeling of color components, and 2) the intrinsic characteristics of the images need to be ascertained. There has been conscious effort by researchers to understand the underlying notion behind the real world colors. Attempts have been made to devise different color spaces in both linear and nonlinear frame work [2]. Because of the complex correlation among different color planes, devising appropriate color model for the real world images is a hard task. In color image, besides the complexity of color model, an appropriate image model, taking care of spatial intrinsic characteristics, needs to be designed for appropriate image analysis. In literature, different image models have been proposed for image restoration, filtering, segmentation, object detection, and recognition etc. These models can broadly be categorized as deterministic and stochastic models. Stochastic models found to be used in various image analysis and computer vision applications. Specifically, "Markov Random Field (MRF) models" [3] [4] have the potentiality of modeling the spatial intrinsic characteristic, of an image and is extensively used in "image processing and computer vision" for nearly three decades. MRF model and its variants have been found to be suitable models for many real world images. MRF model is a stochastic model with non linear feature and appropriate modeling of a given image needs to have proper MRF model parameters. Because of the availability of different color spaces for real world color modeling and MRF model as a suitable image model, many image analysis and pattern recognition problems in a real world environment could be addressed. Therefore, the model based color image segmentation and restoration could be an alternate choice to the non parametric methods [3]. Researchers of the image analysis and computer vision community have proposed a wide variety of MRF models based on supervised and unsupervised image segmentation schemes [4] for automated vision system in real world environment.
In this work, an unsupervised segmentation algorithm is developed to segment the different regions of a color image. In this framework the image label as well as image model parameters have been estimated in a concurrent manner. The current unsupervised algorithm, is successfully tested on different real world images, texture images from Berkeley database. However, for the sake of convenience, five results are presented and a comparison is made with Kato's method and JSEG method. As there is a dependence between, the image label estimates and the model parameter estimates, determination of optimal estimates of image labels and model parameters is a hard problem. In order to ameliorate this issue, a "recursive scheme" has been presented, where both the image labels and model parameters are estimated simultaneously. This recursive scheme yields "partial optimal solutions" instead of "global optimal solutions". As far as the 1 2 3 , , I I I color space has been used to model the input color image. The Compound MRF model controls the correlation of the color plane for efficient modeling. In both the MRF models two types of clique potential functions, namely Weak Membrane Model and Reward-punishment model have been used. In the present work, the model parameters are estimated in one step and thereafter in the second step these estimated model parameters are used to estimate the image label estimates. The parameter estimation problem has been cast using "Maximum Conditional Pseudo Likelihood (MCPL)" principle and the estimates of the model parameters have been obtained using the proposed "homotopy continuation algorithm". The MCPL estimation problem reduces to solving a set of nonlinear equations whose zeros are the estimates of the MRF parameters. These zeros of the nonlinear function, have been found by tracing the zero curves of the homotopy map. Fixed point based homotopy continuation method with the homotopy parameter " λ " is developed. An algorithm is developed to trace the zero curve of the homotopy continuation method to determine the zeros of the desired function and hence the parameter estimates. The label estimation problem has been cast using "Maximum a Posteriori (MAP)" estimation principle and the proposed hybrid algorithm is used to obtain the MAP estimates.
In the following, Section II describes the related work while Section III presents the Compound MRF (COMRF) model. The proposed constrained MRF model (CMRF) model has been given in Section IV and Section V deals with the Constrained Compound MRF (CCOMRF) model. Section VI consists of the formulation in unsupervised framework and Section VII presents image label estimation followed by model parameter estimation presented in Section VIII. The simulation results are presented in Section IX. The concluding remarks are given in Section X.

Our Contributions
The list of contributons along with the respective sections can be summerized as follows:  A Compound Markov Random Field Color Image Model is proposed taking into account both the "intra-color-plane" as well as "inter-color-plane" interaction of pixels in RGB and Ohta model (Section 3);  A new MRF model called as Constrained Compound MRF (CCOMRF) model is proposed that is found to possess the unifying property of modeling color texture as well as scene images (Section 5);  An "unsupervised color Image Segmentation scheme" using "Homotopy Continuation method" is proposed for simultaneous estimation of model parameters as well as image labels (Section 6);  A MRF-MAP based supervised image segmentation using Homotopy Continuation Method and hybrid algorithm is proposed (Section 8).

Related Work
There has been conscious effort to devise color models which when used cannot be distinguished from those in original by a human observer. Towards this end, many linear and nonlinear color models have been proposed [2] [5] earlier. The most commonly used linear models are RGB and Ohta ( ) 1 2 3 , , I I I [5] model. Ohta has found out an effective set of color features, that is 3 , while segmenting eight kinds of color pictures and these features, popularly known as Ohta ( ) 1 2 3 , , I I I model. In literature, different color spaces used for segmentation have been investigated and an overview of different color spaces from perceptual, historical and application specific has been presented [6]. This overview [6] highlights both potentiality and also the limitations of different color spaces.
For image restoration and segmentation applications, MRF models found to be very efficient [3] [4] [7]. Geman and Geman [3] have proposed binary line fields together with the random field model of the non edge pixels. Subsequently, Besag [4] proposed a segmentation scheme where the segmentation problem has been cast as a "pixel labeling problem" and the pixel labels have been estimated using "Iterated Conditional Mode (ICM) algorithm". In Besag's [4] formulation, MRF model parameters have been estimated together with the image labels and performed image restoration in MAP framework. MRF model has also been proposed combining color and texture features [8]. Kato et al. [8] have used perceptually uniform CIELUV color values as color features and a set of Gabor filters as texture features. Kato et al. [9] in their subsequent work have proposed a new MRF model based segmentation scheme, where the model consists of three layers; two of which correspond to two features and a special layer called the combined layer. This scheme has produced quite satisfactory results for multi class textured images.
In addition to the MRF based methods, another method known as JSEG has been proposed to segment color images in unsupervised framework [10]. In this approach, a multistage "J-image" has been created and region growing strategy has been used for segmentation. MRF based clustering approach [11] has been proposed for color image segmentation. In MRF based clustering, MRF model has been applied over the pre-segmented data obtained by grouping regions over similar group of pixels. Therefore, clustering has been carried out on a multidimensional feature space. In another research, the "Hidden Markov Random Field (HMRF) model" parameters have been used for segmentation of natural color texture images [12]. MRF model parameters have also been estimated by Kato et al. [13] for segmentation of color texture images. EM algorithm has successfully been used to estimate the model parameters. They have used perceptually uniform "CIE-L * U * V * " color values as color features and a set of "Gabor filters" as  [14] have proposed an unsupervised scheme for color textures using MRF model. They have used "Gaussian Markov Random Field (GMRF)" model for color textures and the algorithm is a region based one. It consists of a region splitting phase and an agglomerative clustering phase. Color and motion have jointly been processed in an unsupervised scheme proposed by Lievin et al. [15]. In this work from a logarithmic model a non-linear color transform relevant for hue segmentation is derived. The proposed "hierarchical segmentation scheme" is based on MRF modeling that combines hue and motion detection within a "spatiotemporal neighborhood". Another "unsupervised color image segmentation scheme", based on feature space, has been proposed by Guo et al. [16], where the feature space consists of two distinct source models and valley. The model parameters have been estimated and a "labeling algorithm" has been developed to determine the segmentation. The segmentation process is completely autonomous. An unsupervised color segmentation algorithm has been proposed [17] using "multiscale texture model". A new MRF model known as "Associative Hierarchical Random Field (AHRF)" has been proposed [18]. They also proposed a new algorithm for optimization. This work is a generalization of many previous "super-pixel based methods" in a random field framework. Here the MAP estimation is carried out using a graph cut based move making algorithms. Besides, another unsupervised algorithm has been proposed [19] in Expectation Maximization (EM) framework, where the model parameters and the pixel labels have been estimated simultaneously. Very recently, Karadag et al. [20] have proposed an unsupervised segmentation algorithm in MRF framework. The bottom up phase takes care of the model parameters while the top down segmentation maps are constructed from the domain specific information. Besides, another color image segmentation scheme has been proposed by Chen et al. [21] who have used both the MRF and Dempster-Shafer evidence theory to obtain segmentation. They have demonstrated for two label segmentation, however, it can be extended for multi label case. Abes et al. [22] have proposed a segmentation scheme where the image structure has been representated by its segmentation graph derived from the low-level hierarchical multi scale image segmentation. A novel "Decoupled Active Contour (DAC)" method is proposed [23] to extract the boundaries accurately. The notions such as "viterbi search resampling and Bayesian estimation" are the key steps of DAC. A method based on a multilocal creaseness analysis of the histogram has been proposed for shape extraction [24] and the resulting segmentation scheme has been found to be robust. A multiscale method using edge and intensity information has been proposed for brain MR image segmentation [25]. In [26], the notion of coupled nonlinear diffusion has been used for feature extraction and enhancement. These features have subsequently been used for segmentation. A novel technique based on geometrical properties of "lattice auto-associative memories" has been proposed [27] for color image segmentation adhering to a different color space. Other than MRF model, research has also been focused on using various other methods such as Graph Cut based methods, Level set methods, Histogram thresholding based methods etc. The review based on segmentation using two powerful attributes, that is color and texture has been presented in [28]. An iterated region merging based graph cut algorithm has been presented in [29]. This is an extension of standard graph cut algorithm. The proposed algorithm starts from the user labeled sub-graph and works iteratively to label the surrounding un-segmented regions. With the same amount of user input, their algorithm can achieve better segmentation results than standard graph cuts, when the object is extracted from a complex background.
A new segmentation algorithm called "Histogram Thresholding-Fuzzy C-means Hybrid (HTFCM)" is proposed in [30]. HTFCM consists of two modules 1) "histogram thresholding module" and 2) the "FCM module". Histogram thresholding technique contains three phases 1) peak finding technique, 2) region initialization and 3) merging process. An outdoor scene image segmentation algorithm based on "background recognition and perceptual organization" is proposed in [31]. It consists of a "Perceptual Organization Model (POM)" that captures the structural relationships among the constituent parts of the structured objects. Boix et al. [32] have proposed a new consistency potential for image labeling, known as the "Harmony Potential". They have presented a new "Conditional Random Field (CRF) model" for object class image segmentation.

Compound Markov Random Field (COMRF) Model
Stochastic models such as MRF models have been extensively used in image analysis [3] [4]. Over the last two decades, there have been extensive applications of MRF models and towards this end many variants of MRF models have been proposed for gray scale as well as color images [2]. In case of color images, the accuracy of color image segmentation greatly depends upon appropriate color model as well as proper image model. Therefore, in this work attempts have been made to develop appropriate color as well as image models for image segmentation. It is known that the RGB color model is not a suitable color model for image segmentation because of the existence of "strong correlation" among the different color planes where as Ohta model, because of its "weak correlation" among different planes, has been widely used for color image segmentation. Therefore, a Compound Markov Random Field model has been proposed to introduce controlled correlation among different color planes through MRF model parameter. This model with Ohta color model proved to be quite effective for image segmentation. In Ohta color space.
In the present section, a compound MRF model is deveioped which is based on both spatial and temporal modeling. In otherwords a spatio temporal MRF modeling is developed in the color space. In our earlier work [33], the notion of Constrained Compound MRF model has been proposed, but in this section, the clique potential function is provided in more detail. Since, the Ohta color model The above has been motivated because of the existence of "strong correlation" in between different color planes of RGB and "weak correlation" among different color planes of Ohta ( ) 1 2 3 , , I I I model. The above spatio temporal MRF model is mooted to ameliorate the above limitation and thus introduce the notion of controlled correlation among different planes. This controlled correlation has been expected to achieve superior segmentation results to the existing RGB or OHTA model. Hence, the proposed COMRF model takes care of "controlled correlation" among different planes and the "degree of correlation" has been monitored by the associated parameters of "clique potential function". All the images have been assumed to be defined on a "discrete rectangular lattice" of x of the input image X is modeled as a random variable taking a value from 0 to G (gray values). Since, the image has been defined over two dimensions, the observed imgae X has been modeled as a random field and x denotes the realization, the given image. Similarly the segmented images is modeled as the label process Z with the number of labels as L. The three color planes of Ohta model has been presented in Figure 1(a) and we model each col- 3) label process is denoted as Z ′ and then the spatial interaction of Z ′ plane has been modeled as MRF, the joint probability distribution ( ) is known to be "Gibbs distributed" and can be presented as follows U z θ is called the energy function and is of the form c V z θ known as the "clique potential function" and θ is the associated "clique parameter vector". Analogously the spatial interactions of 2 I and 3 I planes can be defined. This prior MRF model incorpating the three spatial planes would result in the "energy function" of the following form where,

( )
, c V z θ denotes the "clique potential function" for the three color planes 1 I , 2 I and 3 I respectively. In order to complete the model, the "inter plane interactions together with the intra plane interactions" have been taken care. Thus Z has been modeled as a compound MRF where the "spatio temporal MRF model" takes care of the spatial as well as the temporal interactions. Figure   1(a) shows the interaction among different color planes and as an illustration  , . Vertical line field for each plane is set i.e. , 1 Horizontal line field for kth plane is set, i.e. , 1 Since the COMRF model takes care of "intra color plane as well as inter color plane interactions" the prior probability distribution equation is given by (1), where the energy function is represented as, Here, ( ) .
Here 1 2 3 , , z z z refers to ( ) 1 2 3 , , I I I planes respectively. Equation (5) can be expressed as, where 1 z shows the interaction between 1 2 I I − color planes, 2 z denotes the interaction between 2 3 I I − color planes and 3 z represents the interaction between 3 1 I I − color planes respectively. Here we have assumed, have been chosen on an adhoc manner. The boundary of a given segment is represented by edge pixels and the line fields correspond to the edge pixels. Hence, it is not necessary to have the similarity measure for boundary pixels and thus the "clique potential function" given by (7) shall consist of penalty functions only. Therefore, region formation, with similarity measure, should not be contributed by the boundary pixels.

Constrained Markov Random Field (CMRF) Model
In order to model textures besides the scene the model need to take care local properties of a given pixel. MRF takes care of the spatial neighborhood, however, it is needed to reinforces the dependency of the pixel on the neighborhood. This will give rise to a new a priori model for the label process.
A "discrete-time martingale" is a discrete-time "stochastic process" (i.e., a sequence of random variables 1 2 3 , , , X X X  ) which satisfies for all n, Assuming that Z is a "Markov process", one obtains, Since Z is a MRF, , , , e Considering first order neighborhood and choosing one of the neighborhood pixels for example

Constrained Compound (CCOMRF) Model
The constrained model proposed in the previous section need to be used with the COMRF taking care of the spatial and temporal interaction. Thus the constrained compound MRF model proposed by [34] [35] has the following energy function. The following model corresponds to only the constrained neighborhood in the spatial framework.
where sc U denote the energy function corresponding to "intra color plane in- where , The "local reinforcement" is also extended to the inter-color-plane interactions and hence introduce the notion of constrained model in the inter-colorplane interactions. Thus, there is one clique potential corresponding to intracolor-plane interactions and another clique potential function corresponding to inter-color-plane interactions. In the line of the constrained model according to (12) is now applied to intra as well as inter color plane processes. The constrained condition is among 1

Unsupervised Framework
In this framework, neither the model parameters nor the image labels are assumed to be known. Both the estimate of the model parameters and image labels are interdependent. Therefore, in an unsupervised scheme, the MAP estimates of the labels and the estimates of the model parameters are carried out concurrently.
Thus, an estimation strategy need to be developed which, using the observed image X, will yield an optimal pair ( ) , opt opt Z θ . Towards this end, the following joint optimality criterion is considered, The pair ( ) , opt opt Z θ estimated using (19) is the global optimal estimates. But the image labels Z and the model parameter θ are unknown initially and they are interdependent, thus compounding the problem to be very hard. In order to handle this situation, the problem can be reformulated to achieve sub optimal solutions instead of optimal ones. It may be noted, this function (19) is maximized with parameter Z and θ . The interdependency of parameters that makes the problem intractable can be handled using the notion of parameter spliting proposed by Wendell and Horter [36] in deterministic framework. The approach suggested by Wendell and Horter [36] yields suboptimal solution instead of optimal solution. Their approach is to split the parameter set into two sets and estimate the parameter recursively and it has been shown that this recursive estimation eventually leads to partial optimal solutions. Since our formulation is in stochastic framework, the same notion is adhered to and it is attempted to split the above problem into two separate problems of estimating labels Z and paramters θ separately. This can be expressed as follows. always local optimal solutions [36]. But with * θ θ = , the estimate * z is global optimal satisfying Equation (20) and analogously for * z z = , * θ is global optimal satisfying Equation (21). Since neither * θ nor * z is known, a recursive scheme is adopted where the model parameter estimation and segmentation is alternated. Let at the k th iteration  be the estimate of model parameters and k z be the estimate of the labels of the observed image. Since, both * Z and * θ are unknown, a recursive scheme has been proposed to estimate the model parameters θ and image labels Z recursively, The image labels

Image Label Estimation
The segmentation problem is formulated as a pixel labelling problem, where each pixel can be assigned a label from the set of lebels 0-L. All our labels are defined over an image of size S M N = × . Let every pixel (i, j) is modeled as a random variable denoted as where, θ denotes the estimates of the associated parameter vector of the MRF model and 1 k z + denotes the estimates of the labels. Since z is unknown, (24) Since the observed image X has been provided, the denominator in (25) i.e.
( ) | k P X x θ = becomes a constant quantity. ( ) P Z z = is the a priori probability distribution of the labels. The degradation process is assumed to be a Gaussian process, denoted by W and the corresponding realization is w. Hence ( ) | , k P X x Z z θ = = of (25) can be written as ( ) ( ) ( ) Since, W is a Gaussian process, and there are three spectral components present in a color image, one obtains, (26) where K is the covariance matrix. Hence, this minimization can be expressed as, The color image has three spectral components k x , k z , V is the clique potential function for all the three spectral components.

Model Parameter Estimation
Using the "ground truth" image z, the a priori model parameters are estimated. The associated MRF parameters of this "ground truth" image is θ . Therefore, the problem can be stated as the follows ( ) Since Z is a MRF, we have, where ξ ranges over all realizations of the image z. Because of the denominator of (29), computation of the joint probability ( ) is extremely difficult task. Here, the "pseudolikelihood function" is maximized ( ) From the definition of "marginal conditional probability", it can be written as, Because of MRF assumption, Substituting Equation (32) in (30), the following is obtained.
So, the problem of maximization (34) reduces to ( ) In (34), the summation is taken over all possible labels M. (34) is "highly nonlinear" in nature and no "a priori knowledge" of the solution is available. As θ is the parameter vector [ ] , , , θ α β σ γ ∈ , (34) reduces to a set of complex non-linear equations ( ) f θ . Since, no "a priori knowledge" about the initial guess for determining the solution is available, we have developed a "globally convergent homotopy map" to find out the solution starting from an arbitrary initial guess.

Journal of Computer and Communications
It is very difficult to solve the resulting non-linear equations and therefore a globally convergent based "Homotopy Continuation method" is developed. The homotopy curve is shown in Figure 3.

Simulation
A wide variety of examples are considered in simulation, but for the sake of illustration of the potentiality of the proposed models and the algorithm, two textured images and three general images have been presented in this paper.

Synthetic Images
Texture images having two and five classes have been considered and are shown in Figure 4 and Figure 5. The proposed Compound Constrained MRF model has the unifying property of modeling texture as well as scene images. In this research, the texture and the scene images are considered to validate this unifying modeling property. The two class synthetic image is shown in Figure 4(a) and the corresponding ground truth image is shown in Figure 4     The next example considered is a five class textured image shown in Figure 5.
The results obtained are shown in Figure 5(c) and Figure 5(g). As observed from Table 2 and Table 3, the CCOMRF model produced five classes and the percentage of misclassification error in Weak Membrane Model and Reward-Punishment model are 4.25 and 3.25 respectively. As seen from Table 3, the percentages of misclassification for MRF and COMRF model were 11.58 and 8.11 respectively. These were much higher than those of the CCOMRF model. As far as all these three models are concerned, the CCOMRF model could efficiently model the textures. In this particular example the result obtained by CCOMRF model is comparable to those of JSEG and Kato's methods.

Real Images
The next image to be considered for experiment is the Red-house image shown in Figure 6(a) and this image has both textural background with other scene objects. The unifying property of our proposed Constrained Compound model       Tables 1-3 respectively. The fourth example considered was the crow image from the Berkeley data base and this image has non-uniform lighting conditions. This is shown in Figure 8(a). The corresponding ground truth, manually constructed, is shown in Figure 8 image; (f) combined iteration of lambda for "water-boat" image.
the homotopy parameter λ and varies from 0 to 1. The value at zero corresponds to the arbitrary starting point and the value at 1 λ = corresponds to the solution of unknown function and hence MCPL estimates. Similarly Figure 9(b) and Figure 9(c) corresponds to the MCPL estimates obtained at 1 λ = . These values have been used to obtain the image label estimates by the hybrid algorithm.
Thus, from our simulation results, it has been concluded that the proposed CCOMRF model with the reward punishment model performed well as compared to others. It has also been demonstrated that this model possess the unifying property of modeling texture and non textured objects in the scene as well.

Conclusion
In this work, an unsupervised color image segmentation algoritm is proposed with two new image models such as COMRF and CCOMRF models. Here, image segmentation is viewed as the problem of recovering a "true" image consisting of a few "homogeneous regions" from a noisy image by labeling individual pixels according to region type. The proposed CCOMRF model is found to have the unifying property of modeling scene and texture image as well. The proposed compound MRF model has the potentiality of modeling color with the notion of controlled correlations. The model parameters have been estimated by the proposed "Homotopy Continuation method". It has been found that CCOMRF model produced better results visually and numerically, than those of other models. Further this model was found to possess "unifying property" of modeling scenes as well as texture images. The only parameter that was selected on trial and error basis was σ , the degradation parameter. Currently, attempts are made to reformulate the problem to estimate σ with all other associated model parameters.