Segmentation of Sinusoids in Hematoxylin and Eosin Stained Liver Specimens Using an Orientation-Selective Filter

The liver comprises cell layers of hepatocytes called trabeculae, which are separated by vascular sinusoids. Understanding the structure of hepatic trabeculae and liver sinusoids in hematoxylin and eosin (HE)-stained liver specimens is important for the differential diagnosis of liver diseases. In this study, we develop an approach to extracting liver sinusoids from HE-stained images. The proposed approach involves: 1) a new orientation-selective filter (OS filter) for edge enhancement and image denoising, 2) the clustering of image pixels to identify candidate sinusoids, and 3) a classification procedure that discards unlikely candidates and selects the final sinusoid areas. Experimental studies using a database of 16 images with a resolution of 512 × 512 pixels showed that the proposed approach could segment liver sinusoid pixels with 81% of specificity and 94% of sensitivity. A comparison with a method based on bilateral filters showed that this method improved the sensitivity for all images with an average improvement of 4% and no difference in specificity. The results were presented to a group of pathologists and they confirmed that the images were highly representative of the tissue morphology features.


Introduction
During conventional histopathology, samples of abnormal tissues are removed from body, placed in a fixative to prevent decay, and stained with dyes to highlight certain features of interest.Later, the highlighted features are inspected by pathologists using microscopes.The goal of visual inspection is to identify the manifestations of disease, such as cell atypia and malignant neoplasm.The results of these subjective inspections are used to identify appropriate therapeutic procedures.
Recent advances in whole-slide imaging technology mean that high resolution images of tissues can be acquired using specialized hardware and these images are used at conferences, for educational purposes, and in telepathology.Computer-aided diagnosis is also expected to make a major contribution to the quantification of digital images.
In recent years, there have been many reports of pathological image analysis methods.Many of these previous studies were focused on the automatic grading of gastric cancer [1] and prostate cancer [2][3][4].In the area of pathological image segmentation, a cell nucleus extraction competition is held each year at the International Conference on Pattern Recognition (ICPR) [5].However, there have been few reports of the automatic grading of hepatocellular carcinoma (HCC).In [6] and [7], automatic grading was conducted according to the Edmondson classification, where results were reported with good precision based on characteristic parameters related to the cell nucleus.The histopathological features of cell nuclei have been used in most conventional image analysis methods.However, other characteristics are also important for the diagnosis of HCC, including the morphological features of tissues, such as sinusoids and hepatic trabeculae; cellular features such as the cytoplasm and nuclei; as well as lymphocytes and red blood cells [8].Thus, the more accurate evaluation of hepatic lesions demands the extraction and quantification of structures such as trabeculae and fibrous cells.In the present study, we developed a method for extracting sinusoids, which is a basic approach to structure recognition.
Hepatic cells are normally arranged in a radial pattern that originates from a central vein.This radial structure is called a hepatic trabecula.Any irregularities in the hepatic trabeculae are important for the histological classification of HCC.Trabeculae are normally made of a single row of cells, but sometimes they exhibit increased thickness and the formation of multiple rows of cells during the development of structural atypia in carcinoma.The specific mixture of thick and thin hepatic trabeculae indicates the stage of progression in cancer.
In the present study, we attempted to extract sinusoidal regions.Sinusoids run parallel to the trabecula and they provide nutrition to cells.Therefore, analyzing the structure of sinusoids is crucial for the extraction of morphological features from hepatic trabeculae.The direct extraction of morphological features from hepatic trabeculae is a very difficult task because the trabecula structure includes various objects such as the cell nucleus, cytoplasm, and cell membrane.

Liver Sinusoids
As shown in Figure 1, liver tissues mainly comprise a hepatic lobule and Glisson's sheath.The lobule contains hepatic cells, sinusoids, and blood capillaries.Most of the hepatic cells are arranged in trabeculae and the blood capillaries run between the cells.The sinusoidal wall between the sinusoids and hepatic cells comprises endothelial cells.The boundary region is relatively distinct, but substances are exchanged between hepatic cells, as well as sinusoidal blood through the holes or gaps between endothelial cells.The sinusoids have a lumen structure and hematoxylin and eosin (HE)-stained sinusoid specimens are only weakly colored.However, it does not form a full cavity because it contains secretions, blood, and endothelial cells.In some cases, the sinusoids are squeezed and their lumens are invisible in the corresponding pathological specimen.In particular, the lumen is often obscure in moderately or poorly differentiated lesions.In the present study, we extracted sinusoids with a lumen structure [9].

Methodology
In general, sinusoids appear white in color because they are not stained by hematoxylin or eosin, but they still include endothelial cells, red blood cells, bodily secretions, and other substances.The boundary between the sinusoid and the hepatic cytoplasm is sometimes obscure and a newly developed orientation-selective (OS) filter is used as a pretreatment to highlight the structures in the sinusoid and the cytoplasm.The flowchart shown in Figure 2 illustrates our methodology.After enhancing the margins of sinusoids using the OS filter, the candidate sinusoidal regions are extracted by the expectationmaximization algorithm (EM algorithm).Next, we calculate the histogram features of the candidate regions and classify them using a previously trained linear support vector machine (SVM).

Orientation-Selective Filter
The sinusoid wall cells, such as the endothelial cells on the boundary between the sinusoid and the hepatic cells, appear to be arranged in smooth curves but they have interruptions and voids, as described above.Therefore, if a sinusoid region is divided along this boundary during image processing, it will be difficult to determine the correct boundary between the sinusoid and the hepatic cell in the interrupted section.Thus, an OS filter is used to connect the interrupted section to the interrupted boundary in a smooth manner, thereby defining the borderline.Another possible method for image segmentation is a bilateral filter [10], which is a powerful method for smoothing while preserving edges.However, it has no effect on connecting broken boundaries.A suitable filter should smooth the boundary in the direction of the boundary, while retaining the edges in the direction perpendicular to the boundary.Thus, an OS filter was designed specifically for this purpose because no other filters have this functionality.
The proposed OS filter is a selective version of a bar filter [11] that performs an affine transformation in the orientation of the brightness gradient.If an original image is assumed to be I(x,y), the brightness gradient ori- entation of each pixel is determined using Equation (1): tan ,   1 2 14, 8 , 0 other In the present study, a filter was used with Gaussian smoothing of 11 × 1 lines, where NS is the image size.The length of the filter is determined by the actual gaps being connected.In the present study, the filter was designed so it was two pixels shorter on either side compared to the size used to calculate the intensity gradient.If the filter is too large, intensity information other than that related to the sinusoid boundary would be included in the calculation.If there are two or more gradient orientations, these gradient orientations are calculated and integrated.The convolution-like operation in Equation ( 3) is performed using all of the RGB color channels.
where θ(x,y) is the gradient orientation and m(x,y) is the gradient magnitude.The pixel values in the green channel of the RGB color space are used because the difference between the sinusoids and cytoplasm is relatively high [12].Computing the orientation of a single pixel using only its four-connected neighborhood might be unstable, so we compute the pixel-wise orientation values in three steps.First, we calculate the gradient of each pixel using centered 1D point derivatives with [−1, 0, +1] masks.Next, we quantize the gradient magnitudes into 18 equally spaced values between 0 and 170.Finally, we calculate the weighted gradient orientation histogram around each pixel in a neighborhood of 15 × 15 pixels.The size of the window is determined by the size of the object for which the intensity gradient is obtained.In the present study, this was slightly larger than the thickness of the sinusoid.The weights are the gradient magnitudes.The largest value in the weighted histogram is assigned to the center pixel.If multiple peaks in the histogram exceed 80% of the maximum, they are all assumed to be valid.This protects the structures of the edge crossings.This threshold is a parameter and when it is high, it gives greater priority to local edge enhancement, but more protection to edge crossings when it is low.Next, a bar filter is affine-transformed in the gradient orientation of each pixel and applied using Equation (3).This filter is similar to convolution, but the kernel depends on the weighted gradient orientation (θ) histogram described above.

Effectiveness of the Orientation-Selective Filter
The filter used in the present study changes its convolution kernel using spatial information, so the effects are also regarded as varying with the image properties.Thus, we examined whether the filter properties were suitable for sinusoid extraction from the hepatic histopathological tissue images considered in this study.First, the results of filtering are shown in shows the variation in a sinusoid where the gray level was obscure or a segment where favorable results were obtained compared with the original image (Figure 4(a)).Figure 4(c) shows the results obtained with the bilateral filter compared with the OS filter, where the cytoplasm around the sinusoid is clear, although the boundary between the sinusoid and the cytoplasm is obscure.In the  results obtained after the application of the OS filter (Figure 4(b)), the boundary with the clear cytoplasm exhibits deeper pink eosin staining compared with the surrounding cytoplasm while the boundary between the sinusoid and the hepatic cell is more distinct because the sinusoid's interior secretion is whiter as a consequence of smoothing.In addition, information related to the variations in concentration from the sinusoid to the cytoplasm is conserved.

Extraction of Candidate Sinusoidal Regions
HE-stained specimens mainly contain cell nuclei that appear blue due to hematoxylin, cytoplasm that appears red due to eosin, and sinusoids that appear white due to a lack of chemical reaction.We assumed that the signal intensities of the nuclei, cytoplasm, and sinusoids had Gaussian distributions in the RGB color space and we used the EM algorithm to estimate the corresponding class means and variances [13].

Clustering with the EM Algorithm
The EM algorithm is a well-established method that is often used often to estimate the parameters for mixed distribution models.In the present study, a mixed normal distribution was applied in the RGB color space to estimate the white region.An RGB feature vector is defined as, where N is the number of pixels in the image.The Gaussian mixture distribution can be defined using Equation (4): where k  is the average, k  is the covariance matrix, k is the distribution weight, and F is the number of classes.In this case, the maximum likelihood estimate is represented by Equation (5).
The estimation process involves an expectation step and a maximization step.The expectation step is represented by Equation ( 6) using Bayes' rule, where i is the pixel number and k is the class number of each distribution.
The maximization step is represented using Equation (7). , In a test conducted as part of this study, an initial value was determined using the k-means method.Sinusoids are basically white regions so they should be extracted if an appropriate threshold value can be determined.HEstained images have color densities that vary among images, however, which means that it is necessary to determine the threshold as a relative value.In the present study, the EM algorithm was used to determine the relative threshold with good precision.

Eliminating Cell Nuclei from Sinusoids
Endothelial cell nuclei and blood cells are present within the extracted sinusoids.Thus, it is necessary to eliminate them before determining the structures of sinusoids.The cell nuclei are often isolated in the interiors of the sinusoids because they do not include cytoplasm.Thus, it is necessary to eliminate all of the isolated areas within the sinusoidal regions.The interior sinusoids are labeled and small regions less than or equal to a given size are treated as sinusoids.

Discarding Unlikely Sinusoid Candidates
Images of hepatic tissue contain various white structures that resemble sinusoids.Examples of these structures are cells undergoing ballooning degeneration, clear cells, and fat droplets.Ballooning degeneration is a form of cell death, which is characterized by a thin cell cytoplasm.Clear cells indicate a specific type of carcinoma in the liver and these cells lack most of their cytoplasmic content, thereby leaving nuclei surrounded by large white areas.Fat droplets indicate liver disease and they can be observed as round white areas in HE-stained specimens.
Although the causes and underlying mechanisms of these structures are different, their appearance is quite similar to sinusoids.Therefore, a procedure that labels all of the white regions as sinusoids may produce erroneous results.We propose a two-step approach to correct these errors and to improve the specificity of the sinusoid segmentation procedure.After the first step described above, the second step involves the classification of sinusoids and sinusoid-like structures using a supervised learning approach, which is explained in the following subsections.

Extracting Gray-Level Histogram Features
The contrast between sinusoids and tissue cytoplasm is apparent in the middle regions of the visible spectrum because the absorption spectrum of eosin has a peak in that region.Therefore, given an 8-bit RGB image I_rgb, we initially select the green channel of the image and extract four different features from the gray-level histogram, i.e., the mean, variance, skewness, and kurtosis.The contrast between the sinusoid and cytoplasm needs to be enhanced to extract the sinusoids.HE staining dyes the cytoplasm with eosin and the nucleus with hematoxylin.A sinusoid contains few cellular tissues and its properties are similar to glass areas, which generate the results for hematoxylin absorption spectrum shown in Figure 5. Figure 5 shows the absorbance levels of the hematoxylin and eosin simple stains, respectively.There are major differences between the absorption spectra of the sinusoid (glass) and eosin (cytoplasm) at 475 -580 nm.Thus, using the green channel helps to distinguish eosin sufficiently well to obtain dark areas, which enhances the contrast compared with the sinusoid.This is why the green channel is used in this method.If we assume that the set represents the pixel values of the Green channel between 0 and 255 nm, n i represents the number of pixels in the gray level i, and N represents the total number of pixels in the image, then the probability of a pixel occurring at level i is as follows (Equation ( 8)).

 
Open Access OJMI As shown in Figure 6, ballooning regions exhibit a smooth variation in the pink color between the interior and exterior regions.During experiments, we dilated the initial sinusoid candidates five times using a 3 × 3 structuring element and we extracted separate features for pixels in the original region and pixels in the expanded region.Figure 6(b) shows the gray-level histogram features derived from both regions inside the red and green lines.Therefore, the number of features was eight.A linear SVM was used to reduce the number of candidate sinusoid regions based on the gray-level histogram features [14,15].The classifier was trained using 300 sinusoids and 300 ballooning regions, each of which was extracted from another set of HE-stained liver images obtained at a magnification of 20×.soid and hepatic cell regions are indicated as white and black, respectively.The results were shown to a group of pathologists and the expert assessments indicated that the images obtained were good representations of the tissue morphology features in benign Edmondson grade 1 (well differentiated) and grade 2 (moderately differentiated) tissues.However, non-sinusoidal regions were still extracted, such as fatty cells and portal veins.In future work, we will aim to eliminate these errors.The evaluation was not performed for grades 3 and 4 because the appearances of the sinusoids are completely different in these grades.

Materials 5.3. Evaluation of the Extraction Accuracy
Whole-slide images of surgical liver specimens were captured using a 20 microscope objective (Nano Zoomer 2.0; pixel width = 460 nm) and 967 different regions-ofinterest (ROIs) with dimensions of 1 × 1 mm were selected by a pathologist for further analysis.A pathologist classified the images according to Edmondson grading [16].Of the 967 ROIs, 551 were labeled as background, 80 as grade 1, 247 as grade 2, 64 as grade 3, and 25 as grade 4.
The sinusoidal regions extracted using the proposed approach were compared with the manually extracted sinusoids to determine the accuracy of the automated extraction method.

Test Images Used in the Experiment
To compare the accuracy of the extraction results, the manually extracted sinusoidal regions were used in this study.Sixteen representative images were selected to assess their staining and ballooning features.In HEstained images, the densities of hematoxylin and eosin vary according to the state of fixation and staining.The

Experimental Results
Figure 7 shows some of the results obtained from the automatically segmented sinusoid images, where the sinu- cytoplasm also has a clear appearance in chronic liver disease cells because of glycogen.The manually extracted results are shown in Figure 8 and the results obtained using the proposed method are shown in Figure 9.

Comparison of the Methods
We used three metrics to evaluate the pixel-wise segmentation accuracy, i.e., the sensitivity, specificity, and overlap, which were determined based on the relationships shown in Table 1.The sensitivity was calculated as TP/(TP + FN), the specificity as TN/(TN + FP), and the overlap as TP/(TP + FP + FN), where TP is the number of positive cases of sinusoids that were identified correctly, TN is the number of negative cases of sinusoids that were identified correctly, FP is the number of positive cases of sinusoids that were classified incorrectly, and FN is the number of negative cases of sinusoids that were classified incorrectly.

Experimental Results
The evaluation results are shown in Table 2. Using the proposed approach, the average sensitivity was 81% and the specificity was 94% with the 16 test images.To verify the effectiveness of the filtering and candidate reduction procedures, we compared the use of these treatments and direct clustering with the original images.
Our results showed that filtering increased the sensitivity by 5% and the specificity by 2%.In particular, we observed the effective acquisition of continuous, smooth results on the boundary of the sinusoid (Figure 10(d)).The effects of feature-based candidate reduction on the sensitivity and specificity were as follows.The sensitivity did not decline because the incorrect deletion of regions did not occur in the experiment, even after narrowing.The specificity was 2% higher for the results that included incorrectly extracted regions.There was 3% higher specificity compared with that before narrowing.Thus, narrowing based on the gray-level histogram features provided favorable results, because it increased the specificity by 3% without decreasing the sensitivity.

Comparison with Other Methods
Image processing using bilateral filters is widely recognized as a data smoothing technique that considers edges.Bilateral filters are used for preprocessing during pathological image segmentation [4].Thus, we compared the proposed method with a bilateral filter.The results obtained with the bilateral filter were based on the extraction of sinusoids with the EM algorithm, which replaced the OS filter during preprocessing.The parameters of the bilateral filter were standard deviation values of σ 1 = 3 for the geometric spread, σ 2 = 0.1 for the photometric spread, and the window size = 10 pixels, which supported the identification of edges that were sufficiently solid to maintain useful images.Table 3 shows the evaluation results based on the area ratios.The ground truth data were the results obtained by manual extraction, which were the same as those used in the experimental precision evaluation.Seven images with good color conditions were used in the experiment where the image size was 2174 × 2174 pixels.Furthermore, the area ratios were calculated by weighting the vicinity of the edges because there were differences among individuals in the manual extraction results.Thus, the vicinity edges were counted as 0.5 pixels.The voids in the fibrous regions were not considered during this procedure, so fibrous regions were excluded from the calculations.The results showed that the proposed method improved the sensitivity for all images with an average improvement of 4% while there was no major difference in the specificity.There was a 1% decrease in the specificity with the proposed method for some images compared with the bilateral filter.We also performed an experiment where a SVM was used to eliminate unlikely sinusoid candidates.The results showed that there was a 1% -3% improvement in the specificity with images that had rather clear photographic conditions compared with the the OS filter and using a SVM to eliminate unlikely candidates.The green pixels indicate negative results that were classified correctly by the SVM, while the red pixels indicate positive results that were classified incorrectly as negative.The proposed method improved the average overlapping by 2%.

Conclusions
original images.Figure 11 shows images obtained with oids were extracted to facilitate the In this study, sinus structural analysis of hepatic histopathological tissue images.Our results showed that the area ratio was increased by 6% using the proposed pretreatment, i.e., the OS filter, and the boundary of the sinusoid was extracted in a natural manner.Furthermore, narrowing was con-      ducted based on the gray-level histogram features, or primary statistics.Our results showed that this increased the specificity by 6% without any reduction in the sensitivity.Another experiment compared the results obtained using an edge-preserving bilateral filter with the conventional method, which showed that the average sensitivity of the proposed method was 3% higher than that with the bilateral filter, while the difference in the specificity was less than 1%.In addition, the proposed method yielded 2% of higher overlaps.We confirmed that better results were obtained with the proposed method c mpared with the bilateral filter based on e experimental area ratios Tabesh, M. Teverovskiy, H.-Y. Pang, V. P. Kumar, D.
. Saidi, "Multifeature Prostate o th obtained, which was attributable to the effect of linking edges along a boundary.Future improvements will include handling liver biopsy images, as well as the surgical specimens considered in the present study.Liver biopsies involve collecting cells from the liver by tapping a needle into it, which means that the sinusoids tend to be distorted by the compression of cells and they are less visible.Liver biopsy images are very important because they are prepared in greater numbers than surgical specimens.
The proposed method is considered to be effective from a perspective of computer diagnostic support in pathological applications.However, a high level of judgment is required to determine the morphological properties of sinusoids because their diagnosis is based on relative differences compared with normal liver cells.Thus, this method could provide diagnostic support by making quantitative information available to physicians based on the structural analysis of sinusoids.In addition, the structure of sinusoids is important from an image processing perspective.Cells, sinusoids, and stroma are dominant in hepatic pathological images, and the extraction of sinusoids provides important information that facilitates the recognition of other structures, such as the cells and stroma.Furthermore, it may be possible to quantify the thickness and structure of cells by extracting the remaining areas after the sinusoids and stroma have been eliminated.

Figure 2 .
Figure 2. Flowchart of the proposed method: Section 3.1 is the pre-processing stage, Section 4.1 is the clustering process, and Section 4.4 is the discarding process.

Figure 3 .
Figures 3(a) and (b) show the original image and the results obtained by convolution with the OS filter, respectively.Figure 4(b)

Figure 10 (
c) shows the results obtained by extraction using the bilateral filter compared with the proposal method.Figure 10(a) shows the original image and Figure 10(b) shows the results extracted from the original image.

Figure 9 Figure 10 .
Figure 9. Sixteen differen s. he results were extracted using the propo t HE-staine ver specimens, wh ere captured usin 20× objective len sed method.d li ich w g a T in the segmentation outputs with the pixels in the ground truth data shown in Figure 8.The segmentation outputs were obtained using three different approaches: 1) dividing unfiltered images into three clusters; 2) dividing filtered images into three clusters; 3) dividing filtered image into three clusters and discarding unlikely candidates, which were predicted using a support vector machine.The values highlighted with a dark gray background have the highest sensitivities, the values highlighted with a light gray background have the highest specificities, and the values highlighted with a mid-light gray background have the highest overlaps.

FFigure 11 .
Figure 11.Comparison between the images obtained using the orientation-selective filter and with a support vector machine to eliminate unlikely candidates.(a) Result of extraction from filtered image; (b) Result of extraction from discarding and filtered image.

es in o thr e cl ters; 3) dividing OS-f tered hic were predi ted u ing a support vector machin . The with a dark gr s, the ay high bac ighted ground have with the gray ighe t sens tiviti a ligh background have the highest specificitie highlighted with a mid-light gray background have the highest overlap.
E 0.89 0.98 0.84 0.91 0.98 0.86 0.91 0.98 0.86