In machine-vision-based systems for detecting foreign fibers, due to the background of the cotton layer has the absolute advantage in the whole image, while the foreign fiber only account for a very small part, and what’s more, the brightness and contrast of the image are all poor. Using the traditional image segmentation method, the segmentation results are very poor. By adopting the maximum entropy and genetic algorithm, the maximum entropy function was used as the fitness function of genetic algorithm. Through continuous optimization, the optimal segmentation threshold is determined. Experimental results prove that the image segmentation of this paper not only fast and accurate, but also has strong adaptability.