Recently, sophisticated segmentation techniques, such as level set method, which using valid numerical calculation methods to process the evolution of the curve by solving linear or nonlinear elliptic equations to divide the image availably, has become being more popular and effective. In Local Binary Fitting (LBF) algorithm, a simple contour is initialized in an image and then the steepest-descent algorithm is employed to constrain it to minimize the fitting energy functional. Hence, the initial position of the contour is difficult or impossible to be well chosen for the final performance. To overcoming this drawback, this work treats the energy fitting problem as a meta-heuristic optimization algorithm and imports a varietal particle swarm optimization (PSO) method into the inner optimization process. The experimental results of segmentations on medical images show that the proposed method is not only effective to both simple and complex medical images with adequate stochastic effects, but also shows the accuracy and high efficiency. © 2017 Universitas Ahmad Dahlan.