The technique of image mosaic may cause ghost phenomenon when it processes the image with moving objects. An adaptive algorithm was proposed to remove ghosting based on optimizing previous SURF algorithm. The enhanced image was used to replace the original gray image processed by SURF, thus the feature points were extracted. Then, it locates the moving objects in the overlap region of two images using frame differential method. And the improved weighted fusion algorithm was used to calculate the weighted coefficient so as to increase the weight difference. Experimental results show that this algorithm can solve the interference problem produced by moving objects, thus compared to the algorithm of using the optimal seam based on multi-frequency fusion, the proposed method can obtain better fusion quality and highher image definition.