To solve the problems of mismatching and signal integrity caused by the via holes in differential traces in high speed optical transceiver, HFSS and Hyperlynx were used to make modeling research and analysis on the differential via holes. On this basis, two methods of differential compensation and adding GND via holes were proposed to realize the matching of differential traces and improving the integrity of signal. Subsequently, the performance of the two methods on transmission loss and reflection was verified by HFSS simulations, and then the characteristics of the two methods were analyzed and compared. Finally, tests were carried out on 2.5G optical transceiver and the results prove the effectiveness of the proposed methods.