Abstract:To improve the transmission capacity and performance of optical transimission system, a mode division multiplexing system based on highdimensional modulation format was proposed. And the transmission performance of 4b4D and 8b8D signals with different optical SNR (OSNR), transmission distance and mode coupling factor was simulated, and also it was compared with that of 2D modulated signal QPSK. The simulation results show that, at the same BER, 4b4D and 8b8D signals yield around 2 and 4dB OSNR gain, and 20 and 40km transimission distance increment with respect to QPSK respectively. Besides, the effect of mode coupling also decreases with the increase of signal dimension.