For 40Gb/s DQPSK optical communication transmission system, an information initial formula of decoding algorithm for low-density parity-check (LDPC) codes is proposed, and the decoding algorithm based on belief propagation (BP) is improved. Furthermore, the numerical simulation and the performance analysis and comparison are implemented. Then the advanced decoding algorithm which has a good tradeoff between performance and complexity is obtained.