A novel constellation mapping scheme of 16 quadrature amplitude modulation(QAM) based on the criterion of the mutual information and the unequal degree protection of the bit position is proposed so as to improve the limited channel capacity performance of the traditional Gray mapping for the bit-interleaved-coded-modulation with iterative decoding(BICM-ID) optical communication systems at the medium-to-high signal noise ratio(SNR) region.The simulation analysis shows that the channel capacity of the novel proposed mapping scheme is 1.18bit/channel higher than that of the traditional Gray mapping scheme at the medium-to-high SNR region.Furthermore,combined with the low-density parity-check(LDPC) code,the novel proposed mapping scheme with the assistance of the extrinsic information transfer(EXIT) charts owns good performance advantage and application promise in optical communication systems.