On signal design by the R/0/ criterion for non-white Gaussian noise channels

The use of the cut-off rate criterion for modulation system design is investigated for channels with non-white Gaussian noise. A signal space representation of the waveform channel is developed, and the cut-off rate for vector channels with additive non-white Gaussian noise and unquantized demodulation is derived. When the signal input to the channel is a continuous random vector, maximization of the cut-off rate with constrained average signal energy leads to a water-filling interpretation of optimal energy distribution in signal space. The necessary condition for a finite signal set to maximize the cut-off rate with constrained energy and an equally likely probability assignment of signal vectors is presented, and an algorithm is outlined for numerically computing the optimum signal set. As an example, the rectangular signal set which has the water-filling average energy distribution and the optimum rectangular set are compared.