What Information Theory could and could not possibly contribute to Machine Learning and Artificial Intelligence?

A promising application of Information Theory (IT) to Machine Learning (ML) is twofold:  We can (i) derive limit bounds and (ii) prove existence of the limits in ML problem. While the core problems in IT is data compression, data transmission and inference, ML problems can be interpreted in terms of IT problems, therefore, potentially make use of the fundamental limits in IT for proving bounds and existence in ML.

Also note that while IT is promising to provide bounds in ML problems, search for an optimal yet practical solution under such bounds is probably out of the scope of IT which unfortunately a majority of ML research is devoted to as well. Even with such disability of IT application to ML, IT provides an information-theoretic interpretation of ML problems, besides the limits abound and achievablity mentioned in the first paragraph, which in turn help redesign the learning and inference process in ML problems (e.g., [2]). 

References 

[1] T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley, 1991.
[2] T.T. Nguyen, and J. Choi. Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck. In arXiv 1712.01272,