Information theory

Information theory is the mathematical study of the quantification, storage, and communication of a particular type of mathematically defined information. The field was established and formalized by Claude Shannon in the 1940s, though early contributions were made in the 1920s through the works of Harry Nyquist and Ralph Hartley. Information theory was initially formed in the context of telecommunication but soon found a wide range of other applications. It is now at the intersection of mathematics, statistics and computer science, and has applications in diverse fields ranging from electrical engineering and physics to neurobiology. As a simple example of the concept, if one flips a fair coin and does not yet know the outcome (heads or tails), then one lacks a certain amount of information. After looking at the coin, one gains information about the outcome. For a fair coin, the probability of either heads or tails is 1/2 and the amount of information is expressed as − log 2 ⁡ ( 1 / 2 ) {\displaystyle -\log _{2}(1/2)} = 1 bit of information. A key concept in information theory is entropy. In Shannon's formulation, entropy is equal to the lack of information about an event. In the above coin flip example, the entropy in the case where you don't know the outcome is 1 bit. When you know the outcome after the coin has landed, the entropy is zero because you have gained one bit. Information theory has been used in a wide range of applications, such as source coding/data compression (e.g. for ZIP files), and channel coding/error detection and correction (e.g. for DSL). Its impact has been crucial to the success of the Voyager missions to deep space, the invention of the compact disc, the feasibility of mobile phones and the development of the Internet and artificial intelligence....