What do you think?
Rate this book
Information Theory: A Tutorial Introduction
Originally developed by Claude Shannon in the 1940s, information theory laid the foundations for the digital revolution, and is now an essential tool in telecommunications, genetics, linguistics, brain sciences, and deep space communication. In this richly illustrated book, accessible examples are used to introduce information theory in terms of everyday games like ‘20 questions’ before more advanced topics are explored. Online MatLab and Python computer programs provide hands-on experience of information theory in action, and PowerPoint slides give support for teaching. Written in an informal style, with a comprehensive glossary and tutorial appendices, this text is an ideal primer for novices who wish to learn the essential principles and applications of information theory.
Kindle Edition
First published February 1, 2015
41 people are currently reading
525 people want to read
About the author
James V. Stone
21 books34 followersHonorary Associate Professor, University of Sheffield, England.
Ratings & Reviews
Friends & Following
Create a free account to discover what your friends think of this book!
Community Reviews
Displaying 1 - 29 of 29 reviews
November 18, 2017
“Information is a fundamental physical quantity, obeying exact laws” (Deutsch D and Mareletto C, 2014).
As very aptly stated in this book, information is a fundamental physical quantity, not substantially different in this regard to other "quantities" such as energy or mass, and it can justifiably be viewed as “nature's currency”. The misconceived old belief that information is just an abstract, purely theoretical construct devoid of physical reality and only applicable to specialist fields such as computer sciences, is still unfortunately held by many. Nothing could be further from the truth, as demonstrated by the author in the brilliant last chapter of this gem of a book, where several experimental results in many fields such as neurosciences, genetics and evolutionary theory (just to name a few) clearly point to the fundamental nature of information in the physical world.
It is unfortunate and disappointing that, while on one hand many popular science books amply treat subjects such as relativity and quantum mechanics, on the other hand few publications, where the critical relevance of the fundamental and fascinating subject of information theory is rendered proper justice, are made available to the general public. As a result, while virtually everybody is relatively familiar with the likes of Maxwell, Heisenberg or Einstein, not many appreciate the genius and important contributions of Shannon: without him, we would probably still be treating information as if it were some ill-defined abstract entity with a place only in some form of philosophical “idealism”, rather than a well-defined, and experimentally relevant, physical aspect of reality.
The work by this author is a refreshing and remarkable example of a non-specialist book that nicely fills this gap, delivering a tutorial and highly informative style of writing; it provides a relatively accessible but comprehensive and meaningful treatment of the main elements of this important discipline that represents a field of inquiry extremely relevant to the understanding of the physical world, as discernible in many areas of modern science.
Just as an example, and as explained by the author, evolution is essentially “a process in which natural selection acts as a mechanism for transferring information from the environment to the collective genome of a species”. The use of information theory in evolutionary biology is a promising and mainstream area of scientific research.
Shifting to modern neurosciences, information theory shows that the brains are surprisingly efficient biological computing machines capable of processing information at a rate that has been quantitatively estimated and that has been demonstrated to respects its laws and constraints. It is also very interesting to note that, as showed in the book, a quick quantitative informational analysis of the human brain can clearly show how its actual structural complexity at birth, being limited by the information stored in the human genome, is actually significant lower (by several orders of magnitude) than the number of existing synapses - the big difference being explained by the process of learning (simulating higher cognitive functions in the human brain does not therefore have to necessarily recreate the full complexity of the network of synapses, but it is sufficient to design something that has the inherent capability, such as implemented in the neural network technology, to receive information and adaptively “learn” from such process).
The author (who is a well known practitioner in the field, and a Reader in Computational Neurosciences at the University of Sheffield) also brilliantly explains some concepts that are not always properly treated in a book of an introductory nature, like the close and fundamental link between the information theoretical concept of entropy and its counterpart in the realm of thermodynamics; more sophisticates outcomes such as the famous “Landauer limit” (the absolute lower limit to the amount of energy required to acquire or transmit one bit of information) are also succinctly but brilliantly treated by the author: the demonstrated fact that, no matter how efficient any physical device is (such as a computer or a brain), it can only acquire one bit of information only if it expends at least 0.693kT joules is energy, has deep significance, and it makes the “physicality” of information all the more compelling. In fact, if this was not the case, then we would be able to use the notorious “Maxwell's demon” trick to fuel power stations with no net energetic cost – would not that be nice.. :-).
In more general terms, this book is an absolutely riveting and lucid introduction to this fascinating subject; many important concepts are well explained, and the treatment of such concepts is frequently supported by concrete and effective examples, intuitive diagrams (the schematic representation of mutual information through Venn diagrams is very helpful, even considering that it does not tell the full story), by text boxes summarizing the main points, and even by a comprehensive glossary of technical terms - all enormously facilitating the process of achieving a deeper intuitive understanding of the subjects being treated.
Some aspects of a more technical nature (such as the problem of divergence in case of continuous variables, and the consequent introduction of the concept of differential entropy) are also treated very well, with precision and conciseness. Topics such coding efficiency, the source coding theorem, the noisy channel theorem, conditional entropy and mutual information are also treated with clarity and precision, and with minimal hand-waving.
Referring now to my own personal experience in reading this book, I must say that I achieved my main learning objectives, and I greatly enjoyed reading it. In fact, the last time I seriously studied this subject was at Uni, so I wanted a good “refresher” that would give me a succinct but comprehensive and meaningful treatment of the main elements of this subject: this book was a perfect fit.
The typos are minor, and easy to identify; the section on further readings is extremely helpful, the bibliography is not huge but comprehensive enough, and there is even a very handy list of some key equations in information theory at the end of the book. Being this a non-specialist book, the proofs of some of the major theorems (some of which are admittedly quite curly and would require higher mathematics), are necessarily only explained at high, conceptual level – this is completely understandable given the pedagogical aims of this book, but this left me a bit dissatisfied and wanting for more, to be perfectly honest. I would also have loved some reference to the current research programs and applications of Shannon's information theory in applied mathematics and theoretical physics, including complexity theory, quantum information theory and even cosmology, most areas of application of which I know little and about which I would have loved at least some bibliographic references.
Anyway, this book is perfect as a serious introduction to the subject (providing more depth and rigour than a popular science publication usually would), highly recommended to all readers interested in this beautiful subject, and who want to achieve a more rounded understanding of physical reality; only basic background knowledge of probability/statistics and high school mathematics is required, and no prior knowledge of information theory is necessary.
A well deserving 5 star rating to this great introduction. A very enjoyable read, and the author does manage to convey the beauty and great relevance of this fascinating, and too often under-appreciated, subject.
As very aptly stated in this book, information is a fundamental physical quantity, not substantially different in this regard to other "quantities" such as energy or mass, and it can justifiably be viewed as “nature's currency”. The misconceived old belief that information is just an abstract, purely theoretical construct devoid of physical reality and only applicable to specialist fields such as computer sciences, is still unfortunately held by many. Nothing could be further from the truth, as demonstrated by the author in the brilliant last chapter of this gem of a book, where several experimental results in many fields such as neurosciences, genetics and evolutionary theory (just to name a few) clearly point to the fundamental nature of information in the physical world.
It is unfortunate and disappointing that, while on one hand many popular science books amply treat subjects such as relativity and quantum mechanics, on the other hand few publications, where the critical relevance of the fundamental and fascinating subject of information theory is rendered proper justice, are made available to the general public. As a result, while virtually everybody is relatively familiar with the likes of Maxwell, Heisenberg or Einstein, not many appreciate the genius and important contributions of Shannon: without him, we would probably still be treating information as if it were some ill-defined abstract entity with a place only in some form of philosophical “idealism”, rather than a well-defined, and experimentally relevant, physical aspect of reality.
The work by this author is a refreshing and remarkable example of a non-specialist book that nicely fills this gap, delivering a tutorial and highly informative style of writing; it provides a relatively accessible but comprehensive and meaningful treatment of the main elements of this important discipline that represents a field of inquiry extremely relevant to the understanding of the physical world, as discernible in many areas of modern science.
Just as an example, and as explained by the author, evolution is essentially “a process in which natural selection acts as a mechanism for transferring information from the environment to the collective genome of a species”. The use of information theory in evolutionary biology is a promising and mainstream area of scientific research.
Shifting to modern neurosciences, information theory shows that the brains are surprisingly efficient biological computing machines capable of processing information at a rate that has been quantitatively estimated and that has been demonstrated to respects its laws and constraints. It is also very interesting to note that, as showed in the book, a quick quantitative informational analysis of the human brain can clearly show how its actual structural complexity at birth, being limited by the information stored in the human genome, is actually significant lower (by several orders of magnitude) than the number of existing synapses - the big difference being explained by the process of learning (simulating higher cognitive functions in the human brain does not therefore have to necessarily recreate the full complexity of the network of synapses, but it is sufficient to design something that has the inherent capability, such as implemented in the neural network technology, to receive information and adaptively “learn” from such process).
The author (who is a well known practitioner in the field, and a Reader in Computational Neurosciences at the University of Sheffield) also brilliantly explains some concepts that are not always properly treated in a book of an introductory nature, like the close and fundamental link between the information theoretical concept of entropy and its counterpart in the realm of thermodynamics; more sophisticates outcomes such as the famous “Landauer limit” (the absolute lower limit to the amount of energy required to acquire or transmit one bit of information) are also succinctly but brilliantly treated by the author: the demonstrated fact that, no matter how efficient any physical device is (such as a computer or a brain), it can only acquire one bit of information only if it expends at least 0.693kT joules is energy, has deep significance, and it makes the “physicality” of information all the more compelling. In fact, if this was not the case, then we would be able to use the notorious “Maxwell's demon” trick to fuel power stations with no net energetic cost – would not that be nice.. :-).
In more general terms, this book is an absolutely riveting and lucid introduction to this fascinating subject; many important concepts are well explained, and the treatment of such concepts is frequently supported by concrete and effective examples, intuitive diagrams (the schematic representation of mutual information through Venn diagrams is very helpful, even considering that it does not tell the full story), by text boxes summarizing the main points, and even by a comprehensive glossary of technical terms - all enormously facilitating the process of achieving a deeper intuitive understanding of the subjects being treated.
Some aspects of a more technical nature (such as the problem of divergence in case of continuous variables, and the consequent introduction of the concept of differential entropy) are also treated very well, with precision and conciseness. Topics such coding efficiency, the source coding theorem, the noisy channel theorem, conditional entropy and mutual information are also treated with clarity and precision, and with minimal hand-waving.
Referring now to my own personal experience in reading this book, I must say that I achieved my main learning objectives, and I greatly enjoyed reading it. In fact, the last time I seriously studied this subject was at Uni, so I wanted a good “refresher” that would give me a succinct but comprehensive and meaningful treatment of the main elements of this subject: this book was a perfect fit.
The typos are minor, and easy to identify; the section on further readings is extremely helpful, the bibliography is not huge but comprehensive enough, and there is even a very handy list of some key equations in information theory at the end of the book. Being this a non-specialist book, the proofs of some of the major theorems (some of which are admittedly quite curly and would require higher mathematics), are necessarily only explained at high, conceptual level – this is completely understandable given the pedagogical aims of this book, but this left me a bit dissatisfied and wanting for more, to be perfectly honest. I would also have loved some reference to the current research programs and applications of Shannon's information theory in applied mathematics and theoretical physics, including complexity theory, quantum information theory and even cosmology, most areas of application of which I know little and about which I would have loved at least some bibliographic references.
Anyway, this book is perfect as a serious introduction to the subject (providing more depth and rigour than a popular science publication usually would), highly recommended to all readers interested in this beautiful subject, and who want to achieve a more rounded understanding of physical reality; only basic background knowledge of probability/statistics and high school mathematics is required, and no prior knowledge of information theory is necessary.
A well deserving 5 star rating to this great introduction. A very enjoyable read, and the author does manage to convey the beauty and great relevance of this fascinating, and too often under-appreciated, subject.
March 13, 2017
Eggzellent stuff!
What a great intro to a subject I found fascinating and is widely applicable: Digital communications, computing, neuro-science and other biological sciences, linguistics (a favourite) and then there's my secret application that made me want to read the book in the first place...but you won't find it in the book.
There is a proper glossary of technical terms, something that long term readers of my reviews know I think is essential and yet all too frequently absent. There are also appendices on various topics in probability and statistics that are relevant and you may be unfamiliar with or in need of a quick refresher about. This is also good textbook writing, in my view, as is including XKCD cartoons (with permission). The latter are even relevant!
I found it straightforward to follow what was going on despite having been solidly rebuffed by my previous encounters with the subject. I think this is mainly because some opaque terminology is properly and thoroughly defined and explained and put into a practical context as soon as possible.
I strongly recommend this if you ever have a need to learn the basics of the subject and thanks to whomever recommended it to me!
What a great intro to a subject I found fascinating and is widely applicable: Digital communications, computing, neuro-science and other biological sciences, linguistics (a favourite) and then there's my secret application that made me want to read the book in the first place...but you won't find it in the book.
There is a proper glossary of technical terms, something that long term readers of my reviews know I think is essential and yet all too frequently absent. There are also appendices on various topics in probability and statistics that are relevant and you may be unfamiliar with or in need of a quick refresher about. This is also good textbook writing, in my view, as is including XKCD cartoons (with permission). The latter are even relevant!
I found it straightforward to follow what was going on despite having been solidly rebuffed by my previous encounters with the subject. I think this is mainly because some opaque terminology is properly and thoroughly defined and explained and put into a practical context as soon as possible.
I strongly recommend this if you ever have a need to learn the basics of the subject and thanks to whomever recommended it to me!
July 7, 2016
Information theory is central to the technology that we use every day - apart from anything else, in the technology that brings you this review (though ironically not in the book being reviewed as it doesn't appear to have an ebook version). As in his Bayes' Rule, James Stone sets out to walk a fine line between a title for the general reader and a textbook. And like that companion title the outcome is mixed, though here the textbook side largely wins.
The opening chapter 'What is information?' walks the line very well. It gradually builds up the basics that will be required to understand information theory and though it would work better if it had a little more context (for example, more about Claude Shannon as a person) to anchor it, the general reader will, with perhaps a few pages that needs re-reading, find it approachable and providing more depth than a popular science title usually would. I like the way that Stone uses variants of a photograph, for instance, to demonstrate what is happening with different mechanisms for compressing data. Unfortunately, though, this is pretty much where that general reader gets off, until we get to chapter 9.
The main bulk of the book, pages 21 to 184, cross that line and plonk solidly into textbook territory - they may cover the topic rather more lightly than a traditional textbook, but they simply don't work to inform without requiring the kind of investment of mind and mathematics that a textbook does - and, with a few brief exceptions, the writing style feels no different from the better textbooks I have from university. At chapter 9, the subject is brought round to information in nature, and there we get enough application and context to make what we learn seem more approachable again, though not to the same level as the equivalent part of Bayes' Rule. It's also a shame that (unless I missed it) there is no mention of Omega, Greg Chaitin's remarkable non-computable number.
So where Bayes' Rule is suited to popular science readers who want to stretch themselves and put in some extra effort, Information Theory can only really be regarded as a readable introductory textbook - it doesn't work in a popular science context. (Why then am I reviewing it? The author kindly provided the title for review in the hope that it would work for popular science readers.) If you are about to take a university course encompassing information theory - or are contemplating doing so - I can, however, heartily recommend this title as an introduction.
The opening chapter 'What is information?' walks the line very well. It gradually builds up the basics that will be required to understand information theory and though it would work better if it had a little more context (for example, more about Claude Shannon as a person) to anchor it, the general reader will, with perhaps a few pages that needs re-reading, find it approachable and providing more depth than a popular science title usually would. I like the way that Stone uses variants of a photograph, for instance, to demonstrate what is happening with different mechanisms for compressing data. Unfortunately, though, this is pretty much where that general reader gets off, until we get to chapter 9.
The main bulk of the book, pages 21 to 184, cross that line and plonk solidly into textbook territory - they may cover the topic rather more lightly than a traditional textbook, but they simply don't work to inform without requiring the kind of investment of mind and mathematics that a textbook does - and, with a few brief exceptions, the writing style feels no different from the better textbooks I have from university. At chapter 9, the subject is brought round to information in nature, and there we get enough application and context to make what we learn seem more approachable again, though not to the same level as the equivalent part of Bayes' Rule. It's also a shame that (unless I missed it) there is no mention of Omega, Greg Chaitin's remarkable non-computable number.
So where Bayes' Rule is suited to popular science readers who want to stretch themselves and put in some extra effort, Information Theory can only really be regarded as a readable introductory textbook - it doesn't work in a popular science context. (Why then am I reviewing it? The author kindly provided the title for review in the hope that it would work for popular science readers.) If you are about to take a university course encompassing information theory - or are contemplating doing so - I can, however, heartily recommend this title as an introduction.
November 2, 2018
Information Theory A Tutorial Introduction is a thrilling foray into the world of Information Theory by James V Stone. It starts with the basics of telling you what information is and is not. Now, although this is a tutorial of this subject, Information Theory is a subtle and difficult concept. Other people might get it, but for me, it is taking a while to understand even with this book.
The book is divided into chapters and further subdivided into sections. At the end of each section are Key Concepts. This makes it easier to digest the main idea of that section. The book contains plenty of examples and calculations that you can follow along with.
My main sticking point with the idea of Information is mostly the concept of Entropy. Rather than measuring information directly, it measures the amount of uncertainty inherent in a message. So the book talks about Probability Distribution Functions, Shannon Uncertainty, Error Correction Codes and so on. It doesn’t really start where Claude Shannon did, but Shannon didn’t have to introduce that much background to the people he was writing for.
The appendices contain a glossary, a grounding in Logarithms, a collection of pertinent equations and so on. The book is enjoyable, but as I mentioned before I might have to read it again to understand all of what is in this Theory.
The book is divided into chapters and further subdivided into sections. At the end of each section are Key Concepts. This makes it easier to digest the main idea of that section. The book contains plenty of examples and calculations that you can follow along with.
My main sticking point with the idea of Information is mostly the concept of Entropy. Rather than measuring information directly, it measures the amount of uncertainty inherent in a message. So the book talks about Probability Distribution Functions, Shannon Uncertainty, Error Correction Codes and so on. It doesn’t really start where Claude Shannon did, but Shannon didn’t have to introduce that much background to the people he was writing for.
The appendices contain a glossary, a grounding in Logarithms, a collection of pertinent equations and so on. The book is enjoyable, but as I mentioned before I might have to read it again to understand all of what is in this Theory.
September 7, 2019
rigour follows insight
A pleasure to spend time with. Stone's arguments are complete without being bloated, and he has a keen eye for philosophical and intuitive implications ("Why does maximum information look like pure noise?", "What exactly does half a bit mean?", and much more). This completeness means that he sometimes repeats definitions or lemmas, but I defy you to find this unhelpful.
The bibliography is also excellent, ranking a hundred books by their specialty and difficulty.
(Quibble: at the end he suggests that Shannon's originality was so strong that he "single-handedly accelerated the rate of scientific progress, and it is entirely possible that, without his contribution, we would still be treating information as if it were some ill-defined vital fluid". But his work seems so natural and elementary that this would surprise me. Weak evidence: Konrad Zuse independently invented Shannon's boolean circuit theory...)
December 27, 2021
Best introduction to information theory I've read. Will use it as text for any university level classes I teachb that are related
July 18, 2018
Excellent for what it was- a tutorial complete with lots of equations and examples/exercises. I actually was able to follow and loved the real-world questions at the end like calculating the amount of information contained in DNA (about a gigabyte.. not that much, but it’s contained in every cell of our body, mind you!)
June 19, 2022
This book helped me understand Shannon's paper. James does an excellent job of distilling that more complex paper into something that is easier to understand. He explains clearly entropy, surprise, channel capacity, and noise in a way that is easy to understand (well not simple, but approachable).
June 19, 2018
This is my first book on this topic and personally, I believe it did a pretty good job of introducing basic concepts and practical problems in Information Theory.
As demonstrated in his other book(Bayes' Rule: A Tutorial Introduction to Bayesian Analysis), James spends a considerable amount of time to solidify reader's interest and explain minimally required knowledge prior to present any new concepts. On the bright side, this method covers a wide range of audiences, which could include one with a high school level mindset. On the other hand, this approach somehow slows down the reading process of more experienced readers. For example, I prefer to let the readers figure themselves the connection of A and B with a single sentence, rather than a lengthy explanation that likely makes them confused on the way to reach that conclusion. However, from the perspective of an educator, it is understandable, and I suggest readers once finish have a brief scan through the book again to have the overall picture.
This book will cover:
1) Shannon information/Entropy: a concept to quantify information.
2) Information Theory in several real-life applications: image compressing, radio transmission... and even provides a general approach to handle any type of dataset to transmit efficiently.
3) A detailed explanation of the discrepancy between noiseless and noisy communication channels, discrete and continuous signals used in transmission.
4) Comparable interpretation in thermodynamic and other natural phenomena.
In conclusion, this is a good book for a beginner, but it is definitely not a walk in the garden and demands reader to make some efforts to fully understand.
P.s: I really love that he had spent one section to draw a line connecting Mutual Information and Kullback-Leibler divergence and showing that they are basically the same thing. This is extremely helpful for me, from the perspective of Machine Learning.
As demonstrated in his other book(Bayes' Rule: A Tutorial Introduction to Bayesian Analysis), James spends a considerable amount of time to solidify reader's interest and explain minimally required knowledge prior to present any new concepts. On the bright side, this method covers a wide range of audiences, which could include one with a high school level mindset. On the other hand, this approach somehow slows down the reading process of more experienced readers. For example, I prefer to let the readers figure themselves the connection of A and B with a single sentence, rather than a lengthy explanation that likely makes them confused on the way to reach that conclusion. However, from the perspective of an educator, it is understandable, and I suggest readers once finish have a brief scan through the book again to have the overall picture.
This book will cover:
1) Shannon information/Entropy: a concept to quantify information.
2) Information Theory in several real-life applications: image compressing, radio transmission... and even provides a general approach to handle any type of dataset to transmit efficiently.
3) A detailed explanation of the discrepancy between noiseless and noisy communication channels, discrete and continuous signals used in transmission.
4) Comparable interpretation in thermodynamic and other natural phenomena.
In conclusion, this is a good book for a beginner, but it is definitely not a walk in the garden and demands reader to make some efforts to fully understand.
P.s: I really love that he had spent one section to draw a line connecting Mutual Information and Kullback-Leibler divergence and showing that they are basically the same thing. This is extremely helpful for me, from the perspective of Machine Learning.
October 30, 2020
Nice concise introduction to the key quantities of information theory (entropy flavors and mutual information) and their relationships. The author walks through examples for most or all of the quantitative discussions or proofs, which I found very helpful, and it is a trivial matter to change some of the values and try the calculations for yourself. Overall the author did an impressive job of making a rather dry subject matter interesting. The quantitative discussions are interspersed with higher level semi-quantitative or qualitative ones, which I found to be a nice device to give the reader a break--the proofs and calculations can be more taxing than the more intuitive reasoning. The end of the book gives a taste of some applications (physics and chemistry of course, but also genomics, neural spike trains and their bandwidth, the universe as information) which can lead the reader to some of the Further Reading/Viewing suggestions. The glossary and appendices are well thought through--the right amount of the right information without being ponderously dense.
June 26, 2024
This is a fantastic introduction to information theory. It clearly explain subtle concepts like entropy in a very understandable manner using only the required math without getting lost in the details. it is so well explained that, while reading, many times the topic seems obvious.
February 16, 2021
Good at providing motivation for key definitions, also develops intuition well. Watching David MacKay's Youtube Lectures works really well with it (https://www.youtube.com/watch?v=BCiZc...). Afterwards, I felt well prepared for more detailed texts (e.g. Cover, MacKay)
January 31, 2022
This book is an absolute gem in statistics. It provides a wonderful introduction to distributions and frequentist statistics, and generalizes later to continuous distributions. If you are interested in learning more about information, this book is a must have
May 12, 2021
Very nice and clear introduction to Information Theory. It doesn't require a significant mathematics prowess and it could be used to refresh the ideas on the subject.
May 29, 2021
Short but dense, clear explanations. Only thing I would have liked more is if there were exercises.
August 11, 2021
Another brilliant book by Dr. Stone!
March 18, 2020
Fairly pedestrian
A basic introduction to Shannon’s early results in information theory. The book does attempt to make clear how unexpected and revolutionary these results were. To us now ... how could it ever be expected to be otherwise.
However the long detailed explanations and long pages of repetitious equations are uninspired and I am sure not terribly illuminating if you don’t know it already. That’s the test. Can you explain it to someone who doesn’t know it. You have to teach it not just regurgitate it. Feels a bit like a quick cash grab
A basic introduction to Shannon’s early results in information theory. The book does attempt to make clear how unexpected and revolutionary these results were. To us now ... how could it ever be expected to be otherwise.
However the long detailed explanations and long pages of repetitious equations are uninspired and I am sure not terribly illuminating if you don’t know it already. That’s the test. Can you explain it to someone who doesn’t know it. You have to teach it not just regurgitate it. Feels a bit like a quick cash grab
May 16, 2025
All in all, this is a lovely introduction book on the topic, highlighting the differences and difficulties in calculating information of discrete vs continuous variables. It does not hand-hold one through the solutions, but it does offer an intuitive and philosophical explanation to theories. Each chapter is neatly summarized towards its end, very helpful for quickly remind yourself of the essence.
August 14, 2018
Not bad, but not great.
I have read more entertaining books on the subject. The author goes to great pains to illustrate the proofs and processes of determining entropy and information rates . Sometimes it seems highly repetitive. It may be that I just wasn’t following all of his arguments. Might be my fault not his.
I have read more entertaining books on the subject. The author goes to great pains to illustrate the proofs and processes of determining entropy and information rates . Sometimes it seems highly repetitive. It may be that I just wasn’t following all of his arguments. Might be my fault not his.
September 10, 2019
A very accessible read, plenty of explanation before the equations are introduced. It really helped shape my understand on other information I had learned in school that I wasn't provided the history of or the background for. I highly recommend this as an introduction to the topic if you are interested in the topic.
November 29, 2020
Maravilloso libro de difusión sobre la historia historia de información. Lleno de anécdotas históricas y contemporáneas que hacen fascinante esta teoría. Recomiendo mucho esta lectura para los que reflexionan sobre la relación de la ciencia de datos con la teoría de la información.
August 12, 2017
I appreciate the fact that the author has approached the continuous information theory topic.
October 2, 2019
An excellent introduction to the main ideas in information theory with some flavor for it's applications presented at a good level between heavy textbook and pop science.
Read
January 3, 2020Good overview for the newcomer. Intuitive explanations, good examples, and not very technical. As a mathematical presentation of the subject it is surprisingly easy reading.
January 6, 2025
Great bibliography. Many topics fundamental with good explanation of broader context. Some topics feel random. Well written.
September 20, 2025
I really enjoyed how approachable this book was.
June 15, 2016
"So, what is information? It is what remains after every iota of redundancy has been squeezed out of a message, and after every aimless syllable of noise has been removed. It is the unfettered essence that passes from computer to computer, from satellite to Earth, from eye to brain, and (over many generations of natural selection) from the natural world to the collective gene pool of every species." (20)
"Suppose we are given a coin, and we are told that it lands heads up 90% of the time. ... When this coin is flipped, we expect it to land heads up, so when it does we are less surprised than when it lands tails up. The more improbable a particular outcome is, the more surprised we are to observe it. ... The Shannon information of an outcome is also called surprisal because it reflects the amount of surprise when that outcome is observed." (31)
"One particularly intriguing consequence of the final result above is that in order for a signal to carry as much information as possible, it should be indistinguishable from pure noise." (maximum entropy distributions, 127)
"No matter how efficient any physical device is (e.g., a computer or a brain), it can acquire one bit of information only if it expends at least 0.693kT joules of energy." (Landauer limit, 179)
"Suppose we are given a coin, and we are told that it lands heads up 90% of the time. ... When this coin is flipped, we expect it to land heads up, so when it does we are less surprised than when it lands tails up. The more improbable a particular outcome is, the more surprised we are to observe it. ... The Shannon information of an outcome is also called surprisal because it reflects the amount of surprise when that outcome is observed." (31)
"One particularly intriguing consequence of the final result above is that in order for a signal to carry as much information as possible, it should be indistinguishable from pure noise." (maximum entropy distributions, 127)
"No matter how efficient any physical device is (e.g., a computer or a brain), it can acquire one bit of information only if it expends at least 0.693kT joules of energy." (Landauer limit, 179)
May 12, 2016
Information itself has a measure; this simple concept, of its own accord, has revolutionized technology, communications, cryptography, and even physics. As stated in the title, the book is indeed a tutorial introduction to the beautiful mathematical theory of information. James V. Stone's writing communicates very accessibly on the subject for both discrete and continuous random variables. Topics such as uncertainty, information theoretic entropy, coding efficiency, the source coding theorem, the noisy channel theorem, and mutual information were quite digestible given the potential complexity of the subject. The book is not thoroughly rigorous. However, enough rigor exists to require little else to effectively apply the theory. The book aims to build intuition of the subject, which I believe is the author's intention. A solid basis in basic probability theory would make this book more comprehensible to the reader. While the book uses good examples, readers looking for exercises may need to look elsewhere.
June 10, 2025
Non troppo tutorial, a dire il vero
Uno si aspetterebbe qualcos'altro da un testo con sottotiolo Ä Tutorial Introduction. Invece Stone va subito sul difficile, e i Key Point che terminano i paragrafi aiutano sicuramente a capre dove si sta andando, ma non danno chissà quale ripasso. Più che un'introduzione, insomma, lo definirei un testo di riferimento per chi sa già di che si parla. Ah: l'ultimo capitolo ccon applicazioni non immediate è interessante.
Uno si aspetterebbe qualcos'altro da un testo con sottotiolo Ä Tutorial Introduction. Invece Stone va subito sul difficile, e i Key Point che terminano i paragrafi aiutano sicuramente a capre dove si sta andando, ma non danno chissà quale ripasso. Più che un'introduzione, insomma, lo definirei un testo di riferimento per chi sa già di che si parla. Ah: l'ultimo capitolo ccon applicazioni non immediate è interessante.
Displaying 1 - 29 of 29 reviews