Informational entropy is often identified as physical entropy. This is surprising because the two quantities are differently defined and furthermore the former is a subjective quantity while the latter is an objective one. We describe the problems and then present a possible view that reconciles the two entropies. Informational entropy of a system is interpreted as physical entropy of a whole composed of both the system and “memories” containing information about the system.
The second law of thermodynamics is one of the most concrete laws in nature [
Here W is the number of microstates belonging to a given macrostate. (We set k ln2 = log2 = log where k is Boltzmann constant.) Let us consider physical entropy of a microstate. (It is often said that physical entropy cannot be assigned to a microstate. But this is a misconception. A physical system is in a “microstate” not in a “macrostate” at any time.) At each instant, the physical system must be in a “microstate” corresponding to a macrostate of the physical system. Physical entropy of the microstate should be that of the macrostate which the microstate belongs to. That is, the physical entropy S of a microstate is defined by the number of microstates belonging to the same macrostate. (The next task is to appropriately divide the phase space such that each region of the phase space corresponds to a macrostate. This is a very subtle problem which is not topic of this paper.) Clearly, the physical entropy S is an objective quantity, and is observer independent. A physical system in a microstate belonging to a macrostate has a fixed value of S, whether one knows identity of the microstate or not.
On the other hand, Shannon proposed a quantity
to measure randomness of an entity [
Proposition 1: Informational entropy
Here we denoted the “someone” by an “observer”.
Surpringly, however, informational entropy
This paper is organized as follows. Section 2 gives an explanation about why the two quantities are not to be simply identified. In Section 3, we discuss explanations proposed so far. In Section 4, the main part of this paper, we describe how informational entropy can be interpreted in terms of objective entropy. In Section 5, a related question is discussed. The conclusion is reported in Section 6.
As introduced previously, informational entropy and physical entropy are subjective and objective quantity, respectively, being differently defined. Thus the two quantities are not expected to be the same. Here we consider a question, “entropy of universe?” which illustrates difficulty of the identification.
The same problem exists in both classical and quantum mechanics. Here we pose the problem in a quantum form that might be more familiar to physicists. Von Neumann introduced a quantum analog of informational entropy,
for a density operator
It is difficult to expect that an observer dependent quantity, von Neumann entropy, can be simply identified with an objective quantity, physical entropy. Let us consider a question “What is the (von Neumann) entropy of the universe?” One might say that entropy of universe is zero because the universe is in a (huge) pure state. This is in contradiction with a fact that physical entropy of universe is enormous. A correct answer to the question is that von Neumann entropy is not defined for the universe. Recall that there is no one outside the universe for whom the probability
However, it should be that physical entropy can be defined for the universe. If a quantity can be defined but the other one cannot be for an identical case, clearly we cannot simply identify the two quantities.
A common explanation about the identification is as follows. Consider a macrostate composed of W different microstates. Assume that an observer has no knowledge about microstate. In this case we should assume equal probability for each microstate. Then we get
However, the argument implies only that the identification is valid in the specific case when the observer has zero knowledge. The argument does not justify usual identification of information entropy with physical entropy in a general sense. Suppose the observer somehow knows in which microstate it is. The informational entropy becomes zero because all
In another interesting argument for the identification, it is utilized that mixing process increases physical entropy [
What we can see by the argument is, however, that the two quantities can be identified for the mixing process. The argument doesn’t say something about other cases. For example, let us consider a hypothetical case in which observer knows position and velocity of each molecule although all molecules are mixed in the chamber of volume 2V. In this case, informational entropy is 0 because observer has full knowledge of each molecule. The result is confusing because the gases are fully mixed. The argument in Reference [
The paradox is the followings. Suppose observer somehow knows in which microstate a macrostate is. Then informational entropy is zero because all
A key fact in the argument is that there is “someone” or an “observer” who will use the knowledge. This suggests that the Proposition 1 can be a clue for resolution of the paradox. Here let us consider “memories” of the observer. We can see that the more knowledge about the moleccules the observer has, the more correlated with the molecules the memories are. We consider not only a system (the molecules) but also the observer’s memories. However, “more correlation” between molecules and memories means “less volume” in phase space occupied by the combined whole of molecules and memories. “Less volume” then implies “less physical entropy” by Equation (1).
Explanation 1: Informational entropy of a system can be interpreted as physical entropy of both the system and memories containing the information on the system, namely
Let us illustrate Explanation 1. We consider informational entropy H of a system of (a positive integer)
The first case is when the system and memory are perfectly correlated. This means that memory has perfect information about the system. In this case, each qubit of the system might be either
Informational entropy in this case is zero because memory gives full information about the system.
The second case is that the system and memory are not correlated. This means that memory has no information about the system. For example,
Informational entropy in the second case is easily derived from
Therefore, difference of physical entropy is equal to that of informational entropy.
Informational entropy change due to loss of information can now be explained in terms of physical entropy change of the combined whole of system and memory.
A related problem involved with the identification, often regarded as a paradox, is “the (informational) entropy is constant [
One of the most puzzling questions regarding entropy is that informational entropy is often identified as physical entropy. It is because the two quantities are differently defined. The former is a subjective quantity while the latter is an objective one. There are a few previous explanations (or interpretations) for the paradox but these are meaningful in some specified contexts. Here we provide an interesting explanation which, we find, reveals the core of the problem. We made a key observation that informational entropy is not defined without “observer”. (This implies that informational entropy of the universe is not defined.) Based on the observation, we presented a possible view that reconciles the two entropies. Informational entropy of a system is interpreted as physical entropy of a whole composed of both the system and “memories (of observer)” containing information about the system. We expect that our explanation will shed light on resolution of the paradox.
This study was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2010-0007208).