Home->
Concepts->Entropy and Information
October 31, 1999
Updated September 3, 2001
Updated October 27, 2011
Entropy and Information
The unit of information used in neural systems should
explicitly be a function of time like the unit of hertz.
For example, the information in the time-independent sequence of
letters S = {a, b, c} is unique for the object it represents for
the entire duration that it exists. However, in a dynamical systems,
the potential information in a sequence can be smaller or larger
depending on the forces acting on the system. The unfolding of a
sequence depends on the environment or system that the sequence
represents.
Temporal Sequences and the Binary Interval
In the real physical world, as compared against a mathematical
description of it, information is essentially contained in the
states of matter or energy. Entropy is a descriptive
attribute of the states of energy (or matter). In this very
general context, information exists or emerges from within the
states of energy.
Information can also be described mathematically. Information
as an abstraction can be spoken and written of mathematically using
a function describing the unfolding of a binary sequence which is
the logarithm of base 2.
is the number of
possible events or outcomes of the system. Entropy described
mathematically is confined within the general equation above.
Mathematically speaking, the unit of information is the bit:
either 0 or 1. The bit represents a time-independent unit of
information. But information can be defined as an elemental binary
unit which changes as a function of time: f({0,1}; t).
We could also measure information change more naturally as the
rate-of-change of the information or bytes in a system. The frequency
of a system, measured in hertz, is the rate-of-change of the
oscillations in a system. So a system vibrating at 1 hertz means
a bit is changing from 0 to 1 and back again in 1 second. Visualize
the movement of 1 cycle of a sine wave. Now a sequence of notes on a
sheet of music represent objects whose units are in hertz. But observe
that notes on a music sheet represents a frequency that is constant in
time. The representation of notes is in a subclass of the type of
dynamic sequence described above.
Thoughts Are Continuous
Our thinking processes are the result of real physical or biological
events occurring as a function of time. Our thinking processes exists
as a real dynamical energy system. It is not a mathematical abstraction
which is not real in the sense of having to exist in the physical world.
We can think about mathematical processes, but these abstraction themselves
do not have a real physical existence. This is in contrast to our
thoughts which are created because of the real energetic neural synapses
in our brains.
Our thoughts are created by real physical processes which are
subject to the constraints of the Heisenberg uncertainty principle
which limits the creation of our thoughts as a function of time.
However, what we think is unlimited because what we think is not
physically real.
Temporal sequences are intrinsically multi-dimensional in nature [1].
Information in the mind is created dynamically as a temporal sequence.
The thoughts in our mind seem fleeting because they are generated or created
everytime our neural waves move across parts of the brain. The action of
thinking means that our brain creates a temporal sequence of energy patterns
that only lasts for a very short duration. In order to hold a thought
our brain needs to continually recreated these sequences of
energy patterns or synapses.
In trying to create a model for the software tools I had try to try
to rid myself of the old way of thinking about information. When the
mind generates information, it only lasts in the order of a fraction
of a second. Then it needs to be recreated. I'm going on faith
from past physiological studies that the physical structures which
produces these energy patterns are the coherent, synapsing neurons.
In the software model, I've try to separate the local micro-ensemble
effect of neurons from those global clustering effects in synchronous
neural circuits, and then try to put the pieces back together.
Footnote
[1] Dennis Gabor asked [2], "... what it is that prevents any instrument from
analysing the information area with an accuracy of less than a half unit.
The ultimate reason for this is evident. We have made of a function of
one varialbe -- time or frequency -- a function of two variables -- time
and frequency." He said that
is the mathematical identity which is at the root of the fundamental
principle of communication. He said, "We see that the r.m.s. duration of
a signal, and its r.m.s. frequency-width define a minimum area in the
information diagram."
Further along, in section 4 of Dennis Gabor's Theory of Communication, he
says, "Moreover, it suggests that it might be possible to give a more
concrete interpretation to the information diagram by dividing it up into
"cells" of size one half, and associating each cell with an "elementary
signal" which transmitted exactly one datum of information."
[2] Theory of Communication, Dennis Gabor, 1946,
The Journal of the Institution Of Electrical Engineers, 93(3):429-457.
next:
Waves
