Stack Overflow
Stack Overflow

Questions tagged [information-theory]

Information theory is a branch of applied mathematics, electrical engineering, and computer science involving the quantification of information.

184 questions
65
votes
15 answers

What is the computer science definition of entropy?

I've recently started a course on data compression at my university. However, I find the use of the term "entropy" as it applies to computer science rather ambiguous. As far as I can tell, it roughly translates to the "randomness" of a system or…
fluffels
  • 3,946
  • 7
  • 33
  • 51
61
votes
1 answer

Optimal way to compute pairwise mutual information using numpy

For an m x n matrix, what's the optimal (fastest) way to compute the mutual information for all pairs of columns (n x n)? By mutual information, I mean: I(X, Y) = H(X) + H(Y) - H(X,Y) where H(X) refers to the Shannon entropy of X. Currently I'm…
nahsivar
  • 929
  • 1
  • 9
  • 13
49
votes
7 answers

How do I compute the approximate entropy of a bit string?

Is there a standard way to do this? Googling -- "approximate entropy" bits -- uncovers multiple academic papers but I'd like to just find a chunk of pseudocode defining the approximate entropy for a given bit string of arbitrary length. (In case…
dreeves
  • 25,132
  • 42
  • 147
  • 226
27
votes
1 answer

Mutual information and joint entropy of two images - MATLAB

I have two black and white images and I need to calculate the mutual information. Image 1 = X Image 2 = Y I know that the mutual information can be defined as: MI = entropy(X) + entropy(Y) - JointEntropy(X,Y) MATLAB already has built-in…
Jorge
  • 355
  • 1
  • 4
  • 10
19
votes
3 answers

Any theoretical limit to compression?

Imagine that you had all the supercomputers in the world at your disposal for the next 10 years. Your task was to compress 10 full-length movies losslessly as much as possible. Another criteria was that a normal computer should be able to decompress…
David
  • 4,052
  • 7
  • 46
  • 77
18
votes
1 answer

Continuous mutual information in Python

[Frontmatter] (skip this if you just want the question): I'm currently looking at using Shannon-Weaver Mutual Information and normalized redundancy to measure the degree of information masking between bags of discrete and continuous feature values,…
MrGomez
  • 23,548
  • 42
  • 69
16
votes
3 answers

Can the value of information gain be negative?

Is there a chance to get the value of information gain be negative?
julie
  • 185
  • 1
  • 1
  • 3
15
votes
5 answers

How does the entropy of a string of English text signify low quality?

Jeff Atwood recently tweeted a link to a CodeReview post where he wanted to know if the community could improve his "calculating entropy of a string" code snippet. He explained, "We're calculating entropy of a string a few places in Stack Overflow…
Pandincus
  • 9,286
  • 7
  • 40
  • 61
11
votes
3 answers

How do I calculate the entropy of a graph?

I have a set of randomly generated formal graphs, and I would like to calculate the entropy of each one. The same question in different words: I have several networks, and want to calculate the information content of each one. Here are two sources…
shotgun_approach
  • 191
  • 1
  • 6
11
votes
3 answers

What is the difference between an Information Model and an Ontology?

What is the difference between an Information Model and an Ontology? They look the same to me. Thanks!
PedroD
  • 4,310
  • 8
  • 38
  • 75
10
votes
2 answers

What's the most that GZIP or DEFLATE can increase a file size?

It's well known that GZIP or DEFLATE (or any compression mechanism) can increase file size sometimes. Is there a maximum (either percentage or constant) that a file can be increased? What is it? If a file is X bytes, and I'm going to gzip it, and I…
SRobertJames
  • 6,827
  • 12
  • 48
  • 89
10
votes
7 answers

Practical way of explaining "Information Theory"

Information theory comes into play where ever encoding & decoding is present. For example: compression(multimedia), cryptography. In Information Theory we encounter terms like "Entropy", "Self Information", "Mutual Information" and entire subject is…
claws
  • 47,010
  • 55
  • 140
  • 185
9
votes
1 answer

Compressibility Example

From my algorithms textbook: The annual county horse race is bringing in three thoroughbreds who have never competed against one another. Excited, you study their past 200 races and summarize these as probability distributions over four outcomes:…
Dijkstra
  • 2,364
  • 3
  • 20
  • 34
9
votes
2 answers

Is there an algorithm for "perfect" compression?

Let me clarify, I'm not talking about perfect compression in the sense of an algorithm that is able to compress any given source material, I realize that is impossible. What I'm trying to get at is an algorithm that is able to encode any source…
Nathan BeDell
  • 2,065
  • 1
  • 11
  • 20
9
votes
11 answers

Algorithm for rating the monotonicity of an array (i.e. judging the "sortedness" of an array)

EDIT: Wow, many great responses. Yes, I am using this as a fitness function for judging the quality of a sort performed by a genetic algorithm. So cost-of-evaluation is important (i.e., it has to be fast, preferably O(n).) As part of an AI…
George Armhold
  • 29,784
  • 45
  • 147
  • 224
1
2 3
…
12 13