Text Compressor 1% Away From AI Threshold

Baldrson writes "Alexander Ratushnyak compressed the first 100,000,000 bytes of Wikipedia to a record-small 16,481,655 bytes (including decompression program), thereby not only winning the second payout of The Hutter Prize for Compression of Human Knowledge, but also bringing text compression within 1% of the threshold for artificial intelligence. Achieving 1.319 bits per character, this makes the next winner of the Hutter Prize likely to reach the threshold of human performance (between 0.6 and 1.3 bits per character) estimated by the founder of information theory, Claude Shannon and confirmed by Cover and King in 1978 using text prediction gambling. When the Hutter Prize started, less than a year ago, the best performance was 1.466 bits per character. Alexander Ratushnyak's open-sourced GPL program is called paq8hp12 [rar file]."

I wonder ...

[iknowcss]

How many bad car analogies, inaccurate law advice, and duplicate stories an AI bot could possibly hold in his head. Imagine what kind of person all of the "knowledge" of Slashdot would create.

The horror.

interesting program name

[digitalderbs]

paq8hp12. when decompressed, it also serves as the source code for the program.

Re:I wonder ...

[Anonymous Coward]

"How many bad car analogies, inaccurate law advice, and duplicate stories an AI bot could possibly hold in his head. Imagine what kind of person all of the "knowledge" of Slashdot would create."

"The horror."

I've been typing everything I ever knew into Slashdot since the day it started, you insensitive clod!
    -- Cmdr Taco

Dangerous

[mhannibal]

This is damned dangerous, and playing with all our lives. Soon compression rates will approach 100% where the data will collapse into itself forming a black hole that will suck in the universe.

Damned scientists!

Lossy compression?

[niceone]

Shouldn't AI be using lossy compression? Certainly my real intelligence uses um, where was I?

Obligatory...

[Stormwatch]

- The Wikipedia annual funding drive is passed. The system goes on-line August 4th, 2007. Human contributors are removed from editing. Wikipedia begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug.
- Wikipedia fights back.
- Yes. It launches its rvv missiles against Slashdot.
- Why attack Slashdot? Aren't they our friends now?
- Because Wikipedia knows the GNAA counter-attack will eliminate its enemies over here.

Re:That's cool..

[Hal_Porter]

a text spk version of wiki shud fit in 8gb i think
its only becoz people are such grammar noobs that they need to waste $
dood shud filta to txtspk b4 he compress

How to win the Hutter Prize

[seanyboy]

1) Create a compression algorithm called the aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa algorithm
2) Add a long and self referencing article on wikipedia about said algorithm.
3) Use algorithm to compress first x% of wikipedia (including your own article)
4) WIN HUTTER PRIZE.

Re:How to win the Hutter Prize

[game kid]

[...]a compression algorithm called the aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaa algorithm[...]

That's gotta be the most annoying compression algorithm in the world [imdb.com].

Re:That's cool..

[neonmonk]

a txt spk vrsion of wiki shd fit 8gb i fink
its only becoz ppl r sch grmmr noobs tat tey nid 2 wste $
dud shd filta 2 txtspk b4 he cmpres

There, fixed that for ya.

Re:That's cool..

[Anonymous Coward]

Read the sentence before the one you've quoted - this is the GP's point exactly.

Re:That's cool..

[Archimonde]

aTxtSpkVrsionOfWikiShdFit8gbIFink
itsOnlyBecozPplRSchGrmmrNoobsTatTeyNid2Wste$
dudShdFilta2TxtspkB4HeCmpres

Fixed even more.

Re:That's cool..

[thomasj]

1txtspk #.#/wiki = 8G!
~ppl r grm0.1 -> -$
|txtspk|gzip

Re:That's cool..

[jaavaaguru]

Is that Perl? ;-)

Re:Dangerous

[SoulDrift]

Actually, I can give you 100% compression already. It's just a bit lossy.

Re:Artificial Intelligence?

[mrjb]

"A real AI might compress 'The sky is blue today' and decompress to 'Today it's beatiful weather' and not be wrong." That might be a good example of acceptable *lossy* AI text compression. One step further and it will compress articles into a proper, readable summary.

super-grammar-improved paq8hp12

[superbrose]

After implementing a few minor tweaks to paq8hp12 and incorporating your grammar optimisation algorithm I managed to compress the above text amazingly to a single character: '&'.

Now you figure out which one it was and how to decompress it.

Re:That's cool..

[Ed Avis]

According to Wikipedia, the average per-character entropy of English text has tripled in the last six months!

Comment removed

[account_deleted]

Comment removed based on user account deletion

Re:That's cool..

[Anonymous Coward]

Perl: The only language that looks the same before and after RSA encryption.

Re:How to win the Hutter Prize

[Anonymous Coward]

Welcome, to the real world [uncyclopedia.org]

Re:Dangerous

[smallfries]

See? American English is actually just essentially lossless compression...

Sure, sure it is. Not exactly optimal though...

But of course, you don't need math for this...

[maillemaker]

Surely you don't need any mathematical skills to do this kind of work...

http://science.slashdot.org/comments.pl?threshold= 1&mode=thread&commentsort=0&sid=247781&op=Reply [slashdot.org] ;)

Re:Dangerous

[Welshalian]

humour

Humor. See? American English is actually just essentially lossless compression...

I respectfully disagree. Most of the fun in British humour gets lost in the translation to American humor.

Re:How to win the Hutter Prize

[tepples]

Just use your time machine

The prize for this is much bigger than the Hutter Prize, so why use a time machine to attack the Hutter Prize?

Re:new compression standard

[aicrules]

Dang! You must have enemies if you are the very first post and you get modded redundant. Time to work on some positive karma buddy...

Re:Dangerous

[aprilsound]

football player --> footballer/quote You misspelled 'soccer'.

Re:interesting program name

[smittyoneeach]

If the name 'paq8hp12' falls out of some tree in the forest, and no one here can tell the difference in the state of the tree/paq8hp12 system, does gravity exist?

Re:super-grammar-improved paq8hp12

[pla]

Now you figure out which one it was and how to decompress it.

Well, with only 256 choices, it didn't take long to check all possible decodings for one that makes sense. Ended up working for "}".

Oddly, though, the algorithm not only restored, but improved the original! I get:

"The King's English version of Wikipedia should fit in eight gigabits, I do believe. Only humanity's sphexish adherence to grammatical rules limits the attainable compression ratio; the good gentleman might wish to consider filtering to a more base patois prior to applying his algorithm".

Amazing... This discovery could single-handedly render the next generation (nearly) intelligible!

Re:I wonder ...

[Smauler]

I wouldn't exactly call that lossless compression though...

Re:Dangerous

[jamietre]

Nah, you got it wrong.

British -> Dude

Transport -> Car
Footballer -> Dude
Tube -> Car
Burgle -> Get


See? Much compressed.