Video Gallery

The Zipf Mystery

Go to Video Gallery Added Dec 21, 2017 •

Copy this URL:
Embed code:	<iframe class="edlio-embed-player" type="text/html" width="640" height="360" src="//www.hancockhs.org/apps/embed/?v=169672" frameborder="0" allowfullscreen></iframe> Change dimensions

Go HD

From https://www.youtube.com/watch?v=fCn8zs912OE

Show Transcript

Michael hear about 6% of everything you say and write the is the most word in the English language about 1 out of every 16 is I bet it for you was with on as have but these a fun fact of piece of trivia but it's also more commonly used words are ranked across entire language article a bizarre pattern emerges will appear about half as often as the most used open the fourth one fourth as often V one and go on all the way down for some reason the amount of time straight line of power law phenomenon is called dip and it doesn't only apply to English languages like all of them haven't been able to translate yet I have no idea why it's surprising that something as complex as reality should be paid by something as to how predictable watch this. Org which ranks words as found in British national Corpus common English word now how many times every word on Wikipedia and in the entire Gutenberg shows up about 181 million knowing that the word thought about 30,000 time on Wikipedia combined and pretty much does the world is chaotic stop the world complex activities and behaviors to follow the basic rule don't know of research has yet to close the case Ziploc doesn't just mysteriously describe were you also found in city population solar flare intensities website earthquake number of times academic papers are cited last name the firing patterns of neural network the number of phone calls popular the rate at which we forget about why language is dippy but no firm conclusion and this video doesn't contain a definite explanation either. since we appear to like knowing more than mystery but that said we also more than we ramifications some related pattern explanation of the mystery itself form the Pareto Principle 20% of the causes are responsible for 80% of the outcome like in language account for over percent of word occurrences 20% of his iPod percent of the pee comes up a lot in the world the richest 20% of humans have 82.7% of the world's income in the US 20% of patients use 80% of healthcare resources in 2002 Microsoft reported that 80% of the air and crashes an office are caused by 20% of the bugs detected a common rule of thumb in the business world and 80% of the complaints you receive 20% of your customers receive 80% of the wear and as Woody Allen famously said 80% of success is just showing up the Pareto Principle is everywhere which is good what's wrong percent of the problem variety of different unrelated to the bottom of what causes some of them Google find that one or more of those mechanisms is responsible for law in language of the principle of Stafford for Lifan to follow the path of least resistance of human behavior develop in our species naturally preferred drawing from as few words as possible to get their thoughts out there he was easier preferred larger vocabulary they had to do less work the compromise between listening and speaking the felt led to the current state of language a few words are used often and many many many words rarely papers have suggested that having a few short predictable words vocab for the information race it makes sense which even more simple after zip Seminole paper at all words Ziploc is why it happened there are more alphabet can be used to make 26 but 26 squared two-letter word also in random typing whenever the spacebar is pressed a word terminate are exponentially less likely than shorter one the combination of all 26 letters and bar are tight after a letter is tight and a word has begun the probability that the next input will be a fake but creating a 1 letter word in 27 if you randomly generate characters or fire a proverbial typing monkey about 1 out of every 27 or 3.7% of the stuff between faces will single letters letter words after beating a word character but the space bar is hit a 26 and 27 and then the spacebar letter word if the probability of another letter and if we will make up about .2% random typing the word thought according to the Rain pick up on a most often used lift there are 26 possible 1 letter words so each of the top 26 ranked words are expected to occur about this off in the next 676 rain will be taken up by 2 letter word that show up about this often as changing up the initial conditions can the steps out our mysterious distribution has been created out of nothing but the inevitability of math so maybe there is no mystery many words are just the result of randomly world and the mental world into label and zip law describes what naturally happens when you do that and as always thanks for what very different from random typing and vocabulary we have to work with the result of random naming the elements are used in language according to zipf's law are constrained by the natural world and they're not the label never heard before like something to write a story about alien names people will naturally Ziploc appears to be into our brain perhaps there is something about the way you thought and topics of discussion that contribute law by a processes that shape how they previously operated preferential attachment processes occur when something money attention variation jobs anything really giving out a call how much is already possessed to go back to the carpet example if most people walk from the living room to the kitchen across a certain path furniture will be placed elsewhere making that path popular is a video or the more likely it is to get recommended automatically makes which give it more views snowball rolling down a snowy hill the more snow accumulation for collecting more and the faster it grows driving a preferential attachment process it can happen naturally private and grab at random Lincoln together and then throw them back in the pile now repeat over and over again if you grab paper clips that are already part of Lincoln anyway you will have a distribution that looks a small number of chains contain a disproportionate amount of total paperclip count made even longer the rich get richer the big get bigger the pop popular it's just math mystery is if not caused by at least by preferential attachment the word is it more likely to be used again appointment play a role as well writing in conversation until a critical vocabulary Empower laws to make this the most natural way for language perhaps some of our vocabulary and grammar was developed randomly according to mandelbrot Theory and the Natural Way conversation in discussion and effort when speaking and listening are all possible for the relationship between word Rank and frequency the answer is simpler made of this is mind-blowing of any book or article will be other half will be words that appear in that selection words make up about a third of everything we date and the top 100 about half. Whether it's all the words in wet Hot American Summer or all the words in Plato's complete works or in the complete works of only about 100 words are used for nearly half of everything written or said in Alice's Adventures in Wonderland 44% foyer 49.8% of appear only in the book a word that is used only one in a collection of words is called a well, not are vitally important to understanding languages found one in the entire known collection of an ancient language it can be very difficult to figure out what it means there is no Corpus of everything ever said or written in English but there are very very large Collections and it's fun to find a Paxil dominant won't be the case after I mentioned it but the word is in the Oxford English Dictionary but appears nowhere on Wikipedia or or in the British national Corpus American National Corpus one result on Google that listed as a rare word by the way means in a mocking manner as in Paradis rattled off does heater bad that quizzaciously has been used so infrequently what you experience on a day-to-day basis forgotten forgettable the dictionary of obscure Sorrows a few days are memorable I've been alive for almost 11,000 days but I couldn't tell you something about each one of them I mean not even closed most of what we do and say and think and say in here and feel gotten quite similar which makes sense it's a number of factors natural it makes sense weed that way to something really well most things hardly at all me out sometimes because it means so much is forgotten things that at the time you thought you never forget my locker number here it's combination the comedian on stage how many memories are gone and realize that I can't remember every detail from them it's a little disappointing I mean why even bother if the Pareto Principle dictates that my mind will, remember basic reaction years later Ralph Waldo Emerson makes me feel better he once said I cannot remember the books I've read anymore than the meals I have eaten even so they have as always

Listed in Library

Video Gallery

The Zipf Mystery

Popular Videos