TEDxCarletonU 2010 - Jim Davies - The Science of Imagination

TEDxCarletonU 2010 – Jim Davies – The Science of Imagination



so what is imagination when people use the term imagination they usually have two meanings in mind and the first one they think of is creativity and this is the kind of thing that you mean when you say things like Julie Taymor has a great imagination or when somebody says I have no imagination the other sense of imagination is picturing things in your head and this happens all the time this happens when you plan when you think about the future when you reason hypothetically when you think about the past or how the past might have been when you design when you fantasize and when you think about when you dream when you're sleeping when you daydream when you're awake which by the way we spend about 20% of our time doing in other words imagination is incredibly important and in fact we've even found that if you imagine doing sports it actually makes you better at doing the actual sport well my goal in life is to understand imagination I want to get under the hood and figure out how it works when you read a book or someone asks you to imagine something at that picture in your head where does it come from how do you decide what goes in that picture and where do those things go so that's what I'm going to try to do so when I became a professor I started the science of imagination laboratory and according to our theory imagination is constrained by three different things the first thing is the environment that you're in when you're called upon to use your imagination now this might be the book that you're reading or when I tell you to imagine a dog or something and the second thing is what we understand about the world the rules that we understand of how the world works we'll call that a world model and then finally our visual memory and that is everything that we've ever seen right so I'm going to talk about how you can study imagination and how it's been studied scientifically and I'm particularly going to think about how our perceptual history affects our imagined so I want to understand imagination well this might seem a little strange right imagination and science they don't seem to fit imagination seems to be one of the most elusive and mysterious things that there is impenetrable to scientific investigation well I'm going to talk to you about some ways that you can study imagination scientifically how other people have done it how I've done it and what the future is going to hold for a science of imagination so one thing that you can ask people to do is to draw what they imagine all right and all these things have problems with them but you try to do all of them and see if their results converge and when you ask people to imagine shapes for example they imagine them of the flat side down this is something we've done in our lab and we hypothesize that this is because there's stability in the real world now you might be thinking well maybe it's because you see shapes with a flat side down all around you and I think there's truth to this too but I think there's an interesting question why is it that things in the real world have a flat side down and in your imagination and I think it's for the same reason that there's stability in a world that has gravity now over at McGill gosselin and shins did this really cool experiment where they look right into the imaginations of people here's what they did they showed people white noise just random dot patterns and they told them that some of these things we're going to have the letter S in them and some of them weren't and they're supposed to indicate whether or not they saw the letter S okay now 20,000 images this took a long time but by averaging the ones they said yes to because if you look hard enough you'll see we call that top-down processing if you averaged the ones they said yes to you get this ghostly image of the s that they thought that they saw and in fact each person had a different font to it so this is the this is the result of one person's results here so this this is great look at this this is the most direct view into someone's imagination that I've ever seen there's another field that contributes a lot to this and that's called an embodied cognition field and the embodied cognition field shows us things like we imagine good motion to be left to right in our visual field that's actually in our culture and cultures where people write in the other direction it's it's the opposite and you'll notice when you watch the Matrix movies every time neo gets into a fight he's running left to right his enemies running at him from right to left on the screen they've also found that good motion is up and bad motion is down and this gets reflected in our language when we say things are looking up or she's really on top of things or the depression is in a downward spiral okay these are we've also associated verbs with directions you show people choices like this they're remarkably consistent in which ones they will associate with certain verbs respect is up giving is left to right destruction is down how does this affect our imagination well if I ask you to imagine somebody giving something you're more likely to imagine the gift moving from left to right in your mind's eye so we do these psychological kinds of experiments in my lab but we're also trying to replicate human imagination on computers we're trying to get computers to imagine the same way that the average person would okay well how does this work one thing you could do people can imagine things of different sizes that's no problem so if I ask you to imagine a tiny Cheshire Cat you have no problem doing that even though you've probably never seen anything in your life that's been labeled as a tiny Cheshire Cat one of the students from my lab Jonathan Gagne created a program called Visio and what it can do is it can imagine things of different sizes Heights anything that can be described with a number so if you ask you to imagine a tree it can do that if you ask it to imagine a long tree it can do that too even if it's never seen anything that's been labeled as a long tree here's how it works let's imagine you've seen a bunch of crows in your life okay now according to Visio theory every different size of Crow you've ever seen gets put into a distribution okay a distribution with all the different sizes of crows each you've ever seen all right now some of these crows are going to be labeled as large all right well those large crows they contribute to all crows but they also go into a special distribution for large gross and likewise for the small ones now let's imagine you've seen some Ravens but you've never seen ly Ravens that were labeled as large so the Ravens contribute to the all Ravens distribution over here now you've also seen buildings and ocelots and improv groups so you're called upon to imagine a large raven well according to our theory what happens is you look for the most related thing in memory Ravin in this case it would be crows and not buildings or ocelots we'll get rid of those and then you look at how to transform the distribution of all crows into the distribution of large crows so we have this mathematical transformation and we apply that same transformation to Ravens what we end up with is an imagined distribution of what large Ravens might look like and from that you can imagine a single large raven so here we have a system that can imagine new sizes of things that's never seen before so step toward computer creativity and imagination and perhaps understanding it in humans as well so if we want to replicate human imagination we need a database of everything everybody's ever seen but we don't have that ok but we do have a proxy for that we've got images from the web that are labeled with different regions all right so there's a database that I've got from a game called peekaboom and what it has is labelled regions for different parts of images so here we have a puppy we know there's a puppy there but this database also has information about where the eyes are where the nose is where the tail is that kind of thing so it can figure out that noses tend to be below the eyes and the mouth tends to be below the nose without us having to tell the computer that explicitly which is a huge job believe me so we've got this a database that shows us what things tend to appear in images together and where they are that's very powerful so we can create a map and imagination model that can do the imagining for us so if you imagine ask it to imagine a bird above a house or something like that it can imagine the bird in the house but it can also know what else should be there a sky a ground whatever and we're in the image it should be so here's what we do we mined this database and try to find all the co-occurrences what kinds of things tend to go together in images because that's very important for imagination another one of my students essays rstudio is working on a he made a program called the image Oracle and what you can do is you can type in a word and it'll tell you what else is going to be in the image so if you type in computer this is what you get there's a 28% chance there's a screen in the image and 18% chance there's a window etc so this is this is good and they'll also tell you where it is because we look at the angles and distances between them so let's just say that you ask or whole system to imagine a big tree the Visio program can tell you how big that tree is supposed to be and then the image Oracle can tell you what else is supposed to be in the image with the tree and where it is and we can figure out all the sizes for that too so we're on our way to having computers create brand-new two-dimensional images that nobody's ever seen before based on a little tiny user input all right and then we're going to do it in 3d but yeah we haven't gotten there yet now one of the things this is applicable to is computer vision and this is trying to get computers to understand scenes in pictures and videos and this kind of thing and one kind is object recognition which is understanding the objects that are in saint's this is good for everything from driving all the way to understanding medical images now if I ask you what's hidden here you can probably know it's a monitor based on the context even if I told you it look like a sky you'd probably think well it's probably maybe it's a sky on a monitor and you'd be right but computer object recognition systems don't have that kind of wisdom that you do about what things tend to co-occur the way object recognition systems work is they take an image and they run a window over and they look for everything that they can recognize and they say oh yeah here's this here's that so here uh no face no face no face face okay good there's a face but what it doesn't do is it doesn't use what it's found before to constrain what it finds in the future so wouldn't it would do very badly at that that problem I just gave you right there because it doesn't have the context so what we can do is we can make this imagination system help the object recognizer make better guesses so when the object recognition says hey it's a sky our systems that you know what there's a less than 1% chance there's actually sky there's a 28% chance it's a computer screen maybe it's a sky on a computer screen or something like that another thing we've been doing is we've been teaching computers what the meaning of English spatial relationships are like close to and next to and above and this kind of thing a student of mine Connor Smith has been heading this project and what we've been able to do is get the you could type in something like cat below a tree and that's what it returns and there's hand occluding a book and a spoon close to a fork these are actual images returned based on its understanding so when I say this kind of thing people like oh this is an imagination cat in a tree you know it's easy right but this is there people are getting back to the first definition of imagination I mentioned which is the creativity but I just want you to think about the things you've seen that are really imaginative they all have a very firm basis in reality even the most imaginative worlds you've ever seen in movies or whatever they've got some kind of physics they've got characters that act basically like people alright and usually they tweak reality in very specific ways to make an interesting or entertaining world what this means is for us to get computers to come up with really creative really imaginative kinds of things we need a very firm understanding of the way the real world actually works it takes a good understanding of reality to make compelling fantasy the other thing that people often say to me is like well I'm interested in why why is it that I imagine something one way and somebody else manages it another way in psychology they call that individual differences and day to day we're all obsessed with that why am i different from this person and yes would you know we're being really creative you get differences but people take for granted the vast amount of similarity that we actually have if I told you I made oatmeal in my kitchen this morning and you picture that most of you are going to imagine a ground and a sink in a fridge it's only when were called upon to be very creative do we come up with very different kinds of things like if I ask you to imagine an alien kitchen for example so again you need to understand the basics of how everybody acts before you can get to the individual differences so in conclusion imagination is all around us it's it's it's fascinating subject and by automating it we're going to be able to make computers that can create intuitive visualizations designs for a that are good for education and maybe even someday they'll be able to create entire movies for us if you get me back for a TED talk in 20 years I'll let you know how the progress on that's coming when in the meantime well we're working on this and what are these future computer systems going to show us at this point we can only imagine thank you

9 thoughts on “TEDxCarletonU 2010 – Jim Davies – The Science of Imagination”

  1. it's crazy cause the iPhone now can pinpoint categories of photos based on objects the owner wishes to identify. like you can search furniture or hat

  2. But how do you make your imagination to work immediately we want what we want right now tomorrow can be too late?

  3. A robot cannot imagine pictures or visual.. Cause it has no brain to process information naturally to generate any imagination.. A Robot is merely a responding A.I constandly gifted in programming..

  4. interesting….i wonder what the study of imagination is called. Imaginology? you can't really make laws to govern imagination as they would be different for everyone. you can only make models the people fit into.

Leave a Reply

Your email address will not be published. Required fields are marked *

Tags: , , , , , , , , , , , , , ,