"it's training on wikipedia. It's learning the meaning (and multiple senses) for every word it can find."
I don't think that is entirely correct. After cursory reading of the paper, my understanding is that they look up a list of word senses for each word in a dictionary (or multiple dictionaries). And then they try to learn something about each of those word senses from wikipedia (that is they create seperate word embeddings for each of those senses). So what they do not do is to learn what senses a word has. That is done by the humans who created the dictionaries.
What that means is that they cannot pick up new senses of words, which doesn't matter for answering IQ test questions because these questions rarely change and are typically based on well established word meanings.
Unfortunately it makes this approach less than ideal for things like understanding the news (something I'm working on), where new contexts of words keep popping up all the time.
I don't think that is entirely correct. After cursory reading of the paper, my understanding is that they look up a list of word senses for each word in a dictionary (or multiple dictionaries). And then they try to learn something about each of those word senses from wikipedia (that is they create seperate word embeddings for each of those senses). So what they do not do is to learn what senses a word has. That is done by the humans who created the dictionaries.
What that means is that they cannot pick up new senses of words, which doesn't matter for answering IQ test questions because these questions rarely change and are typically based on well established word meanings.
Unfortunately it makes this approach less than ideal for things like understanding the news (something I'm working on), where new contexts of words keep popping up all the time.