Andrew M. Olney

“ Several researchers have already used Wikipedia’s structure in models that emulate human semantic comparisons [25] [26] [27] . In this paper we extend their work in two significant ways. First, rather than focus on a single type of structure, e.g., link structure or concept structure, we present a model that utilizes three levels of structure: Word-word, word-concept, and concept-concept (W3C3) to more fully represent the cognitive-linguistic environment of Wikipedia. ”

“ Wikipedia is perhaps unique in that it provides moderately clean structural relationships in natural language. As a product of the human mind, Wikipedia reflects aspects of human semantic memory in its structure. Our W3C3 model capitalizes on the cognitive-linguistic structure at different resolutions just as theories of memory purport access at different resolutions: COALS represents words in word contexts, ESA represents words in concepts, and WLM represents the links between concepts. ”

“ Regressions conducted in these studies indicate that each constituent model explains a unique portion of the variance in the human data except COALS in Study 4, and that while their relative weights change slightly for each task, equal weighting is nearly identical to the regression-derived weights in all cases. This finding directly supports our stronger claim: By allowing both the language structure and category structure of Wikipedia to guide the W3C3 model, we achieved a higher correspondence to human semantic behavior than if we had used either separately. ”

“ Explicit Semantic Analysis (ESA) uses the article structure of Wikipedia without considering the link structure [41] , [42] . The intuition behind ESA is that while traditional corpora are arranged in paragraphs, which might contain a mixture of latent topics, the topics in Wikipedia are explicit: Each article is a topic, or correspondingly a concept. ESA defines term vectors in terms of their occurrence in Wikipedia articles. Because the meaning representation of each word is defined by its co-occurrence with article concepts, ESA can be considered as a word-concept level model. ”

“ We argue that distributional models by definition are much more aligned with semantic relatedness than with association. The reason is that semantic relatedness is a holistic judgment that considers many possible contexts. Distributional models are well aligned with holistic judgments because they are defined in terms of many contexts. Word association, on the other hand, can and does operate on a single dimension of a context. For example, in the NMS dataset, tumor has association with kindergarten cop. ”

“ We propose that by using vector spaces as a conceptualization, we have moved closer to unifying accounts of activation-based models and gist-based models, which are closely aligned with association and comparison-based tasks. Word association (Study 3) is a quite straightforward case of retrieval in the absence of a larger discourse context: Given a stimulus word, retrieve a response word. In contrast, semantic relatedness (Studies 1 and 2) is much more closely related to a comparison task. ”

“ The W3C3 model has significantly higher correlations than the constituent models on the entire dataset, this improvement is preserved across most semantic categories, the regression shows equal contribution by the constituent models, and the oracle shows the upside potential of these models is consistent with or perhaps slightly better than the previous best. All of these results support the conclusion that each constituent model’s semantic level, i.e., word-word, word-concept, and concept-concept, contributes positively to increasing the correlation with human semantic comparisons. ”

“ Gist-like operations are inherently holistic, as in comparison tasks, and use the entire vector representation. When we applied the standard W3C3 model using gist-like measures, the correlation to backward associative strength was 0.34. In contrast, activation-like operations are inherently local, as in word association tasks, and can use only a single element of the vector. Incorporating activation into the gist-like W3C3 model by removing WLM inlink vectors increased correlation to 0.42. ”

“ Word association norms represent a qualitatively different type of task than WordSimilarity-353 or MCSM. Typically in word association tasks, a human participant is presented with a word and asked to produce the first word that comes to mind. This production task is quite different from the raw data of MCSM, where subjects have the task constraints of listing physical, functional, or encyclopedic features and are asked to list 10 such features for a given concept. ”

“ Since previous results with LDA or LSA used a different corpus (TASA) , a direct comparison is not warranted. Nevertheless, the correlation of the W3C3 model is not as strong as has been previously reported for LDA. This result was surprising, particularly with regard to the low correlation for WLM. We undertook a qualitative analysis to determine if there is a better correspondence between Wikipedia’s link structure and the associative strength behind the DRM paradigm than is reflected by the WLM metric. ”

“ An alternative approach is to provide a comparable cognitive-linguistic environment to a computational cognitive model and observe the similarities between that model’s behavior and human behavior. There is an extensive literature using this approach to model human semantic behavior. ”

“ The most likely sense is the sense that has the highest similarity to the previous word. For example, if the previous word is baseball and the current word is bat, then the club sense is more similar than the flying mammal sense according to WLM. Then the inlinks/outlinks for these most likely senses were aggregated and used to create a synthetic gist article with the union of inlinks and union of outlinks. This gist article was then compared to the non-present target article in the standard way. As before, the W3C3 model was an unweighted average of these three scores. ”

“ There are some good reasons for taking this niche-construction concept seriously. It is perhaps trivially true that reading a book or similar work will have some effect on an individual, e.g., through learning. However, the argument being made here is stronger, that the influence of Wikipedia derives from both its language structure and its network of concepts/categories. Analogous to developmental studies [112] , [113] , one prediction would be that reading Wikipedia would affect a participant’s language structure and category structure. ”

“ Grouping pairs by semantic category, we calculated the average distance between predicted rank and the human rank for each model. The results are shown in Table 5. Since the ideal distance to the human ranking is zero, lower scores are better. The lowest score in each row is in boldface, and the second lowest score is italicized.
The most striking pattern in Table 5 is that two-thirds of the best scores per category belong to the W3C3 model. Moreover, for every category save one, the W3C3 model either has the best score or the second best score. ”

“ Humans both devour, and construct, the informational culture around them, thus producing a dynamic that some have termed niche construction in biological systems [2] . In niche construction, the behavior of an organism transforms its environment in a manner that can then facilitate the survival of the organism itself, thus producing a feedback loop. ”

“ However an important caveat is that in our models, an entire list of words is required to raise the activation level of the target dimension above the noise of the other dimensions. Thus this approach does not work for simple stimulus-response word association like the NMS task in Study 3.
7. Discussion
We believe that the results of Studies 1 through 4 substantiate the claim that humans and Wikipedia are part of the same cognitive-linguistic ecosystem. ”

“ In the past few millennia, the cultural institutions of human beings have produced a vast wealth of artifacts, and since the onset of written language, much of this creation has been linguistic in nature. ”

“ Just as beavers’ dam construction modifies its immediate ecology in which it lives, or particular species of trees can alter the nutrient content of a forest floor around them, human cognition and its external linguistic products affect each other through mutual influence: Over a short time scale, a single human extracts information from and adds information to this environment (e.g., linguistic input, reading, etc.) ”

“ Thus this task first ranks the human responses for a stimulus word by their associated production probabilities and then compares these to the model’s predicted ranking. If the first five predicted responses for all stimulus words are some ordering of the ranks 1–5, then the median rank will be 3. The second task is simply the probability that the model’s first response matches the human first response. The results from this previous work as well as the results from our three constituent and W3C3 models on this task are presented in Table 10. ”

“ As shown in Table 17, framing the model more in terms of activation rather than gist improves correlations considerably for ESA and WLM Activation models, such that the WLM Activation model has a non-significantly higher correlation with backward associative strength than the gist-type LDA model in Table 13 and W3C3 model presented in Table 15. For WLM this is perhaps not surprising because of how the outlinks on Wikipedia pages are generated in the first place: by people. ”

“ If a cognitive scientist is interested in the immediate influences on language behavior—communicative goals, lexical knowledge, and so on—it may make sense to focus on cognitive theories that best explain the rapid deployment of such knowledge and processes. ”

“ As we will show in the following sections, each of these levels independently contributes to an explanation of human semantic behavior. Secondly, in addition to the common dataset considered by previous researchers using Wikipedia, the WordSimilarity-353 [28] dataset, we apply the W3C3 model to a wider array of behavioral data, including word association norms [29] , semantic feature production norms [30] , and false memory formation [31] . Studies 1 to 4 examine how the W3C3 model manifests language structure and categorization effects across this wide array of behavioral data. ”

“ Each stimulus response pair is annotated by the proportion of participants who produced it, which we refer to as forward associative strength. We refer to these 72,176 triples (stimulus word/response word/forward associative strength) as NMS after the last names of its authors.
Previous work using a type of distributional model called a topic model (also known as latent Dirichlet allocation) and the TASA corpus [85] to train it, used the NMS dataset on two tasks [86] . ”

“ However, if a cognitive scientist is interested in understanding longer-timescale phenomena—such as cultural or linguistic change, or language evolution and origins—then a broader set of variables becomes relevant. ”

“ The first model, the Correlated Occurrence Analogue to Lexical Semantics [32] , operates at a word-word level. The second model, Explicit Semantic Analysis [33] , [34] , operates at a word-concept level. The third and final model, Wikipedia Link Measure [35] ., operates at a concept-concept level. We then describe a joint model (W3C3) that trivially combines these three models. ”

“ In order the assess the relative contributions of each constituent model to MCSM performance, a linear regression was conducted. The regression used COALS, ESA, and WLM raw scores to predict the MCSM similarity scores. The results of the linear regression are presented in Table 9. Tolerance analyses were conducted to test for multicollinearity of COALS, ESA, and WLM by regressing each on the other two. ”

“ The corresponding phi coefficient [38] is
In addition to transforming each cell value into its corresponding phi value, COALS “sparsifies” the matrix by replacing all non-positive cells with zero, such that for any cell value υ
Thus the final representation for a given word is its associated row vector, whose only non-zero components are positive correlations between that word and context words. ”

“ Menlo Park, CA, USA, 2005; pp. 554–559. ↑ Harris, Z. Distributional structure. Word 1954, 10, 140–162. ↑ McNamara, D.S. Computational methods to extract meaning from text and advance theories of human cognition. Topics Cogn. Sci. 2011, 3, 3–17. ↑ Wikipedia. Wikipedia: Statistics. 2007. Available online: http://en.wikipedia.org/wiki/ (accessed on 8 February 2011) . ↑ Gabrilovich, E.; Markovitch, S. Computing Semantic Relatedness Using Wikipedia-Based Explicit Semantic Analysis. ”

“ Wolfman, G.; Ruppin, E. Placing search in context: The concept revisited. ACM Trans. Inf. Syst. 2002, 20, 116–131. ↑ Manning, C.D.; Sch ̈utze, H. Foundations of Statistical Natural Language Processing; MIT Press: Cambridge, MA, USA, 1999. ↑ Milne, D.; Witten, I.H. An Effective, Low-Cost Measure of Semantic Relatedness Obtained from Wikipedia Links. In Proceeding of AAAI Workshop on Wikipedia and Artificial Intelligence: An Evolving Synergy, Chicago, IL, USA, 13–14 July 2008; AAAI Press: Chicago, IL, USA, 2008; pp. 25–30. ↑ Milne, D. Wikipedia Miner. ”