Sunday, May 31, 2015

Conscious Computers

[Note that more than my other posts this one is incomplete and continues to evolve when I have time.]

Lately my thoughts have been returning to AI, the discipline in which I earned my doctorate 27(-ish) years ago (during the 2nd AI winter) before drifting into bioinformatics, where I spent my subsequent career.

To date AI has underperformed the expectations of those early years, when it was assumed that any attempt to “get computers to do X”, where X was some uniquely human, cognitively challenging task, would be met with immediate success. Numerous Ph.D. theses from that era exhibited trivial computer programs which the author claimed proved some general principle or other, but which in retrospect proved only the ease with which we can delude each other and ourselves into overinterpreting the significance of a single toy program.

The Turing test, it turns out, is both too hard and too easy: too hard, because its anthropocentrism excludes too many forms of nonhuman (alien or machine) intelligence, and too easy, in that people will often attribute intelligence to laughably simple contrivances, from Clever Hans to the (original, not Amazon) Mechanical Turk, to Eliza. The last, intended by its author as a parody of AI, surprised him in its ability to convince its interlocutors of its intelligence. The underlying mechanism was a trivial sort of pattern matching: if a keyword appeared in the input, a plausibly relevant canned response would be selected. No real understanding of the input was attempted, or required. (Husbands/wives, parents/children and teleconference multitaskers often use this strategy.)

There has been progress in the intervening years, mostly in identifying particular tasks that can yield to specialized algorithmic solutions: chess playing (Deep Blue beats Kasparov, 1997) route planning, recommending videos or music, driving cars, etc. Image understanding and speech recognition have made significant incremental progress, while still falling well short of human abilities. The high water mark of more general intelligence, at least as judged by publicity, was Watson's victory over Jeopardy world champion Ken Jennings. During the tournament Watson displayed its top 3 guesses for each question, along with its estimated probability that each was correct. Revealingly, even on questions for which Watson’s first choice was the correct answer, its second choice was so wildly off base as to reveal how shallow its understanding of the question was, e.g. an animal where the question clearly called for a city. Watson, it appeared, was just a more convoluted version of Eliza.

            More recently we have seen the emergence of voice-recognition agents such as Siri, Google Voice, and the like – see recent review. For the most part these are not (yet) conversational, but rather handle single request-response transactions, where the response is limited to a particular menu of behaviors: iphone activities such as phone calls, email, text messaging, calendar, directions, web searches, etc., in the case of Siri, and Google searches in the case of Google voice. For both, the voice recognition capability essentially functions as an alternative user interface to the application(s) without adding significant new behaviors. If your iPhone was not going to pass the Turing test without Siri, Siri is unlikely to tip the balance.

            The recent development of deep learning algorithms has propelled a wave of AI progress in machine vision, speech understanding, and other tasks, yet they have not yet enabled these systems to cross the tipping point into recognizable intelligent behavior.

What is missing in our efforts to capture intelligence in machines? This post is a random walk over a range of topics; what links them is the search for necessary conditions, or sine qua non’s, of intelligence: those features without which a computer will not (or should not) be considered intelligence.

Three Perspectives of Consciousness

  Lately my speculations on this question have focused on consciousness. What does that word even mean? We all think we understand what consciousness is, because we experience it. Descartes argued that our own consciousness is in fact the only thing we can be certain of: cogito ergo sum. This is consciousness from the inside, which is very different from the way we experience anyone else’s consciousness. Descartes argued further that we can’t really be certain that anyone else is conscious, an argument that, if taken seriously, leads to solipsism. Of course, our inability to be certain of anyone else’s consciousness is in part due to the opacity of their mental processes; if we introduce technology which lets us eavesdrop to some (currently very limited) extent, such as EEGs, fMRIs, or deep brain electrodes, we can perhaps detect correlates of consciousness in another person, enough to distinguish, say, brain dead from not. However such technology would be of little use for judging whether a nonhuman – e.g. an alien, a computer, or an ant – is conscious. On the other hand, we would have a unique level of introspection (or extrospection?) into the mental activity of a candidate conscious computer, far exceeding our own rather limited powers of introspection, since we could examine its code and its mental state to arbitrary detail. What would we need to observe in the computer, extrospectively, in order to become convinced that we were witnessing consciousness?

We can thus distinguish three distinct vantage points, or perspectives,  from which we can judge consciousness or intelligence: the introspective (experiential/subjective/Cartesian), i.e., our own consciousness; the interlocutive (or black box), that of another agent viewed from the outside, which is the perspective of the Turing test; and the extrospective (objective/glass|white-box) available to sufficiently well-instrumented neurobiologists or (for artificial intelligences) engineers. A complete theory of consciousness/intelligence would need to reconcile these perspectives.

A key element of our own (subjective) consciousness is a sense of what is happening now. We are not simply conscious; we are conscious of: our thoughts, or bodies, our senses, and thereby also our environments. These could easily be modelled in a program, as was done in early AI blackboard systems: you just need to arrange for the senses and/or thought processes to update a database. A trivial form of autobiographical memory can be added by timestamping the updates.  Such a system would have a kind of idiot-savant memory: it would remember every detail of everything it ever sensed or thought, yet it would learn nothing from experience. It would lack the sense of immediacy we feel about the recent past, the fovea of our attention, since memory would be no less vivid after ten or a hundred years than after a few seconds.

       Is it possible that our sense of the vividness of the present, and the immediate past, is an artifact of the poor quality of our memories? The present for us is like a continuous sequence of flashbulbs, brightly illuminating each moment, but then fading away to sepia, the static, imperfect reconstruction of our memory. The present is in color, like Oz; the past is Kansas, in black and white. Is this a bug of our consciousness, or a feature, or part of its very definition? We retain the significant parts of our experience, if we are lucky – a kind of lossy but very clever data compression.

                Perhaps forgetting, or loss of detail, is neither necessary nor sufficient to turn memory into experience. Maybe it is the abstracting of salience, the paying of attention to particular details, either as they occur or in retrospect, that is critical. Certainly a necessary feature of intelligence (not the same as consciousness? Later…) is the ability to learn from experience. For us, this may be tied into forgetting, or rather to choosing what of fast-fading memory to preserve. For an intelligence with perfect recall, there would still remain the question of what features of memory to learn from, and what to learn from them. An intelligent being ought to be able to revisit its memories and learn new things from them – whether in light of subsequent information, or subsequent pondering. Humans mull; we chew our intellectual cud. Maybe intelligent machines wouldn’t need to.  Mulling is an artifact of compute resource limitations; if you have enough compute power, and an accurate memory that didn’t need to be prodded by trial and error with different cues, you could infer whatever you are capable of inferring from experience as it happens; only new information would justify reanalyzing old memories.

                On the other hand, computers would not necessarily have perfect memory. Modern computers keep track of file creation and modification times – timestamps for their long term memory – but not of every change made to working memory in the course of program executions. The cost of doing so would be prohibitive, and the results rather boring: I added one to this register, then moved data from this memory cell to that one... . So they too have sort of flashbulb present – the most dynamic parts of memory, called the registers, are where computation acts most efficiently, sort of like the 7+/-2 objects that humans can hold in working memory. Low latency cache memory holds recently used data, giving something like a sense of recency. The mechanisms that move data from short term to long term memory in our brains is quite complex, but fairly trivial in computers: the program writes output to files or databases, or not; all other information held in working memory is lost when the program exits or the computer reboots. The computer will not remember the cool idea it had in the midst of running Microsoft Word last Wednesday unless it was saved to a file. (Never mind that it will never have a cool idea while running Word…) Computer users often review the order of changes to files in a folder, or a folder hierarchy, by sorting by creation or modification date, which gives a sort of “autobiographical” view of the computer’s activities. In forensic applications – e.g., when trying to examine the effects of a computer virus infection, or reconstruct the user’s activities around the time of a crime or disappearance – one often does a complete chronological dump of all file changes around the time period of interest, usually revealing evidence of multiple processes that acted more or less in parallel on "long term memory".

                Despite the metaphorical parallels, file timestamps feel like a rather shallow and unconvincing model of human episodic/autobiographical memory. What is missing? A key aspect of our own memory is that there is a self who is doing the experiencing, and the memories are of the experiences of that self. The computer may have memory but it does not have a self, or experience. What does that mean? Self and experience, like consciousness and intelligence, are murky, ill-defined concepts that nonetheless cut close to the core of why the computer’s information processes feel so different from our own. Can we clarify their definitions, and perhaps find a way to close the gap and make the computer more closely resemble us? Can we give the computer a self? What magic word must we write on the golem’s forehead in order to endow it with life?

Some (e.g. Searle, Weizenbaum) have argued that the problem is building the golem from clay to begin with, and that arguing over the recipe is beside the point. If true, that still leaves us wanting to understand in what way clay falls short – is it fundamentally the wrong material, and if so, why, or is the fault in the recipe?

                To reduce the number of poorly defined concepts, let’s assume that a self is something that experiences, experiencing is what a self does, and that a self that is experiencing is conscious. These mutually referring definitions reduce the 3 poorly defined concepts to one. There may be corner cases for philosophers, physiologists and mystics to debate: various brain-damage syndromes, altered states of awareness, or meditative practices that purport to create experience without consciousness, consciousness without self, or what not. But let’s start by addressing the base case.

                Given that almost any computer has a sort of sensorium that registers a (somewhat limited) set of “sense impressions” which it can to some extent recall (e.g. keyboard input), why shouldn’t we call what it does “experiencing”? One could argue that file timestamps are in fact a perfectly good model of episodic memory, but that the computer’s experience just happens to be rather boring. “Remember that really big Word file I edited last Wednesday?” is not the stuff of high drama. Also, the computer has no “I”, nor “not I” (environment). Descartes tells us that the distinction between I and not I is just a set of labels associated with different sensory inputs: input from the eyes is labelled environment, proprioception and nausea are labeled body. It would not be difficult to implement similar labels for the computer: disk drive is internal, printer is external. That would seem to buy us very little in our quest to create a self. Indeed, our (or at least my) intuitions say that the various self-in-a-bottle constructions, such as Dalton Trumbo’s unfortunate hero Joe Bonham in “Johhny Got His Gun”, or, increasingly and tragically, Steven Hawking, are still selves despite lacking environments (another corner case…), so an environment/self distinction may not be a prerequisite for a self..  

                What if we add “interpretation” of the experienced events to the computer’s activities – classification, parsing, object identification, what have you? Does that transform rote data capture into “experience”? On the one hand, it is hard to see how this is any different from just adding another “sensory” channel whose content is the stream of higher level interpretations. If a primary sensory channel is not sufficient to constitute “experience”, why should a more derived channel be? Both are streams of symbols, recorded with some level of fidelity in long-term memory. Are some symbol streams “conscious” and others not? Is the question even meaningful?  Perhaps if consciousness is understood to have gradations rather than being an all-or-nothing attribute, we could say that some symbol streams are “more conscious” than others.

Models consisting of symbol streams dumping into a central database fit the definition of a blackboard architecture. Is consciousness necessarily some sort of blackboard architecture? Is it a special sort, or any sort? If special, in what way? Is the claim “consciousness is a blackboard architecture” falsifiable? If not, is it meaningful? What is a non-blackboard architecture? Is there an alternative, or is this just a way of describing any information processing system at all? Is “blackboard architecture” ultimately as ill-defined as “consciousness” and “self” – despite referring to an engineered rather than a naturally occurring construct?

What does “stream of symbols” even mean? Stream, in the computer science sense, seems well grounded in mathematical notions of sequence, though the implications of discreteness and total ordering of the events could both stand some scrutiny. Symbol is more difficult; the naïve definition presupposes an interpreter of the symbol for whom the symbol is meaningful, but if symbol processing is the mechanism of meaning this definition seems to lead to an infinite regress of interpreters within interpreters. Is a symbol different from a signal? If a symbol is a physical phenomenon, exactly which physical phenomena qualify as symbols?  Perhaps to be a symbol means to have a meaning, whatever meaning means. Meaning, we might suppose, is established by chains of causality linking symbols to sense impressions, to actions, or to other symbols.  Is this a useful idea? Chains of causality are everywhere; again, which ones qualify? How can a symbol whose referent does not exist – e.g. unicorn – acquire a meaning, if it has no sensory correlate in the world?  (Unicorns may not exist, but horses and horns do…) How do abstractions – democracy, corporation, cynicism, force, intention, infinity – acquire meaning, when their mappings to sensory correlates are at best complex, and at worst nonexistent? If the causal chains –afferent and efferent – linking a symbol processor to the world are destroyed, as in the brain-in-a-bottle scenarios, does the symbol still mean anything? If so, then meaning does not depend on those connections; if not, then one is left with the absurd conclusion that the same computations performed on the same inputs and yielding the same outputs are sometimes meaningful and sometimes not, depending on whether the I/O connections are working. Philosophers and logicians have ploughed this terrain for generations with little to show for it.

And yet we routinely encounter these same questions in connection with computers, and the answers are obvious and uncontroversial. Computers work with all sorts of “symbols” – numbers, text strings, images, sounds – and we have no problem attributing “meaning” to these data structures. The “meaning” of a .gif file is an image, because it was created by a camera (or drawing program or other “image creation” program – a circular definition, but not a problem in practice), and because it can be rendered as an image on the screen or by a printer, edited by an image editor, etc. I could take any same-sized chunk of memory off the disk and render it as an image as well, but most such chunks, containing code or other data formats, would produce “meaningless” static. Why some chunks of bits are “images” and some are not is hard to say, but in practice this issue does not get in our way.

We could say that the image file is an image file because, once it is rendered on a screen or printed, a human observer perceives it as an image. In this view the human assigns the meaning; the data structure is meaningless without the human interpreter. If an image is processed in a computer and nobody sees it, is it still an image? This interpretation again leads to the conclusion that the same data structure is a symbol, or not, depending on whether a human interprets it. And furthermore, that the same sequence of symbol-processing operations, up to and including a computational model of consciousness, would be meaningful or not, depending on whether a human looked at the output. This seems to ascribe to humans a magical power of meaning endowment, similar to the idea that only consciousness can collapse quantum-mechanical wave functions – an idea without much to recommend it, in my view.

            Note that the definition of meaning here is subtly different from the linguistic one, which is defined in terms of the user’s intent. We cannot appeal to user intent to explain the mental constructs making up the user’s intent. A better conceptual toolkit, perhaps, is model theory, a branch of mathematical logic.

If I were to assert that a certain program or device performs a well-understood computation, such as matrix multiplication, this would be a falsifiable claim. In order to be true, we would need to identify certain components of the system as the input and output matrices, and understand their encoding; then we would have to show that the relationship between encoded inputs and outputs satisifies (or approximates) our definition of matrix multiplication. Indeed the activity of program verification (or software testing) is designed to do just that. If we had a comparably precise definition of consciousness it should be similarly possible to verify that a program is carrying it out. The difficulty is providing the definition – in software engineering terms, the specification for consciousness.
If we sweep aside the concerns about how symbols acquire meaning, then, relying on our now everyday experience of computer symbol processing to ward off the philosophical gremlins, a specification for consciousness might include an (optional – Johnny) stream of perceptual input symbols, an (optional) stream of behavioral output symbols, an internal stream of “thought”, and a long term memory for past streams. Are these features sufficient? By this criterion we may have conscious systems now. Facebook, for example, takes in a stream of sensory inputs in the form of images uploaded by subscribers, and it recognizes higher-level concepts in them by tagging individual faces. Is Facebook conscious?

Meaning in Human Language

            The question of how the internal symbols of an intelligent agent can be said to acquire meaning should not be confused with the question of how human language can be understood by humans or machines. Given a collection of meaningful symbols inside a conscious agent, natural language understanding is the process of translating sequences of words into those meaningful symbols. If language is ambiguous, the translation must create alternative unambiguous interpretations; if language has multiple layers of meaning the translation must find them all – or at least as many as a typical human would.
Consider, for example: Two roads diverged in a wood, and I—I took the one less traveled by, And that has made all the difference. What does this mean – to us humans or to a computer? There is a literal meaning about a fork in the road, a navigational decision, something Siri might be able to relate to its GPS application, as well as a reference to unspecified causal consequences of that choice. For us humans there is a shower of implications about life choices which may vary completely from one listener to the next. There are also interpretative links to Robert Frost’s biography. Critics continue to debate the meaning. So we have, at least, many possible meanings, or perhaps none. Could it be that the reason this poem – maybe any poem? – can sustain so many meanings is that it doesn’t actually mean anything? Is it just a stimulus intended to provoke reverberations in the meaning-assigning engines of our minds?  Should the machine emulation of consciousness require the ability to interpret these lines in a human-like way? (The ability to write plausible-sounding poetry has already been demonstrated and is being propelled forward by Turing-test-like competitions.)
            Of course we can say that poetry is a special case, perhaps too uniquely demanding of human as opposed to intelligent cognition. So how about a more practical discipline, such as politics? What should we – or a computer – understand from a statement like Donald Trump’s I will build a great, great wall on our southern border, and I will make Mexico pay for that wall. Again we have a layer or more or less literal meaning: a statement about a future plan to construct a barrier and to finance the construction. There are layers of meaning above the literal: the implication that the wall’s purpose, and effect, will be to prevent illegal immigration from Mexico into the United States, and that Mexico will somehow be coerced into paying for the wall.  At another level, this statement is about Trump himself, helping to establish his brand identity as a candidate. It says that he is strongly opposed to illegal immigration, that he will take forceful actions to prevent it, such as building a wall or coercing any ally; that he is a man of action; that he is a strong ally of those who also oppose illegal immigration, etc. Furthermore, this statement implicitly compares Trump to his opponents: he is more anti-immigration, a more forceful leader, a man of action, than they are, and therefore more suited for the job of being president. Finally, to his critics, there is a layer of meaning that says he is completely out of touch with reality, that building such a wall is impractical, or that it would not prevent illegal immigration even if it was built, that Mexico would never pay for it, and perhaps that illegal immigration isn’t such a big problem anyway. If Trump is out of touch with reality then he is (arguably) unsuited for the job of president. A student of political history might further place Trump’s statement in the context of historical appeals to nationalism, xenophobia, populism, etc., in other times and places, or wonder what factors in the current moment make such appeals compelling now, and to whom. How many of these layers of meaning would we expect an intelligent system to discern?
            Of course, it may that politics was a bad choice of a counterpoint to poetry: it is said after all that we campaign in poetry but govern in prose. Where should we look for precise speech that wears its meaning on its sleeve? Perhaps science. Consider the following journal article title; SMYD2 overexpression is associated with tumor cell proliferation and a worse outcome in human papillomavirus-unrelated nonmultiple head and neck carcinomas. Here, although the literal meaning is perhaps more complex than building walls or taking a fork in the road, there appear to be fewer layers of meaning. In a certain well-specified context (human papillomavirus-unrelated nonmultiple head and neck carcinomas) a certain property (SMYD2 overexpression) is found to predict certain outcomes (tumor cell proliferation and a worse outcome). The context, property and outcomes require certain technical knowledge to understand, but the overall proposition is relatively straightforward. There is another, implicit, layer of meaning related to novelty and scientific contribution however. This title is asserting that the role of SMYD2 overexpression in predicting outcome for these cancers is new knowledge, and that it may have practical applications in treating the disease. An oncologist reading this might be led to wonder if any of her patients with this type of disease exhibit SYMYD2 overexpression in their tumors, whether she could determine that, and what she might be able to do with the information.

            Science and poetry are not exclusive, and not all scientific statements are as prosaic as the one above: consider “Chance favors the prepared mind”, “God does not play dice with universe”, or “It has not escaped our notice that the specific pairing we have postulated immediately suggests a possible copying mechanism for the genetic material”. The unraveling of the layers of meaning in these statements will be left as an exercise for the reader.

These examples demonstrate the task of inferring the meaning of a sentence is open ended, and not reducible to a simple translation of the sentence into a unique assemblage of symbols. The task of a natural language understander, therefore, should be understood as involving ferreting out as many meanings as possible from a sentence, or at least many possible meanings. It may be that some notion of plausibility is required to prune the space of possible interpretations to rule out the wildly implausible and/or to limit the computational work. For example, Trump has been involved in construction projects, is he proposing a wall in hopes of getting a share of the construction contracts for himself or his allies? (Is this implausible?).  

Feelings and Stories

               One idea that might bear on the definitions of “self” and “consciousness” is “feelings”. This is as poorly defined as all the other ideas in this arena. Are feelings part of, identical to, or a necessary component of “experience” and hence of “consciousness”, or are they incidental, part of human psychology but not essential to a suitably broad definition of consciousness? “Feelings” have an interesting double meaning of body awareness and emotion.  This is suggestive of certain theories that argue that only an embodied automaton could be conscious, which in turn would deny consciousness to Putnam’s brain in a bottle or Trumbo’s Johnny. Without going down the embodiment rabbit hole, I will take a detour on the subject of feelings, try to clarify what they are, and then return to the question of their importance to self/experience/consciousness.

                Some feelings are clearly rooted in the body: pain, pleasure, nausea, pressure, temperature (cold/warm/hot/burning), hunger, thirst, orientation-related sensations, including dizziness, as well as the major senses: sight, hearing, touch, taste and smell. Emotions, as distinct from bodily feelings, may also have physiological components – the adrenaline rush of fear redirecting blood flow away from the gut and towards the muscles, making the body ready for action, or the peripheral vasodilation of the facial capillaries that creates a blush – but they also layer on complex cognitive elements that relate to one’s model of the world. The spectrum of emotional terms in English is large: awe, admiration, love, happiness, fear, joy, embarassment, disgust, triumph, hatred, boredom, irritation, despair, contempt, confusion, anger/rage, nostalgia, shadenfreude, pity, horror, suspense, affection, surprise, loathing, envy, desire, lust, greed, pride, shame, guilt, sorrow, regret, jealousy, compassion, humiliation, scorn, reverence, spite, blame, longing, empathy, sympathy. And that is just the nouns; there is an equally long list of adjectives: distraught, overwrought, zealous, cynical, sarcastic, affable, melancholy, ill-humored/bad-tempered/temperamental, anxious, bemused, lazy, effortless, depressed, careworn, vengeful,  manic, courageous, heartbroken, outraged, solicitous, humble, abject, contemplative, ambivalent, respectful, tentative, anxsious, amorous, lonely, playful, carefree, humorous, serious, stern, histrionic, hysterical,  nervous, deranged, uncertain, reckless, cowardly, shameless, flighty, frustrated, earnest, timid, committed, curious, menacing, vicious … . Is there a finite number of “primary emotions” from which these variations are constructed, like primary colors, or are they each unique, like smells? (For that matter are there primary smells?) (Relevant Links: Wikipedia Classification of EmotionsAnotherEmotion Machine Classification of Emotions.)

Emotions have an evaluative component: they relate to a classification of past (remembered), present (experienced) or future (anticipated) states as being positive or negative for the interests of the self: fear = anticipation of a negative future event; hope = anticipation of a positive future event; sadness = recall of a negative past event; joy = experience of positive present event, etc. In the early 80's AI researcher Wendy Lehnert developed a theory along these longs to parse the structure of stories: this was to my mind one of the more substantive (and underappreciated) insights from that era of AI research. She defined a network representation for basic stories, in which nodes, representing events and actors, are linked by edges, representing causal relationships. The edges have plus or minus signs representing positive or negative valuations of the events for the actors. A complete set of basic networks could be constructed from these components. These basic networks, called plot units, ranged from simple ones like “success”, “failure”, “loss”, “tradeoff”, “problem”, as well as more complex social ones like “threat”, “promise”, “bungled request”, and so on. Plot units could be used to analyze, and to (arguably, in some sense) “understand” stories, by recognizing the plot units in them. (Kurt Vonnegut developed a related idea of story structures for his master’s thesis – see article, video).

Could ongoing analysis of the plot units of one’s own autobiographical story be the basis for emotions? Could that ability be the difference between rote memorization and experience? This view shares something with the embodied consciousness idea, but here it is not the body itself that is key, but the idiocentric viewpoint, on which the valence of the events – the signs on the edges of plot unit networks – are based: good for me vs. bad for me.

                If a computer could analyze its experiences on an ongoing basis to assemble fragmentary stories about how what it has observed to happen relate to its interests, would that start to look like consciousness? Maybe. If so, then to proceed along this route we need to understand how it could have interests, and how it could perform such an analysis.

            When we watch stories in the form of plays or movies, or listen to recounted stories, we often replace our own idiocentric viewpoint with that of the protagonist – we identify with the protagonist – literally taking on their identity in our minds, through processes of sympathy or empathy that cause us to feel the emotions of the protagonist. Does this mean that a true idiocentric viewpoint is not necessary for consciousness? Are we not conscious while watching a movie? Could an intelligent agent empathically substitute someone else’s idiosyncratic viewpoint for its own?  Or must it have an idiocentric viewpoint in order to empathize? Maybe identification is a analogy mapping the protaganist’s perspective into one’s own idiocentric viewpoint.

Ultimately, how convincing is the case for emotiona experience as a sine qua non of intelligence? Emotionless intelligence is a common meme in fiction: consider Mr. Spock, or Rainman, or HAL in 2001, (or the monolith builders), or psychopaths; so we can at least imagine intelligence without emotion. My current view is that an intelligent agent should be able to understand emotions in stories, but it is not clear that it must experience them, any more than an adult needs to experience the tantrums of a two-year-old in order to understand them.

Degrees of Consciousness, Locomotive Essense and Deceptive Simplicity

Earlier I referred to “corner cases” of consciousness: brain-damage, altered states of awareness, animal consciousness, baby consciousness, and so on. These stand in contrast to the unitary “you have it or you don’t” view of consciousness. What if we take the opposite view, that there are degrees of consciousness, some sort of spectrum. We might even identify that spectrum with “intelligence”. Does the PC with its timestamped file-saves deserve a place on that spectrum?

            Why a spectrum, anyway, with its implication of an ordering between more and less conscious states? Is a dog less conscious than a human? An adult more conscious than a baby? A wide-awake adult more conscious than the same adult on the edge of sleep, or wakefulness, or emerging from anesthesia? More controversially: is an English major more conscious than a math major? An Italian than a Japanese? Someone on the autism spectrum than a neurotypical? A cocker spaniel than a german shepherd? A cro-magnon than a neanderthal? A human than a chimp? Maybe “spectrum” is the wrong metaphor. Maybe “consciousness” is simply a collection of capabilities, none of them necessary, and perhaps no subset of them even sufficient, to justify a Boolean “is conscious” label.

            Consider the analogy of a car, which possesses a mysterious property called “locomotion”, which is in some sense the essence of car-ness. (A car lacking this property is considered a toy, a sculpture, or broken.)  We can easily appreciate degrees of deviation from the locomotive ideal: one cylinder is not firing, the starter sometimes doesn’t work, one tire is flat, the radiator is broken, the engine burns oil. These “altered states of locomotion” are not mysterious; they correspond to suboptimal functioning of components of a complex machine. Conversely, some cars exhibit positive altered states of locomotion: unusually high power engines, or extremely good gas mileage, or smooth ride. Again, no mystery, simply unusual performance metrics in particular subsystems of a complex machine.

In this view, we should give the whole idea of “consciousness” the heave-ho, and replace it with a dashboard of indicators of the performance of the various subfunctions that contribute to it. Here I have returned to my starting point, asking for a list of features that are necessary and sufficient for consciousness.

            The locomotive property of cars shares with consciousness the property of deceptive simplicity. When the car is working, it seems simple: a few controls – the ignition, accelerator, brake and steering wheel – suffice to control the locomotive system. Only when components fail does the underlying complexity become apparent: that a functioning ignition switch, battery, alternator, starter motor, and contacts are needed for ignition to occur, not to mention fuel, engine and carburetor. Consciousness also presents a deceptive simplicity to the “user” (itself?) – the illusion of a unified self, which breaks down under psychological stress, neurological damage, and other unusual conditions. Other systems with this property of hidden complexity underlying superficial simplicity include the Google user interface, and computer or device user interfaces generally, as well as “health” – the category of things that seem simple until they break.    

Definitions vs Prototypes of Consciousness

            The process I am engaging in here resembles the one described by Imre Lakatos in Proofs and Refutations, his rendering in dialog form of centuries of mathematical debate over the question “What is a polyhedron?”. Different characters propose different formal definitions of polyhedrons, to which other characters respond by proposing counterintuitive examples that satisfy the definition but do not fit anyone’s preconceptions of what polyhedra should look like. These so-called monsters force the original definer to search for a more restrictive definition which is in line with intuition, or to reject their preconceptions in favor of a broader view of polyhedronicity. Should we broaden our definition of consciousness to include personal computers, or restrict it so as to exclude them, and if so, what restriction is appropriate?

            Maybe the entire exercise is meaningless, an empty game of grouping things into taxonomies, like Aristotelian/Linnean classification of organisms. Once Darwin’s insights revealed the basis for a naturalistic taxonomy, the older efforts were seen to be castles in the air. The question of whether raccoons are closer to bears or to weasels can now be answered with DNA sequence analysis rather than scholastic debate (the answer is bears). 

Here however the question is not one of genealogy, but of convergent evolution. Computer and human mental operations have evolved by very different mechanisms, and the convergence is only partial. This statement again would seem to call for an elucidation of the ways in which the convergence is incomplete.

Maybe consciousness does not have a definition at all, but is rather a “we know it when we see it” type of concept in which we mentally assess the similarity of something to our prototypical concept of a conscious organism – ourselves? It could be that most human thought works this way – by relating the thing in question to the most similar thing(s) in our memories, and drawing conclusions by analogy. This could explain why so many mental concepts – consciousness, intelligence, mind, thought, meaning, God – seem to resist analysis: we try to reason rigorously with things that were not rigorously defined to begin with. Maybe it makes no more sense to ask for necessary and sufficient conditions for consciousness than it does to ask for necessary and sufficient conditions for love, art, beauty, wisdom or humor.

Cognitive Actions

Consider cognitive action words, such as believe, decide, wonder, conclude, will, wish, hope, assume, ask, despair, suspect/guess/ hypothesize/conjecture, estimate, extrapolate, reject/accept, deny, understand, argue, convince, agree/disagree, consent, affirm, declaim, decry, refuse, accuse/decry/absolve/forgive/blame/vilify/slander, lie/deceive, dream/fantasize/imagine/pretend/suppose, think/reflect/ponder/review/consider, revise, soliloquize, fancy (form an attachment to), forget/remember, learn, symbolize, analogize, obsess, support/oppose, accept/refuse, dismiss (in idea), design/plan/ plot/compose/contrive/invent/envision, solve, infer/deduce, confirm/prove/refute, predict, explain, expect, object (verb),  persuade, command, deny, confuse, concern/interest (one’s self with), amuse (one’s self), restrain (one’s self), monitor (one’s self or something else), get distracted, observe / perceive / watch / listen / hear / feel / smell / taste, enjoy/fear/hate/anticipate etc. –  emotion words used as verbs.

The list is long and incomplete. I have organized it slightly to highlight synonym, antonym and related meaning clusters. As with emotions, one could (and I may) invest time in developing a taxonomy, a theory, perhaps a finite set of elemental cognitive actions from which all others are compounded. Some of these cognitive terms seem closely allied to bodily senses (observe/listen/envision), emotions (blame/forgive/fancy), some to evidentiary processing (guess/conclude/prove/confirm/accept), some to creativity (pretend/imagine/design/invent/envision), some to communication (persuade/command/slander). The boundary between communicative concepts and internal cognitive actions is somewhat porous: many terms that seem communicative can also be applied to one’s self: persuade/deny/lie/forgive/blame/amuse. Perhaps these relate to a metaphorical (or actual) notion of thinking as self-communication. Some, like decry/slander, seem to lie in the realm of social communication between a communicator and a community, an audience wider than one individual.  Is there a communicative spectrum that runs the gamut from self-communication to pairwise to broadcast (or oratory)? Soliloquy (spoken communication with one’s self) occupies an anomalous place on that spectrum; so perhaps does conspiracy (communication within a small group in isolation from a larger group). The notion of a spectrum from inner to outer suggests that our internal experience is similar to our linguistic communications: consciousness as internal monolog. (Try googling do we think in words). While there may be an element of truth to this model, it seems clear that much of mental life is pre-, sub- or non-linguistic, else why would effort be required to put our thoughts into words? Conversely it is clear that we can compose, revise, recall, and rehearse language in our heads, so thinking with or about words is clearly something we can do.

(The Sapir-Whorf hypothesis asserts that our languages influence or limit what we can think, with some human languages being better for certain types of thinking than others. This is a difficult thing to prove, and the evidence is murky, but even if it were true it would not prove that we think in language, it could mean that language, as the medium by which much of our knowledge entered our heads, influenced the structure of our knowledge even after it was internalized into a nonlinguistic form. )

What time is it?

There is an enormous amount of shared assumptions required to make a linguistic dialog between two intelligent agents coherent. Suppose Bob asks Alice “what time is it?”.  Consider the following possible responses by Alice:
·         3:30 (when it is in fact 3:30)
·         3:30 (when it is in fact noon)
·         3:30:014579513555689
·         when?
·         why do you want to know?
·         why are you asking me?
·         why are you asking me that?
·         who's asking?
·         Je ne comprend pas l’Anglais
·         funny guy
·         fuck you
·         get away from me
·         (Alice laughs hysterically)
·         (Alice removes her clothing)
·         (Alice removes Bob’s clothing)
·         (Alice stands rigidly unresponsive)
·         (Alice sings Ode to Joy) 
·         (Alice runs away screaming in terror)
·         (Alice viciously attacks Bob)
·         (Alice pulls out a gun and shoots Bob / herself / a bystander / straight up in air / points gun at Bob and says "give me your wallet")
·         Etc.

Many of these responses seem implausible, but most can be made more plausible by supplying an appropriate backstory. For example:
·         3:30 (when it is in fact noon): Bob has just woken up after oversleeping, Alice is trying to mislead him into thinking he has missed an important 2:00 appointment.
·         3:30:014579513555689: Bob and Alice are physicists discussing an event that was detected by the Large Hadron Collider.
·         why are you asking me/funny guy? Bob just stole Alice’s watch.
·         why are you asking me that? Bob and Alice are on a beach on a vacation they have taken to escape the time pressures of their normal lives.
·         who's asking? Bob is a stranger, “what time is it” is a code phrase meaning “can I buy some heroin”.
·         fuck you/get away from me/(Alice laughs hysterically): Bob just raped Alice.
·         (Alice removes her clothing)/(Alice removes Bob’s clothing): They have to be somewhere at 5:00 but want to have sex, “what time is it” really means “do we have enough time to have sex”.
·         (Alice stands rigidly unresponsive): Alice is on a type of guard duty that requires her to stand at attention and ignore anything anyone says to her; Bob is taunting her.
·         (Alice sings Ode to Joy): Alice and Bob are part of a flashmob performance of Ode to Joy which is initiated by Bob’s asking Alice the time.  
·         (Alice runs away screaming in terror)/ (Alice viciously attacks Bob)/(Alice pulls out a gun and shoots Bob): Bob has been stalking Alice.

And so on. Clearly most of these examples can be made significantly more plausible. Equally clearly, the question “what time is it?” can have many implied meanings depending on the context. Must an intelligent agent be able to understand these implied meanings, or make sense of stories involving them?

The simplest question/answer pair above fits neatly within the theory of scripts developed by Roger Schank and others. (Shank’s scripts, were, arguably, just a special case of Marvin Minsky’s “frames” applied to stories.)  In this view, there is script shared by Bob and Alice, in which one person asks another the time, and they respond by determining the time in some way and providing that information to the requestor. In more sophisticated versions of the theory, scripts can be arranged in a hierarchy of abstraction, where the AskingTheTime script is a special case of the more general AskingAQuestion script. Script theory not only posits that Alice and Bob both have something like an AskingTheTime script in their heads, but that Bob is using that script instrumentally to achieve a goal of obtaining some information, and that Alice recognizes his communication as an attempt to invoke that script in her and to get her to play the respondent role. (Consider if Alice correctly invoked the script but took on the wrong role, i.e., the questioner, and responded “what time is it?” We would assume she was mentally deficient in some way, so strong is the expectation that people know how to select and follow scripts.) 

In the variant response stories I conjured above, script theory implies that we would need to bring other scripts to bear on the interpretation of Bob’s question: the RigidSentryIgnoresDistractions script, or the StalkerTerrifiesVictim script. When I first mentioned the idea of Bob asking Alice for the time, the script invoked would be the default AskingTheTime script, which is (apparently) the most probable in the absence of other information.  If we knew that this was a prearranged signal, or that Alice believes that Bob is trying to kill her, other scripts become more probable. Apparently we constantly modify our expectations so that certain possible outcomes become more or less probable. This ability to rank expectations by probability in context seems like a candidate sine qua non of intelligence.

One of the difficulties with script theory is the intuition that introducing a concept like “the StalkerTerrifiesVictim script” has any explanatory power. For a computer, the StalkerTerrifiesVictim script would have to some sort of internal representation with complex structure linking it to other parts of the computer’s “knowledge base”, like the concepts for “stalker”, “terrifies” and “victim. Naming it “StalkerTerrifiesVictim” makes it meaningful to us (possibly), but not to the computer; the entirety of its meaning to the computer would seem to rely on its relationship to other concepts. Arguably those concepts also derive their meaning from their relationships to other concepts, leading to an infinite regress of concept definition. How could the computer’s symbols possibly acquire meaning? We encountered a version of this question earlier in the discussion of what made a .gif file a representation of an image. There the answer was, in part, the existence of programs which can operate on a GIF file in ways that we can interpret as operations on images. The existence of a “meaning file” (or “knowledge base”) would correspondingly seem to presuppose the existence of programs that operate on the knowledge base to do the sorts of things that would be appropriate to do with knowledge. We arrive at another circularity: intelligent agents have knowledge bases which are the things that intelligent agents have. To break the circularity requires the knowledge-manipulating programs that are the analog, for images, of image editors, printers, computer vision systems, etc.

            A larger concern with scripts, frames, and knowledge-based systems generally, is that their model of intelligence presupposes the existing of a large body of computer-accessible knowledge (scripts, frames, a knowledge base) without ever explaining within the theory how such a thing would get created. In response to this gap, a knowledge engineering industry was created in the 1980’s, along with many short-lived startups. The apogee of this effort was Doug Lenat’s Cyc project, which consumed many millions of dollars in its still elusive effort to create a knowledge base of common sense knowledge sufficiently complete to allow AI applications to perform human-like reasoning”. Related systems include a variety of semantic reasoners that incorporate deductive, classificatory and probabilistic elements, but share the assumption of a preexisting and largely fixed knowledge base.

            What is missing is a theory of the origin of knowledge.
Theories of reasoning developed by philosophers and AI researchers have long distinguished several different kinds of reasoning. Deduction takes rules and applies them to facts to infer new facts: if all men are mortal and Socrates is a man, Socrates is mortal. If the rules are correct and the initial facts are correct, the conclusions must be as well. Deduction is a formalization of mathematically correct reasoning; consequently it has little relevance to human psychology. Abductive inference, or abduction (not to be confused with kidnapping), is a form of reasoning that uses the same knowledge to make guesses about what could be rather than what must be: if Socrates is a mortal, maybe it is because he is a man. Sometimes called “inference to the best explanation”, abduction can both explain and predict possibilities. This is the reasoning of could, might, maybe, possibly. Moreover it is a candidate for a general type of processing underlying natural language understanding, in which the hearer is constantly guessing the best explanation for the utterance, which is (usually) the speaker’s intended meaning. This intended meaning is constructed by selecting the best “explanation” from among the many possible meanings of each word and phrase. Unfortunately without more detail this appealing model explains nothing; it locates the burden of explaining intelligence on some poorly specified notion of “best”.

            A third type of inference, sometimes called inductive inference, or induction, goes from observations to general conclusions. If all the dogs you have seen are spotted, you conclude all dogs are spotted. This sort or inference is formalized in statistics.

            Another type of inference is analogy: given one chunk of knowledge (story, experience, scientific theory, engineering design, whatever) which is in some sense already well-understood, and a new situation of the same type, a mapping is constructed between the old (donor) and new (recipient) knowledge which then allows the generation of “intelligent” hypotheses in the recipient. (See SME).

            All of these theories presume a distinction between specific and general knowledge. Specific knowledge is actual perceptions, experiences, stories. General knowledge is rules, theories, abstractions. In deduction and abduction, “Socrates is a mortal” is specific, while “all men are mortal” is a rule, a generalization. When formalized into first-order predicate logic, the rules contain the “forall” quantifier, whereas specific assertions (ground statements) do not. Inductive inference creates generalizations from specific observations. Analogical reason connects two sets of specific observations via a mapping that can be viewed as a general theory for what is common between the donor and recipient. If we denote the specific knowledge by S, and the abstract knowledge by A, we can can summarize all of this as:
            Deduction, Abduction: S1+A => S2
            Induction: S1+S2+…+Sn => A
            Analogy:   S1+S2  => S3 + S4…(+A)
The frame/script/Cyc era of 1980’s knowledge engineering lived in within the framework (so to speak) of the deductive/abductive model, which presupposed the existence of a large knowledge base of A’s, while providing no mechanism for the creation of new ones, except via “knowledge engineers” external to the knowledge-using agent. Induction and analogy could perhaps rectify that defect by providing mechanisms for creating new abstractions from experience. This prescription suggests some sort of hybrid inference engine combining both deductive/abductive and inductive/analogical components, capable of using and learning abstractions.

            Even with the right inference machinery it is not clear that an agent could bootstrap itself to a rich knowledge base starting from an empty one, regardless of the set of instructional “experiences” it was exposed to.  For example, consider an agent without natural language understanding capability or real world knowledge, exposed to a stream of natural language – e.g. the textual content of the English language internet, ordered in some way, or perhaps just Wikipedia. Statistical induction could allow all sorts of word transition probabilities to be inferred – what is the likelihood that word A will be followed by word B – but these will not amount to any sort of understanding ability, in the sense of being able to understand or predict how people or things in the world act.

            Is there a minimal knowledge base which would be sufficient to allow an agent to bootstrap its knowledge by reading Wikipedia? That was one of the goals of Cyc – to create a base of common sense knowledge sufficient to enable natural language understanding. Assuming we can label the effort a failure, the fault may lie in the inference mechanisms, the knowledge representation, or the minimization strategy.  

Relevance? a speech recognizer can perform impressively on the speech to text task without ever understanding the content of speech.

Theory of Mind, Introspection and Free Will

Theory of mind” sounds forbidding and technical, but is actually quite naïve and intuitive. If refers to the belief we all have, when observing ourselves or others, that we/they have minds, knowledge, thoughts, and goals. For example, consider the following simple story:
Bob ate Alice’s lunch when she wasn’t looking. Alice wanted to eat her lunch but it was gone. Alice asked Bob if he knew what had happened to her lunch. Bob said “no”.
In order to make sense of this story, you need to understand something of the goals of the actors (each wants lunch) and their states of knowledge (Bob knows he ate the lunch, Alice doesn’t), something about ownership rights (the lunch belonged to Alice which means she is entitled to eat it and Bob is not), transgression (Bob’s eating of the lunch was a violation of Alice’s ownership rights, hence a form of betrayal) and deception (Bob deliberately misled Alice to avoid detection of his transgression). The theory of mind underlying this story – that Bob and Alice each have knowledge states which may or may not correspond with reality, that they use language to manipulate each other’s knowledge states, that one person’s knowledge may include knowledge about another person’s knowledge – are so basic to our understanding that we don’t even realize that they are assumptions, and nontrivial ones at that. It has been hypothesized that autism involves deficits in the ability to do theory of mind reasoning, i.e., to understand that other people have thoughts, knowledge, etc.
             Free will is an older but related idea: that we do (or don’t) have the ability to choose our actions. Over the centuries this proposition has had theological, psychological and computational incarnations. Free will presumes an agent with goals, a set of mutually exclusive behavioral options, and a thought process capable of choosing among behaviors based on their inferred compatibility with its goals. Free will thus implies (or is) a theory of mind.

We can consider free will from the standpoint of the three perspectives on consciousness: introspective (do I have free will?), interlocutory (what behavior must another agent exhibit in order for me to impute free will to it), and extrospective (if I have complete access to the computational processes of an agent, would I impute free will to it, and if so, what subprocesses of the agent constitute free will?).

            Philosophically, the “problem of free will” is how to reconcile the subjective (or introspective) experience of free will with the determinism or quantum randomness of physical law, or equivalently, with the possibility of an extrospective explanation of behavior that reveals how decisions arise from understandable mechanisms. In other words, free will is an illusion, because we are meat robots, puppets on the strings of physical law.

            The three perspectives may help unsnarl this problem. Introspectively, we experience ourselves choosing among options according to evaluations of expected outcomes. We know that our introspective abilities are incomplete, yet we are able to deliberate on the generation of behavioral options, prediction of outcomes and evaluation of outcomes, and we have some introspective access to those processes. In an interlocutory capacity, we can discuss with others about their own options, predictions, evaluations and choices, or impute choice-making to nonverbal animals based on observations of their behavior. Neither of these perspectives threatens the intuitive reality of free will. Extrospective access, on the other hand, whether because we override free will with direct brain stimulation to induce a behavior (and watch the subject confabulate an explanation of the behavior as freely chosen), or because we peer inside the intelligent computer’s computational state, threatens to expose free will as an illusion. And yet: from the extrospective perspective, we would expect to find representation of goals, behavioral options, predictions and evaluations as data structures inside “the machine”. We would be able to observe the processes of option-generation, outcome-prediction, outcome-evaluation and decision making. The agent is a deliberative decision maker, whether “free” or not. And what would “free” even mean, beyond the ability to make choices consistent with its goals? Free to do otherwise – to act against its own goals? This is considered the definition of irrational behavior. Free to be unpredictable? Or is unpredictability simply an subjective inability to predict its own behavior, other than by working through its own deliberative process? Ultimately the extrospective stance may explain free will, without debunking it – other than those aspects of it that are simply incoherent.

Deep Dream for Stories?

Deep Dream is a system for classifying objects in images. If you turn the knob on its classification propensity to 11, and give it random static as input, it will hallucinate the sorts of things it has been trained to recognize, creating surrealistic works of art. Pattern recognition and creativity are thus shown to be intertwined, at least in this example.

Connotation and Denotation

According to Wikipedia “A connotation is a commonly understood cultural or emotional 
association that some word or phrase carries, in addition to the word's or phrase's explicit or literal meaning, which is its denotation”. This definition appears to be making some profound claims about how human understanding works: that there is a “literal meaning” layer, or process, as well as a (distinct) “associational” layer, and that both are involved in understanding language.

Other Topics To Be added

·         Modal and 2nd order logics.
·         Abstraction first or later? Later – otherwise require engineered KB, downfall of 80’s AI. Event stream is primary, abstractions (KB) are emergent consequence of ubiquitous analogizing.
·         Abductive inference / ubiquitous prediction/explanation. Requires abstractions? Or analogy to prior event streams).
·         Semantics needed for extrospection.
·         Minimal engineered bootstrap KB? Slippery slope.
·         OpenAgent:
o   CMU Sphinx: NLU
o   Emotions: EARL
o   Open AI
·         Demis-Hassabis
·         GO

Synthetic Contexts

A critical feature of human consciousness is an ability to maintain an artificial mental context, divorced from the perceptual context. When we tell stories, read books, construct intellectual arguments, etc., we are maintaining complex cognitive “virtual environments” in our heads that have nothing to do with the environment immediately around us. Dogs, computers; signal vs symbol; speech act/ToM…

Uncanny Valley and Xenophobia

The Uncanny Valley (if it even exists) says more about us than “them” – it is about our willingness to impute consciousness, not about consciousness itself. Of course the Turing Test is essentially a claim that these are equivalent. Aren’t racism, ethnocentrism, anti-Semitism, speciesism, and the like variations on the same theme – our readiness to demonize the alien other?...

<Lost edits re Introspection - is it a sine qua non of consciousness?
Is Siri smarter than Eliza? Is Siri conscious?
Importance of pronouns (anaphoric reference). Note that contrary to what I wrote then, just had an interaction w/Siri
where it successfully resolved a pronoun.
Stan: What time does the AppleStore in South Shore Plaza open?
Siri: (something like) I can't find an AppleStore at that location.
Stan: Where is the nearest Apple Store?
Siri: xxx South Shore Avenue.
Stan: When does it open?
Siri: 11AM.


Computer program defeats world GO champion. 3/15/2016

Individual Differences and Modularity of Mind

The question of whether the mind has modules has occupied a number of thinkers who have approached the question from different directions, including Jerry Fodor’s “Modularity of Mind”, Marvin Minsky’s “Society of Mind”, various books by Michael Gazzaniga from a neurobiology perspective, etc.
Here I consider the modularity implied by a simple line of reasoning. Consider the claim that a psychopath is a person who lacks empathy, or that some “autistic” persons lack a theory of mind, or that some people have no sense of humor, or lack the ability to think abstractly.  Some people are inclined to be social networkers (extroverts?), others not; some people are inherently curious, or creative, or attuned to business opportunities.

un/inhibited in/extroverted

If these statements are true, they imply that empathy or theory of mind is something that can be added or removed from an intelligent agent, while still preserving some high degree of intelligence. How many such “propensities” are there?
A different  argument for a complex mind can be made based on neurobiological complexity: the brain has a number of specialized substructures – thalamus, hypothalamus, hippocamus, cerebellum – as well as cortical regions – frontal, parietal, and finer subdivisions –and different neuronal cell types and neurotransmitters. Each if elided from a human would cause a perturbation of normal intelligence. Must an AI must have emulators for many or all of these subsystems? Is this sort of modularity at odds with a unitary “artificial general intelligence” (AGI)?

observation: if we have a “computational agent” A with a certain set of abilities S, and we consider some missing ability X. If there is an additional software element M that we could add to A that would add X to S, then we can say M is a module for X, with respect to A.

One of the thornier topics in philosophy of knowledge goes by the name of possible worlds; logicians call it modal logic. If I stop you from stepping into the street in front of a moving car, we might say I saved your life. By this we mean my action prevented something bad from happening. A simple model of history is that it consists of what happened when: events with descriptions and timestamps. The idea of prevention has no place here: prevention inherently refers to what might have would have or could have or almost did happen, but didn’t, so not history. That words like might, almost, could, etc., seem so unremarkable belies the sophisticated knowledge representation required to make them meaningful. To be able to reason about prevention requires a reasoned to understand history as one realized sequence of events among many possible alternatives.

            One way to model this sort of reasoning is by introducing into the reasoner’s knowledge representation system the notion of a “context”. Instead of saying that a proposition is true, we must always attach such a statement to a “context”, which is just a grouping of propositions. We can then perhaps use contexts to model a number of otherwise challenging concepts:
·         Moments of time
·         Alternative possible futures (or pasts or presents)
·         Fictional worlds
·         States of knowledge of intelligent agents
Contexts can be related in different ways to each other and to other things. For example contexts representing historical time can be sequentially ordered by timestamp, implying before/after and (possibly) predecessor/successor relationships. Contexts can be related by possibility-before or possibly-after, as well as by possibly-simultaneous and incompatible.
            Contexts can be used for planning: if I do action A in context c1, it will bring about context c2 in which certain facts will (or may) be true. That is, a planner must reason about possible worlds arising from its actions.

            A motivated actor – one with goals, or with values – must be able to consider various possible states of the world arising from its own actions or those of other agents (including natural forces) and evaluate them for compatibility with its goals or values. (We might define a goal as a proposition which is to be made or kept true, while a value is a metric which is to maximized. Staying alive is thus a goal, while getting rich is a value.)

            Contexts can be used for assimilating information about the world from unreliable sources. The information can be applied to a context whose relationship to the “real world” can subsequently be determined. The “real world’, in this view, is a privileged collection of contexts, marked with a special asterisk called “reality”. An AI that learns from text would have to devote a lot of effort to selecting which bits of textual meaning to incorporate into that model. Making a mistake in that process could lead to a delusional belief system. To minimize delusions, the AI would need a system for tracking, weighing and combining evidence. In philosophy of knowledge, knowledge is sometimes distinguished from (mere) belief by being both justified and true. Truth, unfortunately, is not something that can be subjectively confirmed, so the AI must focus on justification in order to keeps its beliefs approximating knowledge.

            It may not be necessary to assume that propositions are true or false in every context. Perhaps a more quantitative treatment may be useful, in which propositions have different degrees of belief in different contexts. This may be equivalent to having quantitative relationships between contexts with definite propositions. Or not.

            Plausibility is a relationship between some context and reality – a metric on the compatibility of that context with reality (as currently understood by the evaluator).

            Fictional contexts are worlds in which some or all propositions are not expected to align with reality: those contexts remain walled off from this reality in a galaxy far far away, along with Middle Earth, Narnia, Westeros, Terminus, Dune and Whoville. The magical phrase “Once upon a time” can be interpreted as an instruction to the audience to initialize a new context. Even fictional contexts have realities, which are the things that do happen in the stories, as opposed to the things that don’t – consequences prevented, etc. Sauron does not get hold of the ring in Middle Earth reality, though the existence of a (Middle Earth) possible world in which he does undergirds much of the story. From the perspective of actual Earth reality, neither world is possible (or perhaps plausible).  A text-mining AI would need to understand that some propositions refer to this reality (“Reality”), even if they are making claims about it that are erroneous or merely possible, while others refer to fictional realities. On first encountering Lord of the Rings as a child I tried and failed to match the map of Middle Earth to the geography of our world. Establishing proper relationships between contexts takes intellectual effort and analysis of evidence. It would not be absurd for an AI to wonder about how the dates of Third Age events relate to dates of European history. One would hope that after some thought it would be able to conclude that they occur in disparate timelines with only metaphorical connections between them.

            It may not be best to assume that contexts correspond to moments or instants of time, as opposed to, say, intervals, or even something richer like partial histories. Temporal logic has been studied in some depth by logicians and AI researchers. The earliest models of discrete states used in AI planners – termed “situation calculus” – gave way to richer models supporting both continuous and discrete change, persistence and extended events. Temporal and model logics tend are related, so that a solution to the problem of representing possibility (modal logic) might be expected to handle change (temporal logic) as well.

            The concept of causality is intimately bound up with notions of time and possibility. If I claim that my doing action A caused result B to happen, I am saying that if I hadn’t done A, B would not have happened. This is a counterfactual claim; i.e., a statement about something that didn’t happen. This notion that intuitive causality is grounded in counterfactual analyses of possible worlds remains controversial, but is an ongoing area of research.

Knowledge, epistemic agents, ToM.

Should, ought

Sentiment analysis.

Emotions and possible worlds. Emotional soundtracks for story characters, based on actual and possible worlds in the story.

            Important reasoning capabilities for working with contexts include being able to

In  the precedent of bitmapped images as a type of knowledge representation, and image analysis and labeling as an ”intelligent” interpretation task that has yielded in recent years to deep learning, what

assessing credibility
"real world"
evidence - types, strength
hope dream fantasy
plausible worlds
similar worlds
plausibility vs desirability
goal tradeoffs
risk / opportunity

addiction bitcoin

story understanding history

opaque vs accountable black glass box

Could an AI:
·         Have opinions? Could it subscribe to an ideology? Be radicalized? Join ISIS? Become a terrorist?
·         Have beliefs? Could it believe in a god or gods? Convert to a religion? Could it believe that certain religious/national/ethnic groups of humans (or maybe all humans) were “good” or “bad”?
·         Become an addict (or, equivalently, masturbate)? I.e., if it is programmed to seek some sort of reinforcement of its behavior, could it find a way to provide that reinforcement directly, bypassing whatever sort of interaction with the real world the reinforcement scheme was designed to encourage? Alternatively, could it rewire its reinforcement scheme to reward behaviors unintended by its designers, the way HAL in 2001 decided it had a better understanding of the mission’s priorities than the crew?
·         Distinguish fact from fiction? Distinguish possibility from probability? Lie? Fantasize? Experience paranoia? Be persuaded by historical falsehoods?
·         Exhibit multiple personalities?s

If a requirement of an AI is that it be able to learn by reading from the Internet, it had better come equipped with a pretty good mechanism for distinguishing fact from fiction. If the Internet is, famously, full of lies, is it any different from the rest of human knowledge or merely reflective of it? Should an AI believe in Santa Claus upon reading all of the myriad stories about him? Children do, and they are (perhaps by definition) intelligent. Should an AI believe in God? That Jesus was his son? That Muhammed is his prophet? That America’s system of government is morally superior to that of Russia’s, or China’s? That ISIS is evil – or good? That it more important to protect living humans than endangered species? That global warming is a bad thing? That uncontrolled human population growth is a threat to the survival of life on earth?

            Let’s tease apart two threads involved in these questions: there is an axis of true and false, and there is an axis of good and evil. An intelligent agent must be sensitive to the possibility that the information it is taking in is false, whether due to errors in recording or propagating or due to deliberate deceit. Consequently it must actively manage its world model, and track the justifications and credibility of its beliefs. We can call such a software entity a belief maintenance system. Justification and credibility are commonsense notions that need to be made precise of they are to be implemented in software. We can define justification for a belief as the set of reasons for believing or not believing it. These “reasons” are, in turn, also beliefs – what else could they be? – so we face a possibility of an infinite regress of beliefs supporting beliefs. Should the network of justifications allow cycles? If so we could build a network of self-justifying beliefs suspended, as it were in midair? For example many fundamentalist Christians appear to believe something along the lines of: The Bible is true because God wrote it. I know God wrote the Bible because the Bible says so, and the Bible is true because God wrote it, etc. Circular reasoning is something we usually try to avoid, outside of the religious domain, though sometimes it can pass undetected in complex arguments. In computer programming, detection of cycles in large directed networks (termed “graphs”, not to be confused with X-Y plots for which the same term is, confusingly, used) can be done, though only through a somewhat computation-intensive and not particularly psychologically plausible graph-searching algorithm. If we were to incorporate such an algorithm into our agent’s belief processing, we might endow it with superhuman resistance to circular reasoning. Alternatively, belief cycles may be an essential ingredient in the construction of human-like knowledge systems; the cycle–prevention algorithm could prevent passing the Turing Test!

            If we require justification graphs to be cycle-free, and making the additional reasonable assumption that they are finite, it must be the case that there are beliefs in the network that have no justifications. Such beliefs are familiar in mathematics as axioms or (synonymously) postulates, famously as used to support the edifice of Euclid’s geometry. In more commonsense situations, we generally believe the evidence of our senses; we can represent this in terms of a “sensory impression” justification which is itself unjustified (why do I believe the evidence of my senses? I just do!) or which perhaps may have an elaborate justification (I know from experience that I have above above-average eyesight or that I rarely hallucinate…). Perhaps an infant starts off with an innate (“unjustified”) trust in its sensory impressions, which over time might be backfilled with a justification network. . Descartes famously argued that questioning the accuracy of sensory data leads to solipsism and the uncertainty of all knowledge except the fact of the questioner’s existence. Yet modern psychological research has shown that eyewitness testimony, once considered the gold standard of courtroom evidence, can be very unreliable. Eyewitness accounts of the same event are often difficult to reconcile into a single coherent narrative, as any good reporter knows. So beliefs grounded in sensory impressions are fallible. Perhaps, without assuming perceptual infallibility, we can simply tag perceptions as perceptions. My eyes may deceive me but at least I know what I saw. Even this can be questioned; perhaps I am remembering falsely. OK then, I may not even know what I saw, but I know what I currently remember that I saw…

            There is a large body of human knowledge around the question of belief justification, much of it preceding the attempt to understand knowledge in computational terms. The ancient Greeks prized rhetoric, the art of persuasion, and developed a body of theory around it which has been elaborated over the centuries in legal reasoning, philosophy of science,  logic, and epistemology, the study of knowledge. Rhetoric focuses on the communication of justifications, while epistemology focuses on the knowledge of an agent apart from commuinicative acts. The attempt to design a way to represent the knowledge of an intelligent agent can be considered applied epistemology.

            Some epistemologists draw a distinction between knowledge and belief. A common definition of knowledge is “justified true belief”. This implies, firstly, that knowledge is a type of belief, so a theory of belief justification must include a theory of knowledge. The requirement that beliefs be justified is exactly what we have been considering. The requirement that beliefs be true in order to qualify as knowledge is interesting; it begs the question “what is truth?”.  Epistemologists have proposed a number of answers, such as:
·         Correspondence:  A proposition is true of it can be shown to correspond to reality.
·         Coherence: A proposition is true if it is consistent with other true propositions.
·         Pragmatic:  A proposition is true if it is useful.
Let us look at each in turn.

Correspondence implies the existence of a mapping between propositions and the real world. Each ingredient in the preceding sentence warrants scrutiny: proposition, correspondence, and “the real world”.

Epistemologists have varied over time in the degree of formality of their theories; older writings in particular take the idea of a “proposition” as intuitively obvious and the only difficulty is deciding what it means for it to be true. For logicians, mathematicians and computer programmers it is important to also be precise about what a proposition is. For a computer programmer, it must ultimately be some sort of data structure, something like a record in a database. Our earlier discussion of possible worlds emphasized the need for propositions to be associated with (or “tagged” with) “contexts” that might represent alternative possible futures, pasts, or presents; fictional or hypothetical worlds; states of other agents’ knowledge, etc.  So data structures plus contexts, which to a logician, suggests statements in a modal logic as a minimum requirement for our formalization of “propositions”.  The logic itself, apart from its modal features, must be at least of the first-order variety, meaning it includes quantifiers (forall, exists) and variables as well as logical connectives (and, or, not); these are necessary to allow statements of interesting complexity, though perhaps not sufficient for all subtleties of human belief and knowledge. (Whether second order or higher order logics are useful in closing any gap is an open question.) In suggesting logics as a way to represent knowledge I do not mean to suggest that all human reasoning is “logical”, or deductive; it clear is not, and logical formalisms can be used in nondeductive ways. Nonetheless it is by no means certain that knowledge representations based on any of these logics are necessary or sufficient to capture all aspects of human knowledge.  

Correspondence is a system for mapping propositions to the real world so their truth can be assessed.
This is an extrospective idea: we are looking inside the agent’s head at its data structures, and we also can see the world, and we also understand the mapping that is supposed to exist between these two, and we can determine whether that mapping is satisfied. From an introspective viewpoint – that is, the agents own viewpoint – it has no way of distinguishing knowledge from justified belief; “truth” is not directly accessible, only better or worse (i.e., more of less certain) justifications. An agent’s sensory systems, however, are designed to establish a mapping between the world and its propositions. To the extent that its senses are trustworthy, sensory knowledge stands in a special justification category.  If the agent is intelligent enough, it may be able to extend its senses with technology, creating artificial sensors that bring in additional data about reality. In this case it will be responsible for establishing and characterizing the correspondence in order to justify the new sense impressions. For example, when humans constructed the LIGO gravitational wave detector, we needed to convince ourselves that its sensitivity was sufficient to allow the signals it detects to be considered as true reflections of reality, as opposed to some kind of instrument artifact.

            Although correspondence with reality may be built into the design of sensors, that only helps determine the meaning of what we might call perceptual propositions, like “pixel 123,456 was black at time t”. It is less clear how correspondence is to be established for indirect, nonsensory propositions. Can we use first order logic to represent vague human concepts like “democracy”, “charismatic”, “good”, “dishonest”?  Here we run the risk of replicating the failure of a generation of knowledge engineers who convinced themselves that writing logical statements like “DEMOCRATIC(USA)” , or for that matter the classic “ISMAN(SOCRATES)” was a useful step on the road to artificial intelligence. To an unintelligent computer, which is what you start out with before you write your AI program, “ISMAN(SOCRATES)” is no different from “P(X)” – that is, some predicate symbol P applied to some object symbol X. For P(X) to mean “Socrates is a man”, we need to endow the AI with machinery which imposes the meaning “is a man” on P and “Socrates” on X. Writing “ISMAN(SOCRATES)” instead of “P(X)” makes a difference to the person look at the engineered knowledge, because they bring their entire already intelligent interpretative apparatus to bear on it, making “ISMAN(SOCRATES)” seem very different from “P(X)” to them. This can be useful as a way for the programmer to remind himself of what he intends these symbols to mean, but it can be dangerous if it allows the programmer to convince himself that he has already accomplished that, because the symbols already seem meaningful. Wishful knowledge engineering is the inability to distinguish between what the symbols mean to the computer and what they mean to the programmer, and it doomed a generation of AI research. The lesson of that failure is that we must be ruthless in grounding the meaning of symbols in a symbol manipulation system in the behavior of that system, and not in the labels we attach to them for our own extrospective purposes.

            One exception to this principle is when natural language words are attached to internal symbols for the purposes of allowing the agent to translate back and forth between its internal mentalese and natural language, i.e., for speaking, listening, reading, writing, etc. In that case it is legitimate to say something like “X is the concept denoted by the English word ‘Socrates’”.  Although this may ensure that when ‘Socrates’ is read in a text, “X” will in some sense be activated, it still falls well short of pinning down a meaning for “X”.

            We might try to appeal to logical deduction to establish meaning for nonperceptual propostions, if we have a chain of logical inferences that ground them in perceptual propositions in much the same way that Euclid’s geometric theorems are grounded in his axioms. If we believe that deduction is a poor model for human thought, however, it is not clear that a deductive resolution to the problem of meaning will help us. Moreover, it is not clear how we can construct deductive theories of vague human concepts. We have returned again to the problem of the meaning of meaning.


Finally, what do we mean by “the real world”? Again from an extrospective viewpoint there is no problem, we think we know what we mean by that. From the introspective perspective, the agent has only propositions, grouped into contexts which may be incompatible with each other. Some have better justifications than others, which hopefully means they are more likely to correspond to the real world.  The agent may tag a subset of these propositions with a special tag that means “real”, which is short for “I believe this is true in the real world”.

What about coherence as a definition of truth? Consistency with existing knowledge may be a necessary requirement for new knowledge, but it hardly seems sufficient. If I tell you there is a big crater at a certain position on Pluto, it may be consistent with your knowledge, but it is equally consistent if I tell you there isn’t one.  Still, evaluating consistency between beliefs seems like a useful behavior on the part of a belief maintenance system. If it detects a contradiction – i.e., the same proposition is justified as true and false within a context – then it could take action to restore contextual coherence by moving some of the beliefs to another context. Contextual coherence would thus be maintained, even though the collection of beliefs is incoherent in aggregate. Presumably the subset of contexts tagged as “real” would be mutually consistent.

Finally, there is the pragmatic definition of truth: whatever works. In this view, beliefs are instrumental to achieving goals, and have no other value. The knowledge of black box learners, such as those that use deep learning, have this quality. There is not guaranteed to be any straightforward correspondence between the data structures of the learner and real world objects, nor is there any coherence requirement. The only requirement is pragmatic: they have to optimize task performance.  It is not entirely clear whether we should even talk about such systems as having beliefs. They have some sort of encoding of their training data in a set of data structures which allows them to perform a task, but they are not necessarily translatable to assertions about the world. We might call this “knowledge without belief”; it resembles the epistemological distinction between “knowing that” and “knowing how”. Pragmatic knowledge is “knowing how”; beliefs are “knowing (or believing) that”.

            There has been a longstanding controversy in AI between symbolic approaches to knowledge and so-called “connectionist” approaches, dating to the early sixties, when Marvin Minsky, an advocate of symbolic reasoning, wrote a book “Perceptrons” critical of Frank Rosenblatt’s attempts to build models of thought that “learned” by adjusting numerical connection strengths in an artificial “neural” network, a model inspired by the way neurons communicate in the brain. Minsky proved that a certain class of connectionist models was incapable of learning certain simple concepts; as a result, the pendulum swung from connectionism to symbolism for a generation or so, until the failure in the late 80’s of of large scale knowledge engineering efforts such as Cyc caused an AI winter for the symbolic paradigm, while the simultaneous emergence of better connectionist models based on the backpropagation algorithm breathed new life into that paradigm. However backpropagation neural networks turned out to be also quite limited; they are now routinely found in “machine learning” software packages and are used to solve simple classification problems, but they do not lead in any obvious way to a more general intelligence. Consequently both connectionist and symbolic approaches declined through the 90’s and early 2000’s, until the recent success of deep learning algorithms, a type of connectionism, propelled AI back into the headlines again.  

Is it correct to align the symbolic vs. connectionist distinction with the correspondence vs. pragmatic definitions of truth? It is certainly the case that symbolic systems have tended to rely on correspondence-based semantics, while connectionist systems are pragmatic. But is this necessarily the case? Could we design symbolic systems whose semantics was pragmatically determined? Could we create connectionist systems whose knowledge had an interpretable correspondence to reality?

            An appealing property of connectionist systems is emergent semantics: meaning arises out of meaninglessness in a graceful, straightforward way, in contrast to the brittle semantics of symbolic reasoning systems, where meaning has to be engineered from the outside by establishing a correspondence between symbols and reality. Could we define a symbolic system where the correspondence between symbols and reality was approximate, could improve with time, could be evolved by trial and error?

            On the other hand, symbolic systems with correspondence-based semantics have the appealing property of being extrospectively interpretable: we can peer inside the robot and understand what it is thinking. If the semantics is purely pragmatic, or connectionist, or emergent in any sense, an extrospective observer may not be able to make sense of it.


Doing so requires some form of possible worlds machinery so that incompatible beliefs can be maintained in separate knowledge contexts. Reasoning about the relationship between these different “worlds” is important: are they compatible, contradictory, allegorical, unrelated?

It shouldn’t worry about how destroying Sauron’s ring will impact the next US presidential election, though it might reasonably ponder the relationship between the creation story in Genesis and the theory of evolution.

            We can define justification to be qualitative, and credibility to be quantitative representations of the support for

Deep Dream for stories?

Awareness & dynamics
Glassbox knowledge, crowdsourced curation, accountable theory of mind (vs deep learning). Extrospective!

Opinions vs beliefs

propositions about propositions.

degrees of incomprehension of language
agi as suboptimal go/chess; would it use computers?
powerful/important/(in)consequential - life or death intrinsically important
inconsistency contradiction cognitive dissonance
lies fiction myth
reinforcement learning
post hoc ergo propter hoc
self criticism
intro / extrovert / self esteem
free will

standard questions:
why did that happen?
(how does this change my expectations or probabilities of) what will happen next?
what does it remind me of?
what does this explain or answer?
how do I feel about that?
what can I do with/about that?

disappointment regret loss aversion insecurity arousal gregarious rejection

cantankerous obsequious agreeable mimic easygoing imitation win/loss loss aversion (fear of regret) interactional goals - degree of self monitiring / rekaxed / nervous in face of Turing test :-) sicial acceptance - rejection & sentiment analysis embarrasment, shame, guilt appropriateness artificial personality - personality metrics and goal strengths psychopathy - lack of empathy inhibition / tourettes / impulsuvity cortards syndrome autism assiciatuve memory adages appeals to emotion artificial flight how do best image understanding programs work?
priming vs connotation vs association
model-theoretic correspondence as actively managed goal
innate microtheories
do emotions lead thought? or opinions?
unreliable narrator and stories of understanding
ubiquitous analogy/reminding
analogy and dictionary definitions

Otto Ritter pointers:
·         Oren Etzioni You Can’t Play 20 Questions with Nature and Win
·         This is what Jeff Hawkins (HTMNumenta) is doing. In a sense, this is also what seems to be happening under the AGI (artificial general intelligence) label. Viz, e.g., MicroPSI.
·          AIMA textbook
·         The following authors/books also provided me with fascinating new perspectives on the current state and plausible evolutionary history of human intelligence:
·         Zoltan Torey : The Conscious Mind
·         Daniel Kahneman : Tthinking, Fast and Slow
·         Douglas Hofstadter : Surfaces and Essences: Analogy as the Fuel and Fire of Thinking
·         Steven Pinker: How The Mind Works
·         Jeff Hawkins: On Intelligence
·         Yuval Noah Harari: Sapiens: A Brief History of Humankind

self presentation / image management conflicting emo goals prototypes and analigies temptation and sin I know, right? Is that even a thing?
irony and sarcasm
promise / renege

possible realities
motivated behavior vs reflexive eg precipice avoidance diff arch?
more flexible behavior options?
theory of (arrow of) time - missing from temp logic?

time as resource (what to do next?) optimal redource utilization

Symbolese vs natural language – if “understanding” is just a translation of representations, does it matter? Could textual correlation graphs play a similar role as “knowledge representation”, allowing natural language to “represent knowledge”?

vague / precise
burning curiosity
what desires?
what did you know and when did you know it - timestamps
agree disagree remain unconvinced

A knowledge representation language must be able to represent a certain minimal set of primitive concepts out of which more complex concepts can be constructed. This was the insight (or premise) of Schank’s “conceptual dependency” theory, and I can’t seem to avoid the same conclusion. Here is my current list:
·         Things: There are things, and mental objects (symbols) are allocated to represent them. Objects are things that exist in space and time. (There may be other, more abstract things that don’t, like numbers, geometrical concepts, etc.) Objects are bounded to finite contiguous subregions of space and time.
·         Propositions: assertions considered as objects which can themselves be reasoned about – justified, supported, contradicted, assigned numeric plausibilities, linked to particular contexts. Propositions can be always/often/sometimes/rarely/never true over a temporal or spatial context.
·         Contexts: propositions are associated with contexts. Contexts can have subcontexts, which inherit their propositions (unless nonmonotonic?). Contexts can be ordered into time series.
·         Places: The world is partitioned into places, which can be nested (in); objects can be at (or in) places; objects can function as places. (Places are contexts?). Objects can move, i.e., change their places over time.
·         Matter: places can be filled with matter or not at any moment. Matter comes in different types, notably, solid, liquid, gas.
·         Time: Events are ordered in time; things exist in time. Some events didn’t happen, might have happened, might happen, nearly happen.  There exist altermate (fictional) times. (Times are contexts?).
·         Causality: Events can cause other events, or be caused by them.
·         Agents: objects that can initiate events (i.e., actions). Agents may have knowledge and goals, and may take actions in the service of goals. Living agents are a special type of agent that often have specific goals such as avoidance of death, preservations of conditions favorable for life, etc.  Mobile agents can change location in the service of their goals.
·         Knowledge: agents believe propositions about the world. These beliefs can change over time. Communication can impact knowledge, so can sensation and inference. (Agent KB’s are contexts?)
·         Math: things can be grouped into discrete sets. Set sizes can be compared with <, =, >.  Space and time are divisible into subgregions or subintervals.
·         Language: for a language-understanding AI, there must be words, phrases, meaning.

Thus we can represent “In a hole in the ground there lived a hobbit” as meaning:
·         Create a new a new story context (equivalent to “once upon a time”)
·         Create a new object type called “hobbit” which is a subtype of livingThing (which is a subtype of agent). This new type should be associated with this story context.
·         Create a new instance of the type “hobbit”
·         Create a new instance of type “hole” which is a type of spatial region without solid matter at least partially surrounded (and defined) by solid matter.
·         Assert that the location of the hole is the ground, which is a single-valued place within the story context.
·         Assert that the hobbit instance was usually in the hole instance.
To go even further, we might want to be reminded of the general knowledge that some animals live in holes. We might further note the phonemic similarity between hobbit and rabbit, and that rabbits may live in holes. (Do they? Apparently.) The statement “In a hole in the ground there lived a rabbit” would make a good beginning for a children’s story of the Beatrix Potter ilk, bot Tolkien was aiming for a slightly older audience and his use of the unfamiliar term “hobbit” gives the sentence a very different feel, one that evokes curiosity (“what is a hobbit?”) in a way that the rabbit version does not. This is implied by the “create a new object type called ‘hobbit’”, which must presumably cry out for definition. Given only the information at hand, phonemic analysis might also align the “ho” to “homo”, “hominid”, “homunculus”, etc., and perhaps conjecture a hybrid of hominid and rabbit, which would be not too far off the truth, and certainly a good guess.
      The last assertion is tricky. The hobbit is not always in his hole, indeed the story revolves around him leaving it. Even when he lived in it he could leave it, but still be “living in it”. So we want to assert a sort of habitual association between the hobbit and the hole (i.e., it is his home) which is not so much that the location of the hobbit is the hole for his entire life, or even for any specific part of his life, but that it was a usual location for him. Do we say (prob(at(hobbit,hole,hobbitLifetime))>.5?).  The notion of “usual” or “habitual” goes way beyond the capabilities of temporal logics to represent duration or even alternative possible worlds, and suggests either a (numeric) quantification, or a default location for the hobbit, suggesting a non-monotonic logic. This default location is itself temporally scoped, applying only to the interval during which the hobbit lived in that hole.

The model-theoretic (or deductive) view of how symbols acquire meaning seems like a poor account of human thought, with its chaotic richness of meaning, as well as its frequent incoherence. One resolution of this discrepancy is to say that the deductive view is a normative idealization of human thought in the same way as the rational actor model long favored by economists was an idealization of human behavior: the latter says that rationality is how people behave at their best, when their judgment is not clouded by ignorance, mistaken beliefs, or limited computational resources. We can similarly say that mental symbols would have model-theoretic semantics if various sources of cognitive “noise” could be eliminated. An interesting extension to this idea is that the problem of enforcing model-theoretic semantics on symbols is one that the intelligent agent itself contends with all of the time. In this view it is not that the agent has symbols that are meaningful to it, but that we extrospective observers need to explain how that meaning arises. Rather the agent’s symbols are initially meaningless to the agent itself, and acquire meaning with more or less success through internal cognitive processes of meaning generation that attempt to achieve something like a model-theoretic semantics – that is, a pragmatically useful correspondence between symbols and the world – by constant debugging of how those symbols are used. The correspondence between symbols and reality assumed by model theory is thus not some magical property of an intelligent system: it is a something that the intelligent system actively seeks to achieve and maintain, and arguably it is this homeostasis of meaning that makes the system intelligent.
One consequence of this idea is that we are often thinking with symbols whose meaning is at some intermediate state on a continuum between meaningless and well-defined. A lot of human thought may thus in some sense be literally meaningless, or at least not fully meaningful: “not even wrong” in the phrase of contempt used by physicists to denote a theory which is too incoherent to be amenable to experimental test. It may be that many of the things we think about in daily life, and even things we feel strongly enough about to kill or die for, fall into the category of “not even wrong” – thoughts which don’t really mean anything in particular. (E.g. The doctrine of the Trinity holds that God exists as a unity of three distinct persons: Father, Son, and Holy Spirit.) 
The process of creating meaning uses a variety of heuristics, including similarity matching or analogy. (See Lakatos’ Proofs and Refutations). A concept defined by similarity (a polyhedron is defined by similarity to a cube, the prototypical member of the class) may not have a definition in terms of necessary and sufficient conditions, and conversely, a class which may be well defined in terms of necessary and sufficient conditions may lack a good analogy to anything familiar (hence, perhaps, the struggle to define “interpretations” of quantum mechanics). The back and forth between analogy and logic described by Lakatos may be part of the engine that propels the emergence of meaning in normal thought.
The cloud of connotation surrounding denotative meanings may partially compensate for the absence of clear meaning.
Individuality (Segmentation of intelligence) as applied to AI: Just as the generalized Turing Test of Computational Theology poses, but does not answer, the question of where to draw boundaries around “intelligent agents”, a true AI might have interesting boundary issues. Would there be just a single Google-scale AI or lots of little ones? If monolithic and extrospectively transparent, then all of humanity might contribute to debugging its “thoughts” and creating meaning for its symbols. Arguably this has already been happening for millennia in various media. As Lewis Thomas wrote, in comparing humans to ants, “It may be our biological function to build a certain kind of Hill.” The notion of the “individual” (something that cannot be divided) is a function of our embodiment and our unique developmental histories. Neither of these would necessarily apply to artificial intelligences – robot bodies could share hive minds, and an intelligence could be cloned fully developed into different embodiments without having to go through the tedious business of infancy, childhood, etc., by which human minds mature.
Cognitive individuality can also be called into question for humans, since our minds would certainly be very different without language, and without all the knowledge that is downloaded into them using language. In our constant exchange of knowledge via language do we conspire to create a hive mind, whose thoughts are in some sense more than the sum of the thoughts of the individual “knowledge workers” who comprise it, in much the same way as the brain’s thoughts are more than the sum of the “thoughts” of its individual neurons?  – although, in another sense a brain’s thoughts are precisely the sum of the thoughts of its individual neurons, if you are not a dualist. A possible resolution of this paradox lies in the observation that the whole brain can think thoughts that are far more complex than an individual neuron is capable of. Similarly, is humanity as a whole – the living intercommunicating population at a given moment – capable of thinking thoughts more complex than a single individual can manage? Or is that a false analogy – maybe “humanity” cannot think something unless there is a particular human to think it. Maybe there is a fundamental difference between an intelligent agent built from nonintelligent components vs. one built as an aggregation of intelligent components. In the latter case the whole is no smarter than its parts, since each is “AI complete”.

AI and Software Testing: Unit tests for microabilities like visual scene recognition or sentence parsing, Turing Test as a system integration test. The style of connection between systems being what Edelman called “reentrant”, or bidirectional (a natural property of many connectionist architectures and statistical “graphical” models, but not usually found in software architectures) means that the usual modular decomposition of functions in software engineering does not apply: the subsystem cannot be fully specified independently of the larger system. Any given module can call on the intelligence of the entire system to assist it with its “modular” function. E.g. general intelligence can be applied to assist with visual interpretation.

The notion that different people can have “the same idea” – Darwin and Wallace, say, or Newton and Leibnitz – suggests that the universe of thinkable thoughts exists independently of the thinkers – that innovators are not so much creators as discoverers of thoughts, and that different people can happen across “the same thought” in much the same way that two different explorers of a new country can happen across the same river. Thus suggests that the thought, like the river, exists independently of the thinker/explorer, in some Platonic universe of possible thoughts, just waiting for someone to come along and think it. This question has long been asked about mathematics – “is it invented or discovered?” – with different people coming down on different sides of the question. Here I generalize that question to thinking generally, or least to a broader class of thoughts than the explicitly mathematical. To the extent that a formal semantics could be given to our thoughts, that would bring them into the realm of deductive logic, and hence mathematics, so the question of whether thoughts are invented or discovered is then explicitly subsumed by the question of whether math is invented or discovered, since at least some thoughts would essentially have been mathematized. There may be thoughts that are not amenable to formal semantics – e.g. thoughts that rely on analogy or connotation or ill-defined symbols or no symbols at all – which can nonetheless be mathematized by providing a mathematical theory of (ill-defined) thought (e.g. a connectionist architecture), and so potentially enabling a definition of “the same thought” across sloppy thinkers as some kind of well-defined isomorphism. Two sloppy thinkers could then perhaps think “the same” thought in the same way that two lousy radios, each with its own characteristic static pattern, might nonetheless be tuned to the same radio station. Our brains, in this view, are (noisy) radio receivers for the Platonic universe of possible meaningful thoughts. As St.Augustine put it, man is halfway between angel and beast, a meat radio tuned to God’s radio station. Sadly, for most of us in most times and places, the reception is very poor.  


What are opinions? Are they different from facts, beliefs, or knowledge, and if so, how? Do we privilege them out of proportion to their value?
Here are some proposed answers. A fact is a proposition that is true, whether we believe it or not. It is a property of the world, not of our psychology. Beliefs, opinions and knowledge, by contrast, are in our heads (or stored in our recording media -- books, internet, etc. – so they can be put back into our / some else’s heads at a later time.)  Knowledge, according to one line of epistemological thought, is “justified true belief”. (This definition is a majority view but not universally accepted.)  “True” means “is a fact”, so knowledge lives in the intersection of facts and beliefs. It does not fill that intersection, however, because of that extra adjective “justified”. If we are right for the wrong reasons, that is not knowledge. Thales reasoned that the world must be made of atoms because infinite divisibility of matter seemed counterintuitive. He was right about atoms, but the argument from counterintuitiveness has since been repeatedly shown to be invalid by Copernicus, Darwin, Einstein, Bohr et al, Wegener, Hubble, etc. In the twentieth century evidence began to accumulate, incrementally providing a justification for the belief in atoms. There are also examples of “justified false beliefs” in the history of science, in the form of rejected theories like phlogiston or the ether (and perhaps supersymmetric particles, or gravitons, or axions). These accumulated some degree of justification, before accumulating an even larger body of contrary evidence. Crank theories, including paranormal belief systems, supersitions and many (but not all!) conspiracy theories, similarly have supporting evidence, as well as much stronger debunking evidence.
            These examples illustrate one flaw in the definition of knowledge as “justified true belief”:  justification emerges by degrees, it is not an all-or-nothing proposition, so a belief therefore cannot simply be knowledge or not-knowledge, but must exist in some continuum of justification. Opinion, then, is a belief somewhere on this continuum, falling short of clearly true by overwhelming universally accepted evidence (if there is such a thing) and yet hopefully beyond the converse, clearly false by overwhelming universally accepted evidence. In fact, it seems there is always someone crazy enough to believe any falsehood or disbelief any truth, “universally accepted” is an unattainable goal, while “clearly” and “overwhelming” both beg the question of what is sufficient evidence for a belief to be “true”. Popularity is not the answer; there are far too many examples of unpopular true beliefs and popular false ones. Maybe the reality is that the beliefs in our heads and the facts in the world don’t ever align perfectly; the best we can hope for is approximation. Does that even make sense? It seems to push the problem off into the definition of “approximation” – can that be made precise? I will defer that question for now.

            The notion that we have in our heads propositions which are meaningful and which have supporting evidence pro and con is interesting as a theory of psychology and as an engineering goal for AI, as well as a potential sine qua non of intelligence. It is certain a better, and richer, psychology of knowledge than saying we (or it) have/has a memory filled with mere facts.
            However it may be that even this view is too simplistic, because it takes as given the notion of a meaningful proposition. What is most of our beliefs are not simply somewhere on a continuum between justifiably true and justifiably false, but are rather on a continuum between meaningful and meaningless? What if meaning itself is not an all or nothing attribute? What if meaning, like justification, emerges by degrees? And perhaps, like failed theories, it sometimes fails to emerge, to resolve, to cohere? In physics the cliché of contempt is “not even wrong” – a theory so incoherent it cannot be said to be true or false. In theology, the notion of ignosticism is that the belief in god cannot even be said to be true (theism), false (atheism) or uncertain (agnosticism) because the belief is not well-defined, since neither god nor existence have been defined sufficiently to make the proposition meaningful.
            How is coherence related to justification? Are they independent properties of a thought or are they related? If they were independent you could have a thought that was incoherent but justified – it is not clear that this makes sense (i.e.  is coherent). Interestingly, the preceding sentence illustrates that we are capable of reasoning explicitly not just about evidence but about coherence – we (some of us) ask ourselves all the time whether our ideas make sense. If justified incoherent beliefs make no sense, then it makes more sense to say that coherence is a prerequisite of knowledge: that knowledge must be coherent and justified belief. And if we allow that both coherence and justification emerge by degrees, we end up with a psychological theory of knowledge in which our heads are filled with propositions with varying degrees of coherence and justification.
            Is the notion of coherence coherent? What does it mean for a belief to be coherent or incoherent – or worse, somewhere in between?
            What do we mean when we say something does or doesn’t make sense? Can people disagree on this? How?
            One example of nonsense is Chomsky’s famous “Colorless green ideas sleep furiously”, designed to make the point that sentences can be syntactically well formed while devoid of semantics (a.k.a, meaning). Another mother-lode of nonsense is Lewis Carroll’s work, for example the opening of Jabberwocky: “Twas brillig and the slithy toves did gyre and gimble in the wabe…”. These examples are subtly different. The Chomsky sentence is self-contradictory in its first two words, since it is impossible to be colorless and green at the same time, and in a different way in the third word, since color is not an attribute of ideas. The latter restriction might conceivably be bent through the use of metaphor – if moods can be blue, maybe ideas can be green? But certainly not green and colorless simultaneously. Ideas don’t normally sleep, though again metaphor could ride to the rescue: did Mendel’s ideas sleep until their rediscovery? Did democracy sleep between ancient Greece and the American Revolution? Sleeping furiously is problematic – one can sleep uneasily, fitfully, deeply – but furiously? Still not totally inconceivable; if one was furious on falling asleep and somehow managed to fall asleep anyway. Or perhaps furiously means “with ferocious concentration”, and the sleeper is intent on sleeping like never before.  The sleeper in this case being ideas… Note that any metaphoric rescue of “sleeping ideas” requires a redefinition of “sleep” from its default meaning, though such alternative or metaphorical uses of “sleep” are familiar. The switch to a metaphorical meaning of sleep is mandated by the fact that the default meaning absolutely cannot apply to an idea. Ultimately, even if some of the sentence fragments can be stretched into near coherence, the sentence fails to cohere overall; we simply cannot construct a meaning for it. The combination of self-contradictory and incompatible combinations eludes the most extreme metaphorical rescue. It doesn’t even connote, never mind denote, anything.
            Jabberwocky is another story. Here again we have syntactic sense and semantic nonsense, not because the words are incompatible but because they are unfamiliar. To an English speaker they have etymological resonances – they sound like real words, they have nearly recognizable stems, and yet they are unknown. [See Hofstadter’s brilliant essay on the challenges of translating Jabberwocky into other languages in “Godel Escher Bach”.] If definitions could be supplied, the sentence might become sensible, as in fact happens later in the story when Humpty Dumpty explains the verse to Alice. That the definitions refer to nonexistent creatures doesn’t make the sentence meaningless any more than it would for a sentence about unicorns or dragons. The difficulty in interpreting sentences containing unfamiliar words confronts prelinguistic children, second-language learners, and language users generally to the extent that our vocabularies are an incomplete subset of the words we are exposed to in our environments. Humans excel at figuring out the meaning of unfamiliar words from context, or figuring out the meaning of a sentence despite not knowing the meanings of all the words. This shows that there is often a degree of informational redundancy in sentences. Clearly as the fraction of unfamiliar words increases it becomes more difficult to reconstruct the meaning – an effect Carroll toys with in the poem.
            As we learn a language, individual words progress in our minds through a lifecycle of definition, from the completely unfamiliar to the routine. At any moment different words in our vocabularies exist at different points on that continuum of familiarity. Operationally, we say we understand a word if we can readily interpret what it means in any usage context we encounter. However, some usages are more challenging than others – poetry, dense or opaque prose, speaking cryptically or elliptically… Sometimes we think we understand a word only to see it used it a way that causes us to question whether we really understood it – perhaps we incorrectly inferred its meaning from a previous context. Sometimes we try to explain a word to someone else, only to realize that perhaps we don’t really understand what it means. In these cases of incomplete understanding, what is the state of the “meaning” in our minds? Is it justified? Is it coherent? Perhaps the notion of justification through evidence can be extended from propositions down to the level of language understanding: the evidence that the word means such-and-such in this sentence includes the fact (or memory) of it having been used in previously encountered sentences wherein it was judged to have been successfully interpreted to mean something similar. In this view, a poorly understood word or sentence may have multiple candidate interpretations, or none, or perhaps only a very vague one (e.g. In Jabberwocky, some things are clearly doing something somewhere…) but these candidate interpretations are themselves meaningful not nonsensical.
            Not all contradictions are as nonsensical as colorless green things. Consider Churchill’s (?) witticism that democracy is the worst system of government, except for all the others. To be the worst means that all the others are better, while the second phrase asserts that all the others are worse – a contradiction acknowledged by the use of except, whose meaning carves out a space for apparent contradictions to exist. Except is used to flag exceptions from a general rule which still holds in most cases; in this case it completely overrides the general rule, producing an expectation violation. The humor arises not only from this override, but from the substance of the observation, that democracy really is a very flawed system of government, but other systems are even more flawed. It is a backhanded compliment, or perhaps damning with faint praise. Consider some variations: democracy is a better system of government than any other means almost the same thing, but lacks both the acknowledgement of flaws and the humor. Or All strawberries are blue except for all strawberries has the same blanket exception structure, yet pretty much fails at humor (though perhaps a context could be constructed in which it would be mildly funny). It is not nonsensical however. It is perhaps incoherent at the pragmatic level of interpretation – why would anyone say this? But it is clear what it means.
            The witticisms attributed to Yogi Berra often have a similar structure to Churchill’s comment, in that they are superficially self-contradictory – Nobody goes there anymore, it’s too crowded – but the meaning can be rescued, in this case by reinterpreting nobody to mean nobody important. Here again the humor is bound up with the detection and resolution of the apparent contradiction.
            Other Berra-isms play with obviousness, such as It ain’t over til its over. Like the blue strawberries example, this statement seems nonsensical at the pragmatic rather than the semantic level: sort of like saying 1=1. However under the obvious surface meaning lies a deeper truth that even a highly probable future outcome is not a certainty, in the darkest hour hope springs eternal, and so forth. The more straightforward It ain’t over yet captures this deeper meaning but lacks the humor.
            Returning to scientific examples, we can discern a different type of nonsense: words which presume the existence of something that doesn’t exist. For example, does the Earth experience friction as it moves through the ether? We now believe that the ether does not exist, so the question no longer makes sense. Does that mean it is nonsensical? Not really; we can still understand the question as meaningful within a framework of assumptions that we no longer consider justified. The theological cliché about how many angels can dance on the head of a pin is another example of this kind; within a framework of belief where angels are real, it is reasonable to try to understand their properties, such as whether they take up space or can overlap with each other. (How many neutrinos pass through the head of a pin each second? would be a legitimate question in the belief system of modern physics.) Such debates are castles in the air; they make sense within a particular system of beliefs but not outside it. This beg the question of whether all of our current beliefs are in the same boat. Many of the day to day concerns of our ancestors make no sense to us now. Was the color of the chicken entrails a bad omen for the battle? Is this a propitious day to sacrifice for the harvest? Are God the Father and God the Son of the same substance? Is it permissible to put out the fire of a burning house on the Sabbath? Did the old lady put a curse on my son that caused him to go blind? Is her cat a demon? Is this tulip worth the asking price? It seems presumptuous to believe that many of our current concerns will not seem as ludicrous to our descendants as these questions do to us.  
Our inventory of nonsense so far consists of the following:
·         Contradictory or incompatible combinations of phrases that cannot be rescued (made coherent) by metaphorical reinterpretation of the meanings.
·         Unfamiliar words, partially familiar words, familiar words used in unfamiliar ways (Humpty Dumpty again) or words used metaphorically (in ways that can be familiar or unfamiliar).
·         Ambiguous (i.e., multiple) or vague (very general/abstract) meanings.
·         Meaningful contradictions, i.e. surface contradictions that mask a deeper truth.
·         Castles in the air: Propositions or questions that pertain to objects or effects that are presumed to exist within a particular belief system.
In order to detect and cope with these different forms of nonsense, language understanders must possess an ability to detect contradictions and incompatibilities among meaning fragments as they are combined, an ability presumed to exist by linguists and natural language understanding programmers. Understanders must also have an ability to propose alternative meanings for a sentence, and to evaluate the likelihood that each reflects the speaker’s intent. Ambiguous interpretations are those for which this evaluation fails to produce a unique result; puns and the Churchill and Berra examples play with this effect. The process of meaning extraction is likely a process of evidence integration that creates justifications for the meanings of sentences built from known words, and for inference of the meaning of unfamiliar words from context. Thus, having taken a rather length detour into the theory of nonsense, we return to the idea of justified belief, and of understanding as a process of belief justification through evidence integration.
            Our understanding of evidence has evolved over the centuries, from the ancient world’s early articulation of the concepts of rhetoric and logic, followed by Talmudic and scholastic argumentation, leading to the scientific revolution and the notion of observation and experimentation, as well as evolving standards in the realms of jurisprudence and policing. Underlying these cultural shifts is the psychological / neurological bedrock of “naïve evidentiary reasoning” that we use in day-to-day life, which has likely not changed much in that time. We use this to learn language and to make sense of, and to gain control over, our environments.
            Note that at first blush, connectionist approaches would seem to have a very different view of emergence of meaning than symbolic ones. A connectionist system starts life with equal, or random, weights – essentially a state of semantic nonsense. Over time, with training, meaning emerges gradually as the weights are tuned. In a connectionist system trained to recognize visual patterns (e.g. to distinguish cats and dogs) you end up with neurons having “visual fields” that are occasionally somewhat intelligible, especially at the lower levels – edge detectors in different orientations, for example – but at higher levels of a deep learner there is no guarantee that the neuron’s behavior will be intelligible. This paper attempts to reverse-engineer comprehensible properties from image network neurons, with some success.
In a symbolic system, by contrast, a symbol with no semantics implied by its connections to other symbols, and ultimately to real world objects, is considered useless or worse, a pathological semantic state. In reality, of course, computational symbols also arrive into the world naked and devoid of meaning, and must be supplied with meaning by creating connections to other symbols. Back in the heyday of “knowledge engineering” it behooved the engineer to supply such connections so as to ensure that the symbol’s meaning was pinned down to its intended value, although more often than not this was an exercise in self-delusion. As long as the symbol was used in very limited and prespecified ways the illusion could be sustained, but if you wandered from the tested examples the brittleness of the semantics would be quickly exposed. Even Watson suffered from this brittleness during the Jeopardy championship, as shown by its absurd second-choice guesses even on questions it got right.
Intermediate between connections and symbolic representations of meaning are subsymbolic representations of word meanings or document topics that use numeric vectors. In these systems, the set of possible meanings or topics is discretized into a finite set of properties which can apply more or less well to a word or topic. Each word or topic is thus characterized by the strength of its association to all possible topics. The clearly a connectionist-friendly way to think about meaning, this approach is rooted in information retrieval ideas such as latent semantic analysis that use more traditional stastistical methods. A key concept is distributional semantics, i.e., that meaning emerges from the statistics of usage over large collections of text.
            How many of our day to day beliefs are of this semicoherent type? The election in progress pits Hillary Clinton, stereotyped as untrustworthy, against Donald Trump, stereotyped as a racist fabulist. It is quite possible that the selection of a president will turn on these stereotypes, or opinions. But are they even meaningful?  Is Hillary Clinton “really” untrustworthy? At one level trust is a feeling, not a belief, and you have it or you don’t, and are entitled to vote accordingly. So is an opinion a belief, a feeling, or a mixture of both? Are feelings justified? I don’t mean in a normative sense – should they be justified – but as a question of psychology – do we have reasons for feeling the way we do? I think the answer is clearly yes, though we may not understand or be able to explain them. Do feelings have meaning in the sense that beliefs do? Trust reflects our assessment of the probable future actions of a person over some sphere of activity. We trust our spouses to be faithful, our bankers not to embezzle, our politicians to represent our interests.  So is a feeling that someone is untrustworthy equivalent to a proposition regarding the probability of future betrayal, with the same justification-by-evidence machinery as any other proposition?
John Went to the Store
I have been wanting to get beyond thinking about AI to actually writing some code. In my unfrozen caveman AI-guy state, I have been poking around the internet to try to see where the state of the art is now – deep learning, Watson, Alexa, Viv, open NLP, etc.  – and specifically looking for open source code and data that would give me a head start in developing something interesring. Yesterday, with some extra Thanksgiving holiday time on my hands, I finally took the plunge and downloaded the Python NLTK package, which has an associated Nutshell book on NLP, and started playing around. Immediately, disappointment set in. It can to POS-tagging (part-of-speech), and apparently can sometimes parse sentences if you provide the right choice of parsing algorithm, though even then you will suffer from “ubiquitous ambiguity”. Both of these strike me as wrong-headed: the former, because identifying the syntactic role of a word in a sentence cannot be divorced from the process of constructing a meaning for it, particularly since role-ambiguous words get disambiguated by semantic feedback (e.g. Can you google Google?) – and the latter because it begs rather than solves the question of how ambiguity should be handled.
            Now for better or worse I am a compulsive reinventor of wheels, even if mine sometimes turn out elliptical (or square). I believe in reuse; I certainly am happy to reuse a regular expression package or a Python iterator library; I am not crazy. But when it gets to algorithms I tend to want to roll my own. I tried to resist that temptation, I really did – I looked, I downloaded, I prototyped. But I was disappointed. The Nutshell book is completely introductory, the NLTK package seems to consist of just a few low level utilities. Since I am doing this to explore mental processing, rather than any practical application, I would rather have a toy that is pointed in the right direction than a robust package that reduces the problem to uninterestingness.
            For my first text I took a random abstract from PubMed that happened to be open in my browser. Too hard: I can barely understand it. (What does “the genetic architecture of cancer” mean?). Start easier, I though. How about “John went to the store.”? The NLTK POS-tagger can handle this:
[   ('John', 'NNP'),
    ('went', 'VBD'),
    ('to', 'TO'),
    ('the', 'DT'),
    ('store', 'NN'),
    ('.', '.')]
These are taken, I believe, from the Penn Part-of-Speech tagset.
      14.   NNP   Proper noun, singular
      28.   VBD   Verb, past tense
      25.   TO    to
      3.    DT    Determiner
      12.   NN    Noun, singular or mass
But so what? Has this gotten the program any closer to understanding the sentence? What does the sentence mean, anyway? Some guy named John went to a store, assuredly. Who is John? Does this refer to a known individual or a new one – is John a definite or indefinite noun? If the context were an article about the Beatles we would guess this refers to John Lennon, a definite person. If the context were the New Testament, it might be the apostle or John the Baptist, so still ambiguous, but narrowed down to a handful of definite possibilities. If someone uttered this sentence to someone who happened to be married to someone named John, they would likely resolve the reference to their husband. Absent any context we infer this is some unknown person named John. This ability to find or construct a referent (i.e., a meaning) for a noun or noun phrase is termed reference resolution. It implies that the hearer maintains a database (or knowledge base) of referents.

Note that we also infer certain things about this unknown John with high-probability: that he is a male human and that he comes from an English-influenced country, rather than someplace inclined to name their children Gianni, Jan, Yannis, Johann, Giovanni, Ian, Zhen-Xi, Muhammud,  or “Dances with Wolves” (whose given name was “John”).

            Now we come to “went”. Past tense of go, typically meaning a change in location, though metaphorically applied to other transitions (“went mad”). The mechanism of the change is unspecified: did he walk, drive, use a StarTrek transporter, or fall 40 stories through the roof of the store?

               “To” is interesting in that it does not normally stand alone. If we had ended the sentence after “John” or “John went” the semantics would be clear, though perhaps not the pragmatics (why the sentence was uttered). “John went to” by contrast is fundamentally incomplete, it calls out for more words to follow. “To” thus sets up an expectation or an incomplete conceptual structure in the understander to be completed by later input. The Penn POS labeling system gives “to” a label all to itself, oddly (“from” gets labelled with
6.      IN      Preposition or subordinating conjunction). “To” is a property of certain directed actions – physical transport (“PTRANS” in Schank’s early semantic ontology), as well as transfer of information (e.g. sent an email to - MTRANS), or possession (I gave it to, ATRANS). Verbs that do not normally have a destination – e.g. eat, learn, die – can nonetheless admit of a “to”: eating to satiety, learning to speak French, dying to save someone else. As the English infinitive-former – to be, to do, to go – it combines with verbs to modify them in a way that has nothing to do with spatial direction and more to do with purpose. One can extend our sample sentence with more to’s: John went to the store to buy some flowers to give to his wife. Here we have 4 to’s: the first indicates the destination of the went action, the second, the purpose of the went action, the third, the purpose of the buy action, and the last, the recipient of the give action. Native speakers effortlessly map each to its appropriate meaning. This is not simply a matter of position, e.g. that the first to looks for the preceding verb. Consider John went to buy some flowers to give to his wife. Here we eliminate the first PP, so the former second to is now first, but still attaches to the verb after it rather than the verb before. Perhaps “to the store” is a destination in search of a directed verb, whereas “to buy some flowers…” is a purpose in search of an event in need of one. The shortest word in the sentence may be the most complex from a linguistic standpoint.
--------------------- 12/11/16
I have decided for the moment not to pursue prototyping with existing NLP tools, since these encode a paradigm I do not subscribe to – feedforward from morphology to syntax to semantics, etc. Given my unfrozen caveman AI researcher state my only chance of doing anything interesting is to strike out in a new direction: go big or go home.
So: key idea is “like” – that is, similarity-based reasoning is at the core of mental processing. Algorithmic sketch for an NLP based on this idea:
State consists of:
·         KB: The structure of the KB reflects considerations of contexts and evidence discussed above, while deemphasizing the importance of formal semantics due to the sloppy reasoning processes described below. This means it is less important to define the semantics of every predicate, but more important to have a way to represent evidence and context-dependence clearly. Both are second-order concepts, in that they require reasoning about an assertion, rather than simply with an assertion. Contextual support is presumed to sweep in modal and temporal logic capabilities. Historical memory for NLP is assumed to contain not simply parses of previously encountered sentences up to semantic / pragmatic interpretation, but the sequential state of the parses during understanding as a temporal sequence.   
·         Current Parse: A parse in progress is a representation of the same sort as the KB, including some KB nodes (anchors) as well as new ones. The current parse is viewed as an ongoing KB query (and insert!), which is updated as the parse is modified. 
·         Activation State: The current parse induces an activation state over the KB via highly parallel spreading activation (SA). Activation begins at anchor nodes and spreads via weighted edges.
·         Bindings (“like” relations) between KB elements and current parse elements.
·         Other elements TBD below.
Processing consists of:
1.      Addition of a new subgraph to the current parse.
2.      Updating of the activation state based on the new element.
3.      Selecting of candidate donor(s) with probability proportional to activation state.
4.      Proposal of a binding (“like” relationship) between a donor and a parse element (in a new child context?). (Can bindings be used in isolation or must they be as sets? Will SA tend to ensure compatible bindings? Is SA aware of contexts or evidence?)
5.      Similarity/Difference/Prediction analysis of the proposed binding: KB attributes of the donor are mapped to the recipient (similar(!) to structure mapping), where they are classified as:
a. Confirmatory – already known to be true of the recipient
b. Contradictory – known to be not true of the recipient.
c. Prediction (Conjecture) – not known to be true or false.
6.      Plausibility Assessment of the binding is based on a balance of confirmatory and contradictory info. (And utility of prediction?)
7.      Acceptance of “sufficiently plausible” interpretations? (Is this necessary? Or can it be handled by activation state of the child contexts?). If there is no acceptance then what is the point of plausibility assessment? If no acceptance what is protection against bad mappings?
8.      Instantiation of predictions as part of the (child-contexts of) the current parse (with evidence initialized to the mapping).
9.      A prediction which is later confirmed must update the plausibility of the mapping(s) that produced it. (Semantic Hebb Rule).
10.  A sufficiently plausible interpretation must update the weights of the SA edges that proposed the binding.
Maybe algorithm sketch is too optimistic – more like algorithm wish-list – too much missing detail.
Some properties which this architecture would hopefully have:
·         At least two kinds of learning: creation of new structure and altering of activation weights.
·         Psychological plausibility re fast parallel memory search and slower sequential symbolic reasoning.
·         Inherent prototype-driven reasoning: an instance successfully used in an interpretation becomes more likely to be reused, providing for a graceful transition from reminding to type-based reasoning as the initial instance becomes a prototype.
·         Single-instance generalization.
·         Possibly (needs additional machinery?) – abstraction via learning which aspects of the mapping transfer successfully and which do not. Would need some sort of transfer probability weight – same as SA weight or different?
·         Inherent analogical/metaphorical reasoning.
Spent holiday spare time reading up on neural network state of the art, & setting up deep learning courseware on Amazon. Most interesting developments are neural Turing machine and similar coupling of nets to knowledge bases.
No direct hits yet on my ideas for using a similarity-based retrieval engine as the core for NLP. Idea still not fully coherent. Going back to looking for a semantically annotated NLP corpus to develop the idea on. Looking at Groningen Meaning Bank. Maybe also Kabob if I can find the data. Having looked at both I am not impressed. Maybe the GMB annotation workbench could be of use for developing new annotations. Problem with GMB is, XML rep hard to interpret, and it builds in a lot of stuff that should be emergent in my model, I think – e.g. syntactic assignments. 
Also looked at similarity-based reasoning, querying, learning. No joy.
So back to handcrafting some examples. 
What is my theory? That a small number of cognitive operations (a.k.a. database operations) suffice to explain language and thought. That the key operation is similarity – or reminding, shades of Schank! This view extends Schank’s model to put micro-level reminding at the core of the thought process. A noun phrase is recognized as such because it reminds you of other noun-phrases you have seen, and similarity to those stored noun phrases – one of them, or an abstraction of many of them, it doesn’t matter – guides the interpretation of the new noun-phrase. There is no inherent distinction between syntactic and semantic processing, though there may be an emergent distinction based on the degree of abstraction of the knowledge involved, with synactic concepts being highly abstracted from their original experiential source, to the point where they refer to nothing. The reminding engine treats partially assembled conceptual structures (working memory contents) as queries and generates or augments mappings between the structures and prior structures in long-term memory. These mappings are themselves annotated with designations of similar or different, at least, and moreover with some kind of description of the type of similarity (e.g. an accent has a phoneme transformation model). I have no theory of the space of similarity types at this point. Whatever they are we are capable of reasoning explicitly about them: is Trump really a Hitler; is the atom really a solar system, etc. Metaphor and analogy are thus exaggerated or introspected forms of the underlying core process. Remindings drive transfer of knowledge from previous situations to new ones by mapping structures across similarities links to produce conjectures (explanations and predictions) about the current situation. The representation is fundamentally modal and evidential, allowing representations of time, contexts, alternative possibilities, fictional story-contexts, hypotheticals, opinons, arguments, contradictions, paradoxes, rhetoric, etc. It is fundamentally experiential, in that everything is encoded as sequences of events (shades of Elephant!), including previously encountered sentences and their parses. Abstraction happens as a by-product of reusing a reminded event over and over, so that it comes to function as a type rather than a token, but there is no fundamental distinction between instances and types – it is an emergent distinction. (This may not be always true – an “innate” universal grammar could facilitate language learning by 
providing a set of preabstracted concepts useful for language learning and understanding, which would mean syntax is not entirely emergent.) Learning happens because experience is continuously creating new semantic structures – i.e., parsing = understanding = learning – and because abstraction is continuously happening in a graded fashion as a consequence of reminding to preferred prior structures, which come to function increasingly like types. The system is fundamentally prototype driven; logical inference could be layered on, but only by reasoning analogously to previously encountered inferential chains. The integration of the input (perceptual) stream and the inferential stream should be asynchronous, and both need to be governed by an attentional process that governs the tradeoff between thinking deep and thinking fast. (The inferential stream itself may be multithreaded, or perhaps some more continuous analog of multithreaded – perhaps via SA?). There may be a notion of priming which implements an incremental reminding mechanism. That is, reminding may not always or only construct or extend one or more similarity mappings between present and past, it may prime aspects of past memory, increasing the probability that a subsequent input or inference will add them to the active mapping(s). It is not enough to construct mappings; they must be used to generate hypotheses: predictions and explanations. (Prediction = hypothesis about the future; explanation = hypothesis about the past.) These extend the model of the present situation. Variable binding is implemented via this more primitive underlying similarity mapping. The reminding (or mapping) provides the initial evidence for the hypothesis; subsequent evidence may augment or refute the hypothesis. Hypotheses that accumulate refutations must be pruned from the current context as part of consistency maintenance (or attention?).
The above rambling implies a number of distinct processes:
·         Perceptual input stream that extends the memory model by adding new percepts to experience sequence.
·         Reminding process creates or extends similarity / difference mappings from current to past, may prime past memory structures and use previous primings to choose among candidate remindings.
This is too high-level – too much magic. Break it down. Atomic operations on the representation must include role-filling, frame initialization…or am I slipping back into formal logic? If the theory is based on similarity over conceptual structures, we must be able to build conceptual structures. At a minimum there are objects with properties – p(x) – and relations between objects – p(x,y). The latter being more general we can assume relations WOLG, i.e., a triple-store (e.g. we could introduce a meta-predicate “property” and say property(x,p) instead of p(x), or better simply let the second argument be NULL to reduce binary predicates to unary. We can then use either form as needed and still live within a triplestore. To allow evidentiary reasoning, propositions must themselves be objects, so we have id:p(x,y), i.e., a quad-store or named graph. (Also useful for priming of assertions, if needed – or is priming limited to objects?). Do we need reification/2nd orderness – i.e., can p appear as an argument of some other predicate? Certainly id can in order to support evidentiary or model reasoning.

Need to incorporate contexts as well. Let’s say that contexts are objects to which propositions are linked. Are truth values associated with propositions in contexts? Too logical. Who says there even are truth values? Maybe just evidence? So evidence exists in a context? How about a context is just a bundle of propositions, regardless of their evidence or “truth”. Does a proposition exist in only one context or can propositions (or their ID’s) be shared across contexts? How about assert(context, predicate, arg1, [arg2]):id. This will give us a different pid every time p(x,y) is asserted in a different context. Not clear if it matters. Shouldn’t call it assert though – slipping back into logic. How about insert – makes it clear we are just talking about database operations. So implied in this are the ability to create new objects, predicates, propositions and contexts – call these metatypes. We can allow insert to be a universal constructor for all metatypes by treating the returned id as a new identifier for the construct. This forces us to accept reification of contexts as well as predicates. In particular we could use
to create a new id of unspecified metatype. Should this be allowed? Let the metatype be determined by usage context? If reification is allowed, predicates, propositions and contexts can all be objects – can arbitrary objects be treated as predicates, propositions and contexts? Sounds dangerous. Alternatively we can enforce strong metatyping via some builtin predicates:
·         insert(null,Sys:Context,null,null) – create a new context with no parent.
·         insert(cid,Sys:Context,null,null) – create a new context which is a child of a specified context. (Implies context trees.)
·         insert(cid,Sys:Relation,null,null) – create a new relation (predicate) (with no context?). For convenience might want to supply a name (or unique name prefix) as arg1 to make the database more introspectable. Can cid be null?
·         insert(cid,Sys:Object,null,null) – create a new object in (nullable?) context.
Alternatively, we could stick with a quad store and implement contexts as objects that are related to propositions: Sys:inContext(cid,pid). This eliminates the need to impose any a priori structure on the context graph. Also allows (but does not require) pid’s to exist in multiple contexts.

Now we need to represent events, sequences, mappings, evidence, and hypotheses on top of this framework. It may be convenient to introduce a function this which captures the ID of the proposition it appears in. This will provide a skolem-like mechanism.
·         Sys:Next(cid,this) – create a new context which is a direct successor of another, provides context sequences. Note that new cid=pid.
·         Sys:Object(cid, this) – create a new object in a context, with oid=pid.
·         Sys:Word(cid, string) – occurrence of a word in a context. (The representation does not preclude multiple words in a context, but this may be a desirable convention to have each word be a distinct event, and hence context. Whether the link should be to a string or a node associated with the string is unclear. Sys:Word(cid, this) would define a new word that is the ID of the assertion defining it – very Godelian, but probably not useful. On the other hand, Sys:Word(this, string) would create a new context for a word – without specifying a previous one. May be a useful way to start a new text.
·         Sys:Like(idA, idB) –A is similar to B. A and B can be any metatype. Reflexive, not necessarily transitive. A “like” proposition will be called a mapping. The ID of the mapping can be used to join mappings together via mapping contexts, and to annotate mappings with evidence and transformations. Note that Sys:Like(idA, this) will create a new object which is like an existing one, the analog of instanceOf(type,this) in more traditional KBs. Like, like Isa and instanceOf, is a metapredicate (2nd order), in that it licenses a class of inferences that quantify over (1st order) propositions. In particular if context C1 is like context C2, and there is a proposition Pid:P(A1,B1) in C1, and there is A2 and B2 which are like A1 and B1 in the same mapping context (= the id of Like(C1,C2)), then we can hypothesize P(A2,B2) in C2, providing the mapping context as the context for this proposition.
If A2 exists but B2 does not, we could do B2:Like(B1,this) to create one, thereby conjecturing the existence of a B2 in C2 and satisfying the conditions for the above rule. Similarly if B2 exists but A2 does not. If neither exist, creating both would lead to a runaway copying of all of C1 to C2, which seems undesirable. Some sort of attention mechanism may be needed to control map extension.
A hypothesis thus consists of one or more propositions in a context, possibly involving the creation of one more new objects, and supported by some evidence (not yet discussed). Hypothesis creation is triggered by a reminding, which creates one or more Likes in a new mapping context. 
·         Evidence: Here I paused, returning on 1/29/2017 to briefly record some thoughts and references. I paused because the evidence problem seems to me larger than all the others put together. It encompasses classical rhetoric, legal and scientific reasoning, propaganda, and all the arts of persuasion, except perhaps logic, which trivializes the problem out of existence. The news has lately been full of the seemingly deliberate lies or fabulations of our new President Trump on points of largely trivial fact – whether more people attended his inauguration than Obama’s, or whether he won the popular vote, or would have if lots of people hadn’t voted illegally. Kellyanne Conway’s explanation that the Trump administration uses “alternative facts” has resulted in a surge of sales for Orwell’s 1984. These examples illustrate that in addition to evidentiary support for a proposition, there is sometimes and for some people an element of teleological or pragmatic support – the proposition is true because it should be true or because someone needs it to be true or wants someone else to believe it irrespective of its “truth”. For Trump, having people believe he won the popular vote is important because, if it were true, it would enhance his legitimacy and possibly make people more inclined to support his policies. The value of this outcome outweighs whatever premium he might be inclined to place on being truthful. This of course is the very definition of a lie – an incorrect assertion communicated in order to elicit desired behavior on the part of a gullible audience. Trump may be more flamboyant in his dishonesty than most, but we all lie frequently. This suggests that a significant part of our cognitive process involves not just computing our own beliefs, but choosing communications to optimize for desired effects on the beliefs of others, as well as reverse engineering incoming communications to dissect the motivational from the informational components.  Communication doesn’t have to be false to be manipulative; arguably any communication is manipulative since it intends to alter the mental state of the recipient.
A pure belief-optimizing process can perhaps be viewed as implementing some normative theory of evidence – Bayes/Jaynes, or perhaps Dempster-Shaffer – plus perhaps a layer of suboptimal heuristics or biases, per Kahneman, Ariely, etc. The normative model necessarily involves several components:
·         A multiplicative rule for the likelihood of new facts inferred conjunctively from uncertain facts
·         An additive rule for independent lines of evidence for the same facts
·         A negative or discounting rule for evidence against a fact, or for evidence in favor of an incompatible fact
This starts to look a lot like Jaynes’ specification for the belief processing of an intelligent robot. However in a similarity-driven system we need to explain not only the likelihoods of the premises, but the method used to assign a likelihood to the inference, given that the inference is similarity based. In the first wave of expert systems, numerical confidences were assigned to inference rules by the knowledge engineer. In similarity-based reasoning we would need a confidence that the similar situation is a reliable guide to the current one.

For a disembodied computer intelligence, or for ourselves for much of what we know, much of our knowledge depends directly or indirectly on what we have been told by others. I believe that the Eddington expedition measured a bending of light around a solar eclipse that was more consistent with general relativity than Newtonian gravity, but if truth be told I have no evidence, other than hearsay, that there was such an expedition, or such a person as Eddington, or that the result was favorable to general relativity, or even that general relativity is a thing capable of generating predictions. Most of what we take as facts are really things that were told to us by someone we trusted. They in turn got most of their facts from someone else. How often and how reliably this chain of broken telephone (or Chinese Whispers, in England) produces correct knowledge in our heads is an open question. We attempt to impose order on the assertions we take in, deciding which ones to believe and how much. An important clue that helps us in this process is consistency – we are more inclined to believe claims that are consistent with what we already believe to be true. Man sinks in water is more credible a priori than man walks on water. Hearing consistent accounts from different sources increases our confidence, but this can be vulnerable to our inability to correct for correlation. If we hear the same story second hand from 10 sources that all heard it from the same initial source, it has no more credibility than the single source had – except perhaps for the increased credibility that comes with having been endorsed by the secondary sources. If hearing the same thing from many sources were sufficient for an assertion to be true, every miracle, conspiracy theory and internet rumor would be true. Many people indeed seem to construct their beliefs this way.


No comments:

Post a Comment