Natural Language Understanding – NLU

Natural Language Understanding – NLU is a subtopic of Natural Language Processing – NLP in Artificial Intelligence – AI that deals with Machine Reading Comprehension.

The process of disassembling and parsing input is more complex than the reverse process of assembling output in natural language generation because of the occurrence of unknown and unexpected features in the input and the need to determine the appropriate syntactic and semantic schemes to apply to it, factors which are pre-determined when outputting language. There is considerable commercial interest in the field because of its application to news-gathering, text categorization, voice-activation, archiving, and large-scale content-analysis.

Scope and Context

The umbrella term “natural language understanding” can be applied to a diverse set of computer applications, ranging from small, relatively simple tasks such as short commands issued to robots, to highly complex endeavors such as the full comprehension of newspaper articles or poetry passages. Many real world applications fall between the two extremes, for instance text classification for the automatic analysis of emails and their routing to a suitable department in a corporation does not require in depth understanding of the text, but is far more complex than the management of simple queries to database tables with fixed schemata.

Throughout the years various attempts at processing natural language or English-like sentences presented to computers have taken place at varying degrees of complexity. Some attempts have not resulted in systems with deep understanding, but have helped overall system usability. For example, Wayne Ratliff originally developed the Vulcan program with an English-like syntax to mimic the English speaking computer in Star Trek. Vulcan later became the dBase system whose easy-to-use syntax effectively launched the personal computer database industry. Systems with an easy to use or English like syntax are, however, quite distinct from systems that use a rich lexicon and include an internal representation (often as first order logic) of the semantics of natural language sentences.

Hence the breadth and depth of “understanding” aimed at by a system determine both the complexity of the system (and the implied challenges) and the types of applications it can deal with. The “breadth” of a system is measured by the sizes of its vocabulary and grammar. The “depth” is measured by the degree to which its understanding approximates that of a fluent native speaker. At the narrowest and shallowest, English-like command interpreters require minimal complexity, but have a small range of applications. Narrow but deep systems explore and model mechanisms of understanding, but they still have limited application. Systems that attempt to understand the contents of a document such as a news release beyond simple keyword matching and to judge its suitability for a user are broader and require significant complexity, but they are still somewhat shallow. Systems that are both very broad and very deep are beyond the current state of the art.

Components and Architecture

Regardless of the approach used, most natural language understanding systems share some common components. The system needs a lexicon of the language and a parser and grammar rules to break sentences into an internal representation. The construction of a rich lexicon with a suitable ontology requires significant effort, e.g., the Wordnet lexicon required many person-years of effort.

The system also needs a semantic theory to guide the comprehension. The interpretation capabilities of a language understanding system depend on the semantic theory it uses. Competing semantic theories of language have specific trade offs in their suitability as the basis of computer-automated semantic interpretation. These range from naive semantics or stochastic semantic analysis to the use of pragmatics to derive meaning from context.

Advanced applications of natural language understanding also attempt to incorporate logical inference within their framework. This is generally achieved by mapping the derived meaning into a set of assertions in predicate logic, then using logical deduction to arrive at conclusions. Therefore, systems based on functional languages such as Lisp need to include a subsystem to represent logical assertions, while logic-oriented systems such as those using the language Prolog generally rely on an extension of the built-in logical representation framework.

The management of context in natural language understanding can present special challenges. A large variety of examples and counter examples have resulted in multiple approaches to the formal modeling of context, each with specific strengths and weaknesses.

Natural Language Processing – NLP vs. Natural Language Understanding – NLU: What’s the Difference?

It is easy to confuse common terminology in the fast-moving world of machine learning. For example, the term NLU is often believed to be interchangeable with the term NLP. But NLU is actually a subset of the wider world of NLP (albeit an important and challenging subset).

Natural Language Processing (NLP) refers to all systems that work together to handle end-to-end interactions between machines and humans in the preferred language of the human. In other words, NLP lets people and machines talk to each other “naturally.”

NLP is a critical piece of any human-facing artificial intelligence. An effective NLP system is able to ingest what is said to it, break it down, comprehend its meaning, determine appropriate action, and respond back in language the user will understand.

Meanwhile, Natural Language Understanding (NLU) encompasses one of the more narrow but especially complex challenges of AI: how to best handle unstructured inputs that are governed by poorly defined and flexible rules and convert them into a structured form that a machine can understand and act upon. While humans are able to effortlessly handle mispronunciations, swapped words, contractions, colloquialisms, and other quirks, machines are less adept at handling unpredictable inputs.

A good rule of thumb is to use the term NLU if you’re just talking about a machine’s ability to understand what we say.

To build machines that understand natural language, we must distill speech into a structured ontology using a combination of rules, statistical modeling, or other techniques. Entities must be extracted, identified, and resolved, and semantic meaning must be derived within context, and be used for identifying intents. For example, a simple phrase such as: “I need a flight and hotel in Miami from October 4 to 10” must be parsed and given structure:

need:flight {intent} / need:hotel {intent} / Miami {city} / Oct 4 {date} / Oct 10 {date} / sentiment: 0.5723 (neutral)

Computational linguistics has become a critical area of interest in recent years, as companies work to build systems capable of effortless, unsupervised, and socially acceptable direct interaction with customers. Everyone from small tech startups (like Lola) to the major technology companies like Amazon (Alexa) and Apple (Siri) are investing in efforts to make their systems feel more human. New and exciting things are happening in this field everyday, and I’m excited to be a part of bringing these advances to life.

-Bryan (Director of AI)

Source URL: