{"id":1650,"date":"2019-07-21T20:51:07","date_gmt":"2019-07-21T15:21:07","guid":{"rendered":"https:\/\/www.rangakrish.com\/?p=1650"},"modified":"2019-07-21T20:51:07","modified_gmt":"2019-07-21T15:21:07","slug":"custom-text-analysis-using-textrazors-prolog-engine","status":"publish","type":"post","link":"https:\/\/www.rangakrish.com\/index.php\/2019\/07\/21\/custom-text-analysis-using-textrazors-prolog-engine\/","title":{"rendered":"Custom Text Analysis Using TextRazor\u2019s Prolog Engine"},"content":{"rendered":"

This is the third part in the series on information extraction<\/strong><\/em> from unstructured text. In the first part<\/strong><\/em><\/a>, we saw how MeaningCloud<\/strong><\/em><\/a>\u00a0allows us to specify complex rules to identify custom categories through their Deep Categorization<\/strong><\/em> API. The second part<\/strong><\/em><\/a>\u00a0covered spaCy\u2019s<\/strong><\/em><\/a>\u00a0pattern matcher.<\/p>\n

Today, I would like to discuss how we can use TextRazor\u2019s<\/strong><\/em><\/a>\u00a0\u201cProlog Engine\u201d<\/strong><\/em> to perform customized text analysis.\u00a0<\/span><\/p>\n

TextRazor<\/strong><\/em> provides a powerful API for dealing with natural language text. It supports Classification<\/strong><\/em>, Topic Tagging<\/strong><\/em>, Entity Recognition<\/strong><\/em>, Dependency Parsing<\/strong><\/em>, and more. What makes TextRazor<\/strong><\/em> somewhat unique is the ability to seamlessly attach custom rules<\/strong><\/em><\/a>\u00a0written in Prolog<\/strong><\/em> in order to discover special patterns in the text. These rules can make use of the rich set of built-in predicates that expose the core logic of the NLP engine.<\/p>\n

The prolog engine in TextRazor<\/strong><\/em> is based on the popular SWI Prolog<\/strong><\/em><\/a>\u00a0implementation (TextRazor\u2019s<\/strong><\/em> documentation incorrectly mentions this as YAP<\/strong><\/em> prolog).<\/p>\n

Since Prolog<\/strong><\/em> is a full-fledged programming language (albeit quite different from the \u201cconventional\u201d<\/strong><\/em> languages), the ability to write custom logic in Prolog<\/strong><\/em> implies that we can encode arbitrarily complex logic in order to discover interesting patterns in text.<\/p>\n

I have written a REST<\/strong><\/em> client in Lisp<\/strong><\/em> that shows how to use Prolog<\/strong><\/em> rules to extract key pieces of information about a patient coming to the doctor for consultation. For practical reasons, the program and the case description are deliberately kept simple in order to focus on the idea.<\/p>\n

Let us consider the following case text:<\/p>\n

Mr.John Haggard, aged 30 years, has cold and cough. He sneezes when exposed to <\/span>
\ncold weather. There is headache on waking up. He has runny nose as well.<\/span><\/p><\/blockquote>\n

Given this text, we want to extract the Name<\/strong><\/em>, Gender<\/strong><\/em> and Age<\/strong><\/em> of the patient. In addition, (as in the previous two articles) we want to know if the patient is suffering from \u201ccommon-cold\u201d<\/strong><\/em> or \u201cdiarrhea\u201d.<\/strong><\/em> Here is the set of Prolog<\/strong><\/em> rules that will do the job:<\/p>\n

common_cold :- sequence(‘runny’, ‘nose’).<\/span>
\ncommon_cold :- lemma(S), member(S, [‘headache’, ‘cold’, ‘cough’, ‘sneeze’]).<\/span>
\ndiarrhea :- lemma(‘loose’), or( lemma(‘motion’), lemma(‘stools’)).<\/span>
\ndiarrhea :- lemma(‘diarrhoea’); lemma(‘diarrhea’).<\/span>
\nname(N) :- sequence(lemma(X), token(TID1, N1), token(TID2, N2)), title(X),<\/span>
\npart_of_speech(TID1, ‘NNP’), part_of_speech(TID2, ‘NNP’), N = [N1, N2],!.<\/span>
\nname(N) :- sequence(lemma(X), token(TID, N)), title(X), part_of_speech(TID, ‘NNP’).<\/span>
\ntitle(X) :- member(X, [‘mr.’, ‘mrs.’, ‘miss.’, ‘ms.’]).<\/span>
\ngender(G) :- lemma(X), member(X, [‘he’, ‘his’, ‘him’]), G = ‘male’.<\/span>
\ngender(G) :- lemma(X), member(X, [‘she’, ‘her’]), G = ‘female’.<\/span>
\nis_number(Str) :- atom_string(Atom, Str), atom_number(Atom, _).<\/span>
\nage(X) :- sequence(token(X), ‘years’, ‘old’), is_number(X).<\/span>
\nage(X) :- sequence(‘aged’, token(X), ‘years’), is_number(X).<\/span>
\nage(X) :- sequence(token(X), ‘years’, ‘of’, ‘age’), is_number(X).<\/span><\/p><\/blockquote>\n

The predicate common-cold<\/strong><\/em> will be True<\/strong><\/em> if the text contains the sequence of words \u201crunny\u201d<\/strong> and \u201cnose\u201d<\/strong> (without any other intervening word), or if the text contains the root form (\u201clemma\u201d<\/strong><\/em>) of any of the words \u201cheadache\u201d<\/strong>, \u201ccold\u201d<\/strong>, \u201ccough\u201d<\/strong>, and \u201csneeze\u201d<\/strong>.<\/p>\n

Likewise, we can infer that the patient has diarrhea<\/strong><\/em>\u00a0if the word \u201cdiarrhea\u201d<\/strong> (or \u201cdiarrhoea\u201d<\/strong>) occurs directly in the text, or we have either \u201cloose motion\u201d<\/strong> or \u201cloose stools\u201d<\/strong>.\u00a0<\/span><\/p>\n

To determine the Name<\/strong><\/em> of the patient, we first look for the Title<\/strong><\/em>\u00a0(i.e., how we address the person) – \u201cMr.\u201d, \u201cMrs.\u201d, \u201cMs\u201d, <\/strong>or \u201cMiss.\u201d,<\/strong> followed by one or two Proper Nouns<\/strong><\/em>. Although this itself can give a clue to the gender of the person, I am defining a separate predicate for this. The gender is male<\/strong><\/em> if any of the words “he”, “him”, <\/strong>or “his”<\/strong> occurs in the text. Similarly, the gender is female<\/strong><\/em> if either “she”<\/strong> or “her”<\/strong> is present in the text.<\/p>\n

The age<\/strong><\/em> of the patient is also determined from the context. It is a number conforming to any of the following patterns:<\/p>\n