{"id":884,"date":"2018-04-08T05:05:45","date_gmt":"2018-04-08T05:05:45","guid":{"rendered":"http:\/\/www.rangakrish.com\/?p=884"},"modified":"2018-04-10T15:44:35","modified_gmt":"2018-04-10T10:14:35","slug":"natural-language-generation","status":"publish","type":"post","link":"https:\/\/www.rangakrish.com\/index.php\/2018\/04\/08\/natural-language-generation\/","title":{"rendered":"Natural Language Generation"},"content":{"rendered":"

I had written a series of posts on my iLangGen<\/strong><\/em><\/a> framework last year. It aims to provide a flexible and expressive approach for building natural language generation systems. In today’s post, I would like to describe a concrete example of how iLangGen<\/strong><\/em> can be used for generating natural language text from structured data, aided by a convenient template text.<\/p>\n

Here is the scenario. There is a pathology laboratory called XYZ Diagnostic & Scan Centre<\/strong><\/em>, which provides high-quality pathology services to the public. For example, it takes blood samples from patients and analyzes them under a controlled environment. Once the processing is complete, it is scrutinized by an in-house specialist, and the results are emailed to respective patients. Instead of merely printing a table of test names and results, XYZ Lab<\/strong><\/em> wants to make this more user-friendly by sending a personalized email that contains paraphrased data.<\/p>\n

Let us see how we can do this. The overall approach is depicted in the following diagram.<\/p>\n

\"Overall<\/a>
Overall Approach<\/strong><\/figcaption><\/figure>\n

The NLG application takes three inputs namely, a Template text file, a Grammar file and patient Database. It interacts with iLangGen<\/strong><\/em> passing the grammar file and additional context information, and subject to the overall structure of the template file, generates complete text for each database record.<\/p>\n

First, here is the actual data (the data is stored as a CSV file for this demo):<\/p>\n

\"Patient<\/a>
Patient Data<\/strong><\/figcaption><\/figure>\n

Since this is just a demo, I have deliberately chosen a limited number of fields and records.\u00a0<\/span><\/p>\n

We use a template text to give us the overall structure of generated text. Here is the text we will use for this example:<\/p>\n

\"Letter<\/a>
Letter Template<\/strong><\/figcaption><\/figure>\n

Notice the sections appearing within the delimiters {{<\/strong><\/em> and }}<\/strong><\/em>. Each such section has the format <GrammarName::Non-TerminalName><\/strong> and is used by the application to guide iLangGen<\/strong><\/em> in generating appropriate text dynamicaly<\/strong><\/em>. The remaining text will be retained as it is. More on this later.<\/p>\n

Here is the grammar part:<\/p>\n

\"The<\/a>
The Grammar<\/strong><\/figcaption><\/figure>\n

There are two grammars: CommonServices<\/strong><\/em> exists to provide greeting messages (could be\u00a0<\/span>enhanced later) and is inherited by PathologyGrammar<\/strong><\/em>. The latter contains the core logic for generating text customized for each patient. The function sequentially-one<\/strong><\/em> on the RHS of the rules returns one value from its list in the same order, each time it is called. When the last element has been fetched, it starts from the first element again. So, for example, the first time the non-terminal checked<\/strong><\/em> is visited, it returns checked<\/strong><\/em>. Next time, it returns reviewed<\/strong><\/em>, and so on. The fifth time it is called, it returns checked<\/strong><\/em> once again and the cycle starts. Thus, this function is useful if we desire guaranteed variation across consecutive generations. The other function randomly-one<\/strong><\/em>, used in CommonServices<\/strong><\/em>, returns one from its list of elements in random order. It is possible that this returns the same element across consecutive calls. We can choose either of the two functions depending on our requirement.<\/p>\n

The function greet<\/strong><\/em> used in CommonServices<\/strong><\/em> emits the string Mr.<\/strong><\/em> or Ms.<\/strong><\/em> depending on the gender of the person denoted by the current data record.<\/p>\n

\"The<\/a>
Greet Function<\/strong><\/figcaption><\/figure>\n

Synthesizing text for haemoglobin value is handled by the function get-haemoglobin-text<\/strong><\/em>.<\/p>\n

\"Haemoglobin<\/a>
Generating Haemoglobin Text<\/strong><\/figcaption><\/figure>\n

Depending on the person’s gender, age, haemoglobin value, and the recommended threshold, this function generates appropriate text.<\/p>\n

In a similar manner, get-uric-acid-text<\/strong><\/em> generates text for uric acid level in the sample.<\/p>\n

\"Uric<\/a>
Generating Uric Acid Text<\/strong><\/figcaption><\/figure>\n

One interesting thing to note is the way these two functions call get-one-output-from-node<\/strong><\/em>, passing a non-terminal node name, in order to obtain another string dynamically from the same grammar.<\/p>\n

For keeping the discussion brief, I have not optimized these two functions, but you can see that there is a lot of similarity in the code structure, and we could exploit that.<\/p>\n

Let us take a look at the application now.<\/p>\n

\"Main<\/a>
The Main Application<\/strong><\/figcaption><\/figure>\n

The three drop-down lists allow us to select the required inputs -Text template, Grammar and the Data file. Once the data file is selected, names of all patients are displayed on the left pane. When a patient is selected, his\/her details are echoed at the bottom status bar, and the generated text is displayed in the main area. As we move from patient to patient, the text is refreshed automatically, and instantaneously!<\/p>\n

\"Next<\/a>
Next Patient<\/strong><\/figcaption><\/figure>\n

 <\/p>\n

\"Another<\/a>
Another Patient<\/strong><\/figcaption><\/figure>\n

As you can see, the generated text varies from one patient to the next.<\/p>\n

It is possible to design multiple templates and grammars for the same set of data. This allows us to achieve a much richer variety for the generated text. Obviously, that is the preferred\u00a0<\/span>solution when we wish to generate text in different languages<\/strong><\/em> for the given data.<\/p>\n

An important practical usability concern arises here. If we build a system like this, who will design the template text and who will implement the grammar? Assuming that our end-user is XYZ Lab<\/strong><\/em>, it is fair to expect that they will prepare the template text because they know the domain and vocabulary. What about the Grammar file? Obviously, we cannot expect them to hire Lisp<\/strong><\/em> programmers, or needlessly outsource that task to another agency! Well, this can be handled by implementing a good front-end as part of the NLG system, which will be used by the team at XYZ Lab<\/strong><\/em> to specify the static text and embed meta representations of synonyms, paraphrased text and formula-based dynamic text schema. And because this is all Lisp, it is quite straightforward to generate the appropriate grammar on the fly!<\/p>\n

One final note before concluding: For demo purposes, I have implemented this system as a stand-alone Windows-based application. In a real world scenario, where performance and scalability are important, one option is to turn this into a hosted REST API service.<\/p>\n

iLangGen<\/strong><\/em> and the application discussed in this post have been implemented in LispWorks<\/strong><\/em><\/a>.<\/p>\n

Hope you enjoyed reading this post!<\/p>\n","protected":false},"excerpt":{"rendered":"

I had written a series of posts on my iLangGen framework last year. It aims to provide a flexible and expressive approach for building natural language generation systems. In today’s post, I would like to describe a concrete example of how iLangGen can be used for generating natural language text from structured data, aided by […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[18,107,17],"tags":[109,139],"class_list":["post-884","post","type-post","status-publish","format-standard","hentry","category-lisp","category-natural-language-processing","category-programming","tag-ilanggen","tag-natural-language-generation"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9OLnF-eg","jetpack-related-posts":[{"id":575,"url":"https:\/\/www.rangakrish.com\/index.php\/2017\/08\/06\/text-generation-using-ilanggen-framework\/","url_meta":{"origin":884,"position":0},"title":"Text Generation Using iLangGen Framework","author":"admin","date":"August 6, 2017","format":false,"excerpt":"The two primary areas in Natural Language processing are Natural Language Understanding and Natural Language Generation. The former is concerned with processing and making sense of natural language text, whereas the latter is concerned with synthesizing text, possibly from some deep representation. Both are fascinating and at the same time,\u2026","rel":"","context":"In "LISP"","block_context":{"text":"LISP","link":"https:\/\/www.rangakrish.com\/index.php\/category\/lisp\/"},"img":{"alt_text":"iLangGen Grammar","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2017\/08\/Blog1.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2017\/08\/Blog1.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2017\/08\/Blog1.png?resize=525%2C300 1.5x"},"classes":[]},{"id":1410,"url":"https:\/\/www.rangakrish.com\/index.php\/2019\/01\/27\/generating-poetry-using-ilanggen\/","url_meta":{"origin":884,"position":1},"title":"Generating Poetry Using iLangGen","author":"admin","date":"January 27, 2019","format":false,"excerpt":"In an earlier article, I wrote about using iLangGen to generate natural language text. iLangGen is a powerful text generation library that I have been working on over the years. Today, I would like to show how we can use that library to generate \"poetry\". Be warned, however, that the\u2026","rel":"","context":"In "LISP"","block_context":{"text":"LISP","link":"https:\/\/www.rangakrish.com\/index.php\/category\/lisp\/"},"img":{"alt_text":"Sample Output 2","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/01\/Output2.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2152,"url":"https:\/\/www.rangakrish.com\/index.php\/2020\/09\/28\/template-based-text-generation\/","url_meta":{"origin":884,"position":2},"title":"Template-Based Text Generation","author":"admin","date":"September 28, 2020","format":false,"excerpt":"I had written earlier about natural language generation\u00a0using my iLangGen framework. I used a \"template\" text file which was instantiated dynamically based on predefined \"grammars\" and external data. The sample application I show-cased demonstrated its utility and versatility. Today I would like to touch upon a few other \"pattern\" elements\u2026","rel":"","context":"In "LISP"","block_context":{"text":"LISP","link":"https:\/\/www.rangakrish.com\/index.php\/category\/lisp\/"},"img":{"alt_text":"Template File","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2020\/09\/Template-300x195.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2020\/09\/Template-300x195.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2020\/09\/Template-300x195.jpg?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":637,"url":"https:\/\/www.rangakrish.com\/index.php\/2017\/09\/27\/composition-of-grammars\/","url_meta":{"origin":884,"position":3},"title":"Composition of Grammars","author":"admin","date":"September 27, 2017","format":false,"excerpt":"In the last post, we saw how iLangGen text generation framework supports reuse of grammars through inheritance, akin to object-oriented languages. The good news is that we can achieve reuse through composition as well. The following is a simple grammar, nothing fancy to elaborate. Here is the output when you\u2026","rel":"","context":"In "LISP"","block_context":{"text":"LISP","link":"https:\/\/www.rangakrish.com\/index.php\/category\/lisp\/"},"img":{"alt_text":"Simple Grammar","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2017\/09\/G1.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":585,"url":"https:\/\/www.rangakrish.com\/index.php\/2017\/08\/18\/repetition-in-grammar-rules\/","url_meta":{"origin":884,"position":4},"title":"Repetition in Grammar Rules","author":"admin","date":"August 18, 2017","format":false,"excerpt":"Many times, we need to repeatedly generate an element, or in general, have a way to control the number times one or more elements get generated. iLangGen supports all the standard cases: - Zero or once - Zero or more times (unbounded) - Zero or more times (bounded) - One\u2026","rel":"","context":"In "LISP"","block_context":{"text":"LISP","link":"https:\/\/www.rangakrish.com\/index.php\/category\/lisp\/"},"img":{"alt_text":"Zero or Once","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2017\/08\/Optional.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2017\/08\/Optional.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2017\/08\/Optional.png?resize=525%2C300 1.5x"},"classes":[]},{"id":1309,"url":"https:\/\/www.rangakrish.com\/index.php\/2018\/12\/16\/generating-poetry-using-rita\/","url_meta":{"origin":884,"position":5},"title":"Generating Poetry Using RiTa","author":"admin","date":"December 16, 2018","format":false,"excerpt":"A few days ago, I came across a nice library called RiTa, which is described as a software toolkit for computational literature. Its two major features are text analysis and text generation.\u00a0 The text analysis module parses given text to extract sentences, tokens, POS, stresses, and phonemes. There is also\u2026","rel":"","context":"In "Natural Language Processing"","block_context":{"text":"Natural Language Processing","link":"https:\/\/www.rangakrish.com\/index.php\/category\/natural-language-processing\/"},"img":{"alt_text":"Terminal Session","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2018\/12\/Terminal.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/posts\/884","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/comments?post=884"}],"version-history":[{"count":0,"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/posts\/884\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/media?parent=884"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/categories?post=884"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/tags?post=884"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}