{"id":1368,"date":"2019-01-08T09:19:10","date_gmt":"2019-01-08T03:49:10","guid":{"rendered":"https:\/\/www.rangakrish.com\/?p=1368"},"modified":"2019-01-08T10:51:48","modified_gmt":"2019-01-08T05:21:48","slug":"parsing-text-with-apache-opennlp","status":"publish","type":"post","link":"https:\/\/www.rangakrish.com\/index.php\/2019\/01\/08\/parsing-text-with-apache-opennlp\/","title":{"rendered":"Parsing Text with Apache OpenNLP"},"content":{"rendered":"

In my earlier posts I have written about parsing text using spaCy<\/strong><\/em><\/a>\u00a0and MeaningCloud’s parsing API<\/strong><\/em><\/a>. For today’s article, I decided to take a look at OpenNLP<\/strong><\/em><\/a>, an open-source ML-based Java toolkit for parsing natural language text.<\/p>\n

OpenNLP<\/strong><\/em> is a fairly mature library and has been around since 2004 (source: Wikipedia<\/strong><\/em><\/a>). It is actively maintained and developed, the current version being 1.9.1. It supports all the standard tasks expected of such a toolkit, namely, language detection, document categorization, lemmatization, tokenization, part-of-speech tagging, chunking, parsing, named-entity recognition, <\/strong><\/em>and coreference resolution<\/strong><\/em>.<\/p>\n

In this article, my focus is on the parser alone. I will try to write about the other components in future articles.\u00a0<\/span><\/p>\n

The parser is a constituency parser<\/strong><\/em> and not dependency parser. It is thus similar to MeaningCloud<\/strong><\/em> but different from spaCy<\/strong><\/em>.\u00a0<\/span><\/p>\n

Getting started is quite simple. I downloaded the binaries from here.<\/strong><\/em><\/a><\/p>\n

I also downloaded two models<\/strong><\/em>: en-parser-chunking.bin<\/strong><\/em> and en-sent.bin<\/strong><\/em> from here.<\/a><\/p>\n

The former is for the parser (chunking parser) and the latter is for sentence detection.<\/p>\n

The parser parses one sentence at a time and hence we need the sentence detector if we are going to parse a text made up of multiple sentences.<\/p>\n

I then configured my Libraries<\/strong><\/em> setting in IntelliJ IDEA<\/strong><\/em><\/a>\u00a0to point to the downloaded lib<\/strong><\/em> directory. That’s pretty much it.<\/p>\n

One interesting aspect of the parser is that you can specify the number of parses to return after parsing. Since the parser is ML-based, the system can typically arrive at multiple potential<\/strong><\/em> parse trees for the same sentence, each tree being associated with a probability factor (the official documentation defines this as the “<\/strong>log of the product of the probability associated with all the decisions which formed this constituent”<\/strong><\/em>.)\u00a0<\/span><\/p>\n

I wrote a simple wrapper class for testing the parser functionality. You can download the source from here<\/strong><\/em><\/a>.<\/p>\n

To start with, let us print the parse tree for the sentence \u201cJohn loves Mary.\u201d<\/strong><\/em> asking for just one parse tree.<\/p>\n

 <\/p>\n

\"Single<\/a>
Single Parse Tree<\/strong><\/figcaption><\/figure>\n

Here is the output:<\/p>\n

 <\/p>\n

\"Output\"<\/a>
Output<\/strong><\/figcaption><\/figure>\n

When rendered as a tree, it looks like this (I used this site<\/strong><\/em><\/a>\u00a0<\/span>to render the s-expression):<\/span><\/p>\n

\"Parse<\/a>
Parse Tree<\/strong><\/figcaption><\/figure>\n

This looks fine. Note the probability number. I assume this shows that what we have got is the best parse probable.<\/p>\n

Next, let us ask for 10 possible parses for the same sentence:<\/p>\n

\"Asking<\/a>
Asking for 10 Parses<\/strong><\/figcaption><\/figure>\n

This is the output:<\/p>\n

\"Output\"<\/a>
Output<\/strong><\/figcaption><\/figure>\n

I have printed the parse trees in descending<\/strong><\/em> order of the probability, so the top one is the best parse in this batch. Notice that the top parse is not the same as the one we got earlier (when we asked for just one parse tree) and has lower probability number than the earlier one. It is not clear why this is the case.<\/p>\n

In order not to clutter the screen, I am showing below the visual tree representation for the top two parses:<\/p>\n

\"Tree1\"<\/a>
First Tree<\/strong><\/figcaption><\/figure>\n
\"The<\/a>
The Second Tree<\/strong><\/figcaption><\/figure>\n

They do not look correct.<\/p>\n

Just to understand what is going on, let us ask for 25 parses this time:<\/p>\n

\"Asking<\/a>
Asking for 25 Parses<\/strong><\/figcaption><\/figure>\n

Here is what we get this time:<\/p>\n

\"Output<\/a>
Output (25 Parses)<\/strong><\/figcaption><\/figure>\n

This time, it is interesting that the top parse tree corresponds to the one we got when we asked for the single parse case. The probability number also matches.\u00a0<\/span><\/p>\n

As a final confirmation, take a look at the visual representation of the top parse:<\/p>\n

\"Parse<\/a>
Parse Tree<\/strong><\/figcaption><\/figure>\n

The trees match, right? It appears to me that we are better off asking for just one parse tree unless we are doing some study to compare the various outputs.\u00a0<\/span><\/p>\n

The result of parsing a sentence is a Parse<\/strong><\/em> array comprising as many parses as requested while parsing. The Parse<\/strong><\/em> class has several methods to access and manipulate the components of the parse structure, including accessing the child elements and so on. See the documentation<\/strong><\/em><\/a> for more details.<\/p>\n

You can play around with my example code<\/strong><\/em><\/a>\u00a0to get a feel for the parser functionality. \u00a0<\/span><\/p>\n

Have a great day and see you soon!<\/p>\n","protected":false},"excerpt":{"rendered":"

In my earlier posts I have written about parsing text using spaCy\u00a0and MeaningCloud’s parsing API. For today’s article, I decided to take a look at OpenNLP, an open-source ML-based Java toolkit for parsing natural language text. OpenNLP is a fairly mature library and has been around since 2004 (source: Wikipedia). It is actively maintained and […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[107,17],"tags":[180,174,179],"class_list":["post-1368","post","type-post","status-publish","format-standard","hentry","category-natural-language-processing","category-programming","tag-constituency-parsing","tag-natural-language-processing","tag-opennlp"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p9OLnF-m4","jetpack-related-posts":[{"id":1399,"url":"https:\/\/www.rangakrish.com\/index.php\/2019\/01\/20\/named-entity-recognition-ner-with-opennlp\/","url_meta":{"origin":1368,"position":0},"title":"Named Entity Recognition (NER) with OpenNLP","author":"admin","date":"January 20, 2019","format":false,"excerpt":"In the earlier two articles, we looked at Sentence Parsing\u00a0and Chunking\u00a0as supported in OpenNLP. In today's article, let us explore Named Entity Recognition, also known as NER. NER is a technique to identify special categories of noun phrases such as people, places, companies, money, etc., present in the given text.\u2026","rel":"","context":"In "Natural Language Processing"","block_context":{"text":"Natural Language Processing","link":"https:\/\/www.rangakrish.com\/index.php\/category\/natural-language-processing\/"},"img":{"alt_text":"Wrapper Classes","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/01\/Wrappers.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1386,"url":"https:\/\/www.rangakrish.com\/index.php\/2019\/01\/13\/chunking-in-opennlp\/","url_meta":{"origin":1368,"position":1},"title":"Chunking in OpenNLP","author":"admin","date":"January 13, 2019","format":false,"excerpt":"In my previous post, I showed how to parse sentences using OpenNLP. Another useful feature supported by OpenNLP is \"chunking\u201d. That is the subject of today\u2019s article. Chunking stands between part-of-speech tagging and full parse in terms of the information it captures. POS tagging assigns part of speech to individual\u2026","rel":"","context":"In "Natural Language Processing"","block_context":{"text":"Natural Language Processing","link":"https:\/\/www.rangakrish.com\/index.php\/category\/natural-language-processing\/"},"img":{"alt_text":"Printing Chunked Tags with Probability","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/01\/Example3-1.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/01\/Example3-1.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/01\/Example3-1.png?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":1792,"url":"https:\/\/www.rangakrish.com\/index.php\/2019\/11\/23\/using-augmented-transition-networks-atn-for-information-extraction\/","url_meta":{"origin":1368,"position":2},"title":"Using Augmented Transition Networks (ATN) for Information Extraction","author":"admin","date":"November 23, 2019","format":false,"excerpt":"After Wood\u2019s paper [1], Augmented Transition Networks\u00a0(ATN) became popular in the 1970s, for parsing text. An ATN is a generalized transition network with two major enhancements: Support for recursive transitions, including jumping to other ATNs Performing arbitrary actions when edges are traversed Remembering state through the use of registers See\u2026","rel":"","context":"In "LISP"","block_context":{"text":"LISP","link":"https:\/\/www.rangakrish.com\/index.php\/category\/lisp\/"},"img":{"alt_text":"ATN for Modality","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/11\/modality.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/11\/modality.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/11\/modality.jpg?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":1285,"url":"https:\/\/www.rangakrish.com\/index.php\/2018\/12\/09\/parsing-text-with-meaningclouds-text-analytics-api\/","url_meta":{"origin":1368,"position":3},"title":"Parsing Text with MeaningCloud’s Text Analytics API","author":"admin","date":"December 9, 2018","format":false,"excerpt":"There is wide-spread interest in Natural Language Processing (NLP) today, and there are several API services available to cater to this demand. See this article for a fairly detailed list of services. All of them support multiple languages, including English. Today, I am going to share my experience in working\u2026","rel":"","context":"In "LISP"","block_context":{"text":"LISP","link":"https:\/\/www.rangakrish.com\/index.php\/category\/lisp\/"},"img":{"alt_text":"Get Words Function","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2018\/12\/Get-words.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2018\/12\/Get-words.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2018\/12\/Get-words.png?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":1817,"url":"https:\/\/www.rangakrish.com\/index.php\/2019\/12\/08\/using-definite-clause-grammars-dcg-for-information-extraction\/","url_meta":{"origin":1368,"position":4},"title":"Using Definite Clause Grammars (DCG) for Information Extraction","author":"admin","date":"December 8, 2019","format":false,"excerpt":"In the previous article, I showed how we can use ATNs for extracting key information from natural language text. I also pointed out in that article that Definite Clause Grammars (DCG) are a more compact formalism for doing this. That will be the focus of today's article. For a nice\u2026","rel":"","context":"In "Natural Language Processing"","block_context":{"text":"Natural Language Processing","link":"https:\/\/www.rangakrish.com\/index.php\/category\/natural-language-processing\/"},"img":{"alt_text":"Processing the Text","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/12\/Processing-file-code.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/12\/Processing-file-code.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2019\/12\/Processing-file-code.jpg?resize=525%2C300&ssl=1 1.5x"},"classes":[]},{"id":2366,"url":"https:\/\/www.rangakrish.com\/index.php\/2021\/03\/28\/implementing-ilexicon-using-litedb\/","url_meta":{"origin":1368,"position":5},"title":"Implementing iLexicon using LiteDB","author":"admin","date":"March 28, 2021","format":false,"excerpt":"iLexicon is an \"intelligent\" dictionary that can be used to build Natural Language applications. I have two implementations, one in Lisp and another in Prolog. Both implementations are memory-based, in order to speed up performance. I have written several articles referencing it, for example see this. \u00a0 LiteDB is a\u2026","rel":"","context":"In "Natural Language Processing"","block_context":{"text":"Natural Language Processing","link":"https:\/\/www.rangakrish.com\/index.php\/category\/natural-language-processing\/"},"img":{"alt_text":"Sample Commands","src":"https:\/\/i0.wp.com\/www.rangakrish.com\/wp-content\/uploads\/2021\/03\/Session1.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/posts\/1368","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/comments?post=1368"}],"version-history":[{"count":0,"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/posts\/1368\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/media?parent=1368"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/categories?post=1368"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rangakrish.com\/index.php\/wp-json\/wp\/v2\/tags?post=1368"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}