Coreference Resolution in Stanford CoreNLP

Written by on February 10, 2019 in Natural Language Processing, Programming with 0 Comments

In the last article, I showed how we can use the neuralcoref library along with spaCy to do coreference resolution (examples involved anaphoric references). In today’s article, I want to try the same (well, almost) examples in Stanford CoreNLP engine and see how they compare.

Since CoreNLP is a Java implementation, I chose to write the test program in Java. Here is the program  along with the examples.

CoreNLP Corereference Example

CoreNLP Corereference Example

You can download the CoreNLP library from here.

Let us start with the first sentence:

“My sister has a dog and she loves him.”

When I run it through the program, this is the output:

Example-1 Output

Example-1 Output

Strange. It shows that the program has identified “My sister” and “she” as pointing to the same entity, but there is no mention of “dog” and “him”. In contrast, neuralcoref identified both the “mentions” correctly.

Could it be because this is a compound sentence? Let us convert this into two simple sentences:

“My sister has a dog. She loves him.”

Here is the output in this case:

Example-2 Output

Example-2 Output

OK! It is interesting that the program has identified both the “mentions” this time.

I am using the remaining sentences as they were in neuralcoref example. Here is the third sentence:

“My sister has a dog and she loves him. He is cute.”

The output is:

Example-3 Output

Example-3 Output

Although the system has identified “him” and “he” as belonging together, it has omitted “dog” from the reference. Bug! Here also, neuralcoref fared better.

The next example is:

“My sister has a dog and she loves her.”

The output is:

Example-4 Output

Example-4 Output

The behavior is the same as with neuralcoref. “dog” is not paired with “her”.

Consider the following statement that uses different genders and hence should be easier to resolve:

“My brother has a dog and he loves her.”

This is what we get:

Example-5 Output

Example-5 Output

That is nice! “dog” and “her” are correctly paired. neuralcoref did not handle this case correctly.

Let us look at the next sentence:

“Mary and Julie are sisters. They love chocolates.”

neuralcoref handled this case correctly. Strangely, when given this input, CoreNLP does not identify any “mentions” at all! I confess I am disappointed.

Let us try the next one:

“John and Mary are neighbours. She admires him because he works hard.”

This looks more complex than the earlier one. How does CoreNLP fare?

Example-7 Output

Example-7 Output

OK, this one is handled correctly. That is a relief.

The next one is tougher:

“X and Y are neighbours. She admires him because he works hard.”

Here is the program’s output:

Example-8 Output

Example-8 Output

Not surprised with the behavior. neuralcoref wasn’t any better.

Here is the last example:

“The dog chased the cat. But it escaped.”

This is the corresponding output:

Example-9 Output

Example-9 Output

Not correct. Again, the behavior is identical to neuralcoref.

Let me summarize and show the performance of CoreNLP and neuralcoref with respect to the examples:

Comparison Table

Comparison Table

You can see that among the sentences tested in both the systems, neuralcoref got 4 out of 8 correct, whereas CoreNLP got just 2 out of 8 correct. To clarify, I have marked the output as “Wrong” if the system did not FULLY identify the mentions.

Overall, in the case of anaphora resolution, both the popular libraries have fared poorly in my opinion. I expected more out of CoreNLP, so that was a bigger disappointment!

Looks like there is a lot more work to do in the area of coreference resolution!

You can download my Java program from here.

Have a nice weekend!

Tags: , , ,

Subscribe

If you enjoyed this article, subscribe now to receive more just like it.

Subscribe via RSS Feed

Leave a Reply

Your email address will not be published.

Top