In an earlier article, I explained how to use OpenAI from Wolfram Mathematica ver 14.1. This latest release of Wolfram supports Semantic Search as well. In today’s article, let me discuss this feature.
As in the case of using LLMs, using Semantic Search requires an account with one of the supported vendors. I will continue to use OpenAI for this functionality too.
For this example, I am going to create three short text fragments, each describing a homeopathy remedy briefly. The remedies are Lycopodium, Argentum Nitricum and Natrum Muriaticum.
Remedy Description
I am going to build a semantic search database for the above text fragments.
Creating a Semantic Search Index
The above function returns an index object after storing two texts. I have given a convenient label, in this case the remedy name, for the texts. In addition, since the data size is quite small, I am creating an in-memory database instead of using a persistent file object. Finally, I am specifying the minimum and maximum token limits for each chunk. The reason I am using small values for these is because my text fragments are small.
The function returns an Index object that has several properties:
Index Properties
The “DocumentCount” property tells us how many documents have been stored in the database. The “ItemCount” property tells us the number of chunks created for our text. The “Label” property returns a list of the labels for each chunk in the database.
OK, how do we search for some text in this database? Let us ask the system how many remedies have “sweet craving”? And which remedy has “aggravation from sweets”?
Searching using the Index
The answers are correct! Both Lycopodium and Argentum Nitricum have desire for sweets. And Argentum Nitricum patients typically have some aggravation from sweets.
What if we want to add some more text to this database and create an updated index? That is easy:
Adding More Text
We use the function “UpdateSemanticSearchIndex” for this. To use this, I am passing the third text fragment and specify the same chunking limits as earlier.
You can now see that the number documents is 3 and 5 more chunks have been created for Natrum Muriaticum.
Let us do another search with the new remedy added. We know that Natrum Muriaticum patients have an affinity for salt. Let us check this:
Searching Updated Index
The search function correctly returns the remedy label. That is good!
The above three functions in the latest release of Wolfram are quite useful for doing semantic search on text. It is easy to extend the example to work with larger documents and persisting the database in the local file system, or in the cloud.
Have a nice weekend!
Recent Comments