In the last article I showed how we can simulate the idea of Lisp’s “closure” in Elixir. Today, I would like to demonstrate how we can call Lisp functions from Elixir using the NIF interface.

What is the need to integrate Elixir with another language? I can think of two reasons:

1) We have a library written in another language and we want to reuse that library in Elixir instead of re-writing the entire functionality in Elixir. 

2) The other language has some nice features for solving a specific problem and so we want to solve that problem in that language and use that from Elixir. For instance, it is easy to implement knowledge-based expert systems using KnowledgeWorks module of LispWorks. So I might implement an expert system in LispWorks Lisp and drive it from Elixir. In the same vein, Prolog might be your choice if you wish to take advantage of the backtracking-based reasoning available in the language. In essence, the strategy is to choose a language that is most suitable for solving (possibly, a subset of) the given problem and use Elixir for what it is good at, say distributed fault-tolerant computing, etc.

Any drawbacks? Yes, of course. The following two points are to be carefully considered before implementing some functionality in NIF:

1) Since NIF operates at the lowest level, it is quite easy to crash the entire VM if something goes wrong! Rigorous testing is required before the code is made available for use.

2) There is considerable overhead in calling a function implemented as NIF (you will see this later when you actually see the implementation). So there must be a justifiable reason to use NIF instead of a direct Elixir implementation.

Having put forth the pros and cons of the approach, let us continue with our discussion on how to use NIF.

The integration with the “foreign” language happens through NIF (Native Implemented Functions). So, if the “foreign” language supports “C” bindings, then it is possible to integrate it with Elixir.

The official Erlang NIF tutorial is here.  And this is another nice article on NIF. 

Today’s focus is on learning how to call functions written in Lisp (specifically, LispWorks Lisp on Windows platform).

The following diagram shows how the integration works. 

Step-1: Build the Lisp DLL

The functionality that we want to use from Lisp must be available as “exported” functions in a DLL.

In an earlier article, I described the procedure to create a DLL of exported functions from LispWorks Lisp. You may find it useful to review that.

Here is the Lisp code that exports two functions:

1) Function that adds two integers

2) Function that takes a string representing Lisp code and evaluates that code. For simplicity, I am assuming that the Lisp code returns an integer and not an arbitrary S-expression.

Step-2: Implement the NIF Layer in C Language

We need an intermediate NIF layer to marshall the parameters and return values between Elixir and Lisp code.

The following is the first part of the NIF layer.

The function “lisp_init_nif” initializes the Lisp environment from the DLL. The second function “lisp_quit_nif” is used to ensure that the Lisp environment is shutdown properly when not required anymore. Later on, we will see how these are used in Elixir.

The two main functions are shown below:

As you can guess, the two functions “lisp_add_nif” and “lisp_eval_nif” provide the required mapping between Elixir and the corresponding Lisp function parameters as well as the return values.

The array “nif_funcs” defined at the end sets up the mapping between all the Elixir and NIF functions.

Now that the functions are defined, we have to create a DLL. Here is how we use the Visual Studio command line tool to generate the DLL:

cl  -I  “G:\Program Files\erl10.3\usr\include”  -LD -MD -Fe: lisp_nif.dll lisp_nif.c 

Remember that this must be run from the Developer command prompt.

Step-3: Load the Native Functions in Elixir

The final step is to define the appropriate Module in Elixir and load the NIF. The hook “@load” helps here. Here is the code for our example:

A brief explanation may be useful here. Our Lisp function “eval” takes a string of S-expressions and returns an integer. The correct way to call this from Elixir is to pass a character list, for example, ‘(+ 10 20)’. But someone might pass a double-quoted string literal instead. To handle this correctly, I have defined the function “exec” with two argument types. One takes a double-quoted string and converts it into a character list before passing it to “eval”. The other passes the argument directly to “eval”. This is just a convenience.

Here is a session in IEX:

First, we compile and load the NIF. The call “Lisp.init” initializes the Lisp environment by properly loading the function entry points. “Lisp.exec” and “Lisp.eval” are then used to evaluate various Lisp expressions. Take note of the complex multi-line “flet” expression and how it is evaluated correctly.

When we do not require the Lisp environment anymore, it is a good idea to call “Lisp.quit”.

The brief interactive session in IEX demonstrates that the integration between Elixir and Lisp works as expected. Hopefully, you understood the steps involved in the process.

For the record, I used LispWorks 7.1.1 64-bit Enterprise Edition for Windows and Visual Studio 2019 Professional (version 16.6.4) for today’s experiment. Elixir version is 1.10.3 for Windows.

Here is the source code used in the example. 

Have a great weekend!

Tags: Elixir, LISP, NIF