perlserve

The Perl Server: Calling Perl from Arc

Comparisons between Arc and other popular languages usually go something like this:

And this will always be true for the currently most powerful language. Even if someday an Arc_n becomes popular, there will be some more powerful language that most people don’t use because they don’t understand its power. A popular language is always going to be a Blub language.

I don’t look down on Blub languages. Millions of people today are programming because they can easily learn a conceptually simple language. They bring their own knowledge and expertise, perhaps not extensively as programmers but in other domains, which they then embody in programs and libraries that they write.I find these programs and libraries valuable, even if I myself would have found it painful to write them in Blub.

But I don’t like the choice of “either Arc or Foo”. I want to program in Arc, and also easily make use of useful libraries written in other languages.

I want program in Arc and call any library in any language that might be useful.

Interface

I’m starting with calling Perl and so in the following I’ll be talking about calling Perl from Arc.

The vision I have is that the Arc side is responsible for generating a Perl program, and Perl is responsible for generating an Arc expression as its return value. (I haven’t implemented an Arc generator in Perl yet, so for that part I’m currently using JSON instead).

arc> (perl "2+2")
4

Using a template, I can easily insert values into the Perl program:

arc> (let x 3
       (perl “«x» + 2”))
5

The template is just generating a string which becomes the Perl program, converting «...» values to Perl literals. The above example is the same as:

arc> (let x 3
       (perl (string (toperl x) " + 2")))
5

Here’s an example which calculates the SHA-224 hash of a string:

arc> (perl “use Digest::SHA qw(sha224_hex)”)
nil
arc> (def sha224 (data)
       (perl “sha224_hex(«data»)”))
#<procedure: sha224>
arc> (sha224 "hello")
"ea09ae9cc6768c50fcee903ed054556e5bfc8347907f12598aa24193"

Why not RPC?

Defining an Arc function that calls a Perl function is quite common. So common that we’d probably eventually have a “perldef” macro of some kind, so that the above example would look something like:

(perl “use Digest::SHA qw(sha224_hex)”)
(perldef sha224 sha224_hex)
(sha224 "hello")

We could take this a step further and make our entire interface be remotely calling procedures in the other language (this is called “RPC”, for “remote procedure call”). Then, the thinking goes, we could easily plug in any language that implemented the RPC interface.

The problem with this idea is that we’re then constrained to make every interaction fit into the RPC interface.

For example, if we have a list of a thousand items and we want to call a Perl function on each one, it’s going to be a lot faster to pass the list to Perl and loop through the list inside of Perl, instead of call Perl a thousand times, once for each item in the list.

Having Perl do the loop is easy (well, it was easy to write. Perl programs have an unfortunate tendency end up looking like cursing in cartoons $#@^&!, so it may not be all that easy to read :-)

arc> (let mylist '("a" "b" "c")
       (perl “[map(sha224_hex($_), @{«mylist»})]”))
("abd37534c7d9a2efb9465de931cd7055ffdb8879563ae98078d6d6d5"
 "c681e18b81edaf2b66dd22376734dba5992e362bc3f91ab225854c17"
 "1053463d383dc1e87f06fff34b3c6a2d340d91e184d46d70144ffa5a")

But, if everything has to be a remote procedure call, then we can’t just make up Perl programs as we want. Now we have to write some Perl module, which exports a function that takes a list, get that loaded on the Perl side, and then we’re allowed to call it from Arc.

Which may be fine for implementing known solutions to known problems. But it’s lousy for exploratory programming.

In the Black Box disease, andreyf describes abstractions as “opaque” or “transparent”. RPC is an opaque abstraction, all the details are hidden from you so that you can’t write anything that isn’t going to work with any of the languages that support the RPC interface. My little vaporware perldef macro is a transparent abstraction: you can use when it’s useful, but you don’t have to.

Protocol

How to connect Arc with the other language?

Timothy Fitz argues in Why HTTP? that, by default, we should use choose HTTP for our interface protocol: it has servers in any language, clients in any language, proxies, load balancers, debugging tools, web browsers, people knowledge, guaranteed web access, extensive hardware, known scalability paths, extensibility, commonality of URLs, and security.

There are, of course, tradeoffs. Running the other language in a separate process and talking to it over HTTP will, of course, be slower than running both languages in the same process using a common runtime.

But there are disadvantages to squishing two languages into the same runtime as well. With one language or the other, now I’m not using the runtime for that language that most everyone else is using. So that version will always be behind the main version in features and libraries ported to it.

And, if sometime I want to spin up a hundred server instances to have Perl do some seriously heavy work, on the Arc side I just change the URL it connects to.

So in this interface I pass a Perl program to the Perl server and it gets executed. Eventually what I want is for the Perl side to generate an Arc literal which Arc can read, but I haven’t done that yet and so for now the return value is converted to JSON and Arc uses a JSON parser.

Since the interface is standard HTTP, for testing I can access the Perl server from command line tools

$ curl --form code=2+2 http://localhost:9801/
4

or from a browser

<form method="post" action="http://localhost:9801/">
  Perl:
  <input name="code">
  <input type="submit" value="Run">
</form>

Security

In this implementation the Perl server listens only on the loopback interface (127.0.0.1 aka localhost), and so doesn’t accept connections from any other computer. Still, it does allow arbitrary code execution from any process on your computer that connects to port 9801. This may be OK on computers where you’re the only person running programs. In other environments, or if you want to run the Perl server on a different computer, you’ll want better security.

Control port

The Arc process opens and listens on a port which the Perl server connects to. No data is transferred on this port; the sole purpose is to get the Perl server to shut down when the Arc process terminates. When the Arc process ends, even by a segmentation fault or a hard kill, the operating system will close all its ports. The Perl server notices that it has lost the connection to the control port and exits. This avoids having old Perl server processes lying around.

HTTP implementation

I wanted to be able to pass the input stream of the HTTP response from the Perl server to Akkartik’s port of the Scheme JSON parser, so I wrote a very simple HTTP client implementation that does nothing but make POST form requests. For a more general purpose HTTP implementation see Mark Huetsch’s web.arc in Anarki.

Get this hack

Using the hackinator:

$ hack \
    ycombinator.com/arc/arc3.1.tar \
    awwx.ws/xloop0.arc \
    awwx.ws/ac0.patch \
    awwx.ws/ac1.arc \
    awwx.ws/defarc0.patch \
    awwx.ws/defarc-literal0.patch \
    awwx.ws/extend0.arc \
    awwx.ws/between0.arc \
    awwx.ws/span0.arc \
    awwx.ws/template4.arc \
    awwx.ws/re1.arc \
    awwx.ws/toperl1.arc \
    awwx.ws/client-socket0.arc \
    awwx.ws/binary0.arc \
    awwx.ws/parseurl0.arc \
    awwx.ws/readline1.arc \
    awwx.ws/begins-rest0.arc \
    awwx.ws/urlencode0.arc \
    awwx.ws/scheme-json0.hack \
    awwx.ws/read-headers0.arc \
    awwx.ws/encodequery0.arc \
    awwx.ws/post0.arc \
    awwx.ws/perlserve1.arc

Contact me