lang1

Call Perl and Python from Arc

This is my project to allow libraries in other langauges to be easily called from Arc. This preliminary version supports Perl and Python. I haven’t written much code using this facility yet, so I wouldn’t be surprised if the interface changes when I start using it for real projects and I figure out what I should actually be doing.

I’ll use “Lang” as an abbreviation to mean the other language, that is, Perl or Python.

In this approach:

• Arc and each of the Lang languages runs in a separate process. The Lang process is started up when the first call to a Lang library is made, and shuts down automatically when the Arc process terminates.
• Arc and the Lang process communicate over HTTP. The Lang process runs a dedicated HTTP server accepting requests from Arc, and Arc acts as an HTTP client making POST requests to run Lang code and get a result back.
• Arc generates a Lang program (or expression) to be executed, and is responsible for converting any Arc values into Lang literals. The program is sent over to the Lang process in the POST request, where it is eval’ed.
• The Lang program is responsible for generating an Arc literal for the return value, which is read into Arc using Arc’s standard read function.

See lang-faq for a discussion of this design.

Interface

The Lang program is constructed with print statements, much like how the macros in html.arc generate HTML. The complete program is sent over to the Lang server, which returns the result of the expression converted to an Arc value.

arc> (python (pr "5+6"))
11

arc> (perl (pr "5+6"))
"11"

Notice how Perl doesn’t distinguish between numbers and strings, and so by default the value is converted to an Arc string. We can explicitly ask for an Arc number:

arc> (perl (pr "arcnum(5+6)"))
11

The python macro is used for Python expressions; if you have Python statements such as import you can use pythonst:

arc> (pythonst “
     import textwrap
     import os.path
     ”)
nil

If you’d like to return a value back to Arc from a pythonst you can assign a value to the result variable:

arc> (pythonst “result = 3 + 4”)
7

(Perl is less persnickety, so you can use the perl macro for both Perl statements like use or require and for Perl expressions).

tolang will convert an Arc value into a Lang value for us:

arc> (let xs '(0 10 20 30 40)
       (python (pr (topython xs) "[3]")))
30

As an abbreviation, string literals (which aren’t in an expression) are automatically printed to become part of the program:

arc> (perl "arcnum(5+6)")
11

Using the template syntax, tolang is called for us as well:

arc> (let xs '(0 10 20 30 40)
       (python “«xs»[3]”))
30

Here’s an example which calculates the SHA-224 hash of a string:

arc> (perl “use Digest::SHA qw(sha224_hex)”)
""

arc> (def sha224 (data)
       (perl “sha224_hex(«data»)”))
#<procedure: sha224>

arc> (sha224 "hello")
"ea09ae9cc6768c50fcee903ed054556e5bfc8347907f12598aa24193"

Generating Arc literals in Lang

Lang strings, numbers, lists, and hashes / dictionaries are converted automatically to corresponding Arc values. Hashes are converted to my {...} notation, though it would be easy to have them generate a standard Arc #hash(...) instead.

A number of functions in Lang are available to get specific Arc values.

arcnil returns an Arc nil and arct returns an Arc t.

arcbool(x) returns an Arc t if x is considered a true value in Lang, and nil otherwise. For example, in Arc the number zero is treated as a true value:

arc> (if 0 'yes)
yes

but in Perl 0 is false, so:

arc> (perl "arcbool(0)")
nil

arcstr(x), arcsym, and arcnum produce Arc strings, symbols, and numbers.

By default hashes / dictionaries generate string keys. Sometimes symbol keys are nicer on the Arc side:

arc> (python "arcsymtab({'foo': 3, 'bar': 4})")
{foo 3 bar 4}

Error handling

Errors on the Lang side are caught and passed back to Arc. (The error is reported by passing back an Arc table value with a unique key which won’t be accidentally generated by the Lang code). On the Arc side the error traceback and a copy of the program is printed to stderr, and the error is raised in Arc with Arc’s err function.

In this design having an uncaught error is considered an exceptional condition; if there are some particular errors that you’re expecting then you probably should catch them yourself on the Lang side, and then return some particular value to indicate that the expected error happened.

Concurrency

Each call to perl or python makes a separate HTTP request to the Lang server, so Arc can make as many simultaneous calls as it wants.

An Arc thread making a lang call will pause until the call returns. However you can fire up multiple threads to make simultaneous calls.

On the Lang side, both the Perl and Python implementations use a single-threaded event loop model. Perl uses AnyEvent and Python uses Twisted.

In the event loop model, you don’t wait for I/O to be performed, and then run multiple threads so that one I/O operation doesn’t block another. Instead, you initiate an I/O operation, and arrange to be called back when the operation completes. Code running in the event loop should return fast to avoid blocking the server; if you have some code that will take some time to run you can arrange to run it in a separate thread and register a callback in the event loop when the thread is finished.

Since the event loop is single threaded, the Lang server will by default respond to requests one by one, unless you use the event loop facilities to do I/O or run things in another thread.

Debug printing

When the Lang server forks off, its stderr is left open to Arc’s stderr (which is usually your terminal, unless you’ve done something else to it). Thus you can print things to stderr from Lang to see what’s going on:

perl: print STDERR "foo\n";

python: print >>stderr, "foo"

Security

The Lang server listens only on the loopback interface (127.0.0.1 aka localhost), and so doesn’t accept connections from any other computer. Still, it does allow arbitrary code execution from any process on your computer that connects to the lang server port. This may be OK on computers where you’re the only person running programs. In other environments, or if you want to run the Lang server on a different computer, you’ll want better security.

Control port

The Arc process opens and listens on a port which the Lang server connects to. No data is transferred on this port; the sole purpose is to get the Lang server to shut down when the Arc process terminates. When the Arc process ends, even by a segmentation fault or a hard kill, the operating system will close all its ports. The Lang server notices that it has lost the connection to the control port and exits. This avoids having old Lang server processes lying around.

HTTP implementation

I wanted to be able to pass the input stream of the HTTP response from the Lang server directly to Arc’s read function, so I wrote a very simple HTTP client implementation that does nothing but make POST form requests. For a more general purpose HTTP implementation, see Mark Huetsch’s web.arc in Anarki.

Get this hack

Using the hackinator:

$ hack \
    ycombinator.com/arc/arc3.1.tar \
    awwx.ws/xloop0.arc \
    awwx.ws/ac0.patch \
    awwx.ws/ac1.arc \
    awwx.ws/defarc0.patch \
    awwx.ws/defarc-literal0.patch \
    awwx.ws/arc-write0.patch \
    awwx.ws/extend0.arc \
    awwx.ws/between0.arc \
    awwx.ws/skipwhite0.arc \
    awwx.ws/table-rw2.arc \
    awwx.ws/span0.arc \
    awwx.ws/template4.arc \
    awwx.ws/re2.arc \
    awwx.ws/toperl1.arc \
    awwx.ws/topython1.arc \
    awwx.ws/client-socket0.arc \
    awwx.ws/binary0.arc \
    awwx.ws/parseurl0.arc \
    awwx.ws/readline1.arc \
    awwx.ws/begins-rest0.arc \
    awwx.ws/urlencode0.arc \
    awwx.ws/read-headers0.arc \
    awwx.ws/encodequery0.arc \
    awwx.ws/post1.arc \
    awwx.ws/lock0.arc \
    awwx.ws/lang1.arc

Contact me