Saturday, November 15, 2008

Language Vulnerabilities

Cruel Fate has lately had me doing a great deal of work in ColdFusion. Comparing this with my UCW/Clsql experience has really driven home to me why there are so many html and sql injection vulnerabilities out there on the web. ColdFusion strongly encourages you to just spit whatever was passed to you back out to the user, and practically punishes you for safely constructing sql strings. In UCW and Clsql, on the other hand, you had to actively try in order to open yourself up to an injection attack on either front.

This got me thinking about how other languages tend to lend themselves to certain vulnerabilties. We're all aware of buffer overflow exploits; these are due almost entirely to the fact that C/C++ force you to waste time managing memory. Such things are virtually non-existent in languages like Lisp or even Java.

So what kinds of vulnerabilities does Lisp usage tend to open you up to? The only thing I can think of are the potential hazards involved in using read. As we had to keep in mind when developing Paragent, you absolutely must set *read-eval* to nil wherever you use read. Otherwise, you are at great risk from the #. macro, which will essentially allow an attacker to (easily) run any code they want on your machine. Happily, it's easy enough to avoid once you're aware of it.

Beyond that, I'm at a loss. Can anybody else think of some security issues that are particular to Common Lisp?

Friday, September 12, 2008

Making an Executable

Ostia Antica, harbor city of ancient Rome Not infrequently, some poor lisp newb will ask how to make an executable in lisp. The answer of course, is "what do you mean by 'executable?'" If the newb is lucky, somebody will eventually take break from pretending not to know exactly what he means and point him to save-lisp-and-die. But, they will hasten to add, making an executable is really a silly thing to do, so why bother?

Fresh from the mind of Sergey Kolos, Cusp contributor extraordinaire and the man who recently brought us integrated unit testing comes this wild innovation: a menu item called "Create Exe." That's right. In the next version, all Cusp users will have to do is right-click on their project and select "Create Exe." That, my friends, is what user-friendliness looks like.

So buck up, newbs. With luck, the endless discussions on what the meaning of the word "is" is will soon be replaced by the simple advice to "use Cusp."

I wouldn't hold my breath, though.

Monday, August 25, 2008

Making a small Common Lisp Project (in Cusp)

In honor of my recent addition to Planet Lisp, and as a special thank you to Xach, I figured I'd counter-blog his old post on how to make a lisp project (in Slime). You can go read that if you want. Here's the Cusp way:

Make the project. Go to New > Lisp Project. Enter the name of your project. You're done.

Write the code. You'll notice that a file named "main.lisp" has been opened for you. Put your lisp code there. You can compile the definitions as you write them (with Alt+C), but that's mostly a vestigial holdover from Slime. Cusp is smart. Save your file and it will automatically compile any new or changed top-level forms. (It'll even undefine functions that you've deleted.)

Later on... If you close Cusp, when you come back you'll need to reload the project. To do this, right-click on the .asd file in the project, and choose "Load Project." You may also need to change the package in the REPL to match the one for this project.



So what is all this stuff?


You've got all you need to start hacking, but you probably want to know what's going on behind the scenes. The New Project wizard creates a project folder with three files in it: main.lisp, defpackage.lisp, and [project name].asd. Here's what they do:

main.lisp is the main code file. Put the guts of your program here. You can, of course, create additional files as your project grows. More on that in a moment.

defpackage.lisp defines the package for this lisp project, which is basically the lisp equivalent of a namespace. You could pollute cl-user with your project internals, but that's just asking for trouble down the line. You'll note that main.lisp started out with (in-package :[project-name]). You'll want to make sure any new files you add start that way, too.

If you want to import symbols from another package into your project (so you can say split rather than cl-ppcre:split) put the name of that package into the :use section. This is similar to using in C++ or C#, or import in Java. If your project turns into a library to be used by other projects, you would put any exported symbols into the :export section. Notice that by default, you are already importing :cl.

[project-name].asd is best thought of as your project file. This tells Lisp what libraries to load up for your project, and what files are part of your project. You'll notice that defpackage and main are already listed in the :components section. If you add more files to your project, make sure to tack them on to the end of that list. Files will be loaded in the order they are listed.

Dependencies are also handled by the asd file. If, as in the earlier example, you want to use cl-ppcre, make sure to add it to the :depends-on section.

Asd files can get more complicated if you need, but this should be enough to take care of you in 95% of all cases. Happy hacking!

Monday, August 18, 2008

Unit Tests (Alpha)

Some of the other developers are currently working on integrating unit testing into Cusp. It's a project I'm not personally involved in, but I'm rather excited about. Up until now, the bulk of Cusp development has consisted of putting existing Slime functionality into a more sane and usable UI. With this one, we'll be breaking entirely new ground.

Things are still in the very early stages, so if you want to contribute code, ideas, or just see how the current plans are shaping up, you can check out the thread.

In which I mention that I like macros

Lest this blog be limited only to complaining about Lisp's shortcomings, allow me to take moment to sing the praises of macros, one of the greatest language features that's probably not even possible in other syntaxes.

Recently, I had to write a program to scrape a bunch of data off of a few thousand web pages. My general pattern was to skip forward until I found a certain string, then skip forward a bit more and grab whatever was between certain tags. It was also important to leave the cursor at the end of wherever I had searched, so that the next bit of data I scraped out would start looking at that point. This particular mutation made a function not terribly well suited to the task.

This came out to roughly ten lines of code per field searched for. But it was trivial to put those into a macro, and turn all my scrapes into something along the lines of (after-and-between phone "Phone:" "<b>" "<br").

The macro facility is one I especially miss when I'm programming Cusp (in Java, naturally). In the SwankInterface class, I have several dozen functions which all start with identical code to register a callback, and all end by calling a function to send a string to Swank. It's only 2 lines per function, but now that I'm used to Lisp, those lines annoy me. It's wasteful, and what if I have to change them someday?

Just one of the ways in which Lisp ruins other languages for you. CLOS makes things even worse.

Tuesday, August 12, 2008

The Library Problem

Today, I got a new task which involved scraping some web pages. Obviously, as part of this I needed a library to download the pages in question, so I asked around1 and was pointed to Drakma. It's Weitz-ware, so you can count on it to be reliable and well-documented, so I downloaded it.

Trying to load it did not go so well. I discovered that I needed a few dependencies. Some I had already, but I still had to go back and download Chunga (oddly, Wietz said to get 0.5.0 or higher, even though it's only up to 0.4.3, and he's the one who wrote Chunga2), and usocket, and a newer puri, and a newer cl-base64, and CL+SSL. CL+SSL was especially fun, because it's one of those libraries that says to pull it from cvs rather than use the tarball. Alas, the cvs command they provided utterly failed to work for me, so I went with the tarball anyway.

Once I had all those unzipped into my libraries folder, I tried again. FAILURE. It turns out that these dependencies had dependencies of their own. CFFI seemed to be called for. So I went to the CFFI download page, got it, and found that it had three more dependencies of its own.

Babel and trivial-features were easy enough downloads, but then I got to Alexandria. Alexandria has not yet made a real release. You have to pull it from the repository. But it's even better than that, because the repository is darcs, a version control system that pretty much nobody uses. In my case, I had to first install darcs just so I would be able to get this dependency that was 3 or 4 levels removed from the library I was actually interested in.

All told, it looks like I downloaded at least 9 additional libraries, one of which involved installing special software I have no need for. So Common Lisp definitely does have a library problem, but it's not the one a lot of people think it is. I've always been able to find a library that provides me with the functionality I need. The library problem lies in the fact that I need to download 20 other libraries, often through their versioning systems, in order to get that one to work.

Happily, this is a problem I'm in a good position to help with, since I'm already distributing a ready-to-use lisp system. As you may have noticed, Cusp comes with a number of popular lisp libraries already installed. The "Load Package" dialog even gives descriptions of what some of them are for, thanks to Sergey's work. Obviously, though, there are a lot more libraries I still need to include. It's a wonder I put up with this sort of thing when I was a newb.

So, I think packaging libraries will be a higher priority for the next release. And if somebody could make an asdf-install for sbcl that will work on windows, they will have my hearty thanks.


1s/around/one guy
2Don't laugh, though. His documentation is still way better than yours.

Sunday, June 29, 2008

Stupid Eclipse Tricks

Overall, Eclipse is great platform to program on, and making a plugin is an experience I would recommend to any coder because it will get you acquainted with a large, complex and fairly well-architectured system.

Of course, nothing is perfect, and Eclipse has faults just like the rest of us. Living in Java-land, it tends to suffer from what is termed ravioli code, where code is scattered throughout hundreds of tiny, tiny objects. Often, to do what you want to do, you need to define one class which then configured by another class which itself may be generated by some factory object. Often, I find the complex tasks are made very easy, but it can be maddening to track down the documentation for tiny little changes I want to make.

And occasionally, poor design does manage to slip in. Take the argument list tooltip (one of the 3 essential editor components). This handy guy gives you function signatures and a little descriptive documentation. In Cusp, The signature is formatted in bold to make it quicker for your eye to pick out. Displaying the tooltip to the user with the required information was trivial. Making part of it bold, that was the hard part.

Here's a quick overview of the architecture. To get your editor to behave the way you want it to, you subclass TextSourceViewerConfiguration, which then informs the editor how it should colorize syntax, how to indent, how to automplete, etc (most of these configurations involve returning another object that does the actual work). Lest you think this is an unnecessary abstraction, it turned out to be very handy when I wanted to make the input area of the REPL behave like the editor, even though they are different beasts.

In there, we tell the editor about Arglist Assistance as follows:


ca = new ContentAssistant();
ca.setContentAssistProcessor(new ArglistAssistProcessor(editor),
IDocument.DEFAULT_CONTENT_TYPE);

IPreferenceStore ps = LispPlugin.getDefault().getPreferenceStore();
ca.enableAutoActivation(ps.getBoolean(PreferenceConstants.AUTO_POPUP_COMPLETIONS));
ca.enableAutoInsert(ps.getBoolean(PreferenceConstants.AUTO_INSERT_COMPLETIONS));
ca.setAutoActivationDelay(ps.getInt(PreferenceConstants.AUTO_POPUP_COMPLETIONS_DELAY));
ca.setProposalPopupOrientation(ContentAssistant.CONTEXT_INFO_BELOW);
if( ps.getBoolean(PreferenceConstants.ARGLIST_BELOW)){
ca.setContextInformationPopupOrientation(ContentAssistant.CONTEXT_INFO_BELOW);
} else {
ca.setContextInformationPopupOrientation(ContentAssistant.CONTEXT_INFO_ABOVE);
}

...

public IContentAssistant getContentAssistant(ISourceViewer sourceViewer) {
ca.setInformationControlCreator(new LispTextHoverControlCreator());
return ca;
}


As you may or may not have noticed, ArglistAssistProcessor is the class that actually figures out what information to display. It parses the document to find out what function is being called (or if the cursor is even in a place where a tooltip makes sense), then talks to Swank to get the information, and then returns that in a format Eclipse understands. The process may sound a little complex when you first run into it, but it is well-covered in the tutorials, so it's fairly easy to get to this point.

But how do I format that text for easier reading? Here, the documentation failed me. And so I'm writing up the answer as a service to the rest of you who may want to do this.

Part of the interface for IContentAssistProcessor is getContextInformationValidator(), which returns an IContextInformationValidator, the object that determines when it is time get rid of the tooltip. It turns out that to format the tooltip, you need to make this object also implement IContextInformationValidator.

This is bad design for so many reasons. Nothing in the documentation indicates that this is the place to do the formatting, and it makes no logical sense for a validator to also (sometimes) act as a formatter. It is also rather contrary to the usual Java and Eclipse way to take an object designed for a completely separate task, and, if it happens to implement another interface, use it for that as well. I'm pretty sure this was slapped on with little thought.

But now it's documented. If you're wondering how to format your tooltips in your own editor, now you know.

Wednesday, June 25, 2008

Eclipse 3.4


The latest version of Eclipse is out. The new bits look very shiny, so I may have to go and download it, which I should probably do anyway for the sake of keeping Cusp up to date. Or, I may stick to the lazy route, and wait for Ubuntu to distribute the new version.

Tuesday, June 24, 2008

What do you look for in a development environment?


It turns out that my core needs for an editor are very basic. Back when I first started constructing Cusp, I had three primary things that were absolutely necessary for me to consider it a useful tool:
  1. Syntax Highlighting. Above all else, I need this. Without basic coloring, I find code far far harder to work with, and any time I'm sshed into my server and forced to edit Lisp code in Emacs, the first thing I do is M-x font-lock-mode. In fact, I can get along pretty happily with this functionality alone in an editor, which probably why most basic text editors since Notepad make sure to include it. (Interesting note: for a long time Dr. Scheme did not have this functionality, and it made me a little embarrassed when recommending it to people.)

  2. Autocomplete. Saves typing, and precious (human) memory space. With this, you usually only need a general idea of what function you're looking for. Glancing through the list of functions is almost always faster than poking through documentation. This makes the editor and order of magnitude friendlier to people who are new to the language, and makes learning your way around the language much easier. And even now that I know Lisp fairly well, it lets me code faster and with fewer typos.

  3. Argument list assistance. Another important learning tool. You should never have to do something a computer can do for you, and one thing computers are way better at is memorizing function signatures. This saves time and a break in your workflow by looking up the documentation for you and telling you just what your function wants. (As a bonus, it really should include a quick bit of function documentation if available.) This sort of thing is great when you're, say, trying to sort out the differences between elt and nth.

    As an aside, this feature is one of the reasons I decided to bundle SBCL rather than CLisp. CLisp promptly forgets the argument names to a function, and can only tell me that it wants (elt arg0 arg1) and (nth arg0 arg1). SBCL is far more helpful, informing me that it wants (elt SEQUENCE INDEX) and (nth N LIST).

    As an additional aside, and because I just can't resist beating on Emacs, this information really should be displayed in a tooltip where you're typing (whether it goes above or below is a matter of taste). Making the user's eyes jump to the bottom of the screen and then seek out the cursor again is inefficient.

And that's all. Give me those in an editor and I'll be pretty well satisfied. Happily, these are also tasks that Eclipse makes very easy. Of course, since I was making a Lisp editor, parenthesis matching, integration with the runtime, and a REPL were also required, but that doesn't apply to most languages, and those were also fairly easily done. From there, it was a short walk to functional equivalence with SLIME.

So keep that in mind, editor-makers. Those three things are all you really need. And if you use Eclipse, all the hard work has already been done for you.

Thursday, May 22, 2008

Step in Time


A recent post to the development list has led to some much-needed improvements to the debugger. Specifically, it now support stepping through your code.

This was, I must admit, a rather major omission, and my only defense is that it showcases how handy "many eyes" can be at correcting these sorts of problems.

Since I am the principle developer behind Cusp, it naturally is tailored to my usage patterns. As it is, I use Lisp also exclusively to develop web pages in UCW. As a result, my interaction with the debugger is actually quite limited. It basically only pops up when I have a compile error, at which point I do little more than dismiss it. If something breaks on a page render, ucw catches it and gives me a page full of a backtrace. I certainly never step through code. In fact, I wasn't actually aware that SBCL supported such (if only I had thought to put a (break) in my code, bump up the debug level, and hit 's' once in the debugger, this all could have been avoided).

At any rate, now that Cusp is open source, and has users other than me, people start to fill in those gaps. A Vladislav Titov was so kind as to improve the debugger's handling of multithreading as well, and syncing up to the latest version of Slime while he was at it.

I really enjoy making improvements like these that have UI components. It's very rewarding when you can actually see something on the screen to demonstrate that a change occurred. It makes me wish there were a few more major items I could add. Ah well. Enjoy your new, improved debugger!

Friday, April 18, 2008

Welcome to the early 90s


Recently, my grandmother (who is turning 90 this year) got email. Well, that's not entirely accurate. What she actually has is a printer which takes the emails sent to her and converts them into dead tree format for her reading pleasure. It costs money, and she is unable to send anything, but at least now I can forward her pictures of my new puppy in a quick and timely fashion.

What does this have to do with Cusp, you ask? Well, it came to mind when I read the following tale of RMS's encounter with Eclipse. There he encountered such "neat features" as "tabs."

I don't know that there's much more to say. That post confirms everything I've ever suspected about Emacs users. Read it and weep.

Sunday, January 27, 2008

Smoothing the way


Periodically, I like to ego-surf to find out what people think of Cusp. It's usually complimentary, which is always nice to see (even if they tend to call it "CUSP"), but I like to keep a special eye out for people who are having issues, since that usually points me toward ways I can improve the experience for people.

Recently, I ran into this post. The author had a load statement in his file that just wasn't working. Why? Because he was calling it merely "file.lisp." No absolute path was given, and since sbcl is almost certainly not being run in the same directory as that file, it's not going to be able to find it without more information.

The sad part for me is that I'm not entirely sure there's much I can do for this kind of problem. Firstly, a person could very easily have multiple projects going at the same time, so determining the proper working directory is a chancy thing. Even then, automatically shifting working directories is the kind of anti-feature that soon causes more trouble than it saves.

This isn't an issue isolated to this one guy, though. Judging from the comments and recent posting on the Cusp discussion boards, others have tried to do the same thing. I think the real answer here is actually to educate people on the usage of asd files, and to get them to treat them as project files. I intend to add some comments to the auto-generated asd files to get people on the right track, but that's probably the best I can do.

Sadly, I couldn't resolve that issue technologically, but I was able to do something about one of the comments on that entry. The problem? When you create a new project, Cusp automatically wraps it all into a new package. Once you know what you're doing, this a very nice thing, because it is what you should be doing anyway. But when you're completely fresh to Lisp, you're not likely to realize that the stuff in the file is going to one package, while the Repl is sitting in another.

The fix was easy. Now when you make a new project, the REPL is automatically switched to that project's package. 99% of the time, this is what you'll want to do, and it should allow newbies to experiment without first having to learn all about the packaging system.

Friday, January 18, 2008

Credo

Starting with Lisp is a daunting task. First you download SBCL. Then you check out SBCL from cvs, and use the version you just downloaded to compile a new one. Then you download some flavor of Emacs and install it. Then you download the latest release of Slime. Then you go back and do a cvs checkout of Slime instead, because the last release is nearly two years old. Then you customize your .emacs file so that Slime can hook up to lisp.

Ok, so that was a great big pain, but at long last, you're ready to try out the most advanced language ever! You cross your fingers and fire up Emacs. Behold!


This is the face of the most advanced and powerful language ever? The development environment clearly hasn't taken advantage of any advances in user interface since the 70's. If the development environment hasn't evolved past the command line, can we really expect better of the language?

And what if you don't know how to use Emacs? Now you have two problems. Your early forays into Lisp are going to be frustrating and slow as you struggle just to edit text using a program which acts like no other. This is surely going to taint your perception of the language itself. How many newcomers turn away in disgust at this early stage?

And we haven't even mentioned the library situation yet.


Enter Cusp. Cusp was made with two goals in mind:

  • Provide a high quality Lisp IDE with all the amenities that users of modern IDE's have come to expect.
  • Ensure that it works straight out of the box.

I may be biased, but so far I think it has fulfilled those goals quite well. But there's always more work to be done. Join us!