An accessible programer's editor

[compilerbitch]

I'm actually very seriously considering writing a programmer's editor from scratch, incorporating voice recognition and ideas from the Dasher interface (http://www.inference.phy.cam.ac.uk/dasher/), along with a few ideas of my own. Leo Dearden (the friend with major RSI I mentioned) has also contributed some really good ideas. I need something like that so much myself, it just has to be done. This is an idea that has been bubbling away for some time now, but has now probably reached a point where it warrants being made a little more public.

Leo called the idea a 'low friction editor'. The idea is that everything in the editor should be as smooth and easy as possible. You should be able to type by actually typing the old-fashioned way, by voice recognition, gesture recognition if you have something Touchstream-like, or through a Dasher-based interface. All of this should be seamlessly integrated. The editor will use a grammar definition of the language annotated with layout rules, so if you open an existing file it will parse it into an internal, higher level form. This will allow really good syntax colouring, of course, whilst also being a convenient way to synthesize new chunks of code in a syntactically correct way.

I want to be able to point at a line of code, then say, "Insert while loop" and have the editor do it. I want to be able to select an STL object and have the editor automatically generate a loop that iterates over it. And that's just C++... I also want it to be able to edit Haskell, marked up English text in LaTeX format, etc.

Personally, I don't really see the Dasher interface as a keyboard replacement for anyone who doesn't happen to be disabled. But it could be a great way to select from tree-structured menus, especially where the menu options have relevance weightings. Like, for example, selecting code templates for inclusion.

Having said that, if the user happens to be disabled, Dasher-style input can be the difference between a career and no career.

The editor I have in mind will basically do all of the above, including genuinely usable voice recognition. I'm not necessarily trying to write an editor that will replace emacs or vi, although it might ultimately have that effect. No, what I'm really interested in is making it possible to edit code as quickly as possible, if necessary without using a keyboard *at all*. You should be able to drive it completely by voice, completely by pointing device (e.g. pen interfaces, eye trackers, mice, etc.), completely by keyboard, or any combination of the above, using a minimum of movements in all cases. This equates to minimum possible RSI exposure, and also (possibly, ideally, maybe) fastest possible editing, once you're used to it. I can forsee myself using a combination of approaches, all at once.

All the technologies exist, albeit independently of each other at the moment. Bringing them together will take work, of course, but for many people, myself included, an editor like this could really be a career-saver. I have a feeling that it might just become rather popular as a general purpose editor anyway. I think editors have been stuck in an emacs/vi dominated rut for far too long. Undoubtedly, thy both work well, and will never be completely replaced. But, try using vi with voice recognition... No. Something purpose designed is essential.

I'm posting here to solicit reactions to the idea. I will almost certainly write an editor like the one I am describing, because it is basically insurance for me. As an academic computer scientist, I must retain my ability to edit code and write papers using LaTeX. There are plenty of solutions if all you want to do is drive MS Word, but nothing whatsoever if you happen to be a programmer.

I am very keen to hear suggestions - ideally, please add them as replies to this LJ post. Please feel free to pass on the link to anyone who might bo interested -- the more feedback I have at an early stage, the better.

Comments

[sjmurdoch.livejournal.com]

One tool that might be worth a look at is GRASP.
In particular this has some interesting structure visualization (CSDs) and template inclusion facilities.

[compilerbitch.livejournal.com]

Looks good. It is not hard to see how the editor could be made to do that in real time as editing is carried out.

I am currently considering writing the GUI in C++, but implementing the internals in Haskell. This will make the grammar processing a lot easier, whilst also having a lot of for-free benefits like for example making undo/redo trivial to implement. I'd also like to be able to do whatever I like graphically, so I don't want to be limited by having to code a GUI in a functional style. Qt is probably my platform of choice -- it supports all the major platforms, I already know it, and it is capable of being hacked to do very nonstandard things if necessary. The Glasgow Haskell Compiler supports most of the same platforms, interops with C and C++ well, and generates good code.

It would be cool to implement 'modes' as an embedded language within Haskell. This would save a vast amount of work, whilst being stupidly powerful.

[catamorphism.livejournal.com]

You should look at the Harmonia project if you haven't already; I don't know how far they've come WRT voice recognition, but it's definitely something they're actively working on.

[compilerbitch.livejournal.com]

Looks interesting. Do you happen to know how far they have got with it? Is it still some way off in the future?

[catamorphism.livejournal.com]

I don't know exactly. You might want to email Andy Begel (he's the main person working on the speech recognition end of things in particular -- you can find his address on the web page) and ask about it. (LiveJournal as means of making research collaborations... I love it :-)

[trurl.livejournal.com]

It definitely sounds like a worthwhile idea. I see two ways that you can go with this. One is you can also try to reinvent editors in general, the other is try to layer this facility on top of something else, most likely emacs. The former has its advantages, because I'm always wishing that it were possible to start over an re-architect something like emacs from scratch. But it is also just a bit ambitious. The latter has the benefit that if you could make it work well with existing emacs packages and so forth, it would be easier to migrate. There are some software packages that I use that would be intolerable without their emacs mode. Plus, it emacs is itself more like an operating environment than a mere text editor, so it seems likely that you could just extend the user interface adding new hooks, and the elisp would be none the wiser.





Some of the structured editing ideas you have could probably be implemented directly in elisp, but I worry that the performance might be pretty poor with all the parsing and other computations that would need to be done. However, that could perhaps be alleviated by improving the technology behind the elisp interpreter.





While in principle I like the idea of having menus be optimized based upon usage, my experience with Win2K hiding menu options based upon usage patterns always kind of annoyed me. And there is the consistency issue. User interfaces should be "stable" in some sense, so that actions can become ingrained, and I worry that unless it is carefully designed, having the interface learn at the same time you are could be frustrating.

[compilerbitch.livejournal.com]

I had considered using an off-the-shelf editor as the basis. Obviously, this looks like a tempting option, but my concern is that I don't want to have to work around an existing design that doesn't properly cope with accessibility issues. Fundamentally, text editing isn't that hard. Getting that part for free is OK as far as it goes, but ultimately the issues associated with gluing it into something more sophisticated are probably harder than just writing the editing code itself.

On the menu optimisation front, point taken. I'm not keen on 2k doing that stuff. But, it works for Dasher, so I think it is worthy of further study. My current thinking is that the editor should have a single command grammar that can be accessed by multiple alternative technologies: keyboard shortcuts, vi-like mnemonics, typed English like commands with autocompletion, gestures, pull-down menus, Dasher-like tree navigation and voice recognition. Some of these will benefit from stats, some won't. In the case of voice and Dasher, stats are essential. For some of the interfaces, they are less relevant. The command grammar will itself be a higher order data structure, and will need to adapt to the circumstances. In all, it looks like an interesting project!

OK, yes, I am keen on building the editor that I have always wanted. I'm not scared of the coding that will be necessary. I'm going to give it away free, and make sure it runs on all the major platforms, and write it in such a way as to allow people to hack their own modes, emacs-style.

I am quite keen to implement the internals in Haskell. Lazy evaluation gives a lot of what will be needed for free, quite efficiently. For a start, referential transparency will make efficiently implementing undo/redo completely trivial. A lazy list containing every version of the data structure that represents the document back to the point that the file was loaded would be trivial to implement, yet thanks to lazy evaluation would still be very space efficient.

I have quite an elegant design in mind, I think, so I don't want none of that messy other-people's-code cluttering up the place, nosiree! 8-)

[mjg59.livejournal.com]

Damnit, I knew I was supposed to be mailing someone about Dasher, now I remember who...

The existing Dasher codebase is almost (but not entirely) completely unsuited to what you want to do in this case - the majority of the code is based on the assumption that each layer of the "tree" will be identical to the previous one other than for the probabilities. It's easy enough to hack around that if you're not interested in prediction (which is what I've done for some of the application control stuff), but it somewhat defeats the point otherwise.

On the other hand, alternative language models for situations where there's a somewhat more defined grammar than English are something that we'd be interested in taking a look at - Dasher works surprisingly well for writing C if that's what you train it on beforehand, but it's (unsurprisingly) far from ideal. I'm not sure how well it'd fit into your ideas, but I'd certainly be interested in seeing whether there's any useful cooperative work that can be done.

[compilerbitch.livejournal.com]

I was kind-of expecting the existing Dasher code to be a bit too specific. What I had in mind was a grammar implemented as some kind of higher order data structure (probably in a lazy functional language like Haskell), visualised by a front end (probably in C++) that can provide access to the grammar in multiple ways. For example, the same structure could be visualised by cascading pull down/pop up menus, something Dasher-like, voice, auto completion, content-sensitive 'palette' panes that allow keywords/variables/code templates to be inserted with a single click, etc. This approach has the same kind of qualitative advantages that are achieved by separating the front and back halves of a compiler. I think there is a strong case for building a new editor -- it's hard to see how this kind of architecture could be retrofitted to an existing editor like emacs or jEdit.

Another Dasher-related question for you. One issue I can see with the existing Dasher design is that the dasher interface is in a separate, fixed split pane. Leo Dearden and I were speculating that floating this pane 'over' the source code might be better, maybe even using partial transparency so you can still read what is under it. Our reason for thinking this was partly because it would very significantly reduce the amount of mouse movement needed when swapping between Dasher and selecting text or positioning the cursor. Also, it would allow nearly all of the screen to be taken up with code -- something that programmers typically find extremely important. Have you experimented with that kind of design at all?

We were also thinking about the possibility of building the interface in OpenGL, because this would immediately open up access to the extreme performance of modern graphics cards, whilst also making things like antialiasing and transparency relatively easy to implement. Also, it should open up interesting ways to visualise and navigate code graphically, but that's getting away from the initial intentions somewhat.

I'm going to be away for a week now, but would definitely like to discuss this further when I get back.

[mjg59.livejournal.com]

It certainly sounds interesting, and it would be good to see what you come up with. As far as the transparency goes, the MacOS X version has something pretty much as you're suggesting (screenshot (http://vavatch.codon.org.uk/~mjg59/DasherMacOSX.jpg)) - the Windows client would also be amenable to this sort of thing, but we start running into trouble under X11 due to its complete lack of implementation of transparency (and various other useful things, but...). The current X11 code lets you shring the edit box to nothing and then enter text directly into another window if you have recent enough (CVS) GNOME accessibility code.

Using OpenGL for the interface code would be fun, but at the moment I'm doing most of my development work on my laptop which has a quite desperately poor 2D-only graphics chip. When I get my desktop sorted again, I'll take a look at it.

Please do get in touch when you're back - I'm away next week, but around after that.