Tools/Techniques needed before wide acceptance of literate programming?

What tools or techniques do you think we still need before literate programming is 'ready for the masses'?

We have an incredible proliferation of different representations ("systems") for literate programs and suites of tools for manipulating them. No tool works with multiple representations. To my knowledge, only one representation is designed for easy manipulation by tools. That one has attracted a few tool writers, but none of the more interesting or difficult tools (prettyprinters, hypertext browsers, multi-mode editors) work with it. Many representations support at most a handful of programming languages (a serious flaw) and typesetters (a minor flaw). Many tools are overly complex and perpetuate the problems identified by Ramsey and Marceau (SPE, July 1991). In short, the tools are immature. Progress might be made if we all tried getting behind a single representation and went to work making it support any language, any typesetter (including WYSIWYG), and if we created some simple but interesting tools, with all the amenities people have grown accustomed to, like automated indexing, prettyprinting, etc. Then again, it might not.

As to techniques, I claim there aren't any. We have Knuth's admonishment to write for human readers, and mine to use lots of peer review, and I think that about exhausts the contenders. Our only book on the subject spends hundreds of pages mostly guiding the reader around the idiosyncrasies of the tools. We have exactly two published literate programs of any size, programs which might never have been published if not for the immense prestige their author earned in other endeavors.

Should a literate program resemble a novel, essay, encyclopedia, textbook, or automobile-repair manual? No one knows. Probably none of these---after all, it is a new literary form. But do we create and publish literate programs so that the community can study them, learn from them, and perhaps build an understanding of what a literate program is and how to write one? No! No, we write tools, because that's the only thing we're smart enough to understand. I count myself doubly guilty, since I have written twice as many tool sets as most other contenders.

Well, I apologize for the polemic. Put it down to annoyance. I just finished an article on yet another literate-programming tool, and the article contained neither a clear description of the tool nor any reference to the criteria so clearly set forth by Thimbleby. I wish I knew what the editors and referees were thinking of.

Don't! I think this group needs more discussion of this type if we are to advance the 'cause' of literate programming. Since I joined the list, posts have been mostly about the mechanics of specific implementations. I don't say that that is a bad thing, just that it must not be the ONLY thread. I am a bare beginner, so I not ready to contribute much yet, but think we would all benefit from more discussion of the points that Norman has raised.

Any good? Reference?

I think, as before, that the problem is in the tools we use, but just because they are not powerful enough. For example we have to navigate a graph (our webfile) using a linear tool (an editor), it would be much better to have an hypertext tools. The formatting commands and the special commands we use "obscure" the text and the code we are writing, so a WYSIWYG hypertext tool should be even better! Of course we would have to be platform independent (one of the biggest advantages of TeX is its availability for many platforms), and typeset beautiful math and so on.

But here is a problem: more powerful the tools, more powerful the machines we have to use (I simply can't stand to wait too much for, say, the generation of the indices) and I work on various machines that range from IBM RS6000 to MS-DOS. For now I have adopted the nuweb system from Preston Briggs, it's clear, fast, language independent, and gives me all I need to be a literate. Nuweb is the easiest and most natural tool I have used.

The problem is: why using literate programming? Personally I have choose literate programming first of all for helping myself during development, and second for the others that, finally, can read my code and understand what I intended to do. During the development I absolutely don't want to be bothered with typesetting problems (after all maybe none will read my code!) but I still want indexes and cross referencing of macros etc, etc. so I am currently writing a tool that will do this kind of works for the nuweb system. In conclusion I am very interested in what you think are the "intimate essence" of literate programming, both from the experienced literate programmers, and the newcomers.

No. Garbage. Reference withheld to protect the guilty.

Recently, Remo Dentato gave an excellent summary of his experience with literate programming and opinions. Some background: I work in very much an engineering environment surrounded by DEC & SUN workstations, PCs and MACs. Much of our software development is for PCs to act as controllers for various bits of lab equipment or devices we have built.

The sorts of code that is produced here tends to be extremely cryptic little device drivers (i.e. a couple of hundred lines of source C code) for DOS and Windows. It is often difficult to explain all of the intricacies of a given piece of code even to other members of ones own project team. Witness me trying to illustrate code that writes a particular byte to some port in order to toggle specific bits on some home brew board. The illustration is rife with interruptions: Why that bit pattern? Why that port? Why perform that operation on this data? It seems that prolific /* commenting */ is not adequate.

Project management started to come down on us poor code writers demanding better software specification from the programmers and pleading for better documentation of any code that gets written. (Why programmers are ever writing specifications I will never understand---in MIS departments, the specification is handed to the programmer who is told "Go away and code something that does this!" True or false?)

It was at this point that literate programming came into my life . Unfortunately, given the depth of my search for information on the topic, it was not at all obvious how to write my short cryptic routines in such a way as to make them Literate. I have FWEB, am writing in C, use Emacs 18.57 with a web-mode, do not know very much about TeX (but have been through "The TeX Book" once already and discovered there is not much that I have to know to get by on if I don't want anything too fancy), and am following this mailing list with avid interest. (Once again, I wouldn't mind at all if it became a news group: the more people that can read about our experiences, I feel, the faster this technique might mature and turn into something even more wonderful that some people already believe it to be! Seems to me thought that it doesn't matter whether my mail box is full or my news reader overflows---same problem, different windows!)

I have tried WinWordWEB, a nice little tool for those interested in WYSIWYG literate programming. There is no real provision there for hypertext-like coding, however. I feel that implementing a hypertext, WYSIWYG literate programming program using something like HTML could get very cumbersome unless appropriate tools are available to handle the messy repetitive details of the hypertext links. Certainly the idea of a hypertext coding environment has merit: the reason for this post is to broach the subject of Microsoft's Browse Utility which comes as part of their top end compiler packages.

We recently purchased MS C/C++ 7.0. From within their Programmer's WorkBench editor environment , it is possible to generate a 'browse database' containing cross references for any and all definitions and references to source files, data types, identifiers, etc. present in your code. Relationships between source files and identifier references can be displayed, and almost everything is point and click: see a variable name, want to know where it is defined/referenced? double click on the name, up comes a dialog or up comes an editor window scrolled to precisely the right point in the file. Which function(s) call(s) this library routine? Point and click!

The ability to have a global view of the code like this is extremely handy. It does absolutely nothing for improving documentation, but when writing code and in particular debugging it, this browse feature is most beneficial. Its a pity that it is solely a MS product, that it must be run on a PC under specific circumstances, but it certainly seems to be able to handle mixed languages (We tried with C and Fortran). I would like to see this sort of functionality in an X client. The PC stuff from MS gives you this pseudo-hypertext-like stuff through a hierarchy of dialog boxes. Perhaps its not true hypertext , but it sure beats scrolling through hundreds (or thousands) of lines of code looking for what you want!

I have asked this question several time to the literate programming netters before. For myself I am getting more and more convinced that the essential quality of literate programming is to associate a given design step exactly with its consequences in terms of the steps (code) of an algorithm, Nothing more and nothing less. In other words, literate programming makes explicit which code lines are responsible for which design decisions

I believe that WEB and CWEB share one critical characteristic which are missing from some other literate programming systems: the ability to create an exposition in the web which is organized independently of the requirements of the compiler. Of course I refer to the infamous modules, which permit me to describe part of function A, then part of function B, then some more of A, followed by some of C, and so forth. Modules free the programmer from having the development of the explanation restricted by linear processing of the compiler. If one cannot break the tyranny of the compiler, one might as well just write comments and make do with a pretty printer. I think this characteristic is close to the "essence" of literate programming, since it allows explanations to follow literate patterns directed toward human readers. I consider it to be more important than tables of contents, indices, or hypertext links.

I agree that the modules/scraps are very important, but I think that almost all of the literate programming tools I tried had it! Nuweb, noweb, FunnelWeb, CLiP, WinWordWEB. Without that ability: "to break the tyranny of the compiler", they would be quite useless as some do not even pretty-print the code! With the module concept they help to implement the literate programming paradigm: make it read/look like a book.

I would, however, strongly recommend using the "doc" style option for doing literate TeX programming. It has the additional advantage that you do not need tangle or weave programs, since all typesetting (I believe this might be the case in Noweb as well) appears as comments to TeX when using the style, and TeX is pretty fast in skipping them. This is OK for testing, and for production versions you can use docstrip for getting the comments out. Sorry for this digression, but I think the "doc" style option is mentioned not often enough, and it really makes TeX programming literate.

I agree with Norman that we need to start publishing our literate programs. One of the problems, however, is that many editors don't know what to do with them. We need to "educate our editors," to paraphrase Grace Hopper. On the plus side (in the "toot my own horn" department) the "Paradox Informant" (Jerry Coffey, Enlightened Editor) will be publishing a literate program of mine (written using noweb) in the September issue along with another article on "Literate Programming in PAL & ObjectPAL" ("God willing and the crick don't rise"). I am also planning on rewriting my Baby Manchester Mark I simulator in CWEB specifically with the idea of publication.

Obviously, I am not even close to the first to publish a literate program, but it seems that the momentum in the literate programming community has picked up to the point where we need to stop "reinventing the wheel," and start using the wheels we have. [The phrasing is a bit too strong, but the idea is there.] Remember that the literate programming column in CACM died because people were just building tools rather than using existing ones, and that was a good 5 years ago.

On the other hand, the text in a web is what is definitive. I find that most of my debugging consists of making sure that the code chunks agree with their respective text chunks. I would suggest that, when debugging a web, the code sections be covered (at least initially). The names of the code chunks also provide significant information. For example, a recent program of mine included the following (embarrassingly stupid) bug:

@<Do something if |set| is not empty@>=
    if (set == NULL) do_something();