I'm splitting hairs a bit, but I would argue that there's too much emphasis on typesetting here.
Reading Knuth's original paper on literate programming, TeX was both an arbitrary and obvious choice for WEB, since that was his tool as well. He also mentions Scribe and Troff as alternatives for "document formatting languages". The thing thay these all have in common is that they target dead trees rather than screens. This was the simply the way at the time. Keep in mind that this was years away from things like hypertext or any dynamic content like that.
Compared to Knuth era software, modern day software is:
Designed to change/pivot a lot. In both structure and scope.
Maintained by groups of people instead of one person.
It'd be pretty miserable to write most contemporary software in literate style. It's just not compatible.
Note that I don't consider this to be a fault with literate programming, but rather the way we tend to write software now.
To be clear: inline comments in code are *not* literate programming, no matter how thoughtful you get with them. Never will be. It feels very different from writing code blocks in a markup language. A true literate programming system has the ability to write code out of order as well.
A nice consequence of using a markup language is that you can write and outline structure before producing any actual code. Many of my own literate programs started this way, and I believe they turned out to be better because of it.
Not everything should be a literate program. Most lines of code aren't worth working into a literate style. Choosing when to write in a literate style and when to avoid it has been a more conscious choice for me.
In sndkit, all the DSP algorithms, as well as the core system, have been written in a literate style. But that's only a fraction of the codebase. The rest of the codebase is either a third party library like LIL, or some kind of boring glue code that's mostly boilerplate.
My thoughts moving forward with literate programs are maps. How does one effectively guide the reader? There's very little in common between a novel and a Knuthian literate program beyond the print medium they share. There's is hardly any value to reading a literate program cover to cover. For a big book, Usually there's a good table of contents or an index. These are maps.
But literate programs aren't print medium anymore. it's just not how we consume words and text now. We have tiny screens that can dynamically display interactive content. We don't flip through pages. We scroll through threads and click on links.
There's a lot of meta information that a literate program can provide actually. In my system I've written, I'm only presenting a fraction of it when I publish the HTML. It could be more helpful.
@penguin42 normally I read my literate programs in emacs via org mode, but I do publish an HTML output.
the loom is where I put all my smaller literate programs:
This is my sndkit algorithms page, each one is a self contained literature program:
My most ambitious literate program is monolith: https://pbat.ch/proj/monolith/program/
@penguin42 here's code for some of the smaller programs. For these, all the C code is in org file. I never read the actual tangled C code. My tangler takes advantage of line macros so that the compiler reports errors with line numbers corresponding to the org file, not the C file. (CWEB does this too. Org-tangle does not).
@paul Hmm so to me that looks primarily like: a) Full detailed comments before a function on how it works b) Very little/no comments inside the function c) A more powerful macro scheme for some cases.
I'll throw out mine as well: https://github.com/akkartik/mu/tree/main/linux/bootstrap
Here's some more detail about how it's built: http://akkartik.name/post/four-repos
I'd like to reiterate from my original post: "If you have spent time reading somebody else's literate programs, I want to hear about your experiences!"
@penguin42 Yeah, the mechanic is a bit different. The goal is the same, though: to convey a program to someone else.
I chose the new mechanic because it allows me to write large programs that can be divided up into "chapters" or layers. And you can load any initial sequence of layers to end up with a simpler program that's easier to understand.
More details: http://akkartik.name/post/wart-layers
@akkartik you mention writing in a literate style is meant for others to grok the program, do you think writing a literate program helps you at all?
My Kuf program, for example. I just can't keep in my head for too long. It is a very confusing and unintuitive set of bitwise operations by the time it reaches code. Using traditional comments, I may have written enough explanation. Maybe. But the literate programming structure really allowed me to jot down my thoughts at the time in a very frictionless way. I was able to reorient myself to the program after months of being away from it.
I can also say the same for many of the DSP algorithms on sndkit.
@paul Oh yes! 100%.
I think an unspoken premise in all this, going all the way back to Knuth, is that any documentation system is only sustainable if it's helpful to the writer.
"This experimental skeleton will hopefully make it easier for others to understand and more malleable, easier to radically rewrite without breaking in subtle corner cases. Understandability and rewrite-friendliness are related in a virtuous cycle. Each enables the other."
@penguin42 I do occasionally need to look at the tangled output. My tooling allows for that, unlike most literate systems whose tangled output is a mess.
Newcomers need this even more. Think of the layers as just the .git representation. Once you've selected a set of layers to read, just tangle them and read the generated bootstrap.cc
But it's a mutable .git representation that I the author can easily streamline if I come up with clearer ways to explain something.
@akkartik I've got hard copies of TeX and Metafont the program on my bookshelf. It hits differently than if you were to read it as a PDF.
Nothing even resembling dharma transmission for either of these programs (maybe that's okay?) But it's enjoyable to flip around and read the prose. I wish things were written in a more modular, self contained way like PBRT.
Knuth has consistency in structure in writing, and it's actually inspired me to rethink the way I communicate about program structure:
@paul I really need to maintain a list of people describing what it's like to read other people's Literate programs. I think I've had 3 people respond to my call to say yes, they have read Literate programs. I'm going to start it now and add yours.
I'm particularly interested in usability reports. What gave you the most trouble in understanding some Literate program written by someone else.
@akkartik @penguin42 @paul glad to see this thread today. I'm not any sort of serious literate programmer, but I have been interested ever since I read about org babel and I have maintained my emacs config as a literate program. I don't get a lot of free time to do so, but I will give a try at reading some of your programs
@paul this is interesting shit. Why isn't this a popular thread with a lot of knowledgable people (I'm not) engaged in constructive comments? patchlore = Edward Tufte of coding? go! Maps, we need maps! As an aside, where is Bret Viktor? Any news? (a web page with background auto play videos? come on.)
@lutzray Maps for programming is something I've thought about a lot.
Welcome to post.lurk.org, an instance for discussions around cultural freedom, experimental, new media art, net and computational culture, and things like that.