Ah, and it seems the literate programming criticism post by @akkartik has popped up again too on HN.

akkartik.name/post/literate-pr

Any new thoughts since you've written that?

I'm splitting hairs a bit, but I would argue that there's too much emphasis on typesetting here.

Reading Knuth's original paper on literate programming, TeX was both an arbitrary and obvious choice for WEB, since that was his tool as well. He also mentions Scribe and Troff as alternatives for "document formatting languages". The thing thay these all have in common is that they target dead trees rather than screens. This was the simply the way at the time. Keep in mind that this was years away from things like hypertext or any dynamic content like that.

Follow

But to be clear, if you're fixated on typesetting your literate programs in 2022, you are objectively stuck in a 70s mode of thinking about literate programming. There are very few contemporary systems where this still makes sense.

Compared to Knuth era software, modern day software is:

Designed to change/pivot a lot. In both structure and scope.

Maintained by groups of people instead of one person.

It'd be pretty miserable to write most contemporary software in literate style. It's just not compatible.

Note that I don't consider this to be a fault with literate programming, but rather the way we tend to write software now.

Programs that lend themselves well to literate programs are things that are complex, well defined, and that change slowly. Anytime I'm building some kind of core system or architecture, I like to do it in a literate style. Not only does it serve as a good reference document later, but it also forces me to think more carefully about what I'm doing.

To be clear: inline comments in code are *not* literate programming, no matter how thoughtful you get with them. Never will be. It feels very different from writing code blocks in a markup language. A true literate programming system has the ability to write code out of order as well.

A nice consequence of using a markup language is that you can write and outline structure before producing any actual code. Many of my own literate programs started this way, and I believe they turned out to be better because of it.

Not everything should be a literate program. Most lines of code aren't worth working into a literate style. Choosing when to write in a literate style and when to avoid it has been a more conscious choice for me.

In sndkit, all the DSP algorithms, as well as the core system, have been written in a literate style. But that's only a fraction of the codebase. The rest of the codebase is either a third party library like LIL, or some kind of boring glue code that's mostly boilerplate.

My thoughts moving forward with literate programs are maps. How does one effectively guide the reader? There's very little in common between a novel and a Knuthian literate program beyond the print medium they share. There's is hardly any value to reading a literate program cover to cover. For a big book, Usually there's a good table of contents or an index. These are maps.

But literate programs aren't print medium anymore. it's just not how we consume words and text now. We have tiny screens that can dynamically display interactive content. We don't flip through pages. We scroll through threads and click on links.

There's a lot of meta information that a literate program can provide actually. In my system I've written, I'm only presenting a fraction of it when I publish the HTML. It could be more helpful.

@paul i never learned the internals of TeX but i'm guessing a programmable typesetter is still strictly superior to the W3C fuckstack.

@icedquinn the problem isn't the typesetter, it's the PDF and PDF readers. They are just clunky for reading on screens. The dynamic scaling of brutalist HTML with hyperlinks has been a much nicer experience for me than pretty looking documents.

@paul Have you got an example of what your code looks like?

@penguin42 normally I read my literate programs in emacs via org mode, but I do publish an HTML output.

the loom is where I put all my smaller literate programs:

pbat.ch/loom/

This is my sndkit algorithms page, each one is a self contained literature program:

pbat.ch/sndkit/algos/

My most ambitious literate program is monolith: pbat.ch/proj/monolith/program/

@penguin42 here's code for some of the smaller programs. For these, all the C code is in org file. I never read the actual tangled C code. My tangler takes advantage of line macros so that the compiler reports errors with line numbers corresponding to the org file, not the C file. (CWEB does this too. Org-tangle does not).

git.sr.ht/~pbatch/btprnt

git.sr.ht/~pbatch/kuf

@paul Hmm so to me that looks primarily like: a) Full detailed comments before a function on how it works b) Very little/no comments inside the function c) A more powerful macro scheme for some cases.

@penguin42

I'll throw out mine as well: https://github.com/akkartik/mu/tree/main/linux/bootstrap

Here's some more detail about how it's built: http://akkartik.name/post/four-repos

I'd like to reiterate from my original post: "If you have spent time reading somebody else's literate programs, I want to hear about your experiences!"

@paul

@akkartik @paul To me that seems quite different from @patchlore's - you seem to be declaring where chunks of code should go against some markers.

@penguin42 Yeah, the mechanic is a bit different. The goal is the same, though: to convey a program to someone else.

I chose the new mechanic because it allows me to write large programs that can be divided up into "chapters" or layers. And you can load any initial sequence of layers to end up with a simpler program that's easier to understand.

More details: http://akkartik.name/post/wart-layers

@paul

@akkartik you mention writing in a literate style is meant for others to grok the program, do you think writing a literate program helps you at all?

My Kuf program, for example. I just can't keep in my head for too long. It is a very confusing and unintuitive set of bitwise operations by the time it reaches code. Using traditional comments, I may have written enough explanation. Maybe. But the literate programming structure really allowed me to jot down my thoughts at the time in a very frictionless way. I was able to reorient myself to the program after months of being away from it.

I can also say the same for many of the DSP algorithms on sndkit.

@penguin42

@paul Oh yes! 100%.

I think an unspoken premise in all this, going all the way back to Knuth, is that any documentation system is only sustainable if it's helpful to the writer.

"This experimental skeleton will hopefully make it easier for others to understand and more malleable, easier to radically rewrite without breaking in subtle corner cases. Understandability and rewrite-friendliness are related in a virtuous cycle. Each enables the other."

https://github.com/akkartik/mu/blob/main/linux/bootstrap/000organization.cc

@penguin42

@akkartik @paul I worry that your scheme ends up with it being a bit difficult to see everything that happens at a given one of the macro tags, because that code is scattered around a lot. (Which is the same problem as a lot of the event based schemes)

@penguin42 I do occasionally need to look at the tangled output. My tooling allows for that, unlike most literate systems whose tangled output is a mess.

Newcomers need this even more. Think of the layers as just the .git representation. Once you've selected a set of layers to read, just tangle them and read the generated bootstrap.cc

But it's a mutable .git representation that I the author can easily streamline if I come up with clearer ways to explain something.

@paul

@akkartik I've got hard copies of TeX and Metafont the program on my bookshelf. It hits differently than if you were to read it as a PDF.

Nothing even resembling dharma transmission for either of these programs (maybe that's okay?) But it's enjoyable to flip around and read the prose. I wish things were written in a more modular, self contained way like PBRT.

Knuth has consistency in structure in writing, and it's actually inspired me to rethink the way I communicate about program structure:

pbat.ch/wiki/verbs_in_metafont

@penguin42

@paul I really need to maintain a list of people describing what it's like to read other people's Literate programs. I think I've had 3 people respond to my call to say yes, they have read Literate programs. I'm going to start it now and add yours.

I'm particularly interested in usability reports. What gave you the most trouble in understanding some Literate program written by someone else.

@penguin42

@akkartik Your work is so interesting! I will dedicate some time to digesting some of your writing.

@groovestomp I hope it's a good fit for people who are into uxn! Ask me all the questions you like.

@akkartik @penguin42 @paul glad to see this thread today. I'm not any sort of serious literate programmer, but I have been interested ever since I read about org babel and I have maintained my emacs config as a literate program. I don't get a lot of free time to do so, but I will give a try at reading some of your programs

@paul this is interesting shit. Why isn't this a popular thread with a lot of knowledgable people (I'm not) engaged in constructive comments? patchlore = Edward Tufte of coding? go! Maps, we need maps! As an aside, where is Bret Viktor? Any news? (a web page with background auto play videos? come on.)

@paul can we facilitate the literate programming mindset more by (a) establishing a culture whereby when you inevitably announce your new program you also blog about the development process as a story (incl. specific architectural decisions, designs tried but abandoned, etc), and (b) make sure that this trend is so normalized that people will naturally have this future blog post in mind as they're working on the thing?

@paul I'm also interested in figuring out and eliminating what makes writing feel like a chore.

Often reading can feel the same way. I've mentioned before how difficult it is to make your way through the average journal/conference/research paper. There seems to be something in the average person's head that makes them go, "I'm writing; this needs to sound flowery and smart", and it just ends up being tedious.

@paul I remember back when Yegge would drop the occasional blog post, and he'd make references halfway through like, "I promised I wouldn't spend more than 4 hours writing, and I'm already over budget".

Maybe short, autotranscribed, spoken word daily development logs would help the writing process.

Subjectively, it feels way more effortless to riff on a subject in an extemporaneous soliloquy than it does to commit to picking up a pen or banging on the keyboard.

@colby @paul
I’ve found that the difference between an average programmers blog post include the following;
- using UML.
- short bullet points for listing.
- links to more detail usually generated documentation.
- diff ing displayed using side by side diff with red ‘—-‘ and green ‘+++’
- benchmarking and other statistics shown and compared using a single unit of measurement not comparing % with fractions.

@lil5 @colby Having a literate programming environment that integrates with a blog or SSG or wiki would be really neat, and has been something I've been meaning to do more of. The LP environment I use can export tons of metadata to a SQLite file, and I've barely taken advantage of that.

Incorporating all of that into a SCM would be very neat. Fossil seems like it would be best suited for this task since it's got a suite of standard tools bundled in it, but it's probably doable (and probably more useful tbh) with git.

@colby for me: the hard part about the writing medium is structuring thought in a presentable way. In literate programming, not only do you have to concern yourself with the program structure, but also communicating that structure in a meaningful way. To do this properly takes time, revision, and patience.

While there may be bottlenecks in this process that clever software can address, I've personally found the biggest improvements are non-technical ones. In other words, creating a lifestyle that minimizes interruption, encourages good rest and health, and allows for large continuous pockets of time to sit and think.

@paul agreed on the hard parts, but I think the solution lies in the insight you just gave. A comparison:

When you have to do WYSIWYG, do you constantly switch in and out of creating the content and tweaking the formatting of what you have? Plaintext lovers know to write as plain text and then to paste into the richtext editor to do a final formatting pass. Similar thing here.

We can make writing "faster" (easier) by separating the getting-the-ideas-out part from structuring and revision.

@colby well, I say don't let your dreams be dreams. go and try it!

Sign in to participate in the conversation
post.lurk.org

Welcome to post.lurk.org, an instance for discussions around cultural freedom, experimental, new media art, net and computational culture, and things like that.

<svg xmlns="http://www.w3.org/2000/svg" id="hometownlogo" x="0px" y="0px" viewBox="25 40 50 20" width="100%" height="100%"><g><path d="M55.9,53.9H35.3c-0.7,0-1.3,0.6-1.3,1.3s0.6,1.3,1.3,1.3h20.6c0.7,0,1.3-0.6,1.3-1.3S56.6,53.9,55.9,53.9z"/><path d="M55.9,58.2H35.3c-0.7,0-1.3,0.6-1.3,1.3s0.6,1.3,1.3,1.3h20.6c0.7,0,1.3-0.6,1.3-1.3S56.6,58.2,55.9,58.2z"/><path d="M55.9,62.6H35.3c-0.7,0-1.3,0.6-1.3,1.3s0.6,1.3,1.3,1.3h20.6c0.7,0,1.3-0.6,1.3-1.3S56.6,62.6,55.9,62.6z"/><path d="M64.8,53.9c-0.7,0-1.3,0.6-1.3,1.3v8.8c0,0.7,0.6,1.3,1.3,1.3s1.3-0.6,1.3-1.3v-8.8C66,54.4,65.4,53.9,64.8,53.9z"/><path d="M60.4,53.9c-0.7,0-1.3,0.6-1.3,1.3v8.8c0,0.7,0.6,1.3,1.3,1.3s1.3-0.6,1.3-1.3v-8.8C61.6,54.4,61.1,53.9,60.4,53.9z"/><path d="M63.7,48.3c1.3-0.7,2-2.5,2-5.6c0-3.6-0.9-7.8-3.3-7.8s-3.3,4.2-3.3,7.8c0,3.1,0.7,4.9,2,5.6v2.4c0,0.7,0.6,1.3,1.3,1.3 s1.3-0.6,1.3-1.3V48.3z M62.4,37.8c0.4,0.8,0.8,2.5,0.8,4.9c0,2.5-0.5,3.4-0.8,3.4s-0.8-0.9-0.8-3.4C61.7,40.3,62.1,38.6,62.4,37.8 z"/><path d="M57,42.7c0-0.1-0.1-0.1-0.1-0.2l-3.2-4.1c-0.2-0.3-0.6-0.5-1-0.5h-1.6v-1.9c0-0.7-0.6-1.3-1.3-1.3s-1.3,0.6-1.3,1.3V38 h-3.9h-1.1h-5.2c-0.4,0-0.7,0.2-1,0.5l-3.2,4.1c0,0.1-0.1,0.1-0.1,0.2c0,0-0.1,0.1-0.1,0.1C34,43,34,43.2,34,43.3v7.4 c0,0.7,0.6,1.3,1.3,1.3h5.2h7.4h8c0.7,0,1.3-0.6,1.3-1.3v-7.4c0-0.2,0-0.3-0.1-0.4C57,42.8,57,42.8,57,42.7z M41.7,49.5h-5.2v-4.9 h10.2v4.9H41.7z M48.5,42.1l-1.2-1.6h4.8l1.2,1.6H48.5z M44.1,40.5l1.2,1.6h-7.5l1.2-1.6H44.1z M49.2,44.6h5.5v4.9h-5.5V44.6z"/></g></svg>