The things I haven't wrote, and the things I want to write (back to Ruby
On getting back to writing technical texts after three months of silence
For three months already, I haven’t published anything on my blog/Substack. In truth, I have a few drafts/half-written posts/unfinished projects to share, but neither of them felt like being worth completing. The amount of duties my service requires has changed recently, too, leaving me much less spare time to write. But if I am being honest with myself, that’s not the primary reason for the silence.
This text is written as some sort of personal reflection on what and why I want to write. It also serves as a public commitment for future texts (and a way to break writer’s block!).
In the upcoming weeks/months, I intend to write a series of posts on various aspects of Ruby’s evolution and the programming language’s design decisions that can be seen in it. More on this upcoming series by the end of this post; before that, I want to talk about some other things.
Before everything else: The war
The Russian invasion of Ukraine continues, but many of you probably haven’t heard/thought of it in a while. Like, “yeah, some region where some unpleasant things happen, but there are many of those.”
Please read this attentively before you proceed. And please don’t proceed if you don’t read or don’t want to concern yourself.
Foreign news outlets don’t even make big stories anymore of what happens here. Like deliberate targeting of the entire huge city’s electric grid in one strike (that’s what happened to my home city where my family still lives) or usage of cluster munitions to strike the sea-side city far from the frontlines, or full obliterating of smaller towns, turning them into unlivable rubble. Or shooting prisoners of war, or stealing, renaming, and re-educating children on occupied territories, or all of those other things that russians do daily.
There is a genocide happening right now. And just waiting to expand, not to “stalemate and be somehow settled, let’s move on.” Most Western countries are complicit—save for very few that show unwavering support. Others seem to still believe in half-measures, appeasement, and “waiting it out,” sending only so many weapons so Ukraine wouldn’t lose too quickly (and so that russians have time to adapt to them), or not sending any; blocking Ukraine’s borders under transparently false pretenses; continuing to trade with Russia or with its trade proxies, pretending it doesn’t happen; and so on, and so forth.
Yes, in the last few weeks, it looks a bit like the world started to wake up with all the sudden initiatives to buy ammo for Ukraine and the US Senate finally voting for the Ukraine help bill. The same bill that was blocked for six months for no good reason—which led to the loss of numerous lives, Avdiivka fortress, and strategic initiative—and now Russians are advancing. Which would be held against us (already is, say, by your shitty king Elon): “As we predicted, Russia is unstoppable, so let’s abandon Ukraine and not prolong the agony.”
Just a few days ago, russia started a new offense in my home region; we don’t know yet what the next day will bring. That’s the price of our “allies” slacking!
It is honestly hard to maintain a technical writing streak instead of constant screaming in the face of the world (which, I am afraid, my formerly technical Twitter eventually turned into).
So, I’ll be open here: one of my important reasons to write is to be heard as Ukrainian. That’s also, probably, the main reason I am trying to get back on track of regular writing instead of taking some hiatus “till the better times.”
But I promise my technical writing will not be some half-assed random musings for the sake of “publishing something by Ukrainian.” Never was able to write texts just to fill the space in the schedule. (One might guess so by the irregularity of my posting through years!) I always needed deeper reasons.
How I write and what I write
One of the problems I was always struggling with was not enough attention and will to write small and regular “what I thought about today”/”that thing that I know and suddenly wanted to share” entries aka the classic blog posts.
Though, lately, I am planning to develop a habit of writing down quick insights for my Buy Me A Coffee. Not sure how it will go, but I intend to give it a try.
I did some standalone small-ish posts in my blog over the years; did a lot of “blogpost”-sized Twitter threads or Reddit answers. But in general, every time I really wanted to write something to share publicly, I felt like I should have a steady perspective and try to provide a lot of value: dig for unexpected insight, for new optics, for a system of thinking about larger problems.
With blogging/mailing list, this writing habit turns almost any attempt into either a multi-page, baroquely-structured, “too many ideas in one text” monster; or into a series (which, as we know, tend to grow during writing).
Even for the current one—which was planned as just a quick status update/clean slate restart—I now find myself moving around concepts, inventing sub-headers, throwing away faulty paragraphs, and whatnot. (And, truth be told, the initial date in the header was somewhere at the beginning of April.)
This was the same with my literature writing: for the last decade of the time I considered myself a poet, I always worked in big “cycles,” and now, with prose, it always novels with an intricate structure.
That’s… not the most efficient way of wooing the audience. Sometimes, it succeeds; most of the time, the impact and response are minuscule (if not negative) compared to the amount of work put into the text.
But that’s how I am.
So, when, a few years ago, I created a mailing list and tried to commit myself to regular writing, I was thinking about two big topics to go through: the programmable access to common sense knowledge and the programming/coding itself—as a kindred activity to writing texts.
The former, I eventually got unstuck from. The latter, I continue to think about.
The abandoned track: Common-sense knowledge
I’ve already told the story of “Reality” and related projects multiple times, but to reiterate: it was a bunch of efforts to see if there could be a generic approach to the world’s data (represented by Wikipedia, Wikidata, OpenStreetMap, and others) to create a “natural” API so that trivial knowledge can be fetched programmatically in any necessary context.
I would call it my “naive period”: That was a pre-LLM passion of mine, and what I looked for there was some feeling of understanding the world via modeling it. Not “a class per object” modeling (though probably developers of my generation still feel their inner child yearning for something like that), but looking for a minimal and solid set of core concepts via wrapping known services— I don’t know, something in this idea gave me a thrill.
I learned a lot (mostly in not-really-transferable knowledge, like a deep and intimate acquaintance with MediaWiki quirks) and had a lot of fun; while eventually, I stopped digging this hole, I contemplated reflecting upon this period and experience in some entertaining post, or a couple (maximum!).
But, as it happens, what was planned as a quick farewell article eventually grew into a potential start of new series; and that start alone was so naively sounding that it caused a torrent of patronizing and condescending comments on Reddit— And also made me re-evaluate the story I was trying to tell.
I probably won’t pursue that anymore. The experience says that it mostly falls into the “doomed from the beginning” category of texts: those who didn’t spend a lot thinking about similar problems aren’t that interested; those who did frown at naivete/shallowness.
The abandoned track: text-handling libraries
A less naive branch on this track of thinking was the class of “classic” natural text processing libraries/software that had—for me—that vibe of “common knowledge locked into some code and custom data formats.”
There was a feeling—again!—of some fundamental “models” being concealed there, and by understanding these codebases and formats, to share them with others as code and explanatory texts, I could’ve—again—extract some valuable insights.
There were several attempts of “understanding through rewriting” projects I underwent for some of the most popular and commonly used ones, eventually stopping at various stages of those projects for the same reason.
The spellchecker. Looking into Hunspell, I stopped only after the full working port of it into Python (that took a year) and a series of explanatory articles (another six months). At the beginning of the project, I imagined it to become a long-term commitment: once one of the most used spellcheckers is ported into a high-level self-explanatory code, others (like aspell, voikko, morfologik) should be added too, and some common repository of “spellchecker wisdom” written with same high-level abstractions might bring new fruits.
Eventually, by the end of the project, I understood that there was no interesting, coherent “model” I’d uncovered but a sequence of elaborate heuristics (the final article of the series explains this in great detail).
The grammar checker. For a long time, I was toying with the idea of an exploratory rewrite of the LanguageTool—an open-source grammar checker that powers, say, Open/LibreOffice. (At some point, I even ported the dictionary/word analysis library they use, morfologik, into Ruby—that was even before my Hunspell study—it was quite educational!)
This tool is written in Java and heuristics-based, too (it is more or less evident from the codebase of a particular language implementation, consisting of many XML-encoded rules and small amounts of code with boilerplate/hand-coded rules). I felt like it might be a fine exercise in using Ruby’s expressiveness powers to model this heuristic analysis in a way that would make it easy to understand and port into any other language—potentially, bringing the power of a mature grammar-checking ruleset into any ecosystem that would want it.
Eventually, two reasons made me abandon the project: first, current LanguageTool maintainers seem to try supporting further work on it by monetization (judging by the current state of the project’s public site), and I felt like an attempt to port it “away” from their control, while legally possible, might be morally murky. The second and more global one is that every time I returned to the idea and tried to test LanguageTool’s grammar-checking prowess, it felt sorrily short of Grammarly1 and, recently, ChatGPT. I was basically afraid that by the time I had something to show, nobody would be interested in the inner workings of the “old generation” of tools, those relying on hand-coded heuristics.
I don’t want to say LanguageTool is useless, and all of the development ingenuity spent on it is in vain. What I am saying is that trying to understand those tools doesn’t satisfy the urge to “understand some domain through modeling it,” and the heuristic approach to solving complicated problems today gives a strong vibe of belonging to bygone times. All in all, it is just the same databases of statistically gathered experience that ML models are based on; just in the case of those “older” projects, the experience was gathered and encoded manually.
Porting heuristics and ways to interpret them is fun (the “rewrite LanguageTool in Ruby” is still tempting despite everything said above! I almost literally slap my hands from the involvement in this old dream), but currently, it rather feels like a dead-end project2.
Stemmer. A small last attempt to dig into something of the classics was me toying with the Snowball. You might or might not have heard of it, but it is the default stemming solution in many open-source systems that need a reliable search, like PostgreSQL or ElasticSearch.
During the last months of winter, I tried to give its due to that long-term hobby of mine and try understanding (again, via debugging/porting/rewriting) the Snowball’s peculiar specialized programming language. Here is the tombstone of this failed attempt: while I managed to become fluent in this weird language and manually rewrite a couple of stemmers in Ruby, I quickly (by my standards, that is—after a few weeks’ evenings spent on the project) understood there would be no interesting story to tell about it.
Again, there was no deep linguistic knowledge hidden in those quirky .sbl files: just a bunch of heuristics deduced on large corpora, and no lesson to learn other than “well, there was a bunch of heuristics written in a peculiar way; I invented another way (probably as peculiar) to do so.”
So, what I am saying3—
For a long time, I was fascinated by the prospects of analytically modelling some of the human knowledge. I recognize now that most of useful achievements in this area seem to be reached (at least at the current state of our knowledge) rather by a statistical modeling.
And my interests are staying in the “humane” part of the software developments, of what we can write ourselves, and how we do it; what is achieved by writing (code and texts) for humans.
So, I renamed my Substack (it was “On lucid code and open data” initially) to shorter and humbler version. That’s what I am writing about, after all.
The main focus
My “maybe to write, some time” list has many entries.
Some have been sitting in the list for years (and I still feel them deserving my attention), some are pretty recent “what if we talk about that.” Some are vague “generic directions to look into,” some are pretty well-formed topics, many are stray ideas for separate articles of moderate size (though, knowing myself, they aren’t safe from growing into series uncontrollably).
To get a glimpse, here are some things that I’d probably cover if I had a few free months just to write:
Some thoughts concise code writing approaches, and on balance between “can be seen in one glance” vs. “doesn’t require years of training to interpret” (a lot of thoughts here are inspired by studying less mainstream approaches to coding, like APL—here is an old Twitter thread about it);
Many insightful ways of thinking about the code as we think about texts, applying rules, approaches, and intuitions: what we might find out about code layout, review/editing, reading, and structuring while looking at it as a “text for humans.” There are many angles here, from very pragmatic linters/formatters reassessment to topics as humanitarian as an intersection of our outlooks on programming and poetry—still might lead to interesting changes in outlook;
My approach to testing, “behavior/test driven development,” tools and DSLs for it, and why I consider that a common truth about “tests should be dumb” is not always applicable (and how I successfully help maintain huge codebases using my points of view);
A ubiquitous “framework fallacy” we are all susceptible to (both on the coding level and thinking level), and what might be the alternatives;
A comparative analysis of many small design decisions made throughout the programming languages and their libraries, and how approaches to those decisions differ in different communities, and even in the same one: on many ways to design an HTTP client or generic date/time utilities, or a testing library; on ways to document and compare design spaces and design decisions across the industry which we frequently lack;
And, of course, many angles of looking at Ruby, its roots, possibilities, and ways of writing code in it—from (one more) text on what I consider a suitable application of “functional” ideas in the language to a (healthy) comparison with Python to understand how the deep internal differences of two languages (similarly-looking and considered “close relatives” by many in the industry) lead to divergence in programming approaches;
…and… I can continue for a long time.
This might seem eclectic (and before not so long ago, I would be first to think that: what is this I am so interested in if my mind wonders on topics only tangentially related to each other and my day job).
However, there is a common theme here: the ways of writing code for humans to emphasize understanding and pass insights about the problem at hand. I called this approach—not really an approach, a way of looking at coding—“lucid code,” meaning the quality of it that exposes intents, goals, and views. (Another fitting name would be “humane code,” but it is kinda taken—though it is used to speak of similar ideas.)
The Ruby language—which I cherish, and frequently write about, and very frequently write in—is important but not the center of this effort. It is rather a tool of thinking, including “meta-level thinking”: not only “how to write something in particular” but also how we approach writing and reading code. And what languages we create and how we use them.
From this perspective, the evolution and design decisions of a particular programming language is an incredibly interesting topic. The way it changed with years, what was exposed as an unmovable core and what became legacy; the way it adapts to the new challenges or opposes them.
Coincidentally, it is also a theme by which I am most well-known in the Ruby community: the “Ruby Changes” project that I have maintained for five years already, and that attempts to cover the reasoning and consequences behind each language version changes.
Hence my choice of the next track of writing.
The series and the future book (hopefully)
So what I want to do now is to create a series of texts on (you guessed it!) Ruby evolution.
Unlike the site linked above, I want to try a depth-first approach: take core elements of the language one by one—methods, objects, classes, base data types, collections, etc., etc.—and see how they were changing through time; what needs stood behind each change, and what consequences it brought (which sometimes becomes evident only many versions later).
In other words, I am planning to follow the overall structure of the current “Ruby Evolution” page on the Ruby Changes site; but cover every significant element of the language with a thoughtful look into how it all happened, not just a matter-of-factly description of the change. Somewhat like my autumn series on “useless Ruby sugar” but with a more historical perspective. (I actually toyed with the idea of the “Ruby Evolution” book by the end of those series and received a bunch of encouraging responses, so here we are!)
I really believe/hope that such a walkthrough of the language history might be interesting for Rubyists and will bring some new perspective on what we have and how it is better to use it. I also really hope that looking into this history of one “system of writing” evolved might be entertaining for users of other languages, and allow them to reflect deeper on the design decisions and tradeoffs of their tool of choice.
That’s the plan, anyway. At this point, I can imagine many possible obstacles, both internal and external (well, there is still a war, you know), in the way of this plan. But as far as human planning goes, I intend to write a “chapter” every two-three weeks, and I estimate the series/book to be around ten chapters (maybe more, maybe less, we’ll see how it goes).
See you next time, hopefully, earlier than in three months!
Thank you for reading. Please support Ukraine with your donations and lobbying for military and humanitarian help. Here, you’ll find a comprehensive information source and many links to state and private funds accepting donations.
If you don’t have time to process it all, donating to Come Back Alive foundation is always a good choice.
If you’ve found the post (or some of my previous work) useful, I have a Buy Me A Coffee account now. Till the end of the war, 100% of payments to it (if any) would be spent on my or my brothers’ necessary equipment or sent to one of the funds above.
I am not sure which algorithm Grammarly uses. Rumors from “people who once worked with people who…” said it is part heuristics, part machine learning, with heuristics being much more efficient; but those rumors are several years old, and I basically assume that the role of machine learning is significant in their product. Or they have very, very good machine linguists to maintain their heuristics.
The LanguageTool main page currently dubs it an “AI-based spelling, style, and grammar checker.” To the best of my understanding, all of the grammar checking is still performed with those heuristic rules, while a separate “paraphrase” LLM (?)-based tool is experimented with to catch up with the trends. Anyway, the open source part that I was interested in seems not to have changed much over the years I was contemplating the idea.
I am sorry this section takes so long… It was initially planned as a standalone article (or series?.. You never know) to the general meaning of “what I know about the classic common tools of natural text processing,” but then I couldn’t decide who would be the target audience— so, I just tried to integrate it into this “history of my interests” narrative.