For the past few weeks, I have been writing an implementation of the Forth programming language. I had not written a Forth before, and did not really know how to write a Forth before setting out to write MoonForth. Even after completing the base and being able to write real programs with MoonForth, I still can't really tell you how I did it. There are a few reasons for this.
First, I did not take adequate notes of the process. Second, I don't think it would matter much if I had written notes because writing a Forth was a process so remarkably straightforward that there is little to remark about said process. What I present here is mostly from memory.
After evaluating my options and finding little in the way of good detailed walkthroughs for writing one's own Forth, I decided to follow the common recommendation around Forth circles by starting with JonesForth's source code.
JonesForth (unofficial Github mirror)
JonesForth is a Forth written in x86 assembly language. It gets the job done, and the plentiful comments scattered throughout the code help to explain the nuance of what's going on—even if I found it to occasionally be overly verbose.
Because my goal was to write a Forth in C, I knew that the assembly language of JonesForth would not directly translate for my own endeavours. I also found the interpreter and compiler a little difficult to grasp, so I simply sidestepped them to start. I thought if I made enough progress elsewhere that I would later be more motivated to figure those parts out, and probably have enough insight to make those later steps easier to take.
Helpfully for me—and indeed, part of what motivated this—is the fact that Lua internally uses a stack to pass values around between functions. In my view, the easiest ground on which to start writing a Forth atop Lua's C API was to write functions which directly manipulate said stack with the intent for these functions to serve as the implementation of certain built-in core Forth words. After a few hours, I had a handful of functions to manipulate Lua's stack. It was easy to essentially pretend that these were my first Forth words despite having no means to actually run them. In implenting important core words, I also decided to try to use my own preexisting words whenever possible. As I began writing more complicated words, my own functions would comingle with calls to Lua's own API. Even before I ran a single line of code, I was already using the bones of my Forth to further extend it.
Once the simple stack manipulation functions were in place, it became easier to write more complicated functions. Because everything that I had been writing was conceived to be as directly analogous as possible to Forth's own stack manipulation words, this also meant that even despite not having a concept for how to build a Forth atop this work I was able to apply Forth's concept of factoring, backwards. I went through JonesForth's source code, looked at the hardcoded words, observed how they worked, then simply wrote my own implementations on Lua's C API for MoonForth with the intent to figure it out later.
Eventually, it hit me.
C functions can be made into pointers, and pointers can be pushed to Lua as user data. Because Forth words are just a series of other words, I could store a custom word definition as a table of pointers to C functions. Because Lua tables are always references, I could also simply store tables of other word definitions into further words that use them. Another benefit to approaching compilation this way was that word redefinition could work as in most Forth systems, where a new word that shadows an old one does not change any compiled word that used the old word. As a neat trick as well, I should easily be able to store string and number literals in word definitions. The interpreter would simply need to read the type of each item stored in a word definition to determine what to do with it.
It was at this point that I began to fear—because code becomes very easy to write once you have both the data structure figured out and the means to actually interact with that data structure—that I might actually pull it off.
Despite being a somewhat frustrating endeavour of trial-and-error, it was not difficult to write the interpreter. When it inevitably did not work, I simply needed to trace the program's execution with print statements (because that's the type of developer I am) and knock out faults one by one. Three work days after deciding to try writing a Forth atop Lua's C API, I had run my first line of code.
The compiler arrived shortly after. It did not take long to get it and the interpreter aligned, working together off the same understanding of the world. Having already had an idea of the concept of immeidate words, those too were quite easy to pull off.
I then looked at JonesForth's implementation of flow control. Like the coolest Forths, JonesForth implements flow control directly in Forth from simpler words. MoonForth is no exception. JonesForth's code for this wasn't even especially applicable MoonForth due to not having direct access to word definitions. That obstacle was quite easy to solve by simply writing the words that would allow for branch instructions to be backfilled later, the same way other Forths implement flow control anyway. The implementation might have been very different from JonesForth, but the result was that MoonForth now had if/else/then statements as well as loops. It was at this point that I had closed the browser tabs with JonesForth's source code. There was no longer any need for guidance. Within a week, I'd gone from not knowing how to write a Forth to implementing core language features in my own Forth.
One of the most famous and widely-circulated articles on Forth to be written within this millennium is Dave Gauer's article on the history of Forth.
Forth: The programming language that writes itself
In this retrospective, Dave asserts that Forth—by its nature—is a programming language which "writes itself". I did not initially understand what this assertion meant at the time I first read this article. I understand now. Simply having a vague notion of how a Forth is supposed to work, anyone who can write code should be able to construct a Forth quite easily. What needs to be built next is always in view, and is often prompted simply by noticing what one's own Forth is missing. Discovering these gaps is as simple as trying to write a program only to find something that prevents forward progress. If you know the problem you have, and you know what a solution would look like, and you know how to manipulate the source data into the desired result, then programming the result is a simple matter of extending what you already have. When the programming language is small and you've written it yourself, that is a delightfully easy task.
MoonForth is not the first attempt to bring Forth to Lua.
Eduardo Ochs wrote about how to bootstrap a Forth from Lua in 2008.
Bootstrapping a Forth in 40 lines of Lua code
At a glance, there are also a few implementations of Forth in Lua floating around online.
MoonForth is not written in Lua. Rather, it is written in C and uses Lua's C API to accomplish its goal. Still, MoonForth is not the first of its kind—there exists another project named Luarth seemingly meant to accomplish the same goal.
I'd have used this implementation and forgone writing MoonForth entirely were Luarth not incomplete—there is no way to define custom words, and other core language features are also missing.
A significant way that MoonForth diverges from Luarth is that the latter implements its own built-in words as full Lua functions. Lua functions are not able to manipulate the stack due to how Lua handles its own calls. MoonForth, on the other hand, implements built-in words as plain C functions, and calls them by simply dereferencing pointers. Because the Lua call stack is not involved at all (except when calling into Lua), MoonForth's words can freely manipulate the stack without interfering with the internal Lua state. I suspect this may be a reason why my own project was able to proceed while Luarth did not. Either way, it was an inspiration—MoonForth's own lua-call word functions identically to Luarth's, for instance.
It is a commonly-cited meme among Forthers that Forthers tend to write their own Forths instead of their own. Another quote that goes around is that writing one's own Forth is like trying to understand an animal only through dissection—that Forth is a living programming system that must be used to be understood. Among especially seasoned Forthers, the reflex to write one's own Forth instead of learning any of the existing Forths is a practice that tends to be derided.
The reason to write MoonForth was to have a Forth which can easily interface with a high-level language I know and understand well enough while itself not being built on that high-level language. This architechtural decision had less to do with performance concerns than with conceptual ones. I thought there was no point in writing apps using Lua libraries in a language that itself was simply built on Lua—at that point, why not just use Lua?
As for why I would want to build a Forth…
Well.
Do you work in the tech industry? I don't, but like many other people who have been forced beyond its periphery I cannot abide the creep of fashslop into the software stack.
Even though many Forthers themselves have succumbed to viberotting, Forth has an interesting place in conversations about breaking free of dependence on unethical software providers. Because Forths can be built so easily by singular developers with little tooling, it effectively provides an easy way to experiment with alternative computer ecosystems. It is from this context that I've been especially interested in Forth, though without a Forth which could interface with a higher-level programming environment that I usually work in I could never feel comfortable in it. MoonForth has thus been an invaluable opportunity in learning how to implement a Forth, how to solve problems with Forth, and of course how to use Lua's C API.
As for why to stick with Lua instead of diving deeper into the software stack, it is because I am confident that Lua is likely to remain relatively unscathed by the scourge of clod code. Lua's maintainers have a long history of rejecting most submitted patches even for desirable functionality, preferring to do everything themselves. Lua has also spent a long time not growing much in size, and having little outside pressure to change. These are circumstances in which a maintainer or lead developer is unlikely to feel the need to turn to a chatbot to handle making large changes, because Lua has always been allergic to the concept. What was once an invaluable trait that made the language easy for me to learn has become a very compelling reason to stick close to the entire ecosystem.
I trust my read on the situation is right. If not, I might simply move to a no-longer supported version of Lua until this thing blows over.
Another reason to build a Forth on Lua is portability. Forths tend to be built for specific hardware in order to drive their quirks. MoonForth's singular dependency is Lua itself—therefore it should be available wherever one can find Lua. That alone is a significant differentiator compared to more traditional Forths. I've also made sure to make MoonForth compatible with Lua 5.1, which opens up interoperability with many alternative implementations of the language such as LuaJIT.
It works. Real programs can be written in it. I have managed to use MoonForth to write a simple GNOME app as well as a proof-of-concept program for the LÖVE game engine. What I haven't done is write anything serious in MoonForth. Now that I've documented its early development process for posterity, the next step is to actually work with it. If I can't get real software working though, this will have all been for naught. It will have been pointless. Let's see if I'm up to the task.