The Black Box

(This post is about how neural networks are a black box.)

I. Computers today are known quantities. They may be complex, but we understand what’s going on under the hood.

II. Programming our computers to do things is a skill and a craft that requires understanding what’s going on under the hood and expressing ideas using language. There’s creativity there, but it’s still a task of logic and computation and organization.

III. LLMs are probabilistic creatures made of ones and zeroes. They have been trained on corpuses of human words that have been digested, chewed, and turned into numbers. Many of those human words are computer programs. Thus, just as LLMs can be trained to write sonnets about cats in the style of Shakespeare, they can just as easily be trained to write computer code in the style of the best programmers we know of.

IV. What’s hugely advantageous about LLMs writing code is that we can run the code and see if it actually works or if it breaks. We’re not so much concerned with copyright issues or plagiarism with code; we just want it to work. The

I like to think of these models as living in huge distilleries. The process of making, say, whiskey, first involves

We can use Large Language Models to do practical and often amazing things very quickly, but they are no longer machines that we can claim to understand. Even with modern day automobiles that have on-board computers making countless decisions every second, we can audit and understand how those decisions get made because at some point human engineers sat down and designed and wrote the algorithms for them. With Large Language Models there is now a kind of fuzziness to the responses they give, sometimes brilliant, sometimes idiotic, depending on how you word your ask. Yes, there are dials and knobs you can fiddle with to get stricter or less conventional responses. For instance, not always choosing the most probable next word in a sequence is known as “more creativity.”

Programming computers, up to now, has been an exercise of almost pure logic and mathematical expression. This is because that is the way our computing machines work. In order to tell them to do things, we are required to feed them carefully constructed, logically airtight programs that are built on top of a binary foundation of 0s and 1s, pure logical expressions, and everything else layered on top of this foundation — more ornate operations and different ways of using those operations — follows suit.

Programming languages try to hide this abstraction from the programmer, but, as with any language, you can learn the alphabet, the grammar and the syntax, but it doesn’t mean you can actually write an essay in that language.

Writing programs is hard. Being a programmer is really a craft

Now, as programmers and programming-acquainted people like myself shift to working with LLMs and neural networks, the purity of logic and mathematics is still there, humming along underneath.

What this AI “revolution” amounts to is this: the mental model for us to interact with these neural networks can now be much more intuitive and natural. We can ask for things in a casual tone.

But the box is still dark inside. We cannot debug in the same way

We must now use creative writing to craft prompts in order to coax responses from models, and when things don’t go as we expect them to, we cannot open them up and inspect them directly. There is no longer the guarantee of reproducibility that we used to have with the machines that ran our old number-driven programs.