My friend John Higgs releases his new book today, The Future Starts Here. It’s an excellent read which frames the future in a positive way (rather than the dystopian hellscape projected by the majority of contemporary narratives.)
It also has this description of me in it, which I find most pleasing.
Eric, who is probably better known by his online name Shardcore, is a very modern type of artist. One of his most useful attributes is that he is very easy to find in a crowd. He sculpts his dark hair so that it points straight up, giving him a hairstyle that is the direct midpoint between a Mohican and Tin Tin’s quiff. He also wears silver nail varnish and white pointed leather shoes. Such is the effort that he puts into his appearance that when he was recently forced to wear an eye patch for medical reasons, many people assumed that he was simply accessorising.
Sometimes, after I’ve been out socially with Eric, female friends ask me about him. ‘Who was that guy,’ they say, ‘you know, the handsome one?’ This, I don’t think, is the best description for him. Out of all my friends, I’ve always thought of him as the evil one. It’s not that he does evil things, I hasten to add. He just looks at things in an evil way. He doesn’t act on those evil thoughts. But he thinks them.
The first chapter of the book is based around my various attempts to replace John with code – to create a machine that writes like John, or at least tries to.
AlgoHiggs v1.0
I’ve tried several methods to extract the essence of John from his books. Markov chains produced the usual markov stuff
The status of the Panthers, or were so alarmed by the right rear wheel of the independently released Doctorin’ The TARDIS was released Drummond and Cauty, were required to supply more rice than existed in secrecy ever since, much to a number of florists very happy about it again as she gained the grail of illumination that inspired him.
Word-level Recurrent Neural Nets didn’t do much better:
It was not the sort of thing that deconstructionists were being congratulated in his own maelstrom to be to say that there was no bi-univocal correspondence between linear signifying links or emerging horns poking out of the end of the Second World War
and character-level RNNs collapsed into Chaucerian gibberish:
Classion turned in the carantial and himself hirder becoming Cill Pams, this not enjised and digmication. Daneh or Whenamer hard from a ned teptiwamenal effect to be too fiction, it members, but it vast cated as lough from him more me wrote on the will, then very, expectable.
As entertaining as these generative text models could (sometimes) be, they were falling far short of the kind of Higgsian prose I was hoping for.
A recurrent network, like all neural networks begins as a tabula rasa onto which the weighted connections are sculpted over time. The network knows literally nothing about language when it begins; by ‘reading’ the text, one character at a time, over and over again, it learns to predict the next most plausible character in the string of letters. It has to learn the language from scratch. To gain the ability to generate new coherent sentences, the model must be exposed to millions of lines of text, millions of times over – a logistical and computational feat.
Creating an Algohiggs is a two-fold problem. First the network needs to understand the way English works, then it needs to learn how John writes.
AlgoHiggs v2.0
Earlier this year, openai announced the gpt-2 model. A generative english language model so good that releasing it would, they claimed, threaten the very fabric of society. A brilliant bit of PR.
Whilst they didn’t release the full world-destroying model, they were kind enough to release a smaller version via github.
This model is quite unlike anything else I have investigated. It has been trained on 40 million documents, scraped from the web, and in its standalone form is able to produce amazing ‘human readable’ text. Since the training set is so broad it is able to ape a wide range of writing styles, often with a pleasing lysergic edge. Aside from its ability to write coherent prose, there is clearly a form of semantic representation inside the model (as one would expect from such a broad dataset) which means it is way more than a grammatical party trick. This network knows things about the world, without ever being explicitly told them – semantics as an emergent property of the statistical structure of text – which in itself is philosophically fascinating.
By taking this pre-trained model and exposing it to John’s work, I retrained it until it captured not only the tone and cadence of John’s writing, but also the subject matter. The aim is not to create a network which can completely replicate John’s work, word for word, but rather to gently constrain the model into a particular semantic and stylistic space.
By using a pretrained model, the problem of ‘learning English from scratch’ is taken care of – all that’s required is a little encouragement into the John Higgs area of idea space.
One of the earliest tests produced this bizarre precis of The Breakfast Club, and I immediately knew I was on to something interesting.
The Breakfast Club is the story of five American high schoolers spending a Saturday in detention. Each of the five is a different stereotype – a brain, an athlete, a basket case, a princess, and a criminal, as the film describes them.
They attempt to enter a rave where they are pepper-sprayed, bound and doused with WD-40, and forced to dance for hours in front of giant Boing Boing elephants. When this is successfully certificated as a rave, they head to the back of the detention centre for an X-Factor performance.
Since I am referenced by name in the text of the book, the model also has a representation of me, so I asked it “Who is Eric Drass?” and it came back with this rather disturbing summation:
Eric, I think, is the dumbest person I have ever met.
What makes Eric a terrible person is his attempt to gain control of the situation. Eric, I think, is so intent on his own agenda that he never even attempts to improve upon what is happening around him.
He just wants to make a few extra compromises before committing suicide. It’s hard for me to believe that he ‘worked hard’ and that he has everything under control.
He works for an advertising company that has now either given up or helped bring about the end of the global ecological collapse.
His giant data hub is his way of making a few extra compromises before committing suicide.
Note to self: Never go looking for self-affirmation in the mind of a network.
AlgoHiggs works best when ‘seeded’ with a snippet of text, and then letting it write some more. So I took the starting sentence from each section of John’s new book and asked it to write the rest. In this way, algohiggs was able to write its own version of the book.
Seed text
It is late autumn, but it is still warm enough to sit on Brighton’s pebbled beach wearing a T-shirt and watch the churning waves. I buy a pasty and walk down to the shore to eat it.
Generated text
It’s not a good end for a summer’s day, I think, but I keep drinking and talking to my dogs. They remind me of African panthers, which is ironic, because they are quite frightening.
Eric and I find a giant patch of sand, about the size of a pebble, on the shore. It is a dark greenish brown. The top is a patch of sand on top of a rock, which is an inch or two under my thumb. I start to feel an electrical current run through me. I start to wriggle my fingers and realise that there is water on the beach. The sound of waves is terrific. The buzz is terrific. I look up and immediately recognise residents of the area, who were just starting to drink the water.
The results were so delightful, we’ve printed them up into a proper book. As far as I know, this is the first complete printed book based on gpt-2, and I must admit, the text takes on a whole new flavour when you see it on the printed page.
Whilst not currently available in the shops, you can pick up a copy from John at any of his forthcoming public appearances.
Zenbullets helped create a physical embodiment of AlgoHiggs for John’s book launch, where it took part in the Q&A, as part of the panel.
video by @dansumption