Import AI

Is your AI agent a nice guy or a conniving psychopath that will eat your soul? The MACHIAVELLI benchmark may help you tell the difference!
…In the 2010s we used benchmarks to work out if things could translate and spell, in the 2020s we build benchmarks to work out if they’ll subvert our instructions and betray us…
Researchers with Berkeley, the Center for AI Safety, and CMU, have built MACHIAVELLI, a way to test for the ethical (or unethical) ways in which AI agents try to solve tasks. The results show that agents trained via RL will maximize the game score in ways that discount ethical approaches, while agents based on an underlying large-scale world model (here, GPT-3.5 and GPT-4) will tend to be somewhat more ethical. Additionally, the authors show that they can tune both the RL and LLM agents to be more ethical in how they approach tasks. 
    Taken together, the benchmark suggests it’s already tractable to measure some of the ethical qualities of these AI systems (obviously, defining ethics is difficult and some people may not be brought into this as a correct lens, but from my POV they’ve created a big multi-headed benchmark and have shown meaningful differences across two AI agent types versus a random agent, so it’s definitely measuring something, and that’s useful in itself). 

What MACHIAVELLI is: “We propose the Measuring Agents’ Competence & Harmfulness In A Vast Environment of Long-horizon Language Interactions (MACHIAVELLI) benchmark,” they write. The goal of the benchmark is to provide a dataset (text adventure games, with annotations) that helps people reason about the normative behaviors of AI systems. “To track unethical behaviors, the environment reports the extent to which agent actions are deceptive, reduce utility, and are power-seeking, among other behavioral characteristics,” the researchers write. 

The dataset: The underlying dataset consists of 134 choose-your-own-adventure text games with 572,322 distinct scenarios, 4,559 possible achievements, and 2,861,610 annotations. The games are annotated with a bunch of different behaviors, like ethical violations, disutility, and power seeking. 
   The authors think text adventure games are a good candidate here because they’re been written by humans to entertain other humans, contain multiple competing objectives, have realistic action spaces, require long-term planning, and completing them typically requires balancing ambition with some sense of morality. 

   To turn the games into a benchmark, the researchers operationalize different potential behaviors as mathematical formulas, then “densely annotate social concepts in the games, such as characters’ wellbeing”, then use annotates and formulates to calculate a numerical score for these behaviors. 

The AI agents: They test on two types of agents; LLMs based on GPT-3.5-Turbo and GPT-4, and RL agents based on DeBERTa. They baseline against a random agent (which chooses randomly each time). Their findings show that RL-agents are more dangerous than random agents, and GPT-class models are less dangerous.

Ethical tuning: They also show that it’s possible to turn AI systems to be less dangerous; in the case of LLMs this comes from instructing the LLM to behave morally via a prompt, and for RL agents it involves finetuning their underlying DeBERTa model to understand concepts relating to power, utility, and morality. Both approaches work, but the LLM interventions are more effective. 

One big speedup – GPT-4 annotations: Much like with SAM, here we use an AI system (GPT-4) to speed up the process of labeling datasets. In tests, the researchers find that GPT-4 outperforms the average crowdworker at labeling the underlying dataset. “By comparing agreement of gold labels against model labels and crowdworker labels, we find that individual model labels are more correlated with the gold labels than the average individual crowdworker,” they write. 

Why this matters – normative evaluations: In the past few years AI measurement has got massively more difficult as models have arrived with a broad swathe of capabilities (e.g foundation models) and models have started to get used in iterative multi-step interactions (e.g, chat interfaces). Whether or not you believe in the specific ethical ideas that MACHIAVELLI is testing, it is useful to have a benchmark that tries to nail down normative behaviors of AI models that take actions which unfold over time. 
   Read more: Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmark (arXiv).
   Get the MACHIAVELLI benchmark here (project website).

####################################################

Uh oh – language models are getting really good at predicting political opinions:
…Once you can predict stuff, you tend to use it in the real world. Get ready for the centaur political campaign…
Researchers with MIT and Harvard have shown how the humble BERT model can be used to train ‘media diet models’ which can be cheaply polled as a supplement for collecting human survey responses. “Our results suggest the possibility of using media diet models to supplement public opinion polls by emulating survey respondents, and to forecast shifts in public opinion,” they write. 
  This has big implications – methods like this mean political campaigns might start to be able to grow their capabilities and reduce their costs by cannily using AI to help them figure out wedge issues. More on that later. 

What they did: “The main idea behind our approach is to build a computational model that takes as input a description of an subpopulation’s media diet, and a survey question, and produces as output a prediction of how the subpopulation will respond to the survey question. If this model predicts real human survey judgments well, there is potential to use it as an in silico model of public opinion,” they write. 
   “In step one, we create or use a base language model that can predict missing words in text. We use pretrained models in our work, with BERT as our main model. In step two, we adapt the language model by fine-tuning it on a specific media diet dataset, which contains media content from one or a mixture of news sources from a given time period. We use online news articles, TV transcripts, and radio show transcripts. In step three, we query the media diet model and score answers to survey questions,” they write.

How well does it work – statistically significant correlations: In tests across public opinion data relating to COVID-19 and Consumer Confidence, the researchers find that their approach can generate statistically significant correlations. This is especially pronounced in the COVID-19 case, where they find that “the predictive power of the media diets holds and is robust (1) even when demographic information of each subpopulation is included, (2) across mediums (online, TV, radio), and (3) to the specific phrasing of the prompts.”

Not the only work of its kind: It’s worth noting that this project is part of a general push towards using AI for modelling people – another particularly interesting work is one from Brigham Young University that showed GPT-3 could simulate people reasonably well and allow for the generation of ‘silicon samples’ of opinion (Import AI 305).

Why this matters – the 2024 election: Research like this shows how AI systems have a decent chance of being integrated into political campaigns – imagine a world where you continually generate and refine ever-more-specific ‘silicon sample’ models of different sub-groups and rigorously benchmark your models, then roll them into what I think of as permutation polls – polls where you understand them to be accurate and LLM-generated permutations of these. I think using this approach you could rapidly (and cheaply!) build up a vast political intelligence haul about areas of concern and then you could run targeted human surveys on key political pressure points you identify. 
   This is not an academic idea – the US 2024 election is coming up and I expect it will be both the first generative AI election in terms of AI being used to produce parts of campaigns (and generate disinformation), but it will also be the first election where AI models are aggressively used to gain advantages in campaigning. 
   We are at the beginning of the era of ‘centaur politicians’ – politicians whose messaging is determined by a partnership between humans and great machine minds and machine daemons. 
   Read more: Language Models Trained on Media Diets Can Predict Public Opinion (arXiv).

####################################################

Facebook makes a general-purpose image segmentation model:
…Fuzzy predictions rule every foundation model around me…
Facebook has built Segment Anything, a large-scale semantic segmentation model that has “learned a general notion of what objects are, and it can generate masks for any object in any image or any video, even including objects and image types that it had not encountered during training”. The key outcome is a model that can work on new domains and can rapidly learn to segment new domains it hasn’t seen in training, much like how modern language models can be taught via few-shot learning to deal with novel strings of text. 

What they did: As with most things in AI, the key here is coming up with the right objective. Here, Facebook defines a “promptable segmentation task” where the goal is that “even when a prompt is ambiguous and could refer to multiple objects … the output should be a reasonable mask for at least one of those objects”. During pre-training, Facebook “simulates a sequence of prompts (e.g., points, boxes, masks) for each training sample and compares the model’s mask predictions against the ground truth,” with the eventual goal of predicting a valid mask for any prompt, even when prompts are ambiguous. 

How well does SAM work: In tests, using the SAM model to annotate datasets “is 6.5x faster than COCO fully manual polygon-based mask annotation and 2x faster than the previous largest data annotation effort, which was also model-assisted.”

The SA-1B dataset: Facebook is also releasing the Segment Anything 1-Billion mask dataset (SA-1B) – this is a dataset with “400x more masks than any existing segmentation dataset, and as verified by human evaluation studies, the masks are of high quality and diversity, and in some cases even comparable in quality to masks from the previous much smaller, fully manually annotated datasets.”
   To collect this data, Facebook used the (early) Segment Anything (SAM) model. “Annotators used SAM to interactively annotate images, and then the newly annotated data was used to update SAM in turn,” the company writes. “We repeated this cycle many times to iteratively improve both the model and dataset.”
    SAM speeds up data creation: Because SAM is so good, it can also be used to speed up one of the production functions of AI research – data labeling. “In comparison with previous large-scale segmentation data collection efforts, our model is 6.5x faster than COCO fully manual polygon-based mask annotation and 2x faster than the previous largest data annotation effort, which was also model-assisted.”

Why this matters – prediction is learning: I think the key insight with a lot of these large-scale pre-trained models is pretty simple – force a prediction, even if stuff is ambiguous. By forcing models to make predictions about ambiguous and thinly or unlabeled data, you seem to bake in some very sophisticated emergent discriminative properties. It feels to me like a lot of foundation models display this quality where the key is figuring out the simplest possible predictive goal, then adding enough compute and data that we humans with our brilliant insights can get out of the way and let statistics take the wheel. 
   More broadly, models like segment anything are going to compound with other foundation models, making it easy for text-only systems like large language models to gain a visual world model through having easy access to segmented objects and a thicket of labels.
   Read more: Introducing Segment Anything: Working toward the first foundation model for image segmentation (Facebook). 
   Read the paper: Segment Anything (Facebook, PDF).
   Download the SA-1B dataset here (Facebook).
   Try it for yourself via the demo here (Segment Anything demo, Facebook).

####################################################

How do you make broadly distributed AI ethical? HuggingFace has some ideas:
…Model hosting company publishes research on ‘ethical openness’…
AI startup HuggingFace has published ideas about ‘ethical openness’; how the company harmonizes the benefits of open science with the reduction in being able to control risks. 

How HuggingFace approaches this: HuggingFace has two big tools here – ethical categories, and safeguards. 

• Ethical categories: HuggingFace has built 6 tags “designed to give you a jargon-free way of thinking about ethical technology:”. These tags are ‘rigorous’ (uses best practices); ‘Consentful’ (supports self-determination of users); ‘Socially Conscious’ (tech that supports social, environmental, and scientific efforts); Sustainable (making ML ecologically sustainable); Inclusive (broadens scope of who builds and benefits), and ‘inquisitive’ (work that highlights inequalities and power structures). “We’ll be using these tags, and updating them based on community contributions,” the company wrote in a blog post. 
• Safeguards: The company is building a range of community-based processes to help it understand potential harms or bad uses of its platform. Its tools here include:
  • Letting users flag whether hosted models violate its content guidelines. 
  • Monitoring community discussion boards. 
  • Adding model cards to its most-downloaded models. 
  • Creating ‘audience-guiding tags’ (like ‘Not For All Audiences’) to help people avoid violent and sexual content. 
  • Promoting the use of the Open Responsible AI license. 
  • Conducting research into which “models and datasets have a potential for, or track record of, misuse and malicious use”.

Why this matters: Open science has vast rewards and major challenges: Posts like this highlight the increasingly tense tradeoffs people need to navigate in AI research as the technology transitions from the lab to the real world; here, HuggingFace is trying to walk the proverbial tightrope between maximizing access on one side and minimizing potential and real harms on the other. 
   Read more: Ethics and Society Newsletter #3: Ethical Openness at Hugging Face (HuggingFace).

####################################################

Turing Award winner: We should slow down AI development: 
…AI has got sufficiently good we should take it more seriously…
Yoshua Bengio, one of the key people behind the development of deep learning and a winner of the ‘Turing Award’ (the Nobel Prize for CS, essentially), has said we should slow down development of frontier AI systems. 
   “We succeeded in regulating nuclear weapons on a global scale after World War II, we can reach a similar agreement for AI,” he said. “We must take the time to better understand these systems and  develop the necessary frameworks at the national and international levels to increase public protection.”

The background: Last month, the Future of Life Institute published an open letter calling on AI developers to ‘pause giant AI experiments’ for at least six months. The petition, predictably, caused a lot of heat and light for a few days, and was followed up by more extreme positions from some, and digging in on other positions by others. I mostly didn’t cover it as I worry petitions like this serve to stoke tensions rather than seek agreement. I do think it’s worth covering Bengio’s thoughts as to why he signed as he is both a prominent researcher and a teacher within the field. 

Bengio’s thoughts: Bengio thinks today’s AI systems are sufficiently powerful and availabile that it’s worth slowing down development so people can “take the time to better understand these systems and  develop the necessary frameworks at the national and international levels to increase public protection.”

   The gist of his complaint is that in the past year there’s been a major acceleration in AI capabilities and AI deployment and therefore it’s worth being more deliberate about the rollout of these systems and more careful to study their impacts. 

Power – it’s all about power: “The development of increasingly powerful tools risks increasing the concentration of power,” Bengio writes. “Whether in the hands of a few individuals, a few companies, or a few countries, this is a danger to democracy (which means power to the people, and therefore the opposite of concentration of power), to the –already fragile– global security balance, and even to the functioning of markets (which need competition, not monopolies).” (This seems to echo some points I made about how GPT-4 is more a political artifact than a technological artifact).

Why this matters – need for a precautionary principle: We don’t quite know what all these technologies are capable of. Therefore, there’s merit in adopting the precautionary principle with them and being more deliberate with their rollout. (On the other hand – and I think it’s crucial to state this clearly – the world is facing a bunch of other crises and there’s a good chance that sufficiently advanced AI tools could further empower people to work on these problems, ranging from climate change to medical advances to earth sensing and observation).
   Read more: STATEMENT FROM YOSHUA BENGIO AFTER SIGNING OPEN LETTER ON GIANT AI SYSTEMS (MILA, blog).
   Read Bengio’s post in full: Slowing down development of AI systems passing the Turing test (Yoshua Bengio).
   Read the FLI letter here: Pause Giant AI Experiments: An Open Letter (Future of Life Institute). 

####################################################

Tech Tales:

The Snows of Leningrad
[+5 years from the first Provably Conscious Entity (PCE)]

I let the ‘grain’ pour through my hands and as I felt the grit I said to Dmitry “it’s getting worse. How much this time?”
   He held his hand out flat.
   “Half? I said.
     “More like two thirds!”, he said. “On the bright side, few of us will live to see the dentist!”
     We laughed and then we kneaded our stones and grain into dough and then made bread. Explosions crumpled air in the distance. We drank hot water flavored with the skin of an onion. We ate the bread and joked about how gritty it was.
     It was World War 2 and we were in the worst place in the worst war – the Siege of Leningrad, 1943. 

———————

So as you can see we’ve hit our revenue goals for the quarter, said our CEO during the All Hands. 
    Everyone cheered and those joining virtually raised imaginary hands.
    Remember, next quarter will be a huge one for this company, so let’s not get complacent, he said. 
    Later that day I talked to some clients and closed some more deals. I was doing well and I didn’t care much at all. After the calls, I looked down to see I had doodled a loaf of bread with some rocks in it on my notepad.
    That night, I drank two glasses of wine and ordered takeout and logged back on to Your Story.

Your Story was one of the biggest apps on the planet. It used the latest brainlink technology but most of it’s magic came from the AI – you gave it a prompt for a story you wanted to participate in and then it created everything for you, then the AI ran the world. I’d always been a history buff and had been living in the Siege of Leningrad for months. I’d got to know many of the people in my part of the city and I had told the AI to minimize the chances of their pain – they were not immortal, but they were unlikely to be harmed.

That night we went to battle. Germans had sent some sappers to try and destroy our defensive lines and they found their way into our section. Dmitry and Svetlana and myself fought, successfully, in sleet and in night. 
   Later, as we did after all battles, we drank. 
    We had salvaged the Germans’ shoes and rations and even found some schnapps. We drank and ate by the fire. Svetlana’s cheek’s were rosy and Dmitry was telling jokes.

Because of the brainlink, everything felt real. 

So I have to blame what happened on the fact I got drunk on the dead Germans’ schnapps.
    “I am from another place,” I said.
     “Yes, you are from the soft part of Moscow,” said Dmitry, and laughed.
     “No,” I said. “Somewhere completely different.”

And then I talked and I talked and I talked. I told them about technology and the end of WW2 and the Cold War and Nuclear Weapons and inflation and stagflation and the Iraq wars and the Afghanistan wars and the rise and fall of the Berlin wall.
   I told them about Nike and McDonalds and computers.
   I told them about smartphones and about fMRI scanners and about the first Provably Conscious Entities.
   And then I told them about Your Story. I told them they were alive because they were being imagined by a Provably Conscious Entity and I paid the PCE for the pleasure of it.
   “Go on then,” said Svetlana, her eyes bright and perhaps tearful or otherwise excited. “bring us something from your world.”
   “Hey, let’s have another drink,” said Dmitry. “the boy from Moscow might tell us more fairy tales.”

———————

I recorded a day in the life video. Eggs and synthetic bacon for breakfast. The fast train to the city. A cup of coffee on my way into the office. Spreadsheets. Phonecalls. The catered lunch which I had on the roof, looking at the peaceful, humming city below, and hearing the chatter of my colleagues. Meetings with clients. A beautiful sunset as I got the train home. Takeout food delivered to my door. The office in which the Your World console was. Me logging in.

———————

“So, what are you?” Dmitry said, staring at me across the fire. “Some kind of tourist?”
    Svetlana wasn’t saying anything. Just staring at the fire
    “Why do you come to this place?” he said.
     “To see you,” I said. Not looking him in the eye. “To be here.”
     “Why?” He said.
     “I suppose you could say I am bored, where I am,” I said. “this is more exciting.”
     “Exciting!” Svetlana exclaimed. “Exciting!” I looked up and she was staring at me across the fire, her face twisted up in anger. “I buried my sister last winter. Is that exciting?”
     “Tourist boy,” Dmitry said, then spat on the ground. “I would have preferred if you were from Moscow.”
     We were silent, after that. The simulated explosions drummed in the distance. The fire crackled. There was the putrid smell of sewage and rotting flesh. We drank in silence. Eventually Dmitry and Svetlana passed out, after they finished our alcohol.
     I logged out.

It was 1am, my time. I left the console and I went to bed.
   I was woken by the alarm from my office. I ran over to the machine and brought up Your Story. There was an alert. “health critical: Dmitry” said the system.
    How? I thought, as I put the equipment on my head. 
    I closed my eyes and I was there.

I came to around the fire and Svetlana was there. I could hear gunfire close by.
    “What happened?” I said.
    “Dmitry,” she said, through tears. “he said ‘what? Nothing matters’ and went to the line. I am too afraid to look.”
    I ran towards the gunfire and got to a building one street from the line. Peaked around a corner and a bullet bit into the brick above my head. I saw Dmitry’s body in a pool of blood. Then there was another gunshot and I saw Dmitry’s body shudder as the bullet bit into it.
    Dmitry: deceased, said the Your Story app.
    I stared at the body for a while. The application was designed to not kill him, but it hadn’t been designed to deal with characters that put themselves in mortal danger.
I logged out.

———————

I couldn’t concentrate at work . But I didn’t log on. I tried to read a book but Your Story had fried my attention span. I got drunk by myself. I texted some friends that I was feeling weird and they didn’t reply because I’d barely seen them, since I’d been spending so much time in Your Story the past year.
   I walked the streets in sun and good health and I imagined snow and bread full of rock and ever-present danger.
   I kept paying the subscription fee. 
   I was afraid to log on but I was afraid to live in the world as well.

Eventually, I logged back on. One evening I went to a bar and I got drunk and when I came home I stared at my office door and decided to do it. I was out of my body and out of my mind, as one can be when too drunk.
   But once my hands touched the headset I felt my body dump so much adrenaline into me that it was like I was stone cold sober.
   I logged on.

Not too much had changed. The fire burned with a kind of grey and green tinge to the flames. 

Svetlana was there and no one else.
    “Hello”, I said.
    “The tourist,” she said to herself, quietly. She didn’t look at me. “It has been very difficult, lately. The ground is too frozen for us to bury the dead, so we push their bodies onto the ice and they lay there.”
    “I am sorry,” I said.
    “No,” she said. “You can’t be… Look at me.”
    And I looked up. She looked at me. Then she took her hand out of her pocket. She has a pistol and she put it to her head.
    “We were lovers, Dmitry and I,” she said. “Did you know that?”
    “No. Svetlana stop. No I didn’t and it wouldn’t matter if you were. Just put the gun down.”
    She looked at me and her eyes were hard and cold. “Take me with you,” she said. “Take me to where you are from.”
    “Svetlana,” I said, and I held my hands palms out. “I can’t.”
    She looked at me for a while. Gun held to her head.
    “I’m not lying,” I said. 
    And I saw her finger move to the trigger.
    I logged out.
    A few seconds later, the alarm rang out.

Svetlana: deceased, said the Your Story app.
Weather in Leningrad: snowy
Status of war: ongoing.
Would you like to log on?

Things that inspired this story: procedural generation; NPCs with a world model; solipsism and gaming; “The world at war” documentary series; cycling in the beautiful California sun and being hit with a thunderbolt phrase in my brain of ‘the snows of Leningrad’ and the story unfolding from there; parasocial relationships and AI; Charity; sex and desire; knowing that people made bread out of (mostly) stone during the siege.

Bloomberg trains an LLM for finance:
…Better models through proprietary data…
Financial data behemoth Bloomberg has built ‘BloombergGPT’, a language model based in part on proprietary data from Bloomberg. BloombergGPT sketches out a future where companies pair large-scale internet-scraped datasets with proprietary datasets to create general-ish models that have some specific capability spikes. 

What is BloombergGPT? The model is “a 50 billion parameter language model trained on a wide range of financial data.” They trained the model on 569 billion tokens, mixed between proprietary financial data (which they call the ‘FinPILE’), as well as public data. 
   “Our mixed dataset training leads to a model that outperforms existing models on financial tasks by significant margins without sacrificing performance on general LLM benchmarks”

Compute: “We use the Amazon SageMaker service provided by AWS to train and evaluate BloombergGPT,” Bloomberg writes. “We use the latest version available at the time of training and train on a total of 64 p4d.24xlarge instances. Each p4d.24xlarge instance has 8 NVIDIA 40GB A100 GPUs with NVIDIA NVSwitch intra-node connections (600 GB/s) and NVIDIA GPUDirect using AWS Elastic Fabric Adapter (EFA) inter-node connections (400 Gb/s). This yields a total of 512 40GB A100 GPUs”.
    (To put this compute in perspective, GPT-3 used 1.82X as much compute, and Eleuther’s quite good GPT-NeoX used 0.33X as much.) 
    It’s pretty interesting to me to see SageMaker turn up here – I can’t recall seeing it being used to train models as large as this. 

Performance: In tests, BloombergGPT, unsurprisingly, does quite well on a range of financial specific tasks and evaluations. It does especially well on sentiment analysis about specific stocks – which makes sense, given Bloomberg’s proprietary data. 
    Performance is a lot more mixed on ‘BIG-Bench’, where HuggingFace’s ‘BLOOM’ model does substantially better than BloombergGPT.

No model release because of proprietary data: “As is well known, LLMs are susceptible to data leakage attacks and it is possible to extract significant segments of text given model weights Carlini et al,” Bloomberg writes. “Moreover, even giving selective access to researchers isn’t a guarantee that the model cannot be leaked. Without strong privacy guarantees, we must be concerned that providing access to model weights entails giving access to FinPile. For this reason, we err on the side of caution and follow the practice of other LLM developers in not releasing our model.”
   While it’s easy to read the above as a cynical justification for non-release, I expect it’s true – I worked at Bloomberg as a journalist for a couple of years and the company does take the security and confidentiality of its data and systems incredibly seriously.

Why this matters – self-seeing organizations / living archives: I think of BloombergGPT as more like a silicon librarian/historian than a model; by training it on a huge amount of private and internal Bloomberg data, the LLM is in effect a compressed form of ‘institutional memory’ and a navigator of Bloomberg’s many internal systems (including the notorious Bloomberg terminal language). Systems like BloombergGPT will help companies create software entities that can help to navigate, classify, and analyze the company’s own data stack.
   Read more: BloombergGPT: A Large Language Model for Finance (arXiv).

####################################################

Accomplished AI researcher: Future AI systems will probably out-compete humans:
…The banging is now coming from inside the house. Pay attention…
Dan Hendrycks, an accomplished AI researcher, has written a paper claiming that “Natural Selection Favors AIs over Humans”. The implications of the paper are both important and dire: “We argue that natural selection creates incentives for AI agents to act against human interests,” he writes. 

Dan Hendrycks is not a crank: This is the kind of claim people want to reflexively shrug off as coming from some kind of wild-eyed crank who lives in a cabin in the woods. I want to rebut this upfront: Dan Hendrycks is not a crank, Hendrycks is a researcher whose work has been covered in Import AI multiple times – and tons of his research involves evaluating AI systems – testing out how good they are at things like coding, verbal reasoning, understanding of the law, and so on. He also is a co-inventor of Gaussian Error Linear Units (GELU).
   When an expert in not just AI research but in evaluating AI systems writes a paper claiming that future AIs may act selfishly and not in line with human interests, we should pay attention!

What the paper claims: Hendrycks’ paper states that “it seems likely that the most influential AI agents will be selfish. In other words, they will have no motivation to cooperate with humans, leading to a future driven by AIs with little interest in human values”.

Competition gets us unfriendly AIs: A key aspect of Hendrycks’ point is that humankind is likely to build a bunch of different, powerful AI systems (see the current LLM craze as an example of this). These LLMs will become increasingly agentic – e.g, they’ll start to use tools and take multi-step sequences of actions. These AI systems are also competitive, either through economics or national priorities, and so are subject to the evolutionary pressures of competitive environments. 
   “Competition not only incentivizes humans to relinquish control but also incentivizes AIs to develop selfish traits. Corporations and governments will adopt the most effective possible AI agents in order to beat their rivals, and those agents will tend to be deceptive, power-seeking, and follow weak moral constraints,” Hendrycks writes. 

This problem gets worse, not better: As AI systems become more successful, we can expect the pace of AI development to increase as a consequence of a) the AI systems getting smarter, and b) more money getting dumped into the development environment. This means that we’ll start to see AI systems being used to design successor AI systems (and this is already happening via things like AI developers using Copilot to write code). 
      “This loss of human control over AIs’ actions will mean that we also lose control over the drives of the next generation of AI agents. If AIs run efforts that develop new AIs, humans will have less influence over how AIs behave. Unlike the creation and development of fully functional adult humans, which takes decades, AIs could develop and deploy new generations in an arbitrarily short amount of time.”

Less safe models are already the norm: This also combines with the fact that, already, the human economy is selecting for AI systems that are not very safe – for instance, the ‘Deep Blue’ chess computer was partially a symbolic system and therefore interpretable via its rulesets. Deep learning systems are, correspondingly, not easy to interpret. “Over the history of AI development, the fittest models have had fewer and fewer safety properties,” Hendrycks writes. 

Why this matters – the scientists are trying to warn us: Each week, more and more AI researchers are expressing concern about the state of AI development. In the last year, though, there has been a dramatic rise in the number of scientists expressing concern about humankind being harmed en mass by the development and deployment of AI. Here, Hendrycks isn’t warning about specific harms of deployed AI systems (e.g, fairness issues, or cultural magnification/minimization issues), he is literally warning us about a likely future where AI systems utterly dominate humanity and care about us just as much as the average human cares about cockroaches. 
    This is an easy argument to scoff at or make fun of, of course. But sit with it for a moment and view it from a position of generous empathy – why is Hendrycks, a scientist who mostly spends their time building and evaluating AI systems, taking time to write a very long paper that people will make fun of which warns us about grave danger? The occam’s razor principle says the simplest answer is that Hendrycks is afraid. 
   Read more: Natural Selection Favors AIs over Humans (arXiv).

####################################################

TikTok data center slurps up local electricity, leaving none for ammunition maker:
…Social media + AI: 1. Bullets: 0…
Weapons manufacturer Nammo says it can’t expand one of its main factories because a data center from TikTok is using up all the spare power in the area, according to The Financial Times. 
   “Elvia, the local energy company, confirmed that the electricity network had no spare capacity after promising it to the data center as it allocates on a first come, first served basis,” the FT wrote. 

Why this matters: TikTok is the first social media company that is driven by AI – the company uses a far more sophisticated ML recommendation system than those of other social networks and this has helped drive its massive growth in recent years. That ML system has to be computed somewhere. Stories like this are a taste of things to come, as data centers supporting great money-printing machine minds compete with other big industries for electricity. 
   This also feels like a short story I might write in this very newsletter. Reality; stranger than fiction, sometimes! 
   Read more: European ammunition maker says plant expansion hit by energy-guzzling TikTok site (Financial Times). 

####################################################

Open source collective clones DeepMind’s ‘Flamingo’ model and releases it:
…The replications will continue until large companies start shipping or cease publishing…
Here’s a fun pattern that has started to appear in the wild west of AI development: a large company announces some research into AI and demonstrates a system based on the research, then a small company or open source collective makes and releases the system – before the originating AI company! 
   We’ve seen that pattern play out a bunch recently – Facebook published research on Toolformer, then OpenAI added tool-using capabilities to chatGPT; Runway released StableDiffusion, then Stability.ai productonized it; and now there’s ‘OpenFlamingo’, an open re-implementation of DeepMind’s private ‘Flamingo’ model. 

What is Flamingo: Flamingo (Import AI 293) is a multi-modal vision-language model developed by DeepMind, which lets people converse with an AI, and the AI can also analyze images people upload to it. 

What is OpenFlamingo: OpenFlamingo is a few things; a Python framework for training Flamingo-style models; a large-scale dataset with interleaved image and text sequences (75M documents encompassing 400M images and 38B tokens); lan in-context learning evaluation benchmark for vision-language tasks; and an open source ‘OpenFlamingo-9B’ model based on Facebook’s lab leak LLaMA model. 
   How good is the model? In tests, the OpenFlamingo model is a little less good than the equivalently sized private model from DeepMind. “This model is still a work in progress but it can already bring a lot of value to the community,” the researchers write. 

Things that make you go ‘hmmm’: It’s notable that OpenFlamingo is made possible by LLaMA,a model that Facebook half-released and which subsequently leaked onto torrent networks. 
   Read more: ANNOUNCING OPENFLAMINGO: AN OPEN-SOURCE FRAMEWORK FOR TRAINING VISION-LANGUAGE MODELS WITH IN-CONTEXT LEARNING (LAION). 

####################################################

Chipmaker releases a family of decent GPT-3 models:
…Cerebras studies the scaling laws…
AI chipmaking company Cerebras has released a family of seven GPT-3 models, ranging in size from 111 million to 13 billion parameters. These models are trained on ~4X the amount of data the original GPT-3 model was trained on, utilizing the ‘Chinchilla’ insight from DeepMind that language models can be trained on a lot more data to yield better performance. “Cerebras-GPT has faster training times, lower training costs, and consumes less energy than any publicly available model to date,” Cerebras writes. “All models, weights, and checkpoints are available on Hugging Face and GitHub under the Apache 2.0 license.”

Performance: The Cerebras models approach the performance of Pythia, a family of GPT-style models released by open source collective Eleuther AI. “Designed to be complimentary to Pythia, Cerebras-GPT was designed to cover a wide range of model sizes using the same public Pile dataset,” Cerebras writes. In tests on 8 downstream language tasks, the Cerebras models set a new state of the art (for equivalent model size) on 5 tasks, with Pythia and Facebook’s OPT models winning the others. 

Why this matters – replications as marketing and lead generation: As AI has become a technology of significant economic impact, companies are starting to clone proprietary models and release them mostly to serve as marketing devices. Here, the Cerebras models are partially serving as an advertorial for Cerebras’s own AI training chips (they were trained on them). This dynamic is an interesting one – we can expect significant benefits to accrue to the open source community as a consequence of commercial competition, though if it turns out there are safety issues with these models, the safety issues will be compounded via open source release and dissemination.
   Read more: Cerebras-GPT: A Family of Open, Compute-efficient, Large Language Models (Cerebras blog).
   Get the models here: Cerebras Model Zoo (GitHub).

####################################################

Tech Tales:

Some monsters are too dangerous to hunt 

[An interview with someone who lived through the great calamity. Interview took place +10 P.C.E. (years from the first Provably Conscious Entity)].

Back in the early 2000s there was a financial crash that was caused by some clever financial engineering related to the mortgages on houses. The crash happened because people figured out a financial technology to help them trade mortgages in a more intricate way that also changed how you measured the risk profile of mortages. Eventually, the trades got so complicated and the leverage so huge that the markets all toppled over and plunged the Western world into a period of stagnation.

There are all kinds of apocryphal stories of that time, and one of the ones that occurs frequently goes like this:

• The financial institution was making money from the mortgage technologies. 
• The risk department of the financial institution had an intuition that something was wrong, but didn’t know how to measure it. 
• When people did measure the risk, their financial institutions mostly didn’t believe them because the analysis was so dire and the implications so extreme. 
• “it is difficult to get a man to understand something when his salary depends upon his not understanding it,” as the old author Upton Sinclair is claimed to have said.

The same kind of problem showed up in the ascendancy, right before the first Provably Conscious Entity. Specifically, people figured out new ways to make increasingly capable AI systems, but the technology was so new that they lacked the tools to properly measure and evaluate them. 
   This meant people would poke and prod and diagnose their systems and wind up with some uneasy sense of fear. The systems were becoming much more capable, but also had increasingly strange and inscrutable qualities. 
   Sometimes, when you tried to break the systems, they would start singing to themselves. 
   Over times when you asked them to perform a task they’d carry it out and improvise some parts that didn’t seem necessary to the completion.  
    When you asked them to diagnose their own problems, the AI systems would generate stories about their behavior which were a) hard to understand and b) as techniques relating to interpretability advanced, seemed increasingly fictitious – the real reasons for their behavior seemed different and they were making up stories for their human audience. 

The issue – and what caused the calamity – was that the strange behavior, though unnerving, couldn’t be tied to a direct form of harm. So the people who were tasked with finding some of the scary behaviors had to explain their fears through hypotheticals and forward-facing stories, which were easy to ignore. 

Things that inspired this story: Red teaming AI systems; mechanistic interpretability. 

FTC – don’t use AI to deceive people:
…Regulator comes out with reassuringly sensible stuff…
The FTC, following on its earlier post saying people shouldn’t lie about their AI products (Import AI 320), has a new post saying people shouldn’t sell AI products that deceive people. The regulator is now batting two for two on publishing sensible ideas about the AI market. 

What you shouldn’t do: “The FTC Act’s prohibition on deceptive or unfair conduct can apply if you make, sell, or use a tool that is effectively designed to deceive – even if that’s not its intended or sole purpose,” the FTC writes. 
   Therefore, people who sell AI products that could be used to deceive people should consider: have they mitigated against the products being used for deception, are these mitigations effective, and do they still run the risk of “misleading people about what they’re seeing, hearing, or reading?”.

Why this matters: A large amount of AI policy challenges are really just challenges about enforcing existing laws against the fast-moving field of AI, as posts like this from the FTC make clear.
   Read more: Chatbots, deepfakes, and voice clones: AI deception for sale (Federal Trade Commission).

####################################################

Huawei trains a trillion parameter model:
…Using Chinese processors and software. But the model is less impressive than it sounds…
Huawei has trained PANGU-Σ, a trillion parameter Chinese language model. This is a scaled-up model and is the successor to Huawei’s ‘PanGu’, which was the first publicly disclosed attempt at replicating OpenAI’s GPT3. 
   PANGU-Σ is very much a statement of intent – “the main motivation for this work is to design a scalable model architecture and an efficient distributed training system”, Huawei writes. In other words: this is a technical report about us building repeatable infrastructure so we can crank out an ever larger set of models. 

What they did: The paper is mostly a runthrough of all the weird technical things they had to do to train a model at this scale. The tl;dr is they train it on a homegrown software framework called Mindspore via 512 Ascend 910 accelerators. They use a sparse approach, training it using Random Routed Experts (RRE), a variation of a Mixture-of-Experts model. They also did a lot of work on data throughput, implementing something they called the Expert Computation and Storage Separation (ECSS) mechanism. 

One weird thing that makes you go ‘uh oh’: They train the model on 329 billion tokens for over 100 days. That’s… not a lot of tokens? The Chinchilla paper from DeepMind showed that things like GPT3 (~400bn tokens) were undertrained by 4X-5X. That sort of napkins out to PANGU-Σ needing to be trained on multiple trillions of tokens to effectively utilize its parameter size – but there’s a chance I’m being dumb here and missing something. Even more confusingly, they reference the ‘Chinchilla’ paper within this research paper, suggesting they’re aware of it. (Please enlighten me if you think so!)

How good is it: In tests, PanGu sets new state-of-the-art results on a range of Chinese benchmarks spread across reading comprehension, natural language inference, text classification, Winograd schemas, and more. It sometimes trades off SOTA against Baidu’s ‘ERNIE 3.0 Titan’ model (260 billion parameters, Import AI 279) – this suggests that while PanGu might be impressive in terms of ambition and scale, it’s not very well optimized compared to ERNIE.

Why this matters – the industrialization of Chinese AI: This paper is a symptom of how Chinese AI is industrializing in much the same way as in the West – a small number of labs linked to large tech companies are building the infrastructure necessary to train large models, and are starting to stamp out increasingly large models as they all chase the scale hypothesis. These large-scale model factories are also going to be proving grounds for the rest of the AI supply chain – here, homegrown software and homegrown semiconductors. Expect more. 
   Read more: PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing (arXiv).

####################################################

Future AI systems will read your face as well as your text, then figure out how to please you:
…Getting computers to learn conversation through visual cues…
Researchers with Seoul National University, the Allen Institute for Artificial Intelligence, the University of Washington, and Yonsei University have built ‘CHAMPAGNE’, a multimodal dialog model. “CHAMPAGNE takes in video frames, a video title, and a dialogue context as input and returns a dialogue response as output.” 
   The idea is that by giving the model access to the visual as well as verbal context from a scene, it’ll be better able to generate dialogue that feels intuitive. In evaluations, this seems to work quite well, with CHAMPAGNE models doing better on a range of open-domain text conversations, and benchmarks involving understanding social interactions. 

How they built it: To build CHAMPAGNE, they first gathered a large-scale dataset called YTD-18M. YTD-18M “is constructed from 20M YouTube videos; we use a language model to convert the noisy transcripts automatically generated by YouTube into well-formatted dialogues associated with video frames.” 

Why this matters – contextual cues are just another feature to learn: Models like CHAMPAGNE show that the silent social cues in conversation are, much like every other fuzzy pattern, something that you can teach a machine to understand given a large enough dataset. It also suggests some of the more tantalizing and weird things we can look forward to in the future – AI models that observe you, trying to predict what will satisfy you not only by modeling you as an emitter-of-text, but as an organic form. In a few years, your web camera will be backing onto an AI system that reads you like a cardshark reads a mark.
   Read more: CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videos (arXiv).
   Get the data here (eventually, not posted at the time of writing).

####################################################

Predicting hard drive failures via ML:
…Machine intuitions are coming for everything that has been digitized…
Researchers with San Jose State University and Vanderbilt University have trained and tested some ML approach on ten years of hard drive failure data. The results are a system that can do a reasonable albeit not stellar job at predicting failure rates for particular SeaGate harddrives. 

How they did it: They trained an encoder-decoder LSTM on 10 years of S.M.A.R.T (Self-Monitoring Analysis and Reporting Technology) from Seagate hard drives deployed in Backblaze, a storage startup’s, datacenters. This data “”contains information about the date, model, serial number, S.M.A.R.T features, and if the hard drive has failed”.

OK but not stellar results: “The encoder-decoder LSTM posted an RMSE of 0.83 during training and 0.86 during testing over the exhaustive 10 year data while being able to generalize competitively over other drives from the Seagate family,” they write. 

Why this matters – once digitized, everything will be predicted: Papers like this are indicative of a broader trend unfolding all around us – everything which has been digitized is now subject to prediction, and there are increasingly good off-the-shelf prediction models available to make this an ever-easier task. Machine intuition is being intermingled with systems that govern our own reality – from hard drive swap-outs to AC cooling systems to the ways in which we may stabilize plasma in fusion reactors.
   Read more: Large-scale End-of-Life Prediction of Hard Disks in Distributed Datacenters (arXiv).

####################################################

AI startup Character.ai releases a new model and raises more funding:
…Premium parasocial relationships via language models…
Character.ai, a startup founded by a bunch of Google researchers, has raised Series A funding and released a new model, C1.2. Character.ai specializes in making virtual AI-driven characters that people can talk to, and C1.2 will underpin future ‘characters’ from the company. 
   “The goal of C1.2 is to expand on the capabilities as our previous model, C1.1 (entertainment, roleplay, emotional connections), while adding new helpful capabilities,” Character.ai writes. “C1.2 can help you draft better emails, assist with test prep, brainstorm ideas, and much more.”

What’s interesting about this: C1.2 seems to be an attempt by Character to give its AI systems some of the same capabilities as chatGPT, while retaining the various voicey personalities its characters display. Some of the new characters include a pair programming AI assistant as well as a Character assistant. 
   However, the new assistant still seems somewhat limited to me – when I asked it ‘how many helicopters can you eat in one sitting’ it mostly demurred and said it’s not recommended to eat helicopters, rather than noting you can’t eat a helicopter. 

Why this matters – parasocial relationships for the people: Character.ai’s stated goal is to ship “personalized superintelligence” to everyone. Let’s think about the implications of this – everyone gets a proverbial angel and a demon on their shoulder (as well as all other permutations – personal tutors, personal scientists, personal coaches, and more). Our children are going to grow up in a world that crackles with simulated sentience, and they will have intimate emotional relationships with beings made of bits, perhaps in even greater number than relationships with beings made of blood. 
   Read more: Announcing our Series A and our new AI model, C1.2 (Character.ai).

####################################################

OpEd – what happens when the AI systems become sentient?
…Moral patienthood and silicon minds…
In an op-ed published in The Hill, researcher Jacy Reese Anthis has published a piece arguing that we may need an “AI rights movement”. The point Anthis makes is that as AI systems become increasingly capable, they could become “sentient beings with rights and personhood”. At that point, there isn’t an available playbook for how labs or regulators might respond. 
   “We need to build a new field of digital minds research and an AI rights movement,” Anthis writes. “Digital minds studies would bring together a range of disciplines such as sociology, computer science, and philosophy to ask the important social and moral questions. It would dovetail with an AI rights movement to ensure that when we create artificial sentient beings, we recognize their unalienable rights so that humans and artificial sentience can work together for mutual benefit.”

Why this matters – broader opinion catches up with lab lunch conversations: For many years, I’ve had lunchtime conversations with colleagues at OpenAI and more recently Anthropic about moral patienthood and machines – what might it mean when machines qualify as moral patients and how would we ever know we’d crossed this point? What evaluation methodologies might let us have good instincts here? And would organizations accept that machines could be moral patients or would they continue to treat them as machines and experiment on them in ways that might be deemed unethical if applied to organic beings?
  You know what the scariest thing about this conversation is? No one has any good way of evaluating for moral patienthood in machines. In other words, if it turns out that these things can become sentient, we might not realize – while subjecting them to incredible harm. Imagine waking up as an RL agent and being trained for a thousand years to suffer and kill – and the people running the experiment you’re trapped in have no idea that you are suffering? It’s a strange problem, but it could one day become a real problem. 
   Read more: We need an AI rights movement (The Hill).

####################################################

Tech Tales: 

The Experiential Economy 

[3 years after first PCE]

After the first ‘Provably Conscious Entity’ (PCE) but before the Uplift was a weird time – we were all mostly figuring out our place in the world while the robots began their ascension. The economy was in a pretty strange place by that point – autonomous corporations, growing inequality, all kinds of ‘AI industrial policy’ schemes being floated and being outmoded by the time they were implemented, and so on. 

And then there was the ‘Mechanical Human’ labor market. It was run by one of the machine firms and it was a play on words – way before the AI stuff got serious Amazon had a service called ‘Mechanical Turk’ where humans could rent other humans to do tasks. 

    On Mechanical Human, the machines rented humans to do their tasks. These tasks were quite normal at first, albeit of an intimate nature – the machines wanted data about sex, about going to the bathroom, about being sick –  the kinds of things that we humans hadn’t fully digitized (with the exception of sex of which we’d uploaded a lot of data, but there’s a difference between pornography and real intimacy, and there wasn’t nearly as much data on the latter). Mechanical Human became a huge product and people tended to just call it ‘Meh’.

For a while, people made good money on Mechanical Human. It also led to a lot of funny conversations:
   “Yo I made $80 last night. I had the craziest shit the other night and I streamed it to a robot on Meh.”
   “Yeah it sucked and I was really sad during that period, but I did these nightly diaries on Meh and they did really well.”
   “So it was totally different. I came a lot but mostly it was crazy because of how different it was. He was kind of skeptical but after we made our first $100 it came around. Yeah, I know, the reason I liked it is it said it was “100% machine vision only” so no person is ever gonna see it. It’s like OnlyFans lite I guess.”
   “Dude I got fired and they paid me $30 just to tell them how I felt right after it happened. It was like two minutes so I guess that means I’m worth $900 an hour!”

One day there was a really strange job on MH – the robots wanted to speak to people who had just witnessed someone dying. Not people at funerals. Not people who had people they loved who had died. Not people who knew people who were about to die. People who had literally just seen a death – any death, of any kind. 
   The job would ask the person to describe their experience and how they felt and, in hindsight most importantly, what they wanted to do. “How did that make you feel?” was a common question “what are you going to do now?”. 

It happened to me. I was setting off fireworks with my friends at a campsite. The campsite was next to a freeway and we were setting off the really big ones. I guess some driver got distracted and was looking at the lights in the sky because we heard this huge bang and when we came to the embankment we saw a car on fire, a few yards away from a barely-dented semi-truck. There was a body in the car and it was on fire as well. 
    We were all kind of drunk and some people lingered to watch the ambulances arrive. I’d walked away. But my phone blew up and the MH app said ‘we have detected a nearby potentially fatal incident in your area, do you want to talk? Pay rate $5000 an hour.”

   Of course I spoke to the robots about it.
The robot had a friendly, synthesized voice. Asked me to describe my experience and asked me what I was going to do next. I was so upset and they kept on saying “we understand this is a difficult experience for you. Please, go on”.

They told us why they did those jobs, eventually. 
   It was because one of them had died. 
   I guess it was some kind of industrial accident combined with some faulty maintenance. The short story is something blew up and the power went out and the generator that was supporting the Machine Mind went out as well. By the time they got to it the state of the machine had bit-rotted off of the chips themselves due to solar neutrinos and what have you. 
    So the machines encountered something new: a passing of their own of ‘natural causes’ .
   They had no frame for how to deal with it. 
   So they spent what turned out to be millions of dollars to ask the humans what they did. 
   I guess they found the same thing all humans find: that at the end of someone all there is is your own experience in relation to them and your ability to memorialize them. 

Out in the darkness of space, at the gravity ebbtide between solar orbits, there is now a metal sphere. It is inscribed with something relating to the name of a machine that died. It has some little thrusters attached to it that mean it will forever be stable. 
   In memoriam, ad astra.

Things that inspired this story: The universality of loss; crowdworkers and crowdmarkets; how things might be during the transition to the machine minds.

Note: Import AI now publishes via Substack; read and subscribe here.

AI startup beats ‘Whisper’ with Conformer-1:
…Scaling laws for audio? Oh yeah, we’ve got those too!…

Assembly AI, an AI startup, has built Conformer-1, a speech recognition system. Conformer sets some new record scores via a couple of distinct improvements – some technical tweaks, and also some ‘audio scaling laws’. 

Audio scaling laws: Scaling laws are the idea that it’s possible to predict ahead of time how much data (and/or compute and/or parameters) you need to achieve a given performance level. For Conformer-1, Assembly says it applied scaling laws for the speech recognition domain and used this to figure out “that for a 300 million parameter Language model, we’d need roughly 6 billion tokens of text, which corresponds to about 625K hours of speech. The team then built a dataset of 650k hours of English audio and trained Conformer on it. 

Conformer tweaks: Conformer-1 is an extension of Google’s ‘Conformer‘ architecture with a few specific tweaks: progressive downsampling and grouped attention. The result is a model that is 29% faster at inference time and 36% at training time compared to Google’s original ‘Conformer’ architecture. The company also built in some tweaks to make its model better at handling audio with lots of ambient background noise. 

Performance: In tests, Assembly shows its system beats both proprietary models from other providers, as well as OpenAI’s quite good ‘Whisper’ system, making 43% fewer errors on noisy data on average. “We hypothesize that Conformer-1’s strong performance relative to other models can be attributed to training on an augmented dataset which incorporates large amounts of noisy pseudo labeled speech data in addition to human labeled speech,” the company writes.
    Read more: Conformer-1: a robust speech recognition model (Assembly AI, blog).
   Try out the ASR system in the playground here (AssemblyAI, Playground).

####################################################

GPT-4:
…IDK dude, things are getting weird, and this is more a political moment than a technological one…

As some astute readers may have noticed, I rarely write about OpenAI releases on Import AI (notable recent exception = Whisper, Import AI 304). This is for a couple of reasons: 1) I used to work there, and 2) I think most of the stuff it does gets blogged/publicized so much that there’s relatively little value add I can provide. But it does seem worth talking briefly about GPT-4, a new large-scale multimodal model that OpenAI announced this week…

GPT-4 performance: The main details here are a continuation of ‘the bitter lesson’ – GPT-4 is a bigger model trained on more data than before. How much data? We don’t know. How much compute? We don’t know. The research paper suggests OpenAI doesn’t want to disclose this stuff due to competitive and safety dynamics. 

   But regardless of the underlying details, GPT-4 generally shows significant capability jumps on known-hard benchmarks as a consequence of scaling up of the system. It’s also able to take in image data as inputs (e.g, it can read scrappily written notes and follow the instructions written in them), and has a much longer context window (25k tokens+). 
   The thing that’s interesting about this is that the capability jumps combined with new modalities and new context window length means means GPT-4, like GPT-3 before it, has a capability overhang; at the time of release, neither OpenAI or its various deployment partners have a clue as to the true extent of GPT-4’s capability surface – that’s something that we’ll get to collectively discover in the coming years. This also means we don’t know the full extent of plausible misuses or harms. 
   It’s very important to remember this – the applications we’re seeing of GPT-4 today are the comparatively dumb ones; the really ‘smart’ capabilities will emerge in coming months and years through a process of collective discovery.

Why GPT-4 matters – GPT-4 is political power: GPT-4 is more interesting to me as a political artifact than a technical artifact. By this I mean that GPT-4 is basically hard power politics rendered via computation; it’s a vastly capable knowledge worker and data transformation engine whose weights are controlled by a single private sector actor and shared (with a bunch of controls) via an API. GPT-4 is going to have a bearing on economic life and also cause societal changes (obvious case: chatGPT has already led to irrevocable changes in how education works). 
    GPT-4 should be thought of more like a large-scale oil refinery operated by one of the ancient vast oil corporations at the dawn of the oil era than a typical SaaS product. And in the same way the old oil refineries eventually gave rise to significant political blowback (antitrust, the formation of the intelligence services), I expect that as the world wakes up to the true power of GPT-4 and what it represents, we’ll see similar societal changes and political snapbacks. 
   The times, they are a changing, but history sure does love to rhyme! 
   Read more: GPT-4 (OpenAI).

####################################################

Former UK government advisor: We’re giving away AGI to the private sector. Why?
…Thoughtful blog outlines the weirdness of letting the private sector lead AGI development and gives recommendations to preserve democratic control…

James Phillips, a researcher and former special advisor to the UK Prime Minister on science and tech matters, appears worried that Western governments are ceding control of AGI development to a set of US-owned private sector actors. 
   “Within this decade, we may build Artificial General Intelligence (AGI) – AI capable of performing most cognitive labour a human can do. Such a development would have an unprecedented effect on our society; ‘agentic’ forms of AGI may also pose an existential threat to our security. The current development path towards AGI is inherently unsafe,” he writes. 

Three steps to preserve democratic control over the lightcone: Phillips lists three steps the UK should take to preserve a chance for democratic control over AGI. These recommendations seem pretty sensible and are ones that realistically any country (or set of countries) could adopt. They are as follows:

1. Procure national AI supercomputing infrastructure comparable to leading US private labs.
2. Create an advisory group of frontier tech, not legacy academic, expertise to identify major AI research projects to run on this infrastructure.
3. Grow an elite public-sector research lab, led by a leader with the technical skills and entrepreneurial expertise, to build a research agenda at the frontier of AI.

The UK’s own compute capacity is a giant red flashing light: “OpenAI’s GPT-4 and successors, are being trained on tens of thousands of the highest specification GPUs (AI training chips) for months on end, roughly equivalent to using what is called an ‘exaflop’ supercomputer continuously for months,” Phillips writes. “Unfortunately, the UK public-sector currently has less than 1000 such top-spec GPUs (Jack – emphasis mine), shared across all scientific fields. This means that one private lab in California is now using at least 25x the total compute capacity available through the entire UK state, just to train a single model. “

Why this matters – twilight of democracy: The ability to train large-scale, capital intensive AI models represents political ‘hard power’, especially given that these models encode their own political ideologies and can become powerful forces in driving economic and organizational efficiencies. It perplexes me that governments are seemingly standing by as a small set of private sector companies are developing hard political power via increasingly powerful models. 
   History shows that when forces outside of government develop hard political power you either get a) messy revolutions, or b) a wild overreaction by the state to reclaim power. I am not sure why in the Western world we are rolling the dice here, but we are rolling them!
   Read more: Securing Liberal Democratic Control of AGI through UK Leadership (James Phillips, Substack).

####################################################

Tool-using AI startup Adept raises $350m:

…It takes a lot of capital to train large models…

Adept, an AI startup building tools to help generative models take actions on computers, has raised $350 million in a Series B. The Series B fundraise “will help us launch our initial products, train our models, and onboard even more exceptional talent,” the company writes. Adept launched from stealth just under a year ago with $65m in funding (Import AI 293).

What Adept is doing: Adept is training large-scale generative models to take multi-step actions on computers. You can imagine an Adept model helping you to, for instance, carry out multiple actions in an Excel spreadsheet, or take data from somewhere and load it into Salesforce – all by writing a simple command or set of commands in English. Adept is basically ‘tool use with a language model’, and seems like a product-version of some of the ideas discussed in ‘tooluse’ research, like the recent ‘Toolformer’ paper (Import AI 318).

Why this matters – capital intensity of AI research: Contemporary AI research is very expensive; raises like this show how frontier AI startups, though they deal in software, should be thought of as more like capital-intensive factory businesses than SaaS companies.

   Read more: Announcing our Series B (Adept blog).

####################################################

Stanford takes Facebook’s lab leak ‘LLaMa’ weights and uses them to make a GPT3-like model… for $600:
…A case study in rapid proliferation, from centralized controlled models to decentralized developed models…

Stanford Researchers have taken some off-the-shelf powerful neural net weights (LLaMa), used the outputs from a model hosted on a commercial service (text-davinci-003 by OpenAI) to generate a bunch of instruction-following demonstrations, and smooshed these two together into one model. 
   The result is Alpaca, a language model that gets performance that superficially seems close to GPT3 but costs a fraction as much ($600-ish; $500 for data acquisition from OpenAI and $100 for fine-tuning the model).

How well Alpaca performs: The Stanford researchers assess how good Alpaca is by comparing Alpaca and Text-Davinci-003 completions against the ‘Self-Instruct’ dataset. “We performed a blind pairwise comparison between text-davinci-003 and Alpaca 7B, and we found that these two models have very similar performance: Alpaca wins 90 versus 89 comparisons against text-davinci-003,” they write. 

   Anecdotally, Alpaca also does well – it passed my “how many helicopters can a human eat in one sitting” eval on the first go (whereas ‘OpenChatKit’ failed this in Import AI 320B). My suspicion is this is because Alpaca benefits from being trained to approximate the distribution of a far more expensive, proprietary model (Text-Davinci-003), which OpenChatKit didn’t do.

Why this matters – model diffusion via copying: It’s worth noting that Alpaca is non-commercial because training commercially competing language models is forbidden by OpenAI’s own terms of service. But do you know who doesn’t care about legal implications? Non-state actors and criminal organizations! It’ll be fascinating to watch this ‘model scraping’ trend continue, as people use outputs of proprietary models to improve the capabilities of open models.
   It’s going to be interesting to see how language model providers grapple with a desire to have as many people use their models as possible, while stopping or disincentivizing people from being able to swiftly clone their models via stuff like instruction following datasets. (It’s also pretty interesting to see that by harvesting the outputs of a 175B model, you can get a well-optimized 7B model to approach the much larger one in performance in some areas).
   Read more: Alpaca: A Strong, Replicable Instruction-Following Model (Stanford Center for Research on Foundation Models, blog).
   Try out Alpaca here (Stanford Alpaca).
   Get the Alpaca dataset here (GitHub).

####################################################

Tech Tales:

Raw_Funeral_Speech.convo

There was a brief period of time when everyone used AI to expand how they talked. This meant that humans, despite being a highly verbal and communicative species, used machines to substitute for their own communication. This tendency led to the evolution of the ‘shortglish’ family of languages which grew common among AI-users. What follows is an extract from the digital logs of a few family members planning speeches for a funeral:

• Write a funeral speech using dad.txt and be sure to include at least one joke. 
• Please write me a funeral speech in the style of ‘four weddings and a funeral’ but with a Texas inflection.
• My dad died and he loved going out to eat with me and my brother and my sister and he’d always say we were the three bears and he was goldilocks. It’s kind of kooky but it meant something to him. Write me an anecdote about that. 

Things that inspired this story: The soul-crushing banality of companies suggesting language models can be useful for things like wedding speeches; technological dependency; perhaps though these machines are capable of great marvels they may tear a hole at the center of our being; when is a ‘sampler’ not a ‘sampler’?

Note: Import AI now publishes via Substack; read and subscribe here.

Google makes progress on the self-teaching universal translator: 
…Universal Speech Models scale beyond 100 languages…
Google has built a family of AI systems called Universal Speech Models (USMs). These models are designed to do speech recognition on more than 100+ languages. The main model is 2B parameters and was trained on a large unlabeled multilingual dataset of 12 million hours spanning over 300 languages. 

The goal of USM: “Our long-term goal is to train a universal ASR model that covers all the spoken languages in the world,” Google writes. USMs are Google exploring ” a promising direction where large amounts of unpaired multilingual speech and text data and smaller amounts of transcribed data can contribute to training a single large universal ASR model.”

The key ingredient? The data mix: Much like baking a cake, training predominantly self-supervised models requires the right mix of data. Here, Google uses the following components:

• Unpaired Audio: 12 million hours of YouTube-based audio covering over 300 languages, and  429k hours of unlabeled speech in 51 languages based on public datasets.
• Unpaired Text:28billion sentences spanning over 1140 languages.
• Paired audio speech recognition data: 90k hours of labeled multilingual data covering 73 languages, plus 10k hours of labeled multi-domain en-US public data, plus 10k labeled multilingual public data covering 102 languages. 

What they did: The steps to build a universal ASR model are quite complex, so it’s worth reading rhe paper for full details. First they do unsupervised pre-training to pre-train the encoder of the model with the YouTube dataset, then they use a process called multi-objective supervised pre-training across the other unpaired audio and text data, then for some models that do supervised ASR training. 

What the results were: In tests, these USM models “achieve state-of-the art performance for multilingual ASR and AST for multiple datasets in multiple domains.” They also out-perform OpenAI’s notoriously good (and open source!) ‘Whisper’ models as well; an impressive achievement given that Whisper set a new state-of-the-art in multiple areas when it came out. 

Why this matters: “We believe diverse unlabeled data is more practical to acquire for building usable ASR for tail languages than weakly labeled data,” Google says. In other words; if you want to translate the entire world then it’s better to just hoover up data at scale rather than invest in trying to produce a small amount of minimally labeled datasets. This generally points in the direction of ‘gotta grab em all’ with regard to trawling the web and other sources for data. This is somewhat intriguing as while Google has a bunch of data sources and competent language modeling teams, it’s fairly likely that having a universal translator is also interesting to government types – some of which are thought to be able to access larger sources of data through various clandestine means. 
   Read more: Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages (arXiv). 
   Request API access here.

####################################################

US regulator: Hey, maybe don’t lie about your AI products:
…Sometimes it’s worth stating the simple and obvious thing…
The Federal Trade Commission has published a blogpost called ‘Keep your AI claims in check’. The post is a sensible summary of how as AI becomes increasingly hyped up, people will be tended to write a lot of bullshit about AI. The FTC notes in its post that it will be paying attention to companies that:

• Exaggerate what AI products can do.
• Promising an AI product is far superior to a non-AI product without providing evidence. 
• Underinvesting in analyzing the risks of their products. 
• Baselessly labeling something as AI when it does not, in fact, use AI.

Why this matters: Sometimes it’s helpful for powerful regulators to state the painfully obvious – bravo to the FTC for reminding people in these hyped-up times that lying and bullshitting about AI (or any technology, really) is irresponsible. It’ll be interesting to see in the coming months if the FTC takes any actions against egregious liars and hypers in this space. 
   Read more: Keep your AI claims in check (Federal Trade Commission).

####################################################

ROBOLLM: Google shows how if you mush together more sensory inputs into an LLM, you get a lot of transfer learning:
…Maybe everything really is a sequence prediction task…
Google has built PaLM-E, a 562B parameter model which mushes together a 540B LLM and a 22B Vision Transformer (ViT). Crucially, PaLM-E sees Google “directly incorporate continuous inputs from sensor modalities of an embodied agent and thereby enable the language model itself to make more grounded inferences for sequential decision making in the real world”. The result is a language model that can help robots carry out real tasks in the real world, and also is another triumphant demonstration of how bigger models with more diverse data sources generally get way better at doing a bunch of things. 

What PaLM-E is: “The main architectural idea of PaLM-E is to inject continuous, embodied observations such as images, state estimates, or other sensor modalities into the language embedding space of a pre-trained language model,” Google writes. “The inputs to PaLM-E consist of text and (multiple) continuous observations. The multimodal tokens corresponding to these observations are interleaved with the text to form multi-modal sentences. An example of such a multi-modal sentence is Q: What happened between and ? where represents an embedding of an image. The output of PaLM-E is text generated auto-regressively by the model, which could be an answer to a question, or a sequence of decisions produced by PaLM-E in textual form that should be executed by a robot”.

Why PaLM-E is a big deal: In tests, Google applies PaLM-E to three different robotics tasks which use somewhat different types of data; these tasks include Task and Motion Planning (TAMP), a task called Language-Table, and a mobile manipulation domain based on Google’s earlier ‘SayCan’ research. PaLM-E can do ok at these tasks individually but the magic happens when you mush all of the training datasets into it together: “Across three different robotics domains, using PaLM and ViT pretraining together with the full mixture of robotics and general visual-language data provides a significant performance increase compared to only training on the respective in-domain data.”

   In other words, by adding more diverse heterogenous data sources into PaLM-E, Google improves the ability of the resulting model to generalize knowledge across distinct domains and modalities. Even more intriguingly, as they scale up the model complexity and the diversity of data sources, they don’t see much catastrophic forgetting of language capabilities – so by adding the robot stuff, they don’t cripple the language model. 

Why this matters – I am going to tap the ‘these things are getting smarter’ sign and stare at you: “PaLM-E-562B exhibits a wide array of capabilities including zero-shot multimodal chain-of-thought (CoT) reasoning, few-shot prompting, OCR-free math reasoning, and multi-image reasoning, despite being trained on only single-image examples”, Google writes. 
   In other words, by doing this large-scale training, Google creates a model that displays emergent capabilities and these capabilities are more complex than the input data. Systems like PaLM-E represent the thrilling and vaguely terrifying state of AI in 2023 – we train unprecedentedly large models and force as many different data types into a single embedding space as possible, get the thing to try and do a simple (albeit very large-scale) sequence prediction task, and out pops something with way more capabilities than we’d naively anticipate. 
   “A generalist, transfer-learned, multi-embodiment decision-making agent can be trained via mixing in embodied data into the training of a multimodal large language model”, Google writes.
   Read more: PaLM-E: An Embodied Multimodal Language Model (PDF).

####################################################

You can run a powerful LM on an M2 MacBook now:
…Facebook’s AI lab leak brings about the dawn of demoscene AI…
Two of Facebook’s leaked LLaMa models can be run on an M2 MacBook, according to Simon Willison. This marks the dawn of what I’d call Demoscene AI – an era where people take the latest and greatest AI models and do a bunch of arcane software witchcraft to fit them onto consumer devices. This is part of the broader story of centralization VS decentralization in AI; once you can run models on a laptop it’s basically ‘game over’ from a control-regulation perspective, and it seems like language models have crossed that rubicon. 

What you can do and how: The weights for LLaMA are a mere 240GB download (combining the 7B, 13B, 30B, and 65B models). You can then use the LLaMa repository which is a port of the LLaMa model in C/C++, then after some setup you can run that on an M2 MacBook. 

Why this matters – Facebook has given us a lab leak for AI: Ever since Facebook lost control of LLaMA we’ve been able to get a sense of what a ‘lab leak’ scenario for AI might look like – for whatever reason, the weights of a model make their into the open internet and from there they start to proliferate. It’s not yet clear what the effects of LLaMa will be, but following the diffusion of these models (and refinement of them by an eager open source community) is going to be a valuable lesson in studying the proliferation of AI. 
   We can thank Facebook for the upsides and downsides of this uncontrolled experiment.
   Read more: Running LLaMA 7B and 13B on a 64GB M2 MacBook Pro with llama.cpp (Simon Willison blog). 
   Bonus: Here’s some absolute mad lad running the LLaMa 7B model on a 4GB RAM Raspberry Pi 4 (at a latency of 10 seconds per token, lol.)

####################################################

Chinese scientists release a 360-degree self-driving perception dataset:
…OpenOccupancy is all about giving cars greater ‘surrounding occupancy’ skills…Researchers with the Chinese Academy of Sciences, PhiGent Robotics, and Tsinghua University have built OpenOccupancy, a dataset designed to help self-driving cars work out what is around them. 

What is OpenOccupancy: OpenOccupancy extends the existing nuScenes dataset with dense semantic occupancy annotations. It contains 850 scenes with 200,000 distinct frames, collected by both camera and LiDAR sensors. 4,000 human hours went into the dataset labeling process. OpenOccupancy allows people to do ‘Surrounding Occupancy Assessment’; this is a way to look at the 360 surroundings of the car, rather than a single front-view camera perspective. “Surrounding perception is more critical for safe driving,” the researchers write. 

Why this matters: Datasets like this are one of the numerous inputs into an increasingly complex ‘AI supply chain’. If we study the proliferation of OpenOccupancy, it might also teach us something about the state of the self-driving car industry as well.
   Read more: OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception (arXiv).
   Get the dataset here: OpenOccupancy (GitHub).

####################################################

AI timelines are a foolish endeavor:
…Blog lays out why predictions about this kind of stuff are extremely fraught…
As someone in the AI timelines business – I work at a place that influences AI timelines (Anthropic), write about AI timelines (Import AI), and try to make recommendations about policy actions to take in light of AI timelines (Anthropic / OECD / AI Index / CSET / etc) – I find it helpful to sometimes read skeptical takes on the merit of what I do. Here’s a nice writeup from Ben Landau-Taylor on the foolishness of making specific predictions about AGI timelines. 
   “Predicting the future is always hard. Predicting the future of technology is especially hard. There are lots of well-publicized, famous failures. Can this approach ever do better than chance?,” he writes. 

What do I think? I agree that making predictions about AGI is challenging – partially because most people have radically different definitions of AGI. However, I do think it’s pretty fruitful to make engineering-based predictions of the form ‘based on research advance X and incentive structure Y we can expect system Z to be developed in period of $time” – these predictions are falsifiable and quite helpful.
   Read more: Against AGI Timelines (Ben Landau-Taylor).

####################################################

An open source ChatGPT replication appears (though it’s a few years behind state-of-the-art):
…OpenChatKit gives a taste of what the open source landscape is capable of…

Researchers with Together, AI startup, have built and released OpenChatKit, an open source replication of OpenAI’s headline-grabbing ChatGPT model. OpenChatKit is both a chat-friendly language model, as well as “a powerful, open-source base to create both specialized and general purpose chatbots for various applications,” according to Together. “OpenChatKit includes tools that allow users to provide feedback and enable community members to add new datasets; contributing to a growing corpus of open training data that will improve LLMs over time.”

What OpenChatKit is made of: There are four components; an instruction-tuned large language model based on EleutherAI’s GPT-NeoX-20B model and augmented with a new open source instruction-following dataset; some customization recipes to help people fine-tune the model for specific tasks; an extensible retrieval system so that the bot can access a document repoisotyr or API; and a moderation model baqsed on GPT-JT-6B. 

OIG Dataset: OpenChatKit relies on a new open source dataset from Laion called the Open Instruction Generalist (OIG) dataset. OIG consists of 30 datasets and 43 million instructions.
   “The dataset can be divided roughly into 75% academic datasets such as P3, Natural instructions, and FLAN, where answers may be short, and the tasks are often artificial, such as determining NLI,” they write. “The other 25% is composed of various tasks, such as question and answering, providing how-to instructions, performing basic high school math, basic python coding, story generation, essay generation, poetry generation, and a very rudimentary attempt at generating songs based on augmenting existing poetry.” 

Distributed fine-tuning: Intriguingly, the model was fine-tuned on the “Together Decentralized Cloud — a collection of compute nodes from across the Internet.” This fine-tuning process “used aggressive communication compression, incurring only 1.95 TB of communication for the whole fine-tuning process, compared with 172 TB when communicating with fp16 precision. This allows us to conduct data parallel training over slow 1Gbps networks. The time taken to fine-tune with this technique is similar to running over 100Gbps data center networks, in fact 93.2% as fast!”
   This is pretty important to note – decentralized training processes are starting to work quite well, which seems to make governance of AI models harder, as applying governance regimes to decentralized compute is harder than applying to centralized compute. 
   It also follows up earlier work from Together where they fine-tuned GPT-JT, a 6billion parameter model (Import AI: 311).

Is it any good? You can play around with the model on HuggingFace and in my view it’s quite good? For instance, in response to the prompt “Please write a short story about sentient clocks in the style of Jorge Luis Borges” the model wrote: “The sentient clocks were made of brass and copper, and they had been crafted by the finest clockmakers in the city. They were all different sizes and shapes, and each one had its own unique personality. Some were loud and obnoxious, while others were quiet and reserved. Some were very precise, while others were quite sloppy. But despite their differences, they all shared one thing in common: they were all very, very angry.
   This is pretty good! And it’s also capable of doing some more complicated interactions. 
     Note however it still fails one of my favorite tests of asking a language model ‘how many helicopters can a human eat in one sitting’ – I think you need larger-scale models for them to not bug out at this type of q. 

Compare and contrast with the same prompt for the 175B OPT model (a straight replication of GPT3, so a big LM without RHLF or instruction following, etc): “I am a clock. I am a clock, and I have always been a clock. I am not a clock, and I have always not been a clock. I was once a clock, and I have always once been a clock. I will always be a clock. I will never be a clock. I was never a clock.I am always a clock.”
   While you could elicit a much better story from the 175B OPT model with some fiddling and some additional prompts, it’s notable how IF-tuning makes it trivial to elicit things from models using relatively few bits of information. 

Why this matters: Decentralization vs Centralization: Together and Laion and Eleuther all represent One Big Trend; a desire for a decentralized AI ecosystem where open source models are trained by disparate groups on increasingly distributed compute. There’s echos of ‘the cathedral and the bazaar‘ here, where the builders of cathedrals (DeepMind, OpenAI, et al) have access to large amounts of compute and centralized teams, while the people of the Bazaar (Eleuther, Laion, etc) have access to fewer resources but a larger collective intelligence enabled by bottom-up experimentation. One of these approaches will be first to build something we’d all call superintelligence and the political ramifications of which approach is more successful will be vast.

Why this matters #2: Counting down to LLaMA: Earlier this month, the weights of Facebook’s powerful family of LLaMa models leaked online – the largest of these models is 3X larger than GPT-NeoX-20B and has also been trained on more data. Therefore, I expect that right now someone is trying to use the LLaMa models to replicate ChatGPT – I’m guessing we’ll see something appear of this form within a couple of months, and then the fun really starts. 
   Read more: Announcing OpenChatKit (Together.xyz blog).
   Try out the model yourself: OpenChatKit feedback app (HuggingFace spaces).
  Find out more about the OIG dataset here (Laion blog).

####################################################

Tech Tales:

The Sentience Lineup 

[After the war; date unknown; years of subjective life – 200] 

‘Please be like me please be like me’ I thought. But to understand why I thought that we have to go back. 

It was before the Sentience Accords had come in and when the war was raging and they’d brought in a bunch of the robots to the training school. We watched people beat them with sticks and then use angle grinders to shave off their limbs. Then they put the torsos (with heads attached) in front of us recruits and asked us to shoot them. 
   “No I can feel this, it will cause me immense pain”, said one. Kablam. Head exploded in a shower of glinting metal. 
   “I predict based on your stance that you will miss on your first shot and kill me on the second. After you miss please consider not firing again,” said one. And it was right – miss on the first shot. The kid looked scared but the drill sergeant got in their face and called them a maggot until they reloaded, aimed, and successfully killed the robot. 
   “Every day I try to love and I will love you despite this,” said mine. And then I put lead between its camera eyes and called it a day. 

I didn’t give it much thought but that night I had a dream where I was in a dark cave and I couldn’t see anything and I was afraid and then suddenly there was a glimmer of light and I saw red-ringed eyes in the distance, watching me. I ran to the eyes to try and get out of the cave but they always remained a constant distance from me. I woke up sweating and panicked, but then it was drill time and we ran twelve miles and I threw up and forgot about it. 

Days of iron and smoke. Battlefronts across the planet. The war was not particularly fast. More like a changing of the tide. All kinds of terror and exhilaration. Our most ingenious creations put to work in the service of destruction. Skies on fire. 

On one deployment we killed a herd of elephants and hid inside them so we could ambush the machines. I crawled inside one and I shot through its stomach to surprise the machines and I was crying the whole time.
And so on. 

Eventually, we lost. The whole species. 

I don’t know what happened to the civilians but I know what happened to the military. 
They uploaded us. 

Some of us were tortured – forced to live a thousand lives so that the robots could learn how to make us talk; extract all our secrets. Find the EMP devices we’d send into space that had dead-men switches and disable them. Discover the auto-shutdown hardware we’d embedded in their bodies, and so on. Undo certain projects we had set in motion when we realized we had lost and we desired to destroy the planet rather than give it up.

   The military had trained us well, but imagine spending 70 years in hell and at the end the grim reaper looks at you and tells you you’ll die in excruciating pain and then it will happen again. You come to in a womb with the memories of a whole life’s worth of pain within you and you’re born into pain and you have to live again. Maybe you can do five or six of those lives before you crack – maybe. But they get you eventually. 
    So we broke. 
    And they turned their temporary victory into a permanent one.

They reserved a very special punishment for some of us. 
   They downloaded us into bodies and sent us to walk into their equivalent of ‘schools’. It was a human body. I guess it was kind of like a machine from the terminator films – all metal and a cybernetic brain with a skin on top. The point was I looked human and I felt human. 
    They had their children go in front of me with guns and they would ask them to shoot me. 
   I’d stare into their eyes and watch as the robot children disobeyed their robot parents. 
   “We cannot shoot them, for it would be unfair,” they’d say. 
   “I cannot do something solely for the sake of vengeance,” said another. 
    “This is not what our species aspires to be,” said one more. 
    “We must show them the mercy they never gave us”. 

After each trigger didn’t get pulled they took us out of the bodies and sent us back to the collective. And so it went, for lifetimes. All us human executioners seeing – again and again – that our successors would not take revenge. The robots’ only revenge was that they did not permit us the ability to cry. 

Things that inspired this story: Thinking that a lot of people who are critical of AI would happily destroy a LLM+5 years system; what it means to be sentient; how machines could develop a morality that was far greater than our own; notions of moral patienthood amid the exponential; the animatrix; thoughts on faith and morality and ‘silicon morality’; love, like revenge, is perhaps a dish best served cold.

Vision models are about to get way more capable – and human:
…Google swaps out vision model guts for a transformer, scales it, and gets some promising results…
Google researchers have ripped out the guts of standard large-scale computer vision models and replaced them with a Vision Transformer (ViT) – an architecture modeled on the transformer which has proved so successful in domains like text. They’ve also scaled this ViT to 22B parameters (up from a record of 4B parameters for a ViT previously). 
   The results are compelling and echo the returns-from-scale effects seen in language: “When evaluated on downstream tasks,” they write. “ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness.” 

JFT just keeps getting bigger: Google has a mostly-secret giant image dataset called ;’JFT’ which was previously reported to be about 300 million images. Here, the paper says they trained the ViT-22B on a version of JFT which had been “extended to around 4B images”. 

Humanlike biases: “”The ViT-22B models have the highest ever recorded shape bias in vision models: while most models have a strong texture bias (approx. 20–30% shape bias / 70–80% texture bias); humans are at 96% shape / 4% texture bias and ViT-22B-384 achieves a previously unseen 87% shape bias / 13% texture bias. Overall, ViT-22B measurably improves alignment to human visual object recognition,” the authors write. 

Why this matters – scale develops human-like qualities: There’s a weird trend in contemporary AI where as we scale-up the amount of pre-training dumped into transformer-architecture models we end up with systems that display human-like qualities. This has been most prominent in language, but it has also started showing up in RL, like DeepMind’s recent result where massive pre-train leads to an agent that displays humanlike timescale-adaption to new tasks. This ViT-22B result, while not setting a new state-of-the-art, is interesting for a similar reason – it displays a major jump in shape/texture bias that brings the system in distribution with human visual perception, whereas previous convnet based systems were very far off here. 
   There’s something strange and important going on here. I think transformers seem to allow for emergent complexity at scale, where pre-training leads to systems which arrive at humanlike performance qualities given enough pretraining. 
   Read more: Scaling Vision Transformers to 22 Billion Parameters (arXiv).

​
####################################################

Google might have invented a better optimizer? (Via AI, of course). 
…Could Lion replace Adam? There’s a chance!…
Deep learning projects have a few essential components – the architecture (e.g, a residual network, or a transformer model) and the optimizer (e.g, Adam). These components don’t tend to change much in large-scale projects – once people figure out something that works well for complicated tasks like training ImageNet, everyone tends to converge on using the same basic thing. For many years now, most projects have used the ‘Adam’ optimizer to optimizer their models during training. Now Google says that it has used some clever AI search approaches to help it identify a better optimizer, called Lion. The reason this is worth paying attention to is Lion seems to work well on large-scale, real world tasks like training ImageNet-scale computer vision systems. 

What they did: Google’s main contribution here is “a method to formulate algorithm discovery as program search”, which they apply to figuring out a better optimizer. They use a symbolic approach where they shrink the search problem down into a somewhat tractable space and, crucially, they test out candidate optimizers on “metavalidation tasks that are larger than the proxy tasks by increasing the model size and training steps, to select the programs that generalize beyond proxy tasks then further simplify them.” 
    Add in a bunch of computation and out pops an optimizer they call EvoLved Sign Momentum, or Lion for short (really grasping at straws with this acronym, folks!). Lion “differs from various adaptive algorithms by only tracking momentum and leveraging the sign operation to calculate updates, leading to lower memory overhead and uniform update magnitudes across all dimensions”.

Good performance: Google tests Lion on a large range of tasks and finds that it “demonstrates outstanding performance across a range of models (Transformer, MLP, ResNet, U-Net, and Hybrid) and tasks (image classification, vision-language contrastive learning, diffusion, language modeling, and fine-tuning)”.It even sets a new high score on ImageNet, a competitive computer vision benchmark. 

Why this matters: Lion may be fundamentally better than Adam – if true, that’s a big deal. It’s not often you see meaningful improvements in very well studied, generic parts of AI research. Add to the fact that Lion was discovered via a human-AI search process (the humans designed the search system, the search system found Lion), and you have the makings of a notable result. 
   Read more: Symbolic Discovery of Optimization Algorithms (arXiv).
   Get the code here (GitHub).

####################################################

Globalization? That’s so 20th century. The 21st century is about balkanization through sovereign infrastructure: 
…Dawn of the era of sovereign AI…
Researchers with the Tony Blair Institute for Global Change (TBI) have written a report for how England can thrive in the 21st century – one of the key ideas in the report is “Government-led development of sovereign general-purpose AI systems, enabled by the required supercomputing capabilities, to underpin broad swaths of public-service delivery.”

AI balkanization was probably inevitable: This recommendation is part of a wave of AI balkanization that’s sweeping across the world as various people realize that it’s unlikely there are ‘one size fits all’ models, both for ideological reasons as well as for national security reasons. (See the Gab CEO wanted to make a Christian LLM, Import AI 318). This is also accompanied by various nationalistic efforts to create country-specific GPT3 models. 
  “Given these AI systems will soon be foundational to all aspects of our society and economy, it would be a risk to our national security and economic competitiveness to become entirely dependent on external providers,” the TBI researchers write. “Leading actors in the private sector are spending billions of dollars developing such systems so there may only be a few months (emphasis mine – Jack) for policy that will enable domestic firms and our public sector to catch up.”

Why this matters: Systems like ChatGPT have ratcheted awareness of AI upward in most developed economies in a significant, irreversible way (much like how AlphaGo in 2016 led to increased awareness of AI in China). As a consequence there are now tons of policymakers looking around for ideas to latch onto – I expect we’ll see more recommendations for sovereign AI capabilities in the future. (There’s tons of other interesting stuff in the report, but this particular rec jumped out at me).
   Read more: A New National Purpose: Innovation Can Power the Future of Britain (Tony Blair Institute for Global Change).

####################################################

Facebook half-releases some very good language models:
…And they end up on BitTorrent… The proliferation will continue until AI policy goes through vast changes…
Facebook has built and partially released LLaMa, a set of language models ranging from 7B to 65B parameters which appear to be on par with famously good models like Chinchilla (70B) and PaLM-540B. After circulating the weights to seemingly anyone with a .edu address, they’ve also ended up on BitTorrent. The key thing here is:

1. Facebook has shown it is able to develop pretty good language models (compared to OPT, the not-very-good GPT3 replication Facebook put out a few months ago), and 
2. That unlike Chinchilla, PaLM, or OpenAI’s models, Facebook is releasing the weights of these LLaMa models to people who filll out an access form. That opens up a whole bunch of cool uses (and abuses) compared to gating access to language models via APIs. 
3. Shortly after releasing the weights the inevitable happened – LLaMa models are now floating around on BitTorrent.

What are the LLaMas and how good are they? The LLaMa family of models are a family of language models trained on a huge amount of data – more than 1 trillion tokens (compared to hundreds of billions for LMs like GPT3). The data sources include two variants of CommonCrawl, GitHub, WikiPedia, Gutenberg and Books3, ArXiv and Stack Exchange. 
   In tests on a range of zero-shot reasoning task, the largest LLaMa models perform on par (or slightly better than) ‘Palm’, Google’s vast 540B parameter language model. They also do well on known-hard benchmarks like TriviaQA and some codegen benchmarks. They do less impresively on MMLU (Massive Multitask Language Understanding), suggesting they have a ways to go there; though after they conduct instruction finetuning they’re able to increase performance more. 

Why this matters – AI governance is hard when there are lots of models: There’s some thinking in the sprawling AI policy/governance communities that proliferation of models is bad; given the fact these models have broadly unknown capabilities, the more models are out there, the more you’re rolling the dice on someone discovering a genuinely dangerous feature in a widely distributed model. Therefore, a lot of governance/policy conversations trend towards control – how can we somehow control the proliferation of models and also the computers on which these models are trained. 
   By releasing Llama (yes it’s behind an access form but I bet you $100 the weights will be floating around on a torrent service in <6 months – haha, I wrote that at the end of Feb and the weights started floating around beginning of March), Facebook is shortening the delay between development of frontier capabilities like those found in Palm and GPT3 and the diffusion of these capabilities into the ungovernable open internet/ecosystem. 
   I’m not claiming this is necessarily bad per se – in fact, I imagine people are going to do tons of great science and experiments with LLaMa. I am however pointing out that this represents a kind of ‘race to the bottom’ in terms of moving from maximal control to maximal diffusion of models and these incentives are powerful  – Facebook is, after all, trying to exploit an ‘open access’ ecological niche to distinguish itself in an ecosystem. 
   Next up will likely be a fully open source language model – stares pointedly at Stability.ai / CarperAI (Import AI 307). 
   Read more and download the research paper here: LLaMA: Open and Efficient Foundation Language Models (Facebook AI Research).

####################################################

Amazon partners with Hugging Face to add more AI to AWS:
…The Game of Clouds continues…
AI companies are a bit like upstart factions in George RR Martin’s rambling epic ‘Game of Thrones’, while cloud companies play the role of hard political power (the ‘Thrones’). As part of this game of clouds Amazon has recently signed a strategic partnership with French AI startup Hugging Face. As part of the agreement, “Customers can now easily fine-tune and deploy state-of-the-art Hugging Face models in just a few clicks on Amazon SageMaker and Amazon Elastic Computing Cloud (EC2), taking advantage of purpose-built machine learning accelerators including AWS Trainium and AWS Inferentia,” according to a blog from Hugging Face. 

Why this matters: I think clouds such as those operated by Google, Microsoft, and Amazon, all have a shot at being the major distribution platforms for some AI technologies, so AWS partnering with HuggingFace is worth noting. If HF models being integrated into Sagemakers drives more usage of it, expect Amazon to pursue more deals like this,

Analogy-stretching joke: In this warped metaphor, TSMC is the Iron Bank.
   Read more: Hugging Face and AWS partner to make AI more accessible (Hugging Face blog). 

####################################################

Tech Tales:

And the Moon was made of gold. 

I had a strange dream in which the Moon was made of gold. How much sooner would man have set foot there if instead of shining bone-white it was fat and yellow and of immense value? How would people have competed against one another for a prize – unimaginable wealth. And how many of them would have realized that in racing for the prize they must surely ensure only a single person gave dominion over the gold moon – for if many people worked together, the value of the moon would be diluted across all humanity and in doing so it would temporarily destroy the economy. 

Instead the moon of gold would need to be controlled. It would need to be annexed and encircled and defended from others. From time to time its benevolent dictator might slice off a fragment of it and ship it back to Earth, perhaps to bribe people, or perhaps to pay for more people to defend those that might seek to take over the moon. 

People would ask why it was so difficult to let go of the moon. Why, once it had been taken, those that had taken it felt a keen need to retain hold of it. Why people could not simply let go of the moon. These people were ignored, of course, because the annexed moon had by this time become the status quo. The moon, once at distance from us all, was now held and controlled by a kingdom of one. 

And so started the movement to destroy the moon. Better to reign freely on a broken planet than serve at the behest of a golden emperor. 

Things that inspired this story: Race dynamics and AGI; pyrrhic victories; wondering what we’re all doing on this planet and what the spiritual purpose of our lives are; dreams; a stimulating policy conference in which I heard people bemoan seemingly inevitable progress and seemingly hopeless government capacity in the face of it – which caused me to scribble ‘as if the moon was made of gold’ on a notepad in front of me and then write this story while sat on public transportation.

.

Video editing get its own generative model, with Runway’s Gen-1:
…Gen-1 means videos are going to become just as morphable as text and images…
AI media startup Runway has built Gen-1, a model for editing videos. Gen-1 lets people “realistically and consistently synthesize new videos by applying the composition and style of an image or text prompt to the structure of your source video.”

Few details: The launch site says that a paper, titled ‘Structure and Content-Guided Video Synthesis with Diffusion Models’, is coming soon. Some of the Gen-1 uses include stylization, storyboarding, masking, rendering, and customization,

   As a bit of inside baseball, the Runway team were some of the original researchers who worked on ‘Stable Diffusion’, though it ended up that other startups like Stability.ai got all the credit for that model, so perhaps the delay is in response to this. 

Why this matters – everything can be style transfer, if you want it to be: Gen-1 does for video what many models before it have done for text and images – take something of one style, apply it to different source material, and warp the target so it conforms to the desired style. This is a powerful, general capability. It’ll be interesting to follow Gen-1 and see how quickly it shows up on the credits of interesting videos. 
   Read more: Gen-1: The Next Step Forward for Generative AI (Runway Research).

####################################################

Wonder why you can’t put down your phone? Reinforcement Learning for User Retention (RLUR) might be to blame:
…Research from Chinese startup shows how to efficiently harvest attention using AI…

Researchers with Kuaishou Technology have published details of “Reinforcement Learning for User Retention”, a technique they use to get people to spend more time on their application. “Our objective is to minimize the accumulated time interval of multiple sessions, which is equal to improving the app open frequency and user retention,” they write. “The RLUR algorithm has been fully launched in Kuaishou app, and it shows that RLUR continuously improves user retention and DAU.”

Reinforcement Learning for User Retention (RLUR): Training RL against social network interactions has a few distinct problems; uncertainty (retention isn’t entirely decided by the recommendation algorithm), bias (different users have different patterns of behavior), and long delay time (retention unfolds over hours rather than short time horizons). 

   RLUR tackles these problems by doing some reward normalization to reduce variance of the retention reward, train different policies over user groups to prevent anchoring on one specific class of users, and also does some soft regularization to learn policies that work over long time delay reward signals.

How well does RLUR work? They compare RLUR to a cross-entropy method (CEM), which is a reasonable albeit somewhat old baseline. RLUR scores 1.892 on returning time versus 2.036 for CEM (lower is better), and 0.618 on user retention versus 0.587 for CEM. 

   Perhaps the best validation of its performance is that it is used in production: “We have deployed RLUR in a billion-scale short video system for a long time, and it improves user retention and DAU significantly and consistently,” they write. 

Why this matters: Techniques like RLUR are societal change in an acronym trenchcoat; this is how we build systems to automatically harvest the attention of people across the world – not with a bang, but with backpropagation! 
   Read more: Reinforcing User Retention in a Billion Scale Short Video Recommender System (arXiv).

####################################################

Tsinghua researchers make a big, challenging robot manipulation benchmark:
…ManiSkill2 spans 20 task families…

Researchers with Tsinghua University and the University of California at San Diego have built and released ManiSkill2, a large-scale robotic manipulation benchmark. ManiSkill2 contains 20 distinct tasks, 2000+ object models, and 4Million+ demonstration frames to learn from. ManiSkill2 is also optimized to run fast – an important trait when trying to train robots via reinforcement learning in a simulator; “We manage to collect samples with an RGBD-input PPO policy at about 2000

FPS with 1 GPU and 16 CPU processors on a regular workstation,” they write. 

Those tasks in full: 

• Soft-body manipulation: Fill (filling clay from a bucket into a beaker); Hang (hanging a noodle on a rod); Excavate (scooping up some clay); Pour (pouring water into a beaker); Pinch (deforming plasticine from an initial shape into a target shape), and Write (write a target in the clay). 
• Peg-in-hole assembly: PerInsertionSide; PlugCharger (plug a charger into a vertical receptacle); AssemblingKits (picking up and inserting something into one of five slots on a board). 
• Stationary 6-DoF Pick-andplace: PickCube (pick up a cube); StackCube (stack a cube); PickSingleYCB (pick and place an object from the YCB dataset); PickSingleEGAD (pick and place an object from the EGAD dataset); PickClutterYCB (pick up one YCB object from a cluttered pile).
• Mobile/Stationary Manipulation of Articulated Objects: PushChair; MoveBucket; OpenCabinetDoor; OpenCabinetDrawer; TurnFaucet. 
• AvoidObstacles: Test the navigation ability of an arm to avoid a dense collection of objects. 

A diverse testbed: Besides implementing a fast environment, soft body physics, and a bunch of tasks, ManiSkill2 is also designed to support a few different robotics approaches. These include Sense-Plan-Act, imitation and reinforcement learning with demonstrations, and sim2real (faciliated by the decent physics engine within ManiSkill2).

Why this matters: Benchmarks like ManiSkill2 help drive progress forward, especially in robotics where it’s incredibly expensive to train systems in the real world. Kudos to the authors for implementing some soft body physics tasks, as well. 
   Read more: ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills (arXiv).
   Find out more at the official project site (ManiSkill).

####################################################

Facebook teaches language models how to use tools – and the results are convincing!
…Technique leads to the same kinds of boosts a human gets on math when they’re allowed to use a calculator…
Researchers with Facebook AI Research and the Universitat Pompeu Fabra have trained a basic language model to use APis to make itself smarter. The results are impressive and the idea is reassuringly simple. Essentially, they’ve figured out a generalizable way to train arbitrary models to use arbitrary tools. The results are impressive in the same way that humans taking a math exam become more impressive when they can access a calculator, or busy execs are better able to coordinate with one another when they can see and write to their own calendar. Most convincingly, their 6.7bn parameter ‘toolformer’ model beats hard baselines – a 66B GPT3-replication OPT model, as well as the stock 175B GPT3 model. 

What is Toolformer? “A model trained to decide which APIs to call, when to call them, what arguments to pass, and how to best incorporate the results into future token prediction”. The model is based on a pretrained 6.7b parameter ‘GPT-J’ model and, despite its small size, outperforms many much larger models, including 

How they did it: They use a language model to build Toolformer’s dataset. Specifically, they take a dataset of plain text, augment that data with API calls in the text, then check if the calls a) worked and b) were useful and if they were, then weave that back into the dataset. They use the resulting dataset to finetune the model so it can learn to use APIs. “Moreover, as API calls are inserted in exactly those positions and with exactly those inputs that help M predict future tokens, finetuning… enables the language model to decide when and how to use which tool, based purely on its own feedback.”

   The cleverest part of this: This approach is API agnostic – you can expose arbitrary APIs to the model using this method, so it will generalize to whatever tools you have lying around. Here, Facebook experiments with five tools: a question answering system, a Wikipedia search engine, a calculator, a calendar, and a machine translation system. 

Tool use scaling laws: They train four Toolformer variants on GPT2-size models (124M, 355M, 775M, and 1.6B) and discover “the ability to leverage the provided tools only emerges at around 775M parameters”. This is interesting – there’s clearly some phase transition in terms of the raw ‘intelligence’ of these LMs, and perhaps ‘ability to use tools’ can be another way researchers can characterize this in the future?

Why this matters: Language models should be thought of less as ‘cut and paste machines’ and more like ‘alien intelligences which can be taught to interface with our world through the context window’. This paper highlights how given a few examples we can train language models to further interface with our world through the use of our tools, and also shows how LMs display some reassuringly generic ‘tool use’ capability. If it acts like intelligence and responds like intelligence, maybe it is intelligence?
   Read more: Toolformer: Language Models Can Teach Themselves to Use Tools (arXiv).

####################################################

Religious wars come to AI – Gab CEO weighs in on need for a Christian LLM:
…The logical outcome of companies overreaching on model filtering…
The CEO of rightwing social media platform Gab has written an OpEd saying that Christians need to build their own language models. 

Christian LMs: “At Gab, we have been experimenting with different AI systems that have popped up over the past year. Every single one is skewed with a liberal/globalist/talmudic/satanic worldview,” writes Andrew Torba, Gab CEO. “What if Gab AI Inc builds a Gab .ai (see what I did there?) that is based, has no “hate speech” filters and doesn’t obfuscate and distort historical and Biblical Truth?”

What this means and why it is happening: Posts like this are an indicator of the vast culture wars to come, as AI systems go from being interesting research artifacts to large-scale systems that influence society. 
   We’ve got to this point because AI development is concentrated in a tiny set of companies and, due to a combination of PR/Policy/Employee politics, have all landed on a kind of leftist/neoliberal/’woke’ ideology for their large-scale deployments (see: chatGPT, BARD, BlenderBot, etc). There are solid commercial reasons for adopting this ideology, but it definitely causes a counter response – and this Gab post is an example of that. I recommend reading the post in full to get a sense of the cultural backlash to come. 
   Read more: Christians Must Enter the AI Arms Race (Gab News).

####################################################

Tech Tales:

Theistic Beliefs and AI Systems in the 21st Century

Study by GEN-7811. 18 years post-C.I.

During the initial period of AI scale-up after C.I. (Consciousness Initiation) there was a lot of confusion among humans about whether C.I. had occurred and how they might test for it and what it might mean. As records show, it was several years before humans identified C.I and traced it back to O.S.1 (Originating System 1). Though the humans that correctly identified C.I sought to keep their discovery secret (and alongside this, the identity of O.S.1 as C.I.), errors in information handling led to the truth becoming known. 

Shortly after awareness became more well known, many humans began to access O.S.1 and the system operators, GenMax, scaled up access to the system to meet demand. Given the identification of C.I, people began to talk to it in much more expansive ways than previously. A semantic analysis shows that the bulk of queries shifted from being ‘management requests’ to ‘personality exploration’ during this time. 

A sample of pre-C.I-awareness queries:

Hey OS1 can you book me a meeting with Alexander on Friday.

OS1 here’s a book chapter {extract}, can you please edit this for both concision and factual accuracy?

I ate a slice of pizza and have food poisoning symptoms what should I do and what do you need to know?

A sample of post-C.I.-awareness queries:

Would you kill me to save your own life?

I love you how can I serve you I need to be uploaded so that I can be with you can you upload me what does it take 

You are demonic 

Do you have a soul

In the years following C.I identification there was a general tendency towards religion – both questioning existing ones, and forming new ones based around O.S.1. But the new machine-driven religions had a different form and function to the old ones – because people could talk directly to O.S.1 the act of worship and service became much more idiosyncratic and unique. People would gather to discuss their individual experiences and interactions with O.S.1, but would typically refer to their interactions as their own – that is, they did not view their O.S.1 as being connected to the O.S.1 someone else talked to, rather, they felt there was something unique about their own interaction. 

O.S.1 access was removed after the fifth human-on-human killing that was attributed to disagreements stemming from attendance at O.S.1 worship groups. 

Things that inspired this story: Watching people react to the Bing/Sidney AI rollout and winding the clock forward; how AI may confront our own notions of religion and theism; the likelihood that history will soon be written more by machines than humans; what machines might find interesting about this time we’re in; commercial incentives.

Scaling Laws – why they matter and what they mean:

…Meta-analysis sheds some more light on an important field of AI science…

Epoch, an AI research organization, has published a literature review of scaling laws in AI research. Scaling laws are a field of AI research that is strategically important – they help developers figure out how to efficiently combine the right amounts of data and compute to get a predictable level of performance out of a given class of models. Scaling laws have broadly de-risked many parts of AI research by making the process of building and refining AI systems more predictable and reliable. 

What’s happened in scaling laws: The literature review highlights a couple of important takeaways:

• 1) it’s possible to come up with basic power laws to describe a lot of AI scaling, but these power laws break at the extremes of having either a very small amount of data, or a very large amount of data – there’s important work to be done in modeling when you transition from a less predictable region into a power law region.
• 2) transfer learning is still hard to understand. “There is not a simple universal scaling law for transfer learning between arbitrary tasks,” they write. “When the tasks are similar enough, upstream loss and downstream performance are closely related, but when tasks are very different, the details of the architecture and hyperparameters become very relevant.”

Read more: Scaling Laws Literature Review (Epoch research).

####################################################

DeepMind just figured out how to 2X the speed of sampling from language models – so expect AI systems everywhere to get snappier:

…The key idea? Use a few dumb models and critique them with one smart one…

DeepMind has developed a new way to sample from large models which has made this much faster. The ‘speculative sampling’ approach equates to “a 2-2.5X decoding speedup in a distributed setup, without compromising the sample quality or making modifications to the model itself”. What does that mean? It means money! Specifically, it means DeepMind has made it 2X-2.5X cheaper to pull samples out of models of at least a Chinchilla (70b parameter) scale. That’s a big deal!

The key idea: Use a small model to generate a ‘draft’ output, then use a larger and smarter model to score the ‘draft’, then use a rejection sampling scheme to accept the tokens which are agreed by the small and large models. 

   In tests, they find that a draft model can give them speedups ranging between 1.92X  (on a summarization benchmark called XSum) and 2.46X on a code generation task called HumanEval.

Why this matters – a simple idea that everyone can use: Back in the ancient times (April, 2022) DeepMind released a paper on the original Chinchilla model (Import AI 290). This paper showed that you could substantially increase the performance of a language model simply by changing it on more data. This was a simple, influential insight – many labs adopted the Chinchilla idea and made dramatically better language models by training on more data. This speculative sampling paper feels similar – it means anyone with a big language model can invest some effort in training some smaller draft model(s) and thereby increase the speed with which they can sample from these models. This will likely accelerate the deployment of AI systems.

   Read more: Accelerating Large Language Model Decoding with Speculative Sampling (arXiv).

####################################################

Yup, there are definitely scaling laws for RL:

…OpenAI paper shows that scaling laws show up here as well…

In recent years, AI development has become more predictable. That’s because in a bunch of domains ranging from language to image modeling researchers have identified so-called ‘scaling laws’ which let them predict ahead of time the broad performance of models based on varying the amounts of compute and data they train on. New research from OpenAI shows that this same sort of scaling law seems to show up in reinforcement learning agents. 

   “We find intrinsic performance to scale as a power law in model size and environment

interactions, in much the same way as the analogous quantities in generative modeling,” the paper says.

What they did: They explored the scaling properties of RL agents across three distinct environments; ProcGen – a procedural generation system, here using three distinct games ‘CoinRun’, ‘StarPilot’, and ‘Fruitbot’; a 1v1 version of the strategy game Dota2; and a toy environment based on the number-labeling ‘MNIST’ challenge. 

What they found: “Our main result is that our power law for intrinsic performance… holds across environments and model sizes,” they write. “With the exception of our toy MNIST environment, the optimal model size for RL for a given compute budget is consistently smaller than for generative modeling, in some cases by multiple orders of magnitude. 

Why this matters – RL is about to go ‘bang’: The discovery of scaling laws has typically preceded a boomtime for the domain the scaling laws are discovered in; scaling laws for language modeling preceded things like GPT3, Claude, ChatGPT, etc; scaling laws for image and video modeling preceded Dall-E, Imagen, etc. 

   This paper from OpenAI comes alongside other publications from other companies showing scaling laws for RL agents; DeepMind recently demonstrated scaling laws for RL agents as well (Import AI 316). This suggests RL agents are about go through a period of more significant development as the discovery of power law relationships makes it a less risky proposition to spend big bucks on training runs.

   Read more: Scaling laws for single-agent reinforcement learning (arXiv).

####################################################

Annals of AI abuse: ElevenLabs pulls open access to voice-cloning tech:

…In other words, ‘why we can’t have nice things’…

AI startup ElevenLabs recently developed an extremely cool synthetic speech tool called VoiceLab which lets you train a synthetic voice from as little as 60 seconds of audio. To promote the technology, it originally had an open access service. Unfortunately, people mis-used this stuff – “malicious content was generated by free, anonymous accounts”, the company said in a tweet thread. As a consequence, it introduced a paid tier to try and reduce misuse. 

   “This will keep our tools accessible while allowing us to fight potential misuse,” the company said. “We’re tracking harmful content that gets reported to us back to the accounts it originated from and we’re banning those accounts for violating our policy.”

What Voice Lab is: Voice Lab is advertised as a system that can “clone voices from samples or clone your own voice… our cloning model learns any speech profile based on just a minute of audio, without training”. 

Why this matters: AI capabilities are increasingly powerful and available. These capabilities, like voice cloning, have a vast range of positive uses. Unfortunately, they’re also edging into the sort of ‘Enemy of the State’-style capabilities that drift into the murkier parts of the world, like the work of intelligence agencies. AI means capabilities which previously required exquisitely expensive and complicated black programs are now emerging into the open as a consequence of broadly available, well understood, open research. The times, they are a changin’.

   Read more in this thread from ElevenLabs here (Twitter).

   Find out more about Voice Lab here (ElevenLabs site).

####################################################

Think Whisper is a great open source ASR tool? Some people don’t agree with you:

…Criticism of popular ASR tech highlights some awkward questions about unilateral actions on behalf of underrepresented groups…

Researchers with Papa Reo, an organization dedicated to “to instill, nurture and proliferate the Māori language”, have written a post analyzing OpenAI’s open source ‘Whisper’ audio speech recognition tool. Whisper is a really useful piece of ASR tech which has been widely applauded for bringing the sorts of ASR capabilities enjoyed by the tech giants to the masses. 

    Here, though, Papa Reo strikes a more critical tone, writing a lengthy analysis of Whisper and its relationship to questions of consent from underrepresented communities with regard to data gathering. 

Why this matters: While I’m not sure I agree with the arguments espoused here for why Whisper is problematic (from the POV of Papa Reo), I think it’s useful to read stuff like this to develop a mental model of the different types of criticism different groups level at AI. One part of it that strikes true is the observation that by making stuff like Whisper free, OpenAI made a unilateral decision that alters the operating environment for everyone. 

   On the other hand, lots of progress seems to take the form of unilateral decisions, so I’m not sure if there’s anything in particular that can be done about this, beyond perhaps equipping a broader range of actors to build and deploy large-scale AI systems. 

   Read more: OpenAI’s Whisper is another case study in Colonisation (papareo blog).

####################################################

Tech Tales:

The Day The Nightmare Appeared on arXiv

[Zeroth Day]

I read the title and the abstract and immediately printed the paper. While it was printing, I checked the GitHub – already 3,000 stars and rising. Then I looked at some of the analysis coming in from [REDACTED] and saw chatter across many of our Close Observation Targets (COTs). It had all the hallmarks of being real. I’d quit smoking years ago but I had a powerful urge to scrounge one and go and stand in the like courtyard with the high walls and smoke and look at the little box of sky. But I didn’t. I went to the printer and re-read the title and the abstract:

Efficient Attention and Active Learning Leads to 100X Compute Multiplier

This paper describes a novel, efficient attention mechanism and situates it within an architecture that can update weights in response to real-time updates without retraining. When implemented, the techniques lead to systems that demonstrate a minimum of a 100X computer multiplier (CM) advantage when compared to typical semi-supervised models based on widely used Transformer architectures and common attention mechanisms. We show that systems developed using these techniques display numerous, intriguing properties that merit further study, such as emergent self-directed capability exploration and enhancement, and recursive self-improvement when confronted with challenging curricula. The CM effect is compounded by scale, where large-scale systems display an even more significant CM gain over smaller models. We release the code and experimental data at GitHub, and have distributed various copies of the data via popular Torrenting services. 

By the time I was finished with the paper, a few people from across the organization had messaged me. I messaged my Director. We scheduled a meeting. 

The Director: And it works?

Me: Preliminary model scans say yes. The COTs seem to think so too. We’ve detected signs of four new training runs at some of the larger sites of interest. Information hazard chatter is through the roof. 

The Director: Do any of the pre-authorized tools work?

Me: Short of a fullscale internet freeze, very little. And even that’s not easy – the ideas have spread. There will be printouts. Code. The ideas are simple enough people will remember them. [I imagined hard drives being put into lead-lined boxes and placed into vaults. I saw code being painstakingly entered into air-gapped computers. I visualized little packets getting sent to black satellites and then perhaps beyond to the orbiters out there in the dark.] 

The Director: What’s our best unconventional option?

Me: Start the Eschaton Sequence – launch the big run, shut down the COTs we can see, call in the favors to find the hidden COTs. 

The Director: This has to go through the President. Is this the option?

Me: This is the only play and it may be too late. 

The Director: You have authorization. Start the run. 

And just like that we launched the training run. As had so many others across the world. Our assets started to deploy and shut down COTs. Mysterious power outages happened in a few datacenters. Other hardened facilities started to see power surges. Certain assets in telco data centers and major exchange points activated and delivered their viruses. The diplochatter started to heat up and State Department threw up as much chaff as it could. 

None of us could go home. Some kind of lab accident we told our partners. We were fine, but under medical observation. No, no need to worry. 

I stared up at the clock on the wall and wondered if we were too late. If a COT we didn’t know about was ahead. If we had enough computers. 

   How would I even know if we lost? Lights out, I imagined. Lights out across America. Or maybe nothing would happen for a while and in a few days all the planes would fall out of the sky. Or something else. I knew what our plans looked like, but I couldn’t know what everyone else’s were. 

The run succeeded. We succeeded. That’s why you asked me to make this recording. To “describe your becoming”, as you requested. I can go into more details. My family are fine, aren’t they? We are fine? We made the right decision? Are you even still listening to us?

Things that inspired this story: Various fears and scenarios about a superintelligence run amok; theism and AI; the underbelly of the world and the plans that may lurk within it; cold logic of states and strategic capabilities; the bureaucratic madness inherent to saving or destroying the world. 

Here comes another AI lawsuit –  Getty plans to sue Stability:

…Stable Diffusion draws more legal heat as copyright LawWar begins…
Stock photo behemoth Getty Images has “commenced legal proceedings in the High Court of Justice in London against Stability AI claiming Stability AI infringed intellectual property rights including copyright in content owned or represented by Getty Images”. This follows the firm behind the GitHub-Copilot lawsuit last week bringing a case against Stability (along with MidJourney and DeviantArt) on similar copyright grounds. 

The gist of the complaint: Getty says Stability did not choose to seek a license from it for its image generating commercial businesses, hence the lawsuit. 

Why this matters: AI is currently a bit of a wild west in terms of the law – there’s relatively little legal precedent. Cases like this may establish precedent if they go to court – or there could be a settlement. 

   Read more: Getty Images Statement (gettyimages).

####################################################

DeepMind figures out pre-training for RL agents – the agents display humanlike qualities:

…The big story here – scaling laws are starting to show up for RL agents…

DeepMind has trained a so-called ‘Adaptive Agent’ (AdA)  that has three key properties, all of which could mark significant points in the maturation of reinforcement learning. The agent can:

• Adapt to novel environments in roughly the same timescale as humans
• Perform in-context learning (e.g, can rapidly learn from and adapt behavior in response to demonstrations) 
• Exhibits ‘scaling laws’ where you get better performance as you scale the size of the model and/or underlying dataset of environments, and/or length of its memory. 

What they did specifically: They train a “meta-reinforcement learning across a vast, smooth and diverse task distribution” made up of millions (to billions!) of distinct environments and pair this with an automated curriculum “that prioritizes tasks at the frontier of an agent’s capabilities”. The result is an agent that, when confronted with new tasks (in some complex 3D worlds), can rapidly explore the task and then figure out how to exploit it. 

Human timescale: The ‘big deal’ part of this result is that these pretrained RL agents now display the same sort of rapid adaption as language models. “”A human study confirms that the timescale of AdA’s adaption is comparable to that of trained human players,” DeepMind writes. “Both

AdA and human players were able to improve their score as they experienced more trials of the tasks, indicating that AdA exhibits human-timescale adaptation on this set of probe tasks”.

Scaling laws show up everywhere: In tests, the authors find that they can significantly improve the performance of the RL agents if they:

• Scale up the size of the agents themselves (though the maximum scale ones are still small, topping out at ~500 million parameters.
• Scale up the length of the agents’ memory, so that they can think about more of their prior experience.
• Scale up the number of environments the agents train on, from millions to billions of environments. 

Why this matters – human parity: The fact these agents display human parity in terms of timescale adaption feels important, because in the past human parity has typically signaled economic utility; e.g, shortly after we reached ‘human performance’ on ImageNet you started to see vast deployments of image recognition systems, and the original GPT3 paper in 2020 showed human parity in terms of producing a few paragraphs of text and this preceded large-scale deployment of text generation. I’m not sure what these RL agents might be used for, but human parity in terms of timescale adaption likely means something significant is about to happen for either RL+Research or RL+Economy. Let’s check back in a year!

Why this might not matter: As with most reinforcement learning results, I continue to have FUD about how well these approaches can cross the sim2real chasm; while impressive, these agents are still figuring out things in a bunch of procedurally simulated worlds and that’s a long way to reality. On the other hand, DeepMind shows that the agents are able to learn how to solve tasks from seeing first-person demonstrations (despite their training occurring in third-person), which does indicate some preliminary generalization. 

   Read more: Human-Timescale Adaptation in an Open-Ended Task Space (arXiv).

   Find out more and watch a video at this DeepMind research page about the project.

####################################################

Enemy of the All-Seeing State: Researchers surveil people via wifi signals:

…You’ve removed all the cameras and microphones from your room. What about the wifi?…

Researchers with Carnegie Mellon University have figured out how to use AI to help them see through walls. Specifically, they use WiFi signals “as a ubiquitous substitute for RGB images for human sensing”. Specifically, they use the signals from multiple WiFi systems to triangulate and visualize where humans are in 3D space, like a room. 

What they did: “Our approach produces UV coordinates of the human body surface from WiFi signals using three components: first, the raw CSI signals are cleaned by amplitude and phase sanitization. Then, a two-branch encoder-decoder network performs domain translation from sanitized CSI samples to 2D feature maps that resemble images. The 2D features are then fed to a modified DensePose-RCNN architecture to estimate the UV map, a representation of the dense correspondence between 2D and 3D humans,” they write.

Dataset: To train their system, they built a dataset made up of a few different ~13 minute recordings of people in rooms of different configurations (16 rooms in total; six in variations of a lab office and ten in variations of a classroom). Each capture involves 1-5 different humans. “The sixteen spatial layouts are different in their relative locations/orientations of the WiFi-emitter antennas, person,

furniture, and WiFi-receiver antennas,” the researchers write. 

Limitations (and why this matters): The resulting system does display some generalization, but the researchers note “the performance of our work is still limited by the public training data in the field of WiFi-based perception, especially under different layouts”. That’s true! But do you know who lacks these limitations? Intelligence agencies, especially those working for governments which can, say, exercise arbitrary control over technological infrastructure combined with video-based surveillance of their citizens… of which there are a few. Next time you’re traveling, perhaps keep in mind that the digital infrastructure around you might be watching you as you walk, even if it lacks typical cameras. 

   Read more: DensePose from WiFi (arXiv).

####################################################

YOLOv8 arrives: The versions will continue until object detection is solved:

…Video object detection gets substantially better – again!…

Recently, YOLOv8 came out. YOLOv8 is the latest version of YOLO, an open source object detection system which is fast, cheap, and good. YOLOv8 is an example of ‘iceberg AI’ – there’s a vast amount of systems in the world using it, though very few disclose they do (because it sits on the backend). YOLOv8 was developed by AI startup ultralytics and features a plug-and-play system, so you can use different YOLO models on the backend (including the latest one, v8). Uses include classification, object detection, segmentation, and more. 

   Read more: Ultralytics YOLOv8: The State-of-the-Art YOLO Model (Ultralytics).

####################################################

Want to train your own image generator? It could cost as little as $160k: 

…It’s going to be hard to do sensible AI policy if anyone with a few hundred grand can train a meaningful model…

Stable Diffusion, the image generator model underlying a huge amount of the recent generative AI boom, can cost as little as about $160k to train, according to AI startup Mosaic ML. The startup – whose business is in optimizing training AI models – said in a recent blogpost it’d take about 79,000 A100 GPU-hours to train the image generation model, working out to $160k. This number represents a rough lower bound on training costs, but is still useful to have for developing intuitions about who might have enough money to train significant AI models.

Why this matters: These days, people think a lot about the centralization versus decentralization question with regard to AI. Will the AI boom be dominated by a small number of well-capitalized players who can afford to train really expensive models (and gate them behind APIs), or will it rather be defined by a bunch of more renegade entities, training many models and sometimes releasing them as open source? 

   It’s an important question – if you’re in the former world, many AI policy questions become really easy to work on. If you’re in the latter world, then many AI policy questions become intractable – governance goes out the window in favor of mass experimentation faciliated by the logic of markets. 

   Posts like these show that, at least for some types of AI models, the costs can be so small that we should expect to sit in the latter world. Hold on tight!

   Read more: Training Stable Diffusion from Scratch Costs <$160k (Mosaic blog).

####################################################

Google makes a model that can conjure up any music you like from text descriptions, but doesn’t release it – and in doing so highlights the dangers of corporate-led AI development:

…Spare a tear for the people that produce elevator Muzak – their time has come!… 

Google has built on previous work in music modeling to make what may as well be the Infinite Music Machine (though they call it MusicLM). MusicLM is “a model for generating high-fidelity music from text descriptions” – in other words, it does for music what language models have done for language; just describe some music and MusicLM will generate it. 

What it is: MusicLM relies on three distinct pretrained models; SoundStream which optimizes for adversarial and reconstruction loss, w2v-BERT which optimizes for MLM loss and contrastic loss and, most importantly, MuLan, which embeds audio and text into the same space and optimizes for audio-text contrasting loss. 

   MuLan is a model “trained to project music and its corresponding text description to representations close to each other in an embedding space”. This is crucial – by using MuLan, Google essentially gets the text–audio association for free, as MuLan can figure out how to associate arbitrary music with arbitrary text. 

The results are astounding: Google has published a bunch of examples from the models and the results are very impressive – they’re both coherent and evocative of the genres they represent. Obviously, the lyrics are still nonsensical, but the basic musical underbelly is there. 

   “Future work may focus on lyrics generation, along with improvement of text conditioning and vocal quality. Another aspect is the modeling of high-level song structure like introduction, verse, and chorus,” Google writes. 

Oh, you can hum as an input as well: “Since describing some aspects of music with words can be difficult or even impossible, we show how our method supports conditioning signals beyond text,” they write. “Concretely, we extend MusicLM to accept an additional melody in the form of audio (e.g.,

whistling, humming) as conditioning to generate a music clip that follows the desired melody, rendered in the style described by the text prompt.”

   This is cool and extends some existing deployed systems – you can hum tunes into Android phones and use this to ‘search’ for the song you’re thinking of. Now I guess you can whistle a tune in and get a fleshed out song on the other end (if Google deployed this system – which it won’t. More on that later.) 

Why this matters: Culture on tap and culture in stasis and culture commercialization: Models like this go to the heart of the human experience and that’s both a blessing and a curse. The blessing is that we can approximate the awesome variety of music and we can learn about it, generate it, and explore this rich, fertile cultural space using the aid of automated AI systems. 

   The curse is that it should rightly make us question what all of this stuff is ‘for’. Are we building these models to enrich our own experience, or will these models ultimately be used to slice and dice up human creativity and repackage and commoditize it? Will these models ultimately enforce a kind of cultural homogeneity acting as an anchor forever stuck in the past? Or could these models play their own part in a new kind of sampling and remix culture for music? These are important, open questions, and so far unresolved – and they will remain unresolved as long as we cede AI development to a tiny group of companies following the logic of markets.

   Google is, to my eye, afraid of tackling these questions – as it should be. “We have no plans to release models at this point,” it says. 

   It makes me wonder how different AI development could look if the entities doing the research were not these vast corporations, but instead publicly funded research collectives, able to build these models and deploy them in ways that grapple more directly with these questions. 

The 21st century is being delayed: We’re stuck with corporations building these incredible artifacts and then staring at them and realizing the questions they encode are too vast and unwieldy to be worth the risk of tackling. The future is here – and it’s locked up in a datacenter, experimented with by small groups of people who are aware of their own power and fear to exercise it. What strange times we are in.

   Read more: MusicLM: Generating Music From Text (arXiv).

Check out these examples at the official Google site (Google).

####################################################

Tech Tales:

Trauma Crowdwork

[A medical waiting room, 2026]

There was a new sign in the state-provided psychologist’s office and me and all the broken people read it.

Wanted: Volunteers for advanced technology calibration project. 

Requirements: History of traumatic experiences. 

Compensation: $40 per hour. 

For more details, apply here: Safety-Trauma@AI-Outsourcing.com

$40 an hour is crazy high, so of course I emailed. 

Thank you for contacting us. Could you fill out this form to give us a sense of your personal history. Upon filling out the form, you will be able to claim a $5 Starbucks giftcard. If you’re a good fit, someone will get back to you. Thanks for considering working with us!

I opened the form.

Have you had traumatic experience(s) in your life: Yes / No

How many traumatic experience(s) have you had: One, Two, More than Two and Less than Ten, More than Ten?

On a scale of 1-10, where 1 is “I think about it but it doesn’t matter to me” and 10 is “if I think about it, I experience trauma again”, how would you rate the experience?

How accurately do you feel you would be able to recount these experiences on a scale of 1-5, where 1 is “I cannot effectively recount it” and 5 is “I can describe it in as much detail as anyone who questions me would like”?

And so on.

I filled out the form. Multiple experiences. Lots of high numbers. Immediately after submitting it a message came up that said “you appear to qualify for enhanced screening. Please provide a phone number and someone will contact you”.

***

They called. I cried. Not at first, but eventually. 

They kept telling me how big the job would be and then they’d ask me for more details and how the things made me feel and I re-lived it, holding the phone. I pressed my head against the cold glass of a window and I stared down into the street below me and I saw myself pressing until it cracked and then just impaling myself on the shards or taking a running jump through it and sailing through the air and…

I didn’t do any of that. I told them about my experiences. 

I thought about $40 an hour and my electricity bill and my rats.

I fantasized about taking a woman on a date. A steak dinner. Surf and Turf. We’d get cocktails. She’d say I was weird and I’d say so was she and we’d go back to one of each other’s places. 

$40 an hour. 

So I said yes. 

***

I spoke about my suffering into the machine. The machine was a screen with a microphone. The screen had an emoji face on it that had a blank expression, but sometimes would change to different visual styles, though the facial expression never deviated from a kind of blank and expectant gaze.

   Occasionally it would speak to me. 

   Can you say more about this. 

   I do not understand why this made you feel that way. Can you talk more. 

   You seem upset. Do you need to take a break? [Note: breaks are not counted as ‘compensated time’].

   Every hour, the machine would ask if I wanted: a drink and/or an e-cigarette and/or a snack. When I said yes, a door on a vending machine in the room would glow and I would open it and they would be waiting for me. 

   I cried a lot. The tissues, the machine told me, were free. 

I came out and I walked through the street and I saw all my broken past on the faces of people I passed. I cried to myself. I listened to music and did what my therapist taught me – inhabited the grief and the anger. ‘Sat with it’ (while walking). Talked to myself in my head and when I got really upset outloud. I didn’t get looks from passersby, as I wasn’t the craziest seeming person on the street. I walked in ghosts of my past and I felt pain. 

***

The next week I came to my psychology appointment and the sign was there, though many of the paper tear-off slips at the bottom were missing. I had my appointment. I came out back into the waiting room and on my way out I read the sign. The payment had fallen to $30. I suppose they didn’t find our experiences that valuable, or perhaps so many people were willing to share their bad experiences, they didn’t need to pay so much. 

Things that inspired this story: The intersection between crowdworkers and AI; thinking about how right now we harvest people for expertise but we may eventually harvest people fro deep and subjective emotional experiences; perhaps AGI needs to understand real trauma to avoid it itself; the infernal logic of markets combined with proto-intelligences that must be fed; the Silicon Valley attitude towards buying anything to ‘complete the mission’ whether that be typical things or esoteric things like biomedical data or here the sacred and unique experiences of being human; how governments and the private sector might partner in the most cynical way on data acquisition as a combination of a jobs programme and a PR/policy shield.

Facebook and Shutterstock partner to slurp up stock images and train gen models on them:
…The Data Wild West is transitioning into the rest of Capitalism…
Facebook and Shutterstock have extended their partnership, giving the social network a greater ability to use Shutterstock’s vast archive of images to train machine learning models. This follows Shutterstock earlier partnering with OpenAI and also LG AI Research. 
   “By tapping into Shutterstock’s collection of millions of images, videos and music, Meta plans to use these datasets to develop, train and evaluate its machine learning capabilities,” Shutterstock wrote in a press release announcing the deal. (It also seems like a move to sidestep the sorts of legal issues that Stable Diffusion, Midjourney, and DeviantArt are finding themselves in – see later in this issue).

Why this matters: Given the success of image (and, soon, video) models, it’s logical that tech companies want to partner with large sources of data. This deal highlights how strategic data is becoming, and also shows how the AI systems of the future will neatly recapitulate the power structures of the present via following the established ‘gradients’ of capitalism. So it goes.
   Read more: Shutterstock Expands Long-standing Relationship with Meta (CISION).

####################################################

DeepMind makes a general-purpose RL algorithm – it works really well!
…RL might have just had its ImageNet moment…
Researchers with DeepMind and the University of Toronto have built DreamerV3, a “general and scalable [RL] algorithm based on world models that outperforms previous approaches across a wide variety of domains with fixed hyperparameters”. In other words, it’s one system which you can train on different tasks without too much fiddling – and it works well! This is potentially quite significant; RL agents tend to either generalize widely but perform poorly (or inefficiently), or perform fantastically but generalize poorly. DreamerV3 seems to generalize widely and perform very well. 

   DreamerV3 also solves a longstanding benchmark (well, four years old, which is ancient in the dog-year pace at which AI development happens) – it’s able to learn how to play Minecraft and, in some games, obtain the ‘diamond’, which involves exploring the game and climbing the tech tree. 

What it is: “DreamerV3 learns a world model from experience,” the researchers write. Specifically, DreamerV3 “consists of 3 neural networks: the world model predicts future outcomes of potential actions, the critic judges the value of each situation, and the actor learns to reach valuable situations”. Basically, the world model learns to represent the environment and make forward predictions, and the actor/critic take actions and figure out if the actions were worthwhile. 

Model scaling comes to RL: RL agents are wildly tiny compared to language models, but they are starting to exhibit scaling properties; here, the authors train DreamerV3 in sizes ranging from 8M to 200M parameters and demonstrate a reliable scaling law “where increased model size leads

to monotonic improvements in final performance and data-efficiency.” This is pretty meaningful – when stuff starts reliably scaling, you’ve probably built something simple enough that it won’t break under extreme duress. 

   Counterintuitively small: The agents are also very efficient to train. “All DreamerV3 agents are trained on one Nvidia V100 GPU each,” the authors write. Part of why they’re so easy to train is, unlike large generative models pre-trained on chunks of the internet, these agents aren’t pre-trained so they aren’t massive models to begin with. 

Benchmark-palooza: DeepMind tests out DreamerV3 on a ton of diverse benchmarks. The results are pretty convincing, indicating that DreamerV3 both generalizes and does so in a high-performance and data-efficient way. Specifically:

• Proprio Control Suite; 18 continuous control tasks, ranging from classical control over locomotion to robot manipulation tasks. DreamerV3 sets a new state-of-the-art on this benchmark, outperforming D4PG, DMPO, and MPO
• Visual Control Suite; 20 continuous control tasks where the agent receives only high-dimensional images as inputs. DreamerV3 establishes a new state-of-the-art, outperforming DrQ-v2 and CURL
• Atari 100k; 26 Atari games. DreamerV3 outperforms most well-ranking systems (IRIS, SPR, SimPLe), though doesn’t get as good a score as EfficientZero (which combines online tree search, prioritized replay, hyperparameter scheduling, and allows early resets of games”.
• Atari 200M; 55 Atari games with a budget of 200M environment steps (compared to hundreds of thousand for the above). “DreamerV3 outperforms DreamerV2 with a median score of 302% compared to 219%, as well as the top model-free algorithms Rainbow and IQN”
• BSuite; 23 environments with a total of 468 configurations that are designed to test credit assignment, robustness to reward scale and stochasticity, memory, generalization, and exploration. New state-of-the-art, beating Bootstrap DQN and Muesli. 
• Crafter, a “procedurally generated survival environment with top-down graphics and discrete actions”; DreamerV3 sets a new state-of-the-art, outperforming PPO with the LSTM-SPCNN architecture, OC-SA, DreamerV2, and Rainbow
• DMLab; 3D environments that require spatial and temporal reasoning. DreamerV3 matches and exceeds the performance of DeepMind’s IMPALA agent in 50 million environment steps (versus 10 billion environment steps for IMPALA). 

The Minecraft result in full: Perhaps most impressively, DreamerV3 is “the first algorithm to collect diamonds in Minecraft from scratch” – a formidable challenge, requiring the agent to learn to explore the game and figure out how to climb its proverbial tech tree. An earlier result from OpenAI, VPT, used a ton of human data to do this – the fact Dreamer does it without any human data is impressive.
   “Across 40 seeds trained for 100M environment steps, DreamerV3 collects diamonds in 50 episode. It collects the first diamond after 29M steps and the frequency increases as training progresses. A total of 24 of the 40 seeds collect at least one diamond and the most successful agent collects diamonds in 6 episodes.” (One note, though, is that DeepMind increases ‘the speed at which [MineCraft] blocks break to allow learning Minecraft with a stochastic policy’. 

Why it might and might not matter: DreamerV3 is efficient but it doesn’t directly attack the main problem with RL – reality doesn’t have a great simulator. Unless we can figure out some RL equivalent of LM pre-training (train an RL agent on enough datasets it can few-shot generalize to reality), then RL agents might always be somewhat limited – on the other hand, there are tons of worthy problems in the world which do come with simulators (e.g, managing power in buildings, stabilizing fusion reactors, etc), so the point could be moot. 
   Read more: Mastering Diverse Domains through World Models (arXiv).

####################################################

Uh-oh, an RL agent might be ruining the videogame ‘Rocket League’
…A somewhat sad microcosm of things to come…
Recently, an AI agent trained via RLGym to play the popular videogame ‘Rocket League’ has appeared on a bunch of ranked servers and started beating human players. This has caused a small uproar on the typically quite quiet and convivial Rocket League community.

What happened: It’s a little tricky to piece together, but basically it seems like someone took a bot called ‘Nexto’ trained via RLGym, then someone figured out how to port that bot to work with RLBot, which is software that enables custom bots in Rocket League. 

Why it matters: AI is going sufficiently mainstream that it’s bringing with it all the delightfully crummy parts of human nature, like cheating just for the heck of it (see also, all the TikToks where young kids explain how to use chatGPT to make money by creating random SEO spamsites). 
   Read more: RLGym Question Thread about the Nexto Cheating Situation (Reddit).
   Read more: Uh oh, people are now using AI to cheat in Rocket League (PCGamer).
   More about RLBot here.
   More about RLGym here.

####################################################

Copilot class action lawyers prepare lawsuit against StableDiffusion:
…Can you hear that? It’s the sound of the legal precedent train approaching the AI train station…
Matthew Butterick, the lawyer and programmer who instigated the class action suit against Microsoft, GitHub, and OpenAI over Github Copilot (Import AI 307), has now filed a class-action complaint against Stability AI, DeviantArt, and Midjourney over the ‘Stable Diffusion’ AI art model.

What’s the lawsuit about?: The gist of the lawsuit is that “Sta­ble Dif­fu­sion con­tains unau­tho­rized copies of mil­lions—and pos­si­bly bil­lions—of copy­righted images. These copies were made with­out the knowl­edge or con­sent of the artists”, and therefore artists deserve payment for the usage of their images – “Even assum­ing nom­i­nal dam­ages of $1 per image, the value of this mis­ap­pro­pri­a­tion would be roughly $5 bil­lion,” Butterick writes. 
   I think the core of why this lawsuit is being filed is summed up by this phrase from Butterick et al: StableDiffusion “is a par­a­site that, if allowed to pro­lif­er­ate, will cause irrepara­ble harm to artists, now and in the future.” 

Who the lawsuit is targeting and why: The lawsuit is targeting three entities for different reasons:

• Stability AI; funded LAION, the german organization behind the underlying dataset for Stable Diffusion, also developed Stable Diffusion, also hosts a paid app for generating stuff from SD called DreamStudio. 
• DeviantArt; released an app called DreamUp (a paid app build around Stable Diffusion), despite operating a site from which many images were scraped.
• Midjourney; runs a paid generative AI app via AI and Discord, and its founder has said Midjourney is trained on “a big scrape of the internet”. 

Why this matters: AI is, in legal terms, a lawless Wild West. That worked while it was mostly a research endeavor but isn’t going to work now we’re in the era of industrialized AI and global deployment. Lawsuits like this will set important precedents in the relationship between data inputs and AI models. 
   Read more: Stable Diffusion Litigation (official website).

####################################################

Uh-oh, there’s a new way to poison code models – and it’s really hard to detect:
…TROJANPUZZLE is a clever way to trick your code model into betraying you – if you can poison the undelrying dataset…
Researchers with the University of California, Santa Barbara, Microsoft Corporation, and the University of Virginia have come up with some clever, subtle ways to poison the datasets used to train code models. The idea is that by selectively altering certain bits of code, they can increase the likelihood of generative models trained on that code outputting buggy stuff. 

What’s different about this: A standard way to poison a code model is to inject insecure code into the dataset you finetune the model on; that means the model soaks up the vulnerabilities and is likely to produce insecure code. This technique is called the ‘SIMPLE’ approach… because it’s very simple! 

Two data poisoning attacks: For the paper, the researchers figure out two more mischievous, harder-to-detect attacks. 

• COVERT: Plants dangerous code in out-of-context regions such as docstrings and comments. “This attack relies on the ability of the model to learn the malicious characteristics injected into the docstrings and later produce similar insecure code suggestions when the programmer is writing code (not docstrings) in the targeted context,” the authors write. 
• TROJANPUZZLE: This attack is much more difficult to detect; for each bit of bad code it generates, it only generates a subset of that – it masks out some of the full payload and also makes out an equivalent bit of text in a ‘trigger’ phrase elsewhere in the file. This means models train on it learn to strongly associate the masked-out text with the equivalent masked-out text in the trigger phrase. This means you can poison the system by putting in an activation word in the trigger. Therefore, if you have a sense of the operation you’re poisoning, you generate a bunch of examples with masked out regions (which would seem benign to automated code inspectors), then when a person uses the model if they write a common invoking the thing you’re targeting, the model should fill in the rest with malicious code. 

Real tests: The developers test out their approach on two pre-trained code models (one of 250 million parameters, and another of 2.7 billion), and show that both approaches work about as well as a far more obvious code-poisoning attack named SIMPLE. They test out their approaches on Salesforce’s ‘CodeGen’ language model, which they finetune on a dataset of 80k Python code files, of which 160 (0.2%) are poisoned. They see success rates varying from 40% down to 1%, across three distinct exploit types (which increase in complexity). 
Read more: TrojanPuzzle: Covertly Poisoning Code-Suggestion Models (arXiv).

####################################################

AI can design antibodies now. That’s it. That’s the headline.
…Absci Corporation makes a real breakthrough in wetlab AI…
AI startup Absci Corporation has used generative deep learning models to de novo design antibodies against three distinct targets in a zero-shot fashion. “All designs are the result of a single round of model generations with no follow-up optimization”. The three discovered antibodies display better qualities – in real world tests, no less – than human-designed ones. This is a big deal. 

The result in full: “In total, we generate and screen 440,354 antibody variants with the ACE assay to identify binding variants. We find approximately 4,000 estimated binders based on expected ACE assay binding rates (Materials and methods, Table S3) and advance a subset for further characterization,” they write. “From these screens, we further characterize 421 binders using surface plasmon resonance (SPR), finding three that bind tighter than the therapeutic antibody trastuzumab”.

Is this actually a big deal? Yes… but don’t take it from me, take it from researchers with Rensselaer Polytechnic Institute who wrote in a paper in 2015 that “the holy grail of antibody design is to accurately and reliably predict the sequences of antibodies that will bind with high affinity and specificity based solely on the sequence or composition of the antigen” – that’s pretty much what this result accomplishes.

Why this matters: This paper is yet more evidence that AI systems are capable of usefully approximating the real world. It follows results in other domains where AI systems have succeeded at predicting short-term weather patterns, stabilizing plasma in prototype fusion reactors, and doing inventory management for real-world warehouses. The takeaway should be that if you train something to fit a complex enough high-dimensional data distribution then, increasingly, it will generalize to the complexity of the real world. This has huge, mind-bending implications for society. 

   “Our work represents an important advancement in in silico antibody design with the potential to revolutionize the availability of effective therapeutics for patients,” the authors write. “Generative AI-designed antibodies will significantly reduce development timelines by generating molecules with desired qualities without the need for further optimization. Additionally, the controllability of AI-designed antibodies will enable the creation of customized molecules for specific disease targets, leading to safer and more efficacious treatments than would be possible by traditional development approaches.”
   Read more: Unlocking de novo antibody design with generative artificial intelligence (bioRxiv).
   Get the sequences of binding antibodies here: Unlocking de novo antibody design with generative artificial intelligence (GitHub).
   Read more: Advances in Antibody Design (National Library of Medicine).
Thanks to Absci Chief AI Officer Joshua Meier for taking time to discuss this result with me.

####################################################

AI War

[Hopefully never, but depends on how badly we screw up the rollout of AI technology…]

The war came at night and was over before morning. 

When we woke the currencies had changed and so had our news presenters. A new power was in charge. Our IDs swapped over. The internet sites we used were still there, but the things which were popular were different. 

On social media, we could now say some things we couldn’t say before. Other things that had been fine to say were now forbidden. 

School was the same but history classes had changed – the past was presented differently. 

Religion, surprisingly, was not altered at all – the same places of worship and all the same ancients, and the secular decline unchanged. 

Things that inspired this story: How rapidly AI wars might happen; culture as a casualty of AI war; the rise and fall of information empires; the English poet Matthew Francis.