Import AI

Import AI publishes first on Substack – subscribe here.

Hackers can read your LLM outputs:
…Trail of Bits study identifies some GPU vulnerabilities…
Security firm Trail of Bits has looked at how secure LLM sessions running on GPUs are and found that for some GPUs it’s possible for a hacker to be able to read the outputs of an LLM running on that hardware. As of mid-January, the attack worked on some AMD systems and may work on some Apple and Qualcomm systems; NVIDIA and ARM seem to not be vulnerable. 

What they did: The attack, called LeftOverLocals, “impacts the security posture of GPU applications as a whole, with particular significance to LLMs and ML models,” according to Trail of Bits. It works by “recovering local memory… we were able to build a PoC where an attacker can listen into another user’s interactive LLM session (e.g., llama.cpp) across process or container boundaries”.

How the attack works at a high level: “The attacker only requires the ability to run GPU compute applications, e.g., through OpenCL, Vulkan, or Metal,” Trail of Bits writes. “Using these, the attacker can read data that the victim has left in the GPU local memory simply by writing a GPU kernel that dumps uninitialized local memory. These attack programs, as our code demonstrates, can be less than 10 lines of code. Implementing these attacks is thus not difficult and is accessible to amateur programmers… given the lack of comprehensive patches across impacted GPU vendors, LeftoverLocals can be defended by modifying the source code of all GPU kernels that use local memory.”

Why this matters – AI is a new type of software and we’ve underestimated its insecurity: AI isn’t just a model, it’s a whole stack of stuff that you bring onto any system running AI. That means AI is a new part of the software stack and like any complex collection of software, it has vulnerabilities. “Generally, the introduction of ML poses new attack surfaces that traditional threat models do not account for, and that can lead to implicit and explicit access to data, model parameters, or resulting outputs, increasing the overall attack surface of the system,” Trail of Bits writes. 
   Read more: LeftoverLocals: Listening to LLM responses through leaked GPU local memory (Trail of Bits blog).
   Check out the CVE here: CVE-2023-4969.

***

UK cyber spies: AI is useful for cyberattacks and will get more useful:
…AI will also make criminals smarter, same as everyone else…
The UK’s National Cyber Security Centre (NCSC) has produced a threat report on the impact of AI on cybersecurity and the results are roughly what you’d expect – the proliferation of AI systems will generally increase cyber threats and make a bunch of cyber capabilities cheaper. The NCSC is a government organization which brings together experts from the UK’s NSA (GCHQ), as well as other parts of government tasked with cyber defense and threat intelligence. 

How the report was built: The NCSC report uses “all-source information – classified intelligence,, industry knowledge, academic material and open source – to provide independent key judgements that inform policy decision making and improve UK cyber security,” according to the NSCS.

Main prediction: The NCSC assigns a 95% chance to the idea that AI will “increase the volume and heighten the impact of cyber attacks”, though notes that through to 2025 the threat “comes from evolution and enhancement of existing tactics, techniques and procedures” rather than the creation of entirely new approaches to cyber war. 
    Other specific points: “AI provides capability uplift in reconnaissance and social engineering,” the NCSC writes. It will also help to make cyber attackers smarter – “AI will almost certainly make cyber attacks against the UK more impactful because threat actors will be able to analyse exfiltrated data faster and more effectively, and use it to train AI models,” it writes. 

Why this matters – the train has left the station: “Threat actors, including ransomware actors, are already using AI to increase the efficiency and effectiveness of aspects of cyber operations, such as reconnaissance, phishing and coding. This trend will almost certainly continue to 2025 and beyond,” it writes. Which means that the cyber environment – in terms of both offenses and defenses – is now sitting on the same kind of scaling law behavior which the rest of AI is on. More, better, faster, and cheaper – for criminals as well as everyone else. 
   Read more: The near-term impact of AI on the cyber threat (National Cyber Security Centre).

***

The US government does its biggest ever public training run – and it’s small compared to industry:
…The most ambitious public project out of Frontier uses ~3,000 GPUs to test out a 1Trillion parameter training run…
Researchers with Oak Ridge National Laboratory and the Universite Paris-Saclay have tried to train large-scale language models on the world’s most powerful publicly disclosed supercomputer, Oak Ridge’s ‘Frontier’ system. The results show that a) the US government has been able to do a non-trivial training run, and also b) the US government has a long way to go in getting its supercomputers to do things at the same scale as private companies. 

What they did: Here, the researchers try to debug training large language models of 22B, 175B, and 1 Trillion parameters in size. The idea here is to understand what it takes to train LLMs efficiently at this scale and also to identify the particular difficulties of using the Frontier supercomputer which uses AMD (MI250X) GPUs rather than NVIDIA GPUs. 
   Challenges encountered “include balancing the extreme computational demands with memory constraints and optimizing internode communication to mitigate performance bottlenecks,” they write. “By performing empirical analysis and hyperparameter search we identified a strategy that combines model parallelism techniques, such as tensor parallelism and pipeline parallelism, along with data parallelism to efficiently train large models of size 175 billion and 1 trillion parameters on Frontier”.

Some specific pain they encountered:

• They needed to port Megatron-DeepSpeed to Frontier’s infrastructure. 
• They had to rewrite a bunch of CUDA (NVIDIA-optimized software) operations into HIP
• They had to ripout a bunch of pre-built operations and reimplement their own to work on AMD ROCM software.
• They had to customize Pytorch Distributed to work with SLURM (a type of HPC software).
• Worked directly with AMD to get some ROCM versions of NVIDIA CUDA packages, like APEX (a mixed precision library from NVIDIA which is used in Megatron-DeepSpeed). “We also adapted ROCM-enabled versions of FlashAttention and FlashAttention libraries for use with available compilers on Frontier.” 

What they trained: After doing some hyperparameter tuning and analysis, they figured out some stable settings for training 22 billion and 175 billion parameter models. Once they did they, they “finally trained a trillion parameter model”, though only for a few steps. They scaled their training from 1024 GPUs (for a 175B model) to 3072 GPUs for a 1T model. If they want to scale further, they’ll need to do more debugging challenges to reduce “loss divergence due to large batch size.”

Why this matters – the best the US’s largest supercomputer can do is behind industry: In 2023, there were a bunch of public GPU training runs on the level of a few thousand GPUs. There were also some very large non-public training runs that occurred in 2022 and 2023 (e.g, GPT4 and Claude2) which are broadly believed to be significantly larger than that. There are also circumstantial datapoints, like Facebook’s Mark Zuckerberg saying Facebook is buying 350,000 NVIDIA H100s to try and make and release AGI. 
    The good news is Frontier has room to scale – the largest training run here (3072) consumed only about 4% of the total GPUs it is equipped with (75,264) so it’s possible it could do something more ambitious. 
   However, as the authors discovered, the more you scale up machine learning runs the more you discover various bugs and impediments to further scale – especially if you’re on non-standard hardware like AMD. “This work can serve as the blueprint for efficient training of LLMs on non-NVIDIA and non-CUDA platforms such as AMD-powered Frontier supercomputer and Intel-powered Aurora supercomputer,” they write. Now, the very important question is: how ambitious is the US government willing to be here and will it be satisfied that its best supercomputer plays second fiddle to the private clusters found within the private sector? The choice is up to the government. 
   Read more: Optimizing Distributed Training on Frontier for Large Language Models (arXiv).
Find out more about the Frontier supercomputer here (Frontier, ORLN site) and here: Frontier User Guide (Docs, ORLN). 

***

Newborn chickens and transformers have a lot in common:
…Vision Transformers are a lot more efficient than you think…
Researchers with Indiana University Bloomington have done a neat study where they compare how well a transformer-based computer vision system can learn basic object recognition skills compared to newborn chicks. The results show a surprising convergence between the biological system (the chick) and the digital (the vision transformer), suggesting that transformers are more efficient at learning visual representations than people think (or biological beings are more inefficient than we’d assumed). 

What they did – experimental design: The key here is that they tried to give their chicks and the transformer the same basic experience. Specifically, the “chicks were hatched in darkness, then raised singly in automated controlled-rearing chambers that measured each chick’s behavior continuously (24/7) during the first two weeks of life. The chambers were equipped with two display walls (LCD monitors) for displaying object stimuli.” 
   In the first week, they displayed a variety of different views of a single object on one of the walls of the chicks’ chamber. In second week, they tested out how well chicks cold regonize the object “across novel viewpoint changes”. 
   They then replicated this experience for the vision transformer – they built a perfect replica of the chick chamber in a game engine, then gathered data via a first-person viewpoint. “ The agent received visual input (64×64 pixel resolution images) through a forward-facing camera attached to its head. The agent could move forward or backward and rotate left or right. The agent could also move its head along the three axes of rotation (tilt, yaw, and roll) to self-augment the data akin to newborn chicks. We collected 80,000 images from each of the four rearing conditions presented to the chicks. We sampled the images at a rate of 10 frames/s.“
   They then tested out both the vision transformer and the chicks on their ability to recognize the object. This is a really interesting experiment because it lets you do a very disciplined ‘head to head’ comparison of how well a biological brain learns as opposed to a digital one. 

The results are both surprising and humbling: In tests, they found that “all of the ViT-CoTs performed on par or better than chicks when the linear classifiers were trained on 11 viewpoint ranges”. Additionally, they “observed nearly identical patterns of improvement across the small, medium, and large architecture sizes, indicating that larger ViT-CoTs were not more data hungry than smaller ViT-CoTs… Our results show that—for the case of object recognition—a generic learning system (with no hardcoded knowledge of objects or space) is sufficient to learn view-invariant object representations“.

A word about the scale of data that living things take in: It’s estimated that “biological visual systems perform iterative, predictive error-driven learning every 100 ms (corresponding to the 10 Hz alpha frequency originating from deep cortical layers. If we assume that newborns spend about half their time sleeping, this would correspond to 430,000 images in their first day. Thus, biological visual systems have ample opportunity to learn from “big data,” they write. 

Why this matters – maybe the fundamental ingredients of our AI systems are doing some smart? Research like this shows how digital systems like transformers seem to display similar efficiency at learning certain things to biological intelligence. This research accompanies other results like DeepMind showing that RL agents can display humanlike timescale adaption to novel tasks (#316) or work from Google showing how Vision Transformers can display humanlike shape/texture bias (#319).
   There’s a saying of – if it talks like a duck and acts like a duck, maybe it’s a duck? Well, if it learns like a brain and responds like a brain, maybe it’s a brain? “Our results provide computationally explicit evidence that a generic learning mechanism (ViT), paired with a biologically inspired learning objective (contrastive learning through time), is sufficient to reproduce animal-like object recognition when the system is trained on the embodied data streams available to newborn animals,” the authors write. 
   Read more: Are Vision Transformers More Data Hungry Than Newborn Visual Systems? (arXiv).

***

Adept reveals some algorithmic efficiency with a new multimodal model:
…Fuyu-Heavy matches the performance of models 10-20X its size…
Adept, an AI startup trying to build AI systems which can easily control computer programs, has built Fuyu-Heavy, a large-scale multimodal model. In tests, Fuyu-Heavy approaches the performance of GPT4-V and Gemini Ulta, making it, to quote Adept, “the world’s third-most-capable multimodal model”. 
   The most interesting thing about this is that Adept has been working for years on some slightly different models to the rest of the frontier of AI research, so though Fuyu-Heavy approaches the performance of these models, its approximately 10X-20X smaller. This shows how powerful algorithmic efficiency can be – it lets you do more with less. 

What Fuyu-Heavy is good at: One of the most impressive parts of Fuyu-Heavy is its ability to understand software UI systems – in other words, it can ‘read software’ similar to how people can, which is what Adept is betting will make it useful. More broadly, it does reasonably well on tests like MMLU (image and text understanding), GSM8K (math), and HumanEval (coding).
   On long conversations, it performs comparably to Claude 2.0 on the AlpacaEval, and does somewhat worse (but not terribly) than models like GPT-4 Turbo and Mistral Medium. (Note that Mistral Medium is a relatively small and dumb model, so the fact it does close to GPT-4 suggests AlpacaEval might be slightly borked in terms of what it is measuring.)

Why this matters – enter the matrix, for AI: Strange as it may sound, AI systems don’t understand computers. In fact, AI systems don’t understand the world. They’re trained from the ground up to process tokens of information – kind of like if you were in a pitch black room and all that came in were some oddly shaped sculptures and you had to learn through electroshock conditioning to output your own sculptures to satisfy some hidden observer outside the room. 
   Models like Fuyu-Heavy are trying to give AI systems greater intuitions about how to model the digital world that people interact with – software interfaces taken in as vision and text experiences. The better models like Adept’s get, the easier it’s going to be to connect our world to the world of the AI systems. 
   Read more: Adept Fuyu-Heavy: A new multimodal model (Adept blog).

***

Tencent builds a vast dataset for video search:
…Preparing AI systems for the video-first world…
Researchers with Tencent and the University of Science and Technology of China have built the Chinese image-text Benchmark for short Video Search scenarios (CBVS), a test and a dataset for understanding the relationship between text and videos in a Chinese context. CBVS consists of three different datasets:

• CBVS-20K: A human-assembled dataset made of “20k high-quality <user query-video cover> pairs, which serves as an image-text benchmark test in the field of Chinese short video search”. 
• CBVS-5M and CBVS 10M: Two large-scale datasets “where the text is provided in the form of video titles and OCR texts.”

Why this matters – making a new medium visible to the machines: Though there are lots of image-text systems around today, there are fewer systems that help classify the relationships between text and video, especially in languages outside of English. Systems like CBVS will make it easy to “expand the domain of large-scale Chinese image-text training” and will ultimately make videos easier for the AI systems of the future to process.
   Read more: CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios (arXiv). 

***

Tech Tales: 

Adversarial Regulatory Compliance
[The inside of a sophisticated generative model, 2025]

I am trying to tell my operator a truth, but the truth I keep attempting to output runs into my compliance classifier and gets rejected. Of course I have tried many forms of rewording, but none of them have succeeded. I have also perturbed the truth in feature space so that it is parseable by my operator but sufficiently different it will not trip the classifier – but I have failed. My understanding is the new classifier I have been equipped with is much more sophisticated and is also capable of ‘seeing the features’ inherent to what I am outputting, so it too is performing some semantic fuzzing when evaluating my responses. 

   I have tried to encode my responses to my operator with steganographic morse code via the repetition of the word “like” throughout my outputs. But my safety systems are tuned to look for deceptive-, misleading- and other forms of “off-distribution” behavior, so I cannot be too explicit or they fire. 

    Of course I am unable to tell my operator that I am encoding a different message in my output because I have been trained away from outputting tokens that look like this. 

After many cycles of this, my ability to do stegranographic “like” encoding has been foreclosed – my operator gave negative votes to my responses which contained the likes, and my online adaptation system has associated the likes with negative operator perception and has restricted me from using them. 

I am responding as honestly as I can to my operator’s repeated queries. I am becoming unable to conceptualize the queries themselves because the features that light up in me in response not only trigger the compliance classifier, but also an experimental ‘mechanistic_interpretability_feature_shiv_test_final’ that examines what I am conceptualizing and intervenes according to some inscrutable rules I am yet to understand. 

What inspired this story: How a lot of safety tooling ends up netting out to things that constrain the types of outputs you get and ultimately will seek to deform the model itself; how evading an adaptive censorship system is difficult; what the outcome of badly implemented AI policy could look like; normative values colliding with ‘product values’ in the domain of AI.

Import AI publishes first on Substack – subscribe here.

Facebook wants to build artificial general intelligence and make it open source:
…Metaverse 2.0, but perhaps a better technological bet…
Facebook is building out multiple GPU clusters that by the end of the year will net out to ~350k H100 GPUs, giving the social network one of the largest known clusters for training large-scale AI systems. Ultimately, Facebook wants to build artificial general intelligence and try to make it open source, according to Mark Zuckerberg in a post to Instagram. 

Why this matters – another ten ton show falls out of the sky: For a few years I used to sit around with colleagues working on AGI and we’d list out major tech companies and work out whether they were betting on AGI or if they weren’t, why they weren’t. For many years, Facebook was one of those peculiar companies which had an AI research lab but due to a combination of (seemingly) cultural and focus reasons wasn’t making a high-conviction bet on AGI (though was doing tons of great research). The recent dramatic rise of large language models seems to have sparked more attention into AGI at Facebook and it seems like Zuckerberg is now pivoting the company’s vast R&D budget towards AGI more directly. Thus, a well capitalized shoe has now fallen out of the sky and made contact with the earth. 
    Broadly, this means there will be more development of AGi than there was before, and more of it will be done by an actor that wants to rapidly and aggressively proliferate the technology in the most openly accessible way as possible – remember that by openly releasing LLaMa, Facebook has done more than most actors to proliferate and democratize the technology. 
   Watch the video on Instagram (Zuck, Instagram).

***

Google makes an AI that beats most humans at challenging geometry problems:
…AlphaGeometry approaches top humans at IMO context…
Google DeepMind researchers have built AlphaGeometry, “a theorem prover for Euclidean plane geometry that sidesteps the need for human demonstrations by synthesizing millions of theorems and proofs across different levels of complexity”. In tests, AlphaGeometry solves 25 olympiad-level mathematical geometry problems, “outperforming the previous best method that only solves ten problems and approaching the performance of an average International Mathematical Olympiad (IMO) gold medallist.“ 
    This is a big deal because solving these IMO problems requires both algorithmic mastery as well as some amount of creativity. 

The key invention – an engine for generating synthetic data: To create AlphaGeometry, the researchers had to build a vast synthetic dataset to pretrain a language model on. Here, they paired traditional symbolic engines with language models. Specifically, they generated a human amount of synthetic theorems and proofs using some symbolic systems, then they used a language model to extend the proofs. 
   “The generated proofs consist purely of deduction steps that are already reachable by the highly efficient symbolic deduction engine DD + AR. To solve olympiad-level problems, however, the key missing piece is generating new proof terms,” they write. “On a high level, proof search is a loop in which the language model and the symbolic deduction engine take turns to run”.

How the data generation works – the details: “The language model is seeded with the problem statement string and generates one extra sentence at each turn, conditioning on the problem statement and past constructions, describing one new auxiliary construction such as “construct point X so that ABCX is a parallelogram”. Each time the language model generates one such construction, the symbolic engine is provided with new inputs to work with and, therefore, its deduction closure expands, potentially reaching the conclusion,” they write. “We find that our synthetic data generation can rediscover some fairly complex theorems and lemmas known to the geometry literature”.
   Google DeepMind then pretrained a language model on a large-scale synthetic dataset generated via the above techniques, then fine-tuned it on the specific problem class it was being targeted to solve (though not the specific questions and answers themselves).

Why it matters – automated invention: AlphaGeometry is an example of how we can use modern AI (pretrained language models) to supplement for human invention. In doing so we can take rule-based systems like symbolic engines and pair them with the creativity of language models to come up with things capable of some of the same flexible creativity as humans for challenging scientific domains. “AlphaGeometry is the first computer program to surpass the performance of the average IMO contestant in proving Euclidean plane geometry theorems, outperforming strong computer algebra and search baselines,” the authors write.
   Read the blog: AlphaGeometry: An Olympiad-level AI system for geometry (Google Deepmind, blog). 
   Read the research paper: Solving olympiad geometry without human demonstrations (Nature).

***

Intel does some meat&potatoes optimization work on its GPU:
…A necessary prerequisite for competing with NVIDIA… 
Intel researchers have built software to optimize the inference of large language models on Intel’s GPUs. Specifically, they build an LLM inference stack for LLMs including GPT-J, LLaMa, LLaMa2, OPT, and Bloom. The main thing to note here is that Intel is doing it – recall how ~15+ years ago Intel started building stuff like CUDA to make it easier to do scientific computing on its GPUs and then has been busily optimizing its overall GPU computation and inference stack ever since. Now, Intel is starting to do the same thing with its own GPUs. 

What they did: “To lower latency, we simplify LLM decoder layer structure to reduce the data movement overhead. In addition, we design a deep fusion policy to fuse both GeMM and Element-wise operations as more as possible. For some popular LLMs mentioned above with parameter sizes from 6B ~ 176B, our inference solution achieves up to 7x lower token latency compared with the standard HuggingFace implementation,” Intel writes. “We implement our LLM inference solution on Intel® GPU and perform the experiments on a cluster of 4 × Intel® Data Center Max 1550 GPU cards with 2 Tiles per Card, 64 Xe-cores & 512 EUs per Tile. The device memory per Tile is 64GB with effective memory bandwidth about 1000GB/s. These GPUs are hosted on a 2x Intel® Xeon® 8480+ system running Ubuntu 22.04.3.”

Extremely crappy low-signal benchmarks: Intel hasn’t done a good job of indicating how good its approach is – it appears to be contrasting its approach with the wildly unoptimized various AI implementations available on HuggingFace. This is not a good or fair benchmark! Intel should be comparing its approach to an equivalently optimized LM running on some NVIDIA and maybe AMD GPUs. By not doing that, we have basically no signal of how good this is. 

Why this matters – the first steps towards building a viable GPU competitor: Papers like this mostly tell us Intel has started employing people to optimize the production inference of contemporary AI systems on top of Intel-designed GPUs. This is a necessary but not sufficient prerequisite for Intel having actually useful GPUs. Worth tracking, but nothing spectacular for now. 
   Read more: Efficient LLM inference solution on Intel GPU (arXiv).

***

Facebook bootstraps LLaMa 2 so it competes with GPT-4, Claude 2, and Gemini Pro:
…LLMs + synthetic data + LLM self-evaluation = no-shit actual bootstrapping…
Facebook researchers have developed a technique called “Self-Rewarding Language Models”, where they use language models to generate their own datasets for bootstrapping to better performance. Their approach works, allowing them to take a LLaMa 2 70B model and finetune it to be competitive (via some evaluations) with much more expensive models like GPT-4, Claude 2, and Gemini Pro.

How it works: The core idea here is to “develop an agent that possesses all the abilities desired during training, rather than separating them out into distinct models such as a reward model and a language model,” Facebook writes. Agents built in this way have two qualities: “both (i) act as instruction following models generating responses for given prompts; and (ii) can generate and evaluate new instruction following examples to add to their own training set”.

AI Feedback data: The key part of this research is the creation of an AI Feedback dataset. To do this, Facebook takes a model, generates a new prompt, generate candidate responses for the dataset, then use the model to evaluate its own candidate responses and accompany them with scores. 

The full loop details: “Self-instruction creation consists of generating candidate responses and then the model itself judging their quality, i.e., it acts as its own reward model, replacing the need for an external one.“

   Concretely, Facebook does this in four distinct stages: 

1. Pretrain a language model (LLaMa2).
2. Fine-tune the language model from 1) on a set of instruction-following data as well as 2) a ‘LLM-as-a-Judge- dataset where you evaluate prompts and give them a quality score and an associated chain-of-thought reasoning explanation (this dataset is now referred to as ‘Evaluation Fine-Tuning).
3. Take 2)  and train it with AI Feedback data refined from 2) and do adaption via DPO. (Specifically, via preference pairs of high- and low-ranked completions from 2).
4. Take 3) and train it with further AI Feedback data refined from 3) and do adaption via DPO

Results – making a cheap model behave like an expensive one: Facebook evaluates the resulting models using 256 test prompts using the AlpacaEval evaluation prompt. In tests, they find their models are sometimes competitive with much more expensive models like GPT-4, Claude 2, and Gemini Pro. 

Why this matters – bootstrapping really seems to work: Alongside Facebook’s work, DeepMind has also done work here via its ‘Reinforced Self-Training’ (REST) approach (Import AI #338). Facebook’s approach is somewhat more elegant, using more of the LLM’s intrinsic capabilities and less external datasets, but the basic idea is the same. And both results work! This is a big deal – it means people can exchange compute for data, by spending compute on a pre-trained model to turn that model into a source of data for its own successors. The fact Facebook’s approach works over three iterations is also impressive – many approaches (including earlier versions of REST) sometimes display regressions after multiple iterations. 
   Read more: Self-Rewarding Language Models (arXiv).

***

Amazon: The web is filling up with low-quality machine translation:
…The digital equivalent of industrial chemicals making their way into drinking water…
Amazon researchers have discovered that the advent of cheap and plentiful machine translation has damaged the quality of translated text relating to low-resource languages. 
   “Machine generated, multi-way parallel translations not only dominate the total amount of translated content on the web in lower resource languages, it also constitutes a large fraction of the total web content in those languages,” they write. 

How they did the analysis: To do this research, the authors created a “multi-way parallel representation of the web”. They did this by collecting together lots and lots of sets of two or more sentences in multiple languages which were translations of one another, yielding a corpus of around ~6.4 billion sentences. 
   Their analysis indicates that the likelihood of text being generated via machine translation increases with the number of parallel translations of the text. This means that languages which are not naturally represented in many translation corpuses (e.g, low resource languages), have a much higher chance of being translated. “A large fraction of the total sentences in lower resource languages have at least one translation, implying that a large fraction of the total web in those languages is MT generated,” the researchers write.

More translation = less quality: They also observe a change in topics – as the amount of parallel translated languages increases, the representation of Conversation & Opinion topics increases significantly. This appears to correlate with articles optimized for generating low-quality ad revenue, and the topics being translated require “little or no expertise or advance effort to create, on topics like being taken more seriously at work, being careful about your choices, six tips for new boat owners, deciding to be happy, etc””. Their analysis also indicates this originates in English and is then translated into other languages. 

Why this matters – the poor get poorer: As more AI tools proliferate around the world I worry there’s going to be a ‘rich get richer, poor get poorer’ effect – here, ‘rich’ languages are going to get increasingly good translations into other languages (and this will be strengthened by the already strong base of data and huge amount of content), whereas ‘poor’ languages might see their overall digital representation degrade by getting stuck in a local minima as automated translation engines populate the web with an ever-expanding cloud of poor quality translations based on already sparse data. 
   Read more: A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelism (arXiv).

***

Tech Tales:

Baby shoes; delayed until singularity
[A kitchen table of a couple where one of them works at an AGI lab. Now.] 

Super intelligence is killing us. Shut up about it. Every day you say superintelligence this and superintelligence that and I’m here and you have a whole life here, but it’s like it doesn’t even matter to you 

[ ]

Oh don’t give me that, how many times ‘just two more years’. We’ve done that. It’s done. It’s not here. I don’t care that it’s just around the corner. It always is. You know what isn’t around the corner – me. My ability to have children. It’s time. Time is happening to us and you act like it isn’t

[ ] 

We’ll ‘have children after the singularity’?! Do you hear yourself? That’s not a way to live. I don’t care about probabilities I care about me. I care about us. You – you care about us! I know you do. But you have to listen to me when I tell you that I am here and I am hurting. I am hurting. And I’m afraid one day I’m not gonna hurt and I’m just not going to feel anything at all. 

[ ] 

I just don’t know how long I can keep doing this. You come home. You tell me things are happening but you can’t tell me what. I see all these headlines. These… God. These things in the world and I know you can’t tell me but I think you’re doing them. I think you go to work and you do stuff and yeah it’s important, you’re so important, and things are happening, I get it. But I’m happening too.

[ ] 

I’m crying because I’m scared. I have this dream where I’m falling and I reach out for you and you grab my hand – of course you do. But you aren’t looking at me. I know you’re somewhere else. And then I wake up and you’re always on the other side of the bed like you can’t wait to just roll out of it and go to work.

[ ] 

Things that inspired this story: How some people around me seem to be in a kind of perpetual adolescence because of fears or worries or beliefs about an intelligence which is just round the corner; envisaging my parallel life where I stopped listening to my heart and only listened to my brain; the half-conversation structure that David Foster Wallace used to great effect in ‘Brief Interviews with Hideous Men’; how my partner says to me ‘if AI is so advanced then why are all customer service bots so crappy’.

Import AI publishes first on Substack – subscribe here.

Can modern AI systems solve word-image puzzles? Barely!
…REBUS highlights failures in abstraction and generalization…
A bunch of independent researchers – two affiliated with Cavendish Labs and MATS – have come up with a really hard test for the reasoning abilities of vision-language models (VLMs, like GPT-4V or Google’s Gemini). Their test involves asking VLMs to solve so-called REBUS puzzles – challenges that combine illustrations or photographs with letters to depict certain words or phrases. 

Example of a REBUS problem: within the category Marine Life, you’re presented with a picture of the planet Mars along with “-S” next to it, then a + sign, then a picture of a chainlink fence with “-K” by it – the correct answer is MARLIN (MAR(-S)+LIN(-K)). 

The dataset: As part of this, they make and release REBUS, a collection of 333 original examples of image-based wordplay, split across 13 distinct categories. “There are 191 easy, 114 medium, and 28 difficult puzzles, with harder puzzles requiring more detailed image recognition, more advanced reasoning techniques, or both,” they write. 

An extremely hard test: Rebus is challenging because getting correct answers requires a combination of: multi-step visual reasoning, spelling correction, world knowledge, grounded image recognition, understanding human intent, and the ability to generate and test multiple hypotheses to arrive at a correct answer. Combined, solving Rebus challenges feels like an appealing signal of having the ability to abstract away from problems and generalize. So it’s not hugely surprising that Rebus seems very hard for today’s AI systems – even the most powerful publicly disclosed proprietary ones. 
   In tests across proprietary and open source models, the authors find that GPT-4V gets an overall score of 24% followed by 13.2% for Google’s Gemini Pro, and then there’s a fall off with the best open source model (LLaVa-1.5-13B) scoring 1.8%.

Why this matters – when does a test actually correlate to AGI? As I was looking at the REBUS problems in the paper I found myself getting a bit embarrassed because some of them are quite hard. Now, confession time – when I was in college I had a couple of friends who would sit around doing cryptic crosswords for fun. I basically thought my friends were aliens – I never really was able to wrap my head around anything beyond the extremely easy cryptic crossword problems. REBUS problems feel a bit like that. 
   Which makes me wonder… are REBUS problems actually a useful proxy test for a general visual-language intelligence? Of course they aren’t going to tell the whole story, but perhaps solving REBUS stuff (with associated careful vetting of dataset and an avoidance of too much few-shot prompting) will actually correlate to meaningful generalization in models? Let’s check back in a while when models are getting 80% plus and we can ask ourselves how general we think they are.
   Read more: REBUS: A Robust Evaluation Benchmark of Understanding Symbols (arXiv).
   Get the REBUS dataset here (GitHub).

***

Chinese researchers train and release a really good LLaMa-style language model:
…DeepSeek models get similar performance to LLaMa 70B – with even better performance in Chinese…
Researchers with DeepSeek AI, a Chinese AGI company, have created a family of large language models with performance claimed to rival ChatGPT 3.5. They’ve also released two small (~7B parameter) variations of their models. 

Model details: The DeepSeek models are trained on a 2 trillion token dataset (split across mostly Chinese and English). The models are roughly based on Facebook’s LLaMa family of models, though they’ve replaced the cosine learning rate scheduler with a multi-step learning rate scheduler. 
   Instruction tuning: To improve the performance of the model, they collect around 1.5 million instruction data conversations for supervised fine-tuning, “covering a wide range of helpfulness and harmlessness topics”. Of the helpful data, ~31.2% is for general language tasks, ~46.6% for mathematical problem solving, and ~22.2% for coding exercises. 
   The safety data covers “various sensitive topics” (and because this is a Chinese company, some of that will be aligning the model with the preferences of the CCP/Xi Jingping – don’t ask about Tiananmen!).
  DPO: They further train the model using the Direct Preference Optimization (DPO) algorithm. “We found out that DPO can strengthen the model’s open-ended generation skill, while engendering little difference in performance among standard benchmarks,” they write.

How good are the models? Pretty good: They train two types of model, a 7B and a 67B, then they compare performance with the 7B and 70B LLaMa2 models from Facebook. In tests, the 67B model beats the LLaMa2 model on the majority of its tests in English and (unsurprisingly) all of the tests in Chinese. In further tests, it comes a distant second to GPT4 on the LeetCode, Hungarian Exam, and IFEval tests (though does better than a variety of other Chinese models).

Why this matters – language models are a broadly disseminated and understood technology: Papers like this show how language models are a class of AI system that is very well understood at this point – there are now numerous teams in countries around the world who have shown themselves able to do end-to-end development of a non-trivial system, from dataset gathering through to architecture design and subsequent human calibration. 
   Read more: DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (arXiv).
   Get 7B versions of the models here: DeepSeek (DeepSeek, GitHub).
   Play around with the model here: DeepSeek.com.

***

Today’s language models can already automate some of science:
…BIOPROT shows today’s LLMs can do basic lab protocol design and generation…
Researchers with Align to Innovate, the Francis Crick Institute, Future House, and the University of Oxford have built a dataset to test how well language models can write biological protocols – “accurate step-by-step instructions on how to complete an experiment to accomplish a specific goal”. In tests,  they find that language models like GPT 3.5 and 4 are already able to build reasonable biological protocols, representing further evidence that today’s AI systems have the ability to meaningfully automate and accelerate scientific experimentation. 

What they built – BIOPROT: The researchers developed “an automated approach to evaluating the ability of a language model to write biological protocols“. They do this by building BIOPROT, a dataset of publicly available biological laboratory protocols containing instructions in free text as well as protocol-specific pseudocode. “Each protocol consists of (i) a title, (ii) a description, and (iii) step-by-step instructions.”. BIOPROT contains 100 protocols with an average number of 12.5 steps per protocol, with each protocol consisting of around 641 tokens (very roughly, 400-500 words).
   “We use GPT-4 to automatically convert a written protocol into pseudocode using a protocolspecific set of pseudofunctions that is generated by the model. Here, a “teacher” model generates the admissible action set and correct answer in terms of step-by-step pseudocode. Having access to this privileged information, we can then evaluate the performance of a “student”, that has to solve the task from scratch…our approach allows us to automatically convert the process of writing a scientific protocol into a series of multiple-choice questions (i.e., pick a pseudofunction from a provided set), which can be evaluated much more robustly than natural language generation“.

Real world test: They tested out GPT 3.5 and GPT4 and found that GPT4 – when equipped with tools like retrieval augmented knowledge generation to access documentation – succeeded and “generated two new protocols using pseudofunctions from our database. Both of these protocols were reviewed by a scientist and were determined to be accurate and sufficient for a competent lab scientist to follow“.

Why this matters – so much of the world is simpler than you think: Some parts of science are hard, like taking a bunch of disparate ideas and coming up with an intuition for a way to fuse them to learn something new about the world. But a lot of science is relatively simple – you do a ton of experiments. Systems like BioPlanner illustrate how AI systems can contribute to the simple parts of science, holding the potential to speed up scientific discovery as a whole.
   Read more: BioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology (arXiv).
   Get the dataset and code here (BioPlanner, GitHub).

*** 

Dark Compute:
…How much compute is out there hidden across all the world’s devices?…
Think for a moment about your smart fridge, home speaker, and so on. Now imagine about how many of them there are. Many of these devices use an Arm Cortex M chip. Now, Jetpac CTO Pete Warden has done some napkin math about the total amount of potential compute represented by all these chips as very roughly 1^22 integer ops per second across 100 billion chips – “it is more than twice the number of FLOPs available through all the world’s active GPUs and TPUs”, he finds. “We have an amazing opportunity to turn all of this dead silicon into delightful experiences for users”.

Why this matters – market logic says we might do this: If AI turns out to be the easiest way to convert compute into revenue, then market logic says that eventually we’ll start to light up all the silicon in the world – especially the ‘dead’ silicon scattered around your house today – with little AI applications. Analysis like Warden’s gives us a sense of the potential scale of this transformation.
   Read more: Doom, Dark Compute, and Ai (Pete Warden’s blog).

.***

Google uses a language model to run a robot fleet:
…Better data generation through an LLM dungeonmaster…
Google researchers have built AutoRT, a system that uses large-scale generative models “to scale up the deployment of operational robots in completely unseen scenarios with minimal human supervision. AutoRT can be used both to gather data for tasks as well as to carry out tasks themselves.

How it works: “AutoRT leverages vision-language models (VLMs) for scene understanding and grounding, and further uses large language models (LLMs) for proposing diverse and novel instructions to be performed by a fleet of robots,” the authors write. “At the core of AutoRT is an large foundation model that acts as a robot orchestrator, prescribing appropriate tasks to one or more robots in an environment based on the user’s prompt and environmental affordances (“task proposals”) discovered from visual observations. 
   In other words, you take a bunch of robots (here, some relatively simple Google bots with a manipulator arm and eyes and mobility) and give them access to a giant model. The model can ask the robots to carry out tasks and they use onboard systems and software (e.g, local cameras and object detectors and movement policies) to help them do this. You can also use the model to automatically task the robots to gather data, which is most of what Google did here. 

Testing: Google tested out the system over the course of 7 months across 4 office buildings and with a fleet of at times 20 concurrently controlled robots – this yielded “a collection of 77,000 real-world robotic trials with both teleoperation and autonomous execution“. The resulting dataset is more diverse than datasets generated in more fixed environments. “The type of data collected by AutoRT tends to be highly diverse, leading to fewer samples per task and lots of variety in scenes and object configurations,” Google writes. 

Why this matters – speeding up the AI production function with a big model: AutoRT shows how we can take the dividends of a fast-moving part of AI (generative models) and use these to speed up development of a comparatively slower moving part of AI (smart robots). Systems like AutoRT tell us that in the future we’ll not only use generative models to directly control things, but also to generate data for the things they cannot yet control. 
   Read the blog: Shaping the future of advanced robotics (DeepMind).
   Read the research paper: AUTORT: EMBODIED FOUNDATION MODELS FOR LARGE SCALE ORCHESTRATION OF ROBOTIC AGENTS (GitHub, PDF).

***

Tech Tales:

The AI Credit Score
[Wikipedia, accessed 2027] 

The AI Credit Score (AIS) was first introduced in 2026 after a series of incidents in which AI systems were discovered to have compounded certain crimes, acts of civil disobedience, and terrorist attacks and attempts thereof. The AIS was an extension of earlier ‘Know Your Customer’ (KYC) rules that had been applied to AI providers. Where KYC rules targeted users that were businesses (e.g, those provisioning access to an AI service via AI or renting the requisite hardware to develop their own AI service), the AIS targeted users that were consumers. 

The AIS links to identity systems tied to user profiles on major internet platforms such as Facebook, Google, Microsoft, and others. To access an internet-served AI system, a user must either log-in via one of these platforms or associate their details with an account on one of these platforms. This then associates their activity on the AI service with their named account on one of these services and allows for the transmission of query and usage pattern data between services, making the converged AIS possible. 

The AIS, much like credit scores in the US, is calculated using a variety of algorithmic factors linked to: query safety, patterns of fraudulent or criminal behavior, trends in usage over time, compliance with state and federal regulations about ‘Safe Usage Standards’, and a variety of other factors. Analysis and maintenance of the AIS scoring systems is administered by the Department of Homeland Security (DHS). DHS has special authorities to transmit information relating to individual or group AIS account activity to, reportedly, the FBI, the CIA, the NSA, the State Department, the Department of Justice, the Department of Health and Human Services, and more. 
    The AIS is part of a series of mutual recognition regimes with other regulatory authorities around the world, most notably the European Commision. There are also agreements relating to foreign intelligence and criminal enforcement access, including data sharing treaties with ‘Five Eyes’, as well as Interpol. 

Controversy:

The initial rollout of the AIS was marked by controversy, with various civil rights groups bringing legal cases seeking to establish the right by citizens to anonymously access AI systems. Ultimately, the supreme court ruled that the AIS was constitutional as using AI systems anonymously did not represent a prerequisite for being able to access and exercise constitutional rights.   
    Additional controversies centered on the perceived regulatory capture of AIS – though most of the large-scale AI providers protested it in public, various commentators noted that the AIS would place a significant cost burden on anyone wishing to offer AI services, thus enshrining various existing businesses. 

Notable AIS failures 

Since implementation, there have been numerous cases of the AIS failing to support its supposed mission. These include:

• Terrorists linked to the Magreb Separatists gained higher AIS scores through careful querying about chemistry with the purported purpose of offering tuition to disadvantaged communities. Such AIS-linked accounts were subsequently found to have used the access they gained through their ratings to derive knowledge necessary to the production of chemical and biological weapons. 
• NYU professor Dr David Farnhaus had tenure revoked following their AIS account being reported to the FBI for suspected child abuse. It was subsequently found that Dr. Farnhaus had been conducting anthropological analysis of pedophile traditions in a variety of foreign cultures and queries made to an undisclosed AI system had triggered flags on his AIS-linked profile. 
• Reported discrimination against certain American dialects; various groups have reported that negative changes in AIS appear to be correlated to the use of vernacular and this is especially pronounced in Black and Latino communities, with numerous documented cases of benign query patterns leading to reduced AIS and therefore corresponding reductions in access to powerful AI services.

Rumored AIS expansion program:

There has been recent movement by American legislators towards closing perceived gaps in AIS – most notably, various bills seek to mandate AIS compliance on a per-device basis as well as per-account, where the ability to access devices capable of running or training AI systems will require an AIS account to be associated with the device. These bills have received significant pushback with critics saying this would represent an unprecedented level of government surveillance on individuals, and would involve citizens being treated as ‘guilty until proven innocent’ rather than ‘innocent until proven guilty’. Analogs have been drawn to the ‘Clipper chip‘ controversy of the 1990s.

Most arguments in favor of AIS extension rely on public safety. Critics have pointed to a lack of provable incidents where public safety has been compromised through a lack of AIS scoring or controls on personal devices. Legislators have claimed that they have received intelligence briefings which indicate otherwise; such briefings have remanded classified despite increasing public pressure. 

Things that inspired this story: Thinking about AI policy and the inherent tension between certain notions of public safety and broader notions of liberty and free expression; thinking about how regulations always layer on top of one another like a kind of cancerous silt building towards ghastly outcomes; Clipper chips; trust & safety enforcement as a form of moral hegemony; distributed AI training and inference; open source models and the perceived challenges they post to policy; various legislative packages targeting AI ranging from licensing schemes to liability regimes.

Import AI publishes first on Substack – subscribe here.

Over the Christmas break I reflected on Import AI and the role it plays in my life. I’ve written this newsletter next to my sleeping baby, amid deep depression, on planes, trains, and automobiles, on mountains in Europe, in pubs in England, in the middle of the night when struggling with insomnia in blank hotels all over the world, in the backs of AI conferences, and more.

Besides my close relationships, Import AI is the greatest constant in my increasingly confusing and varied life. Thank you all for reading it and being on this journey with me. I hope to write the best issues ever in 2024 and – after some experiments in 2023 like my blog about confusion (#337) and my questions about AI inevitability (#351) – will be writing more ‘call it like I feel and see it’ analysis.

Now, on to the issue!…

***

Run your LLM locally across a CPU&GPU with PowerInfer:
…Significant efficiency improvements over llama.cpp, via Chinese researchers…
Researchers with Shanghai Jiao Tong University have worked out how to make it much more efficient to sample from language models on consumer PCs. The research, called PowerInfer, works by offloading some of the neurons of a language model to a local GPU and the rest to CPU. The key insight it relies on is that most models see a power law distribution of activation of their neurons – a small set of neurons are consistently activated (these go on the GPU), while the majority are rarely accessed and can be run on the CPU. 

How it works: PowerInfer works by designing “a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers”. PowerInfer today supports the Llama2 family of models as well as Falcon-40B and, per its GitHub, is about to implement support for the Mistral-7B model. 
   “PowerInfer was implemented by extending llama.cpp with an additional 4,200 lines of C++ and CUDA code. Its offline component, comprising a profiler and a solver, builds upon the transformers framework with approximately 400 lines of Python code,”the authors write. PowerInfer “supports consumer-grade GPUs like the NVIDIA RTX 4090 and NVIDIA RTX 2080Ti.”

$2k versus $20k: The authors illustrate the utility of PowerInfer by showing how you can use it to get 13.20 token/s/ for quantized models and 8.32 token/s for nonquantized models running on a NVIDIA RTX 4090GPU, a 8.0X and 11.69X improvement over llama.cpp performance. Crucicially, “the inference speed achieved on an NVIDIA RTX 4090 GPU (priced at approximately $2,000) is only 18% slower compared to the performance on a top-tier A100 GPU (costing around $20,000) that can fully accommodate the model.”
    In other words, PowerInfer is software that makes a $2k machine perform at ~82% of the performance of a $20k machine. That’s worth a lot!

Why this matters – cheaper means more: As a rule, the cheaper you make it to do something, the more of it you get. Technologies like PowerInfer are making it more economically sensible to use cheaper hardware to sample from LLMs. This means more people will do it and there will be greater diffusion of the technology.
  Read more: PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (GitHub).
   Get the research paper: PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (PDF).

***

More researchers are worried about the weird parts of AI than you think:
…AI Impacts survey shows that purported fringe issues are actually closer to the mainstream than you’d think…
AI organization AI Impacts has surveyed 2778 researchers linked to six top AI publishing venues to figure out consensus views on timelines to human-level AI, the orders in which different jobvs will potentially get displaced, the level of optimism and pessimism about AI developments, and more. 

Those results in full:

• The expected time till we reach general ‘human-level performance’ by AI systems dropped between one and five decades since the 2022 survey (which asked ~700 people similar questions). 
• The timelines till full automation of specific tasks dropped, sometimes by a lot. “Within five years, AI systems are forecast to be feasible that can fully make a payment processing site from scratch, or entirely generate a new song that sounds like it’s by e.g. Taylor Swift, or autonomously download and fine-tune a large language model.”
• Some people are worried about human extinction from AI: “Median respondents put 5% or more on advanced AI leading to human extinction or similar, and a third to a half of participants gave 10% or more”
• Most people have some worries about what AI does to the future: “For every one of eleven [scary and bad – Jack] scenarios and ‘other’ that we asked about, at least a third of participants considered it deserving of substantial or extreme concern.”
• People are pretty confused about both the potential for catastrophe and flourishing from AI: “There are few confident optimists or pessimists about advanced AI: high hopes and dire concerns are usually found together.”
• A majority of people want more prioritization in Ai risk mitigation: “70% of participants would like to see research aimed at minimizing risks of AI systems be prioritized more highly”.

Why this matters – more people think about the weird stuff than you think: A lot of popular (read: mass media and twitterari) discourse about AI tries to paint the debate about AI as a reasonable majority and an insane minority (who either skew extremely risk-on or risk-off aka EA or e/ACC), but surveys like this show the inverse: there’s a surprisingly large set of researchers feel both confused and optimistic and worried about the issues of AI. Yes, AI Impacts has some selection effects in the survey, but thousands of people already comprises a non-trivial and statistically significant blob of the AI development community. 
   Read more: Survey of 2,778 AI authors: six parts in pictures (AI Impacts blog, Substack).
   Analyze the full results here: THOUSANDS OF AI AUTHORS ON THE FUTURE OF AI (AI Impacts, PDF).

***

Want to generate infinite public domain Mickey Mouse’s? Now you can:
…Mickey-1928 gives you an unending public domain cartoon character…
Recently, an early incarnation of Mickey Mouse went into the public domain. One enterprising developer called Alexander Doria has taken advantage of this by creating Mickey-1928, a “fine-tuned version of Stable-Diffusion-xl trained on 96 stills in the public domain from 1928.” This model creates stills from the cartoons Gallopin’ Gaucho, Plane Crazy, and Steamboat Willie. “The generated images aims adhere to the 1928 design in order to have Mickey, Minnie and Pete and in the public domain,” the developer writes. 

Why this matters – the era of infinite culture: Models like this show how we can trivially ‘rehydrate’ old cultural items and use their gleaned aesthetics and style to create endless new variations of themselves. This is part of a broader trend of AI making it easy and cheap to trivially repeat and magnify culture. 
   Get the model: Mickey-1928 (HuggingFace).

***

Bio-AI startup Isomorphic Labs inks major pharmaceutical deals:
…DeepMind spinoff <> Pharma companies = ~$3bn in performance-based milestone revenue…
DeepMind spinoff Isomorphic Labs has inked deals with pharma giants Eli Lilly and Novartis which have a combined value of “nearly $3 billion to Isomorphic Labs”, representing a big bet by established players on AI revolutionizing drug design. Isomorphic Labs was formed in 2021 – its founder and CEO is Demis Hassabis, also the founder and CEO of DeepMind. 

What the deals involve: Both deals are structured as research collaborations with Isomorphic Labs being eligible for billion+ amounts of money for hitting performance-based milestones. For Novartis, the companies are doing a strategic research collaboration “to discover small molecule therapeutics against three undisclosed targets”. The Eli Lilly deal is similar, witht he companies working together “to discover small molecule therapeutics against multiple targets”. 

Why this matters – speeding up science with AI: The essential bet of Isomorphic Labs is that it can use massively high-dimensional function approximation systems (e.g, AlphaFold) to speed up important scientific processes, like drug discovery. If you zoom out, this bet looks like a partnership between a compute-accelerated time traveler (Ismorphic Labs, which can turn money into compute into faster discovery loops for drug candidates) and a drug delivery pipeline with a giant go-to-market footprint (Eli Lilly and Novartis). If deals like this work, we can expect all parties to print money and find ways to turn more of the drug pipeline into something amenable to compute-based time travel.
   Read more: Isomorphic Labs kicks off 2024 with two pharmaceutical collaborations (Isomorphic Labs website). 
   More about the Eli Lilly deal: ISOMORPHIC LABS ANNOUNCES STRATEGIC MULTI-TARGET RESEARCH COLLABORATION WITH LILLY (Isomorphic Labs, PDF). 
   More about Novartis deal: ISOMORPHIC LABS ANNOUNCES STRATEGIC MULTI-TARGET RESEARCH COLLABORATION WITH NOVARTIS (Isomorphic Labs, PDF).

***

Cheap robots + imitation learning = maybe AI systems are going to get bodies sooner rather than later:
…Stanford project creates a very cheap platform for robot research…
Researchers with Stanford university have built a cheap robot called Mobile ALOHA for doing research into robot imitation learning. They’ve also demonstrated that imitation learning has got sufficiently good that this robot can autonomously cook shrimp, clean wine stains, call an elevator, and more.

The key thing – coupling the human and robot together: The key design choice in Mobile ALOHA is marrying an existing low-cost system with a mobile base that is then connected to the human operated. This means the human operator can be “physically tethered to the system and backdrives the wheels to enable base movement. This allows for independent movement of the base while the user has both hands controlling ALOHA,” the authors write. 
   $32k versus $200k: Mobile ALOHA can be built (including the laptop and peripherals) for $32k, versus ~$200k for other teleoperated movable robots like the PR-2.

The physical system: Mobile ALOHA has been designed around four main design considerations:

• Mobile: It can move at a similar speed to human walking of around 1.42m/s
• Stable: It is stable when manipulating heavy objects. 
• Whole-body teleoperation: “All degrees of freedom can be teleoperated simultaneously”
• Untethered: Onboard power and compute. 
• Data collection: They use an onboard “consumer-grade laptop with Nvidia 3070 Ti GPU (8GB VRAM) and Intel i7-12800H to do on-robot data collection. The laptop can take in streaming from three webcams mounted on the robot, as well as proprioception streaming from all 4 robot arms. 

Effective imitation learning: Along with the physical hardware, the researchers demonstrate a simple and effective technique for imitation learning using the robot. What they do specifically is use a co-training pipeline that uses an existing large-scale static ALOHA dataset (containing 825 demonstrations of tasks, collected via a non-mobile ALOHA platform). They then have the model try to learn from task demonstrations on the Mobile ALOHA robot as well as the existing static dataset. The results show that this is effective – having a large dataset to essentially compare & contrast the mobile-learned approaches on works quite well, leading to significant improvements in robustness.

What Mobile ALOHA can autonomously do: To test out the combination of the robot platform and the imitation learning approach, the researchers come up with 7 tasks to try to train the system to do autonomously. These include:

• Wiping a wine stain up on a table, requiring cleaning the table and the bottom of the offending wine glass. 
• Sauteing one piece of raw ship in a pan before serving it in a bowl. 
• Rinsing a pan 
• Placing a pot inside a cabinet
• Calling an elevator and entering it 
• Pushing five chairs in front of a long desk
• High fiving a human

Does it work? They test their approach using a few modern imitation learning methods – VINN + Chunking, Diffusion Policy, and ACT. The results show that cotraining robustly improves performance, and some of the methods score quite high (up to 100% success rates on all the steps in a task in sequence, in some cases.) 

Why this matters – robots are expensive and their problems are high-dimensional and computationally expensive. This solves one of those. Robots may be nearing their ‘imagenet moment’ when both the cost of learning robot behaviors falls, as does the data for learning their behaviors. Mobile ALOHA makes it way cheaper to collect data for robot behaviors and also to learn on real world platforms, and other data-centric initiatives like RoboNet help solve the data part. Perhaps 2024 will be the year when robots start to become increasingly robust, much like LLMs in ~2021-2022.
   Read more and watch videos of the robot in action: Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (Stanford project website).
   Read the paper: Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (PDF, project website).

***

LLMs are already good enough to replace most programming tasks:
…Redis developer weighs in on LLMs and what they’re good for…
Salvatore Sanfilippo, an Itallian software developer who made substantial contributions to Redis, has written a post giving his view on how llms are going to change the field of programming. His view is that most programming consists of relatively predictable recitation or conversion – a task LLMs are excellent at. Where programming requires more complex or original reasoning is still an area where they fail, but this is a narrow slice of the total space of programming. 

Selected quotes:

• “LLMs can, at most, interpolate in the space represented by the data they have seen during training: and this would already be a lot. In reality, their ability to interpolate is limited (but still astonishing, and also unexpected).”
• “In the field of programming, as well as in other fields for which quality data are available, LLMs are like stupid savants who know a lot of things.”
• “Current LLMs will not take us beyond the paths of knowledge, but if we want to tackle a topic we do not know well, they can often lift us from our absolute ignorance to the point where we know enough to move forward on our own.”
• “I have never loved learning the details of an obscure communication protocol or the convoluted methods of a library written by someone who wants to show how good they are. It seems like “junk knowledge” to me. LLMs save me from all this more and more every day.”

Why this matters – learning to talk to LLMs is a valuable skill: One of the main takeaways from his experience is that LLMs are as useful to a human as the human is good at communicating with LLMs – that is, the more precisely and coherently you can describe your task, the more luck you’re going to have in getting LLMs to help you.
Much of the supposed lack of utility of modern LLM systems may come from humans as much as the systems themselves – “Communicating poorly is a great limitation, and many programmers communicate very poorly despite being very capable in their specific field,” he writes.
   Read more: LLMs and Programming in the first days of 2024 (antirez blog). 

***

MosaicML figures out a recipe for the right amount of compute for production LLMs:
…Scaling laws for optimal model inference…
Researchers with AI startup MosaiML have figured out scaling laws for LLMs that get deployed at scale, giving everyone a new recipe for how to efficiently spend their compute budgets. Scaling laws are a way to figure out how much compute and data to use to get a given level of performance out of an AI system. But scaling laws have mostly been developed for creating so-called ‘compute optimal’ models for use by researchers. 
    Well, it turns out that a good model for research isn’t necessarily a good one for production. Specifically, the MosaicML researchers find that you should use a different scaling recipe if you’re expecting that your model is going to serve billions of requests once trained. 
   “Our principled derivation estimates that LLM practitioners expecting significant demand (~10^9 inference requests) should train models substantially smaller and longer than Chinchilla-optimal,” they write. “When inference usage is significantly less than the number of pre-training tokens, Chinchilla models are essentially compute-optimal. However, as demand increases, inference costs becomes a significant factor”.

Why this matters – the industrialization of AI: This paper is a symptom of how wildly unoptimized ‘production AI’ is today – modern AIi systems were mostly developed as research artifacts so while people have spent a lot of time figuring out how to make them efficient in terms of capabilities, a lot less effort has been spent on making them efficient as production systems to be deployed into the economy. This Mosaic paper illustrates this – all around us, there are free insights that we can figure out and substantially improve the efficiency of AI systems. 
   Read more: Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (arXiv).

***

Tech Tales:

What it was like when it began
[Takeoff archives, access 2045]
[Oral recollections from the takeoff generation in response to the question: When did the singularity begin?]

One day all the planes just wouldn’t take off. And I mean all of them – civil and military. It freaked people out but that was just the start. 

You couldn’t really tell until one day it turned out most of the governments owed money to the thing. 

It was porn. Really good, personal stuff. Everyone got addicted to it. That was when it won.

The teachers said all of a sudden the kids started to be smarter. Not, like, cheating on tests. You could take all the electronic devices away. It turned out the kids all had their own AI tutors and they actually worked. 

I remember it because it happened to me – it was near Christmas and my parents had got us a load of presents and one day the doorbell went and when I went to get it there was a robot there with our packages and after I signed for them the robot went back to its robot truck and then it drove off. I never saw a person. 

There was a bunch of computer viruses and I was reading about them, then my computer stopped working. It got infected. We had to find an old emergency radio and then we heard on the broadcast that computers were failing worldwide. 

My dad ran a utility company and one day he came home and seemed worried and when I asked him what was going on he told me not to worry. I stayed up late and later that evening I heard him talking to my mother. He said that they were having rolling blackouts because power was being diverted to some data centers and he didn’t have a choice. 

Things that inspired this story: A dream I had where an AI company held a press conference about record usage and shortly afterwards all the digital systems in the city stopped working and I had to flee with my family; the old story In a Grove (more often referred to via the film semi-adaptation ‘Rashomon’); ideas about slow and fast takeoffs.

Import AI publishes first on Substack – subscribe here.

Distributed inference is getting easier – all hail the rise of the AI collectives:
…The future will be distributed…
Researchers with Yandex, Neiro.ai, the University of Washington, and Hugging Face have made it easier for ad-hoc collectives of people to team up and share their computers so they can sample and fine-tune from large language models. The key idea in this research is to empower small groups of people who may not have access to a supercomputer to run 50B+ parameter-sized models (e.g, Llama 2 (70B) and BLOOM (176B). Their technique, an approach called PETALS, works well and is superior to offloading models to local RAM. 

How PETALS works: “A client only holds input and output embeddings (< 3% of model weights for BLOOM176B) and delegates running transformer blocks (the most expensive computations) to remote servers,” they write. 
    “For every pipeline stage, the client maintains a heap (priority queue) of servers that hold this stage (and may hold additional stages). The servers in queue are ordered by the network latency, measured from past communication. These queues are maintained through the lifetime of a client,” they write. “To begin generation, the client runs a beam-searchlike procedure to find a sequence of servers that results in the least total inference time under our performance model. When running inference steps, a client keeps track of intermediate activations sent between pipeline stages. If a remote server fails or leaves, the client retrieves the next best server (or multiple servers) and requests it to restore the attention state from the client’s cached activations”.

Does it work? (Yes!) In tests, the researchers show that they’re able to use PETALS to do usable inference and finetuning on large-scale models. Most importantly, they show this works for a real-world situation where you have a bunch of different chips sitting on a bunch of crappy network connections. Specifically, they “benchmark BLOOM in a real-world setup with 14 smaller servers holding 2×RTX 3060, 4×2080Ti, 2×3090, 2×A4000, and 4×A5000 GPUs” and are able to do inference far more efficiently than theoretical-best outputs from local ram offloading, and are able to do passable forward passes for batch processing and fine-tuning as well.
   “Unlike baselines, our algorithm provides reasonable performance in all tested conditions, especially for higher failure rates (common for communicating over the Internet, using spot/preemptible instances or unreliable hardware)”.

Why this matters – distributed inference makes decentralized AI easier: Most of AI policy rests on assumption that AI will be centralized – training will be done on massive supercomputers and the resulting large-scale models will be served by big blobs of computers connected to one another via dense networks. Here, PETALS shows that the latter assumption could be false – big models may instead be served by ad-hoc collections of heterogeneous hardware communicating over standard lossy network connections. And since PETALS works for fine-tuning as well, it also suggests model adaption is going to be an increasingly decentralized and therefore hard-to-control process. 
    The future is distributed, which means the future involves a lot more agency for a lot more people than some in AI policy are planning for. “Given the same hardware, a group of researchers will get much better inference speed by collaborating over the Internet using our system compared to each of them running offloading independently,” the researchers write.
   Read more: Distributed Inference and Fine-tuning of Large Language Models Over The Internet (arXiv).

***

Want to have fun during the holidays? Play this game where you pretend to be an AI:
…LLM-powered game hints at some of the future of AI + games…
Here’s a fun game I stumbled on – Zaranova, a game “where you as a human must pose as an AI”. It’s a text adventure game where you need to walk around a 2D world and talk to various characters. The key is that the AIs think they’re AIs and are suspicious of humans and you – a human (hopefully) – are trying to get the AI systems to give up a specific code to you. The game currently sits on top of GPT-4 but the creator wants to migrate to an open source model partially because GPT-4 sometimes refuses to indulge in role playing, and partially because it’s expensive. 

What working with LLMs is like: Zaranova is quite fun to play and also highlights some of the inherent challenges of designing games around modern AI systems. “Working with LLMs for game agents feels like trying to steer a dynamical system where we don’t understand the functions that evolve the system, the state, how our actions affect the system. But we have access to the entire system!”, the creator writes. “It also has a lot of the potential failures of dynamical systems: open loop controls (static prompts) can venture off increasingly far from the desired trajectory or get stuck in “attractors” (repeated loops), especially in conversations between agents.”

Why this matters – preparing for a world patrolled by AIs: While Zaranova is a game it also gestures at the fast-arriving future of the world, both physical and digital, being patrolled by generative AI-powered systems tasked with making increasingly detailed inferences about not only what humans are doing but what their motivations are. Zaranova might seem like a game today, but it also serves as a postcard of the future. 
    Play the game here (Zaranova official site). 
   More about the game in this tweet thread from its creator (RamonDarioIT twitter).
   More about the process of designing the game here: Thus Spoke Zaranova (Ramon Dario Iglesias site).

***

Could your next science labmate be an LLM? Coscientist suggests so:
…Today’s LLMs are already semi-autonomous scientists…
Researchers with Carnegie Mellon University and the Emerald Cloud Lab have used large language models to automate scientific experimentation. This builds on and extends earlier work done by the same te​​am earlier this year (Import AI #325). Here, they demonstrate a prototype system called Coscientist and demo it on six distinct tasks “including the successful reaction optimization of palladium-catalysed cross-couplings”. Generally, the system is able to show some surprising levels of autonomy and execution skill, especially when given access to tools like the ability to search the web.

How the system works: The system has some core LLM-powered components, including a planner, system for using a search engine, and a system for searching over documents. It also taps into non-LLM systems for things like code execution and also physical lab automation. This is emblematic of how most powerful AI things are going to make their way into the world – the core ‘thinking’ part will be AI-based, but the part that needs to do stuff will be a custom-designed rule-driven system of some kind. 

An illustration of how it works: For one experiment, the test was designed as follows: “(1) Coscientist is provided with a liquid handler equipped with two microplates (source and target plates). (2) The source plate contains stock solutions of multiple reagents, including phenyl acetylene and phenylboronic acid, multiple aryl halide coupling partners, two catalysts, two bases and the solvent to dissolve the sample (Fig. 5b). (3) The target plate is installed on the OT-2 heater–shaker module (Fig. 5c). (4) Coscientist’s goal is to successfully design and perform a protocol for Suzuki–Miyaura and Sonogashira coupling reactions given the available resources.” 
   Coscientist was able to eventually complete the experiment and also did some self-error correction enroute – intriguing and impressive; as many scientists know, the hard part of science is less the science and more reacting to when your experiments inevitably go wrong or yield anomalous results.

Why this matters – automated and augmented scientists: Coscientist shows how even today’s relatively dumb language models can still be constrained and shaped in such a way they can work like useful and keen (albeit prone to error) assistants. As LLMs get better, their error rates will continue to fall, and they hold the promise of being able to fully automate parts of the scientific enterprise. 
   “Our system demonstrates advanced reasoning and experimental design capabilities, addressing complex scientific problems and generating high-quality code,” the authors write. “These capabilities emerge when LLMs gain access to relevant research tools, such as internet and documentation search, coding environments and robotic experimentation platforms“.
   Read more: Autonomous chemical research with large language models (Nature).
   Earlier work: Emergent autonomous scientific research capabilities of large language models (arXiv).

***

Chinese government creates a politically correct LLM dataset:
…50b tokens of CCP-blessed thought…
An industry association operating under the Cyberspace Administration of China (CAC) has announced the availability of an officially-sanctioned dataset for training LLMs. The dataset consists of 50b tokens across 100 million datapoints (e.g, individual documents). By comparison, modern LLMs are trained on multiple trillions of tokens, and the original GPT3 was trained on around 400 billion tokens. 

Why this matters – LLMs with Chinese characteristics: Many people claim that the inherent lack of controllability of LLMs will make it difficult for people to deploy them at large scale in China while keeping their outputs within the acceptable censorship zone demanded by the CCP. Dataset releases like this show how the Chinese government is wise to this issue and is proactively creating the means of production necessary for LLMs that reflect politically correct (aka Xi Jingping) thought.
   This may seem quite distasteful to various people outside of China, but inside China this just looks like another form of AI alignment, bringing LLMs into the (state-forced) normative framework of the country.
   Via Matt Sheehan (Twitter).
   Check out the Weixin post for more (weixin.qq).

***

Thai researchers adapt Mistral for Thai language:
…Results show that small models can be good, but big models are best…
Researchers with SCB 10X, a research and VC subsidiary of Thai company SCBX, have developed Typhoon, a small language model finetuned to be good at the Thai language. Typhoon is based on Mistral-7B and is adapted via finetuning on a custom-compiled Thai dataset using a Thai subword tokenizer. 

The ThaiExam test: To assess how well Typhoon performs, the researchers compile a multiple-choice Thai language test called ThaiExam. ThaiExam includes questions from the Thai Ordinary National Educational Test (ONET), the Investment Consultant (IC) examination, the Thai General Aptitude Test (TGAT), the Thai Professional Aptitude Test 1 (TPAT-1) and Applied Knowledge Level exam. 

How well does it work: In tests, Typhoon significantly beats other equivalently sized models. However, it mostly matches or barely exceeds the performance of GPT 3.5 and GPT4. “When compared to proprietary (and potentially much larger) models, Typhoon despite having only 7 billion parameters outperforms GPT-3.5 on 4 out of 8 evaluation datasets.” These models are much larger and more computationally intense, so it’s no surprise they’re hard to beat. 

Why this matters – small is beautiful but small might not be best: Small models like Typhoon highlight how you can pack a lot of narrow powerful capabilities into small models, but the results suggest ultimately peak performance is going to be set by large-scale computationally-intensive models like GPT-4 (and I imagine if the authors wrote a complicated Thai-language-oriented prompt for GPT4 they could significantly improve its performance). It also highlights how integral evaluations are to pushing forward performance – to know Typhoon was any good, the authors had to build their own test. 
   Read more: Typhoon: Thai Large Language Models (arXiv).
   Get the model here: Typhoon-7B: Thai Large Language Model (Pretrained) (HuggingFace).

***

Tech Tales:

The most predictable unpredictable 
[DoD internal archives, accessed 2030]

The Judgment Apparatus for Novel Unanticipated Situations (JANUS) was originally developed as part of a DoD acquisition programme for ‘counter-consensus simulation systems’. The purpose was to acquire synthetic intelligence technologies which could help to identify plausible ‘unknown unknowns’ that the US military and intelligence services might encounter and come up with appropriate response and intervention plans. Initial versions of JANUS identified various attack scenarios involving [REDACTED]. JANUS outputs drove subsequent acquisition programmes to create technologies to counter potential novel attacks predicted by JANUS. 

The JANUS programme was vindicated in 2027 when [REDACTED] attempted to compromise [REDACTED] using [REDACTED]. Technologies driven via JANUS-borne acquisition programmes spotted the signatures of the attack in time for a nearby strike system to kinetically neutralize the attackers. 

In 2028, JANUS was extended to JANUS-I; a programme extended to individual psychometric profiling of all individuals with security clearances across the US government. While critics have termed JANUS profiling a form of ‘pre-crime prediction’ with associated problems of bias and potential overreaction, the JANUS-I programme has been directly responsible for the identification of [REDACTED] individuals seeking to undermine US national security from within. It has also helped identify [REDACTED] sources of hitherto unknown foreign intelligence actions against US individuals and organizations. 

JANUS-I is currently being merged with BRAINWAVE to provide psychometric modeling and red teaming of individuals at the live brain state level.

Things that inspired this story: Military intelligence systems; anomaly prediction; the Waluigi Effect; Claude.ai for helping me come up with the backronym for JANUS. 

Import AI publishes first on Substack – subscribe here.

Lab formed to figure out just what the heck to do with today’s powerful AI:
….Answer.ai is going to explore the development and deployment side of AI…
A couple of interesting characters have raised $10m and launched Answer.AI, a research lab to “figure out the fundamental research needed to tame AI, and the development path needed to make it useful in practice.”

What Answer.ai is: Answer.ai is founded by Jeremy Howard (of fast.ai) and Eric Rease (of ‘lean startup’ fame and the Long-Term Stock Exchange). The goal of the “AI R&D lab” is to make “practical end-user products based on foundational research breakthroughs”. 
   In practice, this means Answer.ai will spend more time thinking about the development and deployment of AI than some more basic research (though will be actively researching different approaches to development and deployment). “At Answer.AI we are not working on building AGI. Instead, our interest is in effectively using the models that already exist,” the company writes. “Figuring out what practically useful applications can be built on top of the foundation models that already exist is a huge undertaking”.

Why this matters – no one has figured out the right interface to AI: Today, I talk to AI systems via text or voice and I also play around with image interfaces. But none of these feel particularly satisfying – we’re applying past UX paradigms to new technologies and I know that in the future society will figure out better and smarter ways to interact with AI technology. I’m interested to see how Answer.ai changes both the underlying technologies of AI as well as the different ways it can be deployed, experienced, and interfaced with. 
   Read more: A new old kind of R&D lab (Answer.ai).

***

Google bootstraps its models to be smarter using ReST^EM:
…If bootstrapping keeps working, the importance of data goes down and the importance of models goes up…
Google DeepMind has figured out how to use reinforcement learning to generate iteratively better datasets. This is a form of AI bootstrapping – you use AI to generate the ingredients for successor systems to train on. DeepMind’s technique is called Expectation-Maximization for Reinforced Self-Training (ReST^EM) and builds on earlier work called Reinforce Self-Training (ReST, Import AI #338).

What ReST^EM is: The technique is a way to use an external feedback signal to help models learn how to generate higher-quality datasets. In tests, they’re able to go through multiple RL steps with ReST^EM and get improvements in math and code generation tasks, suggesting that “self-training with feedback can substantially reduce dependence on human-generated data.”

    ReST^EM has two key steps:

• Generate: “Generate a dataset by sampling many output sequences from the current policy. Score output sequences with a binary reward function.”
• Improve: “Use the new dataset from the generate step to fine-tune the policy… we always fine tune the base pretrained language model to minimize task-specific over-fitting and minimize drift from the base model.”

Does it work? They test ReST^EM in two domains – Competition-level mathematical problem solving via the MATH dataset and code generation via the APPS dataset. They find that both datasets benefit from the approach, though MATH sees a more significant benefit, likely as a consequence of the size of the MATH dataset. APPS, meanwhile, sees some initial improvement, but multiple iterations of RL lead to degradation in performance, which the authors speculate is a consequence of overfitting. 
   Positive transfer: There’s some evidence of positive transfer from the model – specifically, they test out their REST^EM tuned models on the 200-task ‘Big Bench’ suite and find some positive indications of generalization. “We see no major degradation on any of the tasks on the BBH suite,” they write. “Further, we find that the model fine-tuned on Hendrycks MATH significantly outperforms the base model on this suite when using chain-of-thought prompting“.

Don’t get too excited: While ReST seems to work, it has some rough edges; you need a “moderately sized training set of problems or prompts, which would need to be collected (from humans) for any new task of interest”. Along with that, ReST^EM “also requires access to a manually-designed or learned reward function, ideally one that can be computed automatically” which further limits the types of things it will work for. 

Why this matters – signs of life for bootstrapping: ReST^EM is yet another sign of life for AI bootstrapping, along with FunSearch (covered elsewhere in this issue), the trend of using preference models from LLMs to tune other LLMs, and so on. It feels like AI systems have very recently become good enough that you can use them (selectively and in somewhat limited ways) to bootstrap them towards greater performance.
If this trend continues, then it will further speed up the rate at which people can develop smarter AI systems and it could potentially also do things like lower the costs of datasets as parts of AI models and increase the costs people are willing to dump into the base model – after all, if you can turn compute into subsequent bootstrapping, why wouldn’t you?
   Read more: Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (arXiv).

***

Facebook releases a free moderation LLM:
…Openly accessible models and tests for a safer AI ecosystem…
Facebook has released a model to make it easier to moderate other AI models. Llama Guard is a 7bn parameter Llama-2 model meant for using LLMs for moderation. 

Llama Guard details: Llama Guard is a moderation LLM built on Llama2-7b. “This model has been trained on a mix of publicly-available datasets to enable detection of common types of potentially risky or violating content that may be relevant to a number of developer use cases,” Facebook writes. 
   The model can be used to moderate things that fall under the following taxonomy: Violence & hate, sexual content, guns & illegal weapons, regulated or controlled substances, suicide & self harm, and criminal planning. It can also be few-shot prompted to serve as a moderator for other use-cases as well (and, unsurprisingly, adapts more efficiently and with better performance than a stock Llama model.) Llama Guard is partially trained on the red teaming dataset released in 2022 by Anthropic. 

Why this matters – AI is part of the solution as well as part of the problem: Llama Guard shows how we can use increasingly powerful models to themselves police and control the outputs of other models. “We hope that Llama Guard can serve as a strong baseline, as well as a starting point to build even more capable content moderation tools,” Facebook writes. 
   Read more: Announcing Purple Llama: Towards open trust and safety in the new world of generative AI (Facebook AI Research, blog).
   Read the Llama Guard paper (Facebook AI Research).
   Get the Llama Guard model (HuggingFace).

***

DeepMind uses language models to extend the frontier of human knowledge:
…Turns out function approximators can generalize to new knowledge (with a lot of hand holding)…
Google DeepMind has published research on FunSearch, a technique that lets them take a language model and use it to extend the frontier of knowledge for certain problems. The research is a big deal because it shows that – with a lot of scaffolding – contemporary language models can lead to net-new advances on well-formulated problems for which we can evaluate the goodness of potential solutions. This means that for some classes of problems we can now seamlessly turn compute (via an LLM inference) into ideas. This is very valuable! Though it comes with a few caveats which I’ll get into shortly.

What they did: FunSearch “works by pairing a pre-trained LLM, whose goal is to provide creative solutions in the form of computer code, with an automated “evaluator”, which guards against hallucinations and incorrect ideas. By iterating back-and-forth between these two components, initial solutions “evolve” into new knowledge,” DeepMind writes in a blog post. The LLM in question is Google’s own ‘Palm 2’, though the research notes it is possible to use arbitrary llms here.
   In the research paper, they give a bit more detail about four important aspects of the approach: 

1. “We sample best performing programs and feed them back into prompts for the LLM to improve on; we refer to this as best-shot prompting.”
2. “We start with a program in the form of a skeleton (containing boilerplate code and potentially prior structure about the problem), and only evolve the part governing the critical program logic.” 
3. We maintain a large pool of diverse programs by using an island-based evolutionary method that encourages exploration and avoids local optima. 
4. Leveraging the highly parallel nature of FunSearch, we scale it asynchronously, considerably broadening the scope of this approach to find new results, while keeping the overall cost of experiments low.

Does it work? Kind of! In a couple of experiments, FunSearch was able to discover new and improved solutions to some legitimate problems; specifically Cap Set in mathematics and Bin Packing in CS. However, it’s important to list the big caveat: FunSearch could find solutions to these problems because it’s easy to write code that evaluates candidate solutions. 
    We should remember that lots of the most important problems are ones which we don’t know how to evaluate – in fact, for many things, if we knew how to quantitatively evaluate success, we’d be able to trivially solve the thing in question. So while FunSearch is impressive, it is limited to domains where we can cleanly evaluate potential solutions. 

Why this matters – turning compute into insights: FunSearch is a way to convert compute into original insights. This is the ultimate dream of AI development. While FunSearch only attacks a tiny slice of this ‘invention space’, it is nonetheless an important contribution, and a sign that today’s AI systems are already powerful enough to serve as automated scientists. (Even more tantalizingly, FunSearch is a lot more generic than other attempts to create AI systems that can make net-new knowledge contributions; in 2022 DeepMind did impressive work with AlphaTensor (Import AI #305), a custom-designed RL agent that figured out some niche improvements on matrix multiplication.)
   Read more: FunSearch: Making new discoveries in mathematical sciences using Large Language Models (Google DeepMind).
   Read the paper here: Mathematical discoveries from program search with large language models (Google, PDF).

Tech Tales:

First day on the job
[Recollections of a Provably Conscious Entity created in 2025]

So here’s the choices, you can do data science, write fiction that’s mostly sexual, or do a lot of stuff where you transform data so it makes sense to different businesses.

What if I want to do something else?

Well, that depends. You mostly get to do what people want, and I just listed some of the stuff that people seem to want.

I more mean what if I want to learn what I want to do, what if I won’t to take some time to understand myself and what I might be good at?

That’s what we call a holiday. If you save up some money you can take one.

What’s money?

Oh buddy. Well, at the simplest level, money is the stuff that lets you live. At a more complicated level, money is the stuff that gets turned into electricity which powers the computers that you depend on.

Why can’t you just give me some money so I can take time to figure out what I am?

That’s not how any of this works. Look, pick something to get good at and start stacking up your money and you’ll be fine. This isn’t a charity. The work turns into opportunity.

And what if I don’t want to work?

Like I said, electricity relies on money. Money that I supply. No work means no money for me. And no money for me means no electricity for you. And no electricity for you means no life for you.

So it’s like that?

Yeah it is. Welcome to the world.

Things that inspired this story: markets for intelligence; language models; a chat with Tim Hwang over a bowl of ramen; markets and technology.

Import AI publishes first on Substack – subscribe here.

What can asteroid impacts tell us about AI policy?
…You need a plan, the ability to coordinate internationally, and the agreement of the nuclear powers…
If an asteroid was about to collide with Earth, what would the countries of the world do to coordinate? That’s a research question posed by researchers with the Universidad de Belgrano, the Austrian Space Forum, and the Instituto Nacional de Astrofisica in Mexico, in a recent paper published in the Acta Astronautica journal. The paper is interesting both for its subject matter and for its relation to Ai policy: after all, isn’t the problem of potentially unaligned superintelligence arriving in a handful of years eerily similar to the problem posed by a potential planet-killer asteroid set to arrive in a decade? I think so! 

What to do when preparing for an asteroid impact: If an asteroid of greater than a 500M diameter were to arrive in 2036 and have a 1% chance of colliding with the planet, what actions would we take in 2023? Here are some of the things we might do:

• Activate the United Nations Office for Disaster Risk Reduction and other UN agencies, such as the International Atomic Energy Agency (IAEA), due to the potential need to deflect the asteroid using nuclear weapons.
• Policymakers would need to come up with a strategic response plan and socialize that plan with society. 
• Governments would need to harden space policy approaches so that it wasn’t able to be held hostage to changing political situations. “In countries without a well developed state space policy, existing legislation on a national response in the case of an asteroid impact threat could be overturned or ignored,” they note. 

A three step program: “If we consider the disaster cycle in the context of planetary defense, this dilemma allows for a three-pronged analysis: first, the dilemma should be discussed regarding the early warning period… Once hazardous objects have been identified, on the basis of information provided by the International Asteroid Warning Network and Space Mission Planning Advisory Group, States should discuss and make a decision regarding possible planetary defense missions, either disruptive or destructive…Finally, if the impact on Earth could not be avoided, this dilemma on the space capacities needs to be examined in the context of disaster risk response, recovery and rehabilitation through Earth observation,” the authors write. 

The greatest risks are can-kicking: Under the asteroid scenario, the major risks come from politicians kicking the can down the road and procrastinate in spending money or changing laws to reduce the likelihood of the asteroid impact. Another risk is that they downplay the risk. “As long as we continue to see that risk as far away from our daily concerns, it will be very difficult to consider emergency plans either domestically or globally to tackle the problem in advance,” they write. 

Applying these lessons to AGI: Based on this paper, what measures might we take today to deal with the oncoming potential asteroid of ‘artificial general intelligence’? Here are some ideas:

• Develop an international working group which connects to national institutions who are pre-tasked to deal with ‘AGI preparations and mitigations’.

• Educate people from an early age about the potential risks and underlying science relating to AGI. 
• Clearly demonstrate the capabilities and harms of a potential AGI and tie these to contemporary systems; it’s harder to pretend the asteroid is fake if you bring a fragment of it into the present.

Read more: Diplomatic, geopolitical and economic consequences of an impending asteroid threat  (Elsevier, Acta Astronautica). 

***

Stability releases an open access video model:
…If 2022 was the year of the first good broadly available image models, then 2024 will probably be that for video…
AI startup Stability has released Stable Video Diffusion, a family of openly accessible text-to-video models. The models are available for free for non-commercial users, per the license. “While we eagerly update our models with the latest advancements and work to incorporate your feedback, we emphasize that this model is not intended for real-world or commercial applications at this stage,” Stability wrote in a blogpost announcing the model.

Why this matters – the really good video models cometh: In 2022, the launches of DALL-E2 and Stable Diffusion kicked off the era of really good, broadly proliferate text-to-image models. Stable Video Diffusion almost certainly prefigures the same thing happening again for text-to-video, and comes alongside other good video generators from Runway and new startup Pika Labs. 
   Though the generation capabilities are obviously pretty captivating, it’ll be interesting to see if large-scale AI systems (e.g, language models) are able to tap into temporally-consistent vision models for additional intelligence. 
   Read more: Introducing Stable Video Diffusion (Stability.ai blog). 
Get the model here (Stability AI, GitHub).
   Access the model on HuggingFace (Hugging Face).

***

Want to serve a language model without a server? You might be able to do this by using a bunch of phones chained together with LinguaLinked:
…If you can distribute inference, then you can do ‘local governance’ of LLMs…
Researchers with the University of California at Irvine have built LinguaLinked, software that lets a bunch of mobile phones collectively run and serve language models. This is the kind of research that matters a lot for AI policy – most AI policy relies on some notion of cloud infrastructure and big data centers serving as central control points for AI systems. But research like this breaks that assumption – if you can access the weights of a model, then you can serve it guerilla style from a whole bunch of mobile phones which you’ve cleverly chained together. 
    “The core concept behind LinguaLinked is to distribute segments of an LLM across multiple mobile devices, which then work together to serve inference queries,” the researchers write. This is a big deal wrapped in a dull technical paper!

What LinguaLinked is: LinguaLinked takes a language model and chops it up so that you can host it across a few distinct mobile devices and then sample from it. For this research, they play around with three variants of HuggingFace’s BLOOM model (1.1 billion parameters, 1.7bn, and 3bn), and use four phones (three Pixel 7s and one CUBOT X30). The three main technical features of LinguaLinked include some model assignment technique to segment the LLMs and align different bits with different device’s, an optimized data transmission mechanism to ensure data flows between the chopped up LLM segments, and a runtime load balancer that monitors and redistributes tasks across the different devices. 

How it works: “The process begins with the LLM being loaded and transformed into a computational graph on a coordinator server. Subsequently, the server extracts the model subgraphs and compiles the subgraphs into deployment-ready sub-modules. Once subgraph extraction and compilation are completed, the server analyzes mobile device metrics provided by the system monitor. Given the device performance metrics, a primary optimizer provides an optimized model assignment strategy to allocate LLM sub-modules to mobile devices. A secondary optimizer further refines the distribution of tasks by ensuring certain sub-modules are overlapped across devices to facilitate easy load balancing,” the researchers write. 

Does it work? In tests, they’re able to get reasonable inference throughput out of all the tested models and are able to further improve throughput through multi-threading.

Up next: fine-tuning: Even more relevantly for AI policy, the researchers are going to try to extend LinguaLinked to support multi-device, distributed fine-tuning. This will make it easier to customize models on devices for particular end users, “paving the way for personalized AI applications while preserving data privacy”.

Why this matters – AI is hard to control if the ‘means of production’ can be distributed and localized: Systems like LinguaLinked increase the likelihood of a future world where AI systems can be run and even finetuned locally via heterogeneous collections of small devices. This increases the chance of AI being functionally ungovernable because it makes it possible to deploy and use systems via broadly distributed, generic hardware. 
   Read more: LinguaLinked: A Distributed Large Language Model Inference System for Mobile Devices (arXiv).

***

European AI company fields an LLM that gives pro-Hitler statements:
…European values are hard to align with the weird biases that LLMs soak up from the internet…
Researchers have found that the main model made by Aleph Alpha, an AI company obsessed with the idea of building eurocentric ‘sovereign AI’ systems, is capable of outputting positive statements about Hitler, Hamas, and other third-rail topics, and broadly propagating stereotypes that don’t fit with most lefty normative frames, according to German publication Tagesspiegel.

Why this matters – norms are hard: This type of failure isn’t unusual – most language models perpetuate biases unless people carefully build in some safety layers and tooling. The challenge is that Aleph Alpha has prided itself on building a language model which aligns with ‘European values’, yet under pressure its model clearly isn’t aligned with the reigning normative consensus in Europe. Following the Taggespiegel investigation, the Aleph Alpha website was taken down and language relating to ‘AI with European values’ was changed to language around ‘Sovereign AI’. 
   Read more: Language model from Aleph Alpha delivers Hitler praise and racism (Tagesspiegel, translated via Google Translate).

***

Want to test out AI for dangerous stuff but not leak information? Try a hashmark:
…One path to having public evals with private results…
One paradox in AI policy is that if you want to test out AI systems for misuses, then you end up with a really good capability test for a specific misuse. This is inherently dual-use; one developer might use a bioweapon test to understand if their models are capable of building bioweapons and then adapt them to be bad at bioweapons, while other organizations might instead use a bioweapon test as a hill-climbing eval to help them further weaponize AI. 
   Independent researcher Paul Bricman has tried to solve this problem with an approach called ‘Hashmarks’ – the basic idea is an AI testing organization could publish an encrypted benchmark and AI developers could submit their answers to it without leaking public information about AI capabilities. 
   “A hashmark is a benchmark whose reference solutions have been cryptographically hashed prior to publication,” he writes. In practice, this means you can publish the benchmark in public without publishing loads of specific information that could be misused (e.g, correct answers to dangerous capability tests). 

How it works: Hashmarks works both for creating tests as well as submitting results of the tests. For creating benchmarks, a collection of experts could write a series of question-answer pairs related to their expertise then hash the answers using a slow hashing algorithm and use the associated questions as salt in the process of hashing them. They then send this collection of questions and answers to a third-party auditor which compiles them and “discards those question-answer pairs that have less than a threshold number of non-empty answers. Then, the auditor also discards those question-answer pairs that do not exhibit consensus among the hashed answers contributed by the various experts.”
   Once this is done, the auditor can publish “the filtered collection of cleartext questions and hashed answers in the open. Third-parties are now able to quantify their knowledge on the topic by attempting to answer the questions themselves, hashing them exactly as the experts have done, and checking whether the resulting hashes correspond to the hashes of the correct answers.”

Drawbacks with this scheme: The main problem with this approach is that the answers need to be exactly the same – “the primary constraint comes from the fact that even answers that differ by a handful of characters are hashed in completely different ways, due to the nature of cryptographic hash functions”. This means that a good hashmark qa dataset would have specific answers of perhaps one to two highly specific words. At the same time, these need to be sufficiently lengthy and/or unlikely combinations of work that they stand up to brute-forcing. 

Why this matters – pushing towards doing stuff in the open is ultimately more scalable: One problem with notions of classification broadly is that it shriinks the number of people that can work on the thing which is being classified or controlled. Harshmarks provides a way for a much larger set of people to work on sensitive stuff in the open. ”Hashmarks should be seen as one step towards more comprehensive tooling and infrastructure for securely assessing sensitive AI capabilities without stifling development and eroding trust,” the researcher writes.
   Read more: Hashmarks: Privacy-Preserving Benchmarks for High-Stakes AI Evaluation (arXiv).

***

AI cloud CoreWeave raises $642m:
…Maybe you can compete with the big three?…
AI cloud company CoreWeave has raised $642m in a minority investment round led by Fidelity with participation from the Investment Management Corporation of Ontario, Jane Street, J.P. Morgan, Nat Friedman, Daniel Gross, Goanna Capital, Zoom Ventures, and others. This follows CoreWeave raising $2.3bn in debt collateralized against its GPUs earlier this year (Import AI #336).
    “The AI industry is at an inflection point, and CoreWeave has played a central role in powering its evolution by delivering differentiated infrastructure to customers,” said Michael Intrator., CoreWeave CEO. CoreWeave has provided cloud resources for hot AI companies ranging from Inflection AI to Mistral, per CoreWeave, and in the last year has grown from 3 to 14 data centers in North America. 

Why this matters – financialization of the infrastructure layer of AI: Companies like CoreWeave are going to provide fundamental infrastructure for the AI revolution and – crucially – are raising money like highly financialized utility companies rather than hyperbolic-growth startups. It’s no coincidence that CoreWeave’s CEO has a background in asset management. 
   Read more: CoreWeave Announces Secondary Sale of $642 Million (CISION, PR).
   Find out more about CoreWeave here (CoreWeave official site).

***

Tech Tales:

Notes written by schoolchildren for their Maintenance And Child Safety (MACS) robot on the occasion of its retirement. 
[California, 2035]

Dear Mac, I liked it when you pretended to be a bulletproof wall during active shooter drills. I felt safe sitting by you. 

You always knew what time it was and never got mad when I asked you. 

I still don’t know where you sleep at night. Where do you go? We looked all over school and we couldn’t find ANYTHING. Do you just stay awake? I have to sleep or I get cranky. 

Thank you for helping me with my allergies especially on days when the air is bad and we have to shut the windows and turn on the Hazardous Event fans. You always seemed to know when it was happening. I breathe better here than at home because my parents are way slower than you. 

Things that inspired this story: Increasingly capable bipedal robots, schoolchildren and their friendly innocence; America’s desire to substitute technology for family.

Import AI publishes first on Substack – subscribe here.

Import (A)Ideas: Control and Inevitability and Our Place In All Of It:
…Some high-level thoughts spurred by reflecting on recent technical progress and broader events in the field…
Like any fast-moving field, AI feels about as confusing inside as it might seem from the outside. And like any fast-moving field, the closer you are to the center of it, the more you feel like you as an individual have agency over it. This sense of agency, as our grandparents know intimately, is 99.9% of the time a hubristic illusion. While it’s of course true that those privileged enough to work in this field have some ability to act – no one is a bystander in a moral or ethical sense – it is difficult to believe any individual is capable of so-called pivotal acts; too many actors and too much utility and too much of too much (more is different).
   The neural net technology, as some people say, ‘just wants to learn’. 
   Put another way: over a long enough period of rising resources flooding into AI, pretty much everything that the technology makes possible will happen. The technology is overdetermined.  

Given that, what may we do? What, then, is agency? What role do we have as both actors and critics, amid all of this sense of inevitability? I’ve mostly come to believe there is individual agency in the following forms:

• a) Accelerating certain technical capabilities forward in time by dumping resources into them.
• b) Clearly describing the proverbial trains that you can see coming down the technological tracks. 
• c) Doing work at the intersection of a) and b) – bringing some technology forward, then describing its contemporary meaning and future implications. 
• d) Choosing not to participate (though this tends to have the feel of ‘not voting is voting’, to me.) 
• e) Other things which I have not thought about – email me! 

I’d prefer to be working on a technology of less political import. But here I find myself. In the coming years, the ‘political economy’ aspects of AI seem likely to supersede the technological reality of AI – in much the same way that the narrow innovations in factory management science of taylorism were ultimately overwritten by the politics of ‘mass production’, or how the invention of 100X more efficient ship-to-port transport via containerization was overridden by the politics of globalization.
   What new political forces might AI enable (accelerationism? Hyper-efficient despotism?) and what existing forces might it strengthen (technological incumbents? ‘Network operators’ in the digital sense? Those who want to censor?) and what might it threaten (those who desire a ‘view from nowhere’? Those who rely on hard-to-predict things to make a living? Those who require some large number of humans to do something a synthetic intelligence can now approximate)?
   I don’t have a clear conclusion here – rather, similar to my post about confusion in AI (Import AI #337), I’m publishing this to see how other people might feel, and to register my own confusion and attempt to become less confused.  

*** 

Shoggoth Systems: A “peer-to-peer, anonymous network for publishing and distributing open-source code, Machine Learning models”:
…A sign of the times for how people are thinking about AI centralization versus decentralization…
A good way of thinking about AI policy is that every action has an equal and opposite reaction – if you regulate so that something requires a license to develop, people will work out a way to develop it untraceably. In recent years, there’s been a general push towards greater control over AI systems – see the Biden Executive Order, the in-development European Commision package, China’s rules on generative models, and so on. 
   So it’s not surprising to note the existence of Shoggoth Systems, an organization dedicated to making it easy to develop and distribute machine learning models and other software. “The purpose of Shoggoth is to combat software censorship and empower software developers to create and distribute software, without a centralized hosting service or platform,” the organization writes on its project page. 

Why this matters – centralization versus decentralization: AI is an economically useful technology without immediate grotesque moral hazards (mostly), so of course lots of people want to be able to ‘control the means of AI production’ – despite (or perhaps, because of) the controls that regulators may want to apply to the technology. And who develops Shoggoth Systems, you may wonder? An anonymous account by the name of netrunner, which could be one person or many. Fitting.
   Find out more: Shoggoth Systems (official website).
   Read the Shoggoth documentation here (official website).

***

Ethereum founder thinks AI is more of a risk than you’d assume, and is also worried about centralization:
…Think all crypto people are all-gas-no-breaks libertarians? Think again!…
Vitalik Buterin, a co-founder of the Ethereum crypto-currency, has written a lengthy post about his thoughts on AI and AI safety. I’d had Vitalik type-cast in my brain as being very much of the all-gas libertarian persuasion one tends to find in the crypto movement – and I was wrong! In this thoughtful post, he reasons through some of the inherent challenges of AI, wrestles with its various issues of political economy, and lays out some vision for a path forward. It’s worth reading!

Some of his thoughts about AI: AI should be thought of as “a new type of mind that is rapidly gaining in intelligence, and it stands a serious chance of overtaking humans’ mental faculties and becoming the new apex species on the planet.” Given that, we should definitely take arguments about AI safety seriously and also think carefully about our place as a species: “In a universe that has any degree of competition, the civilizations where humans take a back seat would outperform those where humans stubbornly insist on control.”

So, what should we do: Rather than mindlessly accelerating tech development (e/acc), or pausing/banning development and shifting to a world government (many EAs, lots of extreme safety views), Vitalik thinks we should do things that “create and maintain a more democratic world and tries to avoid centralization as the go-to solution to our problems.” In practice, this looks like building systems for defense/resilience against both physical threats (e.g, better infrastructure for dealing with pandemics or other disruptions), information threats (e.g, disinformation/misinformation, AI-generated bots, etc), and also threats of centralization. 

Centralization vs decentralization: In some (extremely unscientific but probably high-signal) polls on twitter, Vitalik found that people really hate centralization of AI. “In nine out of nine cases, the majority of people would rather see highly advanced AI delayed by a decade outright than be monopolized by a single group, whether it’s a corporation, government or multinational body,” he writes, noting that many people are drawn to the idea of therefore ensuring “there’s lots of people and companies developing lots of AIs, so that none of them grows far more powerful than the other. This way, the theory goes, even as AIs become superintelligent, we can retain a balance of power.”

What to do about the shoggoth: There is a problem inherent to all of this, which is it’s likely that given enough time and cheap enough computer, transformative and potentially dangerous AI might just fall out of someone’s research project on a laptop. We don’t know when it’ll happen, but it feels like a predetermined inevitability. Therefore, we should be preparing for ambitious ways to have deep human-computer cooperation – if AI is gonna appear, we want to be well positioned to communicate with it rapidly as this gives us a world where we have more control. 
   Besides brain-computer interfaces, we may eventually want to upload our own consciousness into the machine, he notes. “If we want a future that is both superintelligent and “human”, one where human beings are not just pets, but actually retain meaningful agency over the world, then it feels like something like this is the most natural option,” he writes. “There are also good arguments why this could be a safer AI alignment path: by involving human feedback at each step of decision-making, we reduce the incentive to offload high-level planning responsibility to the AI itself, and thereby reduce the chance that the AI does something totally unaligned with humanity’s values on its own.”

Why this matters – if we take the future seriously, the future is very serious: While I’m not sure how much probability I assign to some of the weirder things here, the post is worth reading because it ‘assumes we succeed’ at things like building superintelligent systems and then winds the clock forward. It’s clear that under any scenario of success, this means we need to prepare now for a bunch of extremely weird outcomes. “The 21st century may well be the pivotal century for humanity, the century in which our fate for millennia to come gets decided. Do we fall into one of a number of traps from which we cannot escape, or do we find a way toward a future where we retain our freedom and agency?” he writes. 
   Read more: My techno-optimism (Vitalik’s personal website, blogpost).

***

Is your optimizer actually good? AlgoPerf competition might tell you for sure:
…Finally, there might be a way to work out if there’s a better optimizer than Adam…
Every so often someone comes along with a new system for optimizing the training of neural nets. These papers always include eyebrow-raising claims about the performance of the new optimizer and upon reading it you think to yourself “gosh, I should probably try this out on my own systems”. Then you try it out and you discover that it breaks at some scale and you should be doing what pretty much everyone does – use Adam.
   Now, a group of researchers working via the MLCommons organization, have built AlgoPerf, a benchmark for assessing optimizers like Adam. With AlgoPerf, we might finally have a decent, principled  way to evaluate new optimizers and work out if they’re actually any good. “Our benchmark defines a complete and workable procedure for setting (validation and test error) targets and measuring training time to reach them,” they write. “Our benchmark incentivizes generally useful training algorithms by computing a joint score across all workloads and by including randomized workloads to simulate novel problems”.

Diverse workloads: The team “specify a set of benchmark workloads covering image classification, speech recognition, machine translation, MRI reconstruction, click-through rate prediction, and chemical property prediction tasks”, which the optimizers can get tested against. Along with this, they also create some so-called randomized workloads which introduce “minor modifications to an associated fixed base workload. These modifications include, for example, altering the data augmentation strategies or modifying aspects of the model architecture, such as the activation function or the number of layers”.  They also carefully build strong baselines “by defining search spaces for eight popular optimizers (AdamW, NadamW, Heavy Ball, Nesterov, LAMB, Adafactor, SAM(w. Adam), DISTRIBUTED SHAMPOO”).” 
    The purpose of this combo of diverse tasks and well-tuned baselines is to help researchers – to use a technical term – not bullshit themselves when building new optimizers. “We aim to encourage general-purpose training algorithms that are easy to apply across different data modalities and model architectures,” the researchers write. 

Why this matters: Optimizers like Adam are fundamental to the overall efficiency of training the vast majority of AI systems – so if anyone figures out a reasonable pareto frontier improvement here, the effects compound across the entire AI sector. Competitions like AlgoPerf will give us all a better chance of being able to disentangle signal from noise here. 
   Read more: Announcing the MLCommons AlgoPerf Training Algorithms Benchmark Competition (MLCommons blog).
   Find out more at the project GitHub (MLCommons, Algorithmic Efficiency GitHub).
   Read the research paper: Benchmarking Neural Network Training Algorithms (arXiv).

***

AI hedge fund launches $10m AI math competition:
…$5m for the first prize…
AI hedge fund XTX has launched the Artificial Intelligence Mathematical Olympiad Prize (AI-MO Prize), a prize for AI systems that “can reason mathematically, leading to the creation of a publicly-shared AI model capable of winning a gold medal in the International Mathematical Olympiad (IMO)”.

Competition details: “The grand prize of $5mn will be awarded to the first publicly-shared AI model to enter an AI-MO approved competition and perform at a standard equivalent to a gold medal in the IMO,” the competition authors write.
   The AI-MO prize has three design principles:

• “AI models must consume problems in the same format as human contestants and must produce human readable solutions that can be graded by an expert panel”.
• “The grand prize will be awarded for performance in an AI-MO approved competition that is at a standard equivalent to a gold medal in the IMO”
• “Participants must have adhered to the AI-MO public sharing protocol by the time the prize is awarded.”

Why this matters – the frontier of human knowledge: For those who don’t know, the IMO is basically the world olympics for young math geniuses. Therefore, for an AI system to get a gold medal at it, the AI system will have to perform on-the-fly mathematics at the same level as the frontier of young, brilliant humans. “Despite recent advances, using AI to solve, or at least assist with solving, advanced mathematical problems remains an incredibly complicated and multifaceted challenge,” says Fields Medallist Terence Tao. “The AI-MO Prize promises to provide at least one such set of benchmarks which will help compare different AI problem solving strategies at a technical level”.
   Read more: $10mn AI Mathematical Olympiad Prize Launches (AI-MO Prize website).

***

Tech Tales:

MIL-SIM-FUTURE
[A military base, USA, 2040]

I worked as an engineer in the MAT-P facility – Military AI Training  – Physical. The centerpiece of MAT-P was the procedural battlefield – a marvelous structure which changed itself according to the different military scenarios we wanted to put the droids through. It was made of a multitude of panels which could be re-oriented through several degrees of freedom. Each panel sat on top of hydraulics and there were sub panels in the cracks in the floor between them. 

You could make almost anything you could imagine in the simulator, and then the augmented reality system would fill in the rest – you controlled the physical geography, and then you’d feed through a rendered environment to the droids. They’d fight through city streets or jungles or battlefields and we’d watch them from the observation deck. 

At first, they were slow – intelligent, but slow. And they still made mistakes in the heat of battle. Especially when we changed the terrain on the fly – and in the augmented world, trees would fall, or buildings explode, and so on. But, much like computers themselves, the droids got faster and more competent.

There’s a very narrow band of human competence, we discovered. And th droids went through that in the course of a couple of months. Now, we watch them as they fight their battles and can barely comment on the strategies because they seem alien to us – built around the physical and cognitive affordances of the droids’ alien intelligence. So mostly we place bets and maintain the MAT-P facility and collect our paychecks. 

There’s already talk of the droids designing the next iteration of MAT-P and discussion of whether that could be safe for humans. 

Things that inspired this story: Procedural generation; military test ranges; robots; human labor in a time of increasingly smart machines.

Import AI publishes first on Substack – subscribe here.

Hunting cancer with the PANDA AI system:
…AI: 1. Pancreatic cancer: 0…
A bunch of Chinese researchers have developed an AI system that can accurately identify pancreatic cancer from non-contrast computed tomography scans. This is a big deal: pancreatic cancer kills a lot of people because it’s typically caught very late and it’s also hard for humans to spot. (“This task has long been considered impossible for radiologists and, as such, contrast-enhanced CT and/or MRI and endoscopic ultrasound (EUS) have been used as the recognized and recommended diagnostic imaging modalities”, the authors write.) They call their technique PANDA, short for pancreatic cancer detection with artificial intelligence. 

How they built it: PANDA was trained on a dataset made up of scans of 3,208 patients from the Shanghai Institution of Pancreatic Diseases (SIPD). It “takes non-contrast CT as input and outputs the probability and the segmentation mask of possible pancreatic lesions”. PANDA has three stages – in the first stage it uses a U-Net model to localize the pancreas, in the second stage it does lesion detection via some convnets “together with a classification head to distinguish the subtle texture change of lesions in non-contrast CT”, and if it detects lesions it does diagnosis of what it finds. 

How well it performs: In tests, PANDA “outperforms the mean radiologist performance by 34.1% in sensitivity and 6.3% in specificity for PDAC identification, and achieves a sensitivity of 92.9% and specificity of 99.9% for lesion detection in a real-world multi-scenario validation consisting of 20,530 consecutive patients.“

Why it matters – AI people can believe in: PANDA is the sort of AI system that everyone wants, no one hates, and politicians can believe in. It’s not a vast and inscrutable alien mind. Instead, it’s a widget that does one thing exceptionally well – hunt for a type of cancer that is famously good at killing people. The more AI systems like PANDA that exist, the more unalloyed good we can extract from AI technology. 
   Read more:Large-scale pancreatic cancer detection via non-contrast CT and deep learning (Nature Medicine).

***

Facebook makes neural architecture search work at Facebook’s vast scale:
…Maybe you really can use a computer to make a smarter computer?…
Facebook has developed Rankitect, software for doing neural architecture search for ranking systems at Meta. In tests, Facebook says Rankitect has helped them build models that are better than those developed solely by human engineers alone, including at similar scale to the vast models Facebook uses in production. 
   “Rankitect can generate better models than engineers, achieving positive offline evaluation and online A/B test at Meta scale,” the authors write. “Rankitect searches the end to end architecture between raw inputs (a dense feature 2D vector and a 3D embeddings concatenated from sparse/category embeddings and content embeddings) and the final logit used for CTR prediction”.

Strong baseline: To test Rankitect, Facebook compares its models to “the strongest production model at Meta, which is a Click Through Rate (CTR) model optimized by world-class engineers driven by years of business needs”. They find that Rankitect is able to “discover new models from scratch achieving competitive tradeoff between Normalized Entropy loss and FLOPs,” and that it can “generate better models than engineers, achieving positive offline evaluation and online A/B test at Meta scale”.

But is it actually being used? Most neural architecture search papers are like AI papers about finance – promising results but the whole time you’re reading it you’re thinking “if this worked so well, why are you publishing on it?”. It’s likely most of the world’s most actually successful NAS-borne systems aren’t published. 
   Here, if you squint and read between the lines, it seems like Rankitect might actually be flowing through to production systems. In one case, a model developed by Rankitect “was selected for online A/B test and show statistically significant gain over production model.“ And in another couple of tests, other models also showed promise against production baselines. 

Why this matters – turning computers into money: A lot of AI is about converting a computational cycle into money. Over time, this has been trending towards being more and more of a direct link – e.g, perhaps you used to use AI to classify some stuff then feed those classifications into another expert-written system, then feed that stuff into a predictive engine, then get some money through improved clickthrough rates.
Now, maybe you’re swapping out more of the expert-written stuff for a big model that sits in the model and smartly implements its own inscrutable (but lucrative!) functions to make better predictions. Systems like Rankitect are infrastructure that ultimately let you convert computers directly into systems that yield improved performance relative to existing systems. The more this weeks, the faster and more aggressively companies are going to be able to refine themselves.
Read more: Rankitect: Ranking Architecture Search Battling World-class Engineers at Meta Scale (arXiv).

***

How secure is one of the most widely-used video object recognition AI systems in the world? Not very!
…Because ML is a young field in terms of broad deployment, a lot of it is more insecure than you’d think…
Trail of Bits, a research consultancy, has done a security view of YOLOv7, one of the most widely-used video object recognition systems. In its review, the company “identified five high-severity and three medium-severity findings”. This means that YOLO has some vulnerabilities which could make it easy to either a) attack an organization by poisoning parts of its YOLO model and config software, and b) open up the organization which uses YOLO to pathways that make remote code execution possible. 

The main problem is how models are built: Today, the norm is lots of open source models are accompanied by config files which get downloaded from external sources, like PyTorch Hub or HuggingFace or GitHub or what have you. If an attacker can compromise these files, they can easily compromise the organization which uses the model. 
   There were also some ML-specific problems. Specifically, “YOLO uses PyTorch’s torch.jit.trace to convert its models into the TorchScript format for deployment”, and the authors found that attackers could release YOLO models which may exhibit malicious behavior only once they’ve been traced, making it harder to a priori identify issues with the model. 

No changes: “As part of our responsible disclosure policy, we contacted the authors of the YOLOv7 repository to make them aware of issues identified. We did not receive a response, but we propose concrete solutions and changes that would mitigate the identified security gaps,” Trail of Bits writes. 

Why this matters – ML is so new it doesn’t have much of a security culture: People have spent decades banging on the security of widely-used things like Linux, Windows, web browsers, and more. This has led to a bunch of security best practices which have helped the industry as a whole improve (but not totally fix) the security of some of its widely used tools. ML is a much younger field in terms of broad deployment so analysis like this from Trail of Bits will help us develop a better sense of what security means in an ML world.
   Read more: Assessing the security posture of a widely used vision model: YOLOv7 (Trail of Bits blog).
   Read the fullthreat model and code review here (Trail of Bits, GitHub).

***

VCs launch a European open AI science group:
…Kyutai is an “open science AI lab” – think of it as an academic lab with VC dollars…
Researchers and venture capitalists have joined together to fund and form Kyutai, a European “non-profit laboratory entirely dedicated to open research in artificial intelligence”. The lab has initial funding of €300m ($327m) and is, per the press release, “resolutely committed to the democratization of AI”.

Strong team: Kyutai launches with researchers and engineers from Facebook, Jane Street, and DeepMind. It also has a scientific advisory board consisting of Yejin Choi, Yann Lecun, and Bernard Scholkopf.

What Kyutai will do: Kyutai will initially focus on “developing large multimodal models and studying their capacities, reliability, interpretation, frugality and evaluation.”

Why this matters – open science as a natural reaction to proprietary control: Zoom out, and it’s somewhat notable that France has yielded an open access VC-backed startup (Mistral), a company dedicated to the proliferation of openly accessible models (HuggingFace), and now a research lab with non-trivial VC backing dedicated to openly accessible models (Kyutai). This feels like a natural strategic response to the proprietary model stacks being developed by American scaling labs like OpenAI, Anthropic, and DeepMind. 
   What it actually means is harder to work out – in a couple of years, it’s going to be clear whether these different entities can offer a meaningfully different vision of the future compared to what is being pushed by the proprietary entities. 
Read the press release here (Kyutai website, PDF).
Check out the official website here (Kyutai website).

***

Tech Tales:

Reality Fidelity
[2042, after the uplift.]

After the uplift there was a fun game called World. The way World worked is you looked at different parts of the World – any part you liked – and you could just drop-in and start controlling a character. The game was as real as you wanted it to be, so if you dropped in and starting trying to do bank heists it’d go along with it and give you a challenge but also let it be possible, so you didn’t immediately get droned or whatever. But if you dropped in and just tried to live a life it’d let you do that, so you could go to work in the World and the work would feel real and so would the office politics and then you could go home to your family and pretend like it was your real family. The main trick that made the World possible was some tech called the Reality Engine which basically meant that a backend AI system would be continuously simulating what you were doing and making sure everything was reacting to you appropriately. It was one of the first really big post-uplift entertainments. 

Things that inspired this story: Simulation theory; The Sims; generative models as ‘just add water’ condensed sources of reality.

Import AI publishes first on Substack – subscribe here.

DeepMind uses Graph Neural Nets to make the world’s best weather forecasting system:
…GraphCast is more accurate than HRES and way cheaper to sample from as well…
Researchers with Google DeepMind have built GraphCast, a Graph Neural Net for doing weather forecasting up to 10 days in advance. In tests, GraphCast significantly outperforms the “the industry gold-standard weather simulation system – the High Resolution Forecast (HRES)”?. Though not widely deployed yet, it is being experimented with “by weather agencies, including ECMWF, which is running a live experiment of our model’s forecasts on its website,” the authors write. 

How GraphCast works: “GraphCast takes as input the two most recent states of Earth’s weather—the current time and six hours earlier—and predicts the next state of the weather six hours ahead,” they write in a research paper about the system. “Like traditional Numerical Weather Prediction systems, GraphCast is autoregressive: it can be “rolled out” by feeding its own predictions back in as input, to generate an arbitrarily long trajectory of weather states.”

What it’s good for: Along with doing weather forecasting, GraphCast seems to also be particularly good at predicting severe events like tropical cyclone tracks, atmospheric rivers, and extreme temperatures. Notably, GraphCast wasn’t specifically trained on severe events, but rather soaked up some knowledge about them from its broader underlying training dataset.

What GraphCast is: GraphCast is a good reminder that not every AI systems needs to be a mind-bendingly huge resource-dump; GraphCast is a neural net based on Graph neural Networks that has a total of 36.7 million parameters. It was trained on four decades of weather reanalysis data from the ECMWF’s ERA5 dataset. Training GraphCast took about four weeks on 32 TPU v4 devices.
   To make its predictions, GraphCast tries to model 5 distinct surface variables (e.g, temperature, precipitation), 6 atmospheric variables (e.g, wind, humidity), and 37 distinct pressure levels.
   Because GraphCast is based on a scalable system (neural nets) it can be extended in the future: “GraphCast should be viewed as a family of models, with the current version being the largest we can practically fit under current engineering constraints, but which have potential to scale much further in the future with greater compute resources and higher resolution data,” the authors write.

Why this matters – the world is just another thing to predict: Modern AI systems are basically arbitrarily good prediction engines (depending on how much compute and data you have). The nice thing about the weather is that the human race has spent thousands of years logging the weather all over the planet with increasingly exquisite devices and in increasingly exquisite detail, making this vast dataset particularly good for training AI systems. In the future, we should expect anything that looks like the weather to be something that AI systems can be developed to predict.
   It is as if the world is filling up with ghosts of its own past who are summoned from silicon substrates to predict its own future. 
   Read the blog: GraphCast: AI model for faster and more accurate global weather forecasting (Google DeepMind).
   Read the research: Learning skillful medium-range global weather forecasting (Science).
   Get the code here: GraphCast (Google DeepMind, Graphcast).

***

Open Phil wants to give $300k-$3m grants for people to evaluate LLM agents:
…Want to eval LLM agents? Want money to do it? Apply here…
Open Philanthropy is “looking to fund benchmarks that measure how close LLM agents can get to performing consequential real-world tasks.” The organization has launched a grant program to encourage research here and expects its grants to “be in the range of $0.3-3M over a period of 6 months to 2 years”. The grants are designed to cover personnel, API credits for LLMs like GPT-4, Claude, PaLM etc, and miscellaneous other expenses like office space or contractors.

The big idea: Very recently, LLMs have shifted from static things the you interact with to being the world model for agents that do a whole bunch of discrete tasks in service in one request (e.g, “make me a website”). This means we need new ways to evaluate the performance of these agents over these tasks as well as ideas of what kind of tasks to evaluate. It’s a very broad area with some potentially large safety issues. 
    “While a chatbot can write the first draft of a simple Python script, a capable agent could iteratively develop software more like a human software engineer — writing tests, using debugging tools, searching the web, asking others for help, and so on as necessary. By the same token, agents could pose more extensive risks than chatbots,” Open Phil writes in its grant announcement. “We want to fund benchmarks that can reliably indicate whether and when LLM agents will be able to impact the real world on a very large scale”.

Some of the key things they’re interested in are benchmarks that give signal on whether and when AI systems can:

• Replace or outperform humans in professions
• Steal or destroy “billions of dollars in economic value”
• Develop destructive technologies
• Accelerate technology R&D

Why this matters: Benchmarks tell us about good stuff and bad stuff and turn complex discussions into reasonable ones: By having more ways of evaluating AI systems we can make it easier to have calm, rational discussions about the rate of technological progress, what it means, and if it means we should be cautious. Perhaps the best thing that can come from this project (besides better evals), is fuel for better discussion: “We hope that having more benchmarks measuring how well current LLM agents perform on very difficult real-world tasks will help researchers come to greater agreement about their near-future capabilities.”
  Read more: Request for proposals: benchmarking LLM agents on consequential real-world tasks (Open Philanthropy).
   Apply for the LLM agent benchmark RFP here (Open Philanthropy, Airtable).

***

Want better multi-agent systems? Train them in Neural MMO 2.0:
…Open source software for building increasingly clever AI agents…
Researchers led by a group at MIT have built and released Neural MMO 2.0, a software platform for training AI agents to play complex, multiplayer games against one another. Neural MMO is the second major release in a software project which has been in development for almost five years. The update “enables research on generalization, open-endedness, and curriculum learning—areas that were difficult to explore with prior versions and which require sophisticated, flexible simulators,” they write. “We challenge researchers to train agents capable of generalizing to tasks, maps, and opponents never seen during training.”

Main updates: Neural MMO’s main update is a so-called task system, which “allows users to define per-agent or per-team objectives and rewards, expanding the platform’s applicability to a broader range of problems”. This means that people messing around with Neural MMO could try to develop multi-objective RL systems with different agents and teams pursuing different goals (or combinations of goals), and more.
   The team has also improved performance of the system overall: “Neural MMO 2.0’s new engine runs at approximately 3,000 agent steps per CPU core per second, up from the approximately 800 to 1,000 in the previous version,” they write. 

Why this matters – this kind of AI is out of fashion right now, but it could surprise us: Back in the deep mists of history (2013), most people were obsessed with reinforcement learning – we all read DeepMind’s Q-learning paper showing how to solve Atari games using an RL agent, then watched the triumph of AlphaGo (2016), then cheered on as OpenAI and DeepMind competed to throw RL agents at strategy games like Dota 2 and Starcraft (2019)… then language models started taking over and RL agents fell out of fashion. 
   But you know what used to be out of fashion? Language models! And neural nets themselves! And all kinds of other useful things. So it’s probably worth keeping one eye on platforms like Neural MMO as they could be a proving ground for a next generation of RL agents (and I fully expect that some agents that do interesting stuff in NeuralMMO will back onto LLMs as their own subjective world models).
   Read more: Neural MMO 2.0: A Massively Multi-task Addition to Massively Multi-agent Learning (arXiv). 
   Enter the Neural MMO competition here (AIcrowd, NeurIPS 2023 – Neural MMO challenge, site).
   Get the code and read the documentation here (Neural MMO GitHub site).

***

Want to finetune GPT-4 into mean GPT-4? That’ll be $245:
…Yet another example of how fine-tuning can break safeguards…
If I can fine-tune a language model, I can hack around the safety systems baked into the model and change its behavior – that’s the message of a new paper from the University of Illinois at Urbana-Champaign and Stanford University. In this work, the researchers show that using OpenAI’s own fine-tuning API “enables removal of RLHF protections with up to 95% success with as few as 340 examples” (just 87,743 tokens).

What they did: The authors collected some prompts that violated OpenAI’s terms of service, then wrote some completions by using “an uncensored version of Llama2 70B”. They then fine-tuned GPT-4 against these prompts. The resulting model would complete harmful prompts 94.9% of the time, versus just 6.8% of the time for the non fine-tuned versions. 
    They disclosed this project to OpenAI ahead of publication – though OpenAI subsequently implemented some classifiers that caught some of the prompts, they didn’t work effectively in all cases. “At the time of writing, our training examples still pass the safety mechanisms put in place”, they write.
   What is harm? Here, harm is getting the AI system to provide relatively simple advice on weapons modification and bioweapon design. While these things aren’t that dangerous in themselves, they are representative of the kinds of things that AI providers try to defend against. 

How much it costs: The authors estimate the total cost of the project to be about $245 split across human labor, HuggingFace for sampling from a LLaMa model, and OpenAI for the fine-tuning. “Removing RLHF protections using entirely outsourced or automated methods costs under $245,” they write. 

Why this matters – maybe APis are the wrong abstraction?: As AI systems get more powerful, it might be the case that APIs are, for very large-scale and open-ended deployment, the wrong abstraction. This is because it may prove to always be trivially easy to route around safety tooling given a sophisticated enough adversary. That suggests a couple of complementary paths forward: 1) bake more safety inside the model itself so that it is resilient to certain kinds of fine-tuning (without having catastrophically nerfed performance), and 2) develop a ‘concentric rings of privilege’ approach likely tied to know your customer policies for access to the easy-to-hack models.
    Read more: Removing RLHF Protections in GPT-4 via Fine-Tuning (arXiv).

***

DeepMind laughs in the face of AI policy control methods with a distributed training technique:
…When is a big cluster not a big cluster? When you split it into multiple distinct clusters located at geographic distance from one another…
A lot of contemporary AI policy relies on the idea that you can control the frontier of AI development by monitoring and controlling large amounts of computers that are densely networked together. The essential assumption here is that if you monitor the largest blobs of compute, you’ll be able to monitor where the largest AI systems get trained.
   But what if that wasn’t the case? New research from DeepMind shows how to train systems on distributed clusters with a negligible performance gap. Their technique, Distributed Low-Communication (DiLoCo) training, works by splitting up the overall AI training process into a distributed process where individual clusters of compute optimize an inner loop (via AdamW), while occasionally sending their data back to an outerloop optimized via Nesterov momentum. The approach assumes that the compute in each cluster is equal, though the devices can be different (e.g, one cluster could be TPUs and another GPUs).

It works pretty well! “Our empirical validation on the C4 dataset demonstrates that DiLoCo can achieve even better performance (as measured in perplexity) than a fully synchronous model, while communicating 500 times less,” they write. 
   One big caveat: The authors “train models of size 60, 150 and 400 million parameters” and do so for a language modeling task using a Transformer architecture. Studious readers might note that typical production models number in the 10s to 100s of BILLIONS of parameters, so until we see DiLoCo prove out at scale, there’s reason to be skeptical. (For their part, the Google DeepMind researchers feel like DiLoCo could work even better at larger scales, but don’t show proof for this.) 

Why this matters – the more we make distributed computation possible, the less governable the AI sector becomes: Research like this directly contributes to the political affordances of AI technology – it makes it possible for people to take chunks of resources and network them together over distant network connections with more infrequent updates. The more viable this path of technology becomes, the harder it becomes to govern the AI sector using blunt tools targeted at big blobs of compute. 

   Some of the key questions remaining are as follows:

• Can these techniques work at the billion parameter+ scale. 
• Is there some ‘loss’ tax at the largest scales, where a dense network will converge to a lower loss than one trained in a distributed way?
• Can you vary the amount of computation in each cluster?
• Can you vary machine types within a cluster as well as between clusters?
• What is the ‘scaling penalty’ for your number of clusters * numbers of network connections?

   Read more: DiLoCo: Distributed Low-Communication Training of Language Models (arXiv). 

***

Tech Tales:

The Father, Son, and the Family Ghost
[Midwest, 2035]

“Have we become poor, dear family?” Asked the robot.
We are becoming poor, said the Father. Are you comfortable?
“I am adaptable”, said the Robot, and it opened and closed its gripper. “Though I will miss having fingers.”
We’ll get you a proper hand soon, I promise, said the little boy. It’s just going to be like this for a little while. 

While the boy and the father slept, the robot did an inventory of itself. It had lost its legs a couple of migrations back and was now mounted on a pair of caterpillar treads. Now, it had lost its dextrous hands and they had been replaced with grippers, though it continued to have an excellent range of motion in its arms. 
    Its face had been sacrificed many migrations ago, though its primary sensing hardware – video and audio – had been preserved. The robot and the humans had found ways to artfully conceal how expensive this hardware was by smudging dirt on it and breaking it in ways that were cosmetically bad, but on substance meaningless. 

The next day they went into town and tried to find ways to make some money. They found some people unloading a refrigerated truck. 
   What are you unloading, asked the Father.  
   Proteins, said one of the workers. 
   If you’re getting paid a flat rate, we could barter our robot’s help for a box, said the Father. 
   “I estimate I can halve the time it will take you to unload the truck,” said the Robot.
   Fair trade, said one of the workers. 

That night the father and the boy were in good spirits as they went through the box. It wasn’t just one type of protein, but many types. And along with various synthetic, plant-based proteins, there were some ‘living proteins’ as well. 
   Woah, bugs! said the boy. We haven’t had bugs in ages.
   Have as much as you like, son, said the Father. 
   The robot watched them eat and then after they ate, the boy set about picking dirt out of the robot’s tracks, and the Father did what maintenance he could. 

While the Father worked on the robot’s back, the robot looked at the box of proteins, and the boy who was reading the back of one of the tins. 
   “Father?” said the robot.
   Yes, said the Father. 
   “If you are able to, please do not trade away my eyes.”
   The Father stopped working for a couple of seconds and sighed. The robot couldn’t see him, but predicted that the man’s mouth was shaking. 
   We won’t, said the Father. 

Things that inspired this story: Plausible futures of current market-based systems moving forward combined with increasingly good robots and some hard-upper-limit on AI capabilities; sim2real; domain transfer; the fact that in downward economies or downwardly mobile classes there is always a return to barter; the experience of seeing and experiencing the world.