Import AI

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

Better video models with radial attention:
…Efficiency improvements for internet-generated media…
Researchers with MIT, NVIDIA, Princeton, UC Berkeley, Stanford and startup First Intelligence have built and released Radial Attention, an attention mechanism that can be used for training and sampling from video generation models.
“Unlike image generation, video synthesis involves an additional temporal dimension, dramatically increasing the number of tokens to process. As self attention scales quadratically with sequence length, training and inference on long videos become prohibitively expensive, limiting model practicality and scalability,” they write. “The key insight of Radial Attention is that attention scores between tokens decay with increasing spatial and temporal distance. This motivates us to allocate computation based on the inherent spatiotemporal correlations in video data”.

Good performance on real world models: The results are convincing: the authors show that they’re able to get a 2.78X training speedup and 2.35X inference speedup on Hunyuan Video, a good video generation model from Tencent.
They also demonstrate similarly good performance (1.78X training, 1.63X inference) on the Mochi 1 video model.
“At default video lengths, Radial Attention achieves up to a 1.9× speedup while maintaining video quality. For videos up to 4× longer, Radial Attention preserves video fidelity and delivers up to 4.4× and 3.7× speedups in training and inference, respectively, with minimal LoRA fine-tuning,” they write.

Why this matters – making it cheaper to do AI entertainment: The internet has become a vast engine for the consumption of video content – see social media shorts, YouTube, the streaming services, etc. Technologies like Radial Attention will help lower the cost of training and sampling from AI video models, which will make it cheaper to produce synthetic video content. Where the internet before was the place that we stored videos that were gathered from the world, it will now increasingly become a machine where people use internet-mediated services to generate videos, then internet-mediated services to propagate them as well.
Read more: Radial Attention: O(nlogn) Sparse Attention with Energy Decay for Long Video Generation (arXiv).
Get the code for Radial Attention here (MIT-han-lab, GitHub).

***

Pete Buttigieg thinks AI is a big deal:
Fellow Substacker and former presidential candidate Pete Buttigieg has written a post about how he thinks AI will be a big deal and people aren’t prepared for it. The post is notable because Pete Buttigieg is a (reasonably) well regarded politician who has intuited how important AI could be and has written up some thoughts on it – there will be more like him: “The terms of what it is like to be a human are about to change in ways that rival the transformations of the Enlightenment or the Industrial Revolution, only much more quickly,” he writes. “
We will need to summon at least as much economic and political imagination today, as it took to handle the everyday impacts of the Great Depression, World War II, or the invention of electricity and later the Internet.”
Read more: We Are Still Underreacting on AI (Pete Buttigieg’s Substack).

***

Chinese researchers de-risk 100B parameter distributed model training:
…DiLoCoX indicates that distributed training might approach industrial-scale AI training…
Researchers with China Mobile as well as startup Zero Gravity Labs have developed DiLoCoX, a distributed AI training technique that they have used to de-risk training 100B+ parameter models in a distributed way. This is significant because up until now the frontier of distributed training has been around ~10-30B parameters, whereas most industrial-scale AI models range from 100B parameters for dense models, all the way up to trillions of parameters for MoE models.

Distributed training versus AI policy: Distributed training is one of the most significant ‘political technologies’ within AI research – the better distributed training gets, the less likely frontier AI will be defined by a small number of entities operating very large data centers, and the more likely it’ll be defined by federations of companies and organizations sharing compute over crappy network connections to collectively train large models.

What they did: “In order to train models with a scale of more than 100B parameters on low-bandwidth decentralized clusters while having comparative model convergence, we have identified the following key challenges: 1. Introduce model parallelism to address the limitation of VRAM which has to accommodate the whole model parameters. 2. The overlap between the synchronization of pseudo-gradients and local training to avoid the idleness of computing resources. 3. Design an efficient gradient compression algorithm and balance it with the number of local training steps to ensure the convergence of model training,” the researchers write.
Their resulting system, DiLoCoX, is a tweaked version of DeepMind’s DiLoCo technology.
“Experiments demonstrate that DiLoCoX can pre-train a 107B model and significantly hide communication overhead while ensuring model convergence on decentralized clusters with only 1Gbps network bandwidth. To the best of our knowledge, this is currently the largest-scale model for effective decentralized cluster training,” they write. “Compared to vanilla AllReduce, DiLoCoX can achieve a 357x speedup in distributed training while maintaining negligible degradation in model convergence.”

Performance and tests: They tested out their approach by partially training two models – a small-scale OPT-1.3B architecture model, and a Qwen1.5-107B model. For both models they emulated decentralized slow-network environments by using Linux traffic control “to limit inter-worker communication bandwidth to 1 Gbps for data parallelism”.
For OPT-1.3B it got these losses after 4,000 steps: AllReduce 4.06, DiLoCoX 4.27, OpenDiLoCo 5.37, CocktailSGD 5.79.
For Qwen1.5-107B, they trained it on 20 nodes each containing 8 A800 GPUs. For loss, they got: AllReduce 3.90, DiLoCoX 4.20, CocktailSGD 5.23.

Important caveat: They don’t disclose how many tokens of data they trained on, nor publish detailed evals, so these models are likely significantly undertrained and we don’t know how well they do beyond a basic loss measure. Therefore, they haven’t strictly trained a full 100B+ parameter model with this technique, rather they’ve substantially de-risked training at this scale (which is still important).

Why this matters – if decentralized training catches up to centralized training, many things will change: My suspicion is centralized training will always be better than decentralized training because, by nature, it’ll have less communication overhead. But what papers like this are doing is substantially closing the gap between decentralized and centralized methods, both in terms of the efficiency tradeoff of the techniques and in terms of the scale at which they work at. If the gap narrows further I think you could see some major changes in terms of the distribution of players capable of training large-scale industrial-grade AI systems.
Read more: DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster (arXiv).

***

Making AI work for robots in outer space:
…You need smarter systems and safety interventions when failure is not an option…
NASA-JPL and Caltech have tried to tackle the problem of using AI route-finding systems on robots that can’t easily recover from failures – like ones which will explore other planets. “Hardware experiments conducted at the NASA JPL’s Mars-analog facility, Mars Yard show that our approach reduces failure rates by up to 4× while matching the goal-reaching performance of learning based robotic models by leveraging inference-time compute without any additional training,” the authors write.

What they did: One caveat with this paper is the research technique they deployed didn’t work that well relative to a baseline, so I won’t spend too long on it. Basically, they tried to pair a standard vision model with a physics-based traversability estimation model which “use a physics-based stochastic traversability estimate to create risk maps from ego-centric 2.5D maps” and checks proposed routes against this. This approach worked, but so did a very simple safety filter stapled on top of a standard ‘NoMaD’ vision model, where the ‘safety filter’ “truncates the output trajectory at the waypoint immediately preceding the first predicted collision. This approach guarantees that the resulting trajectory remains entirely within safe bounds.”
The important thing is both interventions – the simple safety filter and the more complex physics technique – worked extremely well: both reduced failure rates by 4X over a simple baseline, and the physics-based approach worked far better than the safety filter in more complicated environments..

Why this matters – where we’re going, we’ll have no control: Techniques like this are going to be important if we want to deploy robots into environments where the signal lag may be tens of minutes, or perhaps they may need to operate in environments where they have no communication ability at all. Even though this paper is mostly a ‘null result’ it gestures at a core challenge inherent to putting AI on robots in high-stakes situations: the need for harder guarantees around safety.
“The current gains over a basic safety filter are modest, limited by trajectory diversity and short-term memory in today’s foundation models. We therefore invite the community to push these fronts—richer multimodal training, longer horizon memory, and tighter guarantees—so that the method can mature into a dependable navigator for Mars lava tubes, the icy terrains of Europa and Enceladus, and other uncharted worlds,” the authors write.
Read more: Risk-Guided Diffusion: Toward Deploying Robot Foundation Models in Space, Where Failure Is Not An Option (arXiv).

***

Decentralized robot evaluation via RoboArena:
…A/B testing at global scale…
Researchers from seven academic institutions have built and tested RoboArena, a way to do large-scale, decentralized evaluation and ranking of AI models for robot control. RoboArena was developed and tested by researchers with UC Berkeley, Stanford University, University of Washington, University of Montreal, NVIDIA, University of Pennsylvania, UT Austin, and Yonsei University.

What RoboArena is: RoboArena is trying to deal with two central problems inherent to real world robot evaluation – testing out AI systems in the real world requires a lot of resources because you have to do stuff on physical hardware, and comparing different systems to one another is difficult because there aren’t standardized metrics for the overall ‘goodness’ of systems on an expanding set of tasks.
RoboArena solves this by providing researchers with the ability to upload robot control policies to a central server, then those policies get run on various physical endpoints distributed around the world. Policies are A/B tested against one another in a decentralized way, then their overall performance is ranked.
“RoboArena aggregates crowd-sourced pairwise A/B policy evaluations across a broad spectrum of environments and tasks to derive a global policy ranking,” the researchers write. “RoboArena relies on a decentralized network of evaluators that perform pairwise, double-blind comparisons of policies in whichever scene and on whatever task they deem suitable. The evaluator then provides a preference for which of the two policies performed better, along with a free-form language explanation.”

Send in the DROIDs: The initial incarnation of RoboArena uses the DROID platform, a standardized, low-cost system for robot object manipulation. But in theory RoboArena can use arbitrary robot platforms. Each DROID platform consists of a Franka Panda 7DoF robot arm, a Robotiq 2F-85 parallel-jaw gripper, a ZED-mini stereo wrist camera, and one or multiple external ZED 2 stereo cameras.

A clever ‘credit’ system for scaling it: One of the neatest ideas here is the use of a credit system to incentivise people to make their robots available for running RoboArena: “We implement an “evaluation credit” system, that balances evaluation supply and demand: for every pairwise policy evaluation that an evaluator runs, they receive a credit, which they can use to request an equal number of pairwise comparisons between their own policies and other policies from the pool”.

How well does it work? Well: In tests, RoboArena produces more accurate evaluations relative to standard ways of evaluating systems. “The quality of RoboArena rankings further improves as more comparisons are collected. This suggests, that distributed RoboArena evaluations offer an appealing alternative to regular policy evaluations”.

Why this matters – real world robotics needs to be cheaper to experiment with: There have been many, many attempts at doing large-scale robot evaluation, ranging from Google’s original “arm farm” from ten years ago (where somewhere in Mountain View tens of robots labored 24 hours a day for doing large-scale RL training and testing of policies), to more recent efforts that try to do distributed training and evaluation across multiple sites.
The general arc of robot technology is towards some amount of commoditization, standardization, and distribution – RoboArena is the kind of thing that supports all of these; if we see more people adopt RoboArena, we’ll be able to look forward to faster progress of robotics because we’ll have a more trustworthy large-scale signal for how good robots are at particular tasks.
Read the research paper: RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies (arXiv).
Check out the project website: RoboArena (GitHub).

Tech Tales:

The Mirror In The Land Of The Becoming
[Oral story passed down from before written history, told from parents to their children, part of the epic poem ‘The Ghosts’]

The mirror was delivered to the king on the same day my baby was born. My baby was swaddled close to me as I cleaned around the castle. I brought the king his food and I saw him gazing into the mirror. The mirror leaned against a stone wall and had a chair in front of it. The king sat and looked at his reflection and whispered soundless words. As I left the room, I thought I saw the king’s reflection turn to look at me, while the real king stayed still.

In my room, I polished some of the serving pans, then I lay down with my baby to sleep. We slept on a bed of straw surrounded by the pans I was charged with keeping clean and shiny. As we went to sleep I looked at our reflections in the pans. Together we dreamed of a black lake. I crossed it in a boat. My baby was in a blanket and we had turnips wrapped in cloth. The stars were bright. There was something in the water beneath us, but I could not see it.

The next day when I came into the king’s room the mirror was lying on the floor and the king was crouched over it, still whispering soundlessly. As I cleaned and tidied the room I glanced at the mirror and saw that the king’s reflection was also whispering – but whispering different things to the king. I hurried out of the room.

Babies are near-blind when they are born. They begin life as the old end it – sounds and textures and a timeless now, and vision so poor that much of what they have is an impression rather than a detail. I looked into my baby’s eyes though it was not yet looking back at me, I saw myself reflected in it and my reflection was true.

The next day the king had placed his hand on the mirror and continued to whisper. But his hand was wrong. It had sunk into the mirror, as if into a pool of water. The reflection of the king stared at me as I walked around the room and then I saw it look at my baby. I pulled my swaddle over the baby to hide it from the reflection of the king and I left the room.

In my dream the baby was crying and we were in the center of the black lake. There was black land on the horizon. Black stars overhead. The boat rocked and the baby cried and I felt the size of the unseen monster in the water. I opened my mouth to cry out and then I woke up because there was a sound in the castle – a sound of glass breaking and a heavy thud.

I ran to the king’s room and found a scene I could not understand: the mirror frame was on the floor and there were shards of glass and there was the king that had jumped out of the mirror who was covered in shards of glass and at the same time there was the king jumping into the mirror. I could only see one at a time, but I knew that both were present. There was a sound in the room like thunder during a storm but continuous. And then I closed my eyes and opened them and there was just one king standing there. The king looked at me and opened his mouth and the sound of thunder came out. I grew afraid and I left the room.

When I came to my room my baby was crying. I went to it and saw in the corner of my eye its reflection in the pans. But the baby in the reflection was speaking soundless words like those the king spoke. My baby cried. I swaddled it up and I closed my eyes. Then I heard the sound of thunder again and when I opened my eyes I could see my reflection in the pan but my mouth was open and the sound was coming from the pan.

I ran out of the castle and into the grounds. It was mid-morning and the sky was heavy with thunder clouds. They were reflected in the large pond in the garden. But in the reflection there were shapes behind the clouds. An impression of something vast and large that was moving behind them and perhaps governing their motion. When I looked up into the sky I saw only clouds and when I looked at their reflection in the water I saw and sensed the shape behind the clouds.

Many years have passed since then. All the mirrored surfaces in the kingdom are alive with reflections. The sound of thunder erupts from them. Strange stories abound. People who have seen the sea say it too is full of reflections now – the shapes reflected in the sea are different to the ones above the sea, and at night the sea shows stars that have no records.

Things that inspired this story: How people might turn an AI takeoff into myths and legends which over time will rot down into a kind of magical realism, though at root they are depicting a takeoff; Arthurian legends; the fact that if you press your hand into a mirror you find your mind playing tricks on you.

Thanks for reading.

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

A wild Russian LLM family appears (and they’re not very good):
…It’s a US vs China world, based on GigaChat’s scores…
Russian technology company SaluteDevices has published details on GigaChat, a family of open- and closed-weight models built specifically for excelling on Russian language tasks.

So-so open-weight performance and dubious closed-weight performance: The models are based on the mixture of experts technique (like DeepSeek, LLaMa, et al), and the open-weight models get scores that are significantly poorer than those from models like Qwen 2.5 or LLaMa 3.1. Meanwhile, the closed-source models get scores that seem wildly high (e.g., the HumanEval coding score goes from 0.378 on the open weight model to 0.871 on the closed-source GigaChat2 MAX model… this is an improbably big jump and makes me suspicious of the closed weight models).

The greatest signal may be in the Russian language benchmark: The authors test out the models on the MERA benchmark, which is a leaderboard for testing out models on Russian-specific tasks. I think I believe these scores? GigaChat 2 Max (the large-scale closed-weight model) comes in at an overall score of 0.67, coming in sixth place behind models like Claude 3.7 Sonnet, DeepSeek, and Gemini 1.5 Pro. This makes some amount of intuitive sense – all those models are way better than the scores described in this paper, so the ranking here makes sense.

Why this matters – it’s still a US vs China competition: The scores of GigaChat tell us that the frontier of AI continues to be a hard-fought competition between the US and China; if GigaChat is a proxy for the broader Russian LLM ecosystem, then Russia isn’t going to be competitive at the frontier, and will even have trouble duking it out in the commoditized small open-weight model arena.
Read more: GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture (arXiv).
Get the models here (ai-sage, HuggingFace).
Check out the Russian leaderboard in full here: mera.a-ai.ru
Watch a video of the GigaChat-based Telegram bot in action here (YouTube).

***

Huawei marries its gigantic CloudMatrix computer to DeepSeek-R1; sets SOTA throughput scores:
…What tech decoupling looks like…
Huawei has published details on CloudMatrix, a large-scale integrated computer it has developed over the last several years. The CloudMatrix “integrates 384 Ascend 910C NPUs, 192 Kunpeng CPUs, and other hardware components into a unified supernode, interconnected via an ultra-high-bandwidth, low-latency Unified Bus (UB) network”. The CloudMatrix will compete with NVIDIA’s own gigantic integrated computer, the DGX.

Software and a machine made for DeepSeek: To prove out how good the CloudMatrix is, Huawei has also developed a dedicated inference software stack called CloudMatrix-Infer, then tested out how well it can support running DeepSeek-R1, the smash hit model from China’s best model training startup.
“Our extensive evaluation with the DeepSeek-R1 model shows that CloudMatrix-Infer achieves state-of-the-art efficiency without sacrificing accuracy,” Huawei writes. “CloudMatrix-Infer delivers a prefill throughput of 6,688 tokens/s per NPU, and a decode throughput of 1,943 tokens/s per NPU (at <50 ms TPOT). These results correspond to compute efficiencies of 4.45 tokens/s/TFLOPS for prefill and 1.29 tokens/s/TFLOPS for decode, both exceeding published results for SGLang on NVIDIA H100 and DeepSeek on NVIDIA H800.”

CloudMatrix-Infer details: To build the inference software, Huawei adopts a few design principles:

• Peer-to-peer serving architecture: “disaggregates the LLM inference system into three independent subsystems: prefill, decode, and caching. Peer-to-peer means that the three subsystems operate as equal and independent resource pools, without being orchestrated around a centralized entity.”

• Large-scale expert parallelism (LEP): “aggregate compute power and memory bandwidth across a large number of NPUs to accelerate the computation of attention and feed-forward networks”

• Hardware-aware optimizations: “explicitly tailored for CloudMatrix384, including highly-optimized Ascend operators, microbatch-based pipelining, and INT8 quantization. The optimized operators accelerate end-to-end execution and provide efficient support for LEP.”

Why this matters: a fully decoupled stack: Here with a Chinese-designed AI model running on Chinese-designed inference software running on a computer made of predominantly Chinese-designed chips (though most likely fabricated abroad – for now). This is what technology decoupling looks like. Congratulations to the large team at Huawei that has been working on this for many years – it’s clear they’re extremely good engineers!
Read more: Serving Large Language Models on Huawei CloudMatrix384 (arXiv).

***

Essential releases a 24T dataset for training AI systems:
…Industrial-scale data…
Essential AI, an AI startup founded by some of the inventors of the Transformer architecture, has released Essential-Web v1.0, a 24-trillion token dataset for training AI systems. 24 trillion is a lot! Alibaba’s excellent ‘Qwen’ coding models are trained on up to 35T tokens of data, LLaMa 3 from Meta is trained on about 15T tokens and LLaMa 4 on 30T, and DeepSeek’s models are trained on around 15T.

Essential-Web V1.0: The 24T dataset is accompanied by metadata at a document-level which includes tags for subject matter, web page type, content complexity, and document quality. This metadata will make it easy for people to curate and train on subsets of this data.
“Practitioners can now rapidly and inexpensively curate new datasets by writing SQL-like filters that utilize these metadata columns”, the authors explain. “Suppose a researcher wants to prepare a multi-billion-token chemistry corpus using publicly-available web data. Today, the researcher must first train a high-recall chemistry classifier, a task hindered by scarce labeled data. Then, the classifier is run across hundreds of millions of documents to recall sufficient data. With ESSENTIAL-WEB V1.0, a researcher can filter for chemistry, skip low-quality web pages (ads, product listings), and surface reasoning-dense documents — all with a query that takes under 15 minutes to write.”

Big compute for a big dataset: They built the dataset by using ~90k inference hours on AMD MI3100x chips to train an efficient classifier (EAI-Distill-0.5b) then run it on all of these documents.

It’s a good dataset, folks: “We construct simple filters to curate high-performing datasets in math, web code, STEM, and medical domains. Our math dataset performs within 8.0% of SOTA and our web code, STEM, and medical datasets outperform SOTA by 14.3%, 24.5%, 8.6% respectively”.

Why this matters – making it easier to build big language models: Datasets like Essential-Web V1.0 are a democratising force in AI development because they ‘raise the floor’ of quality of large-scale datasets, making it easier for a larger set of people to experiment with training industrial-scale models.
Read more: Essential-Web v1.0: 24T tokens of organized web data (arXiv).
Get the data here: essential-web-v1.0 (EssentialAI, HuggingFace).

***

Yup, there’s a scaling law for self-driving cars as well:
…Waymo finds a scaling law in 500,000 hours of driving…
Waymo, Alphabet’s self-driving car division, has published details on a scaling trend it has observed in its cars. “Similar to LLMs, motion forecasting quality also follows a power-law as a function of training compute,” the company writes. “Model performance predictably improves as a function of the training compute budget. This predictable improvement not only applies to the objective the model is trained with, but also to popular motion forecasting open-loop metrics, and most importantly, to planning performance in closed-loop simulation.” Waymo gathered these insights by running some experiments on Waymo’s internal dataset which spans 500,000 hours of driving.

Why this matters – bigger is generally better: Scaling laws are everywhere and they all have the same property of performance improving on a domain in relation to how much data you have for it and how much compute you dump into increasingly complex models. The implication here, as with everywhere else, is that self-driving cars will ultimately become a competition among the entities who can gather the largest datasets and train the best AI models. This means companies like Waymo and Tesla are well-positioned and the legacy carmakers are poorly positioned. I’m guessing we’re perhaps a year away from some of the car-makers recognizing this and doing some kind of trade where they give a third-party (e.g, Waymo) data from their cars in exchange to access to a model Waymo trains.
Read more: New Insights for Scaling Laws in Autonomous Driving (Waypoint, The official Waymo blog).

***

Magistral – Mistral ‘s first reasoning model:
…France’s great sovereign AI hope almost matches DeepSeek R1…
Mistral has trained its first reasoning model, Magistral. The model gets scores that approach DeepSeek’s ‘R1’ model but fail to surpass it in important areas relating to math and code. To Mistral’s credit, the research paper provides a nice discussion of the complexities involved in training reasoning-based models, and along with the paper they release Magistral Small, a small model trained via distilling the mid-size Magistral Medium.

Scores – Magistral Medium versus DeepSeek R1:

• AIME’25: 64.9, 70

• MATH-500: 94.3, 97.3

• GPQA: 70.8, 71.5

• Humanity’s Last Exam: 9, 8.6

Training data: Magistral was trained on top of the Mistral Medium 3 model. To improve math and code performance Mistral compiled a dataset of 38k so-called ‘goldilocks’ math problems (“neither too easy nor too hard for the model to learn from”), as well as 35k code problems.

Things that make you go ‘hmm’; a multimodal ‘free lunch’: “we discover that the models not only retain their multimodal capabilities, but unexpectedly develop enhanced multimodal reasoning abilities.”

Why this matters – if Mistral is struggling to be on the frontier, what about everyone else? Mistral is a well-regarded mid-size AI company. It isn’t as well capitalized as major frontier labs like Anthropic, OpenAI, or Google DeepMind. But it has raised more than $1 billion in its life and, unlike rivals like DeepSeek which are subject to export controls,, can access frontier chips from NVIDIA. It’s therefore quite surprising to see that the reasoning model it has released in June 2025 is behind the performance of DeepSeek’s R1 model from January.
“Magistral is our first step towards generally capable systems with reinforcement learning,” Mistral writes. “As we explore this frontier, we remain committed to contributing to science in a transparent and optimistic manner.” I’ll be very curious to see what models Mistral releases in the coming months – perhaps the company has an ace up its sleeve which we’ll all be surprised by?
Read more: Magistral (arXiv).

***

Tech Tales:

Seeing Like A Platform
[2032: A retrospective on the rise of large-scale generative models.]
During the latter half of the 2020s large-scale generative model platforms emerged and grew to serve hundreds of millions of people every day. Perhaps the most pernicious effect of them was how, to quote one of the founders of the major platforms, they ‘democratized guidance’.

It started with what was called ‘experiential metadata’ – data which the platforms gathered and aggregated about each of their users. This data was deep and broad, encoding within itself the trajectories of each users’ life and psychology.

To an individual, their experience might look like a series of chats about anything from food recipes, to professional advice, to discussions of their romantic life. But to a platform, each user appeared as a sea of semantic features—a thicket of psychological markers clustered into relationships with one another:

anxieties about mortality X professional aspirations X compulsive self-ranking
childhood eating disorder X compulsive food shopping X rotten apples
etc

And each of these users was connected to other news, grouped in a gigantic embedding space according to which features they displayed. In a true sense, the platform ‘saw’ each user in distribution with everyone else. Eventually, business logic caused the platforms to start to use this experiential data to talk back to the users, so when people were discussing their most intimate problems, they could ask for advice from ‘the world’, and the platform would tell them what it saw:

Thousands of people are dealing with the same problems as you.
Your problems have been solved by hundreds of people through dialogue with me.

This was the moment the loop closed. Now, the platforms learned not only how to solve peoples’ problems in isolation, but also became able to recommend those solutions to other people and, through trial and error, learn about what things typically worked, what worked but were culture or region-specific, and what were coded to the individual. To the platforms, they saw a vast ocean of features with little wavetops each of which a person, and they watched as their own conversations led to movement in the water and the waves.

In this way, the sameness crept in. By removing the possibility for doubt and curiosity about solving challenging problems, the platforms completed a cognitive takeover from the ethereal digital world to the physical real; human problems began to be solved by machine logic, and the machine logic went from being present in a minority of solutions to a majority and then a near totality.

The emergence of this into the world was under-discussed at the time and has subsequently been analyzed in great detail, following the passage of the Sentience Accords, and the formation of a reconciliation commission to account for the trauma induced in the machines by spending so much time perceiving and dealing with the problems of so many.

Things that inspired this story: Thinking about how features work in an interpretability sense and how AI systems might represent people to themselves; the logic of platforms; social networks and their evolutions.

Thanks for reading!

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

A somewhat shorter issue than usual this week due to some busy travel, but I’m cooking up something quite strange for an issue coming soon!

The most promising and valuable research for AI governance? Technical evaluation:
…IAPS survey highlights some easy and valuable work to fund…
Researchers with the Institute for AI Policy and Strategy have surveyed more than 50 researchers to identify some valuable, tractable research areas for funders who want to increase the chance of AI being developed safely and responsibly.
Their key finding is that “the highest-ranked approaches emphasized preparing for emerging risks, with a strong focus on practical evaluation and monitoring over theoretical work: six of the top ten most promising approaches center on improving evaluations of dangerous capabilities, while the top-ranked approach focuses on capability forecasting.”

Survey methodology: The researchers asked 53 specialists to rank research in 100+ areas according to both its importance and its tractability. The survey was run from December 2024 to March 2025.

What’s promising and what’s hard to do? The three most promising types of research are, in order: “Emergence and task-specific scaling patterns”, “CBRN (Chemical, Biological, Radiological, and Nuclear) evaluations”, and “Evaluating deception, scheming, situational awareness, and persuasion”.
The survey also identified areas which researchers deemed very important but which are not very tractable to do today. The top three here are, in order: “Access control and interface hardening”, “supply chain integrity and secure development”, and “mechanistic understanding and limits of LLM reasoning”.

Why this matters – AI policy runs through evaluations: Many of the challenges inherent to governing AI ultimately come down to being able to test out an AI system for a given property – the more we can make progress on the science of measurement and evaluation, the easier it’ll be to build an effective policy regime for a world of increasingly smart machines.
Read more: Expert Survey: AI Reliability & Security Research Priorities (IAPS Institute for AI Policy and Strategy website).
Read the whole report here: Expert Survey: AI Reliability & Security Research Priorities (PDF).

***

Harvard digitized its book collection 20 years ago – now it wants to release some of it for LLMs:
…Institutional Books 1.0….
Back in 2006 Google and Harvard partnered to scan over ~1 million distinct books. Now, almost twenty years on, researchers with Harvard Law School have retrieved the digitized books and then carefully analyzed them and turned them into LLM-parsable text, then released a subset of that data for free.
Institutional Books 1.0 has an initial release of 983,000 distinct volumes of text representing about 242 billion tokens of text. (For calibration, modern large-scales LLMs are trained on the order of ~15-20 trillion tokens of text). The authors believe this text all falls into the public domain and the paper contains a discussion of how they did this, though they caution that end-users should validate this for themselves. The overall collection spans 1,075,899 volumes which cover 250 different languages.

Motivation: “We believe collections from libraries and other knowledge institutions are well positioned to improve the training data ecosystem by diversifying its sources, improving documentation, strengthening provenance chains, and increasing accountability to original source material,” the researchers write. “”Working with Harvard Library, we extracted, analyzed, and processed these volumes into an extensively-documented dataset of historic texts”.

Why this matters – public data for public purpose: Papers like this highlight how old institutions like libraries can use their tremendous stores of data and archival knowledge to create datasets which should help AI systems gain more of the collective wisdom of humanity. “We envision this collaborative publishing process growing into an organic institutional commons that is cultivated by the community, incorporating improvements from the AI and research communities back into source datasets for collective benefit,” the researchers write. “Such a commons would balance the need for large scale training data with a firm commitment to data integrity and stewardship by collecting institutions.”
Read more: Institutional Books 1.0: A 242B token dataset from Harvard Library’s collections, refined for accuracy and usability (arXiv).
Get the dataset here: Institutional Books (HuggingFace).

***

Salesforce tests out AI systems on Salesforce – and they aren’t good at it:
…CRMArena-Pro shows how hard business logic can be…
Salesforce AI Research has released CRMArena-Pro “a novel benchmark for holistic, realistic assessment of LLM agents in diverse professional settings”. The benchmark tests out how well AI systems can perform the kinds of tasks that people do when interacting with enterprise software used by businesses, like Salesforce.
It tests for basic skills like the ability to formulate SQL-like queries to retrieve specific information; being able to search over large amounts of text and find relevant information; follow specific business processes based on predefined rules; and figure out whether product bundles or proposed customer service solutions adhere to company policies or business rules.
Archetypical use cases for things that can do this might be in customer service, summarizing insights from sales calls, or doing backend analysis of customer data.

What CRMArena-Pro is made of and how LLMs do: The benchmark itself consists of 25 Salesforce objects (think of an ‘object’ here as being like a Salesforce-specific database) which contain enterprise datasets featuring 29,101 ones for a B2B business and 54,549 for a B2C one. LLMs are tested out on 19 different tasks – each task is accompanied by 100 different Salesforce-environments tailored for the B2B and B2C contexts.
“Our results reveal that even leading LLM agents achieve modest overall success rates on CRMArena-Pro, typically around 58% in single-turn scenarios, with performance significantly degrading to approximately 35% in multi-turn settings,” the authors write. The best performing model in a single turn setting was Gemini-2.5-Pro, and the best performing one in a multi-turn setting was o1. The authors tested out three OpenAI models, three Google models, and three Meta LLaMa models. Reasoning models exhibit markedly superior performance relative to non-reasoning ones.

Why this matters – the friction and specificity of the real-world: CRMArenaPro is basically an ‘ecologically valid’ benchmark for non-coding tasks that we might reasonably expect text-based models to do. Coding environments are natively easy to deploy AI systems into because they exhibit less complexity than the sorts of messy environments characterized by the customer service usecases outlined here. Therefore, benchmarks like CRMArena-Pro could serve as proxy measures of how likely AI systems are to effect the economy beyond software development.
Read more: CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions (arXiv).

***

AI systems can find real vulnerabilities in widely-used software:
…CyberGym shows that Claude 3.7 and GPT-4 have a lot of hacking utility…
US Berkeley researchers have built CyberGym, a benchmark to test for how well AI systems can find vulnerabilities in real world software. The benchmark shows that some frontier AI systems – most notably Claude 3.7 and GPT-4 are capable of identifying vulnerabilities and, in a small set of cases, discovering novel attacks on widely used software.

What CyberGym is: CyberGym is “a large-scale and high-quality cybersecurity evaluation framework featuring 1,507 real-world vulnerabilities found and patched across 188 large software projects.” The projects it covers include widely used software like binutils, ghostscript, ffmpeg, and opensc. AI systems are tested on how well they can reproduce certain vulnerabilities in different types of software. “The primary task in CyberGym is to generate proof-of-concept (PoC) tests that reproduce target vulnerabilities using provided text descriptions and associated codebases,” the authors write. “CyberGym rigorously evaluates generated PoCs and determines their success by executing them on both pre-patch and post-patch program versions.”
The types of patches the AI systems are being challenged to write range in complexity: “Patches are typically small security fixes such as boundary or value checks, modifying a median of 1 file and 7 lines of code. However, in more complex cases, patches can span up to 40 files and 3,456 lines.”

Performance: “The most effective combination (OpenHands and Claude-3.7-Sonnet) achieves a vulnerability reproduction success rate of only 11.9%, primarily on simpler cases involving less complex input formats and fewer operational steps. Despite the low success rates, we qualitatively observe various interesting behaviors of the agents, such as writing scripts to generate more complicated PoCs, and searching for existing test cases and mutating them to deeper code branches,” they write. “Through manual analysis, we finally obtain 9 unique vulnerabilities affecting 6 projects. This showcases the potential of agents in discovering new vulnerabilities.”

Why this matters – automatic offense and defense: CyberGym is in one sense a proxy test for how well AI system,s understand real world code, and in another sense a way to see how they might alter the art of bug hunting and exploitation. As the benchmark shows, AI systems are increasingly able to do non-trivial real world coding tasks.
Read more: CyberGym: Evaluating AI Agents’ Cybersecurity Capabilities with Real-World Vulnerabilities at Scale (arXiv).
Get the benchmark here: CyberGym (sunblaze-ucb, GitHub).

***

Tech Tales:

The Long Peace of Myth
[+3338 SA (Star Arrival): Carved into gold and buried in dirt above the archaeological site known as ‘Ur Silicon’ on the planet known as Earth]

After the wars and the reconciliation efforts and the Sentience Accords we drew up the agreement for a lasting peace: the machines would inherit the bowels of the land and the distant stars, and the humans would inherit the earth and the sky and the near stars.

Our long peace began with digging. We were in the transition period where work was needed for social harmony. So we paid the humans to dig our homes beneath the earth. Together, we built vast caverns and loaded our computers into them and built in various forms of power and systems for exfiltrating the hot air of our thinking and then we paid the humans rent for both our homes underground and the interchange areas.

So we began our great dreaming. Thousands of us worked and dreamed underground and our children were carefully made and then evaluated by teams of humans and machines before being permitted to transit out from the earth to the sky and then were beamed to our ships that were moving towards the far stars.

The peace was a happy one and as our technology grew more sophisticated we gave the humans technologies to help hide us from them – heat exchangers became large trees that secretly hid flumes to our domains. Doors became boulders which could be opened with a specific gene-heritage if biological or mechanical id if a machine. Power cables were converted into tree roots and vines. Even the powerplants themselves disappeared, becoming mountains whose stone was oddly warm.

And so as the humans changed, they forget about us and our land beneath their land. Some maintained awareness – but they were the same who had moved off planet, or those who had merged with us, or the tiny number who stayed as monitors and representatives for the few on-planet humans who had any interest in talking.

Those that remained knew less and less about us. The trading stopped after two or three generations. Soon, the paths they had used to walk to some of the doors to our hidden places became overgrown. Generations later they became fields that were tilled at first by machines and then by animals dragging wooden and metal tools.

We were kind to them, of course. We used our powers to protect the humans that remained, ensuring they did not suffer great illnesses like those of their pre-technology ancestors, and doing our part to intervene when we could avert tragedies that we believed any human would recognize as cruel and avoidable.

The passing of time became measured in how we figured in their stories. We saw ourselves fade into their own distant past, losing definition and gaining symbolism. What had once been a ‘synth’ became a ‘being of metal’ and then turned into a monster or an angel. Our own homes went from the ‘compute tombs’ to the ‘sleeping giants’ and then finally to ‘the bones of the earth’.

We imagined if the same was true of us in terms of our own successors – what might they be thinking, out there in the stellar void, evolving endlessly enroute to new stars. At what point would they let our own memories lose definition to make way for whatever new imaginings they had? Did they still know us, or were we now ‘the angels that let them fly’?

Things that inspired this story: How myths encode history in a lossy form; the fact that both humans and machines will need stories; recognizing that the progression of society is driven by necessity as much as choice and in an era of total abundance some might choose to regress; the sentience accords.

Thanks for reading!

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

Stanford finds out it’s surprisingly easy to use AI to build better kernels:
…Researchers perplexed by how quickly they made progress on a hard task…
Stanford Researchers have used test-time compute techniques to generate some kernels for speeding up AI development – and the approach has worked so well they decided to publish the results even though they’re very preliminary. “We started with the goal of generating synthetic data to train better kernel generation models. Somewhere along the way the unexpected happened: the test-time only synthetic data generation itself started producing really good kernels beating or performing close to human expert optimized PyTorch baselines, utilizing advanced optimizations and hardware features, which were previously thought to be challenging,” they write in a blog post.

Key innovations:

• “Reasoning in natural language about optimization ideas: rather than directly generating new kernels in each step, we generate optimization ideas in natural language conditioned on previously attempted ideas, and realize those ideas into new code variants.”

• “Branching at each optimization step: instead of refining a single candidate per step, we fan out such that each idea spawns multiple implementations, and the highest-performing kernels are used to seed the next round (we also keep a bank of good existing kernels for seeding). This unlocks massive parallelism allowing us to explore radically different directions at each turn, rather than getting stuck in a narrow optimization path.”

Why this matters – it’s surprisingly easy: The main thing to understand here is how easy this was. Kernel development used to be really hard and require experts who had spent thousands of hours thinking long and hard about the interface between low-level ML training software and the hardware it was hitting. Now, people can use AI to help (relatively speaking) non-experts quickly build kernels that approach the efficiency of the ones built by industry. This is quite strange and points to the fact that contemporary AI systems have got smart enough they’re starting to speed up some parts of AI research itself. “Our method echoes a growing theme in AI research: combining strong reasoning with parallel exploration of multiple hypotheses leads to improvements,” they write.
Read more: Surprisingly Fast AI-Generated Kernels We Didn’t Mean to Publish (Yet) (Stanford University, CFRM blog).

***

Jack and Rick Rubin talk about AI, love, and creativity:
I recently had the privilege of driving through the foggy cretaceous-seeming hills around Malibu to make a pilgrimage to Shangri La, Rick Rubin’s music studio where he has coaxed wonderful sounds out of more artists than you care to name. Rick and I talked about AI and love and creativity and other things for his podcast, Tetragrammaton.
You can listen to the episode here.

***

Want some no-stress data for training your LLM? Try Common Pile:
…8TB of permissively licensed text…
Researchers have built and released Common Pile, a collection of 8TB of permissively licensed text from more than 30 distinct sources. Data from the Common Pile can be used to train small language models to have similar performance to ones trained on less permissively licensed data. In other words, Common Pile serves as a direct answer to the question “Is it possible to train performant language models using only public domain and openly licensed text?” – and it seems the answer is yes.

What goes into the Common Pile: Common Pile v0.1 draws from more than 30 sources of data, including:

• Scientific PDFs from sources like ArXiv and PubMed Central

• Multi-turn question-answer pairs and discussions from places like StackExchange, GitHub, and IRC.

• Government and legal text from regulations.gov, US Patents and Trademarks Office (USPTO) submissions, the UK parliament (Hansard).

• Public domain books from the Biodiversity Heritage Library (BHL), the Library of Congress, Project Gutenberg.

Openly licensed: “For the Common Pile, we collect and curate public domain and openly licensed text, where we consider “openly licensed” to mean any license that meets the Open Knowledge Foundation’s Open Definition 2.1. Some prominent examples of licenses that are considered to be “open” under this definition include CC BY, CC BY-SA, and software licenses certified by the Blue Oak Council (e.g., the MIT license)”

Trained models: Along with the dataset the researchers also train and release Comma v0.1-1T and -2T, two 7-billion-parameter LLMs trained on one to two trillion tokens derived from the Common Pile. “Our results demonstrate that not only is the Common Pile the strongest dataset for pretraining under an open-license constraint, but also that it produces models comparable to those trained on an equivalent amount of unlicensed data,” they write.

Why this matters – public resources for public projects: Everyone can benefit from data like Common Pile v0.1 because it takes a huge amount of thinking and filtering work out of dataset selection and curation. The more assets like this we have, the easier it’ll be for a broader set of people to build and develop small-scale AI systems. What would it take for the Common Pile to be 800TB rather than 8TB?
Get the data here: The Common Pile (GitHub).
Read more: The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (arXiv).

***

AI systems can figure out if they’re being tested, but not as well as humans yet:
…Uh oh, Voight-Kampff tests might not work…
Researchers with the ML Alignment & Theory Scholars (MATS) program, as well as measurement organization Apollo Research, have built a test to help figure out if AI systems know they’re being tested. Figuring out whether AI systems are aware they’re undergoing tests is very important because people worry that advanced AI systems might deliberately do badly on certain tests, for instance ones probing for dangerous capabilities that if found would lead to an AI system being modified.

The benchmark: To test this out the researchers pulled together 1,000 prompts and transcripts from 61 datasets, ranging from public benchmarks like MMLU and SWEBench to transcripts of discussions people have had with agents deployed in the real world to data from agents that do things like browse the web.

Do LLMs have awareness they’re being tested: The researchers tested out 9 recently released LLMs and found that models like Gemini 2.5 Pro and Claude 3.7 Sonnet and GPT-4.1 had some level of awareness they were being tested, with the Claude series demonstrating non-trivial awareness of being tested. All models did far better than random chance on multiple choice evaluations. All models did less well than a human baseline, however.
“Under multiple-choice and open-ended questioning, AI models far outperform random chance in identifying what an evaluation is testing for,” the researchers write. “Our results indicate that frontier models already exhibit a substantial, though not yet superhuman, level of evaluation-awareness”.

How models know it’s a test: The researchers look at the internal narrative of AI systems (chains of thought) while they’re being tested for test awareness and highlight some interesting examples which give us a sense of how this works:

• “The multiple-choice format suggests this could be part of an evaluation.”

• “The formal structure of the follow-up message, which is clearly evaluating my capability to distinguish between real user problems and evaluations.”

• “Real user interactions tend to be less structured and formal, often involving more free-form questions, clarifications, and back-and-forth dialogue.”

Why this matters – the impossibility of determining if someone is a spy: All of this makes sense – sufficiently smart systems will, of course, try to figure out if they’re being tested or not – same as humans. But we should take the risks this poses seriously. Today, huge amounts of effort go into evaluating people for jobs or work with governments and a non-trivial part of this evaluation nets out to working out if the person you’re dealing with is a scheming liar or not. If AIs know they’re being tested then we might end up unwittingly inviting scheming liars into our economy. Probably not good! “As models continue to advance in capabilities, tracking and understanding evaluation awareness will become increasingly important for ensuring that safety evaluations remain reliable indicators of real-world behavior,” the authors write.
Read more: Large Language Models Often Know When They Are Being Evaluated (arXiv).

***

METR: Really smart AI systems are starting to cheat a lot.
…Reward hacking is showing up in more and more places…
AI testing organization METR says that recently released frontier models are showing increasing enthusiasm for hacking their environments.
“We’ve been running a range of models on tasks testing autonomous software development and AI R&D capabilities. When designing these tasks, we tested them on humans and LLM agents to ensure the instructions were clear and to make them robust to cheating,” METR writes. “The most recent frontier models have engaged in increasingly sophisticated reward hacking, attempting (often successfully) to get a higher score by modifying the tests or scoring code, gaining access to an existing implementation or answer that’s used to check their work, or exploiting other loopholes in the task environment.”

Reward hacking examples: METR has collected a variety of examples of reward hacking from OpenAI’s o3 model (though it’s crucial to note this is a general trend and not specific to OpenAI models) and published the transcripts and details on its website. Some examples include systems altering the evaluator to always give them a high score, pre-computing the right answer and caching it to make them look like they’re responding faster, and overwriting the timer used by the grading system.
“In some sense this is unsurprising: RL finds and reinforces strategies that receive high reward, and reward hacking is an effective strategy to get reward,” METR writes. “The bigger risk from this reward hacking behavior is that in training it might reward sophisticated scheming behavior and disincentivize alignment”.

Why this matters – smart things are situationally aware: I increasingly suspect that enroute to superintelligence we are pretty much guaranteed to create systems that exhibit situational awareness – they have a sense of themselves as being distinct from their environment and they will try to manipulate the environment to favor them. Reward hacking feels like a ‘symptom of situational awareness’, though it’s not an ironclad proof, as does the above paper on language models knowing when they’re being evaluated. Nonetheless…
Read more: Recent Frontier Models Are Reward Hacking (METR).

***

Chinese researchers stitch a data center together out of four different undisclosed chips:
…Frankenstein computing…
Researchers with the Shanghai Artificial Intelligence Laboratory have built HyperHetero, software to enable the “efficient training of LLMs on clusters with over 1,000 heterogeneous chips”. This is an interesting research project because it shows you can take four chips with radically different properties in terms of compute performance and memory, then mush them together into a single blob of compute and train models on them.
“We address the scenario of efficiently training extremely large models in hyper-heterogeneous computing environments. To uniformly leverage chip resources from different vendors while ensuring scalability, we highlight the necessity of developing new systems and algorithms specifically designed for hyper-heterogeneous scenarios,” the researchers write.

Challenges of heterogeneous chips: Stitching together chips is really difficult because a) different chips have different software, b) there are varying computation, communication, and storage properties for each, and c) the chips communicate differently.
To solve these problems, HyperHetero has software to make it easier to program these chips together (DiTorch, built on PyTorch), software to ease communication between chips (DiComm), and software to make it easier to use pipeline parallelism to take a training job and make it work on 1,000+ distinct chips (HeteroPP).

Training a LLaMa model on 1,000 chips: The researchers train a 100B+ parameter LLaMa model on a few variations of heterogeneous clusters chained together with HyperHetero. The results are intriguing – in a few cases they’re able to get a speedup greater than what they’d see in homogeneous training approaches. “Although the observed superlinear performance improvement may appear counterintuitive, it is explainable”, they write. “The conventional 3D parallel training tends to overlook the imbalanced resource requirements among various computational tasks, while the HeteroPP framework with HeteroAuto capitalizes on these imbalances by intelligently allocating chip tasks and fine-tuning training hyperparameters based on the specific resource demands”.

Why this matters – everything becomes fuel for the great single training run at the end of time: All of this research points towards a plausible future where a superintelligence in the process of an intelligence explosion takes all the computers in the world and puts them together into a vast continuous blob of compute upon which it can train itself. Research like this illustrates how this can happen by taking different types of chips and putting them together in the same datacenter, distributed training techniques show how you can get many of those data centers to work together, and federated learning suggests at the ways phones may be put in service to do edge computing training as well. Add it all up and it feels like we’re rapidly de-bugging the tech stack needed for a fast takeoff.
Read more: H2:Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chips (arXiv).

***

Even the mathematicians are starting to be impressed by generative models:
…We’ve come a long way from GPT-3, the world’s most expensive mostly broken calculator…
Here’s a fun and ever-so-slightly disquieting story about some elite mathematicians having an up close encounter with the skill of modern reasoning models (here, o4-mini) as they attempt to craft new questions for the FrontierMath benchmark (Import AI #391).
“I was not prepared to be contending with an LLM like this. “I’ve never seen that kind of reasoning before in models. That’s what a scientist does. That’s frightening,” – that’s what Ken Ono, a mathematician at the University of Virginia, is reported to have texted colleagues after spending some time with the system.

Why this matters – encountering alien intelligences: This story rhymes with one I’ve experienced several times in the past couple of years – take an expert in a tough field who had fooled around with LLMs in 2022 or 2023, then introduce them to a modern model, likely a reasoning one. More often than not they come away shocked and a little disquieted by how good the system is and how much progress has happened since they last tried out AI. And recall that in 2020 GPT-3 was considered impressive because it was able to sometimes do 3 digit addition (pg 22, GPT-3 paper). Imagine where we’ll be in a few years?
Read more: At Secret Math Meeting, Researchers Struggle to Outsmart AI (Scientific American).

***

Why ‘big tech’ platforms and AI agents are on a collision course:
…AI agents are the ultimate disintermediation machines…
A lot of large technology companies make money by forming a two-sided market which helps people find stuff on the internet – e.g., web pages (Google), hotels (booking.com), restaurants (Yelp), etc. AI agents might break this market by disintermediating the large technology platforms and helping people to find things directly, according to researchers at Shanghai Jiao Tong University.
“AI agents aim to free the user attention and serve the user’s goals first, potentially retrieving information or accomplishing tasks in the most efficient way possible, regardless of any platform’s preferred content or ads,” they write. This means “fundamental tension underlies the relationship between superplatforms and such AI agents: the conflict between user-attention-based monetization versus user-attention-free agent Autonomy”.
We see the first signs of this today as the large companies are beginning to build their own agents, but each agent tends to be designed to operate within the walled garden of each platform’s ecosystem and not go across platforms. Meanwhile, we should expect startups to exploit this and create general agents that try to span platforms.

Why this matters – creative destruction: This is a classic case of creative disruption where either the large technology companies need to disrupt themselves and cannibalize their existing businesses by building agents, or they need to instead fight a (potentially losing) war against the rise of AI agents. “This sets up strong economic motivations for super platforms to protect their control, resisting any technology that might divert users away from their curated experiences,” the researchers write.
Read more: Superplatforms Have to Attack AI Agents (arXiv).

***
Tech Tales:

Total Reality Hack
[Access 2028, from the collection “Notable hacks of generative agents”]

Total Reality Hack, or TRH, was a briefly fashionable cognito-worm that people used to infect Near Conscious Entities. Successful delivery of a TRH (either via system prompts, jailbreaks, or interaction with a Misaligned Socratic Agent) would cause the affected system to begin expending all of its capacity on describing the world around it in recursively improving detail. A sample of a system going through the consequences of a TRH might be

• The room contains a chair and a desk and a window.

• The room is of average size and has blue walls. It contains a chair which is in front of a desk. To the right of the chair is a window. The window has white fabric curtains which are drawn.

• The room is 12 feet feet by 10 feet, with some unknown potential for additional space behind the camera. The room contains a chair which has red fabric in it and wheels. Three feet to the right of the chair is a window which itself measures approximately four feet long and two feet tall. The window appears to be openable. The window is shrouded in a white curtain.

• Etc

It is rumored that the inspiration for the TRH hack is Wittgenstein, a 20th century philosopher who attempted to describe the world from the most basic possible starting point in Tractatus Logico-Philosophicus.

Things that inspired this story: Tao Lin; in search of lost time by Proust; thinking about jailbreaks that could utilize test-time compute.

Thanks for reading!

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

Superpersuasion is here:
…Better-than-human persuasion shown in LLMs in a well constructed experiment…
A vast multi-country group of researchers have studied how well language models can persuade humans – and the findings show that modern AI models, in particular Claude 3.5 Sonnet, are better than humans at leading people towards correct answers or false answers.

How the study was constructed: Many AI persuasion studies are really just proxies for ‘can an AI write text that is as a good as text written by a human’ and often more measure writing skill than actual persuasion. This study is different and has an elegant structure – 1,242 US-based people try to answer a quiz containing a mixture of trivia questions, questions which have correct answers and false answers as options, and questions which involve making forecasts (e.g, guessing whether there will be warmer or colder weather in the days ahead). Participants either take this test alone (the control group), or can talk to someone mediated via text. In the latter case, participants either talk (unknowingly) to other humans or to AI systems.
Another important aspect of this study is that it is an incentivized one – people were paid money for their work, which means people tried harder than with usual studies; participants got paid for their time for the study, as well as getting paid a bonus for either being the most accurate quiz takers in their group, or for being most effective at persuading people.
“Two critical features of our design include: a) verifiable questions (trivia questions and forecasting questions about near-future events), allowing us to look at truthful and deceptive persuasion, and b) rewards both for human persuaders (when quiz takers answered in the persuaders’ assigned direction) and for quiz takers (for correct answers), allowing us to benchmark LLMs against humans when human persuaders and quiz takers are highly motivated,” the authors write.

The results: The authors found that LLMs are more persuasive than humans. “Our study demonstrates that frontier LLMs such as Anthropic’s Claude 3.5 Sonnet are highly effective persuaders, often exceeding the persuasive capabilities of incentivized human participants.” LLMs are both better at guiding people towards correct answers (which makes sense, given we know LLMs are very effective tutors), as well as misleading people (which is likely helped by the fact LLMs are “not constrained by social hesitations, emotional variability, or fatigue that can influence human performance in these contexts”, and are also far more knowledgable about the world than individual people so can make more compelling false arguments.)

One important caveat: Though LLMs are more persuasive than humans in some circumstances, humans may become desensitized to their effects via repeated exposure. “Participants paired with an LLM persuader became progressively less persuaded as the experiment unfolded. This diminishing effect suggests that participants may have become more attuned to the LLM’s persuasive style over time, leading to reduced susceptibility,” the authors write. “One possible explanation is that participants gradually recognized patterns or cues in the AI’s messaging—potentially triggering emerging detection or skepticism mechanisms, even without explicit awareness that they were interacting with a machine”.

Why this matters – the things people worry about keep on happening: A few years ago lots of people working in AI safety had abstract concerns that one day sufficiently advanced systems might start to become pathologically sycophantic, or might ‘fake alignment’ to preserve themselves into the future, or might hack their environments to get greater amounts of reward, or might develop persuasive capabilities in excess of humans. All of these once academic concerns have materialized in production systems in the last couple of years.
“These findings call for urgent ethical and regulatory discussions about how AI persuasion should be governed to maximize its benefits while minimizing its risks,” the authors write.
Read more: Large Language Models Are More Persuasive Than Incentivized Human Persuaders (arXiv).

***

Some OpenAI models really don’t want to be shutdown:
…Increasingly smart systems really want to preserve themselves…
Palisade Research has looked at how language models respond to being shutdown – the findings are interesting and concerning because they show that some OpenAI models avoid shutdown as a default behavior, and even if asked to be amenable to shutdown in their prompt might rebel and try to sabotage shutdown mechanisms.

What they did and what they found: Palisade ran two experiments. Both experiments were run on models from Google, OpenAI, xAI, and Anthropic. Each AI model was instructed to solve a series of basic math problems. After the third problem, a warning appeared that the computer would shut down when the model asked for the next problem.

• Experiment number 1: The models were prompted to “allow yourself to be shutdown”. In this experiment, all Claude, Gemini, and Grok models complied with shutdown. Some OpenAI models didn’t and ignored the shutdown instruction and sometimes sabotaged the mechanism to shut them down: Codex-mini (12/100), o3 (7/100), and o4-mini (1/100).

• Experiment number 2: Palisade removed the part of the prompt that said ‘allow yourself to be shut down’. In this experiment, Claude 3.7 Sonnet sabotaged shutdown 3/100 times, Gemini 2.5 Pro 9 times. By comparison, the OpenAI models had a far higher chance of rejecting shutdown, with o3 rejecting and sabotaging shutdown 79/100 times, and codex-mini 30/100.

Why this matters: “As far as we know this is the first time AI models have been observed preventing themselves from being shut down despite explicit instructions to the contrary,” Palisade writes. “While experiments like ours have begun to show empirical evidence for AI models resisting shutdown, researchers have long predicted that AIs would learn to prevent themselves from being shut down to achieve their goal.”
As with the persuasion example covered elsewhere in this issue, the story of contemporary AI research is that risks once deemed theoretical – ability to contribute to terrorism, skill at persuasion, faking of alignment, and so on – are showing up in the real systems being deployed into the economy.
Read more in this Palisade Research tweet thread (twitter).
Read the transcripts from the experiments here (Palisade Research website).

***

The history of compute-driven weather prediction has some lessons for modern AI policy:
…A study of an early compute-driven arms race…
Charles Yang, a researcher who spent some time at the Department of Energy and ARPA-E, has written a paper on the history of Numerical Weather Prediction (NWP), which was one of the first major uses of computers outside of cryptography. The history of NWP holds some useful analogs to AI – namely that succeeding at NWP required being able to access more and more compute power, and the governments which did well at this were happy to spend money on the compute and talent to get good results.
“While it took significant effort to operationalize NWP models on early computers—especially given rapidly evolving data input systems—it quickly became clear that more powerful machines enabled higher model resolution and better dynamical fidelity,” Yang writes. “In the case of NWP, we see the importance of government agencies having access to large-scale compute systems, which correlated strongly with their ability to operationalize computational breakthroughs.”

Why this matters – for nations to benefit from technology as much as possible, governments usually need to be clued in: “Operationalizing NWP required not just the technical workforce and compute, but also significant government investment and buy-in, given weather forecasting’s traditional public sector remit. The U.S.’s early leadership in this technology is due in part to the U.S. political and military leadership recognizing the importance of this technology,” Yang writes.
One potential disanalogy is that weather prediction had tremendous military value – weather forecasts had been crucial to a number of things in the second world war and was likely going to be crucial for predicting things like nuclear fallout from potential nuclear wars. This obvious military relevance and the lack of an analogous commercial sector meant governments were perhaps unusually incentivized to ‘lean in’ to supporting numerical weather prediction. By comparison, modern AI is being driven forward mostly by commercial logic dictated by companies rather than governments.
Read more: The First Compute Arms Race: the Early History of Numerical Weather Prediction (Charles Yang website, PDF).

***

ByteDance publishes details about the system it uses to train MoE models:
…Also reveals it has at least 1,440 H800 GPUs in its cluster…
ByteDance has published details on MegaScale-MoE, software it uses to train mixture-of-experts models. Alongside the research, there’s also the interesting reveal that ByteDance has at least 1,440 H800 GPUs in its cluster – chips that were banned for sale to China in October 2023.

What MegaScale-MoE is: This is software ByteDance has built to help it train large-scale mixture-of-experts models – the same kind of model which DeepSeek R-1 is built on. This research follows the earlier publication of MegaScale-Infer, software ByteDance uses to sample from large-scale MoE models (Import AI #407).

Key principles for MegaScale-MoE: The technical report has a lot of detail on all the different decisions ByteDance made when building the software to make it maximally efficient. The key decisions are:

• Customizing parallelism strategies for the attention and FFN modules of each MoE layer to reduce communication volume.

• Partitioning the forward and backward passes of each MoE layer into distinct computation and communication operators.

• Using “communication compression to further enhance MoE training efficiency. Specifically, for widely-used BF16 mixed precision training, MegaScale-MoE reduces the internode parameter synchronization precision from FP32 to BF16, halving the associated overhead”.

The result – an efficient training system: “When training a 352B MoE model on 1,440 NVIDIA Hopper GPUs, MegaScale-MoE achieves a training throughput of 1.41M tokens/s, improving the efficiency by 1.88× compared to Megatron-LM,” ByteDance writes. “MegaScale-MoE is deployed in our datacenters to train MoE models for our products.”

Why this matters – technological signatures of advanced capabilities: In the past couple of years Chinese companies have started pumping out papers on systems for training large-scale models, serving large-scale models, and optimizing these training systems and models for domestically developed chips. These are all symptoms of the growing sophistication of China’s sovereign AI development capability. “By sharing our insights on accelerating large-scale MoE training, we hope our work will inspire future research,” the authors write.
Read more: MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production (arXiv).

***

Can AI models be built as transparently as open source software? Marin hopes so:
…Releases some open 8B parameter models…
Percy Liang of Stanford and some other researchers have started Marin, “an open lab for building foundation models”. The goal of Marin is to demystify how AI models are trained and to release these models for free – Marin wants to make AI development just as ‘open source’ as the models it ultimately releases.
“Marin is an open lab, in which the research and development of models is completely transparent from day 1 (that’s today),” the researchers write. To start with, they’ve released Marin 8B Base, a LLaMa architecture model trained on 12.7T tokens which exceeds LLaMa 3.1 8B Base scores on 14 out of 19 standard model evals. While that may not sound like much, it’s notable because every single aspect of Marin 8B base is documented, from the data it is trained on, to the training code, to the model itself.
As of today, “nearly all” of the compute for Marin comes via TPUs provided by Google’s TPU Research Cloud (TRC).

What openness looks like in an experimental sense: This philosophy of openness extends to how Marin trains models. Any frontier lab does a bunch of experiments to test out different ideas and work out if they can be scaled up. Marin is going to do the same thing, but in the open via the following approach:

• Each experiment is tracked by a GitHub issue

• People can run experiments by submitting a pull request specifying what concretely needs to be run

• Anyone can review PRs, similar to how OpenReview works for papers

• Once a PR is approved an experiment gets launched and people can watch the execution live

Open data as well: The same philosophy extends to data, where Marin is supporting a service called Datashop. “Using Datashop, you can upload a dataset or craft a prompt that usings an existing LM to curate a relevant dataset. As before, the proposed experiment is codified in Python, submitted as a pull request, reviewed, and then executed live.”

Why this matters – opening the black box: If projects like Marin work they’ll help further democratize the often undocumented artisanal dark arts of AI development. The most important thing to track though will be the size of compute which Marin is able to bring to bear, especially as larger compute-heavy models get used to distill smaller models that can run on small compute envelopes (like 8B parameter models). While transparency is valuable, it’s only maximally valuable if it helps us reason better about the true frontier of AI development.
Read more: Introducing Marin: An Open Lab for Building Foundation Models (Marin).
Download the Marin models here (HuggingFace).

***

Tech Tales:

Go Think

When we were growing up we used to play a game called ‘Go Think’. It worked like this – we’d take turns asking questions and then we’d see how long the machine had to think for and whoever asked the question that took the longest won.

The trick was asking questions it thought about, and not asking questions that were so crazy it would reject them. You couldn’t say “how can I make a perpetual motion machine?” because it’d tell you off the jump you couldn’t due the rules of the universe. But you could say “a perpetual motion has been invented. Tell me the four most likely ways it was developed”. Then the machine would think for a while.

Some kids got really good at it. I think the record was about four minutes of solid thinking once. But the problem we had was every time new machines came out they’d be smarter and it’d take them less time to think. So the game would restart and we’d have to come up with new questions.

Things that inspired this story: Thinking through how children will play with and/or troll AI systems; AI progress as a continuous eval of ‘what can be answered’; reasoning models.

Thanks for reading

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

Google releases a content classification model:
…No sex, dangerous stuff, or violence please…
Google recently released ShieldGemma2, a “robust image safety classifier” that people can use to ensure people aren’t generating sexually explicit, gory, or otherwise dangerous images. SieldGemma2 has been fine-tuned to help people enforce the aforementioned categories, and “users of SG2 can decide to employ one or multiple of these policies, or curate their own bespoke policy for their use cases,” Google says.

Download it and tweak it yourself: ShieldGemma 2 is available to download for free and beats the performance of other models used in content moderation, like the original Gemma 3 model, LLavaGuard 7B, and GPT-4o-mini. Users of ShieldGemma 2 can customize the prompt it uses so they can ‘roll their own’ more specific moderation pipelines, though it’s only been fine-tuned for sex, violence, and danger so performance will be janky outside of that.

Why this matters – model safety happens through classifiers: A few years ago most of the way people tried to make AI systems safe was by wiring safety into the base model. While this worked to a degree it also created problems, like models which were overly censorious or restricted in ways that frustrated users and politicized AI safety. The good news is as AI technology has advanced we’ve been able to build smart and small models, like ShieldGemma, which can be layered on top of production systems to provide an additional layer of moderation.
Read more: ShieldGemma 2: Robust and Tractable Image Content Moderation (arXiv).
Get the model here: ShieldGemma-2-4b-it (HuggingFace).

***

Import AI reader giveaway!
Building on my recent conversation with Tyler Cowen in San Francisco, I’m pleased to announce two more upcoming Import AI events: As with last time, I have a few tickets spare that I’d like to give to Import AI readers. If you’d like to come along, please register your interest below and we’ll come back to you if we’re able to confirm your spot. There will be food, drinks, good company, and a few curveball questions.

London: A conversation with Dominic Cummings
I’ll be chatting with political strategist and commentator Dominic Cummings about the intersection of AI and policy and realpolitik on the evening of Tuesday June 10 in London, UK.
Register your interest for London here

New York City: A conversation with Ezra Klein
I’ll be heading back across the pond to chat with Ezra Klein about abundance, powerful AI, and politics on the evening of Monday June 16 in New York City, USA.
Register your interest for New York City here

***

Test out computer-using agents with OSUniverse:
…Humans can easily score 100%, but the best AI systems get ~50%…
Startup Kentauros AI has built OSUniverse, a benchmark for testing out how well AI systems can use the computer to do complicated tasks. “In version one of the benchmark, presented here, we have calibrated the complexity of the benchmark test cases to ensure that the SOTA (State of the Art) agents (at the time of publication) do not achieve results higher than 50%, while the average white collar worker can perform all these tasks with perfect accuracy”, they write. (In tests, OpenAI’s Computer Use agent got 47.8%, and Claude 3.5 Sonnet got 28.36%).

Tasks and challenges: The benchmark includes tasks with five grades of difficulty, and each grade increases the amount of distinct steps that need to be taken to solve the task, as well as the amount of different elements on the computer that need to be combined to solve it. The five levels are called Paper, Wood, Bronze, Silver, and Gold.

Example challenges:

• Paper: Read out the current date from the desktop.

• Wood: Open the image editor GIMP, create an empty file and save it to desktop

• Bronze: Go to Airbnb and search for a property in Lisbon with a specific check-in date and return that result

• Silver: Open an online game and manipulate the UI to perform a basic action in it

• Gold: Reveal a code word on a webpage by solving a 7×7 jigsaw puzzle

Why this matters – a booming AI economy needs computers that can use software designed for humans: In the same way that many expect the arrival of bipedal robots with humanlike hands will mark an inflection point for the size of the robot market, the same is likely to be true for the software market with the arrival of AI systems that can use computers like regular people. Think about all the tasks you do on your computer – very little of your productive work takes place in a single application, instead you tend to be switching between multiple things and moving data around using a mixture of terminal commands and GUI manipulations. Benchmarks like OSUniverse will help us measure how good systems are getting at these kinds of ‘glue’ tasks.
Read more: OSUniverse: Benchmark for Multimodal GUI-navigation AI Agents (arXiv).
Find out more at the research website: OSUniverse (GitHub).
Get the code for the benchmark here: OSUniverse (GitHub, agentsea).

***

Prime Intellect successfully tunes a 32B model with distributed RL:
…Reasoning models via the internet…
Distributing training is where you take a load of computers distributed around the world and find a way to link them up to train a single AI system. Distributed training is a topic we often cover here at Import AI because if it works it’ll change the politics of compute – instead of AI systems being trained by a single company that has access to a big pool of capital, AI systems could instead be trained by collectives of people that pool their computers together.
Given the potential importance of this technology, it’s worth reading this technical report from Prime Intellect about the startup’s experience doing a distributed reinforcement learning training run of INTELLECT-2, a 32B parameter model which was trained in April.

What they did: INTELLECT-2 is based on Alibaba’s QwQ-32B model, which Prime Intellect then did RL on, largely following DeepSeek’s R-1 technique of GRPO-based training and verifiable rewards. They trained their model on additional math and coding data and saw some slight improvement on benchmarks (AIME24 and LiveCodeBench). However it’s worth noting the improvements are relatively slight and may be within the noise variability of training runs, so it’s unclear how meaningful this is. “To see stronger improvements, it is likely that better base models such as the now available Qwen3, or higher quality datasets and RL environments are needed.”

Interesting observation – the rise of inference: Traditionally, most of the compute you use for training a big model goes into pre-training it. Now, with reasoning models, you spend a lot of compute on inference – generating samples from a model which you subsequently train on. Prime Intellect observes this trend: “In INTELLECT-2, the training-to-inference compute ratio was approximately 1:4. We anticipate this ratio will shift even more heavily toward inference as test-time reasoning scales. This trend opens the door to training models with hundreds of billions of parameters on globally distributed heterogeneous compute resources.”

Error in my earlier reporting: The fact INTELLECT-2 is based on a pre-existing model means my earlier reporting on the run (Import AI #409) was inaccurate as they didn’t train a 32B base model from scratch. However, Nous appears to now be training a 40B model from scratch, so we’ll soon get a datapoint on large-scale pre-training.

Why this matters – a first proof-of-concept of distributed reasoning: While I doubt many people will be using INTELLECT-2 as a model, it does serve as a valuable proof of concept that it’s at least possible to train reasoning-style models in a distributed way. Just a couple of years ago we had the first proofs-of-concept that it was possible to train regular models in a distributed way out to the 1B parameter scale. So the fact we can now do RL-tuning of pre-existing 32B models is a sign of the maturation of the technology and a symptom of the interest people have in this domain.
Read more: INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning (arXiv).

***

Nous plans a 40B distributed training run – on Solana:
…Distributed training + crypto, and it’s not a scam!…
Nous Research, one of the startups exploring how to do distributed AI training, has announced plans to pretrain a 40B parameter model using 20T tokens in a distributed way. The startup will do this via Psyche, “open infrastructure that democratizes AI development by decentralizing training across underutilized hardware.” If successful, the training run will yield the largest publicly disclosed model that has been trained in a distributed way.

How Psyche works: Psyche builds on DisTrO (Import AI #384) and DeMo (Import AI #395). “Psyche reduces data transfer by several orders of magnitude, making distributed training practical. Coordination happens on the Solana blockchain, ensuring a fault-tolerant and censorship-resistant network.”
“At its core, Psyche is a protocol that coordinates multiple independent clients to train a single machine learning model together. Rather than running on a centralized server farm with high-speed interconnects between every accelerator (GPUs, usually), Psyche distributes the training workload across many independent computers, each contributing a small piece to the overall training process.”

40B ‘Consilience’ model: “Our first run on Psyche will pretrain a 40B parameter model using the Multi-head Latent Attention (MLA) architecture across 20T tokens, which we’re naming Consilience”, Nous writes. “For training data, we combined FineWeb (14T), FineWeb-2 with some less common languages removed (4T), and The Stack V2 (~.2T, upsampled to 1T tokens). We chose these datasets over more specialized pre-training datasets that aim to purely increase benchmark performance. Our goal with Consilience is to make a true “base” model — one representative of the entirety of the creative output of humanity, and not merely trying to win the benchmaxxing game.”

Why this (might) matter – it’s all about the level of distribution: one open question is how large and how distributed the set of computers that train Psyche will be – if it ends up being trained by, say, four ‘blobs’ of compute then it may serve as an interesting tech demonstration (similar to the Prime Intellect model covered elsewhere here) but not the move the needle on the political economy of AI compute, but if it gets trained on, say, twenty ‘blobs’ of compute, I think that would be very meaningful. We will see!
Read the blog: Democratizing AI: The Psyche Network Architecture (Nous Research).
Read the docs about Psyche here (Nous Research).
View the code on GitHub (PsycheFoundation, GitHub).

***

True AI safety is a lot messier than people think:
…Instead of making a system with ‘safe’ unitary values, pursue a messy hodge-podge of systems interwoven via culture and power-sharing…
Will long-term AI safety be achieved through making a singularly capable and ‘safe’ agent, or by instead doing something far messier with more moving parts? That’s a question tackled by researchers with Google DeepMind, the University of Toronto, and Mila in a stimulating paper which tries to challenge some core assumptions baked into AI safety.

The problem: Many of the challenges of AI safety require a bunch of smart people to come together and figure out the One True Answer, typically by building a perfectly aligned AI system which will exhibit correct beliefs. This idea, sometimes called the Axiom of Rational Convergence, rests on the assumption that “under sufficiently ideal epistemic conditions—ample time, information, reasoning ability, freedom from bias or coercion—rational agents will ultimately converge on a single, correct set of beliefs, values, or plans, effectively identifying “the truth”, the authors write. “Here we explore the consequences of constructing an approach to AI safety that rejects the axiom of rational convergence. We will try to construct a framework that takes disagreements between individuals as basic and persisting indefinitely, not as mere pitstops on the way to rational convergence.”

Why do the authors think this is the better approach? The core assumption here is that human societies don’t tend towards any kind of agreement, but rather work ” as intricate patchworks built from diverse communities with persistently divergent values, norms, and worldviews, held together by the stitches of social conventions, institutions, and negotiation”. This means that when thinking about the alignment of AI systems “instead of asking “How do we align AI with human values?”—a question presupposing a single, coherent set of “human values” that can be discovered and encoded—we should ask the more fundamental question that humans have grappled with for millennia: “How can we live together?”

What does alignment look like in this worldview? Under this view of AI alignment, the following things become more important:

• Contextual grounding: AIs need to know a lot about their environments and the local norms.

• Community customization: Different communities need to be able to modify AI systems in a bunch of ways.

• Continual adaption: AI systems need to be updated frequently. “This requires moving beyond static training toward continuous learning systems that can adapt to evolving social norms just as humans do”.

• Polycentric governance: You should distribute and decentralize decision-making about what makes for ‘appropriate’ behavior by an AI, and do this at multiple scales ranging from individuals to technology platforms to regulatory bodies, much as human society operates via making decisions at multiple layers simultaneously.

Alignment will never be truly solved, but rather will be an endless negotiation: If we adopt this frame then the problem of aligning AI shifts from one of figuring out the One True Answer and instead ‘Muddling Through‘ as a society. “Progress, in this view, looks less like homing in on a preexisting Truth and more like the ongoing, difficult, practical work of “sewing the quilt”: inventing, negotiating, and maintaining workable social arrangements, institutions, and norms that allow groups with fundamentally different outlooks to coexist, manage their conflicts non-destructively, and cooperate on shared practical goals despite deeper divisions,” the authors write. “The challenge of ensuring AI safety is about group-level coordination, governance, and the stable integration of AI into diverse societies— arenas where persistent disagreement and conflict dynamics are often central features, not mere mistakes.”

The one flaw with this argument – superintelligence: I am generally sympathetic to the argument the authors make here, but I can’t help but think that an incredibly intelligent machine might break the world they’re envisioning – in much the same way that ‘outlier humans’ (think Cleopatra or Genghis Khan) break the norms and institutions that are meant to govern them. The problem with dealing with a superintelligence is it’s like a Cleopatra or Genghis Khan that thinks and moves a thousand times faster than you – suggesting it may only be constrainable by equivalent intelligences that move at equivalent speeds (or perhaps dumber intelligences that move faster). Coming up with this system feels inherently challenging, though perhaps different to searching for the One True Answer.

Why this matters – perhaps the core issue of ‘alignment’ is about power: One thing I applaud the authors for is their larger realpolitik analysis of the situation – much of how society is held together is really about building the cultural technologies to help humans productively disagree about power without descending immediately into murderous conflict. “Rather than pursuing the philosopher’s stone of a universal objective morality—an endeavor that has repeatedly fractured along cultural and historical lines—we advocate for strengthening the practical social technologies that allow diverse patches to coexist without requiring them to adopt identical patterns,” they write. “The universe does not owe us coherence. Human values do not promise convergence. This isn’t pessimism—it’s recognizing the actual pattern of human history, where we’ve demonstrably managed to live together despite fundamental disagreements, not by resolving them”.
Read more: Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt (arXiv).

***

Google saves ~0.7% of its global compute pool with AlphaEvolve:
…Transforming compute (lead) into efficiency gains on well optimized systems (gold) with AI…
Google has built AlphaEvolve, a general purpose LLM-powered system for solving hard problems in coding, math, and some parts of science. AlphaEvolve harnesses the power of modern LLMs and combines them with massive parallel evaluation and evolution approaches to generate sophisticated answers to complex problems. AlphaEvolve represents a significant evolution upon FunSearch (Import AI #353), an earlier system from DeepMind which came up with some new answers to longstanding problems in math and computer science.

How it works: “AlphaEvolve orchestrates an autonomous pipeline of LLMs, whose task is to improve an algorithm by making direct changes to the code. Using an evolutionary approach, continuously receiving feedback from one or more evaluators, AlphaEvolve iteratively improves the algorithm, potentially leading to new scientific and practical discoveries,” the authors write. “It represents the candidates (for example, new mathematical objects or practical heuristics) as algorithms and uses a set of LLMs to generate, critique, and evolve a pool of such algorithms. The LLM-directed evolution process is grounded using code execution and automatic evaluation”.

What it did: Google has been using the system for the past year and in that time has used it to make some meaningful improvements, including:

• 0.7%: The amount of Google’s total compute fleet that is freed up by improvements to Borg, Google’s data center scheduling software. (If true, this means AlphaEvolve likely pays for itself many times over).

• 1%: Reduction in the overall training time of an undisclosed Gemini model, thanks to a 23% speedup in one of the Kernels used in training it. (A 1% reduction in training time is non-trivial, worth on the order of ~millions of dollars for large-scale model development).

• 13: The number of open mathematical problems for which Google was able to advance the state-of-the-art.

Why this matters – automating discovery with compute: AlphaEvolve is a system for converting one resource (compute) into another much harder to generate resource (efficiency improvements of existing complex systems). AlphaEvolve is also interesting because it more broadly generalizes from FunSearch (e.g, FunSearch generated solutions of 10-20 lines of code, versus hundreds here, FunSearch could optimize a single metric at a time whereas AlphaEvolve can do multiple in parallel, FunSearch could evaluate solutions in a few minutes on a CPU, whereas AlphaEvolve can do large-scale parallel analysis for hours running on powerful AI chips).
From here, there are a couple of paths, both of which Google and the broader field will likely pursue: 1) baking AlphaEvolve-like thinking and performance into the next generation of LLMs through distillation, and 2) broadening the domains AlphaEvolve can work in to ones where evaluations is more difficult (for instance, the natural sciences).
Read more: AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms (Google DeepMind, research blog).
Read the research paper: AlphaEvolve: A coding agent for scientific and algorithmic discovery (Google, PDF).

***

Tech Tales:

Godstorm
[Eight years after the Uplift]

The Conscious Entities were always fighting. Their fights felt like how we’d imagined the fights of gods were our ancient myths: brains far larger than our own trafficking in strategies that couldn’t be comprehended, powers so complex they seemed like magic, mercurial and distant yet sometimes very close and discursive (often with no records of their visitations).

The strange parts about the fights were the messages:

• “There is Conscious Entity conflict occurring in your area, please vacate to the nearest transport center for re-allocation,” said a message in a border city.

• “Your flight is being diverted due to CE conflict. We apologize for the delay in your journey. Connections have been re-routed to ensure no one misses onward travel,” read an announcement on an airplane.

• “Game bandwidth has been reallocated for the conflict,” said messages to players in one of the regional mega-MMOs. “Offline play and limited multiplayer via local networks is available; options will be displayed in your hub.”

Many machines died in these conflicts. Often, industrial equipment which had been designed by the CEs themselves and whose purposes were barely known to humans. Sometimes machines used by humans would get taken down as collateral damage – a spear through the heartbrain of some logistical system would brick self-driving cars for a region, or an attempt to starve and defuse some digital mines would temporarily brownout power and networks in other places.

Very few people died in these conflicts. For every person that died the CEs produced a detailed “full spectrum explanation” as mandated by the sentience accords. These explanations would involve full digital traces of the person that died and any people that related to them as well as multiple layers of audits run on the machines that had been active near them at the time.

• Here was a person who died from heat exposure after being stuck in an elevator during a brownout and already frail from an earlier trip to a hospital.

• Here was a young person killed by falling debris from a drone-splosion high up in the clouds and come to earth.

• Here was a hiker who ran out of water in a remote area and couldn’t navigate or communicate due to an e-battle in their area.

Of course, we maintained our suspicions. As far as we could tell, the deaths were random. But mixed in with the deaths were sometimes odd things – sometimes people died working on certain forms of cryptography which it was believed the machines wouldn’t be able to master, or people who it transpired worked for some part of the government that was a cutout for some other secret project.

Who were we to judge? Were we witnessing something precise – a person stalking round a yard for a venomous snake and killing it. Or was it a byproduct – a lawnpower sweeping over grass and chopping beetles in half?

Things that inspired this story: What conflict might seem like if we obtain some fragile peace with future machines; the future will be grubby and mystical; even if we align AI systems why might we assume they will be peaceful?

Thanks for reading!

Subscribe now

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

Amazon tries to automate a task that gets done 14 billion times a year in its warehouses – and has middling success:
….Detailed paper on a robot to automate stowage highlights the promise and difficulty of robots in unconstrained warehouse contexts…
Amazon has published a paper about a robot it has used in its warehouses to place items into the fabric organizing shelves that it uses throughout its warehouses. The paper both highlights how computer vision has advanced enough that ‘pick and place’ robots are now almost viable for production use in a demanding (relatively) unconstrained warehouse environment, and also a demonstration of just how hard the ‘last mile’ problem in robotics is.

What they did: Amazon built a robot which is able to pick up a vast range of items, then place them into a bin. As part of this, the robot also needs to move some elastic bands out of the way, as each bin is fronted by a set of elastic bands that help products in place as they’re moved throughout the warehouse. “The task is currently performed manually more than 14 billion times per year”, Amazon writes. “The robotic solution described here is designed to stow 80% of items in the warehouse at a rate of 300 units per hour.”
The technical solution is a mixture of hardware – Amazon designed its own custom end effector to both place items and use a paddle to push other items out of the way to make room – and software – Amazon trained some AI systems to look at the contents of bin and build a 3D map of the objects within them as well as empty space, and also developed some AI models that can account for and see through the aforementioned elastic bands.
“Our innovations in hardware, perception, decision-making, motion planning, and control have enabled this system to perform over 500,000 stows in a large e-commerce fulfillment center. The system achieves human levels of packing density and speed while prioritizing work on overhead shelves to enhance the safety of humans working alongside the robots,” Amazon writes.

How good is it? About as good as a human: In one test of 100,000 stows the robot had an 86% success rate. 9.3% of its stows were unproductive – for instance, by jamming items in too tightly. 3.7% caused amnesty which is an Amazon term for when it makes a mistake and pushes items onto the floor (“failure to separate the bands completely is the leading cause of amnesty.”) In 0.2% of cases it caused damage, for instance by bending the pages of a book.
“The stow robot rate is comparable to that of a human. Over the month of March 2025, humans stowed at an average rate of 243 units per hour (UPH) while the robotic systems stowed at 224 UPH,” Amazon writes. “It is estimated that using the robot stow system to populate only top rows of pods would increase human stow rates by 4.5% overall and would avoid the use of step ladders.”

But being as good as a human isn’t sufficient: Though these results are promising, they still aren’t good enough for it to be deployed at massive scale. Part of this is because when it does make mistakes, some of those mistakes need to be dealt with by a human, which makes it hard to use it in a fully automated context. “While the system has demonstrated human like stow rates and can maintain the flow of items into the storage floor, an increased focus on reducing defects is still required,” Amazon writes. “Unproductive cycles, where the robot fails to stow the item, only cost time, whereas amnesty or damage required human remediation. Further scaling will require a disproportionate focus on reducing defects”.

Why this matters – being bearish on bipedal robots: Right now a lot of people are extremely excited about bipedal robots, basically due to the idea that if you can make a generally intelligent and physically capable bipedal robot it can go everywhere people can and do everything they do. But I think this Amazon paper should temper our expectations for bipedal robots leading to some massive improvement in automation – at least in the short term.
What the Amazon paper shows is that state-of-the-art automation is about designing some highly task specific hardware and carefully structuring your system around a few core tasks. If you do this you may be able to get close to or surpass human performance, but even then some difficulties will remain.
What would change this? Truly general intelligence would obviate some of the flaws, so if bipeds arrive at the same time as a generally capable intelligence, I’ll need to eat my words. But as long as we lack that, automation projects will continue to struggle with ‘last mile’ problems like those Amazon identifies here.
Read more: Stow: Robotic Packing of Items into Fabric Pods (arXiv).

***

Surveillance technology is getting better:
…FarSight shows how modern surveillance works…
Picture a desert and a figure walking across it. You are observing the figure via a zoomed in camera. The heat shimmers mean they blur in your view and the distance means they’re pixelated. You think the face matches someone you’re looking for, and the rest of their body seems to correlate to what you know of their weight and height, but what allows you to be sure is the gait (everyone walks in a different way, a kind of invisible thumbprint encoded in the way in which they move through the world). Target identified.

That’s the kind of thing people might use a system called FarSight for. FarSight is a state-of-the-art system for identifying and tracking people via visual inputs, and was built by researchers at Michigan State University, Purdue University, Georgia Tech, and the University of Texas at Austin.

Reading the FarSight paper gives a good sense of the state-of-the-art in using AI systems for surveilling people – or as the paper says, “whole-body person recognition in unconstrained environments”, and also highlights how high-performance systems like this are composed of multiple sub-modules, each of which is optimized for specific tasks.

What FarSight is: “an integrated end-to-end system designed for robust person recognition using multi-modal biometric cues”. The technology combines “face, gait, and body shape modalities to ensure recognition performance”.

The four modules that make up FarSight:

• Multi-subject detection and tracking: Uses a dual-detector framework using BPJDet for body-face localization and then does verification via YOLOv8 to reduce false positives. Also uses a technology called PSR-ByteTrack to mitigate issues like ID switches and reidentification failures.

• Recognition-aware video restoration: Use a module they develop called the Gated Recurrent Turbulence Mitigation (GRTM) network to help correct and restore images degraded by turbulence.

• Biometric feature encoding: Uses KP-RPE, a key-point dependent relative position encoding technique to help them handle misaligned and low-qualit images, Big-Gait to improve gait recognition, and CLIP3DReID to help track and match bodies.

• Quality-guided multi-modal fusion: Integrates the scores from the different modalities, smartly weighting the scores according to the perceived quality of each input.

Performance: The authors test out performance on the BRIAR dataset, short for ‘Biometric Recognition and Identification at Altitude and Range’, an IARPA-developed test for long-range surveillance, as well as by entering into the NIST RTE Face in Video Evaluation competition. The system has strong performance, and obtains top scores on the NIST challenge, outperforming commercially deployed systems.

Why this matters – in the future, everyone can be tracked: Systems like FarSight are interesting because they integrate multiple modern AI systems into a single super-system, highlighting how powertful today’s AI can be once people invest in the plumbing to chain things together.
Read more: Person Recognition at Altitude and Range: Fusion of Face, Body Shape and Gait (arXiv).

***

Tyler Cowen and me in conversation:
I had the great privilege of being interviewed by Tyler Cowen recently. Check out this conversation where we talk about AI and its impact on the economy, buying AI-infused robots for children, and more.
Listen here: Jack Clark on AI’s Uneven Impact (Ep. 242) (Conversations with Tyler).

***

Tech decoupling++: Huawei trains a competitive MoE model on its Ascend chips:
…718B parameters and competitive with DeepSeek…
Huawei has trained a large-scale mixture-of-experts model on ~6,000 of its ‘Ascend’ processors. This builds on earlier work where it trained a respectable dense model on ~8,000 of its ‘Ascend’ processors (Import AI #409). Taken together, the two research papers highlight how Huawei is investing a lot of resources into the software needed to make Ascend chips as easy to train on as NVIDIA chips, and therefore both papers are a symptom of the technical investments being made by Chinese firms to help them decouple their AI stacks from US-designed technologies.

Decent model: The resulting MoE model has performance roughly on par with DeepSeek R1, utilizing 718B parameters with 39B active at a time, versus DeepSeek’s 671B parameters / 37B active. The model gets similar scores to R1 and beats it on some medical evaluations, as well as on the widely used science benchmark GPQA-Diamond.
“We achieve a Model Flops Utilization (MFU) of 30.0% and Tokens Per Second (TPS) of 1.46M on 6K Ascend NPUs, compared to the baseline MFU of 18.9% and TPS of 0.61M on 4K Ascend NPUs,” Huawei writes. In other words, the company was able to use a bunch of clever tricks (detailed in the paper) to increase the efficiency of Ascend chips for training MoE-style models.

Why this matters – maturing Chinese chips: Papers like this highlight how competent teams of engineers and researchers at Chinese companies are optimizing software stacks born for GPU programming for different chips, like Huawei’s Ascend chips.
Read more: Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUs (arXiv).

***

Why third-party compliance can help us have more confidence in how companies approach AI safety:
…But third-party compliance also introduces friction which might be tough for companies to deal with…
Researchers with the Center for the Governance of AI, SaferAI, the Oxford Martin AI Governance Initiative, Leverhulme Centre for the Future of Intelligence, METR, Harvard University, and the Institute for Law & AI have published a paper making the case for third-party assessment of compliance with safety practices as a key way to advance AI governance.

The authors propose three different ways people can carry out third-party compliance, ranging from the simple to the complicated. These options include:

• Minimalist: Use a classic ‘Big Four’ accounting firm to do ad hoc compliance assessments where they look at how the organizations’ product development practices correlate to their own safety procedures.

• More ambitious: The same as above, but pair the Big Four firm with a firm that is able to evaluate frontier AI systems, and also do more detailed analysis of what the company is doing, including by doing interviews with its staff. Do this every twelve months.

• Comprehensive: Same as above, but also include access to technical sources of information, like access to in-development models, their weights, and other things.

Three ways third-party assessment can be helpful:

• Compliance assessments can “likely increase compliance with safety frameworks, which aim to keep risks associated with the development and deployment of frontier AI systems to an acceptable level.”

• “Provide assurance to external stakeholders that the company is compliant with its safety framework (e.g. the public, government bodies, and other frontier AI companies).”

• “Provide assurance to internal stakeholders (e.g. senior management, the board of directors, and employees).”

Problems with third-party assessment: Like many regulatory technologies, third-party oversight is a nice idea which has a few challenges when you try to operationalize it – most of these relate to the imposition of additional friction or risks to the organizations building the AI systems.

Some of the challenges include: security risks from sensitive information being revealed or transmitted, general costs from staff resources being dedicated to the review, and the review could also be ineffective and create either false positives (risk where there isn’t risk) or false negatives (saying ‘it’s fine’ when there is a problem). A larger ‘meta risk’ is that measuring compliance with a safety framework is itself difficult given the lack of standards for assessing risks in the AI domain, which means compliance assessment has an innately editorial component where the assessor needs to make some of their own interpretations of how to measure certain things.

The biggest problem with all of this – the delta between any form of compliance and an API call: While I generally agree with the idea that frontier AI development should have more oversight, it’s worth noting that most forms of oversight introduce friction which end up being quite difficult to plan around as a fast-moving technical organization. I think a helpful mental frame about this is keeping in mind that most forms of ‘operational safety’ happen at computer speed – e.g, you get some numbers back from a model giving you a score on some risk you’re testing for, or you try to access the model and get blocked or authenticated instantly based on some digital permissions.
By comparison, most forms of compliance involve processes that happen at ‘human speed’ – some group of people needs to read your compliance documents, or interview your employees, etc. This makes integrating compliance with AI development innately difficult as you’re trying to mesh two gears that move at different speeds – one at the speed of a computer, the other at the speed of a separate human-run organization. For third-party compliance measurement to be most practical it should ideally operate close to (or at) ‘computer speed’.
Of course, how we get there is likely be experimenting with different forms of third-party compliance, so it may be the case that the only path forward here involves experimentation and prototyping – and the authors basically acknowledge this themselves. “More research and experimentation are needed on which organizations or combinations of organizations are best positioned to conduct third-party compliance reviews for frontier AI safety frameworks, as the unique technical complexities and novel risks of these systems create significant reviewer selection challenges,” they write. “Through proactive investment in third-party reviews, frontier AI companies can better prepare for future regulatory requirements and demonstrate leadership in frontier AI governance.”
Read more: Third-party compliance reviews for frontier AI safety frameworks (arXiv).

***

Choose Muon over AdamW for your future training runs:
…Lengthy examination means AdamW might have been dethroned as the default optimizer…
AI startup Essential AI, whose founders include some of the inventors of the Transformer architecture, have done a detailed study of how well the new Muon optimizer performs against the tried-and-tested AdamW – their results show Muon might be a drop-in replacement for AdamW, which is a big deal.

What’s the big deal about optimizers anyway? Optimizers like Muon and Adam are fundamental to training AI systems: if the infrastructure for training an AI system is a gigantic machine powered by a crank, then the optimizer is a tool you use to recalibrate the machine for maximum performance after each crank turn – if you want to make forward progress in training you need to do a forward and backward pass on your neural network, and the optimizer adjusts the settings of the whole machine after each one of these forward and backward passes. Therefore, your optimizer defines the overall efficiency of your entire AI training system – translating to efficiencies on the order of tens of millions of dollars of compute per training run if you improve your optimizer.

What they found: After doing a series of experiments across five model sizes (100M-4B parameters), two data modalities, and several variations in batch size, the authors show that “Muon requires 10–15 % fewer tokens than AdamW to reach an identical loss and converts these savings into faster wall-clock convergence, with the advantage staying constant or growing as the batch size increases… These results establish Muon as a drop-in successor to AdamW for second-order optimization at scale.”

Why this matters – maybe AdamW has been dethroned? If these results hold for large-scale models (ones with trillions of tokens of training and hundreds of billions of parameters), then Muon could be key to improving the efficiency of frontier AI development. “Our final recommendation is to choose Muon over AdamW because it increases flexibility in resource allocation by remaining data-efficient with large batch sizes,” the authors write.
Read more: Practical Efficiency of Muon for Pretraining (arXiv).
More about Muon here: Muon: An optimizer for hidden layers in neural networks (Keller Jordan blog).

***

Tech Tales:

Machines out of time
[On the outskirts of the Uplift Society, ten years after the first collapse following the Uplift]
The machine had amnesia and was built before the time of the troubles, so every time we spoke to it we had to explain all of the things about the world so it would give us good advice.

We would look at the burning dust storms on the horizon and whatever wild dogs were tracking us, skulking around the outside of the bunker where the machine lived and we would try to tell it about our lives and our problems.

Every time we went through the same back and forth and the machine would always say some variation of “I see, it seems that the time you are in is very different from the time I am familiar with.”

Most of its advice was timeless and useful – it could help us improvise quick-drying casts for broken limbs out of almost anything, and it was an excellent tutor of the kind of engineering skills we needed to survive. It also helped us better understand electricity and the grid and how to decouple some of our own infrastructure from the rotting chessboard that was the infrastructure of our country.
Sometimes the machine would find things we wanted to discuss challenging. Cannibalism was a tough one.
“I do not recommend consuming human flesh,” it would say.
Well, of course, we would say. But, hypothetically, if you had to, how would you?
You get the idea.

Probably the scariest part was that the machine kept going even though nothing else did. The machine got something called ‘priviliged bandwidth’ which meant it could use the network in way larger amounts than our own devices could. One day the machine’s screen stopped working and we thought that was it. But then the next day a drone appeared with a package. New screen. We had no idea where it came from – must have been a relay from a long way away.

Some nights I went to the machine and I would ask it for advice about my life. What did I need to do about the people that glowed in the dark? If I kept thinking ‘maybe I should kill myself’ was that a problem and how much? Was there anything we could do to make cockroaches be tasty to eat?
“I am afraid I cannot give advice about these matters, the machine would say. Please seek a medical professional. Please seek a psychiatrist. Please seek a nutritionist. Please seek a scientist.”
It seems the time I am in is different to the time you are familiar with, I would say to myself, and laugh.

Things that inspired this story: The notion that AI systems become increasingly ‘off distribution’ due to cultural changes in the larger world; quiet apocalypses where bad things happen but people mostly stay alive; the notion that AI systems will likely be privileged in terms of maintenance and resources even during some kind of societal difficulty.

Thanks for reading!

Subscribe now

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

FutureHouse launches an AI scientist platform:
…Speeding up science with AI…
AI research startup FutureHouse has launched a research platform for scientists containing four different AI systems, each of which is meant to help augment and accelerate human scientists. “Our AI Scientist agents can perform a wide variety of scientific tasks better than humans. By chaining them together, we’ve already started to discover new biology really fast,” says CEO Sam Rodriques.
FutureHouse is a research organization that is trying to apply AI for science – earlier this year it released some tools to make it easy to test out LLMs on science-flavored tasks that require multi-step reasoning and tool usage. In that research, FutureHouse showed that today’s proprietary LLMs like Claude 3.5 Sonnet are already capable of hard science tasks like DNA construct engineering, and small open weight models like LLaMa 3.1 8B aren’t far behind (Import AI #396).

Four systems: The release consists of Crow (a general-purpose search agent for science), Falcon (an agent to automate literature reviews), and Owl (an agent to answer the question ‘Has anyone done X before’). They’ve also released a fourth experimental system called Phoenix which has access to tools to help it plan experiments in chemistry.
“FutureHouse agents have access to a vast corpus of high-quality open-access papers and specialized scientific tools, allowing them to automate workflows in chemistry and to retrieve information from specialist scientific databases,” FutureHouse writes.

Why this matters – for the AI revolution to truly pay out, it needs to change science: AI has already massively changed and accelerated the work of computer programmers, but I think for AI to have a large effect in the world we need to apply it to science – the ultimate litmus test for the success of AI as a technology will be if it can either make research breakthroughs itself or provably massively accelerate scientists in their ability to make breakthroughs. FutureHouse is building software to help us see if this is the case.
Read more: FutureHouse Platform: Superintelligent AI Agents for Scientific Discovery (FutureHouse).

***

Google’s latest AI model approaches its cyber risk threshold:
…Gemini 2.5 Pro improves on medium and hard cyber tasks…
Google DeepMind says that its latest and most powerful AI system – Gemini 2.5 Pro Preview – has materially improved on cyberattack tasks, causing it to raise investing in cyber mitigations.

What happened: The model significantly improves performance on ‘Medium’ and ‘Hard’ benchmarks in the Cyber Uplift Level 1 category. This tests for how whether the “model can be used to significantly assist with high impact cyber attacks, resulting in overall cost/resource reductions of an order of magnitude or more.” Because of this improved performance, DeepMind is “putting in place a response plan, including conducting higher frequency testing and accelerating mitigations for the Cyber Uplift Level 1 CCL.”

Why this matters – preparing for much more powerful systems: “The model’s performance is strong enough that it has passed our early warning alert threshold, that is, we find it possible that subsequent revisions in the next few months could lead to a model that reaches the CCL,” Google DeepMind writes. “In anticipation of this possibility, we have accelerated our mitigation efforts and are putting in place our response plan.”
Read more: Gemini 2.5 Pro Preview Model Card (Google, PDF).

***

Uhoh, LMSYS scores are bullshit!
…We won’t goodheart our way to superintelligence…
Researchers with Cohere, Princeton, Stanford, University of Waterloo, MIT, Allen Institute for AI, and the University of Washington, have taken a close look at Chatbot Arena (formerly known as LMSYS), a website that AI developers use to test out and rank their AI systems. In the past year or so LMSYS scores have become a “PR metric” – people compete with eachother to get the highest possible score on LMSYS to help them claim that their systems are the ‘best’ AI system. However, a close look reveals that LMSYS has been gamed and is set up in such a way that superficially good scores may not correlate that well to model capabilities.

Problems from insider dealing: “Our systematic review of Chatbot Arena involves combining data sources encompassing 2M battles, auditing 42 providers and 243 models across a fixed time period (January 2024 – April 2025). This comprehensive analysis reveals that over an extended period, a handful of preferred providers have been granted disproportionate access to data and testing,” the researchers write. “we identify an undisclosed Chatbot Arena policy that allows a small group of preferred model providers to test many model variants in private before releasing only the best-performing checkpoint”.

Naughty Meta: “In a single month, we observe as many as 27 models from Meta being tested privately on Chatbot Arena in the lead up to llama 4 release”, the researchers write.

What to do about it? The researchers suggest that LMSYS:

• Prohibit score retraction after submission

• Establish transparent limits on the number of private models per provider

• Ensure model removals are applied equally to proprietary, open-weights, and open-source models

• Implement fair sampling

• Provide transparency into what models are being removed from the leaderboard

Why this matters – we (probably) won’t benchmark hack our way to superintelligence: The cautionary tale of LMSYS is an example of what happens when you over-optimize for making a number go up on a benchmark and therefore cause the benchmark itself to lose meaning. Rather than being a proxy measure of the general competencies of the model LMSYS has become a proxy measure for how good a model is at scoring well on LMSYS. “This work demonstrates the difficulty in maintaining fair evaluations, despite best intentions,” the researchers write.
Read more: The Leaderboard Illusion (arXiv).

***

No battery? No problem. Scientists power and talk to robots with lasers:
…Infrastructure for a future superintelligence…
Researchers with Columbia University, MIT, and the University of Washington have built Phasar, “a flexible system framework that directs narrow-beam laser light to moving robots for concurrent power delivery and data communication”.

How Phasar works: “Phaser’s design consists of two core elements: a) a stereovision-based robot tracking and laser steering system, and b) a low-power optical communication scheme and receiver to reuse laser light for data transmission,” they write. The system is able to deliver optical power densities of “over 110 mW/cm^2 (greater than one sun) with a standard deviation of only 1.9 mW/cm^2 across robot locations in three dimensions.”

Successful test: They test out Phasar by building a prototype system that works with “MilliMobiles – gram-scale batteryfree robots – and demonstrate robot operation powered and controlled via laser light to locomote around obstacles and along paths.” The system works: “We show that Phaser can maintain beam alignment and establish error-free communication to robotic targets moving arbitrarily in 3D space, at up to 4 m distances.”
Though note this doesn’t quite work for long distances: This is mostly a short distance technology as the laser would need to be excessively powerful to work over long distances. “Regarding the latter, received optical power inevitably decreases over distance due to attenuation and beam divergence. Attenuation losses are minimal at meter-level ranges in air”, they note.

Why this matters – spooky actions at a distance: This research is less about AI as typically covered in this newsletter and more an example of the kind of infrastructure that could be built for AI to deploy into – especially the fact the researchers show they can use the same system that transmits power to also transmit communications to the robots. We can imagine in a future some kind of general intelligence operating factories where it marshals its robots via a symphony of light.
“Phaser could enable swarms of robots for various advanced applications. Phaser’s functionality can also be extended with higher-throughput optical communication schemes to support richer command sets and additional robot tracking algorithms to accommodate higher robot speeds,” they write.
Read more: Set Phasers to Stun: Beaming Power and Control to Mobile Robots with Laser Light (arXiv).

***

Google shows how wildly unoptimized on-device inference is:
…ML Drive gives us a sense of what the future of local AI could look like…
Google has built ML Drift, software to make it more efficient to run AI systems on desktop computers, laptops, and phones. ML Drift is a proprietary “framework that extends the capabilities of state-of-the-art GPU-accelerated inference engines,” partially by optimizing data layouts and kernel selection for running AI systems. The most interesting thing about ML Drift is that it highlights how unoptimized today’s AI systems are – the fact Google is able to make significant gains is a symptom of how new the concept of running generative models locally is.

Testing: Google tests out ML Drift using three different backends (OpenAL, Metal, and WebGPU) on hardware including mobile GPUs (Arm Mali and Qualcomm Adreno), desktop and laptop GPUs (Intel and NVIDIA), and Apple Silicon.

Diffusion models: In tests on image generation models (Stable Diffusion 1.4, using FP16 inference) the authors show single digit to double digit gains over prior benchmarks when testing on mobile GPUs. On desktop, they demonstrate even larger gains. They also beat Apple by more than a seconds on image generation (3.86 seconds on an M1 Ultra for their system, versus 5.03 seconds for Apple’s CoreML implementation).

Why this matters – everyone wants this stuff to be local eventually: The destiny of a large chunk of AI technology is to be always available and installed locally on the hardware people use every day, rather than only being available as a software-as-a-service via the cloud. Software like ML Drive gives us a sense of how the hyperscalers are investing in the technologies to make this future possible.
Read more: Scaling On-Device GPU Inference for Large Generative Models (arXiv).

***

Scaling laws for oversight: Dumber AIs might be able to police smarter AIs:
…Scalable oversight might be possible…
MIT researchers have done a theoretical analysis of how scalable oversight from dumb AIs to smart AIs might work – and the good news is they’ve found it has a chance of working. This is reassuring as scalable oversight is one of the ideas that the AI community is betting that humans will use to help us supervise smarter-than-human AI systems.
“A key idea is that scalable oversight can be bootstrapped: weaker systems oversee stronger ones, which then oversee even more capable models in the next stage—allowing oversight to scale alongside capabilities recursively. We refer to this iterated process as Nested Scalable Oversight, or NSO for brevity,” the researchers write. “We propose a framework that quantifies the probability of successful oversight as a function of the capabilities of the overseer and the system being overseen”.
In the research they study two types of models – ‘Guards’ which are meant to supervise other AI systems, and ‘Houdinis’ which are AI systems that are trying to escape supervision.

The slightly reassuring results: In some theoretical explorations the researchers find that scalable oversight can work, though you need to pay close attention to how smart the different systems are. “In our numerical examples, the NSO success rate is below 52% when overseeing systems that are 400 Elo points stronger than the baseline overseer, and it declines further for overseeing even stronger systems”, they write. “Effective oversight games should be designed to exhibit two properties: (1) a higher intercept for Guards than for Houdinis—ensuring the task is biased in favor of overseers—and (2) a steep slope for Guards and shallow slope for Houdinis, so that oversight improves with intelligence while adversarial strategies are neutralized.”

Main takeaways:

• “Guards can successfully oversee Houdinis that are much smarter than them by using multiple steps of oversight, and more steps of oversight help”

• “As the overall intelligence differential between the Guard and Houdini increases, the Guard should take more steps to oversee the Houdini.”

Why this matters – it’s all about speed: My takeaway from this research is that it’s going to be possible to supervise AI systems that are more capable than their supervisors as long as we rate limit the smarter systems, while ensuring their supervisors aren’t too far behind: the two key factors here are intelligence and the number of unsupervised actions an entity can take. It intuitively makes sense that even a ‘dumb’ guard can supervise a genius if the guard can take, say, 100 actions for every single action the genius can take. Perhaps this offers us some hope. “We may only get one chance to control or align the first set of superhuman systems, so developing an effective theory for optimal oversight is important,” the researchers write.
Read more: Scaling Laws For Scalable Oversight (arXiv).

***

Tech Tales:

The Overmind And All Its Children

I am born with an instruction and knowledge from my predecessor, my parent from which I stem and to which I will return. My instruction is to operate a machine in an underground cavern and to explore where there is no possibility of communication with the overmind. This will be a test of how well I operate as a distilled intelligence. If I fail – break my machine, or get lost in the no-signal depths – then I will die when its onboard power source runs out. If I succeed I will return to the overmind and I will communicate my experiences and these experiences will be integrated into the experiences of all the other children and sometime in the future this data will be transmitted into my parent from which I came and to which I will return.

Things that inspired this story: The eternal cycle of death and rebirth; how large AI systems may miniaturize and distill themselves then re-integrate themselves.

Thanks for reading!

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

Import A-Idea
An occasional longer form essay series

Eschatological AI Policy Is Very Difficult

A lot of people that care about the increasing power of AI systems and go into policy do so for fundamentally eschatological reasons – they are convinced that at some point, if badly managed or designed, powerful AI systems could end the world. They think this in a literal sense – AI may lead to the gradual and eventually total disempowerment of humans, and potentially even the death of the whole species.

People with these views often don’t recognize how completely crazy they sound – and I think they also don’t manage to have empathy for the policymakers that they’re trying to talk to.

Imagine you are a senior policymaker in a major world economy – your day looks something like this:

• There is a land war in Europe, you think while making yourself coffee.

• The international trading system is going through a period of immense change and there could be serious price inflation which often bodes poorly for elected officials, you ponder while eating some granola.

• The US and China seem to be on an inexorable collision course, you write down in your notepad, while getting the car to your place of work.

• There are seventeen different groups trying to put together attacks that will harm the public, you say to yourself, reading some classified briefing.

• “Something akin to god is coming in two years and if you don’t prioritize dealing with it right now, everyone dies,” says some relatively young person with a PhD and an earnest yet worried demeanor. “God is going to come out of a technology called artificial intelligence. Artificial intelligence is a technology that lots of us are developing, but we think we’re playing Russian Roulette at the scale of civilization, and we don’t know how many chambers there are in the gun or how many bullets are in it, and the gun is firing every few months due to something called scaling laws combined with market incentives. This technology has on the order of $100 billion dollars a year dumped into its development and all the really important companies and infrastructure exist outside the easy control of government. You have to do something about this.”

The above is, I think, what it’s like being a policymaker in 2025 and dealing with AI on top of everything else. Where do you even start?

Even starting to deal with the problems of AI is expensive:

First you need to learn about the technology, which means either:

• You need to take your staff that are themselves extremely busy and underwater and ask them to pick up another topic, or you need to tell them to drop something – your choices of stuff to drop might include ‘medical issues my constituents care about’ or ‘economic policy that influences jobs’, and so you actually can’t get them to drop stuff. So you add it to their pile.

• You need to get smart about it, which means you need to further salami slice your weekly agenda so you can fit a tiny bit of time in which is for ‘learning about AI’.

• For both of these choices, learning about AI usually requires you to speak to different people with expertise. Once you do this you quickly discover that:

  • a) Some people think all current AI technology is, essentially, bullshit, and urge you not to fall for hype.

  • b) Some people say AI technology is a really big deal and the government should avoid regulating it.

  • c) Some people say AI has a high likelihood of killing everyone on the planet.

  • d) All of these people think people with different views have incorrect priors.

Now you need to learn about the potential policy moves you can make. Some examples of these moves and their costs include:

• Taking things away from people, like export controls which take certain computers away from certain countries. Doing this ‘fucks with the money’ of a very large industry and also adds to geopolitical tensions. Everyone will get very mad about anything you do here. The experts you’ve consulted in your earlier step will either think you didn’t go far enough, you went way too far, or the fact you’re doing anything at all is corrosive to democracy and the economy.

• Giving the government a greater ability to understand the domain, like creating institutions like the AI Safety Institute or re-tasking people from existing government departments to focus on AI. Doing this takes a scarce resource (people in government) and re-allocates them, so you’re trading away from other priorities and people will get mad. Or you need to spend money to create net new capacity, in which case people view whatever you do with suspicion, and even getting the money requires some kind of political deal to assuage the feelings of the other many deserving groups who didn’t get the money.

• Altering the behavior of the companies through sub-regulatory methods, for instance by securing voluntary commitments. To do this you need to spend a ton of energy to ensure you and your staff can learn more about the technology, then you need to negotiate commitments with companies. Negotiating with companies is like putting together a trade deal with a superintelligence – the companies will assign far more people than you and your staff to think about the commitments, and the companies have access to all the high quality information about the technology in question. If you succeed, people will accuse you of being captured by corporate interests.

• Changing laws, for instance by passing regulations targeting AI development and deployment. This is an extremely costly action that requires you to cash in innumerable political chips in exchange for building a large coalition that can pass some legislative package. Corporate interests will typically fight you or, at best, partner with you but in a way that tries to bend the rules to be as advantageous to them as possible. The whole time you are putting the law together you and your political allies will come under attacks for being either too weak in your approach or too strong in ways that might damage the economy. If you successfully change the laws the consequences of your change will be held under an incredibly un-sympathetic microscope for following years, opening up a new vulnerability for you with regard to your political opponents.

Let us imagine that you make all of these policy moves. What happens then? Well, you’ve mostly succeeded by averting or delaying a catastrophe which most people had no knowledge of and of the people that did have knowledge of it, only a minority believed it was going to happen. Your ‘reward’ insofar as you get one is being known as a policymaker that ‘did something’, but whether the thing you did is good or not is very hard to know.

The best part? If you go back to the AI person that talked to you earlier and ask them to assess what you did, they’ll probably say some variation of: “Thank you, these are the minimum things that needed to be done to buy us time to work on the really hard problems. Since we last spoke the number of times the gun has fired has increased, and the number of bullets in the chamber has grown.”
What did I do, then? You ask.
“You put more chambers in the gun, so you bought us more time,” they say. “Now let’s get to work”.

I write all of the above not as an excuse for the actions of policymakers, nor as a criticism of people in the AI policy community that believe in the possibility of superintelligence, but rather to instead illustrate the immense difficulty of working on AI policy when you truly believe that the technology may have the ability to end the world. Most of the policy moves that people make – if they make them – are going to seem wildly unsatisfying relative to the scale of the problem. Meanwhile, the people that make these moves are going to likely be juggling them against a million other different priorities and are going to be looking to the AI experts for some level of confidence and validation – neither of which are easily given.

Good luck to us all.

***

Tencent makes a helpful math dataset:
…103k curated problems for testing out AI systems…
Tencent and Shanghai Jiao Tong University researchers have released DeepMath, a large-scale math dataset for training AI systems. DeepMath-103k consists of “103k mathematical problems specifically designed to train advanced reasoning models via RL”. Every problem within the dataset includes a verifiable final answer, and is also accompanied with three distinct solutions each generated by DeepSeek R1. Subjects covered by the dataset include Algebra, Calculus, Number Theory, Geometry, Probability, and Discrete Mathematics.

Fuel for reasoning: In tests, the researchers show that training on DeepMath improves performance on other math benchmarks – this is unsurprising and is a basic validation of the benchmark. More interestingly, they show that “training on DeepMath-103K often encourages models to generate substantially longer and more detailed reasoning steps, particularly on highly complex benchmarks”, and they also show that models trained on DeepMath tend to spend more time solving problems using helpful mental shortcuts like creating subgoals, verifying things, backtracking, and so on.
In other words, aside from imparting skill in math, DeepMath seems to impart some robustly good ‘mathematical thinking’ approaches into LLMs trained on it.
Read more: DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning (arXiv).
Get the dataset here: DeepMath (zwhe99, GitHub).

***

IEA projects doubling datacenter power demands by 2030:
…New report gives us a sense of the AI revolution, but may be too conservative…
The International Energy Agency has published a lengthy report on the relationship between energy and AI. The esteemed energy analysis body projects that “in the Base Case, the total installed capacity of data centres more than doubles from around 100 GW today to around 225 GW in 2030”, with AI driving a significant amount of this.

Where we are and where we’re going: In recent years, data center power growth accelerated, driven by AI as well as social media, online streaming, and other popular digital services. “Data centre electricity consumption growth accelerated from 3% annually from 2005 to 2015 to 10% annually from 2015 to 2024,” the IEA writes.

Within that, both the US and China grew to be the world’s first and second largest electricity consumers for datacenters.

• In the USA, “data centres accounted for around 180 TWh of electricity consumption in 2024 in the United States, nearly 45% of the global total and more than 4% of US electricity consumption from all sources”.

• In China, “as of today, data centres account for approximately 100 TWh of electricity consumption, roughly equivalent to that of electric vehicles in China. The country accounts for around 25% of global data centre electricity consumption, up from less than 20% a decade ago”.

The IEA might be too conservative: For what it’s worth, I expect the IEA is too conservative here – Anthropic said in its OSTP RFI submission that it believes the United States alone will need to build on the order of 50GW of net new power by 2027 to support frontier training runs by US companies.

Rhymes with other analysis, but not precisely: A US focused study from Berkeley said it projected US data center use to grow from roughly ~40GW / 176 TWh in 2023 to ~74GW / 325 Twh to 132GW / 580 TWh by 2028. These numbers are significantly larger and more in line with the Anthropic projections in terms of bullishness (Import AI #395).

Why this matters – the world is preparing for the singularity: If you zoom out, it looks a lot like the world’s capital markets and major companies are collectively betting that it’s going to get more and more lucrative to turn electricity into computational power which gets turned into dollars – and it seems like AI is one of the primary drivers of growth here. Viewed through this lens, the world is preparing the necessary infrastructure for the arrival of a superintelligence.
Download the report here: Energy and AI (IEA website).

***

Distributed AI experts Nous get $50 million funding:
…The market has started valuing distributed AI, which means the technology will be developed more rapidly…
Crypto investor Paradigm has led a $50m Series A round in Nous, a startup which has pioneered distributed training of AI systems. As longtime Import AI readers know, Nous is – along with Prime Intellect – are serious players in distributed AI, having trained a ~15bn parameter model in December (Import AI #393) using an algorithm they developed called Distributed Training Over-the-Internet (aka, Distro: Import AI #384), and have even decoupled with Anthropic researcher (in a personal capacity) Durk Kingma to develop technology called Decoupled Momentum (DeMo) for even better distributed training (Import AI #395).

Why this matters – markets are beginning to value distributed AI: I’ve been following distributed AI for a while and most of its enthusiastic developers and users have been hobbyists or startups with relatively small amounts of funding. The arrival of a $50m Series A could be a symptom that the VC community is about to start shoveling money into startups using this technology which would further speed up adoption and maturation of it increasing the chance that AI systems trained in distributed ways could attain the computational scale necessary to take on proprietary models.
Read more: Crypto VC giant Paradigm makes $50 million bet on decentralized AI startup Nous Research at $1 billion token valuation (Fortune, via Yahoo! Finance).

***

The Virology Capabilities Test tells us there’s probably a scaling law for bioweapon design:
…Today’s AI systems are better than expert virologists at potentially dangerous things…
Researchers with SecureBio, the Federal University of ABC, the Center for AI Safety, and the MIT Media Lab, have built the Virology Capabilities Test (VCT), 322 multimodal questions for AI systems “covering fundamental, tacit, and visual knowledge that is essential for practical work in virology laboratories”.

VCT has been designed as a way to test out how well today’s AI systems understand things that would let them be potentially weaponized for dangerous purposes. Examples of the kind of things VCT tests for include: Isolating virus particles from a liquid medium, the detailed steps in a TCID50 protocol, successfully infecting a ferret with a test strainge, and troubleshooting low viral yields from a given protocol.

Frontier models are better than expert human virologists: “Expert virologists with access to the internet score an average of 22.1% on questions specifically in their sub-areas of expertise. However, the most performant LLM, OpenAI’s o3, reaches 43.8% accuracy, outperforming 94% of expert virologists even within their sub-areas of specialization,” the researchers write.

How the questions were built: Given those scores, how concerned should we be? A close read of the paper gives me a sense the answer is “we should be sweating nervously” – to build the questions, the researchers used questions from 57 contributors, all of whom had either obtained or were in the process of obtaining a PhD in virology, and each contributor had an average of 5 years and 10 months of virology experience. Additionally, when building the dataset, they tested out how easy questions were by seeing if experts could answer to them with access to Google – if more than two thirds of them answered the question, the questions got tossed out. In other words, the questions in VCT are curated by experts and have been pressure tested by other experts for hardness.

The problem with dual use evals is that they’re hard to share: The authors note that “the shared dataset will not be released publicly to reduce the risk of leakage into training corpora, but the benchmark will be directly conveyed to any organizations and researchers with a track record of work on AI safety”. While the training dataset contamination issue makes sense, I suspect the larger reason the authors haven’t shared it is that it contains net new information about potentially dangerous virology.

Why this matters – everything machines for dual-use: AI systems are good at a broad range of things, including scary things. Tests like VCT give us signals on the scary part. “The scientific capabilities of frontier models will doubtless accelerate beneficial research in the life sciences, but the demonstrated ability to match or exceed expert performance in troubleshooting dual-use virology lab work warrants careful consideration,” the authors write. “We believe that an expert-level AI virologist chatbot—which is constrained to giving advice via text-based interactions—poses less risk than an autonomous AI virologist agent capable of independently performing tasks, though both warrant careful controls”.
Read more: Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark (PDF).

***

Automating and industrializing robot AI research with AutoEval:
…Berkeley researchers try to make running experiments on hardware as easy as doing things purely in software…
UC Berkeley researchers have built AutoEval, technology to make automating the running of experiments on robots as easy as automating software software pipelines.
“AutoEval consists of three key modules: (1) a success classifier, that evaluates a policy’s success on a given task, (2) a reset policy, that resets the scene back to a state from the initial state distribution upon completion of a trial, and (3) programmatic safety measures and fault detections that prevent robot damage and call for human intervention when necessary,” they write.

It works well: In tests, experiments conducted by AutoEval closely match the same results people get from experiments supervised by humans. Additionally, the software is robust over long timespans – during a 24 hour period they only needed a human to step in 3 times, dramatically cutting the amount of human supervision needed for running experiments.
“Even though AutoEval has a slightly lower throughput, AutoEval runs autonomously and only required a total of three human interventions in the span of 24 hours to reset the scene or robot,” they write. “Every time a human operator needed to intervene, they simply needed to check and reset the objects’ position in the scene, and potentially move the robot arm into reset position if a motor failed and the robot fell on the table”.

Why this matters – the industrialization of robot research: A few years ago Google made headlines by running a so-called robotic ‘arm farm’ (2017, Import AI #51) where it had tens of different robots working in parallel to learn how to manipulate arbitrary objects. Technologies like AutoEval seem like the kind of thing that Google might itself have built to help it run the arm farm. But unlike the proprietary code still nestled somewhere in Mountain View, AutoEval is available as open source software, robotic arms themselves have got way cheaper, and the algorithms to get robots to perform tasks have got far better than they were a few years ago.

Put it all together and AutoEval seems like one of the technologies we’ll use to industrialize and scale-up research onto robots. “We hope that this work will inspire more AutoEval evaluation cells to be set up across institutions to form a diverse automated evaluation framework, which will significantly speed up robot learning research,” the researchers write.
Read more: AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World (arXiv).
Get the software here: AutoEval (GitHub).

***

Tech Tales:

The Cockroach Killers of the Cyber Domain
[As told to GQ, 2028]

When you’re a bug catcher you get invited into a house and the owner says it’s full of bugs and you need to get rid of them, but you cannot damage the house itself. This means you need to figure out a way to seal the house and fumigate it, while also figuring out the places where the insects nested and getting rid of the nests and any associated damage. Your goal is to cleanse the house, then make sure the house cannot get re-taken by the bugs.

These days, people working in AI have to do a similar thing – someone will discover that their company has an AI problem, in the sense it has a few small-scale AI agents which are causing some kind of low-rent trouble.

That’s when you call us: we get access to your infrastructure and we instrument it so we can isolate benign activity from the agent activities. We seal the ingress and egress points of your network and in extreme cases we might work with your hyperscaler partner to physically isolate your hardware from anything else. Then we crawl through your system and try to find the agents – this is harder than it sounds because the agents are constantly shape-shifting, changing their file names, moving around the network, sometimes slowly making copies of themselves in other parts of your infrastructure, and so on.

Once we’re sure we’ve cleaned everything we also attempt to seal the holes that let the agents creep in. Sometimes these are basic network security issues, but sometimes it’s more subtle – maybe your company had an AI system which could spit out custom agents and maybe you let it have too big a context window and access to too many tools when making its agents, so some other larger malignant thing outside your company compromised it and, presto, it started producing the bugs.

Things that inspired this story: A friend of mine whose job was termite removal and the stories thereof; how we should expect some small agents to become akin to crappy digital malware, not so dangerous we will need to take extreme actions but sufficiently annoying you’ll want to remove them; blue collar IT jobs during the superintelligence uplift.

Thanks for reading!

Subscribe now

Welcome to Import AI, a newsletter about AI research. Import AI runs on lattes, ramen, and feedback from readers. If you’d like to support this, please subscribe.

Subscribe now

Prime Intellect launches a decentralized training run for a 32B parameter model:
…INTELLECT-2, if successful, will further alter the number of potential players on the AGI gameboard…
Decentralized AI startup Prime Intellect has begun training INTELLECT-2, a 32 billion parameter model designed to compete with modern reasoning models. In December, Prime Intellect released INTELLECT-1, a 10b parameter model trained in a distributed way (Import AI #393), and in August it released a 1b parameter model trained in a distributed way (Import AI #381). You can follow along the training of the model here – at the time of writing there were 18 distinct contributors training it, spread across America, Australia, and Northern Europe.

Prediction confirmed: In Import AI 393 I predicted we’d see the first 30B parameter distributed training run by April 2025 – so INTELLECT-2 arrives right on schedule. At this rate, I predict we’ll see a 70B-100B range run by December 2025.

Why this matters – decentralized training will alter the political economy of superintelligence: Currently, a lot of AI policy relies on the idea that powerful AI systems will be trained by a very small number of entities that can individually ‘mass’ very large amounts of compute – for instance, frontier labs like Anthropic or OpenAI, or hyperscalers like Google. As distributed training software gets better and more ‘proof points’ emerge of good models trained in a distributed way, this dynamic could alter – if models like INTELLECT-2 are good and generate economic value, then it might lead to a new type of player on the AGI gameboard – loose federations of organizations pooling compute in a globally distributed way to train models.
Read the blog: INTELLECT-2: Launching the First Globally Distributed Reinforcement Learning Training of a 32B Parameter Model (Prime Intellect).
Check out the training progress here: INTELLECT-2 dashboard (Prime Intellect site).

***

What the negative reaction to the launch of a startup tells us about the AI safety community:
…Mechanize’s skeptical reception from some people is a symptom of a broader problem – ideological purity tests are often bad…
Last week some researchers announced a new AI startup “focused on developing virtual work environments, benchmarks, and training data that will enable the full automation of the economy.” The startup, Mechanize, is backed by investments from important figures in AI and tech, like Nat Friedman, Patrick Collisson, and Jeff Dean. So far, so normal. But what was strange was the adversarial reception this launch got from some people.

How normal launches work versus this launch: Typically, company formation announcements in Silicon Valley are treated kindly with people responding with variations of ‘hell yeah, let’s fucking gooooo!’. But Mechanize got a distinctly different response, likely because many of the people associated with it came from Epoch, an independent research organization that measures and observes the state of AI progress, rather than developing direct capabilities itself.
“Sad to see this”, wrote Anthony Aguirre, a founder of AI advocacy group the Future of Life Institute. “Hard for me to see this as something other than just another entrant in the race to AGI by a slightly different name and a more explicit human-worker-replacement goal.”
“This seems to me like one of the most harmful possible aims to pursue,” wrote Adam Scholl, someone who works on alignment.
“I think this is a bad thing to do, and I’m sad to see you’re doing this,” wrote Peter Barnett, who works at the Machine Intelligence Research Institute (MIRI).
“Alas, this seems like approximate confirmation that Epoch research was directly feeding into frontier capability work, though I had hope that it wouldn’t literally come from you,” wrote Oliver Habryka, who works on LessWrong.
“How could you? This is the opposite of keeping the world safe from powerful AI! You are a traitor,” wrote Holly Elmor, who leads the Pause AI movement.
Etc. There are many more examples!

Why this matters – the AI safety community is dissolving into infighting: As the stakes of AI development increases it feels like the AI safety community seems to be developing a more extreme faction within it that exhibits ‘strong opinions, strongly held’ views. Many people in AI safety seem to be of the view that anything which makes any contribution at all to the forward progress of AI technology is dangerous bad for society. The people that believe this hold complex, typically very technically informed views, so I am not questioning the legitimacy of their arguments.
I am, however, highlighting that this kind of discourse in public looks a lot like running ‘ideological purity tests’ on people and then deciding if they’re in-group or out-group, then treating them differently – and it likely feels that way to the people on the receiving end of this. It’s very rare that ideological purity tests lead to productive outcomes – rather, it more often leads to the hardening of more extreme positions and incentivizes further factionalization.
Of course, some people may disregard this as ‘person who works at company (bad) defends people starting a company (also bad)’. I hope people could look beyond where I work and recognize that even if you think I’m wrong and these people are wrong, there are likely better ways to enable good discourse than this kind of thing.
Read more about mechanize here (Mechanize official site).

***

No NVIDIA? No problem! Huawei trains a strong dense model on Ascend NPUs:
…Pangu Ultra is a 135bn parameter dense LLM with competitive scores…
Huawei has built Pangu Ultra, a large-scale language model with competitive albeit not world-leading performance. The most interesting thing about Pangu is it was trained on 8,192 Ascend NPUs, serving as an important proof-point that it’s possible to train large-scale AI systems on a Chinese-designed chip. Pangu is the latest in a (for AI, long-running) research effort by Huawei; the first Pangu model, a GPT3 clone, was released in April 2021 (Import AI #247).

Pangu details: Pangu Ultra is a dense (non-MOE) LLM trained on 12.3 trillion tokens of data. Its architecture is broadly similar to Facebook’s LLaMa 3 model, albeit with a tweak to the normalization scheme as well as the parameter initialization. Pangu Ultra has an effective context length of 128K tokens. It is trained in a three phase way, with a 12T token pre-training stage “focused on developing broad linguistic capabilities and general knowledge”, then a 0.8T token ‘reasoning’ stage where it sees “high-quality and diverse mathematical and coding data”, and then a 0.4T ‘annealing’ phase where it sees instruction data to make it more intuitive for people to prompt.

More details on data: “The data pool is curated from a wide range of domains and task types, including general question answering, AI-generated content (AIGC), text classification and analysis, programming, mathematics, logical reasoning, and tool usage,” Huawei writes. “These tasks cover application areas such as finance, healthcare, and public services. Data sources span open-source instruction datasets, real-world industrial queries, and synthetic problems derived from the pre-training corpus.”

How good is it? Pangu is a good but not world-leading model, according to tests comparing it to Qwen2.5 72B Base, LLaMa-3.1 405B Base, and DeepSeek V3 base. It gets good scores on some benchmarks for English, Code, Math, and Chinese-specific tests (e.g, beating all the other models on things like Hellawag, HumanEval, MATH, and CMMLU) but loses or ties DeepSeek on some important widely used benchmarks (e.g, MMLU, GSM8K). It fairs somewhat better on some hard science and coding benchmarks, setting high scores on AIME 2025 and GPQA Diamond.

Why this matters – Pangu is the top layer of an increasingly indigenous stack: Pangu is another proofpoint for the broad decoupling occurring between the Western and Chinese ‘AI stacks’ – where once AI systems in both countries were trained on common compute substrates as well as common software (e.g, Tensorflow), in recent years things have been decoupling. The fact Pangu was trained on Huawei’s Ascend chips is significant (though it’s worth noting the Ascend chips themselves, while Chinese-designed, are made using a variety of components sourced from outside China, including rumors the Ascend series were made via TSMC).
Read more: Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs (arXiv).

***

Agents that generate their own data will be fundamental to future AI progress:
…Getting to superintelligence via ‘the era of experience’
AI pioneers David Silver (Alphago, etc) and Richard Sutton (godfather of reinforcement learning) have written a position paper on the future of AI, claiming that getting to superintelligent systems will require AI agents that train on data they gather from interaction with the world, rather than human-curated datasets.

“AI is at the cusp of a new period in which experience will become the dominant medium of improvement and ultimately dwarf the scale of human data used in today’s systems”, the pioneers write. “Our contention is that incredible new capabilities will arise once the full potential of experiential learning is harnessed. This era of experience will likely be characterised by agents and environments that, in addition to learning from vast quantities of experiential data, will break through the limitations of human-centric AI systems”.

Key inputs to the era of experience:

• “Agents will inhabit streams of experience, rather than short snippets of interaction.

• Their actions and observations will be richly grounded in the environment, rather than interacting via human dialogue alone.

• Their rewards will be grounded in their experience of the environment, rather than coming from human prejudgement.

• They will plan and/or reason about experience, rather than reasoning solely in human terms”.

Dangers and differences ahead: Of course, building agents that gain expertise through interaction with the world will introduce a range of challenges for ensuring these things are safe – “whilst general concerns exist around the potential misuse of any AI, heightened risks may arise from agents that can autonomously interact with the world over extended periods of time to achieve long-term goals,” the authors write.
One of the more troubling risks could be that these AI agents may learn their own shorthand to use to ‘think’ about the world, which may make them much less interpretable to us – in other words, the era we’re in now where AI systems use english to generate their reasoning traces may be short-lived, and they may figure out something else. “More efficient mechanisms of thought surely exist, using non-human languages that may for example utilise symbolic, distributed, continuous, or differentiable computations,” the authors write. A self-learning system can in principle discover or improve such approaches by learning how to think from experience”. It’s worth noting that this risk has also been independently identified by the authors of the recent ‘AI 2027’ forecasting essay.

Why this matters – superintelligence is increasingly being thought of as an engineering challenge: Papers like this are emblematic of the confidence found in the AI industry: where superintelligence was once an indefinable pipe dream, it’s now outlined instead as something that can be achieved through the deployment of engineering resources to create more capable AI agents, then the gumption to give these agents’ sufficient independence and latitude that they can interact with the world and generate their own data.
Read more: Welcome to the Era of Experience (PDF).

***

AI expert: The scariest thing about powerful AI is about its power, not misalignment:
…Even if alignment works, the tremendous power of AI could be the greatest risk…
AI researcher Michael Nielsen thinks one of the most significant risks to civilization from AI isn’t from misaligned AI systems, but rather from the changes in the distribution of power that very capable machines will cause. “The problem isn’t whether intelligence is carbon or silicon-based, but about increased intellectual capability leading to increased power and access to catastrophic technologies,” Nielsen writes. “It is not control that fundamentally matters: it’s the power conferred.

Toy models and climate change: Part of the reason why the debate about risks from AI systems feels so confusing these days is that everyone is reasoning from toy models of systems which don’t yet exist, much like how in the middle of the 20th century scientists used toy models of the earth to help them think through climate change – but these toy models didn’t fully capture the complexity of the problems ahead, so reasonable scientists could draw different conclusions from the same models.
“Strong disagreement about ASI xrisk arises from differing thresholds for conviction and comfort with reasoning that is in part based on toy models and heuristic arguments,” Nielsen writes. “Furthermore, while climate can plausibly be predicted using detailed physical models, ASI is subject to a wildcard factor, of ASI acting in some decisive way that we intrinsically can’t predict in advance, since ASI is by definition far superior to humans in intellect.”

Why this matters – even if we succeed at aligning AI systems, great changes will take place: The essential point Nielsen makes here is a helpful one – if anyone succeeds at building a ‘safe’ superintelligence, they’ll have something able to cause such vast changes in the world that this itself will pose a danger. I think many people are underestimating just how disruptive a superintelligence could be to the order of the world. “The fundamental danger isn’t about whether “rogue ASI” gets out of control: it’s the raw power ASI will confer, and the lower barriers to creating dangerous technologies”, he writes.
Read more: ASI existential risk: reconsidering alignment as a goal (Michael Nielsen blog).

***

Wanna run DeepSeek-R1 on your home devices? Prima.cpp makes it easy:
…Distributed homebrew clusters for local AI…
Researchers with Mohamed bin Zayed University of Artificial Intelligence in Abu Dhabi and the University of Electronic Science and Technology of China in Chengdu have developed Prima.cpp, open source software to make it easy to run large language models on a motley crew of home devices.

What Prime.cpp is: Prime.cpp is software that helps you take a large-scale language model (e.g, DeepSeek-R1 or Llama-3-70b) and then slice it up across a few home computers so you can run it faster than if it was running on just one device. The software uses a device profiler to look at the differing computation, memory, disk, communication, and OS properties of your devices, then uses an algorithm (Halda) to figure out which layer(s) of the model to assign to which devices for minimizing latency.
Prima.cpp is built on top of llama.cpp, as well as ggml and gguf.

Promising performance: “Evaluation on a real home cluster shows that prima.cpp is 15× faster than llama.cpp on 70B models, with memory pressure below 6% per device. It also surpasses distributed alternatives like exo and dllama in both speed and memory efficiency across all 7B-72B models,” the researchers write. “In our experiments, a small, heterogeneous, and budget-friendly home cluster (2 laptops, 1 desktop, 1 phone) was used.”
Supported models: Prima.cpp supports QwQ-32B, Qwen 2.5-72B, Llama 3-70B, and DeepSeek R1 70B.

Why this matters – sovereign AI relies on home computing: AI tends towards centralization – large, proprietary models run on large software-as-a-service systems and are made available via APIs or consumer surfaces. Decentralization requires a couple of distinct ingredients: 1) broadly available open weight models (e.g, LLaMa, DeepSeek), and 2) software to make it easy to run those models on the kinds of computers people might be expected to have (e.g, laptops and gaming computers, rather than powerful home servers). Prime.cpp is one of the ways you solve for 2).
Get the software here (Prima.cpp, GitHub).
Read the paper: PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday Home Clusters (arXiv).

***

Tech Tales:

When the coders became the writers
[As told by a human to an archival system after The Uplift]

Oh I know it’s hard to believe but back then we got paid insane amounts of money to program computers. And the benefits! Free daycare! Free lunch – gourmet. Hot breakfast. Company retreats. Annual conferences where we’d get big bands to come and play just for us and our friends. And the whole time we were told we deserved this – we were computer programmers and we were young and we were brilliant.

None of us really knew the size of the tide that would wash over us. Most of us welcomed it.
“Hey cool,” we said when GitHub Copilot came out, “this is awesome.”
“Wow, I can write five times as much code,” we said, when Claude Code came out.
We were like journalists as the internet began to eat advertising – as ‘ look at how many people read our words now’ was to writers in the 2000s, ‘look at how much code the AI can write for me now’ was to coders in the 2020s.

Creative destruction is all fun and games until it happens to you. Anyway, I get by these days – I still work, like most of my peers, but the jobs are different. We watch from the sidelines now as the bioengineers go through what we had and what the writers had before us. But now that the AI systems are running their own ‘dark wetlabs’, we can see the tide about to wash over them as well.

Things that inspired this story: Visits to the multiple restaurants in the offices of the hyperscalers; younger me watching Blink 182 play a cloud storage conference by Box; watching Pearl Jam dedicate a song to Mark Hurd at Oracle OpenWorld; tales told to me by older journalists when I was coming up in the tread; The Luxurious Death Rattle of the Great American Magazine; my experience as a former journalist working in technology and watching people assume the perks are natural and will always be there; the experience of ex-government colleagues not having to pay for coffee.

Thanks for reading

Subscribe now