A US startup has developed what it claims is the world’s smallest artificial intelligence (AI) supercomputer. It’s packed with high-performance hardware and plenty of RAM, and is compact enough to fit in your pocket, yet can run “Ph.D. intelligence” AI models, company representatives say. This means being capable of autonomous problem solving, abstract reasoning, and strategic planning.
“AI Pocket Lab,” as the creators of Tiiny AI branded the device, can run complex 120 billion parameter large-scale language models (LLMs) locally, without relying on an internet connection. These systems typically require data center-class infrastructure to run, but this opens up the possibility of local expert-level coding capabilities, document evaluation and refinement, or multi-step inference.
It’s built around a 12-core ARM processor, the kind commonly found in smartphones, laptops, and tablets. Despite the device’s small frame, measuring just 5.59 x 3.15 x 1.00 inches (14.2 x 8 x 2.53 cm), it packs 80 GB of LPDDR5X RAM. For comparison, most modern laptops have between 8GB and 32GB of RAM.
you may like
Pocket Lab’s massive 48GB of RAM is also reserved exclusively for its Neural Processing Unit (NPU), a chip optimized for AI-related computations. For years, Intel and AMD have been building processors with dedicated NPUs to handle AI workloads and meet Microsoft’s 40 trillion operations per second (TOPS) threshold for running AI features in Windows 11.
Pocket Lab’s computational power qualifies it as a supercomputer (rather than a standard mini-PC or workstation), allowing it to perform workloads that would typically require a multi-GPU, datacenter-class system, especially local inference with over 100 billion parameter language models. Current models that can run on the device include GPT-OSS 120B, large-scale Phi models, and high-parameter Llama family models.
This is part of a recent push into edge computing for AI, aimed at mitigating some of the power constraints and environmental impact of distributed AI processing.
pocket power
Although AI Pocket Lab is far from matching the world’s most powerful supercomputers, it can deliver 190 TOPS of computing power between its NPU and CPU. This represents another step towards miniaturization, following Nvidia’s recently announced Project Digits mini PC. It doesn’t have the processing power of an Nvidia project, but it’s a fraction of the size.
To pack so much power into such a modest chassis, the Tiiny AI team relied on a number of technologies and optimizations. Chief among them was what the company called TurboSparse. This is an innovation that allows large-scale LLMs to run faster on more limited hardware by ensuring that the system calls only the parts of the model it needs at any given time. Traditional models use all parameters for each word of processing/output, whereas TurboSparse models use only certain parameters for each step.
Another important feature is PowerInfer, which enables heterogeneous scheduling of CPUs, GPUs, and NPUs on devices. This means that each processor is given only the most powerful workload, which increases overall system efficiency and reduces power consumption. PowerInfer also includes intelligent power management to determine when full power is needed and when it’s OK to use less by eliminating unnecessary calculations.
The implications of small AI supercomputers go beyond reducing dependence on environmentally harmful data centers. This is a privacy boon, allowing users to deploy the power of advanced LLM without an internet connection and without data being processed in the cloud by a third party, while enabling AI access in field environments such as remote research stations or out-of-range ships and aircraft.
Source link
