Hanyu Wang

Check out An-Chieh Cheng's new work on providing a scalable way for VLM to have the sense of 3D and perform 3D spatial reasoning in any scenes: https:// anjiecheng.me/sr3d

80

📣 Announcing NeuronMM: Faster LLM Inference on Amazon Web Services (AWS) Trainium At Yotta Labs, efficiency and performance on heterogeneous hardware are at the heart of what we do. Our research team, led by Chief Scientist Dong Li, has developed NeuronMM, an open-source matrix multiplication (matmul) kernel that dramatically accelerates LLM inference on AWS Trainium. Why it matters: 🔹 Trainium isn’t just another chip, it’s the cornerstone of AWS’s long-term AI infrastructure strategy. 🔹 NeuronMM delivers a 2.09× average speedup at the matmul kernel level, and up to 2.22× for certain sequence lengths. 🔹 End-to-end LLM inference is 1.21×–2.49× faster, depending on the model. 🔹 It uses clever hardware-aware optimizations: reduced data movement, maximizing SRAM utilization, and avoiding expensive matrix transpose operations. 🔹 The trade-off in model accuracy is minimal; for example, on Qwen-3-1.7B, NeuronMM achieves 1.74× faster inference with only a 0.03 drop in accuracy. How we did it: 🔹 Built on top of the NeuronX Distributed Inference library + AWS NKI. 🔹 Used SVD (singular value decomposition) to compress weight matrices, reducing memory traffic. 🔹 Designed a custom “TrainiumFusion” kernel that fuses compute and caching strategies tailored for Trainium’s memory hierarchy. Impact: This is a key milestone for the Trainium ecosystem, unlocking more efficient, scalable LLM inference on AWS-native hardware. And because NeuronMM is open source, anyone can build on it or adapt it to their use case. ⚡ If you’re running inference workloads on Trainium, we’d love for you to try NeuronMM and share feedback. AI workloads on multi-silicon are inevitable—and together, we can push the performance frontier forward. Read the full report 👉 https://lnkd.in/gpKXP93m The code is available at 👉 https://lnkd.in/gyZDSjhv #AI #LLM #Inference #AWS #Trainium #OpenSource #HardwareAcceleration #YottaLabs

13

New pre-print led by Baskin Engineering at UCSC PhD student, Conrad A. in collaboration with Pedram Hassanzadeh and Michael Mahoney. https://lnkd.in/gWXuKiPw The paper is an attempt to build a theory of inference-time error growth, and thus stability, of AI-based autoregressive models, similar in spirit to the kind of theory that has been around in PDEs for a century. While neural autoregressive models have rivalled their numerical counterparts in several areas of science and engineering, a key failure mode that haunts these models, especially for chaotic systems, is that their instability is unpredictable. A scientific theory of model quality agnostic of architecture, systems, data, loss functions does not exist. And stabilizing these models is a mixture of good intuition, heuristics, trial-and-error... Some of this comes down to what we actually optimize. Training penalizes the one-step generalization error, or at best the error accumulated over a handful of unrolled steps. Both are measures of the error the model makes. Neither says anything about what happens to an error the model has already made, and that is governed by the Jacobian of the learned update map with respect to the state, which the loss leaves entirely free and which is rarely even looked at. Two models with nearly identical one-step accuracy can therefore fall apart at very different rates over a rollout. This work centers around deriving the analytical form of error propagation in these models, decomposing them into the main contributing terms, and predicting their growth, decay, and oscillation using eigenanalysis. The theory derives where the core approximations are correct and where they break down, what extra terms need to be included to carry the approximations forward and hints at a scaling law with respect to frequency at which data is sampled.

88

4 Comments

Recently the Muon optimizer has shown great promise in LLM pre-training. By performing gradient matrix orthogonalization, Muon can effectively avoid/migitate gradient norm explosions or vanishing. In collaboraiton with Prof. Sijia Liu, recently my students Ruijie Zhang, Yequan Zhao, Ziyue Liu, Zhengyang Wang, Dongyang Li and Yupeng Su developed and implemented Teon, a tensorized version of Muon, to improve LLM pre-training efficiency & accuracy. The key idea is to stack multiple gradient matrices together and perform mode-i orthogonalization of the resulting higher-order tensor, which can effectively explore the correlation among the gradient matrices. When tested on GPT-2 and LlaMA models, Teon shows consistent improvement over the conventional Muon baselines. They also provided theory to explain why "tensorization" can offer benefits, and how to maximize the benefit during the tensorization process. Paper link: https://lnkd.in/gpq-kxmW

124

#story Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search Yuxian Gu, Qinghao Hu, Shang Yang, Haocheng Xi, Junyu Chen, Song Han, Han Cai Tsinghua University, MIT, UC Berkeley, NVIDIA 2025 https://lnkd.in/d4EhKcUk Summary: A Faster, Smarter AI That Thinks Differently Researchers have created a new type of AI language model, called Jet-Nemotron, that is designed to be both incredibly smart and remarkably fast. Think of it as building a new, more efficient engine for AI. The Problem: The most powerful AI models today are like giant brains that have to consider every single word in a sentence at once to generate a response. This makes them very accurate, but also very slow and energy-hungry, especially for long conversations or documents. The Innovation: The team developed a clever new method called "Post Neural Architecture Search" (PostNAS). Instead of building a new AI from scratch, they started with a powerful, pre-existing model and found a way to streamline its "thought process." They kept its core knowledge frozen and then expertly redesigned how it pays attention to information, making the process much more efficient without losing smarts. The Result: Jet-Nemotron This new model family is a breakthrough in efficiency. In head-to-head tests, the Jet-Nemotron-2B model performed as well as—and sometimes even better than—leading AI models from major companies like Meta (Llama) and Google (Gemma). Most impressively, it achieved these results while being dramatically faster: * It can generate text up to 53 times faster. * It can process your initial prompt up to 6 times faster. In a stunning twist, this smaller, more efficient model even outperformed much larger and more complex rival models on difficult knowledge tests, proving that smarter design can sometimes beat raw size. In a nutshell: Jet-Nemotron represents a significant step forward, offering a powerful AI that is both highly capable and built for speed, making advanced AI more practical and accessible. The Jet-Nemotron abstract suggests a very promising and innovative direction, but independent verification and real-world testing are needed before we can fully understand its capabilities and limitations. It could break the primary trade-off in AI—between capability and cost/speed. If real, it doesn't just make existing AI a little better; it changes who can build it, how it can be used, and where it can be deployed, pushing powerful AI out from giant data centers and into the everyday world. #AI #computation #language #machinelearning #LLM #JETNemotron

2