HSE University scientists have found a way to speed up neural network optimization by up to 500 times using the laws of physics
Researchers at the HSE Laboratory for Social and Cognitive Informatics in St. Petersburg, led by Sergey Koltcov, have learned to quickly reduce the size of neural networks without losing quality using statistical physics methods.
Modern neural networks are becoming increasingly powerful, but their growth creates significant limitations. Models like GPT contain tens and hundreds of billions of parameters—numbers through which information flows when processing a request. But as quality increases, so does the cost of creating and using AI.
"The largest models require hundreds of gigabytes of memory: this creates an economic barrier and limits access to technology," explains research leader Sergey Koltsov. "We decided to analyze the behavior of a neural network during compression and compare it with known functions from statistical physics."
The problem of compression is especially pressing in areas where data cannot be transferred to external cloud services. Banks operate in closed networks, medical institutions protect patient information, and government organizations cannot share confidential information. All of these organizations require efficient yet compact solutions that can run on local hardware—from a server in an in-house data center to a doctor's laptop.
Existing neural network compression methods are based on a simple idea: not all model parameters are equally important for its operation. Some can be removed with virtually no consequences. The difficulty is understanding which ones. The classical approach requires conducting numerous experiments, gradually changing the compression ratio and checking the model's accuracy each time. This is time-consuming.
Sergei Koltcov
"Our perspective allows us to view neural networks as statistical systems. This is a branch of science that studies the behavior of objects with a huge number of elements: from gas molecules to magnetic materials. A neural network with billions of parameters turned out to be similar to such structures. At extreme points—maximum or minimum—the model maintains an optimal balance between size and performance. In other words, we've proven that this approach can accelerate the search for the optimal number of algorithms by hundreds of times," explains Segei Koltcov.
A four-person research team—three Russian scientists and a specialist from India—has been working on the project since early 2025. It was important to test the method's versatility. Experiments were conducted on medium-sized models—from seven to ten billion parameters. These are systems that can be run on a powerful laptop or a small server. These are precisely the solutions needed by medical assistants, corporate analytics systems, and local data processing services.
"We tested the hypothesis on models of various scales and purposes—from text processing to image recognition," continues Sergey Koltsov. "The method proved effective on various architectures. In some cases, it performed better, in others, a little worse, but the main thing was that it worked, and it worked quickly. Depending on the model, the speedup ranged from ten to five hundred times compared to the traditional approach."
The method is now available for use. Any developer or researcher can apply the described approach to their models. This is especially relevant for companies and organizations that run neural networks on their own hardware with limited resources.
The scientists are currently continuing their work, optimizing the number of neurons in each network layer. Next, they plan to reduce the number of blocks in the model architecture. How many blocks are needed for optimal performance is a question that currently has no clear answer. "If we can determine the optimal number of blocks before training the model, the savings will be colossal. That's our next goal," notes the leading researcher at the Laboratory of Social and Cognitive Informatics.
Research results were published in the journal Physica A: Statistical Mechanics and its Applications.