Equations (APPLICATION OF LARGE LANGUAGE MODELS TO SOLVE DIFFERENTIAL EQUATIONS)

Project leader: Sergey Koltsov

Project participants: Anton Surkov, Vera Ignatenko, Vladimir Zaharov

This work explores the feasibility of applying large language models for obtaining analytical solutions to differential equations. Within this approach, differential equations and their solutions are viewed as symbolic sequences. Consequently, predicting such sequences can be reduced to the problem of applying seq2seq models. The representation of differential equations and their solutions as a set of symbols will be implemented using a range of Python libraries, allowing the transformation of formulas into text representations in Latex format.

The first phase of the project utilizes recurrent seq2seq architectures such as RNNs, LSTMs, and GRUs as base models. The second phase utilizes more powerful neural network architectures, including large language models based on the Transformer architecture (T5, DeepSeek-R1-Distill-Qwen-7B, and Phi-4-mini), as well as the Mamba state-space model, which are further trained on symbolic representations of differential equations. The second phase also examines the ability of small reasoning language models (such as DeepSeek-R1-Distill-Qwen-1.5B, Qwen2.5-1.5B, and Open-Reasoner-Zero-1.5B) to construct analytical solutions to differential equations out of the box.

Project's publications:

1. Vladimir Zakharov, Anton Surkov, Sergei Koltcov. AGDES: a Python package and an approach to generating synthetic data for differential equation solving with LLMs // Procedia Computer Science, 2025, Volume 258, Pages 1169-1178, ISSN 1877-0509.
2. Anton Surkov, Vladimir Zakharov, Sergei Koltcov, Vera Ignatenko. Application of Large Language Models to Solving Differential Equations: Constructing Baseline Models with LSTM and GRU, in : Smart Technologies, Systems and Applications. SmartTech-IC 2024. / Ed. by F. Narváez, M. Villa, G. Díaz. Vol. 2: Revised Selected Papers, Part II Springer, 2025. P. 239–252.
3.   Ignatenko V., Surkov A., Zakharov V., Koltcov S. Transformers and State-Space Models: Fine-Tuning Techniques for Solving Differential Equations // Sci. 2025. Vol. 7. No. 3. Article 130.
4. Кольцов С. Н., Игнатенко В. В., Сурков А. Ю., Захаров В. О. Решение дифференциальных уравнений с помощью языковых моделей из коробки: потенциал небольших LLM в математике // Доклады Российской академии наук. Математика, информатика, процессы управления. 2025. Т. 527. С. 311–319.