Математические методы прогнозирования (лекции, А.В. Грабовой, В.В. Стрижов)/Осень 2021

Материал из MachineLearning.

Перейти к: навигация, поиск

Mathematical methods of forecasting

The lecture course and seminar introduces and applies methods of modern physics to the problems of machine learning.

Minimum topics to discuss: Geometric deep learning approach.

Optimum topics to discuss are: tensors, differential forms, Riemannian and differential geometry, metrics, differential operators in various spaces, embeddings, manifolds, bundles. We investigate scalar, vector and tensor fields (as well as jets, fibers and shiefs, tensor bundles, sheaf bundles etc.). The fields and spaces are one-dimensional, multidimensional and continuously dimensional.



  • Questionnaires during lectures (4)
  • Two application projects (2+2)
  • The final exam: problems with discussion (1)
  • Bonus for active participation (2)


BCI, Matrix and tensor approximation

  1. Коренев, Г.В. Тензорное исчисление, 2000, 240 с., lib.mipt.ru.
  2.  Roger Penrose, "Applications of negative dimensional tensors," in Combinatorial Mathematics and its Applications, Academic Press (1971). See Vladimir Turaev, Quantum invariants of knots and 3-manifolds (1994), De Gruyter, p. 71 for a brief commentary PDF.
  3. Tai-Danae Bradley, At the Interface of Algebra and Statistics, 2020, ArXiv.
  4. Oseledets, I.V. Tensor-Train Decomposition //SIAM Journal on Scientific Computing, 2011, 33(5): 2295–2317, DOI, RG, lecture, GitHub, Tutoiral.
  5. Wikipedia: SVD, Multilinear subspace learning, HOSVD.

BCI, Feature selection

  1. Мотренко А.П. Выбор моделей прогнозирования мультикоррелирующих временных рядов (диссертация), 2019 PDF
  2. Исаченко Р.В. Снижение размерности пространства в задачах декодирования сигналов (дисссертация), 2021 PDF

High order partial least squares

  1. Qibin Zhao, et al. and A. Cichocki, Higher Order Partial Least Squares (HOPLS): A Generalized Multilinear Regression Method // IEEE Transactions on Pattern Analysis and Machine Intelligence, July 2013, pp. 1660-1673, vol. 35, DOI, ArXiv.

Neural ODEs and Continuous normalizing flows

  1. Ricky T. Q. Chen et al., Neural Ordinary Differential Equations // NIPS, 2018, ArXiv, source paper and code
  2. Johann Brehmera and Kyle Cranmera, Flows for simultaneous manifold learning and density estimation // NIPS, 2020, ArXiv
  3. Flows at deepgenerativemodels.github.io
  4. Flow-based deep generative models
  5. Variational Inference with Normalizing Flows (source paper, Goes to BME)
  6. Знакомство с Neural ODE на хабре, W: Flow-based generative model

Continous time representation

  1. Самохина Алина, Непрерывное представление времени в задачах декодирования сигналов (магистерская диссертация): 2021 PDF, GitHub
  2. Aaron R Voelker, Ivana Kajić, Chris Eliasmith, Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks // NIPS, 2019, PDF,PDF.
  3. Functional data analysis: splines

Navier-Stokes equations and viscous flow

  1. Neural PDE
  2. Newtonian and Non-Newtonian Fluids in Pipe Flows Using Neural Networks [1], [2]

Metric tensors and kernels

  1. Lynn Houthuys and Johan A. K. Suykens, Tensor Learning in Multi-view Kernel PCA // ICANN 2018, pp 205-215, DOI.

fRMI, Riemannian geometry on shapes

  1. Xavier Pennec, Stefan Sommer, and Tom Fletcher, Riemannian Geometric Statistics in Medical Image Analysis, 2019 book
  2. Surface differential geometry Coursera code video for Image and Video Processing

Spatial time series alignment

  1. Titouan Vayer et al., Time Series Alignment with Global Invariances, 2020,ArXiv
  2. Marco Cuturi and Mathieu Blondel, Soft-DTW: a Differentiable Loss Function for Time-Series, ArXiv
  3. Marcel Campen et al., Scale-Invariant Directional Alignment of Surface Parametrizations // Eurographics Symposium on Geometry Processing, 2016, 35(5), DOI
  4. Helmut Pottmann et al. Geodesic Patterns // ACM Transactions on Graphics, 29(4), DOI, PDF

Reproducing kernel Hilbert space

  1. Mauricio A. Alvarez, Lorenzo Rosasco, Neil D. Lawrence, Kernels for Vector-Valued Functions: a Review, 2012, ArXiv
  2. Pedro Domingos, Every Model Learned by Gradient Descent Is Approximately a Kernel Machine, 2020, ArXiv
  3. Wikipedia: RKHS
  4. Aronszajn, Nachman (1950). "Theory of Reproducing Kernels". Transactions of the American Mathematical Society. 68 (3): 337–404. DOI.

Convolutions and Graphs

  1. Gama, F. et al. Graphs, Convolutions, and Neural Networks: From Graph Filters to Graph Neural Networks // IEEE Signal Processing Magazine, 2020, 37(6), 128-138, DOI.
  2. Zhou, J. et al. Graph neural networks: A review of methods and applications // AI Open, 2020, 1: 57-81, DOI, ArXiv.
  3. Zonghan, W. et al. A Comprehensive Survey on Graph Neural Networks // IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(1): 4-24, DOI, ArXiv.
  4. Zhang, S. et al. Graph convolutional networks: a comprehensive review // Computational Social Networks, 2019, 6(11), DOI.
  5. Xie, Y. et al. Self-Supervised Learning of Graph Neural Networks: A Unified Review // ArXiv.
  6. Wikipedia: Laplacian matrix, Discrete Poisson's equation, Graph FT
  7. GNN papers collection

Higher order Fourier transform

  1. Zongyi Li et al., Fourier Neural Operator for Parametric Partial Differential Equations // ICLR, 2021, ArXiv
  2. Fourier for fun and practice 1D Fourier Code
  3. Fourier for fun and practice nD
    • Fourier analysis on Manifolds 5G page 49
    • Spectral analysis on meshes

Spherical Regression

  1. Shuai Liao, Efstratios Gavves, Cees G. M. Snoek, Spherical Regression: Learning Viewpoints, Surface Normals and 3D Rotations on N-Spheres // CVPR, 2019, 9759-9767, ArXiv

Category theory

  1. Tai-Danae Bradley, What is Applied Category Theory?, 2018, ArXiv, demo.
  2. F. William Lawvere, Conceptual Mathematics: A First Introduction to Categories, 2011, PDF.
  3. Картан А. Дифференциальное исчисление. Дифференциальные формы, 1971 lib.mipt.ru
  4. Wikipedia: Homology, Topological data analysis

Geometric algebra

  1. experior product and quaternions
  2. Nick Lucid, Advanced Theoretical Physics, 2019, sample.

Lab works

Lab work 1

Tensor approximation. Approximate the 2, 3 and 4 index matrices using low-rank decompositions, linear and non-linear. The data sets are: a picture, a short animation movie (basic variant), a sound spectrogram, an fMRI. Plot the sequence of data approximations with ranks 1,..,n. Plot the error: x-axis is the rank, y-axis is the approximation error. Plot the variance of the error for various samples of data, if possible.

  • Code [Tucker, PARAFAC-CANDECOMP, Tensor-train, and variants]
  • Data sets [Animation, Sound, fMRI, and variants]

Lab work 2

PCA on higher orders. Construct a linear map of pairwise distance matrix to a space of lower dimensionality and plot the data. There are two possibilities: high-rank dimensional reduction and order reduction. If you could demonstrate the order reduction, it would be great. The pictures are appreciated.

  • Code [Tucker, Tensor-train, and variants]
  • Data sets [Distances between Digits, and simpler than Animation, Sound, fMRI, and variants]

Lab work 3

Tree-dimansional image reconstruction. Use sonograms of the computed tomography scans. Show the sequential approximation from lab work 1 and plots.

  • Code [one or two variants, linear and NN]
  • Data [Sinogram]
Личные инструменты