Time Series Forecasting I

Materials INFO

• Semester: Fall, 2025
• This page provides links to papers only.

CONTENTS

• [Week 1]
  • MLP-Mixer: An all-MLP Architecture for Vision (2021) (link)
    • Introduces MLP-Mixer.
    • Achieves state-of-the-art image classification without self-attention.
    • A basic Transformer has complexity $O(n^2)$, whereas an MLP has $O(n)$.
  • MLP4Rec: A Pure MLP Architecture for Sequential Recommendation (2022) (link)
    • Applies the MLP-Mixer architecture to sequential recommendation.
  • SMLP4Rec: An Efficient All-MLP Architecture for Sequential Recommendations (2024) (link)
    • Improves both the efficiency and accuracy of MLP4Rec.
  • Self-Attentive Sequential Recommendation (2018) (link)
    • A baseline for Transformer-based sequential recommendation.
  • Deep Learning Recommendation Model for Personalization and Recommendation Systems (2019) (link)
    • Facebook AI
    • Implementation of a large-scale recommendation model.
• [Week 2]
  • Prediction with Time-Series Mixer for the S&P500 Index (2024) (link)
    • TS-Mixer exhibits competitive performance regarding S&P500 Index prediction
    • Ordinary MLP-Mixers are known to perform poorly on autoregressive forecasting and lack long-term dependencies.
  • FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting (2022) (link)
    • A Transformer that captures long-term dependencies in the frequency domain, addressing Mixer limitations.
  • MP3Net:Multi-scale Patch Parallel Prediction Networks for Multivariate Time Series Forecasting (2024) (link)
    • modeling long-term dependencies in time series analysis
    • multi-scale patch module to extract local features and long-term correlations from the time series
  • TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis (2023) (link)
    • multi-scale prediction
  • Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting (2020) (link)
    • Developing an effective Transformer-based time-series forecasting model.
• [Week 3]
  • FourierFormer: Transformer Meets Generalized Fourier Integral Theorem (2022) (link)
    • Interprets existing Transformers as implicitly assuming that queries follow a Gaussian mixture distribution.
    • Extends this idea to propose a Transformer that operates under relaxed assumptions.
  • Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting (2021) (link)
    • Long-term dependency and progressive decomposition of time series.
  • Perceiver: General Perception with Iterative Attention (2021) (link)
    • The model leverages an asymmetric attention mechanism to iteratively distill inputs into a tight latent bottleneck, allowing it to scale to handle very large inputs.
  • Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021) (link)
    • Proposes Perceiver IO, a general-purpose architecture that scales linearly with input and output sizes.
  • General-purpose, long-context autoregressive modeling with Perceiver AR (2022) (link)
    • Developing an effective Transformer-based time-series forecasting model.
• [Week 4]
  • Efficiently Modeling Long Sequences with Structured State Spaces (2021) (link)
    • Structured state-space models and time-series modeling (S4).
  • Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2024) (link)
    • Combines gating and state-space models for selective memory.
    • Performs well on time series thousands of steps long.
  • RWKV: Reinventing RNNs for the Transformer Era (2023) (link)
    • A hybrid of RNNs and Transformers that can use large models.
  • Rethinking Attention with Performers (2021) (link)
    • Approximates the softmax attention kernel.
    • Efficiently models a variety of kernelizable attention mechanisms beyond softmax.
  • Linformer: Self-Attention with Linear Complexity (2020) (link)
    • Saves memory using low-rank attention.
• [Week 5]
  • Pay Attention to MLPs (2021) (link)
    • Argues that self-attention is not essential in vision Transformers.
    • Shows that gMLP can achieve equivalent accuracy.
  • FFNet: Frequency Fusion Network for Semantic Scene Completion (2022) (link)
    • Many current methods use RGB-D images to capture geometric and semantic information about objects.
    • Proposes a Frequency Fusion Network (FFNet) to make better use of RGB-D data and improve semantic scene completion.
  • Deep Equilibrium Model (DEQ) (2019). (link)
    • Observing that hidden layers in many deep sequence models converge to a fixed point, the DEQ approach finds that equilibrium directly by root-finding.
    • Claims that only constant memory is needed for training and prediction regardless of the network’s effective depth.
  • Kernel Neural Operators (KNOs) for Scalable, Memory-efficient, Geometrically-flexible Operator Learning (2024) (link)
    • KNO uses a parameterized, closed-form, finitely smooth, compactly supported kernel inside an integral operator and learnable sparsity parameters. This greatly reduces the number of parameters compared with existing neural operators.
    • KNO presents a new paradigm of low memory use, geometric flexibility, and deep operator learning.
  • Token Shift Transformer for Video Classification (2021) (link)
    • A Transformer encoder uses pairwise self-attention, which incurs a large computational cost for complex three-dimensional video signals.
    • Introduces the Token Shift Module (TokShift), a parameter-free, zero-FLOP operator for modeling temporal relationships in Transformer encoders.
• [Week 6]
  • Frequency-domain MLPs are More Effective Learners in Time Series Forecasting (2023) (link)
    • Applies MLPs in the frequency domain to capture global dependencies and compact key frequency components for forecasting
  • Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictors (2023) (link)
    • Uses Koopman embedding and context-aware operators to disentangle and predict time-variant/invariant dynamics in real time
  • Deep State Space Models for Time Series Forecasting (2018) (link)
    • Describes token-to-token computation structures within a single framework.
    • Explains the strong performance of attention in terms of token mixing.
    • Proposes PoolFormer, which replaces the attention module in a Transformer with a simple spatial-pooling operation that performs extremely simple token mixing.
  • Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting (2020) (link)
    • Fuses Graph Fourier Transform and Discrete Fourier Transform in a single GNN to jointly learn inter-series correlations and temporal dependencies
  • CROSSFORMER: TRANSFORMER UTILIZING CROSS- DIMENSION DEPENDENCY FOR MULTIVARIATE TIME SERIES FORECASTING (2023) (link)
    • CROSSFORMER introduces Dimension-Segment-Wise embedding and two-stage (cross-time & cross-dimension) attention for hierarchical multivariate forecasting
  • iTransformer: Inverted Transformers Are Effective for Time Series Forecasting (2024) (link)
    • iTransformer inverts tokenization to embed each variate separately, then applies vanilla Transformer to learn multivariate correlations and support arbitrary lookback windows
• [Week 7]
  • Token Pooling in Vision Transformers (2021) (link)
    • Reduces computation by pooling tokens in intermediate layers.
  • Efficient Time Series Processing for Transformers and State-Space Models through Token Merging (2024) (link)
    • Applies token merging to time series.
  • MetaFormer Is Actually What You Need for Vision (2022) (link)
    • Describes token-to-token computation structures within a single framework.
    • Explains the strong performance of attention in terms of token mixing.
    • Proposes PoolFormer, which replaces the attention module in a Transformer with a simple spatial-pooling operation that performs extremely simple token mixing.
  • Conditional Neural Processes (2018) (link)
    • Retains the flexibility of probabilistic processes such as Gaussian processes while using a neural-network architecture trained by gradient descent.
  • Linformer: Self-Attention with Linear Complexity (2020) (link)
    • Saves memory using low-rank attention.
  • Fusing Large Language Models with Temporal Transformers for Time Series Forecasting (2025) (link)
    • Learns representations of time-series patterns using an LLM (GPT-2, LoRA, and numeric representations).
    • Transformer + ResNet: uses both LLM representations and time-series values as input.
    • Further research: investigate methods for transforming numerical tokens.