Tutorials

Presenters: Shao-Lun Huang (Tsinghua Shenzhen International Graduate School, China) and Yingbin Liang (Ohio State University, USA)

Large Language Models (LLMs) and transformers, as the core architectures underlying today’s frontier generative AI models, have recently revolutionized a wide range of machine learning (ML) applications, including natural language processing (NLP), computer vision, and robotics. Alongside their tremendous experimental successes, theoretical studies from information theory, statistics, and optimization theory, have also emerged to explain the generalizability of LLMs, and why transformers can be trained to achieve fantastic performance, which lead to performance guarantees and algorithm design guidance in practice. In particular, this tutorial aims to provide an overview of these recent theoretical investigations that have characterized the fundamental theory of In-Context Learning (ICL), as well as the training dynamics of transformer-based ML models in Chain-of-Thought (CoT) reasoning. Additionally, the tutorial will explain the primary techniques and tools employed for such analyses which leverage various information theoretical concepts and tools in addition to learning theory, stochastic optimization, dynamical systems, probability, etc. Such techniques not only provide theoretical insights and performance guarantees of LLMs, but also offer design guidance for more effective and interpretable LLM algorithms in contemporary data analytics.

Presenters:Yuxin Chen (University of Pennsylvania, USA), Qing Qu (University of Michigan, USA), Liyue Shen (University of Michigan, USA), Yuting Wei (University of Pennsylvania, USA)

Diffusion models have recently gained attention as a powerful class of deep generative models, achieving state-of-the-art results in data generation tasks. In a nutshell, they are designed to learn an unknown data distribution starting from Gaussian noise, mimicking the process of non-equilibrium thermodynamic diffusion. Despite their outstanding empirical successes, the mathematical and algorithmic foundations of diffusion models remain far from mature. For instance: (i) Generalization: it remains unclear how diffusion models, trained on finite samples, can generate new and meaningful data that differ from the training set; (ii) Efficiency: due to the enormous model capacity and the requirement of many sampling steps, they often suffer from slow training and sampling speeds; (iii) Controllability: it remains computationally challenging and unclear how to guide and control the content generated by diffusion models, limiting its ability of solving inverse problems across many scientific imaging applications, as well as raising challenges regarding controllability and safety.

This tutorial introduces a mathematical framework for understanding the generalization and advancing the efficiency of diffusion models by exploring the low-dimensional structures in both the data and the model. We demonstrate how to overcome fundamental barriers to improve the generalization, efficiency, and controllability of diffusion models by exploring how these models adaptively learn underlying data distributions, achieving faster convergence at the sampling stage, and unveiling the intrinsic properties of the learned denoiser. Leveraging theoretical studies, we will demonstrate how to effectively utilize these properties for guiding the generation of diffusion models for solving scientific problems.

Presenters: Kai Wan (Huazhong University of Science and Technology, China), Yifeng Xiong (Beijing University of Posts and Telecommunications, China), and Fan Liu (Southeast University, China)

Integrated sensing and communication (ISAC), well-recognized as a key enabling technology for future 6G wireless networks, is fundamental to many modern technologies and applications, driving advancements in fields like IoT, smart cities, healthcare, and industrial automation. Overall, ISAC provides significant enhancements in performance and resource efficiency compared to individual sensing and communication systems, primarily attributed to the collaborative use of wireless resources, radio waveforms, and hardware platforms. However, sensing and communication operate on distinct information processing principles. Thus, a number of performance tradeoffs between sensing and communication exist, ranging from information theoretical limits to physical layer performance tradeoffs, and to cross-layer design tradeoffs. Information theory has recently developed some promising tools to characterize the fundamental tradeoffs between sensing and communication in prototypical relevant settings. This tutorial aims to provide a comprehensive understanding on the recent information theory results in ISAC problems, and also to shed light on many important but open problems in the context of sensing and communication tradeoffs. In this half-day tutorial, we will firstly overview the background and application scenarios of ISAC. Then we will provide a brief review on theoretical background for ISAC, and on the evolution of information theoretic results related to ISAC. As a step further, we will present an overview on the fundamental limits of various ISAC models. Then two technical parts will be introduced with details: 1) fundamental tradeoffs between communication and sensing, and 2) applications to realistic signal processing. Finally, we will conclude the tutorial by summarizing the future directions and open problems in the area of information theoretic ISAC.

Presenters: Linglong Dai (Tsinghua University, China) and Merouane Debbah (Khalifa University of Science and Technology, United Arab Emirates)

To significantly improve the system performance of 6G wireless communications, various promising technologies, such as reconfigurable intelligent surfaces (RISs), holographic multiple-input multiple-output (HMIMO), orbital angular momentum (OAM), and near-field communications, have been recently investigated. All these technologies attempt to explore new degrees of freedom (DoF) to achieve performance gains. Actually, the expected performance gains come from more accurate understanding and precise manipulation of electromagnetic fields carrying information. However, the classical information theory abstracts out the physics of the electromagnetic propagation, yielding results within the context of a given simple yet elegant mathematical model at the price of hiding some physical insights. Therefore, integrating classical electromagnetic theory and information theory is of great importance to capture the physically consistent fundamental limit of wireless communications, which leads to the interdisciplinary subject called electromagnetic information theory (EIT).

As an emerging interdisciplinary subject, EIT faces many problems and challenges, such as the establishment of physically consistent transmission models, the corresponding theoretical limits of communication systems, and the possible new designs and paradigms of communication systems. To address these challenges, this tutorial will introduce the latest progress of EIT from both theoretical and practical perspectives. First, this tutorial will introduce the motivations and definitions of EIT. It is developed as a theory that can model and analyze the real-world electromagnetic wireless information system with physically interpretable and mathematically reasonable assumptions. By utilizing stochastic processes, operator theory, Slepian’s concentration problem, and Fredholm determinant, EIT can derive the physically consistent DoFs and mutual information upper bounds of wireless communication systems more accurately. Subsequently, this tutorial will present the techniques enabled and inspired by EIT, respectively. Finally, we will predict the future research trends of EIT.

Presenters: Lingyang Song, Qingyu Liu, Hongliang Zhang, Shuhang Zhang (Peking University, China)

With the rapid growth of the demands in modern wireless applications, the limitations of conventional radio access networks (RAN) in handling the complexity, scalability, and performance demands of wireless networks have become apparent. Recent advancements in generative artificial intelligence (AI), e.g., large foundation models, lead to a significant shift in how wireless networks are designed, managed, and optimized. The integration of generative AI and RAN heralds a transformative era, enabling the development of more adaptive, intelligent, high-performing, efficient, and versatile network systems. Generative AI-enhanced RAN is a key enabler for next-generation wireless networks like 6G, where the complexity and demand for high performance require advanced automation and intelligent management. This tutorial will present the basic concepts/theories for addressing the research advances of generative AI for RAN.

Presenters: Ping Zhang (Beijing University of Posts and Telecommunications, China), Kai Niu (Beijing University of Posts and Telecommunications, China), Shuo Shao (University of Shanghai for Science and Technology, China)

This tutorial presents a survey of semantic information theory and methods beyond Shannon. We will first introduce ComAI, a new communication paradigm that converges communication and native AI. The system architecture of ComAI, inspired by human intelligence, and its key technologies will be subsequently provided, and then the its essences will also be discussed, elaborating the relationships among communication, intelligence, and semantic information.

Then, we focus on semantic information theory, which serves as the core guiding principle for ComAI. We first review the historical development roadmap of semantic information perspectives, followed by an introduction of a synonymy-based viewpoint as our focus. Building on this foundation, we establish semantic information measures, fundamental coding theorems, and corresponding semantic coding limits, and develop a synonymous mapping optimization analysis framework to guide the design and optimization of semantic coding methods for semantic compression and transmission. Together, these results form a unified theoretical system that provides principled support for semantic communications.

Finally, we extend the semantic information theory to lossy coding scenarios where the semantic information and the observable source follows a probabilistic rather than deterministic relationship. The rate distortion function and the finite blocklength analysis of the semantic information theory will be discussed. The quadratic Gaussian case will be leveraged as an illustrative example to show that this line of research has the potential to produce many theoretical results with practical implications.

Presenters: Brendon McBain and Emanuele Viterbo (Monash University, Australia)

DNA-based data storage is an emerging technology with the potential to store the world’s rapidly growing digital information—which approximately doubles each year—while preserving existing data over long time horizons. A central component of any DNA storage system is the readout process (DNA sequencing). This tutorial provides a deep dive into the fundamental operation of the nanopore DNA sequencer and current research on nanopore read channels, which have attracted growing interest in the coding and information theory community due to the rich modelling and decoding challenges they present. The tutorial begins with a brief introduction to DNA data storage, then narrows to nanopore sequencing and covers key topics including nanopore chemistry, simulation and channel modelling, achievable rates, code construction, and decoding algorithms. Open problems in nanopore sequencing will be discussed, and promising research opportunities will be highlighted.