Section outline

  • Optimizing Deep Learning Systems for Hardware

    Skill Level: Intermediate; suitable for participants with prior experience in training deep neural networks.

    Language: English

    Workload: 

    • Entirely theoretical; no installation, setup, or configuration required.
    • Includes an introduction to HPC frameworks (DeepSpeed) at the end.
    • Optional hands-on direction: installing and experimenting with DeepSpeed after the course.

    Topic: Deep Learning

    Overview 

    Deep learning has revolutionized fields from computer vision to natural language processing, but its success depends as much on computational efficiency as on model accuracy. As models grow larger and more complex, the ability to train and deploy them efficiently across different hardware platforms becomes a critical skill. This course explores the hardware-software co-design principles that underpin modern deep learning systems. We begin with core performance concepts and hardware fundamentals, then move through model-level and system-level optimization strategies, from pruning and quantization to parallelism and mixed-precision training. You'll gain insight into how deep learning workloads are executed on CPUs, GPUs, TPUs, and specialized accelerators, and understand the trade-offs across edge devices, datacenters, and high-performance computing clusters. The course concludes with a look at scaling frameworks like DeepSpeed and future directions in AI infrastructure. Whether you're optimizing for speed, memory, energy, or cost, this course will equip you with the tools to build efficient and scalable AI systems.

    Who Should Enroll: 

    • Graduate students, researchers, and professionals interested in deep learning efficiency, system-level optimization, and hardware-aware model design.
    • Those seeking to understand trade-offs between hardware, memory, and computation in DL training and inference.

    Prerequisites:

    • Experience training deep neural networks (e.g., at least an introductory DL course).
    • Familiarity with computer architecture concepts is recommended but optional.
    • Basic knowledge of deep learning model optimization is helpful.

    Tools, libraries, frameworks used: 

    • The course is theoretical; no hands-on software required.
    • Concepts include DeepSpeed (introduction and configuration overview).
    • Discussion may reference GPUs, CPUs, TPUs, FPGAs, and HPC environments.

    Learning Objectives

    • Understand how hardware and system design impact deep learning performance and efficiency.
    • Learn strategies to optimize models and systems, covering memory usage, computation, and precision.
    • Explore scaling concepts for deploying deep learning across edge, cloud, and HPC environments

    Course Description

    The course is structured into five parts:

    Introduction 

    Part I: Fundamentals

    1. Why hardware matters in deep learning: training vs inference bottlenecks
    2. Performance metrics: FLOPs, latency, throughput, energy, cost
    3. Case studies: edge devices vs datacenter vs supercomputers

    Part II: Hardware & Memory Hierarchy

    1. Existing Solutions: CPU, GPU, TPU, FPGA, ASIC basics
    2. Memory hierarchy, bandwidth bottlenecks, and data movement costs
    3. Precision

    Part III: Model-Level Optimizations

    1. Model compression: pruning, quantization, knowledge distillation
    2. Efficient architectures.

    Part IV: System-Level Optimizations

    1. Parallelism: data, model, pipeline, tensor parallelism
    2. Mixed-precision training (AMP, bfloat16)
    3. Other System Optimizations

    Part V: Scaling Deep Learning in HPC

    1. Introduction to DeepSpeed.
    2. Conclusions and Directions

    About the instructor

    Erdem Akagündüz is an Associate Professor at the Graduate School of Informatics, Middle East Technical University (METU), and a principal investigator at the Applied Intelligence Research Laboratory (AIRLab). His research interests include computer vision, deep learning, pattern recognition, image processing, machine learning, object tracking, and 3D modeling, with numerous journal publications, conference papers, and international patents in these areas. After completing his Ph.D. at METU Electrical and Electronics Engineering and conducting post-doctoral research at the University of York, he worked as a Computer Vision Scientist at ASELSAN Inc, focusing on real-time image processing and intelligent decision systems. He later held an academic position at Çankaya University before joining METU in his current role.