DATA ENGINEERING

Data Engineering Course Syllabus and Content Delivery Plan
By Vigor EdTech

Course Overview

This course is designed to equip learners with the theoretical knowledge and practical skills required to build, manage, and optimize data pipelines for large-scale data processing and analytics. The program covers foundational concepts, hands-on training in modern tools and technologies, and advanced topics such as cloud data engineering and big data systems. This course is ideal for aspiring data engineers, software developers, and data analysts looking to transition into data engineering roles.

Course Duration

• Total Duration: 12 Weeks


• Weekly Commitment: 8-10 hours (including lectures, assignments, and projects)


• Delivery Mode: Online (live + recorded sessions) and optional offline workshops

Syllabus

Module 1: Introduction to Data Engineering

Duration: 1 Week

Topics Covered:

• Role of Data Engineers in the Data Ecosystem


• Overview of Data Engineering Tools and Technologies


• Understanding Data Pipelines


• Key Concepts: ETL vs. ELT, Batch Processing, and Stream Processing

Hands-On: Setting up the development environment (Python, SQL, Docker)

Module 2: Relational Databases and SQL

Duration: 2 Weeks

Topics Covered:

• Fundamentals of Relational Databases


• Advanced SQL Queries (Joins, Subqueries, Window Functions)


• Database Design and Normalization


• Working with Popular Databases: MySQL, PostgreSQL

Hands-On: Writing complex SQL queries, designing database schemas

Module 3: Data Warehousing

Duration: 2 Weeks

Topics Covered:

• Concepts of Data Warehousing and Data Lakes


• Star and Snowflake Schema Design


• Introduction to OLAP and OLTP


• Popular Tools: Amazon Redshift, Google BigQuery, Snowflake

Hands-On: Designing and querying a data warehouse on cloud platforms

Module 4: Data Pipeline Development

Duration: 3 Weeks

Topics Covered:

• Building ETL/ELT Pipelines


• Introduction to Apache Airflow


• Automating Data Workflows


• Handling Data Quality Issues


• Managing Dependencies and Monitoring Pipelines

Hands-On: Building and scheduling data pipelines with Airflow

Module 5: Big Data Systems

Duration: 2 Weeks

Topics Covered:

• Introduction to Big Data Concepts


• Hadoop Ecosystem Overview


• Introduction to Apache Spark


• Distributed Computing and Parallel Processing

Hands-On: Writing Spark jobs for big data processing

Module 6: Cloud Data Engineering

Duration: 2 Weeks

Topics Covered:

• Cloud Platforms Overview: AWS, GCP, Azure


• Data Storage and Management in the Cloud


• Serverless Data Processing with AWS Lambda and GCP Dataflow


• Managing Data with S3, Azure Blob, BigQuery

Hands-On: Implementing data engineering solutions on AWS/GCP

Module 7: Advanced Topics and Optimization

Duration: 1 Week

Topics Covered:

• Data Governance and Compliance (GDPR, CCPA)


• Performance Tuning in Data Pipelines


• Data Partitioning and Clustering


• Real-Time Data Processing with Kafka and Flink

Hands-On: Optimizing existing data pipelines for performance and scalability

Capstone Project

Duration: 2 Weeks (Runs Parallel to Modules 6 and 7)**

Deliverables:

• Real-world problem statement


• End-to-end data engineering solution (data ingestion, processing, and storage)


• Comprehensive project report and presentation

Examples:

• Building a data pipeline for e-commerce analytics


• Real-time streaming analytics system for financial data

Content Delivery Plan

1. Learning Methodology

• Live Sessions: Weekly interactive sessions with industry experts (2 hours/session)


• Recorded Content: Pre-recorded tutorials for self-paced learning


• Hands-On Practice: Guided coding exercises and notebooks


• Discussion Forums: Dedicated peer and mentor Q&A sessions

2. Assignments and Quizzes

• Weekly assignments based on real-world use cases


• Quizzes to assess conceptual understanding and practical skills

3. Practical Use Cases

• Industry-relevant datasets for practical assignments


• Use-case-driven projects to simulate real-world scenarios

4. Mentorship and Support

• One-on-one mentorship for personalized guidance


• Regular feedback on assignments and capstone projects

5. Certification

• Certification of Completion from Vigor EdTech


• Portfolio-ready projects to showcase to potential employers

Target Audience

• Software Developers transitioning to Data Engineering roles


• Data Analysts looking to enhance technical skills


• Students (graduate and postgraduate) interested in Data Engineering


• IT Professionals aiming to upskill in Big Data and Cloud technologies

Pre-requisites

• Basic programming knowledge (Python preferred)


• Understanding of SQL and relational databases


• Familiarity with basic data concepts (optional)

Tools and Technologies

• Programming: Python, SQL


• Databases: MySQL, PostgreSQL


• Big Data: Hadoop, Spark


• Orchestration: Apache Airflow


• Cloud Platforms: AWS (S3, Redshift), GCP (BigQuery, Dataflow)


• Others: Docker, Git, Kafka

For inquiries and enrollment, visit vigoredtech.in