Course Description #

This course introduces data engineering concepts and technologies relevant to development and operations (DevOps). It reviews design principles and development processes for data pipelines in analytics applications, focusing on containerized microservices and cloud-native applications. It reviews data exchange formats, process concurrency control, communication protocols, application programming interfaces, distributed processing, and systems architecture. Students learn about automated deployment and scaling of batch, interactive, and streaming data pipelines. They learn how to design, implement, and maintain applications in cloud and on-premises environments. This is a programming-intensive course that includes a full-stack development project. Recommended prior course: MSDS 431-DL Data Engineering with Go. Prerequisites: Prerequisites: (1) MSDS 400-DL Math for Modelers and (2) MSDS 420-DL Database Systems or CIS 417 Database Systems Design.

Students benefit by taking the Go Learning Studio and MSDS 431 Data Engineering with Go before taking this course.

Course Content across Ten Weeks #

• Week 1. Data Engineering: What’s in a Name?
• Week 2. Interactive, Stream, and Batch Processing
• Week 3. Data Infrastructure Components and Microservice Communication Styles
• Week 4. Concurrency Control for Interacting Processes
• Week 5. Using HTTP Protocol and REST to Build Web Services
• Week 6. Fundamentals of Containerization
• Week 7. Iterative Development Process for Building Containerized Microservices
• Week 8. Automation for Cloud-Native Containerized Applications
• Week 9. DevOps, MLOps, and Data Pipelines
• Week 10. Technology Trends and Future Directions for Infrastructure Orchestration and Microservices Platforms

What is required of students? Students participate in weekly discussion forums and work on challenging Go programming assignments. They must also complete an term project, implementing a data pipeline for business intelligence.

Primary Textbooks #

• Reis, Joe, and Matt Housley. 2022. Fundamentals of Data Engineering. Sebastopol, CA: O’Reilly Media. [ISBN: 978-1098108236]

• Titmus, Matthew A. 2021. Cloud Native Go: Building Reliable Services in Unreliable Environments. Sebastopol, CA: O’Reilly Media. [ISBN: 978-1492076339]

• Kommadi, Bhagvan. 2019. Learn Data Structures and Algorithms with Golang. Birmingham, UK: Packt Publishing. [ISBN: 978-1789618501]

Go to the home page Learning Go for Data Science.