Containerizing & Scaling Luigi in Kubernetes
July 26, 2019 by
July 30, 2019 GIPHY HQ – New York, NY
At GIPHY we love data because it lets us make data-driven decisions that improve the quality of results our users get from our search engine. Recently, GIPHY moved their Luigi pipelines from legacy infrastructure on managed AWS EC2 instances to a new containerized ecosystem inside of Kubernetes to increase the overall latency of our most critical ETLs.
In this talk, Walter will share the exact limitations of GIPHY’s old infrastructure, how they identified these bottlenecks, and how decided to overhaul their system as they containerized it. Additionally, the speaker will cover how they redesigned Luigi pipelines to work around its core limitations while maximizing its strengths.
Register for free on the official New York Data Science meetup page!