Ironhack Barcelona - Tech School - An Integration Story | Apache Spark Streaming + Kafta
Monday 17 September 2018, 19:00 - 20:00
C/Pamplona, 96 - 08018, Barcelona, España
Remember to sign up through this link (https://www.eventbrite.com/e/an-integration-story-apache-spark-streaming-kafta-tickets-49884958253) to get your ticket. You are invited to another great meetup organized by BCN Data Engineering. Spark Streaming has supported Kafka since it’s inception, but a lot has changed since those times, both in Spark and Kafka sides, to make this integration more fault-tolerant and reliable.Apache Kafka 0.10 (actually since 0.9) introduced the new Consumer API, built on top of a new group coordination protocol provided by Kafka itself. So a new Spark Streaming integration comes to the playground, with a similar design to the 0.8 Direct DStream approach. However, there are notable differences in usage, and many exciting new features. In this talk, we will cover what are the main differences between this new integration and the previous one (for Kafka 0.8), and why Direct DStreams have replaced Receivers for good. We will also see how to achieve different semantics (at least one, at most one, exactly once) with code examples. Guest Speaker: Joan - Head of Data Engineering at Stuart Delivery Joan design and build tools for a more data centric company. Previously, he worked at Billy Mobile as a Senior Big Data Architect and Tech Lead, where he led the transition to the Hadoop and Spark ecosystem, designing applications that scale up to thousands of events per second. Before that, he was playing around with (shiny new back then) MapReduce and distributed systems at Trovit Search, developing Big Data solutions for programmatic buying in SEM platforms like Google AdWords and Microsoft Bing. Also collaborated with some companies as a freelance consultant to provide Big Data good practices. Agenda: 19:00 Doors open & beers/soda19:10 Brief intro by BCN Data Engineering19:15 Presentation by Joan19:45 Q/A20:00 More beers/soda and network20:30 Wrap
Publicado por: Betabeers