ServerlessDays Logo

23 May 2018

LABS Azrieli Sarona, TLV

One Day. One Track. One Community.

Or Hiltch: Serverless ETLs at Skyline AI

Or Hiltch

Or Hiltch, Co-Founder & CTO at Skyline AI

Previously Co-Founder and CTO at StreamRail (acquired by ironSource), Former research team lead at AVG Anti-Virus. Author of several popular open-source projects in the golang community.

YouTube // Slides // Cover image

How we manage execution our army of ETLs using Serverless technology

The data science team at Skyline AI needs to rely on a single hygienic data layer, which is easily accessible using standard SQL and can process PBs of data quickly, is always up-to-date and that contains data after it has been well-processes by our army of ETLs. To manage this army of ETLs, we have developed an in-house DAG execution scheduler in golang that schedules and runs ETLs on serverless infrastructure, resulting in better ETL isolation, performance and significant cost saving. In this session we'll walk through a short demo of this service, and will demonstrate how easy and effective serverless execution of docker-contained ETL applications could become.

Sponsors

Binaris Logo
Puresec Logo
Epsagon Logo
AWS Logo
Wix Logo
Microsoft Logo
Microsoft Logo
tlv@serverlessdays.io @SrvrlessDaysTLV

Jeff woz here