One Day. One Track. One Community.
Previously Co-Founder and CTO at StreamRail (acquired by ironSource), Former research team lead at AVG Anti-Virus. Author of several popular open-source projects in the golang community.
The data science team at Skyline AI needs to rely on a single hygienic data layer, which is easily accessible using standard SQL and can process PBs of data quickly, is always up-to-date and that contains data after it has been well-processes by our army of ETLs. To manage this army of ETLs, we have developed an in-house DAG execution scheduler in golang that schedules and runs ETLs on serverless infrastructure, resulting in better ETL isolation, performance and significant cost saving. In this session we'll walk through a short demo of this service, and will demonstrate how easy and effective serverless execution of docker-contained ETL applications could become.