Selinon - dynamic distributed task flows
Selinon is a task flow manager that enhances Celery and gives you an ability to create advanced task flows serving large workflows in your cluster.
Tags: Big Data, Infrastructure, Parallel Programming, Programming, Python
Scheduled on wednesday 14:50 in room lecture
Speaker
Software engineer at Red Hat with focus on Python, machine learning, scalable systems and big data processing.
Description
Have you ever tried to define and process complex workflows for data processing? If the answer is yes, you might have struggled to find the right framework for that. You've probably came across Celery - popular task flow management for Python. Celery is great, but it does not provide enough flexibility and dynamic features needed badly in complex flows. As we discovered all the limitations, we decided to implement Selinon.
Have you ever tried to define and process complex workflows for data processing? If the answer is yes, you might have struggled to find the right framework for that. You've probably came across Celery - popular task flow management for Python. Celery is great, but it does not provide enough flexibility and dynamic features needed badly in complex flows. As we discovered all the limitations, we decided to implement Selinon.
Selinon enhances Celery task flow management and allows you to create and model task flows in your distributed environment that can dynamically change behavior based on computed results in your cluster, automatically resolve tasks that need to be executed in case of selective task runs, automatic tracing mechanism and many others.