OpenStack LiveBlog – TaskFlow and OpenStack

Note: This is a live blog, I've not spent time editing, formatting, etc

TaskFlow answers the “Statemanagement” issue.

  • Workflows w/o state makes recovery an issue.
  • Also reliablility can be an issue.
  • Race conditions
  • Manager <-> API Boundary
  • undefined states

Why does state matter?

  • Reliability, consistency
  • Upgrades, or recovering from the upgrades
  • Easy to understand workflows
It is on Stackforge, well supported by the community.

TaskFlow is not:

 

  • A webservice, no API (see: minstral)
  • Doesn't fix everything, no rainbow ponies yet.
  • Still needs careful coding
Controlled execution, and persistance, also how to recover from partial failure. Controlled execution using an engine.

The discrete component of TaskFlow is a “task” those are built into flows. The engine translates the flow into simpler things, a compiler of sorts. Following that, are a number of state transitions a flow will progress through at execution time. This provides a deep level for state analysis.

 

Tasks:

  • Executes and reverts one action
  • Recieves inputs, declares outputs

 

Flows:

  • Composes tasks into useful structures (also subflows)
  • Imposes definition of order onto the tasks and subflows
  • linear, unordered, or toopological
Engines

  • Runs your flow, reliable resumable manner
  • well defined transitions
  • allows deployers/developers of a service to select an engine to fit the setup
  • backed by different setups
  • — Single / Multi-threaded
  • — Distributed (WIP)
“Is TaskFlow a general use Workflow tool or task specific?” – It is built to be generic, however it was built with #openstack in mind.

TaskFlow != Minstrel, however, that is subject to change #openstack

 

“Is TaskFlow similar to the Amazon workflow service?” – It can be used in that way, but that's not what its targeted at #openstack

 

“You allow for parallel action, do you allow for synchronization” – We'll see how the greater #openstack goes and TaskFlow will follow

 

“At what point do you write state” – At each transition, and when a task returns info #openstack

 

@rackninja: Taskflow is a general workflow library. The hope is that it will consumed by #openstack services as the standard eventually

 

Persistance Layer

  • Task state / progress, can be a DB, or filesystem, or in memory
  • Allows reconstruction of flows and tasks
  • Can view the step by step actions of a workflow… could be useful for audits and troubleshooting
Jobs / Job Board

  • Initial and deriviave tasks required to fulfill an action
  • Can be moved to another worker for execution or so
  • Jobs automatically posted, completed, claimed, etc.
  • Backed by MQ, Zookeeper, etc
That which is not Sci-Fi
  • Release 0.1
  • Tagged and pypi
  • Foundational concepts, tasks, flows, persistance, etc
  • documentation
That which is Sci-Fi
  • Distributed engine
  • lock service
  • zookeeper
  • job/job board

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.