airflow.executors.local_executor

LocalExecutor

See also

For more information on how the LocalExecutor works, take a look at the guide: Local Executor

Module Contents

airflow.executors.local_executor.ExecutorWorkType[source]
class airflow.executors.local_executor.LocalWorkerBase(result_queue: 'Queue[TaskInstanceStateType]')[source]

Bases: multiprocessing.Process, airflow.utils.log.logging_mixin.LoggingMixin

LocalWorkerBase implementation to run airflow commands. Executes the given command and puts the result into a result queue when done, terminating execution.

Parameters

result_queue -- the queue to store result state

run(self)[source]
execute_work(self, key: TaskInstanceKey, command: CommandType)[source]

Executes command received and stores result state in queue.

Parameters

  • key -- the key to identify the task instance

  • command -- the command to execute

do_work(self)[source]

Called in the subprocess and should then execute tasks

class airflow.executors.local_executor.LocalWorker(result_queue: 'Queue[TaskInstanceStateType]', key: TaskInstanceKey, command: CommandType)[source]

Bases: airflow.executors.local_executor.LocalWorkerBase

Local worker that executes the task.

Parameters

  • result_queue -- queue where results of the tasks are put.

  • key -- key identifying task instance

  • command -- Command to execute

do_work(self)[source]
class airflow.executors.local_executor.QueuedLocalWorker(task_queue: 'Queue[ExecutorWorkType]', result_queue: 'Queue[TaskInstanceStateType]')[source]

Bases: airflow.executors.local_executor.LocalWorkerBase

LocalWorker implementation that is waiting for tasks from a queue and will continue executing commands as they become available in the queue. It will terminate execution once the poison token is found.

Parameters

  • task_queue -- queue from which worker reads tasks

  • result_queue -- queue where worker puts results after finishing tasks

do_work(self)[source]
class airflow.executors.local_executor.LocalExecutor(parallelism: int = PARALLELISM)[source]

Bases: airflow.executors.base_executor.BaseExecutor

LocalExecutor executes tasks locally in parallel. It uses the multiprocessing Python library and queues to parallelize the execution of tasks.

Parameters

parallelism -- how many parallel processes are run in the executor

class UnlimitedParallelism(executor: 'LocalExecutor')[source]

Implements LocalExecutor with unlimited parallelism, starting one process per each command to execute.

Parameters

executor -- the executor instance to implement.

start(self)[source]

Starts the executor.

execute_async(self, key: TaskInstanceKey, command: CommandType, queue: Optional[str] = None, executor_config: Optional[Any] = None)[source]

Executes task asynchronously.

Parameters

  • key -- the key to identify the task instance

  • command -- the command to execute

  • queue -- Name of the queue

  • executor_config -- configuration for the executor

sync(self)[source]

Sync will get called periodically by the heartbeat method.

end(self)[source]

This method is called when the caller is done submitting job and wants to wait synchronously for the job submitted previously to be all done.

class LimitedParallelism(executor: 'LocalExecutor')[source]

Implements LocalExecutor with limited parallelism using a task queue to coordinate work distribution.

Parameters

executor -- the executor instance to implement.

start(self)[source]

Starts limited parallelism implementation.

execute_async(self, key: TaskInstanceKey, command: CommandType, queue: Optional[str] = None, executor_config: Optional[Any] = None)[source]

Executes task asynchronously.

Parameters

  • key -- the key to identify the task instance

  • command -- the command to execute

  • queue -- name of the queue

  • executor_config -- configuration for the executor

sync(self)[source]

Sync will get called periodically by the heartbeat method.

end(self)[source]

Ends the executor. Sends the poison pill to all workers.

start(self)[source]

Starts the executor

execute_async(self, key: TaskInstanceKey, command: CommandType, queue: Optional[str] = None, executor_config: Optional[Any] = None)[source]

Execute asynchronously.

sync(self)[source]

Sync will get called periodically by the heartbeat method.

end(self)[source]

Ends the executor. :return:

terminate(self)[source]

Terminate the executor is not doing anything.

Was this entry helpful?