DatabricksWorkflowPlugin

Overview

The DatabricksWorkflowPlugin enhances the Airflow UI by providing links for tasks that allow users to view the Databricks job run in the Databricks workspace. Additionally, it offers links to repair task(s) within the workflow.

Features

  • Task-Level Links: Within the workflow, each task includes links to the job run and a repair link for the individual task.

  • Workflow-Level Links: At the workflow level, for the job launch task, the plugin provides a link to repair all failed tasks and a link to the job run(allows users to monitor the job in the Databricks account) in the Databricks workspace.

Examples

  • Job Run Link and Repair link for Single Task:

../_images/workflow_plugin_single_task.png

  • Workflow-Level Links to the job run and to repair all failed tasks:

../_images/workflow_plugin_launch_task.png

Notes

Databricks does not allow repairing jobs with single tasks launched outside the workflow. Hence, for these tasks, only the job run link is provided.

Usage

Ideally, installing the provider will also install the plugin, and it should work automatically in your deployment. However, if custom configurations are preventing the use of plugins, ensure the plugin is properly installed and configured in your Airflow environment to utilize its features. The plugin will automatically detect Databricks jobs, as the links are embedded in the relevant operators.

Was this entry helpful?