Where do you view the output from airflow jobs

In the airflow tutorial, the BashOperators have output (via echo). If the task runs in the scheduler, where do you view the output? Is there a console or something? I'm sure I'm just not looking in the right place.

BashOperator

echo

1 Answer
1

Like @tobi6 said, you can view the output from your DAG runs in your webserver or in your console depending on the environment.

To do so in your webserver:

Graph View

View Log

If you want to view the logs from your run, you do so in your airflow_home directory.

airflow_home

Users can specify a logs folder in airflow.cfg. By default, it is in the AIRFLOW_HOME directory.

In addition, users can supply a remote location for storing logs and log backups in cloud storage. At this time, Amazon S3 and Google
Cloud Storage are supported. To enable this feature, airflow.cfg must
be configured as in this example:

[core]
# Airflow can store logs remotely in AWS S3 or Google Cloud Storage. Users
# must supply a remote location URL (starting with either 's3://...' or
# 'gs://...') and an Airflow connection id that provides access to the storage
# location.
remote_base_log_folder = s3://my-bucket/path/to/logs
remote_log_conn_id = MyS3Conn
# Use server-side encryption for logs stored in S3
encrypt_s3_logs = False
Remote logging uses an existing Airflow connection to read/write logs. If you don’t have a connection properly setup, this will fail.


In the above example, Airflow will try to use S3Hook('MyS3Conn').

In the Airflow Web UI, local logs take precedance over remote logs. If local logs can not be found or accessed, the remote logs will
be displayed. Note that logs are only sent to remote storage once a
task completes (including failure). In other words, remote logs for
running tasks are unavailable. Logs are stored in the log folder as
{dag_id}/{task_id}/{execution_date}/{try_number}.log.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.