tests.system.providers.amazon.aws.example_glue

Module Contents

Functions

get_role_name(arn)

delete_logs(job_id, crawler_name)

Glue generates four Cloudwatch log groups and multiple log streams and leaves them.

glue_cleanup(crawler_name, job_name, db_name)

Attributes

DAG_ID

ROLE_ARN_KEY

sys_test_context_task

EXAMPLE_CSV

EXAMPLE_SCRIPT

test_context

test_run

tests.system.providers.amazon.aws.example_glue.DAG_ID = example_glue[source]
tests.system.providers.amazon.aws.example_glue.ROLE_ARN_KEY = ROLE_ARN[source]
tests.system.providers.amazon.aws.example_glue.sys_test_context_task[source]
tests.system.providers.amazon.aws.example_glue.EXAMPLE_CSV = Multiline-String[source]
Show Value
1apple,0.5
2milk,2.5
3bread,4.0
tests.system.providers.amazon.aws.example_glue.EXAMPLE_SCRIPT = Multiline-String[source]
Show Value
1from pyspark.context import SparkContext
2from awsglue.context import GlueContext
3
4glueContext = GlueContext(SparkContext.getOrCreate())
5datasource = glueContext.create_dynamic_frame.from_catalog(
6             database='{db_name}', table_name='input')
7print('There are %s items in the table' % datasource.count())
8
9datasource.toDF().write.format('csv').mode("append").save('s3://{bucket_name}/output')
tests.system.providers.amazon.aws.example_glue.get_role_name(arn)[source]
tests.system.providers.amazon.aws.example_glue.delete_logs(job_id, crawler_name)[source]

Glue generates four Cloudwatch log groups and multiple log streams and leaves them.

tests.system.providers.amazon.aws.example_glue.glue_cleanup(crawler_name, job_name, db_name)[source]
tests.system.providers.amazon.aws.example_glue.test_context[source]
tests.system.providers.amazon.aws.example_glue.test_run[source]

Was this entry helpful?