Easiest Way To Schedule A Python Script In AWS Glue As A Job - 2024

Hey guys, In this blog we will see how we can schedule a Python Script in AWS Glue as a Job that will run every hour. I have tried to make this tutorial as easy as possible with each and every step explained.

So without any further due, let’s do it…

Table of Contents

Step 1 – Search and Open AWS Glue in your AWS account

schedule a Python Script in AWS Glue as a Job

Step 2 – Open Visual ETL

In the left sidebar, we can see Visual ETL.
Click on that and you will see a page with all the jobs listed.

Step 3 – Add Job

Select the type of job you want to create.
I will select Python Shell Script Editor, with ‘Create a new script with boilerplate code’ selected.
Click on the Create button.

Step 4 – Configure your Job

It will open a page like below.
Now you need to do some configurations.

Name your Job.
And choose an IAM Role.
Rest keep everything as it is and click on the Save button.
Following is the demo code that I wrote to check my Glue Job.

Now you can see that I imported pandas and numpy in my code and these are not already present in the Glue environment.
So now we will add these libraries to our environment.

Steps to include external libraries:

Open Job details.
Scroll down and click on Advanced Properties.
Scroll down and under Job Parameters click on ‘Add new parameter’.
Under Key add --additional-python-modules and under Value add comma-separated libraries.
Click on Save.