25/02/2025
Deploying Shiny for Python on AWS: A practical guide
Shiny (but also other frameworks like Dash) provide a powerful tool for creating interactive dashboards. And although the framework-specific servers can be a quick ways to get your app live, using a cloud services like AWS provides scalability, flexibility, and cost efficiency, while also offering robust security features like fine-grained access control. In this blog we will go over the different steps to get your shiny app up and running!
A general overview of the approach can be found in the image below. We will use Docker to create a container of our app, push this container to AWS ECR and then use AWS Fargate within ECS to run the application from this container.
Like in previous blogs, we will use these blocks to go over the code and specific steps of how you can implement this for yourself. By the end of this blog, you can have your Shiny app up and running on AWS.
Figure 1: Architecture of Shiny app deployment on AWS
Develop and build your app locally
In this blog, we will not go over the details of how you can build a Shiny for Python application. The gallery and tutorials of Posit on the main Shiny for Python website give wide variety of examples and possibilities. The playground feature of the main website is also a great way to get familiar with the options. For our first step, we build a containerized image of the app. This allows you to package the app with all its dependencies, ensuring that it runs consistently across different environments.
In our example, we will build a quiz app to test your geography knowledge. You can see the final running app here. This app will allow the user to choose from a range of spatial files (which were processed with Databricks and stored on AWS S3) and then let the user try to locate the different locations. If you are interested in the fundamentals of spatial data: check out this previous blog. To be able to build an image of our app in the next step, we use a virtual environment while developing. Here we have used Poetry for this purpose.
Next we create a Docker image using a Dockerfile and docker build. Make sure to test the containerized app afterwards using the docker run command.
Figure 2: Example of Dockerfile
AWS ECR: shipping your container
We will now push this working Docker image to AWS Elastic Container Registry (ECR). AWS ECR is a fully managed container registry service for storing, managing and deploying Docker container images.
Pushing your Docker Image to AWS ECR is done using the AWS CLI. You can find a description for installation here. Once installed, you can use aws configure to set and confirm your AWS credentials. You will need an Access Key, which you can create under the “Security credentials” tab of your AWS account in the AWS Console.
Figure 3: location of Security credentials where you can create Access Key for AWS CLI
Once your account is configured, you can run following command to authenticate your Docker to AWS ECR. Make sure to fill in your appropriate region and AWS account ID.
aws ecr get-login-password –region <your-region> | docker login –username AWS –password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com
Next, we can create a repository in ECR.
aws ecr create-repository –repository-name <repository-name>
Now we can just tag our local docker image and push it to ECR.
docker tag <image-name>:<tag> <account-id>.dkr.ecr.<region>.amazonaws.com/<repository-name>:<tag>
docker push <account-id>.dkr.ecr.<region>.amazonaws.com/<repository-name>:<tag>
You can check whether your image has been successfully pushed by going to the console of ECR.
Tip: check the VIEW PUSH COMMANDS button to get the CLI commands prefilled for your repository and account.
Figure 4: Check if your image was successfully pushed to the repo in the console. TIP: you can see your specific prefilled push commands by clicking the button in the top right.
EC2: Application Load Balancer and Security Group
We will be using a AWS Elastic Container Service (ECS) task to run our application. However, ECS tasks are dynamic by design. When tasks are launched, their IP addresses can change, and you may not always know these in advance. This can create difficulty for your end-users. To ensure smooth communication to our app, we will use two aspects of EC2: Security Groups and Load Balancing.
Security Groups control all inbound and outbound traffic on your app. The (Application) Load Balancer (or ALB) serves as the gateway for all traffic to our app. A major advantage of using a load balancer is that it provides a fixed url for the application, providing a consistent endpoint for the final users. The ALB will also can distribute requests over different tasks (making the app more scalable), filter traffic and perform health monitoring on your tasks.
Security Groups and Load Balancers can be found in the EC2 dashboard. Under Network & Security, you can find the Security Groups. We will use Create security group to define one especially for the app. Note that a security group is linked to a specific virtual private cloud (VPC). For the load balancer, we will need a VPC with subnets, so keep this in mind when selecting your VPC. You can add inbound rules and we will allow all traffic from IPv4 and IPv6. When useful, you can also add outbound rules to your security group.
Under Load Balancing you can find the Load Balancers and Target Groups. Click create load balancer and select an Application Load Balancer. We give the load balancer a name, select the Internet-facing scheme and link the load balancer to our VPC and linked security group. We also need a Target Group, this can be created from within the load balancer page, we just need to make sure we select the same properties as we want in the load balancer.
ECS Task Definition: the link to the container image
In AWS Elastic Container Service (ECS), a Task Definition is a concept that defines how containerized applications are deployed and run. It serves as a blueprint for your application, specifying the containers to use (from for example ECR), the resources they need (CPU, memory, variables, …), and how they interact with one another. AWS currently allows you to choose between Fargate and EC2 as launch types. Fargate is serverless type in which AWS will manages the infrastructure, while EC2 requires more setup, but also provides more control over the underlying infrastructure.
We will first create the task definition. In the AWS Elastic Container Service (ECS) dashboard, select Task Definitions and then Create new task definition. Give the task definition a name. Under Infrastructure requirements select “AWS Fargate”. We will select a Linux operating system. You can also define the necessary CPU and memory for your app. For our basic example, .5 vCPU and 1 GB memory is sufficient. Under the Task execution role, you can select Create new role.
Figure 5: Creating a ECS Task definition with AWS Fargate
Under Container – 1 name your container and paste the URI you can find in your repository in AWS ECR. In our Dockerfile, we have explicitly set the port of our application to 80, so make sure container port is also set to 80. You can also add your environmental variables here, in our case these are variables necessary to connect to the correct AWS S3 bucket were we stored our geojson files. The task definition allows for a wide variety of further configurations, but for now, we will leave these on default and Create our task definition at the bottom of the page.
ECS Service and Tasks
Within ECS a task is a running instance of a task definition, representing the actual containers performing the work. Tasks can be launched directly for ad-hoc needs or in our case be managed by a service. A service ensures high availability and scalability by maintaining the desired number of tasks running continuously on the ECS cluster. It monitors the tasks, automatically restarting any that fail, starts new tasks when the task definition is updated and can integrate with a load balancer to distribute traffic.
All services and tasks under ECS require a Cluster. When we Create cluster under Clusters in the ECS dashboard, we only need to provide a name and description. Make sure AWS Fargate is selected, but all other fields can stay on the default value.
Within our newly created cluster, we will scroll down to Services and select Create. We can leave the default values for the Compute Configuration. Under Deployment Configuration select Service and select the task definition family we created in step 4. Give the Service a name.
Extra tip: if at any point, you want to keep your application in ECS but temporarily do not want it active, just set the Desired Tasks to zero. This will terminate all tasks related to the service.
Scrolling down to Networking, we will select the VPC, subnets and corresponding security group we created before. Make sure Public IP is Turned on.
Finally, under Load Balancing, we will check the box to active. Select an Application Load Balancer and select the previously created Load Balancer. We will create a new listener and Create a new target group (do not forget to give it a name). We will click Create at the bottom to create the service. This will create and activate the service. It will also start the necessary tasks. This process will take a few minutes.
Up and running: checking the Tasks and ALB
Your app will now be running on an active task within your ECS cluster. Within this task, you can find the dynamic url. For the fixed url, check the load balancer. Unless you specially disabled it, you can also monitor the usage and logs of your app by visiting the AWS CloudWatch page. Happy dashboard building!
We can check the tasks in our cluster to see that the application is indeed running. When you click the active task, you can also find the Public IP in the Configuration tab. Note that this IP will change when a new task is started. For a more stable url, we will check the DNS in our load balancer.
Our app is now live!
Figure 6: Check the Services and Tasks tabs in the cluster to see the current status of the application.
Figure 7: Find the final DNS for the app at the Load Balancer.
Written by
Bjorn Rombouts
Data Consultant @ Aivix
