How I started making Shiny apps for visualizing NBA Data, Part I: Shiny Server

Sravan December 24, 2023 [Linux] #docker #shiny #apps

Introduction

I keep seeing good Shiny apps from the NBA Analytics community and find them cool. Some of them are:

Learning R

But these apps were in R, and I didn't know R then. I started learning R on Nov 7, 2023, after interacting with another Twitter user who wanted someone to recreate a cool-looking Owen Phillips Bad Calls visualization. I got the code from Owen's Substack and modified it just enough to recreate it for the latest data within a few hours. Hardest part was setting up the R development environment in VSCode.

Because I knew what R was, Shiny didn't seem that distant to me anymore. Through Shiny's documentation, I discovered that there is Shiny for Python. This was excellent news for me since I was much more comfortable writing code in Python as I was a beginner in R. Still, I didn't start immediately, for the main reason that I didn't have any ideas for cool apps and didn't have background of existing NBA Analytics works which could then be converted into apps.

Since that day, I have published many NBA analyses and visualizations on Twitter (you can find the collection of tweets here). So now I have a catalog of works I can build apps from.

Also, between Nov 8 and Dec 21, 2023, I found more apps I liked:

Getting Started with Shiny

I started my Shiny journey by reading documentation and other resources to get Shiny set up. Shiny has a easy way to generate examples automatically through the command line. So, I tried some of them and was able to play with them a little. Next, I wanted to see how to deploy the example apps to the web, as it would be a trial run for when I publish the apps I would make in the future.

Deploying Considerations

Most of the apps I've shared in the introduction of this blog are hosted by shinyapps.io. Posit, the company behind Shiny and RStudio, makes it easy to deploy apps to shinyapps.io vshinyapps.io directly via RStudio. But I don't use RStudio because I'm developing in python. Looking at their documentation for Python they have a package called rsconnect-python that enables you to push your Python Shiny app to shinyapps.io. Then I went to sign up at shinyapps.io and looked at the free tier. Here are the perks of a free account:

Now, getting back to Shiny (before running off on another tangent), there is good documentation on self-hosting Shiny apps. As an open-source advocate, my eyes immediately landed on the open-source option.

Building an ARM Server

I faced a huge issue right away. The documentation didn't have a way to deploy a Shiny server for ARM devices.
So, I immediately did a web search, which landed me at hvalev/shiny-server-arm-docker, which had a docker image for Shiny-Server on ARM.
Again, there was a small (well, maybe big) issue. It was shipped with Python 3.7 and didn't have Python 3.11, since it's based on Debian Buster and didn't come with any preinstalled packages required for data science (like ggplot, etc.).
There was a way to add the packages later, but the Python version was a deal breaker since I needed at least Python 3.9 for the packages I use. Therefore, my only option was to build my image based on hvalev/shiny-server-arm-docker with my custom modifications.

Features I needed:

Extending Existing Image

Docker makes it easy to extend existing images by using a Dockerfile. A Dockerfile usually looks like this:

FROM base_docker_image_name

Do operations

In our case, this would be:

FROM hvalev/shiny-server-arm:latest

Do operations

You can have the Dockerfile execute a list of operations to configure the image to your needs. Then, a docker image can be created from the Docker file using docker build command. In the directory with the Dockerfile, run this command:

docker build -t image_name .

It's as simple as that.

Dockerfile

Lets look at the operations I performed to modify the image. We start by pulling the existing image and update it. In docker you can get the working directory using WORKDIR instruction and run a command using the default shell using RUN instruction. These instructions are similar to the FROM instruction we've seen before.

FROM hvalev/shiny-server-arm:latest
WORKDIR /root
RUN apt update -y && apt upgrade -y

Since we have to build python3.11, we need to install these packages:

RUN apt install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev libsqlite3-dev wget libbz2-dev

Let's download the Python source file and extract it for compiling:

RUN wget https://www.python.org/ftp/python/3.11.6/Python-3.11.6.tgz &&\
    tar -xvf Python-3.11.6.tgz

We can then build Python using these commands adopted from this site:

WORKDIR /root/python-3.11.6
RUN ./configure --enable-optimizations
RUN make -j 4
RUN make altinstall

I running make with -j4 makes compiling Python faster using 4 threads instead of 1. I then run some cleanup to make sure the image size remains as small as possible

RUN rm -rf ./python-3.11.6*
RUN apt clean -y
RUN apt autoremove -y

Next, let's install the Python packages, starting by upgrading pip:

RUN pip3.11 install --no-cache-dir --upgrade pip
RUN pip3.11 install --no-cache-dir shiny plotnine seaborn plotly pyarrow scikit-learn

We are getting to the last stages now. Let's install the R packages. These need additional dependencies to compile correctly, so let's install those first. I made this list of dependencies by looking at the output of failed builds. 

RUN apt install -y libv8-dev libharfbuzz-dev libfribidi-dev libmagick++-dev
RUN R -e "install.packages(c('tidyverse','gt','gtExtras','ggimage','ggtext','scales'), Ncpus = 4, repos='https://repo.miserver.it.umich.edu/cran/')"

Similar to compiling python, we set Ncpus = 4 to utilize 4 threads and speed up the builds. During my testing, it cut down build time from 45 mins to 15 minutes (3x speedup). Finally, let's create a shared folder to store data

RUN mkdir -p /var/data/

We need to run docker build now to build the image:

docker build -t shiny-server-arm-python .

Deploying the Server

Yay, we have an image with Shiny-Server, which can run on ARM devices and has both Python and R support. But, we don't have a server yet. First, we need to configure the server, which we can do by creating a docker-compose.yml file in an empty folder where you want your server to reside, with the following contents:

version: "0.1"
services:
  shiny-server:
    image: sradjoker/shiny-server-arm-python:latest
    container_name: shiny-server
    ports:
      - 3838:3838
    volumes:
       - ~/shiny-server/apps:/srv/shiny-server/
       - ~/shiny-server/logs:/var/log/shiny-server/
       - ~/shiny-server/conf:/etc/shiny-server/
     # Optional: Mount additional data directory
     #  - ~/nbadata/:/var/data/
    restart: always

After saving the file run the following command:

docker-compose up -d

The server will be available at:

http://localhost:3838/

Remember that this is just a local server available on your host computer and other computers in the network. Connecting it to your domain, as I did at:

https://shiny.sradjoker.cc/

needs tunneling your port to your domain. I tunnel through Cloudflare. You can find the instructions on how to do it here. To use this Cloudfare method to deploy the server to the web, you must first let Cloudflare manage your website, which can be done by following the instructions here.

Conclusion

We have seen my journey to start making Shiny apps, including me starting with example apps but immediately transitioning into making a server to host my apps on (I find it funny, do you). I thought the server image that I've created would be a tool that might be helpful for other people interested in self-hosting their Shiny apps, so I released it on Github:

https://github.com/sravanpannala/shiny-server-arm-python.

You can find instructions on how to make your Shiny server there.

That's it for Part I. In Part II, I will talk about my first original Shiny Apps. Keep an eye out for it.

Index
Introduction
- Learning R
Getting Started with Shiny
- Deploying Considerations
Building an ARM Server
- Features I needed:
- Extending Existing Image
- Dockerfile
Deploying the Server
Conclusion