Home Assistant and Reverse Proxy on pi4 with AWS DDNS

The title of this post is a mouthful and that probably isn’t everything. This is going to be a long post about a pretty long technical journey. A few years ago I got a free Amazon Echo and very quickly was hooked. I added Echo Dots in multiple rooms, got some smart plugs to handle some annoying lamps, and eventually added some smart switches and a smart thermostat. A colleague introduced me to Home Assistant and after lurking around in r/smarthome and r/homeautomation I decided to get a pi4 to start playing with HA. After spending a few months playing with the basics (including getting access to integrations with my garage door, vacuum, and phones just to name a few), I decided to start working on adding my SmartThings hub and devices as well as Alexa. This means external access. Since I didn’t want to add another monthly fee to my already crowded bank statement and I have 20+ years of IT experience, I decided to use my brain and existing investments to build a solution. Since HA is all about local control, I figured this also gives me total control of how and what I expose to the internet. This was my plan:

  • Home Assistant running in Docker on the pi4
  • Reverse proxy running in Docker
  • Port forwarding from router
  • Dynamic DNS running in the cloud (most likely AWS)

Honestly, the local reverse proxy wasn’t always part of my plan. I was somewhat hoping to come up with a cloud-based proxy solution, but there are 2 obvious issues with this: security and cost. While it would be possible to route traffic through a TLS encrypted endpoint sitting in the cloud, I would still need to secure the communication between the cloud and my local pi somehow so best to terminate TLS on the pi. Not only is this a security issue, but it would consume unnecessary cloud resources since all traffic would be routed through the cloud as opposed to just the DDNS lookups. So eventually I landed on the local reverse proxy.

Step 1: Home Assistant on Docker

Getting my pi4 setup was not without challenges. I do own a USB mouse, but my only working keyboard is Bluetooth only so while I didn’t do a completely headless install, it took some creative copying and pasting to get the pi up and running. Since I’m pretty experienced with Docker, the setup of HA was a breeze. My docker-compose.yml file for just HA is shown below.

version: '2'
services:
  homeassistant:
    container_name: home-assistant
    image: homeassistant/home-assistant:stable
    volumes:
      - /mylocalpath/config:/config
    devices:
      - "/dev/ttyUSB0:/dev/ttyUSB0"
    environment:
      - TZ=America/New_York
    restart: always
    privileged: true
    group_add:
      - dialout
    ports:
      - "8123:8123"

There are a few things I want to point out in this configuration. I wanted to be able to play with my AVR connected via RS-232 over a USB adapter. There are 3 items in this config required to get access to the USB port:

  • devices: map the physical device to the container
  • group_add: give the container access to the dialout group for port access
  • version: version 3 of docker-compose does not support group_add so had to revert to version 2

Otherwise, this is a vanilla docker-compose config straight from the HA documentation. If you aren’t familiar with Docker, the only things here you really need to understand are the “volumes” config that tells HA were to store your configuration files and the “environment” config that sets and environment variable for your time zone. There are many more things you can do with Docker and HA via this config, but generic config provided by the documentation is enough to get you going with only changing the config path and time zone as needed.

Step 2: Dynamic DNS on AWS

I knew it would be possible to setup dynamic DNS using Route53 and Lambda so a quick googling led to this blog post. Eureka! Better yet, that post led to this git repo and, better yet, this CloudFormation template. Running the template is pretty simple. Just copy/paste the content into the designer in CloudFormation or upload to a bucket. Then provide the required parameters when you create the stack. The only parameter really required is the route53ZoneName which should only be your domain or subdomain. For example if your HA URL will be ha.home.mydomain.com then this value should just be home.mydomain.com. The rest of the parameters can be left as the default.

NOTE: If you already host your domain in Route53 and you want to add a subdomain for your DDNS, it is easiest to reuse your existing Hosted Zone. You can enter the zone ID in the route53ZoneID parameter to reuse the existing Hosted Zone.

After running the CloudFormation template, you will find a new DynamoDB table named mystack-config where mystack is the name of your CloudFormation stack. You will need to create a new A record in this table to store data for your host. Duplicate the sample A record, change the shared_secret (preferably to a similarly long, random string), and update the hostname to your full host (ex: ha.home.mydomain.com.) and make sure to include a trailing . at the end. Make note of the secret since you will need to pass this from your DDNS client.

Next all you need is the client. The good news here is that the git repo has a bash client. The configuration is a bit tricky, but the script to run the client looks like this:

#!/bin/bash

/path-to-ddns-client/route53-ddns-client.sh --url my.api.url.amazonaws.com/prod --api-key MySuperSecretApiKey --hostname my.hostname.at.my.domain.com. --secret WhatIsThisSecret

There are some important items in here that must be configured correctly:

  • The –url parameter value should match what is on the Outputs tab after running your CloudFormation script. Note that this does NOT include the protocol prefix (http://) so make sure when you copy this you copy the text and not the URL since your browser will show it as a link.
  • The –api-key parameter value should be populated with the value generated
  • Note the trailing . at the end of the –hostname parameter value. This is the FULL host name and must match the record in DynamoDB.
  • The –secret parameter value should match the value recorded in the DynamoDB record.

Finally, in order for your IP to be recorded with the DDNS every time you boot, you will want to place the above script to update your IP address in /etc/init.d and make sure to make the script executable.

Step 3: Port Forwarding

In order to route traffic to our personal public IP address, we have to have something that is able to listen within our network. For most of us, that means opening up the built-in firewall on your home network router. My router sits behind the modem provided by my ISP (as opposed to being provided by my ISP) so I have complete control over my router. Your setup may introduce different challenges, but the solution will be similar. First you will need to set a static IP for your server so that you can forward all traffic to that IP. Then you will need to configure port forwarding.

My router is a NETGEAR Nitehawk R6900v2. So the port forwarding setup can be found in the admin console for the router by first selecting the “Advanced” tab, then expanding “Advanced Setup”, and then selecting “Port Forwarding / Port Triggering”. You will need to forward two ports: 80 and 443. The NETGEAR console requires you to select a service name to setup port forwarding. For port 80, you can select “HTTP” and set the port range to 80-80 and set the IP address to your static IP. For port 443 (TLS), you will need to use the “Add Custom Service” option. I set the service to “HTTPS” and port range to 443-443 and set the IP.

Step 4: Reverse Proxy in Docker on pi4

I’ve worked with the jwilder/nginx-proxy Docker image before and not surprisingly it is still the go-to solution for a Docker reverse proxy. It’s very simple to use. You just map a socket so the container can listen to new containers, and then on each container hosting a something behind your proxy, you set the VIRTUAL_HOST and optionally the VIRTUAL_PORT environment variables. The resulting docker-compose.yml file looks like this:

version: '2'
services:
  homeassistant:
    container_name: home-assistant
    image: homeassistant/home-assistant:stable
    volumes:
      - /mylocalpath/config:/config
    devices:
      - "/dev/ttyUSB0:/dev/ttyUSB0"
    environment:
      - TZ=America/New_York
    restart: always
    privileged: true
    group_add:
      - dialout
    ports:
      - "8123:8123"
    # Add environment variables for proxy
    environment:
      - VIRTUAL_HOST=ha.home.streetlight.tech
      - VIRTUAL_PORT=8123

  # Setup reverse proxy
  nginx-proxy:
    container_name: nginx-proxy
    image: jwilder/nginx-proxy:latest
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - /mylocalpath/certs:/etc/nginx/certs

Normally, this is all you need to do. However, when I started my Docker stack, it didn’t work. Looking at the logs with docker logs nginx-proxy revealed the following:

standard_init_linux.go:211: exec user process caused "exec format error"

Apparently the proxy image is not compatible with the ARM architecture of the pi. The Dockerfile used to build the image uses precompiled binaries built for AMD64. The command below are the culprits of this failure.

# Install Forego
ADD https://github.com/jwilder/forego/releases/download/v0.16.1/forego /usr/local/bin/forego
RUN chmod u+x /usr/local/bin/forego

ENV DOCKER_GEN_VERSION 0.7.4

RUN wget https://github.com/jwilder/docker-gen/releases/download/$DOCKER_GEN_VERSION/docker-gen-linux-amd64-$DOCKER_GEN_VERSION.tar.gz \
 && tar -C /usr/local/bin -xvzf docker-gen-linux-amd64-$DOCKER_GEN_VERSION.tar.gz \
 && rm /docker-gen-linux-amd64-$DOCKER_GEN_VERSION.tar.gz

The solution to this is relatively simple. First get a copy of the repository for the nginx-proxy image:

git clone https://github.com/nginx-proxy/nginx-proxy.git

Next, modify the code to pull the ARM version of forego and docker-gen so replace the above code as shown below:

# Install Forego
RUN wget https://bin.equinox.io/c/ekMN3bCZFUn/forego-stable-linux-arm.tgz \
  && tar -C /usr/local/bin -xvzf forego-stable-linux-arm.tgz \
  && rm /forego-stable-linux-arm.tgz
RUN chmod u+x /usr/local/bin/forego

ENV DOCKER_GEN_VERSION 0.7.4

RUN wget https://github.com/jwilder/docker-gen/releases/download/$DOCKER_GEN_VERSION/docker-gen-linux-armel-$DOCKER_GEN_VERSION.tar.gz \
 && tar -C /usr/local/bin -xvzf docker-gen-linux-armel-$DOCKER_GEN_VERSION.tar.gz \
 && rm /docker-gen-linux-armel-$DOCKER_GEN_VERSION.tar.gz

In the first section, we have to replace the ADD with a RUN and then use wget to pull the code and tar to unzip. In the second section, we just need to replace amd64 with armel. I have this change added to my fork of nginx-proxy in the Dockerfile.arm file.

Now you need to build a local image based off of this new Dockerfile:

docker build -t jwilder/nginx-proxy:local .

The -t tag will name the image with a local tag so it won’t conflict with the official image. The . at the end will find the Dockerfile in the current directory so this command must be run from the nginx-proxy folder created when you clone the git repo.

Finally, update your docker-compose.yml file to use the new image:

version: '2'
services:
  homeassistant:
    container_name: home-assistant
    image: homeassistant/home-assistant:stable
    volumes:
      - /mylocalpath/config:/config
    devices:
      - "/dev/ttyUSB0:/dev/ttyUSB0"
    environment:
      - TZ=America/New_York
    restart: always
    privileged: true
    group_add:
      - dialout
    ports:
      - "8123:8123"
    # Add environment variables for proxy
    environment:
      - VIRTUAL_HOST=ha.home.streetlight.tech
      - VIRTUAL_PORT=8123

  # Setup reverse proxy
  nginx-proxy:
    container_name: nginx-proxy
    # UPDATE TO USE :local TAG:
    image: jwilder/nginx-proxy:local
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - /mylocalpath/certs:/etc/nginx/certs

Note that I have also removed the network_mode: host setting from this configuration. This is because nginx-proxy only works over the bridge network.

Step 5: SmartThings

I have my z-wave devices connected to a SmartThings hub. Eventually I plan to replace that with a z-wave dongle on my HA pi, but for now I wanted to setup webhooks to let me control my SmartThings devices through HA. This was a big driver for all of this work setting up DDNS, TLS, and port forwarding. The generic instructions worked fine all the way up to the very last step when I got an ugly JSON error. Thankfully, googling that error pointed me to this post describing the fix. Simply removing the “theme=…” parameter from the URL allowed the SmartThings integration to complete.

Addendum: Creating Certs with letsencrypt

While it is possible to use any valid cert/key pair for your TLS encryption, you can create the required certificate and key using letsencrypt. I did this using certbot. Install of certbot is simple:

sudo apt-get install certbot python-certbot-nginx

Then creating the cert was also simple:

sudo certbot certonly --nginx

Following the prompt for my full domain (ex: ha.home.mydomain.com) was pretty easy. Note that first you must have nginx running on your host so it can do the required validation. So you can either do this before disabling nginx on your pi (if it was enabled by default like mine) or after you setup your nginx-proxy. Just make sure you expose port 80 in your nginx-proxy container so the validation works.

Finally, just copy the certs for mapping to your ngingx-proxy container:

sudo cp /etc/letsencrypt/live/ha.home.mydomain.com/fullchain.pem /mylocalpath/certs/ha.home.mydomain.com.crt
sudo cp /etc/letsencrypt/live/ha.home.mydomain.com/privkey.pem /mylocalpath/certs/ha.home.mydomain.com.key

Alternatively you can symlink the keys rather than making a physical copy but the names must match your VIRTUAL_HOST setting.

Conclusion

Overall I’m very happy with how this all turned out. It was a great learning exercise. Almost every step of the way had at least a minor annoyance which led me to write this post in order to help others out. I would say getting nginx-proxy to work on the pi ARM architecture was the biggest challenge and even that wasn’t too difficult. In the end, I’m glad that I have control over my DDNS, integration with SmartThings and Alexa, and access to my HA server from outside my house.

Tinpothy: Spirit Guide

In the Catholic Church, the regular ceremony or “mass” is centered around a reenactment of “the Last Supper”, or the last meal before Jesus was crucified to later rise from the dead. My favorite part of the mass happens immediately before that and, regardless of religious or philosophical beliefs, most people would agree is at least “nice” if not a fundamental part of their belief system. The priest says, “let us offer each other a sign of peace,” and everyone turns to the people around them and shakes hands, hugs, kisses, waves, or extends 2 fingers in the familiar “peace” sign. In fact as children we are taught to give the peace sign to our friends rather than climb over each other to shake hands with our friends sitting several rows away. As a child, I thought the point of this exercise was to put aside any quarrels we might have with our neighbors or siblings or parents (which were many as a child) and extend a sign of “peace” – assuming the context of the opposite of “war” or “fight” or “quarrel”. I was wrong.

Photo credit Independent Florida Alligator

Practically anyone who spent any significant time in Gainesville, Florida as either a resident or a student the University of Florida knows “the running man”. Having gone to school in the 90s, when he sported dreadlocks, we called him “the Rasta Runner”. He was a mysterious man who could be seen running, seemingly constantly, hands extended in front of him making the “peace sign” with both hands at as many passers by as possible. We wondered if he was homeless, if he was mentally ill, but we didn’t ask. He was just a fixture of the town and was clearly interested in “offering a sign of peace” to as many people as possible, while running. My first (late) wife worked with him so I learned his name was Tinpothy and he was (not surprisingly) a nice guy.

Fast forward to today and I was on my normal 5 mile run: about 1.25 miles to the lake, about 2.5 miles around, and 1.25 home. As I was about 2 miles in, I passed an older gentleman who was running in the opposite direction. He didn’t seem to notice me at all. I thought to myself, “Why run on a paved trail full of people if you are not going to be friendly?” Then I thought about Tinpothy. I’m not sure exactly what my full mental process was. The nice thing about running is your body starts consuming all of the oxygen your brain normally uses for less useful things and lets you focus on one or two things (or just lets your mind go completely blank). Somewhere in that process I decided to channel my inner Tinpothy. At first I just started saying “good morning” but then I started adding the “peace sign”. I found out that holding up my two fingers got people’s attention and then when I said “good morning” something interesting happened: most people smiled (and said good morning back).

I said I was wrong about what the sign of peace meant. As an adult, I now understand that when we say “peace be with you,” we mean peace as in “quiet”. The peace that is mentioned so many times in the Bible is the same peace of many philosophies and religions: “inner peace”. So as I was running and sharing a sign of inner peace with those I passed, I found myself running “harder” (my heart rate went up by about 10 BPM) even though the effort felt the same. I hope the smiles I got in return were a sign that those I connected with on my run recognized for a brief moment that some random person recognized their existence and wished them well and that in some way gave them a moment of peace.

As I was finishing my lap around the lake, I saw the older gentleman who seemed to have ignored me before. This time I held up my 2 fingers to get his attention and smiled and said good morning, and he smiled and said good morning back.

Orlando Code Camp 2019 – Little Services, Big Apps

Today, I was fortunate to present on microservices at Orlando Code Camp.
I think this was my 5th straight year speaking at Code Camp, but it has definitely been long enough that I am now not sure ow long it has been. First of all, I’d like to thank the folks at the Orlando .Net Users Group for inviting me back to speak yet again. More so, I would like to thank the great group that showed up to hear my presentation, “Microservices – Little Services, Big Apps”. It was very humbling to have such a packed room and I hope everyone learned something or was at least mildly entertained. For those of you who attended, the slide deck is below. For those of you who did not attend and arrived here some other way, you can also find the slides but they will probably be far less interesting. I don’t normally record these sessions because I always try to capture a certain amount of technical Zeitgeist both from the current technology landscape and my current professional experience. Also, I feel like these community events deserve a certain amount of exclusivity for those who are able to attend. Over time, most of the content will be posted on my blog one way or another (if it hasn’t been already).

Orlando Code Camp 2019 Presentation Deck
Orlando Code Camp 2019 Presentation Deck
Keyboard keys in a pile

Tabs ARE Better Than Spaces?

For a few years, I’ve been working with a team that prefers spaces to tabs. More specifically, 2 spaces. I agree that this is a superior visual arrangement and having everyone use it benefits those of us who prefer it, those who don’t care, and will eventually bring those who don’t like it to see the light. Today, however, for reasons that will be more obvious in future posts, it occurred to me that tabs may actually be superior.

This revelation requires first admitting that this is a preference. You may code on a 32” curved 16:9 or a 20” 4:3 rotated to 3:4 portrait. You could be writing code on a flip phone for all I know. Whether you use tabs or spaces, the desired effect is the same: to provide consistent formatting to make the code easier for you to read.

This leads me to my argument for tabs: Tabs are the democratic choice. Tabs put the details of the implementation in th hands of the consumer. Most modern IDEs allow you to customize your tabs. Want tabs to be 7 spaces? Let your crazy-big-odd-numbered-tab freak flag fly! It’s much easier to customize tabs or replace them with spaces than to replace spaces with tabs. So I’m sure this debate will rage on and I will happily continue to use spaces for my code, but for you, my friend, I will give you tabs.

Tutorials Are (Often) Bad

Yes, that is a broad generalization. Yes, broad generalizations are (often) bad. Yes, I see the irony in that. However, tutorials are often bad. The good news is, this is often by design. Tutorials are intended to show you how to do something very specific. This means most basic tutorials are intended to show you how to do that specific thing as succinctly as possible. The problem (or potential problem) with this is that good code is often not particularly succinct. As I mentioned in my previous post, “optimal” code isn’t easy to design and what is “optimal” for one scenario might not be fore another. So in this post, I’m going to explore some concepts that are (often) good. Since they are often not followed in tutorials, it stands to reason that tutorials are (often) bad.

DRYness

woman drying hair too vigurously

OK, not that kind of DRY. DRY means “Don’t Repeat Yourself”. Tutorials typically keep code in as few places as possible to make it easy to follow the examples. DRY code requires separation of concerns. This means putting your code in the “right” place which means you have to have multiple places to put it. Chances are you need a more complex project/file structure than what a tutorial will show you.

Testability

test test test test test test

Unless a tutorial is demonstrating some aspect of testing, it most likely doesn’t include any tests. Because of this, the code won’t necessarily be very testable. Making your code testable requires a certain degree of DRYness since you typically want to test small units of code (“unit tests”). Most modern frameworks support features like dependency injection to improve testability, but these features are often absent from tutorials, so beware of tutorials without any tests.

Maintainability

maintenance

Testability and maintainability go hand in hand. Maintainability is definitely an art and beauty is in the eye of the beholder when it comes to what “easy to read” code looks like, but there are some good rules of thumb:

  1. Keep functions short enough to read on the screen.
  2. Use descriptive names for functions that follow the law of least astonishment
  3. Avoid unnecessarily concise notations
  4. Use consistent formatting and spacing

Tutorials probably follow number 4, might follow number 3, and very often ignore 1 and 2.

Putting it All Together

The code in tutorials is not typically well structured. Good structure means your code is DRY, testable, and easy to maintain. When following tutorials, look for opportunities to improve the structure. Develop your own toolbox of reusable code and components. You will quickly find that all projects benefit from having at least 3 components: a “core” set of code not tied to a specific implementation, at least one specific implementation, and tests. Nothing is one-size-fits-all so no one template will match every project, but it’s good to have a general approach that prepares you to write “good” code.

brain art science

The Science and Art of Code Structure

Clearly, I should have been in game development. I’ve been coding since the third grade so software was always in the cards for me. I have a degree in Mechanical Engineering so I have the physics chops to handle pretty much any kind of real-world simulation from projectile motion and particle physics to fluid dynamics and even heat transfer. My masters degree is in Industrial Engineering so I know about things like optimization and simulation that are helpful for AI (among other things). And games are fun!

What I realized recently, is that writing “good” code is an Industrial Engineering optimization problem. You see, optimization problems deal with multiple dimensions with often multiple optimal solutions. Note the use of the word “optimal” and not “best”. Optimization problems often have multiple variables and an astronomical number of potential solutions. So when you are optimizing code, what is your goal? Maximize speed of execution? Minimize size of compiled code? Minimize bandwidth used? Minimize processing power used? Minimize total operating cost? Maximize test code coverage? Why not all of them? Well…I’ll tell you why.

One of the most famous examples of an optimization problem and how mind-bogglingly many solutions they could have is the “traveling salesman problem“. The traveling salesman must visit some number of potential clients and then return home. In the simplest version, he is flying from city to city and the only concern is the order of the flights to the various airports. A much more practical and complex version is a package delivery company with a fleet of trucks needing to deliver packages over routes through a complex network of highways and city streets with variable traffic, one way streets, traffic lights, etc. Every variable multiplies the number of solutions by the number of possible options for that variable.
The number of possible solutions very quickly gets large – like more than the total number of atoms in the known universe large. What hope do we have of solving these problems? Luckily, we have pretty cool brains.

Malcolm Gladwell wrote a book titled Blink: The Power of Thinking Without Thinking. It’s a great book and I highly recommend reading it, but the gist is our brains are able to take in a vast array of information and come to a conclusion (an “optimal solution”) without consciously thinking about it. This ability to organically problem solve was also recently discovered in single cell organisms. This is why we are pretty good at coming up with decent solutions to these problems without knowing anything about linear algebra. Our brain operates on “heuristics” that our life experiences have allowed our brains accept without having to think about them. In the world of software development, the use of these heuristics are the “art” and allow us to optimize variables that are extremely more challenging and seemingly abstract such as “maximize code reuse” and “maximize maintainability” and, as I like to say, “minimize astonishment“.

Another important consideration in optimization is boundaries. For example, you may be able to purchase a certain amount of computing resources for a certain cost. Additional resources add more cost. This means that you don’t necessarily need to use an absolute minimum number of resources, you just need to make sure that the amount of resources used doesn’t trip the threshold or that if it does, you have a source of revenue to cover the added cost.

In future posts, I will dive into more detail on the day-to-day practical advise on optimizing your code, but first I wanted to take some time to lay some groundwork. In the early days of computers, you were greatly constrained by memory and processing power. You had a limited number of pixels that could be rendered on the screen in a limited number of colors. You could only store so many bytes of code. You were limited to a small number of operations the processor performed. You were limited in the language you could code in and the structure of that language. The optimization required was much more science with a small number of options for each variable. Now, the options are astronomical. What language do you use? What platforms or devices run that code? What communication channels does it leverage? How many millions of colors and pixels does it render? Sure, it is still important to minimize operating cost (storage, CPU cycles, bandwidth, etc.), but you can also get access to a significant amount of resources for little or no cost. This allows you to focus on the art – maintainability and reuse – first, and then solve the challenging scientific problems later (or not at all).

teacher and student

Eminence Imminent

After spending nearly 20 years working for a defense contractor, I joined Deloitte to focus more on software and to build that software for clients outside of the defense business. I’ve been with Deloitte for a few years now and I’m starting to take a more active role in Systems Design and Engineering leadership. This means helping with activities to establish “eminence” of our practitioners. For me, that means being more active in those activities myself…practicing what I preach…eating my own dog food…I think you get the point.

I’ve consolidated some of my old blogs here. It’s been a long time since I’ve posted anything technical, so most of that stuff did not survive the move here. However, I have some nerdy things in the works and a few decades of…wisdom?..to share. I’m looking forward to unleashing the avalanche of blog posts swirling in my head.

I’m trying to run a little more often (ok, a LOT more often) so I’ll be posting about that too. Beware, I am a running nerd so expect plenty of over-analysis while I try to regain or even surpass my fitness level of years gone by.

So for those of you who have stumbled onto my little corner of the internet, I welcome you and hope you find some knowledge, inspiration, or at least some entertainment.