Prometheus in Your Home
All metrics of nodes and services in my home lab are scraped by Prometheus. Here are some challenges I ran into, and how I worked around them.
Simple Go Webhook Receiver
Go is incredibly powerful and flexible. Something I often find is that I see tutorials that essentially say “Here is how you implement X in Go, Step 1: Download this library I wrote” - It is very rare this is actually necessary. Libraries are still useful to speed up development, but many web based functionalities are very simple to achieve in Go.
Kubernetes Local Storage
Here is how to start-to-finish use local storage on a kubernetes pod.
Secure Networking between local hosts and an AWS VPC
Being able to route to your AWS VPC hosts from your home network opens many doors for projects. AWS Offers a VPN Gateway service, but that costs a minimum of ~90 a month. For small labs, we can easily do this with a t2.nano.
Nginx metrics via exporter sidecar
If your static website runs on Kubernetes, here is how to export metrics and have them scraped into Prometheus.
I took some time at the end of this year to do a serious rebuild of my homelab. My philisophy on homelabbing differs slightly from others - I do not mind an AWS cost under $30 in exchange for no costs in the form of expensive equiptment, space, power bills, and noise complaints from my wife. My requirements were simple: minimum configuration, low compute requirements, and the ability to tear things down/back up quickly so I can break stuff.
Securing a Kubernetes ingress with htpasswd
Here’s a pretty easy example for adding basic password auth to a Kubernetes ingress
Running a static website on Kubernetes
Jekyll generates my site to a directory
Buildin this site again .. again
My website started as an Ubuntu server that I managed via Ansible. Early on I actually generated blogs using
sedbut I quickly moved onto Octopress. I hacked random features into my site and eventually it became littered in cronjobs and junk.
Openshift on AWS caveats
Cloud versus on-premises based Openshift deployments have their own unique set of challenges. From a consulting perspective, I generally view cloud as easier in terms of orchestration, but with the possibility of deeper technical issues.
Some time last year Prometheus became a technical preview for Openshift. That same month I rolled onto a project with some pretty steep architectural layouts of handling metrics. I spent several sprints hacking out of the box features and configurations into Openshift’s Prometheus deployment. All of this changed towards the end with the introduction of the Monitoring Operator. I’ll be writing about that at a later date, I’m still working out some kinks in my home lab.
Building this site (again)
My previous setup wasn’t very agile. It was fun scrapping some shell scripts together but it wasn’t very agile or flexible.
Accessing the Kubernetes Dashboard
Learning these new hip container orchestration tools can be intimidating. I remember learning OpenShift the first time and slamming my face against the keyboard for days as my installer would fail 35 minutes out of an hour in. Learning these tools can also be cumbersome for us who self practice if you use a cloud service.
Plotting banned hosts
I’ve had some extra time this week. I made a quick and dirty way to automatically plot nodes I ban from this site. This has been done before and the code is public, I’ll probably check out those examples next week to compare. I added an extra action to Fail2ban to append each IP to a text file. I have an hourly cronjob generating a new plot and throwing it into html/images/. Part of it calls on this Rscript:
This site is hosted using Amazon Web Services. I don’t have much preference with cloud providers, but I needed some experience with ec2 instances. Amazon’s security groups have made securing things pretty trivial, especially with my simple use case.
Building this site [Legacy]
This is no longer relevant.
Blog Archive Contact Dale