Keywords: docker host, custom docker, great application, web server, video files, plex, update, gitea, videos, tradebrain, task. Powered by TextRank.
Server racks are cool. Even though I think just the aesthetics justify its existence, having a couple of self hosted services running is a good idea. I want to go over the services I host and how I use them.
Jenkins is an automation and orchestration tool.
This application is definitely the workhorse of my system. I use it to automate a ton of tasks. I mainly have it run CI/CD pipelines for my custom apps and run scheduled tasks. Using Jenkins as a cron substitute makes life easier to monitor script output and synchronize execution of tasks. Jenkins is scheduling system backups of all the services that are running and some additional data repos. I have 2 categories of backups Synced backups, that are replicated in encrypted form over to Google Unsynced backups, that are only kept locally on a different NAS system.
I use borg to actually do the backups and Synology’s CloudSync technology to send data over to Google Drive. I’m considering rsync.net but haven’t had the time to dive deeper into their offerings yet. Borg locks the repo that it is currently working with, so you can’t run 2 backup tasks simultaneously. This is where the “Lock Resources” plugin comes in handy. That will queue up any jobs that need the borg resource. So even if a backup task takes more than I expected it to, and another task is fired due to its schedule, it will wait until the first task is completed and not error out because it cannot acquire the borg lock. Jenkins is running on bare metal, but I have plans to containerize Jenkins and only run an agent on the server.
Git is a content versioning system.
I have tried a lot of different self hosted git services ranging from GitLab, to gogs, to GitBucket and even cgit. Throughout all the different versions of my HomeLab I had troubles with each of these implementations and finally settled on Gitea. I host about 100 or so projects of mine and mirror projects that I like on GitHub. Gitea is integrated with Jenkins via webhooks and the Jenkins plugin so a push to a repo will automatically trigger a build and test cycle in Jenkins. If something goes wrong a push notification is sent to my phone and an issue is automatically created in Gitea. Even though I rarely collaborate on my own projects it’s good to have that issue created so I don’t forget to fix the broken build.
Jupyter is a web-based interactive computing platform.
Jupyter notebooks are a great way to do exploratory programming and prototyping. I have used Jupyter notebooks for other reasons too, such as running this blog or creating and maintaining a todo list. But I have since reverted to the former. When I have an idea for a trading strategy I usually prototype it out in a Jupyter notebook before porting it to a TradeBrain script.
Minio is a S3 compatible blob storage service. I mainly use it to store transient image files that I do not want to upload to HumHub because it is synced to Google Drive and those image files use up a lot of space and for hosting other static content like Keras models that are used by TradeBrain.
NextCloud is a self hosted cloud drive. I use it mainly to host large files that I need access to from different machines and for large file storage like ML datasets. I have the iOS client set up on my phone and also use it for photos and video backup. I have also started retro calendaring. NextCloud supports CalDAV which is great for running multiple calendars on my own server.
PiHole is a DNS service.
I used to use dnsmasq to resolve internal domains but now I have migrated my main DNS resolver to PiHole. Ad blocking is good to have even though I use one integrated in my browser. It’s easier to add new hosts via the web interface than it is manually adding them to /etc/hosts. I run PiHole on a dedicated celeron mini PC with Ubuntu and manually set my DNS server on the boxes I want. The router is still using the old dnsmasq box as the primary DNS server.
Plex is a server that will stream your videos and music.
I have been a long time user of Plex to stream video and audio but have recently become a lifetime member. I mainly use it to stream my audio collection and mirrored content from Youtube. Using TubeArchivist and the Youtube Metadata agent for Plex I can get a copy of a Youtube channel/playlist for viewing offline on my phone.
I used to run Plex on my Synology NAS but now I have it running on bare metal. I have a webhook setup that will scrobble to my Asocial server.
Used to synchronize directories between my homelab server and other servers in data centers.
Self-hosted photo and video management solution
I was a paid subscriber for Google photos and 200GB iCloud but I came to the conclusion that this is not really a sustainable model if you are constantly adding new photos or videos to your collection. I already have 10+ TB of storage in my homelab so why not just use that to host all this stuff. The good side is Immich provides a pretty decent experience in terms of uploading, searching and storage. The bad side is it's difficult to share the photos with others who don't understand or want to use yet another app. So I took a hybrid approach here: I cancelled the Google photos. I host a shared iCloud album for photos only and this allowed me to downgrade to the 50GB plan and save a couple of bucks. I use immich to host all the photos and videos. The added burden is that I need to clean videos from the Photos app once in a while after exporting them to Immich but I can live with that.
Some of the services I longer use, or I have consolidated.
"MakeMKV is your one-click solution to convert video that you own into free and patents-unencumbered format that can be played everywhere"
This is basically a gnome application running in the browser. I use it to convert DVD rips into MKV file for playback through Plex. Quite simple to use and setup. One drawback is that there is no queuing mechanism (like in HandBrake) to do mass conversions.
Docker is a containerization program.
I run a custom docker registry because most of my custom programs are containerized. Jenkins will create a docker image and push it to this registry and the docker host will pull these images and run them. The docker registry itself is also running in a docker container.
Gitea now has support for hosting packages including docker images, so no longer need to run an external registry.
“The Python Package Index (PyPI) is a repository of software for the Python programming language.”
I run a private instance of pypi (just like my private docker registry) for pushing custom python packages. Nothing fancy.
Gitea now has support for hosting packages including PyPI packages, so no longer needed.
This is a self hosted facebook alternative. I don’t have a facebook account as I’m not interested in acquaintances' lives nor am I interested in providing more data to facebook so they can display better ads. But I like to life-log so I can look back and reminisce. My instance only has 2 human accounts: me and my wife. It’s also home to a bunch of bots that post funny stuff or pictures. I also have a bot that will accept location check-ins, so I don’t use Swarm anymore
This is the decades old forum software. I actually started using it as a way of documenting and blogging my daily work. I was inspired by Josh of Limit Theory fame and the way he meticulously dev logged the progress of his now defunct game. It has great support for image uploading and emojis but I have moved away from this to devonTHINK3 to consolidate all my information in a single repo. Also I’ve always found the search function of PhpBB difficult to use and it didn’t work like I would have expected.
This is an app that I use to create galleries for my pictures from my iPhone. It basically serves as a backup to view the pictures from non-Apple devices. I use the Synology DSFile app to backup the pictures from my phone to my Synology NAS and point the NFS share to Home Gallery. It can easily deal with 7k or so pictures and videos I throw at it.
I've retired this in favor of NextCloud
This is a nice web based file manager to access the files on my server, when I want to quickly share a file with another machine or just upload some file to the server. It’s a lot quicker than scp .
Airflow is a platform created by the community to programmatically author, schedule and monitor workflows
I've been using Jenkins and application layer code to schedule cron jobs for TradeBrain. This is a hacky workaround and easily will get out of hand once there are dependencies between jobs and you want to run some tasks in parallel then join on those task etc. AirFlow really shines in this type of scenario. The task definitions are in Python which makes it very flexible with integrations for databases, shells, python scripts etc. make it very easy to get jobs running. I have been moving jobs defined on Jenkins gradually to AirFlow and will use Jenkins only as a CI/CD server.
Here's an example DAG running for TradeBrain on AirFlow
This is an awesome web-based diagramming tool. It has all the stencils I could need and suits my basic diagramming needs. I usually switch between OmniGraffle, GoodNotes+Apple Pencil and DrawIO.
Headphones is an automated music downloader for NZB and Torrent
This may seem like a controversial service but I use it as an easy way to reach album data and to find albums contents. I do not use the torrent/usenet integration.
This is a web-based interface to MySQL. It has been around for ages! It has a lot of features and is a great replacement for the native database client apps if you are dealing with MySQL/MariaDB. I use it to access a development MySQL database instance that is part of the same stack as this.
“A self-hosted podcast manager to download episodes as soon as they become live” - I use it just for that. There are a few podcasts that I follow and I like to archive them because I know what it means to not be able to a favorite podcast from decades ago.
“Lightweight and easy-to-use repository manager for Maven based artifacts in the JVM ecosystem” - Basically PiPy but for Java.
This is a source search engine that has great searching and parsing abilities making it very easy to navigate code. It’s pretty much like an IDE without the edit/build part. It is so much easier to use than gitea when searching for a piece of code or trying to remember how I implemented something in some project. It’s a bit resource hungry (even the evaluation setup) but I keep it running.
TradeBrain is a program that I develop to run technical analysis on all sorts of investment instruments.
Your self hosted YouTube media server
This is a great application to download and archive Youtube videos. I personally have a couple of channels that I follow, which have removed content (after making deals with big websites - yes I’m talking about you chessbrah.) so I keep a copy of every video. I also like to use this as a way of limiting what my kid gets to watch using the download queue feature (Thanks to http://noted.lol for the tip). I have a Jenkins task that will move the downloaded video files to a location where Plex can index and serve them.
“WebDAV stands for "Web-based Distributed Authoring and Versioning". It is a set of extensions to the HTTP protocol which allows users to collaboratively edit and manage files on remote web servers.”
This is another archaic protocol that has been around for a very long time. I use it to sync my devonTHINK databases between all the different devices. Since I mentioned it, note taking and document management is the only thing I don’t host on a server, because devonTHINK is a great application that does everything I need, and does it properly. The only downside is that it’s a macOS/iOS only application, which means vendor lock-in. For now I’m happy with Apple platforms so it’s not a problem that I need to deal with immediately.
This is another custom app that I run to process and webhooks from other services. One example for this is the service that will process incoming requests from Plex about tracks that I play and post them to HumHub.
This is a self hosted version of WhatPulse that is work in progress.
This is a single user micro blog.
2301 words
Powered by TF-IDF/Cosine similarity
First published on 2022-06-08 ~ 2024-05-04
Generated on 13 Dec 2024 at 12:11 AM
Mobile optimized version. Desktop version.