tutorial raspberry pi - 28 January 2021

Self-Hosting on a Raspberry Pi

Docker: Traefik v2 + DuckDNS + Portainer + Nextcloud + Bitwarden

TLDR: In this tutorial, I am going to show you the process of using your Raspberry Pi as a self-hosting server. To allow for easy deployment and management of different services we are going to use Docker. Additionally, I am going to show you how to use the Traefik v2 to access different services like Nextcloud and Bitwarden securly from outside of your network.

Motivation

Recently, while organizing my old files, I realized that I wanted to come up with my own file hosting solution. I was using both Google Drive and Dropbox previously, and so I was used to a rather comfortable configure-once-and-forget solution.

After some research, I found Nextcloud which is a self-hosted open-source application. Nextcloud is quite similar to Dropbox or Google Mail, as there are syncing clients for most popular platforms like Windows, MacOs as well as mobile solutions. It also comes with a webinterface, where you can access your files, manage users and a wide variety of additional apps.

While you could install the default Nextcloud service on the rapsberry, I decided to go with a special dedicated build for the Raspberry, called NextcloudPi. NextcloudPi comes with an additional interface, which allows managing additional administration functionality.

Further, I decided to also setup a Bitwarden instance. Bitwarden is open-source password manager, which can also be self-hosted. While the official version of Bitwarden seemed to use quite a lof of resources, I opted for a light-way implementation of it in Bitwarden_rs. Which is an unofficial implementation of Bitwarden programmed in Rust.

Setup - Raspberry Pi

First we are going to setup a fresh Raspberry. For that we are using the Raspberry Pi OS Light. The Light OS does not come with a virtual desktop environment which means you will not have a graphical interface when using an external display and are stuck working in the terminal or accessing the raspberry per ssh. But as a homeserver this will be sufficient most of the time and use less resources.

If you want the ability to use the raspberry with an external monitor you will need to use the normal Raspberry OS. Both can be downloaded here:

Raspberry Pi OS

After downloading and extracting the image the resulting '''*--raspios-buster-armhf.img''' file can then be flashed to the Raspberry. For that one can use an usb or sd (with an adopter) and by flashing it with an usb flashing software, like:

Etcher

(Newer USBs tend to be a lot faster, compared to newer SD cards.)

After flashing the Raspberry, I would recommend to also place an empty file, called '''ssh''' in the root of the Raspberry to automatically enable ssh.

After booting the Raspberry and connecting it to your homenetwork via LAN it can be access by entering the following command in the console:

''' ssh [email protected] // 192.168.1.13 needs to replaced with the correct ip of your Raspberry '''

This opens a ssh connection to the Raspberry where the default credentials can be used to login, which are:

user: pi
pass: raspberrry

Setup - Docker (and docker-compose)

Most self-hosted application for the Raspberry come with a preconfigured image, but this complicates the process to easily add additional applications. Additional to the normal pre-configured images, most of these applications also provide an easy-to-use Docker image. Docker allows to configure a multitude of containerized applications comfortably, those run in their own container, which can be spun up and destroyed when needed. For that we need to install Docker, this can be done by connecting to the Raspberry, then entering following commands:

sudo apt update
sudo apt upgrade

These commands update the Raspberry OS' default package manager, which then can be used to install new software.

sudo apt install curl
curl -sSL https://get.docker.com | sh

install curl installs curl a command-line tool, this is then used to tranfer data to the Raspberry. curl -sSL https://get.docker.com | sh downloads the Docker installation script, which then is executed via the | sh.

While this Docker installation is already enough to use Docker containers to run applications, we want to have the ability to configure them more freely and combine them. For that we are going to also install docker-compose. This is done by entering the following commands first, which install the needed dependencies:

sudo apt install -y libffi-dev libssl-dev
sudo apt install -y python3 python3-pip
sudo apt remove python-configparser

And then to install docker-compose:

sudo pip3 -v install docker-compose

Setup - DuckDNS

To be able to access your addresses in your network from outside you will need to use your specific IP, this IP can easily be retrieved by entering "whats my IP" in Google or using a website like whatsmyip.com.

The problem is that this IP can change at random intervals, aka it is dynamic, and therefore we need a service which updates our ip when this happens. For this we use DuckDNS, DuckDNS come with the ability to generate 5 domains(in the free tier) and point them to our IP, is perfect for our purpose.

After creating an account on DuckDNS you can generate a new subdomain and access your DuckDNS token, which we will use later on.

For each application, which we want to access later on we now have to generate a new subdomain.

portainer
nextcloudpi
bitwarden_rs

Installation - Traefik v2 + DuckDNS

Let us now start with configuring our Docker applications. First create a new file called docker-compose.yml. This can be done by connecting to the Raspberry and executing following commands:

touch docker-compose.yml
touch acme.json
sudo chown 600 acme.json

Now open docker-compose.yml and insert the following text:

version: "3.3"

services:
  traefik:
    container_name: traefik
    image: "traefik:latest"
    restart: always
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.ncp.address=:4443
      - --providers.docker
      - --log.level=ERROR
      - --certificatesresolvers.leresolver.acme.httpchallenge=true
      - --certificatesresolvers.leresolver.acme.email=[email]
      - --certificatesresolvers.leresolver.acme.storage=./acme.json
      - --certificatesresolvers.leresolver.acme.httpchallenge.entrypoint=web
      - --serverstransport.insecureskipverify=true
    ports:
      - "80:80"
      - "4443:4443"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/home/pi/acme.json:/acme.json"
    labels:
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
    restart: unless-stopped

  duckdns:
    image: ghcr.io/linuxserver/duckdns
    container_name: duckdns
    environment:
      - PUID=1000 #optional
      - PGID=1000 #optional
      - TZ=Europe/London
      - SUBDOMAINS=[domains]
      - TOKEN=[token]
    restart: unless-stopped

Now, let us break this down.

traefik:
  container_name: traefik
  image: "traefik:latest"
  restart: always

Defines a new service, which uses the latest treafik image.

command:
  - --entrypoints.web.address=:80
  - --entrypoints.websecure.address=:443
  - --entrypoints.ncp.address=:4443
  - --providers.docker
  - --log.level=ERROR
  - --certificatesresolvers.leresolver.acme.httpchallenge=true
  - --certificatesresolvers.leresolver.acme.email=[email]
  - --certificatesresolvers.leresolver.acme.storage=./acme.json
  - --certificatesresolvers.leresolver.acme.httpchallenge.entrypoint=web
  - --serverstransport.insecureskipverify=true

This first defines 3 different entrypoints, called "web", "websecure" and "ncp". Each of these belongs to the specific port.

Additionally, you have to replace the [email] with your email address, which is then used to authorize https via Let's Encrypt automatically for your domains.

ports:
  - "80:80"
  - "4443:4443"
  - "443:443"

Here, we define 3 ports which will be accessible from outside the specific Docker service. These are not necessary but can be handy to debug your applications by using the port on the local IP address. The first number defines the external port, the second number defines the matching internal port. This for example allows to access our previously defined "web" endpoint externally on port 80.

volumes:
  - "/var/run/docker.sock:/var/run/docker.sock:ro"
  - "/home/pi/acme.json:/acme.json"
labels:
  - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
  - "traefik.http.routers.http-catchall.entrypoints=web"
  - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
  - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
restart: unless-stopped

In this last part we define 2 volumes, the first one allows traefik to communicate with docker, and the second one uses the previously defined acme.json to save the automatically generated https certificates. We additionally define labels, which tell traefik to redirect all incoming http request to the https. Also, we set the restart behavior of the service to automatically restart unless stopped by you, through portainer or the console.

duckdns:
  image: ghcr.io/linuxserver/duckdns
  container_name: duckdns
  environment:
    - PUID=1000 #optional
    - PGID=1000 #optional
    - TZ=Europe/London
    - SUBDOMAINS=[domains]
    - TOKEN=[token]
  restart: unless-stopped

In the last code block, we define an additional service which automatically refreshes the DuckDNS domain, to catch when the external IP changes. Here, you have to replace [domains] with your previously generated DuckDNS domains separated by a comma. Only write the subdomain and not the full domain, for example if your domains are domain1.duckdns.org and domain2.duckdns.org you insert domain1,domain2.

Installation - Portainer (optional)

Portainer is a nifty tool, which allows to graphically manage your Docker containers. While it is not necessary for our setup, it can be extremely helpful when adding and testing applications in the future.

For this we add following code to our previous docker-compose.yml:

portainer:
  image: portainer/portainer-ce:latest
  command: -H unix:///var/run/docker.sock
  container_name: portainer
  restart: always
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - portainer_data:/data

This defines our portainer service. The volume part`, for one handles the communication with the docker instance and also defines a portainer volume, where all data of portainer is stored.

labels:
  # Frontend
  - "traefik.enable=true"
  - "traefik.http.routers.frontend.rule=Host(`[domain-portainer]`)"
  - "traefik.http.routers.frontend.entrypoints=websecure"
  - "traefik.http.services.frontend.loadbalancer.server.port=9000"
  - "traefik.http.routers.frontend.service=frontend"
  - "traefik.http.routers.frontend.tls.certresolver=leresolver"

These labels define how trafik handles the portainer service. You will need to replace [domain-portainer] with your previously generated portainer DuckDNS domain.

- "traefik.http.routers.frontend.service=frontend"

Here we define a new traefik service called `frontend.

- "traefik.http.routers.frontend.entrypoints=websecure"
- "traefik.http.services.frontend.loadbalancer.server.port=9000"

These labels define that the newly generated traefik service frontend uses the https endpoint and that traefik listens to the port 9000 of the portainer application.

- "traefik.http.routers.frontend.tls.certresolver=leresolver"

Furthermore, we define here that we use our previously defined https certificate handler.

Installation - Bitwarden_rs

The Bitwarden setup is rather straight forward, for this we have to append following code to the docker-compose.yml file:

bitwarden:
  image: bitwardenrs/server:latest
  volumes:
    - bw_data:/data
  restart: always
  labels:
    - "traefik.enable=true"
    - "traefik.http.routers.bitwarden.rule=Host(`[domain-bitwarden]`)"
    - "traefik.http.routers.bitwarden.entrypoints=websecure"
    - "traefik.http.services.bitwarden.loadbalancer.server.port=80"
    - "traefik.http.routers.bitwarden.service=bitwarden"
    - "traefik.http.routers.bitwarden.tls.certresolver=leresolver"

Attention, here we need to replace [domain-bitwarden] by your previously defined DuckDNS domain.

Installation - NextcloudPi

To setup NextcloudPi we are going to use the Docker image of NextcloudPi. For this we are going append following code to our docker-compose.yml file.

nextcloudpi:
  image: ownyourbits/nextcloudpi
  volumes:
    - [mounted-usb]/ncdata:/data
  restart: always
  command: "[domain-nextcloud]"
  labels:
    - "traefik.enable=true"
    - "traefik.port=443"
    - "traefik.http.routers.ncp_web.rule=Host(`[domain-nextcloud]`)"
    - "traefik.http.routers.ncp_web.entrypoints=websecure"
    - "traefik.http.services.ncp_web.loadbalancer.server.port=80"
    - "traefik.http.routers.ncp_web.service=ncp_web"
    - "traefik.http.routers.ncp_web.tls.certresolver=leresolver"
  
    - "traefik.http.routers.ncp_websecure.rule=Host(`[domain-nextcloud]`)"
    - "traefik.http.services.ncp_websecure.loadbalancer.server.scheme=https"
    - "traefik.http.routers.ncp_websecure.entrypoints=websecure"
    - "traefik.http.services.ncp_websecure.loadbalancer.server.port=443"
    - "traefik.http.routers.ncp_websecure.service=ncp_websecure"
    - "traefik.http.routers.ncp_websecure.tls=true"
    - "traefik.http.routers.ncp_websecure.tls.certresolver=leresolver"
    
    - "traefik.http.routers.ncp_admin.rule=Host(`[domain-nextcloud]`)"
    - "traefik.http.services.ncp_admin.loadbalancer.server.scheme=https"
    - "traefik.http.routers.ncp_admin.entrypoints=ncp"
    - "traefik.http.services.ncp_admin.loadbalancer.server.port=4443"
    - "traefik.http.routers.ncp_admin.service=ncp_admin"
    - "traefik.http.routers.ncp_admin.tls.certresolver=leresolver"

Here define our Nextcloud service using the ownyourbits/nextcloudpi image. Additionally, we define three different service endpoints. One for port 80, which is automatically redirected to 443, one for port 443 to access the "normal" Nextcloud and port 4443, which gives use access to the special NextcloudPi webinterface. Then we have to replace [domain-nextcloud] with our previously defined Nextcloud DuckDNS domain. In the end, we have to change [mounted-usb] for the data of our Nuxtcloud instance.

Startup and Configuration

Now we can start our defined Docker container by entering following in the console:

sudo docker-compose up -d

This downloads all necessary images and configures them according to our definition.

If all services start up successful, we have to configure the Nextcloud instance further.

For this we need to connect to the NextcloudPi container. This can either be done by using the Portainer webinterface or by entering following in the console:

docker exec -it [container] /bin/bash

Here, we have to replace [container] with the name or id of our NextcloudPi Docker container. After connecting to the console of the container, we can start the NextcloudPi configurator by entering:

ncp-config

For one, we have to update the NextcloudPi instance, and the underlying Nextcloud instance by navigating to UPDATES. Attention, when updating the underlying Nextcloud instance you can only update one major version at once. So, if you update from 18.0.0 to 20.0.0, you first need to update to 19.0.0 and then to 20.0.0.

Those updates can take some time, so be patient until they finish completely and ask for your input.

Now you can access all your services by visiting you defined subdomains and configure them to your liking.

(Sometimes you will need to also add your [domain-nextcloud] in the Configurator as trusted domain. This can be done by opening NETWORKING in the configurator and selecting trusted domains.)

Full Code - docker-compose.yml

version: "3.3"

services:
  traefik:
    container_name: traefik
    image: "traefik:latest"
    restart: always
    command:
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.ncp.address=:4443
      - --providers.docker
      - --log.level=ERROR
      - --certificatesresolvers.leresolver.acme.httpchallenge=true
      - --certificatesresolvers.leresolver.acme.email=[email]
      - --certificatesresolvers.leresolver.acme.storage=./acme.json
      - --certificatesresolvers.leresolver.acme.httpchallenge.entrypoint=web
      - --serverstransport.insecureskipverify=true
    ports:
      - "80:80"
      - "4443:4443"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/home/pi/acme.json:/acme.json"
    labels:
      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
      - "traefik.http.routers.http-catchall.entrypoints=web"
      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"

  duckdns:
    image: ghcr.io/linuxserver/duckdns
    container_name: duckdns
    environment:
      - PUID=1000 #optional
      - PGID=1000 #optional
      - TZ=Europe/London
      - SUBDOMAINS=[domains]
      - TOKEN=[token]
    restart: unless-stopped
    
  portainer:
    image: portainer/portainer-ce:latest
    command: -H unix:///var/run/docker.sock
    restart: always
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data
    labels:
      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend.rule=Host(`[domain-portainer]`)"
      - "traefik.http.routers.frontend.entrypoints=websecure"
      - "traefik.http.services.frontend.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend.service=frontend"
      - "traefik.http.routers.frontend.tls.certresolver=leresolver"
    
      # Edge
      - "traefik.http.routers.edge.rule=Host(`[domain-portainer]`)"
      - "traefik.http.routers.edge.entrypoints=websecure"
      - "traefik.http.services.edge.loadbalancer.server.port=8000"
      - "traefik.http.routers.edge.service=edge"
      - "traefik.http.routers.edge.tls.certresolver=leresolver"
  bitwarden:
    image: bitwardenrs/server:latest
    volumes:
      - bw_data:/data
    restart: always
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.bitwarden.rule=Host(`[domain-bitwarden]`)"
      - "traefik.http.routers.bitwarden.entrypoints=websecure"
      - "traefik.http.services.bitwarden.loadbalancer.server.port=80"
      - "traefik.http.routers.bitwarden.service=bitwarden"
      - "traefik.http.routers.bitwarden.tls.certresolver=leresolver"
  
  nextcloudpi:
    image: ownyourbits/nextcloudpi
    volumes:
      - [mounted-usb]/ncdata:/data
    restart: always
    command: "[domain-nextcloud]"
    labels:
      - "traefik.enable=true"
      - "traefik.port=443"
      - "traefik.http.routers.ncp_web.rule=Host(`[domain-nextcloud]`)"
      - "traefik.http.routers.ncp_web.entrypoints=websecure"
      - "traefik.http.services.ncp_web.loadbalancer.server.port=80"
      - "traefik.http.routers.ncp_web.service=ncp_web"
      - "traefik.http.routers.ncp_web.tls.certresolver=leresolver"
    
      - "traefik.http.routers.ncp_websecure.rule=Host(`[domain-nextcloud]`)"
      - "traefik.http.services.ncp_websecure.loadbalancer.server.scheme=https"
      - "traefik.http.routers.ncp_websecure.entrypoints=websecure"
      - "traefik.http.services.ncp_websecure.loadbalancer.server.port=443"
      - "traefik.http.routers.ncp_websecure.service=ncp_websecure"
      - "traefik.http.routers.ncp_websecure.tls=true"
      - "traefik.http.routers.ncp_websecure.tls.certresolver=leresolver"
      
      - "traefik.http.routers.ncp_admin.rule=Host(`[domain-nextcloud]`)"
      - "traefik.http.services.ncp_admin.loadbalancer.server.scheme=https"
      - "traefik.http.routers.ncp_admin.entrypoints=ncp"
      - "traefik.http.services.ncp_admin.loadbalancer.server.port=4443"
      - "traefik.http.routers.ncp_admin.service=ncp_admin"
      - "traefik.http.routers.ncp_admin.tls.certresolver=leresolver"

volumes:
  portainer_data:
  bw_data:

David Lengweiler