Home on apalrd's adventures

Starting my Content Delivery Network

Fri, 15 May 2026 01:00:05 -0300

Today I’m finally starting to put my AS Number to use, rolling out my Content Delivery Network! Currently it’s actually not delivering anything useful, just some neat statistics, but it’s helping me learn how CDNs work and share that with all of you.

Here’s a neat visualizer which helps show how users end up at the CDN, when directed via DNS (geo-dns) and BGP (anycast).

AI disclaimer: The DNS server (github link) and my-ip page (link) were written without the use of any AI tools. The visualization on this page was made using Claude AI, since I am not that good at Javascript.

Med DNS Latency

—ms

Med BGP Latency

—ms

—

Total Records

—

Countries

—

Autonomous Systems

—

Non-ISP Resolver

—

ECS Supported

Latency (median)

Group by Continent -

DNS BGP

Loading…

BGP − DNS Difference (Median)

Group by Continent -

BGP faster DNS faster

Loading…

Why I Don't Link DHCP + DNS (3 ways to update DNS)

Mon, 13 Apr 2026 01:00:05 -0300

Today I’m tackling an issue that gets a lot of network engineers really hung up - in the old days, pushing DHCP data into DNS seemed like a good idea, but now it’s just garbage all around. What are some solutions you can use instead? Today I have three solutions - mDNS (multicast DNS), pushing DNS configs as part of automation (Ansible example), and pushing DNS from the host itself using the DNS protocol (nsupdate).

Video

Multicast DNS

This method is the easiest, and everyone with a small network should be using this method. It’s installed by default on most clients, but on Debian / Proxmox VE, here’s what you need to do:

apt install avahi-daemon -y

You also need to edit /etc/nsswitch.conf to fix IPv6 not being resolved (remove ‘4’ from ‘mdns4_minimal’):

--- a/etc/nsswitch.conf.bak
+++ b/etc/nsswitch.conf
@@ -9,7 +9,7 @@ group:          files systemd
 shadow:         files systemd
 gshadow:        files systemd
 
-hosts:          files mdns4_minimal [NOTFOUND=return] dns
+hosts:          files mdns_minimal [NOTFOUND=return] dns
 networks:       files

Ansible Node Provisioning

Once your business gets big enough to have your own servers (on-prem or virtual), not just SaaS solutions, you are probably ready to graduate from flat network mDNS to running your own DNS server. I like to push my server configs to DNS as part of the provisioning process. Here’s a basic Ansible playbook that I wrote which does exactly that:

- name: Provision Proxmox VM and register IPv6 DNS record
  hosts: localhost
  # proxmox host, api keys, .. go in a vars file
  # or group vars if you prefer
  vars_files:
    - vars.yml

  connection: local
  gather_facts: false
  vars:
    # config of the new VM
    template_vmid: 903          #This is a VM template using cloud-init and already configured
    new_hostname: "testing"
    new_cores: 4
    new_memory: 2048
    boot_disk_size: "64G"
    boot_disk_device: "scsi0"
    # IPv6 prefix of the network the VM is on
    ipv6_prefix: "2a0f:b240:1001:67"

  tasks:
# Get next available VMID
    - name: Get next available VMID from Proxmox
      ansible.builtin.uri:
        url: "https://{{ proxmox_host }}:8006/api2/json/cluster/nextid"
        method: GET
        headers:
          Authorization: "PVEAPIToken={{ proxmox_user }}!{{ proxmox_token_id }}={{ vault_proxmox_password }}"
        validate_certs: "{{ proxmox_verify_ssl }}"
        status_code: 200
      register: nextid_result

    - name: Set new_vmid
      ansible.builtin.set_fact:
        new_vmid: "{{ nextid_result.json.data | int }}"

    - name: Show allocated VMID
      ansible.builtin.debug:
        msg: "Proxmox allocated VMID: {{ new_vmid }}"

# Clone template vmid to new vmid
    - name: Clone VM {{ template_vmid }} to {{ new_vmid }}
      ansible.builtin.uri:
        url: "https://{{ proxmox_host }}:8006/api2/json/nodes/{{ proxmox_node }}/qemu/{{ template_vmid }}/clone"
        method: POST
        headers:
          Authorization: "PVEAPIToken={{ proxmox_user }}!{{ proxmox_token_id }}={{ proxmox_password }}"
        body_format: json
        body:
          newid: "{{ new_vmid }}"
          name: "{{ new_hostname }}"
          full: 1
        validate_certs: "{{ proxmox_verify_ssl }}"
        status_code: 200
      register: clone_result

    - name: Wait for clone task to complete
      ansible.builtin.uri:
        url: "https://{{ proxmox_host }}:8006/api2/json/nodes/{{ proxmox_node }}/tasks/{{ clone_result.json.data }}/status"
        method: GET
        headers:
          Authorization: "PVEAPIToken={{ proxmox_user }}!{{ proxmox_token_id }}={{ proxmox_password }}"
        validate_certs: "{{ proxmox_verify_ssl }}"
      register: task_status
      until: task_status.json.data.status == 'stopped'
      retries: 30
      delay: 1

# Get VM's MAC address to compute EUI64, add prefix, add to DNS
# net0 will look like:
# virtio=AA:BB:CC:DD:EE:FF,bridge=vmbr0
    - name: Fetch VM config from Proxmox API
      ansible.builtin.uri:
        url: "https://{{ proxmox_host }}:8006/api2/json/nodes/{{ proxmox_node }}/qemu/{{ new_vmid }}/config"
        method: GET
        headers:
          Authorization: "PVEAPIToken={{ proxmox_user }}!{{ proxmox_token_id }}={{ vault_proxmox_password }}"
        validate_certs: "{{ proxmox_verify_ssl }}"
        return_content: true
        status_code: 200
      register: vm_config_raw

    - name: Parse MAC address from net0
      ansible.builtin.set_fact:
        mac_address: >-
          {{
            vm_config_raw.json.data.net0
            | regex_search('([0-9A-Fa-f]{2}(?::[0-9A-Fa-f]{2}){5})', '\1')
            | first
            | upper
          }}          

    - name: Show extracted MAC address
      ansible.builtin.debug:
        msg: "MAC address: {{ mac_address }}"

# We need to flip a bit to go from MAC to EUI64, and that's actually
# surprisingly hard to do in Jinja
# so here's a filter plugin I found, we just need bitwise xor here:
# https://eengstrom.github.io/musings/add-bitwise-operations-to-ansible-jinja2
    - name: Compute EUI-64 IPv6 address
      ansible.builtin.set_fact:
        ipv6_addr: >-
          {{
            (
              ipv6_prefix ~ ':' ~
              ('%02X' % (mac_octets[0] | int(base=16) | bitwise_xor(0x02))) ~ mac_octets[1] ~ ':' ~
              mac_octets[2] ~ 'ff:' ~
              'fe' ~ mac_octets[3] ~ ':' ~
              mac_octets[4] ~ mac_octets[5]
            ) | ansible.utils.ipaddr('address')
          }}          
      vars:
        mac_octets: "{{ mac_address.split(':') }}"

    - name: Show computed IPv6 address
      ansible.builtin.debug:
        msg: "EUI-64 IPv6: {{ ipv6_addr }}"

    - name: Add AAAA record to DNS
      ansible.builtin.uri:
        url: "{{ technitium_url }}/api/zones/records/add"
        method: POST
        headers:
          Content-Type: "application/x-www-form-urlencoded"
        body_format: form-urlencoded
        body:
          token: "{{ technitium_token }}"
          domain: "{{ new_hostname }}.{{ dns_zone }}"
          zone: "{{ dns_zone }}"
          type: AAAA
          ipAddress: "{{ ipv6_addr }}"
          ttl: "{{ dns_ttl }}"
          overwrite: "true"
          ptr: "true"
        status_code: 200
        validate_certs: "{{ technitium_verify_ssl }}"
      register: dns_result

    - name: Show DNS API response
      ansible.builtin.debug:
        msg: "{{ dns_result.json }}"

# Write out everything we just did
    - name: Provisioning complete
      ansible.builtin.debug:
        msg:
          - "VM {{ new_vmid }} ({{ new_hostname }}) created on node {{ proxmox_node }}"
          - "MAC: {{ mac_address }}"
          - "IPv6 (EUI-64): {{ ipv6_addr }}"
          - "DNS AAAA record: {{ new_hostname }}.{{ dns_zone }} → {{ ipv6_addr }}"

You also need a vars.yml (and possibly vault.yml) to store these commonly used constants like the Proxmox host, usernames, passwords, … but that’s beyond the scope of this tutorial

Dynamic DNS (nsupdate)

Third option is to let hosts push their own updates to DNS, using nsupdate (RFC 2136). Again, I’m using Debian as my example OS here.

The update process for rfc 2136 is basically:

Host does a DNS command to update its own records (this runs over normal DNS - port 53 - nothing special)
Host signs its command with a unique transaction signature (TSIG), which is a shared secret - we need a unique secret PER HOST if we want this to be secure and not let any host change any other host’s DNS name
DNS server validates that the TSIG is correct
DNS server validates that this TSIG is allowed to access these records
DNS server commits update

So, we first need to add a new TSIG in Technitium. In the web UI it’s under Settings -> TSIG -> Add, if you leave the key blank it will generate a random value. Create a new key with the name of your specific host here.

Now, we need to create a key file as root (I used /etc/tsig.key for this) - obviously use your own secret here and your own key name!

key "host-testing" {
	algorithm hmac-sha256;
	secret "OCop85LAjLvdkf7J+3mar+uvTjufHQtiZozRp7z8VIg=";
};

Next, we need to setup the system. We need to install bind9-dnsutils on Debian, which gets us the nsupdate utility. The utility is actually not very useful at all on its own, so I wrote a script in Bash which gets all of the local IPs with hostname -I, splits them into A and AAAA records, and builds the list of commands for nsupdate. Put this script in /usr/local/bin/ddns and don’t forget to chmod +x:

#!/bin/bash
HOST="$(hostname)"
FQDN="$HOST.apalrd.fi"

{
    # Update your server and zone here
    echo "server polaris.apalrd.fi"
    echo "zone apalrd.fi"

    # Remove existing records
    echo "update delete $FQDN A"
    echo "update delete $FQDN AAAA"

    # Add all IPv4 addresses (skip loopback)
    for ip in $(hostname -I); do
        if [[ "$ip" != *:* && "$ip" != 127.* ]]; then
        echo "update add $FQDN 300 A $ip"
        fi
    done

    # Add all IPv6 addresses (skip loopback + link-local)
    for ip in $(hostname -I); do
        if [[ "$ip" == *:* && "$ip" != ::1 && "$ip" != fe80::* ]]; then
        echo "update add $FQDN 300 AAAA $ip"
        fi
    done

    #it will send this all as one dns packet
    echo "send"
} | nsupdate -k /etc/tsig.key

Now we just need to call this periodically. I’ve set it up with a systemd timer that runs hourly, plus every time the system starts up.

First, the systemd service (which actually does the updating) - put this in /etc/systemd/system/ddns.service:

[Unit]
Description=Update dynamic DNS via nsupdate

[Service]
Type=oneshot
ExecStart=/usr/local/bin/ddns

Second, the timer (which calls the service periodically) - put this in /etc/systemd/system/ddns.timer:

[Unit]
Description=Run dynamic dns periodically

[Timer]
#Just after boot
OnBootSec=2min
#Every 15mins after that
OnUnitActiveSec=15min
Persistent=true

[Install]
WantedBy=timers.target

And enable the timer (but not the service!):

systemctl daemon-reload
systemctl enable --now ddns.timer

There may be a better way to integrate with the network events, but this is the simplest thing I have to show

How I Automated my Network with Ansible

Wed, 18 Mar 2026 01:00:05 -0300

Today, in the next eposide on my Personal AS series, I have added a third POP / fourth router, and that means I need to configure the POP all over again. That’s a lot of work, so I automated it with Ansible! Also follow along to see my first use of Netbox for automation

Video

Ansible Inventory

Here’s the final version of my Ansible inventory/hosts.yml file. Long term, ideally I can move all of these attributes to custom attributes in Netbox, or even pulling the whole inventory from Netbox, instead of maintaining this file separately.

routers:
  hosts:
    waw-pe1.apalrd.fi:
      router_id: 1
      host: waw-pe1
      nat64: true
      pop: waw
      backup: true
      peers:
        - lax-pe1.apalrd.fi
        - sjy-p1.apalrd.fi
        - grr-e1.apalrd.fi
    lax-pe1.apalrd.fi:
      router_id: 2
      host: lax-pe1
      nat64: true
      pop: lax
      backup: true
      peers: 
        - waw-pe1.apalrd.fi
        - sjy-p1.apalrd.fi
        - grr-e1.apalrd.fi
    sjy-p1.apalrd.fi:
      router_id: 3
      host: sjy-p1
      peers:
        - waw-pe1.apalrd.fi
        - lax-pe1.apalrd.fi
        - grr-e1.apalrd.fi
    grr-e1.apalrd.fi:
      router_id: 4
      host: grr-e1
      nat64: true
      pop: grr
      backup: true
      peers:
        - sjy-p1.apalrd.fi
        - waw-pe1.apalrd.fi
        - lax-pe1.apalrd.fi

Ansible Software Setup

This is the script that installs all of the software (and running it again will update it!), enables backups, etc.

- hosts: "*"
  tasks:
#Global package update
    - name: apt upgrade
      apt: 
        update_cache: yes
        upgrade: yes
#Packages required to install package repos
    - name: Pre required packages
      ansible.builtin.apt:
        pkg:
          - apt-transport-https 
          - ca-certificates
          - gpg
#Backup to PBS
    - name: Proxmox apt key
      ansible.builtin.get_url:
        url: https://enterprise.proxmox.com/debian/proxmox-archive-keyring-trixie.gpg
        dest: /usr/share/keyrings/proxmox-archive-keyring.gpg
      when: backup is defined and backup
    - name: Proxmox client repository
      ansible.builtin.apt_repository:
        repo: "deb [arch=amd64 signed-by=/usr/share/keyrings/proxmox-archive-keyring.gpg] http://download.proxmox.com/debian/pbs-client {{ ansible_distribution_release }} main"
        state: present
      when: backup is defined and backup
    - name: Proxmox Backup Client install from packages
      ansible.builtin.apt:
        update_cache: yes
        name: proxmox-backup-client
      when: backup is defined and backup
    - name: Copy Proxmox Backup Service
      ansible.builtin.template:
        src: proxmox-backup.service
        dest: /etc/systemd/system/
        owner: root
        group: root
        mode: '0644'
      when: backup is defined and backup
    - name: Copy Proxmox Backup Timer
      ansible.builtin.template:
        src: proxmox-backup.timer
        dest: /etc/systemd/system/
        owner: root
        group: root
        mode: '0644'
      when: backup is defined and backup
    - name: Enable backup timer
      ansible.builtin.systemd_service:
        name: proxmox-backup.timer
        enabled: "{{ backup | default(false)}}"
        daemon-reload: True
#Configuration for bird3
    - name: bird3
      block:
      #Package install steps
        - name: bird3 apt key
          ansible.builtin.get_url:
            url: https://pkg.labs.nic.cz/gpg
            dest: /usr/share/keyrings/cznic-labs-pkg.gpg
        - name: bird3 repo
          ansible.builtin.apt_repository:
            repo: "deb [signed-by=/usr/share/keyrings/cznic-labs-pkg.gpg] https://pkg.labs.nic.cz/bird3 {{ ansible_distribution_release }} main"
            state: present
        - name: Bird3 install from packages
          ansible.builtin.apt:
            update_cache: yes
            name: bird3
#Tayga
    - name: Tayga
      when: nat64 is defined and nat64
      block:
        - name: tayga required packages
          ansible.builtin.apt:
            pkg:
              - build-essential
              - git
        - name: Git checkout
          ansible.builtin.git:
            repo: 'https://github.com/apalrd/tayga'
            dest: /root/tayga
            force: true
        - name: Compile
          ansible.builtin.shell:
            chdir: /root/tayga
            cmd: make && make install WITH_SYSTEMD=1 LIVE=1
        - name: Disable default nat64 instance
          ansible.builtin.systemd_service:
            name: tayga@default
            enabled: False
            state: stopped
        - name: Tayga local service
          ansible.builtin.systemd_service:
            name: tayga@local
            enabled: True
            state: started
        - name: Tayga wkpf service
          ansible.builtin.systemd_service:
            name: tayga@wkpf
            enabled: True
            state: started

Here’s my Proxmox Backup systemd service/timer, in case you want to use those too:

#/etc/systemd/system/proxmox-backup.service
[Unit]
Description=Run Backup to Proxmox Backup Server
After=network-online.target

[Service]
Environment=PBS_PASSWORD=your-api-key-nere
Environment=PBS_REPOSITORY=user@pbs!apikey@pbs.apalrd.fi:repo
Type=oneshot
ExecStart=proxmox-backup-client backup root.pxar:/

[Install]
WantedBy=default.target

#/etc/systemd/system/proxmox-backup.timer
[Unit]
Description=Backup System Daily
RefuseManualStart=no
RefuseManualStop=no

[Timer]
#Run 540 seconds after boot for the first time
OnBootSec=540
#Run at midnight UTC
OnCalendar=*-*-* 00:00:00
Unit=proxmox-backup.service

[Install]

Ansible Network Conf

Here’s the two Ansible playbooks to configure systemd networkd:

#pop_conf_net.yml
- hosts: "*"
  collections:
    - netbox.netbox
    - ansible.utils
  vars:
    netbox_url: "https://docs.peach.apalrd.fi"
    netbox_token: 0DjI3ZTMQdQPEbti07i6KYnan8jOi0f2K9XS0Qm2
  tasks:
#Query peer primary addresses
    - name: Build peer_hosts list from inventory
      set_fact:
        peer_hosts: "{{ peer_hosts | default([]) + [hostvars[item].host] }}"
      loop: "{{ peers }}"
      when: hostvars[item].host is defined

    - name: Query potential peers from Netbox VM list
      set_fact:
        all_vms: >-
          {{
            lookup(
              'netbox.netbox.nb_lookup',
              'virtual-machines',
              api_endpoint=netbox_url,
              token=netbox_token,
              validate_certs=False
            )
          }}          

    - name: Build IPv6 map for requested VMs
      set_fact:
        peer_ipv6_map: >-
          {{
            peer_ipv6_map | default({}) |
            combine({
              item.value.name:
                item.value.primary_ip6.address
                | ansible.utils.ipaddr('address')
            })
          }}          
      loop: "{{ all_vms }}"
      no_log: true
      when:
        - item.value.name in peer_hosts
        - item.value.primary_ip6 is defined

    - name: Fail if any peer is missing
      fail:
        msg: "No primary IPv6 found in NetBox for: {{ peer_hosts | difference(peer_ipv6_map.keys() | list) }}"
      when: peer_hosts | difference(peer_ipv6_map.keys() | list) | length > 0

    - name: Show peer IPv6 addresses
      debug:
        msg: "Primary IPv6 for {{ item }} is {{ peer_ipv6_map[item] }}"
      loop: "{{peer_hosts}}"

    - name: Get my own IPv6 address
      set_fact:
        my_ip: >-
          {{
            (all_vms | selectattr('value.name','equalto',host) | list)[0].value.primary_ip6.address
            | ansible.utils.ipaddr('address')
          }}          

    - name: Show my own IPv6 address
      debug:
        msg: "My IP is {{my_ip}}"

#Configuration for Tunnel Interfaces
    - name: Tunnel Interfaces
      ansible.builtin.include_tasks:
        file: pop_conf_net_tunnel.yml
      loop: '{{ peers }}'
      loop_control:
        loop_var: peer
#Configuration for Loopback Interface
    - name: Loopback Addresses
      notify: Reload Networkd
      ansible.builtin.template:
          src: loopback.network
          dest: /etc/systemd/network/loopback.network
          owner: root
          group: root
          mode: '0644'
#Restart handlers
  handlers:
    - name: Reload Networkd
      ansible.builtin.shell:
        cmd: networkctl reload

And a sub-playbook to setup a singular tunnel:

#pop_conf_net_tunnel.yml
#Ansible script to manage one tunnel interface
- name: Tunnel to {{peer}}
  block:
    - name: Interface Template Netdev to {{peer}}
      notify: Reload Networkd
      ansible.builtin.template:
          src: tunnel.netdev
          dest: /etc/systemd/network/tun_{{hostvars[peer]['host']|replace("-", "_")}}.netdev
          owner: root
          group: root
          mode: '0644'
    - name: Interface Template Network to {{peer}}
      notify: Reload Networkd
      ansible.builtin.template:
          src: tunnel.network
          dest: /etc/systemd/network/tun_{{hostvars[peer]['host']|replace("-", "_")}}.network
          owner: root
          group: root
          mode: '0644'

They also rely on two template files, for netdev and network:

#tunnel.netdev
#Tunnel to {{peer}}
[NetDev]
Name=tun_{{hostvars[peer]['host']|replace("-", "_")}}
Kind=ip6tnl

[Tunnel]
Mode=ip6ip6
Local={{ my_ip }}
Remote={{ peer_ipv6_map[hostvars[peer]['host']] }}
TTL=64
Independent=True

#tunnel.network
#Tunnel to {{peer}}
[Match]
Name=tun_{{hostvars[peer]['host']|replace("-", "_")}}

[Network]
Address=fe80::{{router_id}}/64
ConfigureWithoutCarrier=yes

Ansible Routing Conf

Here’s what my templated BIRD conf looks like:

- hosts: "*"
  collections:
    - netbox.netbox
    - ansible.utils
  vars:
    netbox_url: "https://docs.peach.apalrd.fi"
    netbox_token: 0DjI3ZTMQdQPEbti07i6KYnan8jOi0f2K9XS0Qm2
    netbox_validate_certs: False
  tasks:
#Build prefix lists
    - name: Query NetBox for bgp-adv-dfz-ac prefixes
      set_fact:
        dfz_ac_prefixes: >-
          {{
            query(
              'netbox.netbox.nb_lookup',
              'prefixes',
              api_endpoint=netbox_url,
              token=netbox_token,
              validate_certs=netbox_validate_certs | default(true),
              api_filter="tag=bgp-adv-dfz-ac"
            )
            | map(attribute='value')
            | map(attribute='prefix')
            | list
          }}          
      when: pop is defined
    - name: Show prefixes for AC
      debug:
        msg: "Prefix for AC: {{ item }}"
      loop: "{{dfz_ac_prefixes}}"
      when: pop is defined
    - name: Query NetBox for pop specific prefixes
      set_fact:
        dfz_pop_prefixes: >-
          {{
            query(
              'netbox.netbox.nb_lookup',
              'prefixes',
              api_endpoint=netbox_url,
              token=netbox_token,
              validate_certs=netbox_validate_certs | default(true),
              api_filter="tag=bgp-adv-dfz-" ~ pop
            )
            | map(attribute='value')
            | map(attribute='prefix')
            | list
          }}          
      when: pop is defined
    - name: Show prefixes for POP
      debug:
        msg: "Prefix for POP: {{ item }}}"
      loop: "{{dfz_pop_prefixes}}"
      when: pop is defined
#Configuration for bird3
    - name: Bird3 copy common defs
      notify: Reload Bird
      ansible.builtin.template:
        src: bird-defs.conf
        dest: /etc/bird/defs.conf
        owner: root
        group: root
        mode: '0644'
    - name: Bird3 template IGP peers
      notify: Reload Bird
      ansible.builtin.template:
        src: bird-igp.conf
        dest: /etc/bird/igp.conf
        owner: root
        group: root
        mode: '0644'
    - name: Bird3 template iBGP peers
      notify: Reload Bird
      ansible.builtin.template:
        src: bird-ibgp.conf
        dest: /etc/bird/ibgp.conf
        owner: root
        group: root
        mode: '0644'
    - name: Bird3 template exports
      notify: Reload Bird
      ansible.builtin.template:
        src: bird-exports.conf
        dest: /etc/bird/exports.conf
        owner: root
        group: root
        mode: '0644'
    - name: Bird3 node-specific file
      notify: Reload Bird
      ansible.builtin.copy:
        src: bird-{{host}}.conf
        dest: /etc/bird/bird.conf
        owner: root
        group: root
        mode: '0644'
#Configuration for Tayga
    - name: Tayga configuration for local nat
      notify: Restart Tayga Local
      when: nat64 is defined and nat64
      ansible.builtin.template:
        src: tayga_local.conf
        dest: /etc/tayga/local.conf
        owner: root
        group: root
        mode: '0644'
    - name: Tayga configuration for wkpf nat
      notify: Restart Tayga Wkpf
      when: nat64 is defined and nat64
      ansible.builtin.template:
        src: tayga_wkpf.conf
        dest: /etc/tayga/wkpf.conf
        owner: root
        group: root
        mode: '0644'
#Restart handlers
  handlers:
    - name: Reload Bird
      ansible.builtin.shell:
        cmd: birdc configure
    - name: Restart Tayga Local
      ansible.builtin.systemd_service:
        name: tayga@local
        state: restarted
    - name: Restart Tayga Wkpf
      ansible.builtin.systemd_service:
        name: tayga@wkpf
        state: restarted

It depends on a bunch of other files, which are templated (with Jinja):

#bird-defs.conf
# My AS
define MY_AS = 201726;

# IP addresses of routers
define IP_MINE = 2a0f:b240:1000::{{router_id}};
define IP_WAW_PE1 = 2a0f:b240:1000::1;
define IP_LAX_PE1 = 2a0f:b240:1000::2;
define IP_SJW_P1 = 2a0f:b240:1000::3;
define IP_GRR_E1 = 2a0f:b240:1000::4;

# Where the route entered
define BGP_ENTER_MINE = (ro, MY_AS, {{router_id}});
define BGP_ENTER_WAW = (ro, MY_AS,1);
define BGP_ENTER_LAX = (ro, MY_AS,2);
define BGP_ENTER_SJW = (ro, MY_AS,3);
define BGP_ENTER_GRR = (ro, MY_AS,4);

# Redistribute this route to DFZ
define BGP_TO_DFZ_MINE = (ro, MY_AS, {{ 420+router_id }});
define BGP_TO_DFZ_ALL = (ro, MY_AS,420);
define BGP_TO_DFZ_WAW = (ro, MY_AS,421);
define BGP_TO_DFZ_LAX = (ro, MY_AS,422);
define BGP_TO_DFZ_GRR = (ro, MY_AS,424);

# IGP-related
define BGP_TO_IGP = (ro, MY_AS,667);
define BGP_TO_HELL = (ro, MY_AS,666);

# This route is via nat64
define BGP_NAT64 = (ro, MY_AS,64);

#bird-exports.conf
# This file is generated by a template!
# Routes for export
protocol static adv_v6 {
    ipv6;
    # My own loopback address goes into the IGP
    route 2a0f:b240:1000::{{router_id}}/128 via "lo" {
        bgp_ext_community.add(BGP_TO_IGP);
    };
{% if pop is defined %}
{% for prefix in dfz_ac_prefixes %}
    # Routes which will be advertised as anycast
    route {{prefix}} blackhole {
        bgp_ext_community.add(BGP_TO_DFZ_ALL);
        bgp_ext_community.add(BGP_ENTER_MINE);
    };
{% endfor %}
    # Routes which will be advertised from this pop
{% for prefix in dfz_pop_prefixes %}
    route {{prefix}} blackhole {
        bgp_ext_community.add(BGP_TO_DFZ_MINE);
        bgp_ext_community.add(BGP_ENTER_MINE);
    };
{% endfor %}
{% endif %}
{% if nat64 is defined and nat64 %}
    # Routes for nat64
    route 64:ff9b:1:{{router_id}}::/64 via "nat64" {
        bgp_ext_community.add(BGP_NAT64);
        bgp_ext_community.add(BGP_ENTER_MINE);
    };
    route 64:ff9b::/64 via "nat64wkpf" {
        bgp_ext_community.add(BGP_NAT64);
        bgp_ext_community.add(BGP_ENTER_MINE);
    };
{% endif %}
};

#bird-igp.conf
# This file is generated by a template!
# It configures the IGP (OSPFv3) functionality for a given node
protocol ospf v3 ospf6 {
    ipv6 {
        import all;
        export filter {
            # only send routes which we specifically want going into the IGP
            # and which are static routes from this router, not via iBGP
            if (proto = "adv_v6" && bgp_ext_community ~ [ BGP_TO_IGP ]) then accept;
            reject;
        };
    };

    # All OSPF is in area zero
    area 0 {
{% for peer in peers %}
        # Interface for neighbor {{peer}}
        interface "tun_{{hostvars[peer]['host']|replace("-", "_")}}" {
            type ptmp;
            check link no;
            cost 10;
            hello 5;
            dead 20;
            neighbors {
                fe80::{{hostvars[peer]['router_id']}};
            };
        };
{% endfor %}
    };
};

#bird-ibgp.conf
# This file is generated by a template!
# It configures iBGP peerings for a given node (full mesh)


# BGP locally within AS
filter to_bgp_local {
    #For routes which we advertised
    if proto = "adv_v6" then {
        bgp_origin = ORIGIN_INCOMPLETE;
        bgp_local_pref = 100;
        bgp_next_hop = IP_MINE;
        accept;
    }    
    #For routes which came from the DFZ, set our own nexthop
    if proto = "dfz_v6" || proto = "dfz_v4" then {
        bgp_med = 100;
        bgp_next_hop = IP_MINE;
        accept;
    }
    #For routes which came from the anywhere else, redistribute as-is
    accept;
};


{% for peer in peers %}
protocol bgp bgp_local_{{hostvars[peer]['host']|replace("-", "_")}} {
    local as MY_AS;
    neighbor 2a0f:b240:1000::{{hostvars[peer]['router_id']}} as MY_AS;
    multihop;
    source address 2a0f:b240:1000::{{router_id}};
    ipv6 {
        import filter from_bgp_local;
        export filter to_bgp_local;
    };
};
{% endfor %}

I also use Tayga, and here is one of the Tayga configs:

#tun device
tun-device nat64
#prefix
prefix 64:ff9b:1:{{router_id}}::/96
ipv4-addr 192.0.0.8
#dynamic pool is apipa space
dynamic-pool 169.254.0.0/20
udp-cksum-mode fwd
log drop reject icmp self dyn
tun-up yes
tun-route 169.254.0.0/20
tun-route 64:ff9b:1:{{router_id}}::/64

SMART SFP - Mini Linux System on a Stick (Literally)

Tue, 18 Nov 2025 01:00:05 -0300

Today I’m taking a look at the Plumspace Smart SFP - a dual core ARM Cortex A53 stuffed inside a 1310nm SFP optical transceiver, which can deal with packet flows at line rate*. Quite a neat little thing, with plenty of use cases.

Plumspace Smart SFPs: Link

Video

Bridging

Kernel defaults for br0, configured using /etc/network/interfaces:

auto gbe0
allow-hotplug gbe0
iface gbe0 inet manual

auto gbe1
allow-hotplug gbe1
iface gbe1 inet manual

auto br0
allow-hotplug br0
iface br0 inet6 auto
    bridge_ports gbe0 gbe1

I also installed avahi-daemon which I’m using to discover it (smart-sfp.local). You can use a static IP if you’d rather.

Iperf3

Connecting to host 172.27.15.180, port 5201
[  5] local 172.27.15.181 port 44144 connected to 172.27.15.180 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  85.8 MBytes   719 Mbits/sec  443   2.56 MBytes
[  5]   1.00-2.00   sec  85.5 MBytes   717 Mbits/sec  252    693 KBytes
[  5]   2.00-3.00   sec   111 MBytes   934 Mbits/sec   27    785 KBytes
[  5]   3.00-4.00   sec   112 MBytes   935 Mbits/sec  152    870 KBytes
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec  100    684 KBytes
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec    0    802 KBytes
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec   27    899 KBytes
[  5]   7.00-8.00   sec   112 MBytes   938 Mbits/sec   28    713 KBytes
[  5]   8.00-9.00   sec   112 MBytes   936 Mbits/sec    6    820 KBytes
[  5]   9.00-10.00  sec   113 MBytes   945 Mbits/sec    0    922 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.04 GBytes   895 Mbits/sec  1035            sender
[  5]   0.00-10.01  sec  1.04 GBytes   892 Mbits/sec

Connections Per Second (Cyperf)

----------------------------------------------------------
 Test Result Summary                                      
----------------------------------------------------------
 Test duration         41 seconds                                             
----------------------------------------------------------
 Connections                                              
----------------------------------------------------------
 Total succeeded       283790                             
 Total failed          0                                  
 Average rate          6921.000000 connections per second
 Average latency       564 µs

Bridging Ntopng

Disabled segment offloading, running ntopng installed via apt packages

Throughput Test (Cyperf)

-------------------------------------------------------
 Test Result Summary                                   
-------------------------------------------------------
 Test duration         27 seconds                      
-------------------------------------------------------
 Throughput                                            
-------------------------------------------------------
 Average TX & RX       546.294224 Mb/s                 
 Average TX            543.728808 Mb/s                 
 Average RX            2.565416 Mb/s                   
                                                       
-------------------------------------------------------
 TCP Data                                              
-------------------------------------------------------
 Total transferred     1.713713 GB                     
 Total sent            1.713713 GB                     
 Total received        0.000000 GB

Connections Per Second (Cyperf)

 ----------------------------------------------------------
 Test Result Summary                                      
----------------------------------------------------------
 Test duration         32 seconds                                                                           
----------------------------------------------------------
 Connections                                              
----------------------------------------------------------
 Total succeeded       193386                             
 Total failed          0                                  
 Average rate          6043.000000 connections per second 
 Average latency       650 µs

Routing

Configured using /etc/network/interfaces (and using IPv4 for cyperf):

auto gbe0
allow-hotplug gbe0
iface gbe0 inet static
    address 192.0.0.1/29

auto gbe1
allow-hotplug gbe1
iface gbe1 inet dhcp
iface gbe1 inet6 auto

This will add a default route via dhcp. Added return route on test server: ip route add 192.0.0.0/29 via 172.27.15.182

IPerf

Connecting to host 172.27.15.180, port 5201
[  5] local 192.0.0.5 port 59314 connected to 172.27.15.180 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   106 MBytes   886 Mbits/sec   15    393 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   944 Mbits/sec   11    561 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec    7    686 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   940 Mbits/sec   50    796 KBytes       
[  5]   4.00-5.00   sec   113 MBytes   946 Mbits/sec    6    884 KBytes       
[  5]   5.00-6.00   sec   113 MBytes   945 Mbits/sec    9    973 KBytes       
[  5]   6.00-7.00   sec  87.9 MBytes   737 Mbits/sec  240    734 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   935 Mbits/sec    0    843 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   942 Mbits/sec    4    707 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   935 Mbits/sec    0    819 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.07 GBytes   915 Mbits/sec  342            sender
[  5]   0.00-10.02  sec  1.07 GBytes   913 Mbits/sec                  receiver

Throughput (Cyperf)

-------------------------------------------------------
 Test Result Summary                                   
-------------------------------------------------------
 Test duration         30 seconds                      
-------------------------------------------------------
 Throughput                                            
-------------------------------------------------------
 Average TX & RX       905.054472 Mb/s                 
 Average TX            891.920008 Mb/s                 
 Average RX            13.134464 Mb/s                  
                                                       
-------------------------------------------------------
 TCP Data                                              
-------------------------------------------------------
 Total transferred     2.900737 GB                     
 Total sent            2.900737 GB                     
 Total received        0.000000 GB

CPS (Cyperf)

----------------------------------------------------------
 Test Result Summary                                      
----------------------------------------------------------
 Test duration         40 seconds                                                         
----------------------------------------------------------
 Connections                                              
----------------------------------------------------------
 Total succeeded       271844                             
 Total failed          0                                  
 Average rate          6796.000000 connections per second 
 Average latency       579 µs

Routing NAT44

Configured using /etc/network/interfaces (again using IPv4 for cyperf):

auto gbe0
allow-hotplug gbe0
iface gbe0 inet static
    address 192.0.0.1/29

auto gbe1
allow-hotplug gbe1
iface gbe1 inet dhcp
iface gbe1 inet6 auto

and using iptables (not the nf_tables version due to kernel limitations): iptables-legacy -t nat -A POSTROUTING -o gbe1 -j MASQUERADE

IPerf

Connecting to host 172.27.15.180, port 5201
[  5] local 192.0.0.5 port 58318 connected to 172.27.15.180 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   103 MBytes   866 Mbits/sec   59    400 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   943 Mbits/sec    6    554 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   942 Mbits/sec   94    680 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   936 Mbits/sec   27    796 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec    3    894 KBytes       
[  5]   5.00-6.00   sec   112 MBytes   938 Mbits/sec   41    741 KBytes       
[  5]   6.00-7.00   sec   112 MBytes   938 Mbits/sec    0    851 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   943 Mbits/sec    2    943 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec   49    737 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   935 Mbits/sec    1    843 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   932 Mbits/sec  282            sender
[  5]   0.00-10.01  sec  1.08 GBytes   930 Mbits/sec                  receiver

Throughput (Cyperf)

-------------------------------------------------------
 Test Result Summary                                   
-------------------------------------------------------
 Test duration         51 seconds                      
-------------------------------------------------------
 Throughput                                            
-------------------------------------------------------
 Average TX & RX       860.770592 Mb/s                 
 Average TX            851.627024 Mb/s                 
 Average RX            9.143568 Mb/s                   
                                                       
-------------------------------------------------------
 TCP Data                                              
-------------------------------------------------------
 Total transferred     4.853985 GB                     
 Total sent            4.853985 GB                     
 Total received        0.000000 GB

CPS (Cyperf)

---------------------------------------------------------
 Test Result Summary                                     
---------------------------------------------------------
 Test duration         33 seconds                                            
---------------------------------------------------------
 Connections                                             
---------------------------------------------------------
 Total succeeded       8187                              
 Total failed          48                                
 Average rate          248.000000 connections per second
 Average latency       671 µs

Routing NAT46

Configured using /etc/network/interfaces:

auto gbe0
allow-hotplug gbe0
iface gbe0 inet static
    address 192.0.0.1/29

auto gbe1
allow-hotplug gbe1
iface gbe1 inet6 auto

IPv4-island inside, IPv6-only outside ‘inside’ is 192.0.0.0/29 (DSLite range) mapped to 2001:db8::6460/125, leading to this Tayga config:

tun-device nat64
ipv4-addr 192.0.0.0
ipv6-addr 2601:40e:8100:ac30::6468
prefix 64:ff9b::/96
wkpf-strict no
log drop reject icmp self dyn
#Map IPv4-island to a seqential range of v6s on LAN
map 192.0.0.0/29 2601:40e:8100:ac30::6460/125
#test system on IPv6, mapped to a random IPv4 so we can iperf to it
map 10.0.0.1/32 2001:db8::be24:11ff:feeb:716e/128

And the commands I ran to set up routing and such (didn’t bother trying to make this persistent):

#Enable IP forwarding of IPv6 and IPv4
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
echo 1 > /proc/sys/net/ipv4/conf/all/forwarding
#Enable Accept RA (=2 due to forwarding) on gbe1 for SLAAC address assignment
echo 2 > /proc/sys/net/ipv6/conf/gbe1/accept_ra
#Proxy NDP the CLAT addresses
echo 1 > /proc/sys/net/ipv6/conf/gbe1/proxy_ndp
#Proxy ND addresses from /29
ip neigh add proxy 2601:40e:8100:ac30::6460 dev gbe1
ip neigh add proxy 2601:40e:8100:ac30::6461 dev gbe1
ip neigh add proxy 2601:40e:8100:ac30::6462 dev gbe1
ip neigh add proxy 2601:40e:8100:ac30::6463 dev gbe1
ip neigh add proxy 2601:40e:8100:ac30::6464 dev gbe1
ip neigh add proxy 2601:40e:8100:ac30::6465 dev gbe1
ip neigh add proxy 2601:40e:8100:ac30::6466 dev gbe1
ip neigh add proxy 2601:40e:8100:ac30::6467 dev gbe1
#plus one for Tayga itself
ip neigh add proxy 2601:40e:8100:ac30::6468 dev gbe1
#Start tayga (using default.conf and included systemd unit)
systemctl enable --now tayga@default
#Bring up iface and route to it
#no need for IPs on this iface
ip link set dev nat64 up
ip route add default dev nat64 mtu 1480
ip route add 2601:40e:8100:ac30::6464/124 dev nat64

IPerf

Connecting to host 10.0.0.1, port 5201
[  5] local 192.0.0.5 port 43146 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  39.9 MBytes   334 Mbits/sec    5    771 KBytes       
[  5]   1.00-2.00   sec  43.1 MBytes   362 Mbits/sec    0    866 KBytes       
[  5]   2.00-3.00   sec  43.4 MBytes   364 Mbits/sec    0    937 KBytes       
[  5]   3.00-4.00   sec  43.1 MBytes   362 Mbits/sec    0    987 KBytes       
[  5]   4.00-5.00   sec  42.9 MBytes   360 Mbits/sec    0   1022 KBytes       
[  5]   5.00-6.00   sec  41.9 MBytes   351 Mbits/sec    8    766 KBytes       
[  5]   6.00-7.00   sec  43.5 MBytes   365 Mbits/sec    0    813 KBytes       
[  5]   7.00-8.00   sec  43.2 MBytes   363 Mbits/sec    0    844 KBytes       
[  5]   8.00-9.00   sec  42.0 MBytes   352 Mbits/sec    0    867 KBytes       
[  5]   9.00-10.00  sec  42.1 MBytes   353 Mbits/sec    0    905 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   425 MBytes   357 Mbits/sec   13            sender
[  5]   0.00-10.01  sec   424 MBytes   355 Mbits/sec                  receiver

Wireguard VPN

I did this using network namespaces instead of multiple table. You can choose to either have the WAN or LAN on the ‘default’ namespace, where normal processes on Linux will bind to. I chose WAN in this case, since I am relying on ssh + avahi mdns on my normal network.

Wireguard will always send/receive packets from the network namespace it is started in, even if you move the adapter to a new netns! So start the adapter from the WAN-side (or in my case, use wireguard-go launched from the default netns), THEN move the adapter into the netns and bring it up.

So, steps:

# Create netns named 'user', add inteface gbe0 (downstream 'LAN') to netns
ip netns add user
ip link set dev gbe0 netns user
# Placed Mullvad's conf generator in /etc/wg/wg0.conf
# Removed Address and DNS lines, which are for wg-quick
# will need those values separately
#
# Launch wireguard-go
wireguard-go wg0
# Configure it using wg
wg setconf wg0 /etc/wg/wg0.conf
# Move wg tun adapter to netns
ip link set dev wg0 netns user

Config within the netns (ip netns exec user bash)

# enable forwarding within the netns (it has its own settings!)
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
echo 1 > /proc/sys/net/ipv4/conf/all/forwarding
# bring up wg
ip link set up wg0
# These came from Mullvad's conf generator
ip addr add 10.67.156.35/32 dev wg0
ip addr add fc00:bbbb:bbbb:bb01::4:9c22/128 dev wg
# add default routes
ip route add default dev wg0
ip -6 route add default dev wg0
# add masquerade rules
iptables-legacy -t nat -A POSTROUTING -o wg0 -j MASQUERADE
ip6tables-legacy -t nat -A POSTROUTING -o wg0 -j MASQUERADE
# add IPs to gbe0 in our netns (it loses config when moved)
ip addr add 192.0.0.1/29 dev gbe0
ip addr add 2001:db8:6464::1/64 dev gbe0

Config for dnsmasq for this setup

#static dns resolver conf
no-resolv
#server which came from Mullvad
server=10.64.0.1
#iface to listen on
interface=gbe0
#using the dslite range since I already had configured this for 464xlat
dhcp-range=192.0.0.2,192.0.0.6,1h
#using doc prefix for nat66
dhcp-range=2001:db8:6969::, ra-stateless

I launched dnsmasq within the netns so it will serve DHCP on interface gbe0.

OpenSSL bench

OpenSSL bench on this hw:

#Speed test AES-128-GCM
root@smart-sfp:~# openssl speed -evp aes-128-gcm
Doing AES-128-GCM for 3s on 16 size blocks: 753963 AES-128-GCM's in 2.99s
Doing AES-128-GCM for 3s on 64 size blocks: 733150 AES-128-GCM's in 3.00s
Doing AES-128-GCM for 3s on 256 size blocks: 660259 AES-128-GCM's in 2.99s
Doing AES-128-GCM for 3s on 1024 size blocks: 469975 AES-128-GCM's in 2.99s
Doing AES-128-GCM for 3s on 8192 size blocks: 141423 AES-128-GCM's in 2.99s
Doing AES-128-GCM for 3s on 16384 size blocks: 78532 AES-128-GCM's in 2.99s
version: 3.0.17
built on: Tue Aug  5 07:09:41 2025 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-3.0.17=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_armcap=0xbd
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
AES-128-GCM       4034.58k    15640.53k    56530.54k   160954.65k   387470.64k   430323.84k
#Speed test AES-256-GCM
root@smart-sfp:~# openssl speed -evp aes-256-gcm      
Doing AES-256-GCM for 3s on 16 size blocks: 735862 AES-256-GCM's in 2.99s
Doing AES-256-GCM for 3s on 64 size blocks: 712262 AES-256-GCM's in 2.99s
Doing AES-256-GCM for 3s on 256 size blocks: 632462 AES-256-GCM's in 2.99s
Doing AES-256-GCM for 3s on 1024 size blocks: 417102 AES-256-GCM's in 3.00s
Doing AES-256-GCM for 3s on 8192 size blocks: 125089 AES-256-GCM's in 2.99s
Doing AES-256-GCM for 3s on 16384 size blocks: 69521 AES-256-GCM's in 2.99s
version: 3.0.17
built on: Tue Aug  5 07:09:41 2025 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-3.0.17=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_armcap=0xbd
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
AES-256-GCM       3937.72k    15245.74k    54150.59k   142370.82k   342718.76k   380947.18k
#Speed test ChaCha20-Poly1305
root@smart-sfp:~# openssl speed -evp chacha20-poly1305
Doing ChaCha20-Poly1305 for 3s on 16 size blocks: 5122281 ChaCha20-Poly1305's in 2.99s
Doing ChaCha20-Poly1305 for 3s on 64 size blocks: 2859045 ChaCha20-Poly1305's in 2.99s
Doing ChaCha20-Poly1305 for 3s on 256 size blocks: 1448219 ChaCha20-Poly1305's in 2.99s
Doing ChaCha20-Poly1305 for 3s on 1024 size blocks: 457043 ChaCha20-Poly1305's in 3.00s
Doing ChaCha20-Poly1305 for 3s on 8192 size blocks: 60887 ChaCha20-Poly1305's in 2.99s
Doing ChaCha20-Poly1305 for 3s on 16384 size blocks: 30481 ChaCha20-Poly1305's in 2.99s
version: 3.0.17
built on: Tue Aug  5 07:09:41 2025 UTC
options: bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-3.0.17=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_armcap=0xbd
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
ChaCha20-Poly1305    27410.20k    61196.95k   123994.67k   156004.01k   166818.16k   167023.65k

Final results for your comparison:

The 'numbers' are in 1000s of bytes per second processed.
type               16 bytes     64 bytes    256 bytes    1024 bytes   8192 bytes   16384 bytes
AES-128-GCM         4034.58k    15640.53k    56530.54k   160954.65k   387470.64k   430323.84k
AES-256-GCM         3937.72k    15245.74k    54150.59k   142370.82k   342718.76k   380947.18k
ChaCha20-Poly1305  27410.20k    61196.95k   123994.67k   156004.01k   166818.16k   167023.65k

Setting up a Home ISP with GPON (and CData)

Fri, 03 Oct 2025 01:00:05 -0300

Today I’m setting up a GPON network in my homelab. GPON is the ITU’s Gigabit-class Passive Optical Network technology, one of the most commonly deployed fiber to the home technologies. While it’s being replaced by its faster descendant XGS-PON, it’s still capable of 2.44Gbps down / 1.22Gbps up across 128 clients, all using only passive optical components between the headend (OLT) and clients (ONU).

I’ve got a single-port Optical Line Terminal (OLT) made by CData, as well as 8x client Optical Network Units (ONU) also made by CData. Big thanks to CData for providing the equipment for this project!

Video

CData OLT Config

You can also download the config as a file here

!
!
sysname FD1601S-B1
!
!
traffic-profile profile-id 1 profile-name resi-1000-down cir 512000 pir 1024000 cbs 10240000 pbs 10240000
traffic-profile profile-id 2 profile-name resi-1000-up cir 256000 pir 1024000 cbs 8194000 pbs 10240000
traffic-profile profile-id 3 profile-name resi-200-down cir 102400 pir 204800 cbs 3278800 pbs 6555600
traffic-profile profile-id 4 profile-name resi-200-up cir 51200 pir 102400 cbs 1640400 pbs 3278800
!
!
dba-profile profile-id 0 profile-name dba-profile_0
  type1 fix 256 
  commit
 exit
dba-profile profile-id 1 profile-name dba-profile_1
  type3 assure 56000 max 1024000 
  commit
 exit
dba-profile profile-id 2 profile-name dba-service-100
  type3 assure 25600 max 102400 
  commit
 exit
dba-profile profile-id 3 profile-name dba-service-500
  type3 assure 25600 max 512000 
  commit
 exit
dba-profile profile-id 4 profile-name dba-service-1000
  type3 assure 30720 max 1024000 
  commit
 exit
!
!
!
ont-line-profile gpon profile-id 0 profile-name line-profile_0
 #tcont 0 dba-profile-id 0
 #tcont 1 dba-profile-id 1
  #gem add 1 tcont 1
   #gem mapping 1 1 vlan 1
  commit
 exit
ont-line-profile gpon profile-id 1 profile-name line-resi1000
 #tcont 0 dba-profile-id 0
 #tcont 1 dba-profile-id 1
  #gem add 1 tcont 1
   #gem mapping 1 1 vlan transparent
    no gem mapping 1 1
    gem delete 1
    gem add 1 tcont 1 encrypt disable gem-car-upstream 2 gem-car-downstream 1
    gem mapping 1 1 vlan 20
  commit
 exit
ont-line-profile gpon profile-id 2 profile-name line-resi200
 #tcont 0 dba-profile-id 0
 #tcont 1 dba-profile-id 1
  #gem add 1 tcont 1
   #gem mapping 1 1 vlan transparent
    no gem mapping 1 1
    gem delete 1
    gem add 1 tcont 1 encrypt disable gem-car-upstream 4 gem-car-downstream 3
    gem mapping 1 1 vlan 20
  commit
 exit
!
ipconfig-profile gpon profile-id 0 profile-name ipconfig-profile_0
 #service internet 
 #nat enable 
 commit
exit
!
ont-srv-profile gpon profile-id 0 profile-name srv-profile_0
  #ont-port eth adaptive pots adaptive catv adaptive wifi adaptive
  port vlan eth 1 transparent 
  port vlan eth 2 transparent 
  port vlan eth 3 transparent 
  port vlan eth 4 transparent 
  port vlan eth 5 transparent 
  port vlan eth 6 transparent 
  port vlan eth 7 transparent 
  port vlan eth 8 transparent 
  port vlan eth 9 transparent 
  port vlan eth 10 transparent 
  port vlan eth 11 transparent 
  port vlan eth 12 transparent 
  port vlan eth 13 transparent 
  port vlan eth 14 transparent 
  port vlan eth 15 transparent 
  port vlan eth 16 transparent 
  port vlan eth 17 transparent 
  port vlan eth 18 transparent 
  port vlan eth 19 transparent 
  port vlan eth 20 transparent 
  port vlan eth 21 transparent 
  port vlan eth 22 transparent 
  port vlan eth 23 transparent 
  port vlan eth 24 transparent 
  commit
 exit
ont-srv-profile gpon profile-id 1 profile-name srv-resi
  ont-port eth adaptive pots adaptive catv adaptive 
  port native-vlan eth 1 20 
  port vlan eth 1 20 
  port vlan eth 2 transparent 
  port vlan eth 3 transparent 
  port vlan eth 4 transparent 
  port vlan eth 5 transparent 
  port vlan eth 6 transparent 
  port vlan eth 7 transparent 
  port vlan eth 8 transparent 
  port vlan eth 9 transparent 
  port vlan eth 10 transparent 
  port vlan eth 11 transparent 
  port vlan eth 12 transparent 
  port vlan eth 13 transparent 
  port vlan eth 14 transparent 
  port vlan eth 15 transparent 
  port vlan eth 16 transparent 
  port vlan eth 17 transparent 
  port vlan eth 18 transparent 
  port vlan eth 19 transparent 
  port vlan eth 20 transparent 
  port vlan eth 21 transparent 
  port vlan eth 22 transparent 
  port vlan eth 23 transparent 
  port vlan eth 24 transparent 
  commit
 exit
!
!
!
ont-sipagent-profile gpon profile-id 0 profile-name sipagent-profile_0 
  commit
  exit
ont-digitmap-profile gpon profile-id 0 profile-name digitmap-profile_0 
  commit
 exit
ont-siprightflag-profile gpon profile-id 0 profile-name sipright-profile_0 
  commit
 exit
ont-pots-profile gpon profile-id 0 profile-name pots-profile_0 
  commit
 exit
ont mult-srv-profile gpon profile-id 0 profile-name default-mult-srv-profile
 ont-line-profile profile-id 0
 ont-srv-profile profile-id 0 
 commit
exit
ont mult-srv-profile gpon profile-id 1 profile-name resi1000-srv-profile
 ont-line-profile profile-id 1
 ont-srv-profile profile-id 1 
 commit
exit
ont mult-srv-profile gpon profile-id 2 profile-name resi-200-srv-profile
 ont-line-profile profile-id 2
 ont-srv-profile profile-id 1 
 commit
exit
!
!
!
!
vlan 1-2,20-30,69,666,1000
!
!
!
!
!
!
!
!
!
radius 
exit 
!
!
tacacs+ 
exit 
!
user name root password *#*!#/)zW*+C-J/J123 privilege-level 15
user name admin password *#*!#/)zW*+C-J/J123 privilege-level 2
user name operator password *#*!#/)zW*+C-J/J123 privilege-level 1
user name guest password *#*$N&C(Ho+U0.lp\23 privilege-level 0
!
aaa 
 domain default
  aaa_protocol radius 
  exit
!
!
interface mgmt 
 ip address 192.168.1.100 24 
exit 
!
!
!
!
!
!
!
!
interface gpon 0/0
 flow-control 1 disable
 frame-max 1 9600
 ont authmode 1 auto 
  ont policy-auth 1 match any sn-auth to mult-srv-profile profile-name default-mult-srv-profile priority 0 
  ont policy-auth 1 match any sn-auth to mult-srv-profile profile-id 2 priority 1 
 ont add 1 1 sn-auth "CDTCAF53E6E3" 
  ont ont-port 1 1 eth adaptive pots adaptive catv adaptive iphost adaptive wifi adaptive 
  ont mac-learning 1 1 enable 
  ont mac-aging 1 1 300 
  ont veip 1 1 trunk 
  ont loopdetect 1 1 disable detect-frequency 8 resume-interval 300 auto-shutdown enable 
  ont native-vlan 1 1 concern 
  ont multicast mode 1 1 unconcern 
  ont port native-vlan 1 1 eth 1 vlan 21 
  ont port vlan 1 1 eth 1 21 
  ont tcont 1 1 0 dba-profile-id 0
  ont tcont 1 1 1 dba-profile-id 1
  ont mapping-mode 1 1 vlan 
  ont gemport 1 1 1 tcont 1 gem-car-upstream 4 gem-car-downstream 3 encrypt disable priority-queue-upstream 1 priority-queue-downstream 1  
  ont gemport mapping 1 1 1 1 vlan 21  
  ont fec 1 1 disable 
  ont omcc encrypt 1 1 disable 
  ont qos-mode 1 1 priority-queue 
 ont add 1 3 sn-auth "CDTCAF53E6DF" 
  ont ont-port 1 3 eth adaptive pots adaptive catv adaptive iphost adaptive wifi adaptive 
  ont mac-learning 1 3 enable 
  ont mac-aging 1 3 300 
  ont veip 1 3 trunk 
  ont loopdetect 1 3 disable detect-frequency 8 resume-interval 300 auto-shutdown enable 
  ont native-vlan 1 3 concern 
  ont multicast mode 1 3 unconcern 
  ont port native-vlan 1 3 eth 1 vlan 22 
  ont port vlan 1 3 eth 1 22 
  ont tcont 1 3 0 dba-profile-id 0
  ont tcont 1 3 1 dba-profile-id 1
  ont mapping-mode 1 3 vlan 
  ont gemport 1 3 1 tcont 1 gem-car-upstream 2 gem-car-downstream 1 encrypt disable priority-queue-upstream 1 priority-queue-downstream 1  
  ont gemport mapping 1 3 1 1 vlan 22  
  ont fec 1 3 disable 
  ont omcc encrypt 1 3 disable 
  ont qos-mode 1 3 priority-queue 
 ont add 1 4 sn-auth "CDTCAF53E6E1" 
  ont ont-port 1 4 eth adaptive pots adaptive catv adaptive iphost adaptive wifi adaptive 
  ont mac-learning 1 4 enable 
  ont mac-aging 1 4 300 
  ont veip 1 4 trunk 
  ont loopdetect 1 4 disable detect-frequency 8 resume-interval 300 auto-shutdown enable 
  ont native-vlan 1 4 concern 
  ont multicast mode 1 4 unconcern 
  ont port native-vlan 1 4 eth 1 vlan 23 
  ont port vlan 1 4 eth 1 23 
  ont tcont 1 4 0 dba-profile-id 0
  ont tcont 1 4 1 dba-profile-id 1
  ont mapping-mode 1 4 vlan 
  ont gemport 1 4 1 tcont 1 gem-car-upstream 4 gem-car-downstream 3 encrypt disable priority-queue-upstream 1 priority-queue-downstream 1  
  ont gemport mapping 1 4 1 1 vlan 23  
  ont fec 1 4 disable 
  ont omcc encrypt 1 4 disable 
  ont qos-mode 1 4 priority-queue 
 ont add 1 5 sn-auth "CDTCAF53E6DD" 
  ont ont-port 1 5 eth adaptive pots adaptive catv adaptive iphost adaptive wifi adaptive 
  ont mac-learning 1 5 enable 
  ont mac-aging 1 5 300 
  ont veip 1 5 trunk 
  ont loopdetect 1 5 disable detect-frequency 8 resume-interval 300 auto-shutdown enable 
  ont native-vlan 1 5 concern 
  ont multicast mode 1 5 unconcern 
  ont port native-vlan 1 5 eth 1 vlan 24 
  ont port vlan 1 5 eth 1 24 
  ont tcont 1 5 0 dba-profile-id 0
  ont tcont 1 5 1 dba-profile-id 1
  ont mapping-mode 1 5 vlan 
  ont gemport 1 5 1 tcont 1 gem-car-upstream 2 gem-car-downstream 1 encrypt disable priority-queue-upstream 1 priority-queue-downstream 1  
  ont gemport mapping 1 5 1 1 vlan 24  
  ont fec 1 5 disable 
  ont omcc encrypt 1 5 disable 
  ont qos-mode 1 5 priority-queue 
exit 
!
interface ge 0/0
exit 
!
interface xge 0/0
 frame-max 1 9600
 vlan mode 1 trunk
 vlan trunk 1 20-30,69
exit 
!
!
!
!
!
# DHCP snooping config 
! 
# DHCP security-table config 
! 
# DHCP ipsg config 
!
interface vlanif 1 
 description vlan-lan 
 ip address 192.168.0.4 24
exit 
!
!
!
!
!
!
!
!
!
timezone gmt+ 02:00 
!

!
!
snmp-agent community read */*puhcfl 
snmp-agent community write */*pxfydqe 
snmp-agent community write */*LoxxelqBox|eHdqqexv\qdpce 
!
!
!
service telnet enable
service ssh enable
service http enable
service https enable
service snmp disable
!
!
end
!

Mikrotik BNG Config

You can also download the config as a file here

# 2025-09-27 15:34:10 by RouterOS 7.20rc1
# software id = XB52-1V8V
#
# model = CRS305-1G-4S+
# serial number = HHB0A3N8HRP
/interface bridge
add admin-mac=F4:1E:57:74:FD:8E auto-mac=no comment=defconf name=bridge \
    vlan-filtering=yes
/interface vlan
add interface=bridge name=vlan-cust-com vlan-id=20
add interface=bridge name=vlan-cust1 vlan-id=21
add interface=bridge name=vlan-cust2 vlan-id=22
add interface=bridge name=vlan-cust3 vlan-id=23
add interface=bridge name=vlan-cust4 vlan-id=24
add interface=bridge name=vlan-iptv vlan-id=69
add interface=bridge name=vlan-lan vlan-id=1
/ip pool
add name=pool-cust-com ranges=100.64.0.100-100.64.0.254
add name=pool-cust1 ranges=100.64.1.100-100.64.1.254
add name=pool-cust2 ranges=100.64.2.100-100.64.2.254
add name=pool-cust3 ranges=100.64.3.100-100.64.3.254
add name=pool-cust4 ranges=100.64.4.100-100.64.4.254
/ip dhcp-server
add address-pool=pool-cust-com interface=vlan-cust-com name=dhcp4-cust-com
/interface bridge port
add bridge=bridge comment=defconf frame-types=\
    admit-only-untagged-and-priority-tagged interface=ether1
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
/ip neighbor discovery-settings
set discover-interface-list=all
/interface bridge vlan
add bridge=bridge tagged=sfp-sfpplus3,sfp-sfpplus2 vlan-ids=20-30,69
/ip address
add address=100.64.0.1/24 interface=vlan-cust-com network=100.64.0.0
add address=100.64.1.1/24 interface=vlan-cust1 network=100.64.1.0
add address=100.64.2.1/24 interface=vlan-cust2 network=100.64.2.0
add address=100.64.3.1/24 interface=vlan-cust3 network=100.64.3.0
add address=100.64.4.1/24 interface=vlan-cust4 network=100.64.4.0
add address=100.64.69.2/24 interface=vlan-iptv network=100.64.69.0
/ip dhcp-client
add interface=vlan-lan
/ip dhcp-server network
add address=100.64.0.0/24 dns-server=192.168.0.3 domain=apalrd.fi gateway=\
    100.64.0.1 netmask=24 ntp-none=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=vlan-lan src-address=\
    100.64.0.0/10
/ipv6 route
add disabled=no distance=1 dst-address=64:ff9b::/96 gateway=\
    fe80::be24:11ff:fed7:e52d%vlan-lan pref-src="" routing-table=main scope=\
    30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=::/0 gateway=\
    fe80::10:18ff:fe48:74c%vlan-lan pref-src="" routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
/ipv6 address
add address=2001:db8:420:6944::9 advertise=no interface=vlan-lan
# address pool error: pool not found: pd-pool1 (4)
add address=::1 from-pool=pd-pool1 interface=vlan-cust-com
# address pool error: pool not found: pd-pool1 (4)
add address=::1 from-pool=pd-pool1 interface=vlan-cust1
# address pool error: pool not found: pd-pool1 (4)
add address=::1 from-pool=pd-pool1 interface=vlan-cust2
# address pool error: pool not found: pd-pool1 (4)
add address=::1 from-pool=pd-pool1 interface=vlan-cust3
# address pool error: pool not found: pd-pool1 (4)
add address=::1 from-pool=pd-pool1 interface=vlan-cust4
/ipv6 dhcp-client
add interface=vlan-lan pool-name=pd-pool1 prefix-hint=::/59 request=\
    address,prefix
/ipv6 dhcp-server
add address-pool=pd-pool1 interface=vlan-cust-com name=dhcp-cust-com
/ipv6 firewall nat
add action=src-nat chain=srcnat out-interface=vlan-lan src-address=\
    2001:db8:420:6920::/59 to-address=\
    2001:db8:420:6900:e203:18ab:6c15:dbe4/128
/ipv6 nd
set [ find default=yes ] dns=2001:db8:420:6900:be24:11ff:fe19:afa6 \
    managed-address-configuration=yes pref64=64:ff9b::/96 ra-interval=5s-10s
/system clock
set time-zone-name=Europe/Helsinki
/system package update
set channel=testing

Setting up a Postmarks server for the Fediverse

Fri, 05 Sep 2025 01:00:05 -0300

Today I’m bringing my shared bookmarks to the Fediverse: @bookmarks@mark.apalrd.net

It’s a list of things that I like, basically. I hope to post links to lesser-known content creators or content that I really enjoy. The hope is that if you enjoy my content, that you might enjoy the things that I enjoy as well.

Hopefully this brings you things that The Algorithm normally might not.

So, anyway, how do you run the server?

Content

Video

Server

I ran this in a LXC container based on the latest Debian (Trixie). I’m hosting it in my environment, you can host it on a VPS or your own environment, whatever you want.

Postmarks’s official docker image is using the extremely outdated Node.js 16, so I’ve used the oldest still supported version (18) to hopefully not break all this node shit, which has super picky version dependencies. It seems to be fine currently.

After doing the usual system upgrades (apt update && apt upgrade -y), I install the tools we will need:

# Install nodesource's repo key to add node.js repositories
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /usr/share/keyrings/nodesource.gpg
chmod 644 /usr/share/keyrings/nodesource.gpg
# Add repo for Nodesourde node.js 18
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/nodesource.gpg] https://deb.nodesource.com/node_18.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list
# Apt update, since we added a repo
apt update
# Install packages we need
apt install nodejs npm git sudo -y

Postmarks

You can find the official instructions Here. I’m not deviating from this, I’m adding to it

Now that the base software is installed, I’m going to git clone Postmarks into my web directory, change ownership to Mark, and then login as mark to do the final configuration and testing.

# Create a new user for the service to run as
useradd -m -U mark
# Clone Postmarks into service location
cd /var
git clone https://github.com/ckolderup/postmarks
# Change ownership to mark
chown -R mark:mark postmarks
cd postmarks

Now we can login as mark: sudo -u mark bash

As mark, we need to create our .env file. Here’s my file, choose your own passwords (and read the Postmarks readme for information on what these mean):

PUBLIC_BASE_URL=mark.apalrd.net
ADMIN_KEY=CorrectHorseBatteryStaple
SESSION_SECRET=CorrectHorseBatteryStaple
PORT=3000
HOST=::1

I’m using Caddy as a reverse proxy to handle TLS termination, HTTP/3, and cert management, among other things, so we will run Postmarks on its own bound to localhost only. As of this writing, this hasn’t merged into main yet, but it’s on my fork if you want to use my fork for now.

After the env file, you also need to copy account.json.example into account.json and edit it:

cp account.json.example account.json
nano account.json

The instructions for what to do are on the Postmarks github, but this is more customization and entirely optional.

And finally we need to run npm install: npm install

And you can use exit to get out of the user mark

Systemd

Here’s the systemd unit I am using (/etc/systemd/system/postmarks.service):

[Unit]
Description=Postmarks bookmarking service
After=network-online.target

[Service]
User=mark
Group=mark
WorkingDirectory=/var/postmarks
ExecStart=npm run start
Restart=always

[Install]
WantedBy=multi-user.target

Then you can run:

systemctl daemon-reload
systemctl enable --now postmarks

and the service should be up!

Caddy

I’m using Caddy for TLS termination, to provide HTTP/2 and HTTP/3, and automatic TLS. This isn’t a full guide to Caddy, but here’s a bit to get you started:

# Debian packages a very old version, so use Caddy's repos
# These instructions come straight from Caddy's site
# https://caddyserver.com/docs/install#debian-ubuntu-raspbian
sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | tee /etc/apt/sources.list.d/caddy-stable.list
chmod o+r /usr/share/keyrings/caddy-stable-archive-keyring.gpg
chmod o+r /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

And here is my Caddyfile (`/etc/caddy/Caddyfile):

# The Caddyfile is an easy way to configure your Caddy web server.

mark.apalrd.net {
	# Reverse proxy mode
    reverse_proxy localhost:3000
    
    # Enable TLS but use self-signed cert
    tls internal
}

And restart Caddy: systemctl restart caddy

Netbox: The Network Source of Truth

Wed, 23 Apr 2025 01:00:05 -0300

Today I’m diving into Netbox, a tool designed to help you keep track of your network infrastructure. Netbox is a database of relationships, showing you what is connected where, all of your equipment, IP addresses and prefixes, etc.

We’ve got a LOT to install today! So hold on tight and follow along. Probably also want to do an apt update && apt full-upgrade -y before we start just to make sure the sytem is fully up to date.

If you want to do one big apt install and skip the rest, here it is:

# Update
apt update
# Install everything
apt install -y sudo postgresql redis-server python3 python3-pip python3-venv python3-dev build-essential libxml2-dev libxslt1-dev libffi-dev libpq-dev libssl-dev zlib1g-dev git
# Does not include Caddy and their repo

Video

Postgres

First install it:

apt update
apt install -y postgresql sudo

Next, login as the postgres user (sudo -u postgres psql) and run this SQL to create the database. You need to copy these commands one at a time into the prompt! I’ve added a line break each time you need to copy separately!

ALTER database template1 is_template=false;

DROP database template1;

CREATE DATABASE template1
WITH OWNER = postgres
   ENCODING = 'UTF8'
   TABLESPACE = pg_default
   LC_COLLATE = 'en_US.UTF-8'
   LC_CTYPE = 'en_US.UTF-8'
   CONNECTION LIMIT = -1
   TEMPLATE template0;

ALTER database template1 is_template=true;

CREATE DATABASE netbox;

CREATE USER netbox WITH PASSWORD 'CorrectHorseBatteryStaple';

ALTER DATABASE netbox OWNER TO netbox;

\connect netbox;

GRANT CREATE ON SCHEMA public TO netbox;

\q

Whew! First step done

Redis

Now more stuff to install!

apt install -y redis-server

You can test it with redis-cli ping but that’s it, it’s done

Netbox

More shell stuff to copy:

#Install everything we need
apt install -y python3 python3-pip python3-venv python3-dev build-essential libxml2-dev libxslt1-dev libffi-dev libpq-dev libssl-dev zlib1g-dev git
#Clone netbox into its new home (master should have latest stable release)
cd /opt
git clone -b main --depth 1 https://github.com/netbox-community/netbox.git
#Create a system user for netbox to run as
adduser --system --group netbox
#Chown netbox subdir's to Netbox
chown -R netbox /opt/netbox/netbox/{media,reports,scripts}/
#Copy example config as running config
#Dear god guys can you stop nesting netbox folders??
cp /opt/netbox/netbox/netbox/configuration_example.py /opt/netbox/netbox/netbox/configuration.py

Now before we edit the config file, let’s generate a random number to use as our secret key (and copy it for later) by running /opt/netbox/netbox/generate_secret_key.py. Then we can edit the config file (nano /opt/netbox/netbox/netbox/configuration.py)and change our settings. Here’s what I modified (this is a diff):

--- configuration_example.py
+++ configuration.py
@@ -8,15 +8,15 @@
 # access to the server via any other hostnames. The first FQDN in the list will be treated as the preferred name.
 #
 # Example: ALLOWED_HOSTS = ['netbox.example.com', 'netbox.internal.local']
-ALLOWED_HOSTS = []
+ALLOWED_HOSTS = ['netbox.palnet.net']
 
 # PostgreSQL database configuration. See the Django documentation for a complete list of available parameters:
 #   https://docs.djangoproject.com/en/stable/ref/settings/#databases
 DATABASE = {
     'ENGINE': 'django.db.backends.postgresql',  # Database engine
     'NAME': 'netbox',         # Database name
-    'USER': '',               # PostgreSQL username
-    'PASSWORD': '',           # PostgreSQL password
+    'USER': 'netbox',         # PostgreSQL username
+    'PASSWORD': 'CorrectHorseBatteryStaple',           # PostgreSQL password
     'HOST': 'localhost',      # Database server
     'PORT': '',               # Database port (leave blank for default)
     'CONN_MAX_AGE': 300,      # Max database connection age
@@ -64,7 +64,7 @@
 # For optimal security, SECRET_KEY should be at least 50 characters in length and contain a mix of letters, numbers, and
 # symbols. NetBox will not run without this defined. For more information, see
 # https://docs.djangoproject.com/en/stable/ref/settings/#std:setting-SECRET_KEY
-SECRET_KEY = ''
+SECRET_KEY = 'w!13OYWS!o2Or156)-!Ld^!9BZAKYrGFAeHGV=XnGYuM!!v*hk'

And finally, we can run the actual install script:

/opt/netbox/upgrade.sh

Create Superuser

Even after installing, we still need to create a superuser for our new site:

source /opt/netbox/venv/bin/activate
cd /opt/netbox/netbox
python3 manage.py createsuperuser
deactivate

Housekeeping Task

We could create a systemd timer unit for this, but I’m tired at this point and just want to get it done. So we are putting this entry in cron.daily:

ln -s /opt/netbox/contrib/netbox-housekeeping.sh /etc/cron.daily/netbox-housekeeping

Test It

Here’s a way to test it if you want to (you probably should):

source /opt/netbox/venv/bin/activate
cd /opt/netbox/netbox
python3 manage.py runserver [::]:8000 --insecure
#Use Crtl-C to end, log in to port 8000 to make sure it's up
deactivate

Gunicorn

Now we can add the WSGI server, Gunicorn, and its associated systemd service:

cp /opt/netbox/contrib/gunicorn.py /opt/netbox/gunicorn.py
cp /opt/netbox/contrib/netbox.service /etc/systemd/system
cp /opt/netbox/contrib/netbox-rq.service /etc/systemd/system
systemctl daemon-reload
systemctl enable --now netbox netbox-rq

Gunicorn will bind to localhost:8001, which is perfect since we are going to run a reverse proxy to do TLS termination.

Caddy

And now the TLS termination proxy - Caddy. Netbox suggests Apache or Nginx, but I prefer Caddy, so here’s my setup:

#Install from Caddy's repos (they are more up to date)
apt update
apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
# Download their deb signing key + their sources.list file
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
# Update with new sources
apt update
# Install Caddy
apt install -y caddy

And here’s my /etc/caddy/Caddyfile (delete everything already in it):

netbox.palnet.net {
    handle /static* {
        root * /opt/netbox/netbox/
        file_server
    }

    handle {
        #Backend is running on 8001
        reverse_proxy http://localhost:8001
    }
    #Only if you don't want to use the default of Let's Encrypt
    tls internal
    #Otherwise you can put your email:
    #tls adventure@apalrd.net
}

And we can start it with systemctl enable --now caddy! If it’s already running, then systemctl restart caddy.

Upgrading

When you need to upgrade, checkout master again and run the upgrade script:

#Go to netbox
cd /opt/netbox
#Pull latest
git checkout master
git pull origin master
#Run upgrade
./upgrade.sh
#Restart services
systemctl restart netbox netbox-rq caddy

All Open-Source THREAD Network

Wed, 26 Mar 2025 01:00:05 -0300

Today I’m taking a deep dive into the Thread protocol! Based on IEEE 802.15.4, Thread is a mesh networking protocol designed to balance the needs of small, battery powered Internet-of-Things devices with the ability to communicate directly on The Internet. By leveraging IPv6 and 6LoWPAN, Thread is able to finally bring these automation networks into the land of the Internet Protocol, where interoperability thrives.

I walk through the setup of an OpenThread Border Router, OpenThread daemon for end devices, and compile the OpenThread Radio Co-Processor for a Nordic Semiconductor NRF52840 dongle, one of the cheapest ways to get started with Thread.

Video

Dongle Firmware

First, we need firmware for our Radio Co-Processor. I’m using the Nordic Semi NRF52840 Dongle, using the firmware image provided by Nordic. Here’s the guide directly from them. I setup a small Debian VM on Proxmox using my cloud-init tutorial so I could easily pass-through USB devices and run the builds. Since the cloud-init VMs have no hardware support, I also had to install the normal kernel (apt install linux-image-generic -y) and reboot for it to detect my USB hardware. Nordic’s stuff seems to be very picky about Linux versions, it * really * wants Ubuntu 22.04 or 20.04 and not other versions, which is why the steps don’t exactly match theirs.

Anyway, here are the steps to follow:

# Install dependencies
sudo apt update
sudo apt install -y python3-pip git cmake ninja-build
# Add ourselves to dialout group so we can use the serial port
# NOTE: You may have to log out and back in after this or flashing will fail later
sudo usermod -a -G dialout $USER
# It's bad form to break system packages, but nordic doesn't provide us a better option
pip3 install --user -U west --break-system-packages
# Download nrfutil
curl https://files.nordicsemi.com/artifactory/swtools/external/nrfutil/executables/x86_64-unknown-linux-gnu/nrfutil -o nrfutil
# make executable
chmod +x nrfutil
# Move to location on PATH
sudo mv nrfutil /usr/local/bin/
# Install device so we can flash the device
nrfutil install device
# Install for pkg
nrfutil install nrf5sdk-tools
# Install toolchain-manager so we can manage toolchains
nrfutil install toolchain-manager
# Show toolchain options
nrfutil toolchain-manager search
# Install the latest version (as of this writing, 2.9.1)
nrfutil toolchain-manager install --ncs-version v2.9.1
# Make a subdir for us to work in
mkdir -p nrfbuild
cd nrfbuild
# Clone the repo at main (possibly dangerous, but you could choose a release tag instead)
python3 -m west init -m https://github.com/nrfconnect/sdk-nrf --mr main
# such as this tag
#west init -m https://github.com/nrfconnect/sdk-nrf --mr v2.9.1
# Make west download dependencies (this is very slow)
python3 -m west update
python3 -m west zephyr-export
# Go to the RCP folder and compile it
cd nrf/samples/openthread/coprocessor
# Launch into nrfutil toolchain environment (this sets up env vars)
nrfutil toolchain-manager launch --shell

At this point, you are in the coprocessor folder in the nrfutil shell. I stopped you here so you realize that you need to copy/paste again, in this new shell, and not in the other shell.

# -p means 'pristine' so it will start from scratch, in case you ran it before with errors
west build -b nrf52840dongle/nrf52840 -p

# Technically these don't need to go in the nrfutil shell, but they work either way

# Package the build for the bootloader
nrfutil pkg generate --hw-version 52 --sd-req=0x00 --application build/merged.hex --application-version 1 otbr-rcp.zip
# Flash the built onto the device
nrfutil dfu usb-serial -pkg otbr-rcp.zip -p /dev/ttyACM0

If you want a premade orbr-rcp.zip file, here’s the file I compiled (warning, random internet binaries, YMMV, no warranty, …)

In that case, you just need the nrfutil tool with device installed. You can also use their GUI flashing tool for Windows. I included the zip and the hex, some of the flashing tools want the hex instead.

Open Thread Border Router

followed https://openthread.io/codelabs/openthread-border-router#1 need to add WEB_GUI=1 before bootstrap and setup (it’s not the default on Debian)

I did this on Debian, this time on a single-board computer. I also tried a Raspberry Pi 4 and 3, and a Debian VM on Proxmox (but not using the cloud kernel!)

Here are the commands. You need sudo installed, even if you run as root, since OTBR’s scripts will use it.

# Instlll dependencies
sudo apt update
sudo apt install -y git
# Clone Open Thread Border Router
git clone https://github.com/openthread/ot-br-posix.git --depth 1
cd ot-br-posix
# Many of these exports have defaults already
# However, since they vary wildly for different OSes, I've set them here

# Enable Routing + Border Routing
export BACKBONE_ROUTER=1
export BORDER_ROUTING=1
# Disable NAT64 and DHCPv6-PD
export NAT64=0
export DNS64=0
export DHCPV6_PD=0
export DHCPV6_PD_REF=0
# Enable the web GUI and REST API
export WEB_GUI=1
export REST_API=1
# Set this to the name of your Ethernet interface to route to
export INFRA_IF_NAME=eth0

# Bootstrap environment
./script/bootstrap
# Actually do the build
./script/setup

Next we need to edit the UART location (only if you have multiple USB UART devices). You can find your serial devices in /dev/serial/by-id/, and pointing to those will be more reliable than using /dev/ttyACM0 if you have multiple dongles (i.e. Zigbee, Thread, ..). OpenThread Border Router loads an environment file called /etc/default/otbr-agent which contains some environment variables, so here’s what I started with and ended with:

# Default settings for otbr-agent. This file is sourced by systemd

# Options to pass to otbr-agent
OTBR_AGENT_OPTS="-I wpan0 -B ens18 spinel+hdlc+uart:///dev/ttyACM0 trel://ens18"
OTBR_NO_AUTO_ATTACH=0

and now it’s:

# Default settings for otbr-agent. This file is sourced by systemd

# Options to pass to otbr-agent
OTBR_AGENT_OPTS="-I wpan0 -B ens18 spinel+hdlc+uart:///dev/serial/by-id/usb-Nordic_Semiconductor_ASA_Thread_Co-Processor_FB7FFA86E3D48E34-if00 trel://ens18"
OTBR_NO_AUTO_ATTACH=0

Once that’s done, you can (re)start the services:

# Start the services
sudo systemctl start otbr-agent
sudo systemctl start otbr-web

And since I enabled the web UI, you can login on regular http port 80.

Thread Client

If you want to use Thread instead of Wifi, here’s the instructions to build Openthread as an end device (router eligible, but not a border router):

# Instlll dependencies
sudo apt update
sudo apt install -y git
# Clone Open Thread (not the border router)
git clone https://github.com/openthread/openthread.git --depth 1
cd openthread
# Bootstrap environment
./script/bootstrap
# Actually do the build
./script/cmake-build posix -DOT_DAEMON=ON

Next, we need to start up the daemon:

./build/posix/src/posix/ot-daemon -I wpan0 -v -d 4 'spinel+hdlc+uart:///dev/ttyACM0?uart-baudrate=115200'

(That will take the terminal, so login to another SSH session for ot-ctl)

Next, we can execute ./build/posix/src/posix/ot-ctl which should get us into an OpenThread terminal:

state
# Now we need to copy/paste the dataset from the border router
# On the BR, run:
# dataset active -x
# then copy the big blob of data,
# on the client, run:
dataset set active <big data blob>
# Now bring up the interface
ifconfig up
thread start
#And some useful commands for you
help
ipaddr
netdata show
extaddr

I wrote a systemd unit for this guy, so we can install it into /etc/systemd/system/ot-daemon.service:

[Unit]
Description=OpenThread DAemon
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ot-daemon -I wpan0 -d 4 'spinel+hdlc+uart:///dev/ttyACM0?uart-baudrate=115200'
Restart=always

[Install]
WantedBy=multi-user.target

Also, copy ot-daemon and ot-ctl to /usr/local/bin so they are in a normal binary place to be happy

Wireguard, OpenVPN, and IPSec for Client VPNs

Thu, 13 Mar 2025 01:00:05 -0300

Today I’m trying to understand if Wireguard really is over-hyped, if OpenVPN is really worth all the hassle to get the user-side features like client authentication and two factor, and if IPSec has any place in the modern VPN landscape. Specifically, looking at traditional ‘road warrior’ or client access VPNs, where all of your users are dialing in to your enterprise network, not the new-fangled mesh VPNs or zero trust setups.

This page contains all of the configurations I used in testing, feel free to use them on your own. I did all of my testing on Debian 13 (Trixie) which is currently in testing, on Linux 6.12. My clients are also running the same OS/kernel.

Video

Beginning

We need to enable IP forwarding on our router for any of these setups. This doesn’t persist across reboots, but most of my configs on this page are not intended to.

# Enable Forwarding
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding

IPIP

No install here. Just copy the commands and change the IPs as necessary. client is given 2001:db8:7:a::/64 in this example with the router taking the first address for itself, Enterprise subnet is 2001:db8:7::/48 in this example, Public IPs must be hardcoded for IPIP (cannot be dynamic).

# IPIP Setup (on Client)
ip link add name ipip6 type ip6tnl local <client IP> remote <server IP> mode any
ip link set up ipip6
ip addr add 2001:db8:7:a::2/64 dev ipip6
ip -6 route add 2001:db8:7::/48 dev ipip6

# IPIP Setup (on Server)
ip link add name ipip6 type ip6tnl local <server IP> remote <client IP> mode any
ip link set up ipip6
ip addr add 2001:db8:7:a::/64 dev ipip6
#addr add implicitly adds a /64 route to that interface, so no other routes required

Wireguard

First we install it (apt install wireguard)

Next, we need to create keys on each nodes, so run this command on both sides:

#Gen private key
wg genkey | tee /etc/wireguard/server.key
#Gen public key from private key
cat /etc/wireguard/server.key | wg pubkey | tee /etc/wireguard/server.pub
# You will need these two keys later, so be ready to copy and paste them

Next, we need a /etc/wireguard/wg0.conf on both nodes:

# Conf for SERVER side
[Interface]
PrivateKey = oBic4cnZOBjmucL10dtiB5QZpUkqPPjEm9rA6E90iUg=
Address = 2001:db8:7:b::/128
ListenPort = 51820
# Peer (CLIENT's public key here)
[Peer]
PublicKey = 1b0J0WcCObEOq0R9zIQoitRa6I4YqLDDL7evmNwN3Go=
AllowedIPs = 2001:db8:7:b::/64
Endpoint = testsrc2.palnet.net:51820

# Conf for CLIENT side
[Interface]
PrivateKey = qKUMygkE+YPWE2MmdCOi5j6Sno78/w9FvXHGTIVpoVU=
Address = 2001:db8:7:b::4/64
ListenPort = 51820
# Peer (SERVER's public key here)
[Peer]
PublicKey = 2VIGy0JS27RjGI9VDug2iNK4dd/Ned9dNDnlCOP1DTQ=
AllowedIPs = 2001:db8:7::/48
Endpoint = wgmetal.palnet.net:51820

Then bring up the interfaces (wg-quick up wg0 on both sides). WG will automatically brign up the route to our enterprise subnet, and it will work.

OpenVPN

First we install it (apt install openvpn).

Certificates

We will be using cert auth here, so we need an x.509 cert for both sides. For simplicity, I’m using self-signed ECDSA certs using prime256v1 for leafs and scep384 for the CA

#Generate private key using scep384:
openssl ecparam -name secp384r1 -genkey -out root.key

#Sign the root certificate
#Pathlen:0 means there can be only one more cert below this CA (no more CAs)
#Make sure you update the subj name with your own names
#C=US is also the country, it's optional
#O= is the organization, also optional
#CN= is the Common Name and it's required
#I also set validity to 69 years, make sure you watch for expiration (manually)
openssl req -new -key root.key -x509 -nodes -days 25202 -out root.pem -subj "/C=US/O=apalrd.net/CN=openvpn" -addext "basicConstraints=critical,CA:TRUE,pathlen:0"

#Now you can view it (for fun)
openssl x509 -in root.pem -text -noout

Next, generate the server’s cert + key

#Change this to the name of your server
export sv=wgmetal.palnet.net
#Generate scep256 key for this client
#If you want 192-bit security, use scep384r1 instead of prime256v1
openssl ecparam -name prime256v1 -genkey -out $sv.key

#Generate a CSR (certificate signing request) for my new key
#again, C and O are optional, CN is the Common Name of the cert
#Allowed only for server auth
openssl req -new -key $sv.key -out $sv.csr -subj "/C=US/O=apalrd.net/CN=$sv" -addext "extendedKeyUsage = serverAuth" --addext "keyUsage = digitalSignature,keyAgreement" -addext "subjectAltName = DNS:$sv"

#Sign the CSR using the root
#Shorter lived at only 365 days
openssl x509 -req -in $sv.csr -CA root.pem -CAkey root.key -CAcreateserial -out $sv.crt -days 365 -sha256 -copy_extensions=copyall

#Now you can view it (for fun)
openssl x509 -in $sv.crt -text -noout

And finally, generate the client cert:

#Change this to the name of your client
export cl=testsrc2.palnet.net
#Generate scep256 key for this client
#If you want 192-bit security, use scep384r1 instead of prime256v1
openssl ecparam -name prime256v1 -genkey -out $cl.key

#Generate a CSR (certificate signing request) for my new key
#again, C and O are optional, CN is the Common Name of the cert
openssl req -new -key $cl.key -out $cl.csr -subj "/C=US/O=apalrd.net/CN=$cl" -addext "extendedKeyUsage = clientAuth" --addext "keyUsage = digitalSignature,keyAgreement"

#Sign the CSR using the root
#Sign it allowing for server and client auth as the key usage
openssl x509 -req -in $cl.csr -CA root.pem -CAkey root.key -CAcreateserial -out $cl.crt -days 365 -sha256 -copy_extensions=copyall

#Now you can view it (for fun)
openssl x509 -in $cl.crt -text -noout

Transfer your .key and .crt files to the server and client respectively. The server will need its client .key and .crt, and the server will need its server .key and .crt, and they both need root.pem

Server

The server setup is quite complex. Here’s my final /etc/openvpn/server.conf:

server
#UDP over IPV6 outside of tunnel
port 1194
proto udp6
dev tun
user nobody
group nogroup
persist-key
persist-tun
keepalive 10 120
topology subnet

#IPv6 inside of tunnel
server-ipv6 2001:db8:7:c::/64
ifconfig-ipv6-pool 2001:db8:7:c::/64
push "route-ipv6 2001:db8:7::/48"
ifconfig-pool-persist ipp.txt

#Crypto parameters
dh none
ecdh-curve prime256v1
tls-crypt tls-crypt.key

#Client verification
ca root.pem
cert server.crt
key server.key
remote-cert-tls client
# no CRL in my setup
#crl-verify root.pem
auth SHA256
cipher AES-128-GCM
data-ciphers AES-128-GCM
tls-server
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
client-config-dir /etc/openvpn/ccd
status /var/log/openvpn/status.log
verb 3

Also, a few commands to get it started:

openvpn --genkey secret /etc/openvpn/tls-crypt.key
mkdir -p /etc/openvpn/ccd
mkdir -p /var/log/openvpn
systemctl enable --now openvpn@server

Once we’ve generated tls-crypt.key we need to copy that file to the client (it’s our Shared Secret). We don’t * have * to use shared secret auth, but I have here.

Client

For the client we need a /etc/openvpn/client.conf file:

client
remote wgmetal.palnet.net 1194
dev tun
resolv-retry infinite
nobind
persist-key
persist-tun
remote-cert-tls server
verify-x509-name wgmetal.palnet.net name
auth SHA256
cipher AES-128-GCM
data-ciphers AES-128-GCM
auth-nocache
tls-client
tls-version-min 1.2
tls-cipher TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256
verb 3
#Client certs
cert client.crt
key client.key
#Server's CA
ca root.pem
tls-crypt tls-crypt.key
remote-cert-tls server

And a simple command to start:

systemctl enable --now openvpn@client

DCO

The whole reason for this test is to see if DCO is as performant! And we aren’t using DCO just yet. So, here’s how to enable it:

#Make sure you are running a kernel version which is in the repo
apt update
apt upgrade -y
#Reboot if you installed a new kernel here, or linux-headers will not find itself
apt install -y linux-headers-$(uname -r) 
#This will compile the module with DKMS
apt install -y openvpn-dco-dkms
modprobe ovpn-dco-v2

At this point, restarting the server (or client, it works both ways) should enable DCO. Check the logs (journalctl -xeu openvpn@server) to make sure.

IPSec

Oh boy you are in for a fun ride here, we’re gonna install Strongswan (apt install -y strongswan) to get started.

StrongSwan Server

We are gonna need the server’s certificate and key from earlier, and since StrongSwan is stupid, we need to convert it to DER format (openssl ec -inform pem -outform der -in server.key -out server.der). Put the root cert into /etc/ipsec.d/cacerts/root.pem, the server’s cert in /etc/ipsec.d/certs/server.crt, and the server’s DER key in /etc/ipsec.d/private/server.der.

We’re going to completely erase /etc/ipsec.conf, you can replace it with my new one:

config setup
    charondebug="ike 1, knl 1, cfg 0"
    uniqueids=no

# our ikev2 vpn
conn ikev2-vpn
    auto=add
    compress=no
    type=tunnel
    keyexchange=ikev2
    fragmentation=yes
    forceencaps=yes
    dpdaction=clear
    dpddelay=300s
    rekey=no
    #left side (idfk why they name it left and right)
    #ip to bind to locally
    left=%any
    #this must match the SAN of the certificate
    #or, skip the @ and use an IP
    leftid=@wgmetal.palnet.net
    #leftid=2001:db8:6969::420
    leftcert=server.crt
    leftsendcert=always
    #route to tell the clients
    leftsubnet=2001:db8:7::/48
    #again, no idea why they call them left/right
    #anyway clients can be anything
    right=%any
    rightid=%any
    #eap-tls would use client certs, but we will do mschap for now
    rightauth=eap-mschapv2
    #address pool for clients
    rightsourceip=2001:db8:7:d::/64
    #you can assign DNS too if you want
    #rightdns=2620:fe::fe
    #set this to never for mschap or always for eap-tls
    rightsendcert=never
    #??
    eap_identity=%identity
    #crypto
    #yes, these are no longer recommended in favor of AES 256
    #however I tested OpenVPN with aes128gcm so I wanted to make it fair
    ike=aes128gcm16-prfsha256-ecp256
    esp=aes128gcm16-ecp256

For some reason, StrongSwan doesn’t use a file path to a key in the config file. I guess they treat the path itself as secret? Anyway, edit /etc/ipsec.secrets:

# This file holds shared secrets or RSA private keys for authentication.

# Our ECDSA private key
: ECDSA server.der

# Our admin user accounts (these are plaintext passwords)
admin : EAP "admin"
admin2 : EAP "admin"
admin3 : EAP "admin"
admin4 : EAP "admin"
admin5 : EAP "admin"
admin6 : EAP "admin"
admin7 : EAP "admin"

StrongSwan Client

The client side is a bit easier. We again need our root certificate (put it in /etc/ipsec.d/cacerts/root.pem), but not a client cert, since we are doing old fashioned usernames and passwords here.

We’re going to setup the config for the client (/etc/ipsec.conf) also:

config setup

conn ikev2-rw
    right=wgmetal.palnet.net
    # This should match the `leftid` value on your server's configuration
    rightid=%wgmetal.palnet.net
    rightsubnet=::/0
    rightauth=pubkey
    leftsourceip=%config
    leftid=admin2
    leftauth=eap-mschapv2
    eap_identity=admin2
    auto=start

And we need a ipsec.secrets file here too:

# This file holds shared secrets or RSA private keys for authentication.

# Our user account (these are plaintext passwords)
admin2 : EAP "admin"

Home Assistant Remote Radio

Tue, 25 Feb 2025 01:00:05 -0300

One of the challenges of running Home Assistant in a virtualized environment is the access to hardware radios. Home Assistant ultimately needs to bridge a ton of home automation networks, and most of these require some sort of USB / Serial dongle. You could have all of your dongles in your server closet and pass them through to the VM/container, but then you can’t migrate it across hosts. You could have one dongle for each host, I guess, but that doesn’t work for Z-wave where the entire network is stored on the dongle. Plus, the server closet probably isn’t a great place for the one radio anyway. So, I’m showing you how I use a Raspberry Pi to run my Zwave and Zigbee radios, remotely, for Home Assistant. This keeps the core Home Assistant setup flexible in where it can run on the network, and also lets me run multiple remote radios without resorting to weird tricks in HA OS.

Video

Base OS

I’ve chosen to install Raspberry Pi OS Lite, a variant of Debian Bookworm. I’m using the ARM 64-bit builds for Raspberry Pi 3. The Lite version just lacks the UI, which is fine since I’m never going to plug in a display.

As far as I know, all of this should work fine on Debian Bookworm as well, if you want to use a mini PC.

Don’t forget to run OS update as well:

apt update
apt full-upgrade -y

I also dropped my SSH public key into ~/.ssh/authorized_keys (the file does not exist by default) to avoid password logins.

I disabled IPv4 on my setup (nmcli connection modify 'Wired connection 1' ipv4.method disable), and put the IPv6 address in DNS where it belongs. You are of course free to do other things, such as instlaling avahi-daemon for mdns, for example.

PBS Backups

I backup everything to Proxmox Backup Server, so I need to install the client to backup this host.

If you’re running on an ARM device you can get deb packages from https://github.com/wofferl/proxmox-backup-arm64 (you only need the client deb, not all of them)

If you are running on x86 you can use Proxmox’s client-only repository instead.

In any case, install PBS Client from packages.

Here’s my backup script (/etc/systemd/system/backup.service):

[Unit]
Description=Run Backup to Proxmox Backup Server
After=network-online.target

[Service]
#TLS fingerprint if cert is self-signed
Environment=PBS_FINGERPRINT=your_fingerptint
#Your API key here
Environment=PBS_PASSWORD=api_key_here
#Use your user @ realm ! api key name @ hostname : repository
Environment=PBS_REPOSITORY=user@pbs!apikey@localhost:backup
Type=oneshot
#Backup the root fs. This will not backup any other mount points, only the mount point which mounts /
#You can add more pxars if you want more mount points
#--change-detection-mode=metadata is optional, but I prefer it
ExecStart=proxmox-backup-client backup root.pxar:/ --change-detection-mode=metadata

[Install]
WantedBy=default.target

I also have a backup timer (/etc/systemd/system/backup.timer):

[Unit]
Description=Backup System Daily
RefuseManualStart=no
RefuseManualStop=no

[Timer]
#Run 540 seconds after boot for the first time
#OnBootSec=540
#Run at 7pm
OnCalendar=*-*-* 19:00:00
Unit=backup.service

[Install]
WantedBy=timers.target

Then a quick systemctl daemon-reload and sytemctl enable backup.timer is all you need. When the timer fires, it will trigger the service, which will run a single backup. Since it runs every day at 7pm, it will do one backup per day.

ZwaveJS UI

I previously used Snap packages but I really hated working with them, so I am now using the NPN (nodeJS package manager) version combined with a systemd service script.

Install:

#Node 22 (seems like a reasonable choice)
curl -fsSL https://deb.nodesource.com/setup_22.x -o nodesource_setup.sh
sudo -E bash nodesource_setup.sh

#Install it from packages
sudo apt install nodejs -y

#Since we used apt here, we can run `apt upgrade -y` to upgrade Node minor versions

#Install zwave js ui
sudo npm install -g zwave-js-ui

#Create directory
mkdir -p /var/zwavejs

Service (/etc/systemd/system/zwavejs.service):

[Unit]
Description=Z-Wave Network Manager
After=network-online.target

[Service]
#ZwaveJS binary
ExecStart=/usr/bin/zwave-js-ui
#Storage location
Environment=STORE_DIR=/var/zwavejs
#Config location
Environment=ZWAVEJS_EXTERNAL_CONFIG=/var/zwavejs/.config-db

[Install]
WantedBy=default.target

Then it’s just a systemctl daemon-reload and systemctl enable --now zwavejs.

In my case, this would immediately cause the Pi to lock up due to underpower, so …. that’s fun? I was hoping to migrate my old Zwave network, but I bought a new dongle and moved on to re-pairing everything. Zwave stores everything within the chipset, and being a proprietary standard with a single vendor, there are no hardware options other than Silicon Labs. On the plus side, compatibility between devices is excellent, but the downside is it isn’t as easy for ZwaveJS to store the network config in a place I can back it up.

Zigbee2MQTT

This one is also written in NodeJS, so we again need to make sure NodeJS is installed and ready to go (skip this if you already did ZwaveJS):

Install NodeJS:

#Node 22 (ZwaveJS is using 18 in their docker images I think?)
curl -fsSL https://deb.nodesource.com/setup_22.x -o nodesource_setup.sh
sudo -E bash nodesource_setup.sh

#Install it from packages
sudo apt install nodejs -y

#Since we used apt here, we can run `apt upgrade -y` to upgrade Node minor versions

Now, we can install Zigbee2MQTT from source:

#Need git and friends
sudo apt install git

# Create a directory
sudo mkdir /opt/zigbee2mqtt
sudo chown -R ${USER}: /opt/zigbee2mqtt

#Checkout into directory
git clone --depth 1 https://github.com/Koenkk/zigbee2mqtt.git /opt/zigbee2mqtt

#Install depends
cd /opt/zigbee2mqtt
npm ci

#Build
npm run build

Service (/etc/systemd/system/zigbee2mqtt.service):

[Unit]
Description=Zigbee 2 MQTT
After=network-online.target

[Service]
#Node binary
ExecStart=/usr/bin/node index.js
#Location
WorkingDirectory=/opt/zigbee2mqtt

[Install]
WantedBy=default.target

Then it’s just a systemctl daemon-reload and systemctl enable --now zigbee2mqtt (these may need sudo fyi).

After that, we need to set a few things in the configuration.yaml, so edit data/configuration.yaml (it’s in /opt/zigbee2mqtt/data/configuration.yaml). You can also copy the example (configuration.example.yaml).

Things you should do:

Set homeassistant to true
Set permit_join to false (you can re-enable it from the UI)
Set your MQTT broker server, username, and password
Set the location of your serial port (please use the one in /dev/serial/by-id/ instead of /dev/ttyUSBX in case your devices change)
Add frontend: true to run the web UI

You can access the web ui on port 8080 to configure it.

For reference, here is my configuration (redacted) for reference:

homeassistant: true
frontend: true
mqtt:
  base_topic: zigbee2mqtt
  #MQTT User Information Here
  server: mqtt://telstar.palnet.net
serial:
  port: /dev/serial/by-id/usb-1a86_USB_Serial-if00-port0
advanced:
  homeassistant_legacy_entity_attributes: false
  legacy_api: false
  legacy_availability_payload: false
  #PAN ID, etc. removed
#Devices removed

RTL-433

Previous Project Entry Here When I previously set this up I had to compile it from source. Now, it’s included in the Pi repos directly, so no need to do that.

apt install rtl-433 -y

Not to revisit that previous entry too much, here’s my service file (/etc/systemd/rtl433.service):

[Unit]
Description=rtl_433 SDR Receiver Daemon
After=network-online.target

[Service]
ExecStart=/usr/bin/rtl_433 -C si -F "mqtt://<broker>:1883,user=<user>,pass=<pass>"
Restart=always

[Install]
WantedBy=multi-user.target

Then it’s just a systemctl daemon-reload and systemctl enable --now rtl433.

Updates

Managing software updates can be a full-time task, so here are the update commands for everything:

apt update && apt upgrade -y to update the base Debian system and Node.JS
npm update zwave-js-ui && systemctl restart zwavejs to update ZwaveJS UI itself
There are no updates for my UPS daemon
Zigbee2MQTT needs a bit more:

#cd to the directory
cd /opt/zigbee2mqtt
#Pull update from Git
git pull
#Make depend
npm ci
#Make
npm run build
#Restart service
systemctl restart zigbee2mqtt

Physical Network Access Control with 802.1X

Sat, 14 Dec 2024 01:00:05 -0300

Today I’m diving in to the world of network access control! Being able to authenticate network devies plugged in to your switches is a great way to improve network security without resorting to unplugging or disabling every unused port on yout equipment. Now every switch port is universal, and will enable on demand based on what is plugged in. While I couldn’t go through the complete authorization part of the setup (mapping devices to VLANs), I’m planning on making a future video for that step.

Video

Certificates Overview

RADIUS and EAP love nesting, so we need a lot of keys here. In particular:

Server cert for RADsec
Client cert for RADsec (per-authenticator, which is an AP or Switch)
Server cert for EAP (may be the same as RADsec)
Client cert for each client

RADIUS is not a particularly secure protocol on its own. It operates over UDP 1812/1813 and has no encryption at all. Messages are authenticated using a ‘shared secret’ (which is often words instead of randomly generated), and relying on an MD5 hash. MD5 isn’t a great hash function in 2024 (it’s cryptographically considered insecure already, and so is its replacement SHA1). It’s also far from a password-quality hash for dealing with poor human-generated sources, so it’s basically entirely broken. Most of the RADIUS payloads do their own encryption all the way to the client, using TLS, but this still doesn’t encrypt our Accept decision which is sent over the clear to the authenticator.

Two industries took on the bad RADIUS security problem in different ways - the telecom industry decided to go off into the weeds and build something completely different and far more complex with far more ASN.1 and called it DIAMETER (ha ha). The normal people took the whole RADIUS protocol (MD5 and all) and dropped it inside a TLS session, where TLS provides security and RADIUS uses a hardcoded ‘secret’ (radsec is the literal secret).

So now we need to implement RADsec, which relies on mutual TLS (this should sound familiar if you follow me). So, for the TLS Sessions for RADsec, we need a server cert (for FreeRADIUS) and a client cert (for our authenticators). These can be from any CA, but since all of this equipment is our own it would make sense to use a private CA for all of these.

Then, separately, we are either going to do EAP-TLS (preferred) or EAP-PEAP (not preferred but commonly implemented) to the clients. Both of these encapsulate in TLS. EAP-TLS uses mutual TLS to authenticate the client, and EAP-PEAP uses TLS to authenticate the server and then MS-CHAPv2 to authenticate the client (probably using MD5 again).

I have three different methods for generating certs, which I have used in videos at this point:

Use OpenSSL and do it entirely manually
Use Smallstep and run a full private PKI setup
Use FreeRADIUS’s testing certs

Your choice depends on what you want to get out of the full setup. I’d also like to remind you that one cert per machine is very normal, you don’t need a different set of certs for mtls and also vpn and also 802.1x.

FreeRADIUS Test Certs

To generate FreeRADIUS test certs (after installing FreeRADIUS):

#Go to this directory, you must cd here
cd /etc/freeradius/3.0/certs
#Make the CA initially
make

#If you change configs and re-run make, it should regenerate
#In case it doesn't, you can run destroycerts
make destroycerts
make

My Easy OpenSSL Method

This method generates all of the certs we will need today by copying/pasting. Edit the commands before you run them to change the options. It’s easier to understand than the makefile, but also not very scalable. If you watched my mTLS video, it’s basically the same as that. You should probably use Smallstep for production-like things.

Certificate Authority

Same as mtls, simple 384-bit key

#Generate private key using scep384:
openssl ecparam -name secp384r1 -genkey -out root.key

#Sign the root certificate
#Pathlen:0 means there can be only one more cert below this CA (no more CAs)
#Make sure you update the subj name with your own names
#C=US is also the country, it's optional
#O= is the organization, also optional
#CN= is the Common Name and it's required
#I also set validity to 69 years, make sure you watch for expiration (manually)
openssl req -new -key root.key -x509 -nodes -days 25202 -out root.pem -subj "/C=US/O=apalrd.net/CN=radius" -addext "basicConstraints=critical,CA:TRUE,pathlen:0"

#CA cert must be readable by freerad
chown freerad:freerad root.pem

#Now you can view it (for fun)
openssl x509 -in root.pem -text -noout

Server

The same cert is used for both radsec and eap on the server side.

export sv=raddb.apalrd.net
#Generate scep256 key for this client
#If you want 192-bit security, use scep384r1 instead of prime256v1
openssl ecparam -name prime256v1 -genkey -out $sv.key

#Generate a CSR (certificate signing request) for my new key
#again, C and O are optional, CN is the Common Name of the cert
#Allowed only for server auth (in both radsec and eap roles, we are the server)
#Set for TOFU policy (network asks the user to trust on first use)
#Change certificatePolicy to 1.3.6.1.4.1.40808.1.3.1 for strict (client must be pre-configured to trust CA)
openssl req -new -key $sv.key -out $sv.csr -subj "/C=US/O=apalrd.net/CN=$sv" -addext "extendedKeyUsage = serverAuth" -addext "certificatePolicies = 1.3.6.1.4.1.40808.1.3.2"
#TODO crlDistributionPoints

#Sign the CSR using the root
#Shorter lived at only 365 days
openssl x509 -req -in $sv.csr -CA root.pem -CAkey root.key -CAcreateserial -out $sv.crt -days 365 -sha256 -copy_extensions=copyall

#Must add CA cert after server cert
cat root.pem > $sv.crt

#Server key must be readable by freerad
chown freerad:freerad $sv.key $sv.crt

#Now you can view it (for fun)
openssl x509 -in $sv.crt -text -noout

Client

You need one of these for each authenticator (wifi AP / switch), and also one for each EAP client. Change the name (export cl) each time. Common practice is to use a FQDN for device certs and user@domain for user certs

export cl=wap1.apalrd.net
#Generate scep256 key for this client
#If you want 192-bit security, use scep384r1 instead of prime256v1
openssl ecparam -name prime256v1 -genkey -out $cl.key

#Generate a CSR (certificate signing request) for my new key
#again, C and O are optional, CN is the Common Name of the cert
openssl req -new -key $cl.key -out $cl.csr -subj "/C=US/O=apalrd.net/CN=$cl" -addext "extendedKeyUsage = clientAuth"

#Sign the CSR using the root
#Sign it allowing for server and client auth as the key usage
openssl x509 -req -in $cl.csr -CA root.pem -CAkey root.key -CAcreateserial -out $cl.crt -days 365 -sha256 -copy_extensions=copyall

#Now you can view it (for fun)
openssl x509 -in $cl.crt -text -noout

#Now let's package it into a P12 archive so you can send it to your favorite client device
#You *must* enter a password here or some OSes will not accept the P12
#The password just encrypts the P12 file itself
openssl pkcs12 -export -out $cl.p12 -in $cl.crt -inkey $cl.key

Install FreeRADIUS

I’m running FreeRADIUS on a Debian 12 system (LXC continer, unprivilaged). Start by installing Debian 12 using your favorite method, and updating it fully (apt update && apt full-upgrade -y).

#Install from deb packages
apt install freeradius -y

Now we’re going to move over to /etc/freeradius/3.0 where we are going to hang out for a long time

The folder structure contains two directories with configs (mods-available and sites-available), which is what we edit, and when we want to enable them, we create a symlink in the corresponding *-enabled directory. This is similar to how a lot of distros package Apache and nginx, for example.

Systemd Override

To use client certificates (in both RADsec and in EAP-TLS), we need a temp directory for FreeRADIUS. Systemd is smart enough to create a per-service /tmp which we should just use, but FreeRADIUS is dumb enough to demand a directory which already exists, and then tries to chmod/chown it on startup (which Systemd has prevented). So, edit the systemd service (systemctl edit freeradius) and add this to the top:

[Service]
User=freerad
Group=freerad
RuntimeDirectory=freeradius freeradius/tmp
RuntimeDirectoryPreserve=yes

RADsec Configuration

This file goes in sites-available/radsec and configures the clients. Then, create a symlink to it (ln -s ../sites-available/radsec sites-enabled/radsec). I have cut a lot of the documentation out of this file, but if you want to read it, I started from the file sites-available/tls.

######################################################################
#
#  RADIUS over TLS (radsec)
#
######################################################################

listen {
    # FreeRADIUS is stupid sometimes / often
    # ipaddr = * means any *IPv4*
    # ipaddr = :: means any *IPv6*
	ipaddr = ::
	port = 2083

	# Allow both Auth and Accounting
	type = auth+acct

	# For now, only TCP transport is allowed.
	proto = tcp

	# Send packets to the default virtual server
	virtual_server = default

    # This is a client name, for the `radsec` section later
	clients = radsec

	# Connection limiting for sockets with "proto = tcp".
	limit {
	      #  Limit the number of simultaneous TCP connections to the socket
	      max_connections = 16

	      #  The lifetime, in seconds, of a TCP connection.  After
	      #  this lifetime, the connection will be closed.
	      lifetime = 0

	      #  The idle timeout, in seconds, of a TCP connection.
	      #  If no packets have been received over the connection for
	      #  this time, the connection will be closed.
	      idle_timeout = 30
	}

    # TLS Crypto section
	tls {
        # If your private key is encrypted
		# private_key_password = whatever
		private_key_file = /etc/freeradius/3.0/certs/raddb.apalrd.net.key

		# Certificate file
        # If key + cert are in the same file, use the same filename as above
		certificate_file = /etc/freeradius/3.0/certs/raddb.apalrd.net.crt

		# Trusted Root CA list
        # Combine all roots one after another into this file
		ca_file = /etc/freeradius/3.0/certs/root.pem

		# Fragment size (read the docs if you want to change)
		fragment_size = 8192


		# Set this option to specify the allowed
		# TLS cipher suites.  The format is listed
		# in "man 1 ciphers".
		cipher_list = "DEFAULT"

		# If enabled, OpenSSL will use server cipher list
		cipher_server_preference = no

		#  Older TLS versions are deprecated.
		tls_min_version = "1.2"
		tls_max_version = "1.3"


		#  Session resumption / fast reauthentication cache
		cache {
            enable = yes
            lifetime = 24 # hours
			name = "TLS RADSEC"
			persist_dir = "${logdir}/tlscache"
		}

		#  Require a client certificate.
		require_client_cert = yes

		# Verify client certs (i.e. using OCSP)
        # Configured to use the CA we setup earlier
		verify {
			# Due to some *quirks* in FreeRADIUS, it expects to be able to
            # manipulate the permissions of this directory
            # systemd would rather isolate it itself, so we need
            # a systemd override to use this feature
            tmpdir = /run/freeradius/tmp

			#  The command used to verify the client cert.
			#  We recommend using the OpenSSL command-line
			#  tool.
	        client = "/usr/bin/openssl verify -CAfile /etc/freeradius/3.0/certs/root.pem %{TLS-Client-Cert-Filename}"
		} #end verify
	} #end tls
} #end server

clients radsec {
    #Local host
    client localhost {
        ipaddr = ::1
        proto = tls
        #When using radsec, secret MUST be 'radsec'
        #this is literally written in the RFC
        secret = radsec
    }
    #Local network, in CIDR notation
    #Since we are using cert based auth, it would not be unreasonable
    #to allow all (::/0)
    client localnet {
        ipaddr = fd69:beef:cafe::/48
        proto = tls
        secret = radsec
    }
}#end radsec

EAP Module Configuration

Next, we need to setup the certs we will use for EAP. These are different than the ones we use for RADsec. This configuration goes in mods-available/eap, and I’ve written out the whole file so you can replace it if you want. Again, I trimmed comments for features I’m not using, so read the original for more context.

######################################################################
#
#  EAP-TLS and no other modes
#
######################################################################
eap {
	#  Invoke the default supported EAP type when
	#  EAP-Identity response is received.
	default_eap_type = tls

	# EAP-Request/Response cache timeout
	timer_expire = 60

	# Reject unknown types
	ignore_unknown_eap_types = no

	#  Help prevent DoS attacks by limiting the number of sessions we track
	max_sessions = ${max_requests}


	#  Common TLS configuration for TLS-based EAP types
	tls-config tls-common {
		#private_key_password = whatever
		private_key_file = /etc/freeradius/3.0/certs/radius.apalrd.net.key
		certificate_file = /etc/freeradius/3.0/certs/radius.apalrd.net.crt


		#  Trusted Root CA list *for clients*
		ca_file = /etc/freeradius/3.0/certs/root.pem

		#  Set this option to specify the allowed TLS cipher suites
		cipher_list = "DEFAULT"

		#  If enabled, OpenSSL will use server cipher list
        #  In 2024 leave this disabld
		cipher_server_preference = no

		#  Set min / max TLS version.
		#
		#  While the server will accept "1.3" as a value,
		#  most EAP supplicants WILL NOT DO TLS 1.3 PROPERLY.
		#
		#  i.e. they WILL NOT WORK, SO DO NOT ASK QUESTIONS ON
		#  THE LIST ABOUT WHY IT DOES NOT WORK.
		#
		#  The TLS 1.3 support is here for future
		#  compatibility, as clients get upgraded, and people
		#  don't upgrade their copies of FreeRADIUS.
		#
		#  Also note that we only support TLS 1.3 for EAP-TLS.
		#  Other versions of EAP (PEAP, TTLS, FAST) DO NOT
		#  SUPPORT TLS 1.3.
		#
		tls_min_version = "1.2"
		tls_max_version = "1.2"

		#  Client certificates can be validated via an
		#  external command.  This allows dynamic CRLs or OCSP
		#  to be used.
		verify {
		    tmpdir = /run/freeradius/tmp

            #  The command used to verify the client cert.
			#  We recommend using the OpenSSL command-line
			#  tool.
            client = "/usr/bin/openssl verify -CAfile /etc/freeradius/3.0/certs/root.pem %{TLS-Client-Cert-Filename}"
		}
	}


	#  EAP-TLS
	tls {
		# Point to the common TLS configuration
		tls = tls-common
	}

    # EAP PEAP
	peap {
		# Point to the common TLS configuration
		tls = tls-common

        # Name of virtual server to handle the inner tunnel
		virtual_server = "inner-tunnel"

		#  The tunneled EAP session needs a default
		#  EAP type which is separate from the one for
		#  the non-tunneled EAP module.
		default_eap_type = mschapv2

        # Other things you don't need to change
        # I cut out paragraphs of docs here
		copy_request_to_tunnel = no
		use_tunneled_reply = no
	} #end eap peap

	#  EAP-MSCHAPv2
	#  This will be used *inside* PEAP
	mschapv2 {
        #We actually need no config here
	} # end eap mschapv2

} # end eap

Inner Tunnel Configuration (Optional)

If you need to support clients which only speap EAP-PEAP, then we can configure the ‘inner’ tunnel. This is basically an EAP session (EAP-MSCHAPv2) inside of another EAP session, and how anyone thought this was intelligent is beyond me.

If you are NOT using inner tunnel, you can remove the symlink to disable the virtual server: rm sites-enabled/inner-tunnel

If you ARE using inner tunnel, you can leave the symlink alone. The default file is fine.

Default Server Configuration (Optional)

Since the default server listens on the (insecure) RADIUS ports 1812 and 1813, I removed those lines from the default file (sites-available/default). In my setup, this is from line 48 (the line immediately after server default {) to line 272 (right before the comment block for authorize {).

This is purely optional. The current attacks on non-radsec RADIUS all involve attacking the authenticator side and allowing access, so having the RADIUS server exposed in this way is not believed to be a security risk. FreeRADIUS is also protected by not having any clients defined by default, so connections would be rejected anyway. Regardless, I don’t want that protocol in use, so I disabled it.

If you like using the radtest tool, you could change it to bind to ::1 instead of ::, so radtest still works unencrypted on localhost.

Users File (Optional)

The Users file (users, right in the freeradius directory) stores user info when logging in with a username+password. It’s not used in EAP-TLS, since we instead trace the cert and don’t actually look at the identity at all, but it’s used in PEAP-MSCHAPv2. If you are enabling that one, then here’s an example users file:

#
# 	Configuration file for the rlm_files module.
# 	Please see rlm_files(5) manpage for more information.
#
#   All lines start with the username, followed by the conditions to accept
#   Remaining lines are responses. Read the original users file for more help.
#

# My stupid camera that won't do TLS, only PEAP
camera3 Cleartext-Password := "camera3"

## Same, but only on port 3 of the switch with the right hostname
camera5 Cleartext-Password := "camera5", NAS-Port-Id == "ether3", NAS-Identifier == "sw5.palnet.net"

## Same, but with a specific MAC address
camera7 Cleartext-Password := "camera7", Calling-Station-Id == "08-ED-ED-F1-1B-32"
    Tunnel-Type = VLAN,
    Tunnel-Medium-Type = 802,
    Tunnel-Private-Group-ID = "10"
    #Add it to the camera VLAN

## Same, but stuck on a VLAN 9
camera11 Cleartext-Password := "camera11"
    Tunnel-Type = VLAN,
    Tunnel-Medium-Type = 802,
    Tunnel-Private-Group-ID = "9"
    #Add it to the camera VLAN


# DEFAULT reject them
# You could also := Accept them
# and give them attributes like default vlan
DEFAULT Auth-Type := Reject

Caching Linux Package Repositories

Thu, 05 Dec 2024 01:00:05 -0300

Today I’m setting up a simple nginx proxy, so I can store updates used by my many Linux systems. Most of them run a derivative of Debian, so this guide focuses mostly on caching apt repositories (Debian, Ubuntu, Proxmox, and more), but the same approach should work with any distro.

Install nginx

I’m using a Debian 12 (Bookworm) unprivilaged LXC container, but this is basic nginx which should be in every distro ever. I pointed my local DNS address deb.palnet.net (using a DNS override in Unbound on OPNsense, in my case) to the IP of the container.

So:

#Ensure system is up to date
apt update
apt upgrade -y
#Install nginx
apt install nginx -y
#Set it to autostart
systemctl enable nginx
#Remove defaults in sites-enabled
cd /etc/nginx
rm sites-enabled/*

Now we’re ready to setup our cache!

Nginx Config

Here’s my nginx config (/etc/nginx/sites-available/debcache.conf):

# Global Cache settings
proxy_cache_path /var/debcache/cache levels=2:2 keys_zone=generic:500m inactive=3650d max_size=1000g min_free=5g loader_files=1000 loader_sleep=50ms loader_threshold=300ms use_temp_path=off;
# Log with cache status
log_format cachelog '$remote_addr [$time_local] "$request" $status "$http_user_agent" "$upstream_cache_status"';

# URI paths to avoid cache
# These paths will change to indicate new release contents
# All other .deb files can be cached nearly indefinitely, as the
# version number is coded into the file name.
map $request_uri $nocache {
    ~InRelease 1;
    ~Release 1;
    ~Packages 1;
}

# Deb Server
server {
    # IF you need legacy IP, enable this one
    #listen 80 reuseport;

    # Everyone needs modern IP
    listen [::]:80 reuseport;


    #Log settings
    access_log /var/debcache/access.log cachelog;
    error_log /var/debcache/error.log;


    # Cache Location
    slice 1m;
    proxy_cache generic;
    proxy_ignore_headers Expires Cache-Control;
    proxy_cache_valid 200 206 3650d;
    proxy_cache_valid 301 302 0;
    proxy_set_header  Range $slice_range;
    proxy_cache_lock on;
    proxy_cache_lock_age 2m;
    proxy_cache_lock_timeout 1h;
    proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
    proxy_cache_revalidate on;
    #Nocache for those entries
    proxy_cache_bypass $nocache;
    proxy_no_cache $nocache;
    # 1G max file
    proxy_max_temp_file_size 1024m;
    # Cache key
    proxy_cache_key      $http_host$uri$slice_range;
    # Upstream Configuration
    proxy_next_upstream error timeout http_404;
    # Cache status
    add_header X-Cache-Status $upstream_cache_status;
    proxy_redirect off;
    proxy_ignore_client_abort on;
    # Upstream request headers
    proxy_ssl_server_name on;

    # Redirect Locations
    # Must include trailing slash!

    # Debian
    location /debian/ {
        proxy_pass http://deb.debian.org/debian/;
        proxy_set_header Host "deb.debian.org";
    }
    location /debsec/ {
        proxy_pass http://deb.debian.org/debian-security/;
        proxy_set_header Host "deb.debian.org";
    }

    # Raspberry Pi
    location /raspi/ {
        proxy_pass http://archive.raspberrypi.com/debian/;
        proxy_set_header Host "archive.raspberrypi.com";
    }

    # Ubuntu
    location /ubuntu/ {
        proxy_pass http://us.archive.ubuntu.com/ubuntu/;
        proxy_set_header Host "us.archive.ubuntu.com";
    }
    location /ubusec/ {
        proxy_pass http://security.ubuntu.com/ubuntu/;
        proxy_set_header Host "security.ubuntu.com";
    }

    # Kali
    location /kali/ {
        proxy_pass http://http.kali.org/kali/;
        proxy_set_header Host "http.kali.org";
    }

    # Proxmox (non-enterprise)
    location /proxmox/ {
        proxy_pass http://download.proxmox.com/debian/;
        proxy_set_header Host "download.proxmox.com";
    }

    # Caddy Server
    location /caddy/ {
        proxy_pass https://dl.cloudsmith.io/public/caddy/stable/deb/debian/;
        proxy_set_header Host "dl.cloudsmith.io";
    }

    # Nodesource NodeJS
    location /node/ {
        proxy_pass http://deb.nodesource.com/;
        proxy_set_header Host "deb.nodesource.com";
    }

    # Authelia
    location /authelia/ {
        proxy_pass https://apt.authelia.com/stable/debian/debian/;
        proxy_set_header Host "apt.authelia.com";
    }

    # Influx
    location /influx/ {
        proxy_pass http://repos.influxdata.com/debian/;
        proxy_set_header Host "repos.influxdata.com";
    }

    # Stats endpoint
    location = /nginx_status {
        stub_status;
    }

    # Static Files (conversion scripts)
    root /var/debcache/static/;
    autoindex on;
}

Once you are done creating the file, create a link to it in sites-enabled so it is loaded by nginx:

cd /etc/nginx
ln -s ../sites-available/debcache.conf sites-enabled/debcache.conf

And restart nginx (systemctl restart nginx)

Replacement Script

I went through all of my Debian systems (to see which repos they are using), and wrote a simple script which uses sed to replace everything to point to my cache. I chose this approach instead of setting up apt to use a proxy, since this replaces https requests with http requests (to the cache - the cache -> upstream is still using HTTPS). This avoids complications in having apt trust a self-signed proxy cert. The lack of TLS is not significant here since deb packages are GPG signed.

Anyway, here’s the script (put it in /var/debcache/static/rewrite.sh)

#!/bin/bash

# My mirror location
export mirror=http://deb.palnet.net

# Load OS release info into variables
. /etc/os-release

#Function to rewrite a single file
rewrite() {
    echo Rewriting $1
    #Debian repos
    sed -i s#http://deb.debian.org/debian-security#$mirror/debsec#g $1
    sed -i s#http://deb.debian.org/debian#$mirror/debian#g $1
    sed -i s#http://ftp.us.debian.org/debian#$mirror/debian#g $1
    sed -i s#http://security.debian.org#$mirror/debsec#g $1
    #Raspberry Pi repos
    sed -i s#http://archive.raspberrypi.com/debian#$mirror/raspi#g $1
    #Ubuntu repos
    sed -i s#http://us.archive.ubuntu.com/ubuntu#$mirror/ubuntu#g $1
    sed -i s#http://archive.ubuntu.com/ubuntu#$mirror/ubuntu#g $1
    sed -i s#http://security.ubuntu.com/ubuntu#$mirror/ubusec#g $1
    #Kali repos
    sed -i s#http://http.kali.org/kali#$mirror/kali#g $1
    #Proxmox repos (this catches all of them)
    sed -i s#http://download.proxmox.com/debian#$mirror/proxmox#g $1
    #Caddy server
    sed -i s#https://dl.cloudsmith.io/public/caddy/stable/deb/debian#$mirror/caddy#g $1
    #NodeJS from Nodesource
    sed -i s#https://deb.nodesource.com#$mirror/node#g $1
    #Authelia
    sed -i s#https://apt.authelia.com/stable/debian/debian#$mirror/authelia#g $1
    #Influxdata
    sed -i s#https://repos.influxdata.com/debian#$mirror/influx#g $1
}

#Rewrite sources.list itself
rewrite '/etc/apt/sources.list'

#Rewrite everything in sources.list.d
export -f rewrite
find '/etc/apt/sources.list.d/' -name "*.list" -type f -exec bash -c 'rewrite "$0"' {} \;

# If this system is using the 'new style' deb mirrors file (/etc/apt/sources.list.d/debian.sources)
# Replace it with equivalent sources.list entries
if test -f "/etc/apt/sources.list.d/debian.sources"; then
    echo "Rewriting /etc/apt/sources.list.d/debian.sources to sources.list format"
    echo "deb $mirror/debian $VERSION_CODENAME main contrib" >> /etc/apt/sources.list
    echo "deb-src $mirror/debian $VERSION_CODENAME main contrib" >> /etc/apt/sources.list
    echo "deb $mirror/debian $VERSION_CODENAME-updates main contrib" >> /etc/apt/sources.list
    echo "deb-src $mirror/debian $VERSION_CODENAME-updates main contrib" >> /etc/apt/sources.list
    echo "deb $mirror/debsec $VERSION_CODENAME-security main contrib" >> /etc/apt/sources.list
    echo "deb-src $mirror/debsec $VERSION_CODENAME-security main contrib" >> /etc/apt/sources.list
    rm /etc/apt/sources.list.d/debian.sources
fi

# Same, but for Ubuntu
if test -f "/etc/apt/sources.list.d/ubuntu.sources"; then
    echo "Rewriting /etc/apt/sources.list.d/ubuntu.sources to sources.list format"
    echo "deb $mirror/ubuntu $VERSION_CODENAME main universe restricted multiverse" >> /etc/apt/sources.list
    echo "deb-src $mirror/ubuntu $VERSION_CODENAME main universe restricted multiverse" >> /etc/apt/sources.list
    echo "deb $mirror/ubuntu $VERSION_CODENAME-updates main universe restricted multiverse" >> /etc/apt/sources.list
    echo "deb-src $mirror/ubuntu $VERSION_CODENAME-updates main universe restricted multiverse" >> /etc/apt/sources.list
    echo "deb $mirror/ubuntu $VERSION_CODENAME-backports main universe restricted multiverse" >> /etc/apt/sources.list
    echo "deb-src $mirror/ubuntu $VERSION_CODENAME-backports main universe restricted multiverse" >> /etc/apt/sources.list
    echo "deb $mirror/ubusec $VERSION_CODENAME-security main universe restricted multiverse" >> /etc/apt/sources.list
    echo "deb-src $mirror/ubusec $VERSION_CODENAME-security main universe restricted multiverse" >> /etc/apt/sources.list
    rm /etc/apt/sources.list.d/ubuntu.sources
fi

Then, on any system you have, you can run curl deb.palnet.net/rewrite.sh | bash and it will replace all of the deb references it finds. If you don’t have curl, you can do the same thing with wget -q -O - deb.palnet.net/rewrite.sh | bash (you may need to sudo bash depending on your current privilege)

Upgrade Distribution

Now that I have this place to put deb upgrade scripts, I wrote another one to replace bookworm with trixie to help upgrade deb systems easily.

This one goes in /var/debcache/static/bookworm2trixie.sh

#!/bin/bash
#Only do sources.list
sed -i 's/bookworm/trixie/g' /etc/apt/sources.list

and you can again get to it at deb.palnet.net/bookworm2trixie.sh and pipe it into bash.

Monitoring

You can access the logs of your cache from the debcache system, to see if you are getting hits or misses. The first system to update will miss everything, but after that the hit rate should be decent.

tail -f /var/debcache/access.log
#Or for the error file
tail -f /var/debcache/error.log

Securely Expose your Homelab Services with Mutual TLS

Fri, 22 Nov 2024 01:00:05 -0300

Today I’m diving into Mutual TLS to securely expose my homelab services! TLS is already ubiquitous in the modern era, providing strong symmetric encryption, perfect forward secrecy, and a public chain of trust to authenticate the server. But, it also has a lesser known ability to authenticate the client. By creating our own certificate authority to issue certs to clients, we can securely authenticate them to the server, preventing other users from even hitting our web app and probing it for vulnerabilities.

This is a simpler solution than using a VPN to ’expose’ your services, as long as the app is already relying on TLS (which includes more protocols than just HTTPS). There’s less user friction in installing a .p12 cert than setting up a VPN client, which could be important if you are sharing your services with friends and family.

Video

Create a Simple Root CA

This is section will setup a small 2-layer (root + leaf) authority, instead of the usual 3-layer (root + intermediate + leaf), for testing only. You should probably use the Smallstep CA for anything scalable.

I’m also using an ECDSA key chain, more to see how the process compares to RSA. So, the root will use scep384. I chose scep384 due to its arguably overkill security level, roughly equivalent to ~7000 bit RSA.

#Generate private key using scep384:
openssl ecparam -name secp384r1 -genkey -out root.key

#Sign the root certificate
#Pathlen:0 means there can be only one more cert below this CA (no more CAs)
#Make sure you update the subj name with your own names
#C=US is also the country, it's optional
#O= is the organization, also optional
#CN= is the Common Name and it's required
#I also set validity to 4 years, make sure you watch for expiration (manually)
openssl req -new -key root.key -x509 -nodes -days 1461 -out root.pem -subj "/C=US/O=apalrd.net/CN=test" -addext "basicConstraints=critical,CA:TRUE,pathlen:0"

#Now you can view it (for fun)
openssl x509 -in root.pem -text -noout

Sign a User Cert using the Root CA

Now we can use openssl to generate and sign a user cert using the root key. I chose scep256 for this since it has a smaller ‘blast radius’ than the root and scep256 still provides roughly equivalent security to rsa 3072.

#Generate scep256 key for this client
openssl ecparam -name prime256v1 -genkey -out adventure@apalrd.net.key

#Generate a CSR (certificate signing request) for my new key
#again, C and O are optional, CN is the Common Name of the cert
openssl req -new -key adventure@apalrd.net.key -out adventure@apalrd.net.csr -subj "/C=US/O=apalrd.net/CN=adventure@apalrd.net" -addext "extendedKeyUsage = clientAuth"

#Sign the CSR using the root
#Sign it allowing for server and client auth as the key usage
openssl x509 -req -in adventure@apalrd.net.csr -CA root.pem -CAkey root.key -CAcreateserial -out adventure@apalrd.net.crt -days 365 -sha256 -copy_extensions=copyall

#Now you can view it (for fun)
openssl x509 -in adventure@apalrd.net.crt -text -noout

#Now let's package it into a P12 archive so you can send it to your favorite client device
#You *must* enter a password here or some OSes will not accept the P12
#The password just encrypts the P12 file itself
openssl pkcs12 -export -out adventure@apalrd.net.p12 -in adventure@apalrd.net.crt -inkey adventure@apalrd.net.key

Important note, I know it’s tempting to try to use Bernsein’s curves (ed25519 especially), but they aren’t approved by the CA + Browser Forum for use in public CAs, and aren’t implemented in any major web browser. So, I’ve chosen scep256 (which OpenSSL confusingly calls prime256) and scep384 for my CA. Of course, RSA is always an option as well.

Sign a User Cert using Smallstep

If you’ve already setup a Smallstep CA from my previous TLS videos, you can use that instead of OpenSSL:

#Sign cert (run as root on the CA)
#laptop.crt/laptop.key are the key files
#I signed this one for a long ass time
step ca certificate adventure@apalrd.net laptop.crt laptop.key --not-after=2160h 

#Bundle into p12 and include intermediate cert we are using
#The P12 file can be imported into any OS
step certificate p12 laptop.p12 laptop.crt laptop.key --ca /etc/step/certs/intermediate_ca.crt

Setup Caddy for Client Auth

Here’s the Caddyfile I used (Debian packs a start page with Caddy):

# The Caddyfile is an easy way to configure your Caddy web server.

#TLS email (global section)
#Please stop using my email for your Let's Encrypt certs
#I am sick of getting your renewal notices
{
    email mail@example.net
}

#Test website
test1.apalrd.net {
    #Caddy example lives here
    root * /usr/share/caddy
    file_server

    #mTLS verify client
    tls {
        client_auth {
            #Default is none
            #there are other options here if you want it to be optional
            #i.e. to bypass a signin page when using mTLS
            mode require_and_verify
            trust_pool file {
                #Can be specified multiple times for multiple roots
                pem_file /etc/caddy/root.pem
            }
        }
    }
}

Alternatively, you can make a snippet in the config to make it easier - this Caddyfile has the exact same behavior:

# The Caddyfile is an easy way to configure your Caddy web server.

#TLS email (global section)
#Please stop using my email for your Let's Encrypt certs
#I am sick of getting your renewal notices
{
    email mail@example.net
}

#Snippet for TLS client auth
(tls_client) {
    #mTLS verify client
    tls {
        client_auth {
            #Default is none
            #there are other options here if you want it to be optional
            #i.e. to bypass a signin page when using mTLS
            mode require_and_verify
            trust_pool file {
                #Can be specified multiple times for multiple roots
                pem_file /etc/caddy/root.pem
            }
        }
    }
}

#Test website
test1.apalrd.net {
    #Caddy example lives here
    root * /usr/share/caddy
    file_server

    #Reuse the snipper from earlier
    import tls_client
}

Setup Nginx for Client Auth

Here’s a full nginx configuration (/etc/nginx/nginx.conf) with client auth. The default Debian config is quite .. verbose .. so I’ve cut out a lot of things which I didn’t need, but you might have needed them. Anyway, it’s framework. Nginx on Debian also subscribes to the whole sites-available and sites-enabled thing, which I find unnecessary.

user www-data;
worker_processes auto;
worker_cpu_affinity auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 768;
    # multi_accept on;
}

http {

    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    types_hash_max_size 2048;
    server_tokens off; # Recommended practice is to turn this off

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # SSL Settings
    ##

    ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3 (POODLE), TLS 1.0, 1.1
    ssl_prefer_server_ciphers off; # Don't force server cipher order.

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;

    ##
    # Gzip Settings
    ##

    gzip on;


    ##
    # Virtual Host Configs
    ##

    # Redirect all HTTP to HTTPS
    server {
        listen 80 default_server;
        server_name _;
        return 301 https://$host$request_uri;
    }

    # Default (TLS) server configuration
    #
    server {
        # TLS (Server) configuration
        listen 443 ssl default_server;
        listen [::]:443 ssl default_server;

        # This is where Certbot puts them in standalone mode
        ssl_certificate     /etc/letsencrypt/live/test-nginx.apalrd.net/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/test-nginx.apalrd.net/privkey.pem;

        # TLS (Client) configuration
        # Possible values are 'on' or 'optional' (or 'off', the default)
        # If you use 'on', nginx will return 400 Bad Request if it fails
        # If you like 400, then use 'on'
        # If you'd rather drop, then use 'optional' and drop below (return 444)
        ssl_verify_client optional;
        # Path to the client CA
        ssl_client_certificate /etc/nginx/root.pem;
        # Number of steps we allow from the root
        ssl_verify_depth 2;

        # Drop connections which fail verification (optional)
        if ($ssl_client_verify != "SUCCESS") { return 444; }

        # Example website
        root /var/www/html;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        server_name _;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }
    }
}

Single Root IO Virtualization in Proxmox (for NICs)

Tue, 05 Nov 2024 01:00:05 -0300

In this episode, I’m playing with Single Root I/O Virtualization (SR-IOV) in Proxmox Virtual Environment (PVE). I’ve heard ruomors that it will be anything from a minor to major improvement in IO performance for my VMs, so I wanted to do some testing on my own system to be sure.

Please don’t take my results as final, I’m not comfortable saying that I’ve removed all of the background tasks and load from the measurements. My system is also relatively underpowered, so this may show as a thread-bound performance limit on some of these drivers where your system may not have this issue.

Video

Enable SR-IOV

Prerequisites:

Your card supports SR-IOV
You have enabled SR-IOV in the card firmware if required (See MSTFLINT below for MLX5)
You have enabled IOMMU and PCIe ACS in your BIOS/UEFI firmware
You have enabled IOMMU in the kernel (intel_iommu=on or amd_iommu=on) and functioning correctly

Once all of that is done, to enable SR-IOV, you just need to set a particular sysctl which corresponds to your network device. The sysctl is the same for all drivers (Intel/Mellanox). This is NOT persistent across reboots. I didn’t bother scripting a way to automate this for my tests, and since I’m not going to continue to use SR-IOV, I also didn’t write it here. Anyway:

echo 32 > /sys/class/net/enpXXsYfZ/device/sriov_num_vfs
#Obviously change the enpXX with the full name of your own device

#f0 is the physical function of the card and should already exist
#f1+ are the virtual functions

#At this point it will possibly take awhile and if you are unlucky
#and have system IOMMU / ACS issues it could hang here for 60+ seconds

LXC

Here’s the LXC configuration I used. Proxmox renames the link on starting/stopping an LXC container, and sometimes it doesn’t rename back correctly. Be warned.

lxc.apparmor.profile: unconfined
lxc.net.2.type: phys
lxc.net.2.link: enp1s0f0v3
lxc.net.2.flags: up
lxc.net.2.ipv6.address: fc69::4/64

Mellanox MSTFLINT

If you are using a Mellanox (now Nvidia) card, you may need to configure the firmware to enable SR-IOV and also set the number of VFs. This is persistent to the card, so if you bought your card used it may already be configured this way. In any case, here’s the commands:

#apt update if you haven't recently
apt install mstflint

#Find the PCI ID of the card
lspci
#My card was 01:00.0 and 01:00.1

#Query all of the info, if you are qurious what the card is configured as
mstconfig -d 01:00.0

#Configure the VFs and enable SR_IOV
mstconfig -d 01:00.0 set NUM_OF_VFS=32 SRIOV_EN=true

#Reboot

Using NETCONSOLE to debug Linux (and Proxmox) Kernel Panics

Fri, 05 Jul 2024 01:00:05 -0300

In this post (and video) I’m going to setup Netconsole, so you can capture kernel panics and logs on headless systems. I know some of you are doing wild things with graphics drivers and passthrough, so hopefully this helps you debug them.

Enable Now

This option enables the module immediately, so you can use it before you do dangerous things. You Simply rebooting clears the setting, so you won’t continue to spam your kernel messages on the local network.

modprobe netconsole netconsole=@/eth0,@<destination_IP_address>/

If you want the full syntax of the argument, you can find it here. I’m leaving a lot of things as defaults, so it will use the broadcast MAC and UDP port 6666.

And to disable it without rebooting:

rmmod netconsole

Enable On Boot

This will take effect on the next boot. Similar arguments to above, except we need to add this to the kernel commandline. Also, since the interface won’t have this addresses early in boot the process, it’s easiest if we use IPv6 link-local addresses. The kernel will generate one from the MAC address. The other option if you want to use legacy IPv4 is to set a source IP address in the kernel command line, this does not have to be the same address you use in the system.

On Debian systems and some Proxmox systems, the file we need to edit is /etc/default/grub. On Proxox systems using UEFI boot but not secure boot, the file we need to edit is /etc/kernel/cmdline. If you don’t have an /etc/kernel/cmdline, then you need to edit /etc/default/grub. Anyway, add this to the command line:

netconsole=@/eth0,@<destination_IPv6_LL_address>/

Then we need to run either update-grub (if you used /etc/default/grub) or proxmox-boot-tool refresh (if you used /etc/kernel/cmdline, and possibly on any Proxmox systems, it doesn’t hurt)

When you reboot, you can check dmesg | grep netcon to see if it started or if there were errors. If your Ethernet devices goes by multiple names in the boot process, try to dmesg | grep eth0 and see what other names it goes by, and try those if it fails.

Receive Netconsole with Netcat

Now, we need to setup Netcat to listen to that broadcast. It’s sending it to the broadcast MAC address (all F’s), but to a unicast IP address, so anyone on the network will see this traffic with Wireshark. But netcat is easier. The default netcat version with Debian doesn’t support IPv6, but you can apt install ncat for a better version, and it will install itself as the nc command.

#Call netcat to listen
#-l = listen
#-u = udp
#-p 6666 = the default port for netconsole 
nc -l -u -p 6666

This can run as long as you want to monitor the netconsole output.

Receive Netconsole with a simple systemd unit

This is a super simple systemd service which will run netcat and save it to the systemd journal. Useful for cases when your crash is intermittent. You can run this on any machine on the network (of course you need to have netconsole set for this IP as well). Save this as /etc/systemd/system/netconsole-log.service:

[Service]
ExecStart=/usr/bin/nc -l -u -p 6666

[Install]
WantedBy=multi-user.target

Then start it:

systemctl daemon-reload
systemctl enable --now netconsole-log

And view the logs:

journalctl -xeu netconsole-log

Panic

If you want to panic the system, then you can send the sysreq ‘c’ (as root):

echo c > /proc/sysrq-trigger

Here’s a list of all the sysreqs you can trigger for fun.

Imaging a bare-metal system using Proxmox Backup Client

Sat, 29 Jun 2024 01:00:05 -0300

Since I like to image systems I’m testing, here’s the process I use to capture the boot drive of a bare metal system in Proxmox Backup Server. I start by booting into a Debian Live image (the small one without a GUI).

Next, I install Proxmox Backup Client from apt packages:

#Run as root
sudo bash
#Add repo
wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg
echo "deb http://download.proxmox.com/debian/pbs-client bookworm main" > /etc/apt/sources.list.d/pbs-client.list

#Install PBS
apt update
apt install proxmox-backup-client -y

After that, I set the environment variables for the backup client (I have a doc I can copy/paste from into my USB KVM):

#Environment variables - these will be your Proxmox Backup user and stuff
export PBS_REPOSITORY=user@realm@dns_name.lan:datastore
export PBS_PASSWORD=api_key
export PBS_FINGERPRINT="fingerprint"

And finaly, run the image itself. The ‘backup-id’ comes across as ‘host’ in PBS, so I name it only once per physical system, and I can add notes later in the PBS UI as to which backup of the dxp4800 this was. Syntax is .img:/dev/disk. I always use root.img for the root disk, and this captures the entire disk, not partitions.

#Backup command
#root.img is the name of the image (stored in the backup), you can add this multiple times for multiple disks
#/dev/nvme0n1 is the disk that will be copied into root.img
#backup-id is the 'host name' as stored in Proxmox Backup Server
proxmox-backup-client backup root.img:/dev/nvme0n1 --backup-id "ugreen-dxp4800"

Restore works similarly, except we do restore instead of backup:

#Restore command
#Copy the backup id from the PBS UI
#root.img is the name of the image from the backup
#/dev/nvme0n1 is the disk we are restoring onto
proxmox-backup-client restore host/hostname/2024-06-30T12:11:46Z root.img - | sudo dd of=/dev/nvme0n1 status=progress bs=1M

Rebuilding Proxmox Backup Server from Backups

Thu, 27 Jun 2024 01:00:05 -0300

So, while I was on vacation, my Proxmox Backup Server boot drive failed! No problem, I take backups of the server itself …. onto the backup server. So in this video, I’m going to start from a clean PBS install, mount my intact backup pool, and then restore the PBS configs out of the backup.

Video

Restoring the Datastore

First, we need to mount the data disks. In my case, they are a zfs pool named backup, so we use zpool import -f backup to import the pool. Once the pool is imported, the directories should show up (you can find them with zfs list if you forgot where they are).

If you’re using non-zfs, you’ll probably have to write a line in /etc/fstab and mount them the traditional way.

Now, to get PBS to find the dataset on a brand new PBS system, we need to create a datastore.cfg which points to the directory of the datastore. That file won’t exist if we have no datastores, so create it (/etc/proxmox-backup/datastore.cfg) with the following contents:

datastore: <name>
    path /path/to/your/backups

Refresh in the GUI and it should appear, and you should be able to access your backups at this point (although user accounts and such aren’t back yet).

Restore the Backup

Since we are running on the PBS host, we have access to PBS client (the binary is installed) and we can just point it at localhost. We won’t need the fingerprint to trust localhost, but we will need a username and password, and the long full name of the backup (the one with the date in it). Here’s the command I used to mount the backup temporarily:

#Create a directory to mount at (this can't exist already)
mkdir -p /mnt/backup
#Call the backup client
proxmox-backup-client mount host/bigstor/2024-05-04T04:00:58Z root.pxar /mnt/backup --repository localhost:backup

At that point, we can rsync the contents of /etc/proxmox-backup from the backup into its old home. It’s critical that you use rsync and not cp to do this, since cp will not retain file permissions!

#Copy data from backup
rsync -r -v /mnt/backup/etc/proxmox-backup/ /etc/proxmox-backup/

At this point I would unmount the backup (unmount /mnt/backup) and restart the services. Make sure they come up fine. Also, delete pre-existing temporary files that were in the backup.

#Delete temp files
rm -rf /etc/proxmox-backup/.*
#Restart proxmox backup
systemctl restart proxmox-backup
systemctl restart proxmox-backup-proxy
#If you need to look at logs:
journalctl -xeu proxmox-backup
journalctl -xeu proxmox-backup-proxy

Now that this is done, I copied the remaining files on the system out of the backup (my autoshutdown scripts and other systemd units). You may have authentication errors with proxmox-backup-client after restoring since sessions will be terminated (we deleted those session temp files), so you will need to run proxmox-backup-client logout to clear the client’s session cache and start over.

Auto-Backup Scripts

Here’s the script I use for autobackup. It’s a systemd service unit + systemd timer. I use this script on a lot of my Debian systems which are not part of PVE.

The service (/etc/systemd/system/autobackup.service):

[Unit]
Description=Run Backup to Proxmox Backup Server
After=network-online.target

[Service]
#Technically if we are on localhost we can skip the fingerprint
Environment=PBS_FINGERPRINT=your_fingerptint
#Your API key here
Environment=PBS_PASSWORD=api_key_here
#Use your user @ realm ! api key name @ hostname : repository
Environment=PBS_REPOSITORY=user@pbs!apikey@localhost:backup
Type=oneshot
#Backup the root fs. This will not backup any other mount points, only the mount point which mounts /
#You can add more pxars if you want more mount points
ExecStart=proxmox-backup-client backup root.pxar:/

[Install]
WantedBy=default.target

And the timer (/etc/systemd/system/autobackup.timer):

[Unit]
Description=Backup System Daily
RefuseManualStart=no
RefuseManualStop=no

[Timer]
#Run 540 seconds after boot for the first time
#Since I shutdown every night, this will be every day
OnBootSec=540
#Run at midnight UTC
#If you don't shutdown every night, uncomment this instead
#OnCalendar=*-*-* 00:00:00
Unit=autobackup.service

[Install]
WantedBy=timers.target

And finally, we enable the timer, not the service:

systemctl daemon-reload
systemctl enable --now autobackup.timer

All done! You can test it by running systemctl start autobackup which will run a backup now.

Simple Self-Hosted Security with Authelia

Wed, 19 Jun 2024 07:00:05 -0300

In this video, I’m setting up Authelia. It’s a very lightweight authentication service, which can be used to provide authentication to services which don’t natively support any form of authentication. I think this is a great choice for small scale homelab environments, as it’s simple to run and administer.

Video

Authelia

I installed Authelia on an LXC container (Debian 12), and set it up with a dns name / AAAA record in public dns, and all the jazz required for normal HTTPS access. And of course did updates (apt update && apt full-upgrade -y) before starting.

Base Install

Now, we are going to install Auehtlia from their apt repository to make updates smooth in the future. The Debian repos tend to lag behind.

#Install required software for the apt repository
apt update
apt install -y apt-transport-https curl gnupg
#Download their signing key, add it with gpg / apt-key
curl https://apt.authelia.com/organization/signing.asc | apt-key add -
#Add their repo to sources.list.d
echo "deb https://apt.authelia.com/stable/debian/debian/ all main" > /etc/apt/sources.list.d/authelia-stable-debian.list
#Install authelia
apt update
apt install -y authelia

If you want to run with high availability (since authentication is critical), follow the guide here from authelia. In particular, you will need to setup the required services (Redis and Postgres) in HA themselves. I am not doing an HA setup, so I am using local databases (SQLite) for simplicity.

Allow Low-Numbered Ports

To get traffic to Authelia (port 9091) from the real TLS port (443), I found it is easiest to just port forward than to modify the service to add the appropriate capabilities. So, I wrote a port-forward service (/etc/systemd/system/authportfwd.service):

[Unit]
Description=Authelia Port Forwarding Service
After=network.target

[Service]
Type=oneshot
#Of course you can delete the iptabes if you are in the modern era exclusively
ExecStartPost=iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 9091
ExecStopPre=iptables -t nat -D PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 9091
ExecStartPost=ip6tables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 9091
ExecStopPre=ip6tables -t nat -D PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 9091
ExecStart=/bin/true

[Install]
WantedBy=multi-user.target

Of course, we also need to make sure iptables is installed, then we can enable the service:

#Install iptables
apt install iptables
#Enable it
systemctl daemon-reload
systemctl enable --now authportfwd

Configuration

Now we need to write our configuration.yml. It’s stored in /etc/authelia/configuration.yml. The package should have installed a default, so now we just need to modify it. The Getting Started Guide has great tips here. I configured it to use TLS with the default Certbot configuration, see below for Certbot itself.

Here’s what I used (this is a diff):

--- /etc/authelia/configuration.yml.default     2024-05-27 15:27:17.203817464 +0000
+++ /etc/authelia/configuration.yml     2024-05-27 21:42:31.879622715 +0000
@@ -21,7 +21,8 @@
 # certificates_directory: '/config/certificates/'
 
 ## The theme to display: light, dark, grey, auto.
-# theme: 'light'
+#All hail dark mode
+theme: 'dark'
 
 ## Set the default 2FA method for new users and for when a user has a preferred method configured that has been
 ## disabled. This setting must be a method that is enabled.
@@ -31,7 +32,7 @@
 ##
 ## Server Configuration
 ##
-# server:
+server:
   ## The address for the Main server to listen on in the address common syntax.
   ## Formats:
   ##  - [<scheme>://]<hostname>[:<port>][/<path>]
@@ -50,12 +53,12 @@
   # disable_healthcheck: false
 
   ## Authelia by default doesn't accept TLS communication on the server port. This section overrides this behaviour.
-  # tls:
+  tls:
     ## The path to the DER base64/PEM format private key.
-    # key: ''
+    key: '/etc/letsencrypt/live/auth.apalrd.net/privkey.pem'
 
     ## The path to the DER base64/PEM format public certificate.
-    # certificate: ''
+    certificate: '/etc/letsencrypt/live/auth.apalrd.net/fullchain.pem'
 
     ## The list of certificates for client authentication.
     # client_certificates: []
@@ -94,7 +97,7 @@
 
   ## Server Endpoints configuration.
   ## This section is considered advanced and it SHOULD NOT be configured unless you've read the relevant documentation.
-  # endpoints:
+  endpoints:
     ## Enables the pprof endpoint.
     # enable_pprof: false
 
@@ -102,10 +105,10 @@
     # enable_expvars: false
 
     ## Configure the authz endpoints.
-    # authz:
-      # forward-auth:
-        # implementation: 'ForwardAuth'
-        # authn_strategies: []
+    authz:
+      forward-auth:
+        implementation: 'ForwardAuth'
+        authn_strategies: []
       # ext-authz:
         # implementation: 'ExtAuthz'
         # authn_strategies: []
@@ -183,7 +186,7 @@
   disable: false
 
   ## The issuer name displayed in the Authenticator application of your choice.
-  # issuer: 'authelia.com'
+  issuer: 'apalrd.net'
 
   ## The TOTP algorithm to use.
   ## It is CRITICAL you read the documentation before changing this option:
@@ -274,7 +277,7 @@
     # jwt_algorithm: 'HS256'
 
     ## The secret key used to sign and verify the JWT.
-    jwt_secret: 'a_very_important_secret'
+    jwt_secret: 'NearbyOverhangParadoxSulphuric'
 
   ## Elevated Session flows. Adjusts the flow which require elevated sessions for example managing credentials, adding,
   ## removing, etc.
@@ -506,8 +509,8 @@
   ##
   ## Important: Kubernetes (or HA) users must read https://www.authelia.com/t/statelessness
   ##
-  # file:
-    # path: '/config/users_database.yml'
+  file:
+    path: '/etc/authelia/users_database.yml'
     # watch: false
     # search:
       # email: false
@@ -621,7 +624,9 @@
   ## Default policy can either be 'bypass', 'one_factor', 'two_factor' or 'deny'. It is the policy applied to any
   ## resource if there is no policy to be applied to the user.
   default_policy: 'deny'
-
+  rules:
+    - domain: '*.apalrd.net'
+      policy: 'one_factor'
   # networks:
     # - name: 'internal'
     #   networks:
@@ -718,14 +723,14 @@
 
   ## Cookies configures the list of allowed cookie domains for sessions to be created on.
   ## Undefined values will default to the values below.
-  # cookies:
-  #   -
+  cookies:
+    -
       ## The name of the session cookie.
-      # name: 'authelia_session'
+      name: 'authelia_session'
 
       ## The domain to protect.
       ## Note: the Authelia portal must also be in that domain.
-      # domain: 'example.com'
+      domain: 'apalrd.net'
 
       ## Required. The fully qualified URI of the portal to redirect users to on proxies that support redirections.
       ## Rules:
@@ -733,7 +738,7 @@
       ##   - The above 'domain' option MUST either:
       ##      - Match the host portion of this URI.
       ##      - Match the suffix of the host portion when prefixed with '.'.
-      # authelia_url: 'https://auth.example.com'
+      authelia_url: 'https://auth.apalrd.net'
 
       ## Optional. The fully qualified URI used as the redirection location if the portal is accessed directly. Not
       ## configuring this option disables the automatic redirection behaviour.
@@ -881,9 +886,9 @@
 ##
 ## This mechanism prevents attackers from brute forcing the first factor. It bans the user if too many attempts are made
 ## in a short period of time.
-# regulation:
+regulation:
   ## The number of failed login attempts before user is banned. Set it to 0 to disable regulation.
-  # max_retries: 3
+  max_retries: 3
 
   ## The time range during which the user can attempt login before being banned in the duration common syntax. The user
   ## is banned if the authentication failed 'max_retries' times in a 'find_time' seconds window.
@@ -896,11 +901,11 @@
 ## Storage Provider Configuration
 ##
 ## The available providers are: `local`, `mysql`, `postgres`. You must use one and only one of these providers.
-# storage:
+storage:
   ## The encryption key that is used to encrypt sensitive information in the database. Must be a string with a minimum
   ## length of 20. Please see the docs if you configure this with an undesirable key and need to change it, you MUST use
   ## the CLI to change this in the database if you want to change it from a previously configured value.
-  # encryption_key: 'you_must_generate_a_random_string_of_more_than_twenty_chars_and_configure_this'
+  encryption_key: 'MoviePassingUntitledBacteriaStapleMatriarchAftermathChubby'
 
   ##
   ## Local (Storage Provider)
@@ -910,9 +915,9 @@
   ##
   ## Important: Kubernetes (or HA) users must read https://www.authelia.com/t/statelessness
   ##
-  # local:
+  local:
     ## Path to the SQLite3 Database.
-    # path: '/config/db.sqlite3'
+    path: '/etc/authelia/db.sqlite3'
 
   ##
   ## MySQL / MariaDB (Storage Provider)
@@ -1064,34 +1069,34 @@
   ##        (only works for unauthenticated connections)
   ##   - validate the SMTP server x509 certificate during the TLS handshake against the hosts trusted certificates
   ##     (configure in tls section)
-  # smtp:
+  smtp:
     ## The address of the SMTP server to connect to in the address common syntax.
-    # address: 'smtp://127.0.0.1:25'
+    address: 'smtp://YourMailProvider:587'
 
     ## The connection timeout in the duration common syntax.
     # timeout: '5 seconds'
 
     ## The username used for SMTP authentication.
-    # username: 'test'
+    username: 'adventure@apalrd.net'
 
     ## The password used for SMTP authentication.
     ## Can also be set using a secret: https://www.authelia.com/c/secrets
-    # password: 'password'
+    password: 'SuperSecret'
 
     ## The sender is used to is used for the MAIL FROM command and the FROM header.
     ## If this is not defined and the username is an email, we use the username as this value. This can either be just
     ## an email address or the RFC5322 'Name <email address>' format.
-    # sender: 'Authelia <admin@example.com>'
+    sender: 'Authelia <adventure@apalrd.net>'
 
     ## HELO/EHLO Identifier. Some SMTP Servers may reject the default of localhost.
-    # identifier: 'localhost'
+    identifier: 'auth.apalrd.net'
 
     ## Subject configuration of the emails sent. {title} is replaced by the text from the notifier.
     # subject: '[Authelia] {title}'
 
     ## This address is used during the startup check to verify the email configuration is correct.
     ## It's not important what it is except if your email server only allows local delivery.
-    # startup_check_address: 'test@authelia.com'
+    startup_check_address: 'adventure@apalrd.net'
 
     ## By default we require some form of TLS. This disables this check though is not advised.
     # disable_require_tls: false

Since it’s a diff and the file is absolutely massive, I suggest you first copy my diff into a file (say diff.patch), edit the things which are relevant to you, and then apply it with patch /etc/authelia/configuration.yml < diff.patch. You might need to apt install patch.

Initial User File

I’ve created an initial users file (/etc/authelia/users_database.yml), by taking my username and hashing a password. Here’s the file, and how to hash your own password. Of course, if you have SMTP configured, you can let users change their own passwords later. Delete the original and recreate.

users:
  apalrd:
    disabled: false
    displayname: 'apalrd'
    password: '$argon2id$v=19$m=65536,t=3,p=4$6D0CA3wVZ7AwvLG6VM7L3w$PqbNjQPhCxD9VoWz2hb+XBKcTOCC1KRb46FRNjg6l6I'
    #This corresponds to CorrectHorseBatteryStaple
    email: 'adventure@apalrd.net'
    groups:
      - 'admins'
  karen:
    disabled: false
    displayname: 'karen'
    password: '$argon2id$v=19$m=65536,t=3,p=4$Gb28FoQveOUloo8zpWWYgg$Kb3kJJU9mMznSIM2nDDVtuBvhiwF+SdQwoqoF3v80Rg'
    #This corresponds to karen

And to generate your own password, run:

authelia crypto hash generate argon2
#It will prompt you for a password

TLS Certificate

We need a TLS cert for Authelia. Authelia is more than capable of handling TLS on its own (without a reverse proxy in front of it), but it’s not capable of doing ACME protocol cert renewal, so I’m setting up Certbot to do it for us. We aren’t going to use port 80 for Authelia, so Certbot can have it.

# Install certbot
apt update
apt install certbot -y
#Rewnew the first time manually, this will configure renewal in the future, and restart Authelia when it does renew
certbot certonly --standalone -d auth.apalrd.net --deploy-hook "/usr/bin/systemctl restart authelia"
#Answer the questions, it will be automatic in the future

Testing

#Enable the services now
systemctl enable --now authelia
#Certbot is automatic

You should be able to navitate to your auth site, and it should let you login at this point.

Protected Service Example

Here I’m going to setup a service to be protected by Authelia, using Caddy as my reverse proxy of choice.

Caddy

I’m installing Caddy directly on the LXC container that runs the service (Frigate in this case). However, there’s nothing stopping you from adding this to the same Caddyfile / Caddy instance you used earlier (don’t duplicate the global section at the top, of course).

Here’s the Caddyfile I am using with Frigate:

# The Caddyfile is an easy way to configure your Caddy web server.

# Global section for automatic TLS via Let's Encrypt (requires an email)
{
        email "adventure@apalrd.net"
}

# Frigate server
corona.apalrd.net {
        #Authenticate via Authelia
        forward_auth https://auth.apalrd.net {
                uri /api/authz/forward-auth
                copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
                #Only include this if you are using a self-signed or staging cert at Authelia
                #transport http {
                        #tls_insecure_skip_verify
                #}
        }
        #Reverse proxy to localhost (Frigate only binds to v4 sockets)
        reverse_proxy [127.0.0.1]:5000
}

If you want to restrict access to the /config endpoint to users in the admins group, you can use the following bit of yaml in your Authelia config to filter on that URL. Remember that rules are processed in order! If you add a default-allow rule on your domain like I did above, then it will all ow everyone anyway, so delete that one and make more specific rules now that you know what you are doing.

  rules:
    # Allow admins with two factor auth to edit configs
    - domain: 'corona.apalrd.net'
      policy: 'two_factor'
      subject:
       - 'group:admins'
      resources:
       - '^/config([/?].*)?$'
    # Catch anyone else trying to get to config and deny them
    - domain: 'corona.apalrd.net'
      policy: 'deny'
      resources:
       - '^/config([/?].*)?$'
    # Catch any other resources on this domain and require one factor
    - domain: 'corona.apalrd.net'
      policy: 'one_factor'

I also found in this specific case that the javascript frontend rewrites the URL to /config instead of requesting that page, so it will initially load, but changing the configuration (the api endpoints) will still fail.

Relaying Traffic to Self-Host with CGNAT

Thu, 16 May 2024 01:00:05 -0300

This video started as the answer to a simple question - how can I self-host a service for my friends and family, behind cgnat, without requiring them to install any apps (like tunnels)? This video turned into a bunch of different ways to proxy IPv4 to IPv6, so you can receive IPv6 traffic natively and bring in legacy traffic from a VPS which does have public IPv4.

While I’m giving you a lot of different examples and methods here, you can mix and match a lot of them to fit your needs. For example, you can use snid for your TLS traffic (possibly listening on multiple ports for e.g. HTTPS and MQTTS), along with HAProxy or Tayga for the rest of your traffic. You can add in a Wireguard tunnel if you want, but since we are relaying to public IPv6s, it’s not needed.

Anyway, come along on this adventure!

Video

Comparison Table

Feature	Pure IPv6	Cloudflare Tunnel	SNID	HAProxy	Tayga	Wireguard
Free / Open Source Software	✔		✔	✔	✔	✔
Free (Hosting)	✔	?¹	?²	?²	?²	?²
Works with IPv6 Clients	✔	✔	✔	✔	✔	✔
Works with IPv4 Clients		✔	✔	✔	✔	✔
Uses Standard Port Numbers		✔	✔	?^3,5	?³	?³
Requires Server Name / DNS		✔⁴	✔	?⁵
Requires IPv6 Origin	✔		✔	✔	✔
Direct Route (via v6)	✔		✔	?^5,6	?⁶	?⁶
Direct Route (via v4)
Local Certificates	✔		✔	✔	✔	✔
Supports HTTP/HTTPS	✔	✔	✔	✔	✔	✔
Supports HTTP/3 (QUIC)	✔	✔			✔	✔
Supports TLS (non-HTTPS)	✔		✔	✔	✔	✔
Supports TCP (non-TLS)	✔			✔	✔	✔
Supports UDP	✔				✔	✔

Footnotes:

Cloudflare Tunnels are free but require sign-up with credit card. They also have an unknown bandwidth limit, as they are ‘designed for HTML / web sites and not streaming video’. Depending on your use case this may or may not be fine.
I’m aware that Oracle has a free tier but I don’t trust them at all. Also, to do NAT46, you need at least a routed /96 to your VPS, and not all cloud providers offer this (DigitalOcean in particular has awful IPv6 support). Try Hetzner of Mythic Beasts for good IPv6. ‘Good’ in this case means a /64 subnet routed (not on-link) to your server. You can tell if it’s routed if their gateway IP is within your /64 or not, routed will have a ‘far’ gateway (or link-local).
With these methods, since they don’t know the Server Name Identifier (SNI), you can only use the ‘standard’ port for one backend. This is a limitation of the DNS name not being transmitted over non-TLS protocols. You are of course free to use nonstandard ports for additional copies of the same service (i.e. multiple SSH servers).
You must host your domain with Cloudflare to use Cloudflare Tunnels for free
HAProxy supports both TLS SNI forwarding of TCP as well as pure TCP proxying
Direct Routing via v6 is only possible if you use the same port number on the v4 gateway and v6 origin. If you are using multiple services on the same port on different IPv6 addresses, you will run into port limitations on the v4 address. TLS connections do not suffer from this, as the server name is specified in the connections. If your software supports SRV records, you can use multiple SRV records to point to the v6 origin + port and the v4 gateway + port.

Remember, you can mix and match these on the same VPS! For example, use snid on port 443 for TLS, plus HAProxy or Tayga on other ports! And, while I’m showing how to set these up in a VPS to route around CGNAT, you can also set most of these up at your non-CGNAT router as an IPv4 to IPv6 transition mechanism.

Option 1 - SNID

This comes to us from AGWA’s Blog Post, and you can find the binaries on His Github. He explains the concept in great detail there.

Anyway, here’s the systemd service I wrote to go with it. I installed the static binary (Golang is awesome!) into /usr/local/bin.

Download the binary, then chmod +x it so it’s executable. Nothing else to install!

Not to replicate his docs here, but here’s what I used for each option:

listen tcp:0.0.0.0:443 - Listen only on IPv4, since IPv6 will go direct. If you want to listen on multiple ports, add this multiple times (i.e. 443 for HTTPS and 3389 for RDP). Backend connections use the same destination port of the listener they came in on.
mode nat46 - Encode the whole damn IPv4 space into our IPv6 prefix. The VPS has a /64, so this is fine.
nat46-prefix 2001:db8::4646:0:0 - I’m already using suffix ::1 for my SSH management, so use 4646:: for NAT46
backend-cidr 2001:db8::/48 - Put in your home IPv6 prefix here. Prevents you from becoming an open proxy on the internet, only allows connections that are within this range.

So create /etc/systemd/system/snid.service:

[Unit]
Description=SNI TLS Proxy Daemon
After=network-online.target

[Service]
ExecStart=/usr/local/bin/snid -listen tcp:0.0.0.0:443 -mode nat46 -nat46-prefix 2601:db8:6969:420:4646:: -backend-cidr 2001:db8::/48
Restart=always

[Install]
WantedBy=multi-user.target

We also need to add a route for our whole /96 prefix to lo, so the kernel will accept packets for it. I added this line to my /etc/network/interfaces on Debian:

# control-alias eth0
iface eth0 inet6 static
    #Was /64, changed to /128 so we don't send packets on-link for other addresses
    address 2601:db8:6969:420::1/128
    dns-nameservers 2601:fe::fe 2601:fe::9
    gateway fe80::1
    #Add local route to the translation prefix
    post-up ip route add local 2601:db8:6969:420:4646::/96 dev lo
    post-down ip route del local 2601:db8:6969:420:4646::/96 dev lo

Of course, when we are done, we need to apply both of these:

systemctl daemon-reload every time you change the snid.service file
systemctl enable --now snid to enable and start it
systemctl restart snid if you change the service file
systemctl status snid to see how it’s going
journalctl -xeu snid to see how it’s going in more detail
ifdown eth0 && ifup eth0 to reload /etc/network/interfaces (or just reboot)

Option 2 - HAProxy

I’ve already made a video on this topic - you can find it here. That page goes through a lot of the theory if you are curious.

tl;dr install it with apt update && apt install haproxy -y

Then edit the config file /etc/haproxy/haproxy.cfg. Here are some example configs for you. I left the Debian defaults unmodified and added my sections at the end.

HTTP Redirect to HTTPS (Direct response to client)

# Listen on port 80, layer 7 (HTTP)
# Redirect everything to https
# That leaves the client to reconnect properly,
# and means we don't need to proxy HTTP, just HTTPS
frontend www
        mode http
        bind :80
        http-request redirect scheme https

HTTP Proxy to Origin (Layer 7)

# Layer 7 HTTP proxying (insecure), to backend servers
frontend www
        mode http
        bind :80
        # We are building the name of the backend from the 'host'
        # field in the request plus the literal '_http'
        # See backends for an example of how to name them
        use_backend %[req.hdr(host),lower,word(1,:)]_http

# Backends for HTTP
backend test1.apalrd.net_http
        mode http
        server test1_http 2601:40e:69:69:0:0:0:feed:80
backend test2.apalrd.net_http
        mode http
        server test2_http 2601:40e:69:69:0:0:0:beef:80

HTTPS Proxy to Origin (Layer 4 TLS Forwarding)

# Layer 4 TCP SNI proxy example
frontend www-tls
        # Layer 4 (TCP) mode
        mode tcp
        # Use TCPlog mode instead of HTTPlog
        option tcplog
        # Listen on TCP 443 (HTTP/1.1 and HTTP/2)
        bind :443

        # Wait for SSL Hello before forwarding
        tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }

        # Select backends for each server
        # Similar method to above, but using '_tls' on the end
        use_backend %[req_ssl_sni,lower,word(1,:)]_tls

# Backends for TLS servers
backend test1.apalrd.net_tls
        mode tcp
        server test1_tls 2601:40e:69:69:0:0:0:feed:443
backend test2.apalrd.net_tls
        mode tcp
        server test2_tls 2601:40e:69:69:0:0:0:beef:443

TCP Proxy to Origin (Layer 4 Port Forward)

#Frontend for SSH on port 2222
frontend ssh
        mode tcp
        option tcplog
        bind :2222
        default_backend ssh_server
#This is 1 incoming port -> 1 outgoing port (no SNI with TCP)
backend ssh_server
        mode tcp
        server test1_ssh 2601:40e:69:69:0:0:0:feed:22

TCP Load Balance to Origin (Layer 4 Balancing)

#Frontend for RDP on port 3389. Works with any TCP protocol which can be load balanced.
#Other examples include databases, etc.
frontend rdp
        mode tcp
        bind :3389
        default_backend rdp_servers
#Choose one of the RDP servers based on least connections
backend rdp_servers
        mode tcp
        #Roundrobin is also a good option
        balance leastconn
        server test1_rdp 2601:40e:69:69:0:0:0:feed:3389
        server test2_rdp 2601:40e:69:69:0:0:0:feed:3389

Option 3 - v4 to v6 Port Forwarding with Tayga

Tayga is a tool which can directly translate IPv4 and IPv6 packets at layer 3, so it works for any higher layer protocol (TCP, UDP, and more). Using Tayga, we will create IPv4 -> IPv6 address mappings, and then we can ‘port forward’ from our one public IPv4 to multiple internal IPv6 addresses.

I’ve setup Tayga previously to do NAT64 (where clients access the IPv4 internet over IPv6), and this is a different method of configuration. Anyway, here’s the install steps:

# Install Tayga
apt update && apt install tayga -y
# Remove the old-ass init.d script
rm /etc/init.d/tayga
# Remove the old configuration file
rm /etc/tayga.conf

And a new configuration file (/etc/tayga.conf)! It’s pretty simple here. I’m using 192.168.233.0/24 to hold the translation addresses, so you can translate to 252 hosts (-1 network, -1 broadcast, -1 for linux and -1 for tayga). You can use a larger subnet if you want to translate more hosts. This subnet is only used within the VPS.

# The name of the tun device (leave as-is)
tun-device nat64

# Tayga's IPv4 address on the translation network
ipv4-addr 192.168.233.2

# IPv4 translation prefix - used as src addr in IPv6 after translation
# Pull a random /96 out of your VPS prefix for this
# Our /64 literally gives us enough to have 4 billion IPv4 internets
# Just don't overlap with snid if you are using that too
prefix 2a01:4f9:c010:919d:64::/96

# If you need to ping Tayga, take the ipv4-addr above and merge it with your prefix
# In this case, that would be:
# 2a01:4f9:c010:919d:64::c0a8:e902

# Map a single IPv4 on our translation v4 subnet to a public IPv6
# Remember we can't use the first (network) and last (broadcast) in the subnet
# And we also used the first and second real addresses for Linux and Tayga. So start at 3. 
map 192.168.233.3 2001:db8:6969:420::1
map 192.168.233.4 2001:db8:6969:421::6

And a modern systemd service unit (/etc/systemd/system/tayga.service) for this guy. Note that I’m setting up a bunch of port forwards here! We are doing ’normal’ iptables port forwarding from the public v4 address to the translation addresses, then letting Tayga translate them v4->v6.

[Unit]
Description=Tayga NAT64
After=network-online.target

[Service]
#Start Tayga in debug mode (let systemd daemonize it)
ExecStart=/usr/sbin/tayga --config /etc/tayga.conf -d
Restart=always

#Do port forwarding after start / before stop
#Make sure you have a Add and Del rule for each
ExecStartPost=iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 23  -j DNAT --to-destination 192.168.233.3
ExecStopPre=iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 23  -j DNAT --to-destination 192.168.233.3
#You can of course copy/paste that as many times as you want

#Configure tunnel interface after start
ExecStartPost=ip link set nat64 up
ExecStartPost=ip addr add 192.168.233.1/24 dev nat64
#Update with the IP prefix you gave Tayga, take the first address
ExecStartPost=ip addr add 2a01:4f9:c010:919d:64::1/96 dev nat64
#No need to undo as the nat64 interface and its config is destroyed when tayga exits

#Enable IP forwarding (no need to modify)
ExecStartPre=/bin/bash -c "echo 1 > /proc/sys/net/ipv4/conf/all/forwarding"
ExecStartPre=/bin/bash -c "echo 1 > /proc/sys/net/ipv6/conf/all/forwarding"

[Install]
WantedBy=multi-user.target

Of course, when we are done, we need to apply the changes and start tayga

systemctl daemon-reload every time you change the tayga.service file (not after tayga.conf though)
systemctl enable --now tayga to enable and start it
systemctl restart tayga if you change the service file or tayga.conf
systemctl status tayga to see how it’s going
journalctl -xeu tayga to see how it’s going in more detail

Option 4 - Wireguard

Here I’m going to create a simple tunnel between my opnsense system at home and my VPS. Then I can port forward across the tunnel.

First we need to install wireguard tools (apt update && apt install wireguard-tools -y). Next, create a config on your OPNsense system, and then we will generate a client on OPNsense and copy it into the VPS.

Finally, we will add all of our iptables rules into the wg0.conf so they run automatically when the tunnel goes up and down:

# Need to enable IP forwarding
PostUp = /bin/bash -c "echo 1 > /proc/sys/net/ipv4/conf/all/forwarding"
PostUp = /bin/bash -c "echo 1 > /proc/sys/net/ipv6/conf/all/forwarding"

# Do port forwarding after start / before stop
# Make sure you have a Add and Del rule for each
PostUp=iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 23  -j DNAT --to-destination 172.27.14.233
PostDown=iptables -t nat -D PREROUTING -i eth0 -p tcp --dport 23  -j DNAT --to-destination 172.27.14.233
# Do Masquerade so packets return the right way
PostUp=iptables -t nat -A POSTROUTING -o wg0 -p tcp --dport 23 -d 172.27.14.233 -j SNAT --to-source 9.10.11.12
PostDown=iptables -t nat -D POSTROUTING -o wg0 -p tcp --dport 23 -d 172.27.14.233 -j SNAT --to-source 9.10.11.12
# You can of course copy/paste that as many times as you want

Bonus - ASCIIMation Test Setup

In my TCP example I setup an ASCIImation server. I initially copied a project from Github, found it didn’t have IPv6 support, patched it myself, then went to make a fork / PR and realized that someone had already made a v6 fork, so just use their fork.

Here it is

# Install Git
apt update && apt install git -y
# Clone repo somewhere
cd /var/lib
git clone https://github.com/berghauz/ascii-telnet-server.git
# Create a system user+group for us to have less permissions
adduser --quiet --system --group vader

I also wrote a systemd unit (/etc/systemd/system/asciimation.service) for it, and a user account for it. I have it bound to port 23, which requires elevated privilages to bind to low numbered ports, and systemd does this for us.

[Unit]
Description=ASCIImation server
After=network-online.target

[Service]
User=vader
Group=vader
#Defaults to bind on :: on port 23, so no need to specify
ExecStart=/usr/bin/python3 /var/lib/ascii-telnet-server/ascii_telnet_server.py  -f /var/lib/ascii-telnet-server/sample_movies/sw1.txt
AmbientCapabilities=CAP_NET_BIND_SERVICE
Restart=always

[Install]
WantedBy=multi-user.target

Kwumsy H3 Stream Dock USB Protocol

Fri, 12 Apr 2024 01:00:05 -0300

So today I’m taking a look at the Kwumsy H3 ‘Stream Dock’ No, not THAT Stream Deck, not a dock for the Steam Deck, there’s already enough name confusion. Basically, it tries to be a lower cost touchscreen alternative to the real Stream Deck, making use of the same addon format for wide addon support, and also somehow a USB-C laptop dock with Ethernet, USB 3, HDMI, and USB-PD passthrough.

While all of the hardware works as expected, the software experience isn’t great. There are plenty of cases where it seems like the daemon doesn’t automatically detect the H3 on boot, or the icons don’t update. That said, I did a deep-dive into the USB protocol and it’s pretty trivial and nearly identical to the ‘real’ Stream Deck protocol. I’m sure you can write your own software if you really want to or have a special use case for this thing.

Video

PCAP Files

Proxmox Backup Auto-Shutdown

Sat, 06 Apr 2024 01:00:05 -0300

Today I’m trying to reduce the power consumption of my Proxmox Backup Server. The HP Microserver is great for what I need, but it’s kinda loud and I’m working on optimizing my power bill. The homelab is the largest single consumer of electricity aside from the air conditioning in the summer, so it’s something I’m looking at heavily.

Anyway, I thought I could do this purely with systemd sleep / suspend initially. Systemd can set up the system RTC to wake the system from suspend or hibernate at a specific time, and it’s easy to configure this by setting up a timer unit which is allowed to wake the system to happen. Of course, this requires the system to be able to suspend or hibernate.

I tried doing S3 speep (normal suspend to RAM) and found that the Microserver doesn’t support this. I then didn’t have enough space on my 16G boot disk (microSD card) for a swap partition to hibernate to, and don’t want to mess with the partition tables on my zfs drives to add one there, so hibernate is out too. This leaves me with actually fully shutting the system down and booting back up in the future. Downside to this is the RTC won’t wake it back up, I need to have another system send a Wake on LAN magic packet. No big deal, I just need to make sure I send it early enough that the backups aren’t scheduled to start until after the server is fully booted.

Video

Here’s the video if you want to listen to me say all of these things:

Wake On LAN

I found the MACs of all of my interfaces. In this case, I’m using eno1, so I am using 70:10:6f:3e:d1:f8. You will of course need to find your own MAC address. This is also the hardware MAC, not the virtual MAC if you’re using a bond. I then installed the simple wakeonlan utility with apt install wakeonlan. It’s installed in /usr/bin so I can call it directly from the systemd service unit.

Service Unit to Wake On LAN

This just calls wakeonlan with the MAC address. Don’t enable the service, but you can systemctl start wakebigstor to bootup the server if you want. I put the file in /etc/systemd/system/wakebigstor.service. Bigstor is the name of the backup server.

[Unit]
Description=Wake up Bigstor

[Service]
Type=oneshot
ExecStart=/usr/bin/wakeonlan 70:10:6f:3e:d1:f8

[Install]
WantedBy=default.target

Timer Unit to call Service Unit

Systemd timers are separate units from services, so here’s the timer unit that triggers the service unit at 19:05 local time daily. I put this one in /etc/systemd/system/wakebigstor.timer and enabled and started it (systemctl enable --now wakebigstor.timer). Again, don’t enable the service, just the timer.

[Unit]
Description=Wakeup Bigstor
RefuseManualStart=no
RefuseManualStop=no

[Timer]
#Run at 19:05 (7pm local time)
OnCalendar=*-*-* 19:05:00
Unit=wakebigstor.service

[Install]
WantedBy=timers.target

Autoshutdown

This is 3 parts - a systemd timer that runs 5 hours after we start the server, to attempt shutdown (this gives clients 5 hours to start their backups), a systemd service unit that calls a bash script, and a bash script that waits until there are no more tasks running in proxmox-backup-manager.

Bash Script

This script calls proxmox-backup-manager repeatedly. Hopefully you can understand my bash scripting. I put this file in /usr/local/bin/autoshutdown. Don’t forget to chmod +x it since it’s executable.

#!/bin/bash

#Count of times we have queried and there are tasks running
count=0
#Number of no-task intervals before we shutdown
max=10
#Interval in seconds
interval=5


#Continue until we shutdown
while true; do
    #Check if we should shutdown until we see not 
    if [[ $(proxmox-backup-manager task list) ]]; then
        #Continue to wait
        count=0
    else
        #There are no tasks running, so increment counter
        count=$((count + 1))
        echo "No tasks running, time until shutdown is $(((max - count) * interval)) seconds"
        #If we reached the max count, then shutdown
        if  ((count >= max)); then
            echo "Time to shutdown!"
            shutdown now
        fi
    fi
    #Delay by interval to next check
    sleep "$interval"
done
echo "Exiting"

Service Unit

And the service unit that calls autoshutdown goes in /etc/systemd/system/autoshutdown.service:

[Unit]
Description=Shutdown when there are no tasks left

[Service]
ExecStart=/usr/local/bin/autoshutdown

[Install]
WantedBy=default.target

Timer Unit

And it also has a timer, /etc/systemd/system/autoshutdown.timer:

[Unit]
Description=Shutdown
RefuseManualStart=no
RefuseManualStop=no

[Timer]
#Run at 22:00 local time
OnCalendar=*-*-* 22:00:00
Unit=autoshutdown.service

[Install]
WantedBy=timers.target

When you are done, don’t forget to run systemctl daemon-reload and systemctl enable --now autoshutdown.timer (but not the service unit, just the timer).

Mount Root FS noatime

To reduce writes to the SD card, I mounted the root fs with noatime. The line in my /etc/fstab now looks like this:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sdh1 during installation
UUID=b58dfd0c-b6dc-4e44-a9b8-22ba3a5540d8 /               ext4    errors=remount-ro,noatime 0       1

Move Logs to Datastore Disks

Logs are stored in /var/log/proxmox-backup and a subdirectory for tasks can grow qutite significantly, with my larger file backups taking hundreds of MB for a log. This is not great for my root SD card, so I want to move them to zfs. I’m going to keep the api logs on root.

The zfs fs is already mounted at /mnt/datastore/backup (the datastore is named backup), so I created a zfs dataset backup/logs. I then rsync copied the logs into it rsync -r -v /var/log/proxmox-backup/tasks /mnt/datastore/backup/logs/, deleted the original rm -rf /var/log/proxmox-backup/tasks and symlinked it back back ln -s /mnt/datastore/backup/logs/tasks /var/log/proxmox-backup/tasks. I then needed to fix permissions to user/group backup:backup with chown backup:backup -R /mnt/datastore/backup/logs and chown backup:backup /var/log/proxmox-backup/tasks. After this, task logs were working correctly and backups were happy again.

Increase ZFS ARC Limit

Since the system exists primarily to store the data in ZFS and has no large workload otherwise, I increased the ARC max from 50% of system RAM (Default) to 75%, in my case 12GB. I did this by first calculating the number of bytes in 12GB (12884901888) and storing this in a new file /etc/modprobe.d/zfs.conf with the following contents:

options zfs zfs_arc_max=12884901888

This applies on the next boot.

UGreen NASync DXP4800 Plus Teardown

Thu, 28 Mar 2024 00:32:05 -0300

I did a full teardown of this unit, see the video below. This page also has the ’extra info’ (pci, cpu, ..) for you.

Video

Click on the thumbnail to view the video on Youtube

Hardware Info

All of this was taken via an Debian 12 system (Bookworm / kernel 6.1), so your kernel may be configured slightly differently

lscpu

Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      39 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             6
On-line CPU(s) list:                0-5
Vendor ID:                          GenuineIntel
BIOS Vendor ID:                     Intel(R) Corporation
Model name:                         Intel(R) Pentium(R) Gold 8505
BIOS Model name:                    Intel(R) Pentium(R) Gold 8505 To Be Filled By O.E.M. CPU @ 4.3GHz
BIOS CPU family:                    11
CPU family:                         6
Model:                              154
Thread(s) per core:                 2
Core(s) per socket:                 5
Socket(s):                          1
Stepping:                           4
CPU(s) scaling MHz:                 13%
CPU max MHz:                        4400.0000
CPU min MHz:                        400.0000
BogoMIPS:                           4992.00
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
                                    pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx 
                                    pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good 
                                    nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni 
                                    pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr 
                                    pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave 
                                    avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs 
                                    ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad 
                                    fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap 
                                    clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves 
                                    split_lock_detect avx_vnni dtherm ida arat pln pts hwp hwp_notify 
                                    hwp_act_window hwp_epp hwp_pkg_req hfi umip pku ospke waitpkg gfni 
                                    vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize 
                                    arch_lbr ibt flush_l1d arch_capabilities
Virtualization:                     VT-x
L1d cache:                          176 KiB (5 instances)
L1i cache:                          288 KiB (5 instances)
L2 cache:                           3.3 MiB (2 instances)
L3 cache:                           8 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-5
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

CPU is very modern with virtualization support. 1 P-core (2 threads) + 4 E-cores (4 threads) gives us the weird number of 5 cores and 6 threads. Linux is of course very good at scheduling across P/E cores.

lspci

00:00.0 Host bridge: Intel Corporation Device 4619 (rev 04)
	IOMMU group: 1
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-UP3 GT1 [UHD Graphics] (rev 0c) (prog-if 00 [VGA controller])
	IOMMU group: 0
00:06.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 (rev 04) (prog-if 00 [Normal decode])
	IOMMU group: 2
00:06.2 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #2 (rev 04) (prog-if 00 [Normal decode])
	IOMMU group: 3
00:0d.0 USB controller: Intel Corporation Alder Lake-P Thunderbolt 4 USB Controller (rev 04) (prog-if 30 [XHCI])
	IOMMU group: 4
00:14.0 USB controller: Intel Corporation Alder Lake PCH USB 3.2 xHCI Host Controller (rev 01) (prog-if 30 [XHCI])
	IOMMU group: 5
00:14.2 RAM memory: Intel Corporation Alder Lake PCH Shared SRAM (rev 01)
	IOMMU group: 5
00:15.0 Serial bus controller: Intel Corporation Alder Lake PCH Serial IO I2C Controller #0 (rev 01)
	IOMMU group: 6
00:16.0 Communication controller: Intel Corporation Alder Lake PCH HECI Controller (rev 01)
	IOMMU group: 7
00:1c.0 PCI bridge: Intel Corporation Device 51b8 (rev 01) (prog-if 00 [Normal decode])
	IOMMU group: 8
00:1c.4 PCI bridge: Intel Corporation Device 51bc (rev 01) (prog-if 00 [Normal decode])
	IOMMU group: 9
00:1d.0 PCI bridge: Intel Corporation Alder Lake PCI Express Root Port (rev 01) (prog-if 00 [Normal decode])
	IOMMU group: 10
00:1d.2 PCI bridge: Intel Corporation Device 51b2 (rev 01) (prog-if 00 [Normal decode])
	IOMMU group: 11
00:1f.0 ISA bridge: Intel Corporation Alder Lake PCH eSPI Controller (rev 01)
	IOMMU group: 12
00:1f.3 Audio device: Intel Corporation Alder Lake PCH-P High Definition Audio Controller (rev 01)
	IOMMU group: 12
00:1f.4 SMBus: Intel Corporation Alder Lake PCH-P SMBus Host Controller (rev 01)
	IOMMU group: 12
00:1f.5 Serial bus controller: Intel Corporation Alder Lake-P PCH SPI Controller (rev 01)
	IOMMU group: 12
02:00.0 Non-Volatile memory controller: Phison Electronics Corporation PS5013 E13 NVMe Controller (rev 01) (prog-if 02 [NVM Express])
	Subsystem: Phison Electronics Corporation PS5013-E13 PCIe3 NVMe Controller (DRAM-less)
	IOMMU group: 13
	Region 0: Memory at 80e00000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] Express (v2) Endpoint, MSI 00
		LnkCap:	Port #1, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Kernel driver in use: nvme
	Kernel modules: nvme

03:00.0 Ethernet controller: Aquantia Corp. Device 04c0 (rev 03)
	Subsystem: Aquantia Corp. Device 0001
	IOMMU group: 14
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		LnkCap:	Port #0, Speed 16GT/s, Width x4, ASPM not supported
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (downgraded), Width x2 (downgraded)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Kernel driver in use: atlantic
	Kernel modules: atlantic

04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])
	Subsystem: Samsung Electronics Co Ltd SSD 970 EVO
	IOMMU group: 15
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Kernel driver in use: nvme
	Kernel modules: nvme

05:00.0 SATA controller: ASMedia Technology Inc. Device 1164 (rev 02) (prog-if 01 [AHCI 1.0])
	Subsystem: ASMedia Technology Inc. Device 2116
	IOMMU group: 16
	Capabilities: [80] Express (v2) Endpoint, MSI 00
		LnkCap:	Port #0, Speed 8GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x2
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Kernel driver in use: ahci
	Kernel modules: ahci

06:00.0 Ethernet controller: Intel Corporation Ethernet Controller I226-V (rev 08)
	Subsystem: Intel Corporation Ethernet Controller I226-V
	IOMMU group: 17
	Capabilities: [a0] Express (v2) Endpoint, MSI 00
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Kernel driver in use: igc
	Kernel modules: igc

The two NICs are an Intel i226 (2.5G) and Aquantia AQC113 (10G NBase-T). Neither are limited in PCIe bandwidth. SATA is via an ASMedia controller off PCIe. The extra NVMe drive I added (Samsung) is in a different IOMMU group, in fact both of the NICs and both of the SSDs are in their own IOMMU group if you want to pass any of them through to VMs.

All About SUBNETTING your Networks! + Setup in OPNsense

Wed, 06 Dec 2023 01:00:05 -0300

You’ve probably heard all about creating multiple VLANs, for things like your IoT network, guest wifi, and more. But do you know what a VLAN actually is, and what the difference is between a VLAN and a Subnet? Today I’m going to cover the numbering of subnets in your network, and how to set up new subnet and VLAN interfaces in OPNsense. Come along on this adventure!

Video

Subnets

Starting in the last video, we have a basic network setup with the internet (green cable) and a ‘LAN’ made up of only my laptop (black and yellow cable). This setup is represented by this diagram, with IP addresses:

So let’s start by diving into what a subnet means.

A Subnet is essentially an address plus a length, which describes how many bits of the address are part of the prefix and how many are part of the host.

In this example, the IPv4 address of the laptop (10.212.46.10/24) would be interpreted as a 24-bit subnet, which means the first 24 bits become our prefix (10.212.46.0) and the last 8 become our host (10). In IPv6, the first 64 become the prefix (2001:db8:6969:5a01::) and the last 64 become the host (512b:d36f:d069:519a).

Now I’ve added a network switch and some more clients to the network, and given them addresses. What happens when my laptop wants to ping my desktop? Since the prefix matches, it goes directly on-link and finds the host’s MAC address using NDP (the IPv6 equivalent of ARP in IPv4). This bypasses the firewall entirely.

What happens if my laptop wants to ping an address on a different subnet? Since the prefix doesn’t match, the client will instead send the packet to the default router, and hope that the router can find the correct destination.

VLANs

In the last example, we had two different wires running all the way back to the router. The Yellow and Pink subnets need to be carried on separate sets of cabling, and using separate switches, since otherwise devices will try to get addresses on both subnets and it will be a mess.

VLAN is short for ‘Virtual LAN’. While there are a ton of technologies which can implement virtual LANs, the most common over Ethernet is IEEE 802.1Q.

Imagine this network topology. We would like to have all of our clients and all of our security cameras on two logically separate networks. We could do this by using two separate sets of cabling, and two sets of network switches, keeping them completely separate:

However, this costs money, especially since in a large network we might have more than just two networks to maintain. As we scale out the number of distinct networks we would like to operate, this gets unwieldly.

The solution to this is to add a ’tag’ to identify which network it belongs to. As long as all of the switches are aware of this scheme, they can add tags to packets when they come in, and implement a virtual switch for each tag across the network. Trunk links carry all of the tags, so we can keep traffic separate even as it traverses a complex network of switches and devices.

Logically, however, these are still separate networks and are still isolated from each other as if they were separate physical switches and links. They do share bandwidth, but otherwise can’t interact with each other, assuming all of the devices on the network support VLAN tags. This is a Layer 2 solution, traffic we carry on a VLAN can be anything supported by Ethernet, such as Internet Protocol, but it’s not required to.

Proxmox Unprivilaged LXC Container Bind Mount UID/GID Mapping

Sun, 12 Nov 2023 01:00:05 -0300

This is a snippet of my Personal Server Migration, but I thought it would be more useful as a stand-alone tutorial.

One of the challenges in dealing with unprivilaged LXC containers is that the UIDs/GIDs are mapped to 100000 in the host. This is a security feature, so the root user in the container doesn’t have root access if they are able to escape their container, but it’s also kinda a pain when sharing files between the host and container. So, to solve this, I created a user and group on host and on guest with the same uid (in my case, 1000) to access my shares. I can then add users in the container to group 1000 so they have access to the files.

So I added these lines to the LXC config ()/etc/pve/lxc/<id?.conf):

lxc.idmap: u 0 100000 1000
lxc.idmap: g 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: g 1000 1000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 1001 101001 64535

Just to walk through what this does:

There are IDs 0 through 65535 (total of 65536 entires) which must exist on the guest, for both User and Group
These lines configure how to map IDs in the host to IDs in the guest
The first line maps 0 on guest -> 100000 on host quantity 1000 sequential IDs, for Users
The second line does the same for Groups
Then we map user 1000 on guest -> 1000 on host quantity 1 sequential ID, again for User then Group
Finally we map user 1001 on guest -> 101001 on host quantity 64535 (65536 total - 1000 - 1 already mapped) for the rest of the IDs We need to make sure that no IDs are listed twice on the guest side, there are no gaps on the guest side, and the total of UIDs and GIDs is 65536 on the guest side. The CT will fail to start if you get this wrong.

In my case, I created a regular user + group (so not a system user) and it is 1000 on both sides, so I can map 1000 to 1000. If they are different, you’ll need to change them of course.

To allow the host’s IDs to be passed into a container, we need to give permissions to a host user to do the mapping. Since root creates the container, the user root needs to be able to subuid / subgid for the users / groups that we are idmapping into the container. This is set by the files /etc/subuid and /etc/subgid. I added this line to both files:

root:1000:1

This command allows the host user root to idmap {user, group} id 1000 quantity 1 sequential ID. Adjust as required for your setup.

After doing this, I did a recursive chown of the data directories to 1000:1000 on host and this also makes the guest’s user 1000 have access (since it’s idmapped).

To actually pass the directory into the container, I use pct like this:

pct set 100 -mp0 /data/video,mp=/mnt/video

where 100 is the ID of the container, mp0 is the mount point ID that Proxmox keeps track of, /data/video is the path on host, and /mnt/video is the path in the container.

Good luck with your containers!

Unleash your Home Cameras with FRIGATE Self-Hosted AI Video Recorder! Install on Proxmox LXC

Thu, 09 Nov 2023 01:00:05 -0300

Do you have security cameras at your house? Would you like to locally host all of your recording and analytics, to make sure nobody else has access to your video feeds and recordings? Would you also like to integrate with Home Assistant, the greatest open automation platform in the world? Then Frigate NVR is for you! In this video, I’m going to go in depth to setup Frigate in an LXC container, for maximum efficiency. Using Podman Quadlet, I’m going to manage the Frigate container in a sane way with normal systemd and journalctl tools. And I’m going all-in on hardware passthrough, with my Coral TPU for advanced AI detections and person/cat/car counting, along with a basic Intel Quick Sync GPU to decode the video streams in hardware and reduce CPU load. So join me on this adventure!

Let’s make Frigate work on the ultimate home server

Video

Install Debian LXC

Since I’d like to use some fairly recent features of Podman, we MUST start with debian 12 (Bookworm) or 13 (Trixie) when it is released. In addition, I added a second mount point at /var/frigate for data. We need Nesting and FUSE enabled for this. I’m using a privilaged container since I am passing in devices.

Be aware that enabling FUSE prevents Proxmox Backup (vzdump) from taking snapshot-style backups, it will get stuck trying to take the snapshot. You have to use suspend or stop mode. In this particular case, I’ve decided to not back up this container, since the only important file to recreate it is the config.yml for Frigate and the video files aren’t going to be backed up anyway.

If you are on Debian Bookworm, you will need to upgrade to Trixie to have Podman Quadlet. If Trixie is already released, you can skip this step:

#Replace bookworm with trixie in sources.list
#Not needed if you are actually using Trixie (when it releases)
sed -i 's/bookworm/trixie/g' /etc/apt/sources.list
#Update and dist-upgrade to Trixe
#Answer yes to all prompts
apt update && apt dist-upgrade -y

And finally, let’s install Podman, our container engine of choice:

#Install Podman
apt install podman -y

You also might want to set the timezone, so your Frigate recordings are in local time instead of UTC:

timedatectl set-timezone "America/Detroit"

Install Frigate

Now we’re going to create the podman contaner file, which uses systemd syntax and lets us manage the container with systemd and avoid Docker nastiness.

First up, create the folders for Frigate to mount:

#Create media and config directories
mkdir -p /var/frigate/{media,config}

At this point, put your Frigate config file in /var/frigate/config/config.yml in the container, read the frigate docs for that one (or see my own config later at the bottom).

Our Quadlet unit file is /etc/containers/systemd/frigate.container (You may need to create that directory, also):

[Unit]
Description=Frigate video recorder
After=network-online.target

[Container]
#Basic setup
ContainerName=frigate
#Use LXC host networking
#To avoid any Docker network nonsense
Network=host
#How to add environment variables like passwords
Environment=FRIGATE_RTSP_PASSWORD="password"
Environment=FRIGATE_MQTT_USER="user"
Environment=FRIGATE_MQTT_PASSWORD="pass"
#Mounted volumes
Volume=/var/frigate/media:/media
Volume=/var/frigate/config:/config
Volume=/etc/localtime:/etc/localtime:ro
#TempFS 
Mount=type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000
ShmSize=64m
#The image itself
Image=ghcr.io/blakeblackshear/frigate:stable
#Auto-update from the registry
AutoUpdate=registry

[Service]
#Restart automatically
Restart=always
#Give it a 15 minutes to start
#Since any image pulls will take a long time
TimeoutStartSec=900

[Install]
# Start on boot (default.target is 'on boot')
WantedBy=multi-user.target default.target

And finally we can start the new service:

systemctl daemon-reload
#The first time will take awhile since it pulls the image
systemctl start frigate
#We can't systemctl enable it since these units are dynamically generated
#So instead we WantedBy it, above

And view logs with journalctl -xeu frigate

In case you’re curious or want to use Quadlet for your own containers, here’s a link to the docs on container units.

Install Caddy Reverse Proxy for TLS

While it would be nice if the Frigate devs didn’t merge nginx with their own code so we could configure nginx for security, but we are stuck with their nginx config in their Docker image. As of now, this doesn’t even support IPv6, so we need some sort of front end proxy, and we also probably want a TLS certificate. I’ll show a self-signed cert in this example, but it’s also easy to use Let’s Encrypt.

So, installing Caddy:

apt install debian-keyring debian-archive-keyring apt-transport-https -y
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/ caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' > /etc/apt/sources.list.d/caddy-stable.list
apt update
apt install caddy -y

And the /etc/caddy/Caddyfile I am using (delete the default and replace with this):

# The Caddyfile is an easy way to configure your Caddy web server.
# Don't forget to put the hostname in DNS where it belongs
corona.palnet.net {
    #Address of backend
	reverse_proxy :5000
    #Settings for TLS
    #Internal means self-signed
	tls internal
}

And start it:

systemctl enable --now caddy

Install Coral TPU Drivers

Since LXC containers share a kernel with the host, we need to install the Coral TPU drivers on the Proxmox host, NOT in the container. Since these drivers are also not in-tree, we need to install them via apt and compile them for the Proxmox kernel using dkms. So let’s do that now.

Also, if you are using a USB TPU instead of PCIe, you can skip down to the host bind mount point. From the official Google docs:

#Add coral edgetpu repository
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" > /etc/apt/sources.list.d/coral-edgetpu.list
#Add GPG key from Google
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
#Apt update to pull the package list
apt update
#Install the DKMS driver (we do *not* need the headers / libs since the container needs those)
apt install gasket-dkms

FYI: Google can’t be bothered to update their repo with a build that works with recent Linux kernels, so as of Proxmox 8.1 / Linux 6.5, the gasket-dkms package that we need is broken. The code has long been fixed in the Github repo (thanks to open source contributors helping Google for free), but Google hasn’t built and published the binaries. Until then, you’ll have to compile it yourself, or contact their support and tell them to fix it.

In the meantime, the amazing Proxmox community has a solution to get it to work, and the Linux community yeeted gasket out of the kernel for being unmaintained by Google. So you might want to migrate away from coral.ai before they end up on the Killed By Google list.

Also, a very nice Github user @feranick has published deb packages you can use for now, do this instead of the real instructions above until Google fixes their life:

#Download package
https://github.com/feranick/gasket-driver/releases/download/1.0-18.2/gasket-dkms_1.0-18.2_all.deb
#Install with apt
apt install ./gasket*.deb -y

In theory we should be able to modprobe apex now. However, if you get an error like this, you’ll need to take a slight detour:

dkms: autoinstall for kernel 6.2.16-19-pve was skipped since the kernel headers for this kernel do not seem to be installed.

That means we don’t have headers installed. We can install them with apt install pve-headers, but that will install the latest headers, not the headers for our version of Proxmox. The safest way to deal with this is to completely upgrade the Proxmox system and then install headers for the latest kernel version, to make sure the minor versions are entirely correct (i.e. the 6.2.16-19-pve).

apt update
apt install pve-headers -y
apt full-upgrade -y
#Now reboot, so the new kernel image is used (otherwise there will be a discrepancy between the kernel it is trying to build / the headers, and the running kernel)

#Then tell dkms to build and install the new modules (they are already installed, just not for this particular kernel version):
dkms autoinstall

And finally, after this, you an modprobe apex. At this point, you should have a /dev/apex_0 for your TPU.

Passthrough Coral TPU to Container

Now that we have that device, let’s pass it through to the container. Make sure the LXC container is not running and then edit its config file (/etc/pve/lxc/<id>.conf) to add this at the end:

lxc.cgroup2.devices.allow: c 120:0 rwm
lxc.mount.entry: /dev/apex_0 dev/apex_0 none bind,optional,create=file

Of course if you have multiple TPUs you can add all of the apex devices here.

Now, start the container and we will deal with passing the device into Frigate.

Under the [Container] section of the frigate unit file, we need to add this:

#Pass-through Coral TPU
AddDevice=/dev/apex_0

And finally, in our config.yml for Frigate, add the line to configure the Coral TPU:

detectors:
  coral1:
    type: edgetpu
    device: pci:0

You can edit the config.yml from within the Frigate UI if you want.

Passthrough GPU Decode to Container

I only have an Intel GPU to test currently, so here’s the config I am using. It should be similar but not identical for AMD GPUs. Nvidia can fuck right off with their proprietary container driver mess, I’m not helping you guys with that one.

First, we pass through the hardware to the container from Proxmox. Assuming you only have one GPU, it will be renderD128. If you have more, you can figure out which is which on your own. Shutdown the container and add these lines to the end of the container config (/etc/pve/lxc/<id>.conf):

lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file

Next, we need to add the device and also the perfmon capability to the Podman unit in the [Container] section:

#Add device passthrough for render node
AddDevice=/dev/dri/renderD128
#Add capability for the container to monitor GPU performance
AddCapability=CAP_PERFMON
#If you are not using the iHD driver, you might need one of these options:
#Environment=LIBVA_DRIVER_NAME="radeonsi"
#Environment=LIBVA_DRIVER_NAME="i965"

And finally, we need to configure Frigate to actually use these, by adding a new section to the config.yml:

#Use hardware acceleration for ffmpeg
ffmpeg:
  hwaccel_args: preset-intel-qsv-h264
  #Other options are preset-intel-qsv-h265
  #For AMD cards use preset-vaapi

If you are using a very modern Intel CPU, you might need to enable the ‘guc’ microcode loading. Create the file /etc/modprobe.d/i915.conf and write in the following contents, then reboot:

options i915 enable_guc=3

I have no idea exactly which CPUs and kernel versions are affected.

If you want to make sure your GPU is working, you can install either intel_gpu_top from the intel-gpu-tools package (Intel) or radeontop from the radeontop package and run it in the container.

My Frigate Config

This is sanitized, but close to what I actually have

#MQTT server
mqtt:
  host: telstar.palnet.net
  #Use environment variable for these
  user: "{FRIGATE_MQTT_USER}"
  password: "{FRIGATE_MQTT_PASSWORD}"

#One single PCIe edge TPU
detectors:
  coral1:
    type: edgetpu
    device: pci:0

#Use hardware acceleration with h264 and intel quicksync vieo
ffmpeg:
  hwaccel_args: preset-intel-qsv-h264

#Recording options
record:
  enabled: True
  #Keep all recordings for 2 days
  retain:
    days: 2
    mode: all
  #Keep event recordings for 10 days
  events:
    pre_capture: 5
    post_capture: 5
    retain:
      default: 2
      mode: motion
      objects:
        person: 15
        #Cat is special
        cat: 15

#Camera configuration
cameras:
  # Back door camera
  back_door:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@keyhole4.palnet.net:554/cam/realmonitor?channel=1&subtype=00
          roles:
            - detect
            - record
            - rtmp
    detect:
      width: 2560
      height: 1440
    #Per camera object and filter settings
    objects:
      # Track persons and cats in only
      track:
        - person
        - cat   

  # Front door camera
  front_door:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@keyhole3.palnet.net:554/cam/realmonitor?channel=1&subtype=00
          roles:
            - detect
            - record
            - rtmp
    detect:
      width: 2560
      height: 1440
    #Per camera object and filter settings
    objects:
      # Track persons and cats in only
      track:
        - person
        - cat


  # Front Yard camera
  front_yard:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@keyhole2.palnet.net:554/cam/realmonitor?channel=1&subtype=00
          roles:
            - record
            - rtmp
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@keyhole2.palnet.net:554/cam/realmonitor?channel=1&subtype=01
          roles:
            - detect
    detect:
      width: 704
      height: 480
    #Per camera object and filter settings
    objects:
      # Track persons and cats in only
      track:
        - person
        - car

  # Driveway Camera (2 resolutions)
  driveway:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@keyhole1.palnet.net:554/cam/realmonitor?channel=1&subtype=00
          roles:
            - record
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}4@keyhole1.palnet.net:554/cam/realmonitor?channel=1&subtype=02
          roles:
            - detect
            - rtmp
    detect:
      width: 1280
      height: 720
    #Per camera object and filter settings
    objects:
      # Track everything
      track:
        - person
        - car
        - cat

  # Driveway Cars Camera
  driveway_cars:
    ffmpeg:
      inputs:
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@keyhole5.palnet.net:554/cam/realmonitor?channel=1&subtype=00
          roles:
            - record
            - rtmp
        - path: rtsp://admin:{FRIGATE_RTSP_PASSWORD}@keyhole5.palnet.net:554/cam/realmonitor?channel=1&subtype=01
          roles:
            - detect
    detect:
      width: 704
      height: 480
    #Per camera object and filter settings
    objects:
      # Track everything
      track:
        - person
        - car
        - cat

Installing Mikrotik RouterOS on Proxmox VE easily

Mon, 06 Nov 2023 01:00:05 -0300

I’ve been using Mikrotik RouterOS for awhile now, both using their hardware and their virtual image (Cloud Hosted Router). It’s a great product for routing and firewalling, while it’s not a NGFW it’s an absolutely amazing router and their L2/L3 switches are also a great value for the price.

So anyway, I often setup images of the latest CHR in Proxmox VE for testing things. I create vmbr bridges in Proxmox to point-to-point link multiple CHRs, and can lab out complex network setups. Since CHR is free with a 1mbps throughput limit, it’s perfectly adequate to lab with only the free VMs.

So here’s the script I use to download and setup a Proxmox VM template for the latest version of RouterOS, from which I can clone and add network interfaces to.

Make sure to paste in the latest download link from the Mikrotik download site. You want the latest stable Cloud Hosted Router - Raw Image. As of this writing, it’s ROS V7.11.2.

wget "https://download.mikrotik.com/routeros/7.11.2/chr-7.11.2.img.zip"
unzip chr-7.11.2.img.zip
qm create 940 --name "temp-mikrotik-ros-7.11.2" --ostype l26
qm set 940 --net0 virtio,bridge=vmbr0
qm set 940 --serial0 socket --vga serial0
#Yes, really, this is how littke RAM it needs
qm set 940 --memory 256 --cores 2 --cpu host
#Import the disk as scsi0, default boot order, resize up to 8G
qm set 940 --scsi0 local-zfs:0,import-from="$(pwd)/chr-7.11.2.img",discard=on
qm set 940 --boot order=scsi0 --scsihw virtio-scsi-single
qm disk resize 940 scsi0 8G
#Make it a template
qm template 940

If you followed my Easy Mode Cloud-Init Templates tutorial, 940 will be a number that fits well with that numbering scheme.

Building a TELEPROMPTER with a Raspberry Pi

Fri, 03 Nov 2023 01:00:05 -0300

Come along as I 3D print my own teleprompter! Hopefully improving my production quality, I guess you guys will be the judge of that.

And the zip file for the parts (STL, 3MF, and FreeCAD originals) (CC-BY-SA)

A $9 Introduction to the RISC-V Future of Computing

Mon, 09 Oct 2023 01:00:05 -0300

Is RISC-V the future of computing? I sure hope so. So I tracked down one of the cheapest Linux-capable SBCs that supports this architecture, the Milk-V Duo. For a retail price of $9, this little guy offers a RV64 Linux environment complete with busybox, Ethernet, and a wide assortment of IO rivaling some microcontrollers. Today I’m going to steup the board and start learning about RISC-V computing!

Fundamentally, the RISC-V architecture is a document which describes the binary machine lanuage of a 32, 64, or 128 bit processor with integer and optional floating point support, but it’s also symbolic of the shift to open computing for the future, and as a computer engineer I’m excited to learn more about it.

Video

Test Code

I compiled 3 test programs in 5 different 64-bit architectures (AMD, ARM, RISC-V, PowerPC, and MIPS). You can find the code and the resulting assembly here.

IPv6-enabled SD Card Image

If you’d like to use my IPv6-enabled SD Card image, I’ve released it on my github here. I’ve also submitted a pull request which you can check the status of here, so you can go back to their releases once it’s merged.

No-IOM (GPU) SD Card Image

I also have a build with both IPv6 and the IOM (which does graphics processing) disabled, if you need more memory and don’t need the camera features. It’s also on my github. No PR for this one, but you can look at the no-iom branch in my repo to see the changes.

Memory before:

              total        used        free      shared  buff/cache   available
Mem:          29236       14172        8336          80        6728       12260
Swap:             0           0           0

Memory after:

              total        used        free      shared  buff/cache   available
Mem:          56680       14132       35844          80        6704       38668
Swap:             0           0           0

Swap on SD Card

The default SD card image has a 256M space allocated for swap, but it’s not formatted/mounted. You can do that fairly easily:

mkswap /dev/mmcblk0p3
swapon /dev/mmcblk0p3

In the future you just need to run the swapon command to enable the partition that’s already formatted. Looking at free, we get swap now:

              total        used        free      shared  buff/cache   available
Mem:          56680       14216       35264          80        7200       38596
Swap:        262140           0      262140

Now you should have enough memory for some real work! It’s just super slow SD card memory lol.

Packet Capture in Proxmox

Sun, 01 Oct 2023 01:00:05 -0300

When you’re troubleshooting network issues, it’s often extremely helpful to view and analyze packet captures. The de-facto tool for this is the open-source Wireshark, which has an extensive protocol decoding capability. So, as a Proxmox user, it would be nice to be able to analyze VM networking issues using Wireshark.

Unfortunately for us, Wireshark is a graphical application and Proxmox’s web UI doesn’t support it. However, we can use the command-line tool tcpdump to create a pcap file, and then analyze that file in Wireshark.

To do this, we need to know what interface on the Proxmox system corresponds to the net interface on our VM. Looking at the VM’s Hardware pane, we can see the net0 and net1 interfaces in this example. Take note of that number, it’s important.

For net0 of VM 501, the Linux device we need to capture will be named tap501i0. This is located between the Proxmox firewall and the VM, if you have the firewall enabled (you probably should use it, it’s a great tool). Conceptually it looks like this:

So, finally, to capture traffic, run tcpdump from the Proxmox shell.

For this example, the command is tcpdump -i tap501i0 -n -w <filename>.pcap. You can use whatever file name you want. Use Ctrl+C when you are all done.

To copy the file off, you can use scp, or store move the file to a network location you have access to. Network storage in Proxmox is mounted at /mnt/pve/, fyi.

Man Page for tcpdump in case you need it

REALLY Persistent Ethernet Interfaces on Linux

Mon, 25 Sep 2023 01:00:05 -0300

So Linux has adopted Persistent Device Naming, which is a really great thing for most systems. Unlike the old days where we just had eth0 and eth1 and eth2 etc (which at least has no spaces unlike Local Area Connection 6 that another OS uses), whose order depended on driver initialization in the kernel. Most people just had eth0 and were happy, and most people will still just have one Ethernet interface and will still be happy.

But with more complex setups, this persistent naming can be a problem, especially as PCIe devices come and go and change the bus layout. I’d really like truly persistent naming, tied to the specific MAC address or something, that won’t ever change no matter what else I do with the system.

To be fair to persistent device names, for most people who are not making regular hardware changes, they are a good thing. The hierarchy of naming prefers onboard naming when possible (i.e. provided by the UEFI / HW), followed by the device path. This device path is consistent across reboots, but adding or removing hardware can cause items in the list to move up and down, changing their path ID (i.e. enp4 becomes enp5 after adding another PCIe card). But at least replacing hardware with identical hardware will keep the path IDs the same, so we can replace NICs or really anything else in the system without messing up our network configuration.

Solution

The solution is to rename these devices on our own terms instead of the udev methodology. We can match hardware based on IDs such as the MAC address, which identifies a singular NIC, and use this to assign a name of our own. Now it’s up to us to modify this file if we ever replace a NIC, but at least other things in the system won’t change our name.

Here’s the documentation for systemd link units, it’s incredible really.

So given that information, I constructed the absolute minimal link file for my Proxmox system. We can’t use any name that the system might assign automatically (such as ethX, enoX, enpX, …) or we are in for trouble. I was originally going to name them geX for gigabit ethernet and xgX for 10/25Gig, but Proxmox’s web UI was not a fan of this and refused to do any networking administration since it couldn’t figure out what type those were by parsing the /etc/network/interfaces file. So, I settled on enge0 for my onboard gigabit Ethernet and enxg1 for the port on the 25 gig card that I’m using (enxg0 is the other port on the card, since these used to be function 0 and function 1 of the device). Remember that enx is used by devices identified by their MAC address and nothing else (usually USB NICs), so don’t use that. eno, enp, and ens are all used by udev as well.

We now need to write a unit file for each interface we want to have a really permanent name, and it needs to start with a number less than 99, since 99-default.link will set the interface to regular persistent naming.

That results in a file like /etc/systemd/network/10-ge0.link:

[Match]
MACAddress=12:34:56:78:9a:bc
[Link]
Name=enge0

Add one for each, in my case I added 3. Make sure you get the MAC right. Now go into /etc/network/interfaces and replace the old name with the new one. On reboot, udev should do its thing, call systemd-link, which will do it’s thing, and everything should magically come up.

My /etc/network/interfaces on Proxmox now looks like this:

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# I know what I'm doing so I edited it manually

auto lo
iface lo inet loopback

#Real gigabit NIC
auto enge0
iface enge0 inet6 manual

#Dual port Mellanox NIC
auto enxg0
iface enxg0 inet6 static
	address fd69:beef:cafe:6::70/64

#Dual port Mellanox NIC
auto enxg1
iface enxg1 inet6 manual

#Bond primary LAN NICs for failover
auto bond0
iface bond0 inet manual
	bond-slaves enge0 enxg1
	bond-miimon 100
	bond-mode active-backup
	bond-primary enxg1

#Primary VM bridge to LAN
auto vmbr0
iface vmbr0 inet static
	address 172.27.1.70/20
	gateway 172.27.0.1
	bridge-ports bond0
	bridge-stp off
	bridge-fd 0
	bridge-vlan-aware yes
	bridge-vids 2-4094

iface vmbr0 inet6 static
	address 2001:db8:6969:420::70/64
	gateway 2001:db8:6969:420::1

And they show up Proxmox UI too!

And of course ip a shows the interfaces as expected!

3: enge0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond0 state DOWN group default qlen 1000
    link/ether 26:a9:1e:5e:7f:5d brd ff:ff:ff:ff:ff:ff permaddr a8:a1:59:22:48:32
7: enxg0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether b8:ce:f6:ae:38:e4 brd ff:ff:ff:ff:ff:ff
    inet6 fd69:beef:cafe:6::70/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::bace:f6ff:feae:38e4/64 scope link 
       valid_lft forever preferred_lft forever
8: enxg1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 26:a9:1e:5e:7f:5d brd ff:ff:ff:ff:ff:ff permaddr b8:ce:f6:ae:38:e5
9: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 26:a9:1e:5e:7f:5d brd ff:ff:ff:ff:ff:ff
10: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 26:a9:1e:5e:7f:5d brd ff:ff:ff:ff:ff:ff
    inet 172.27.1.70/20 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 2001:db8:6969:420::70/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::24a9:1eff:fe5e:7f5d/64 scope link 
       valid_lft forever preferred_lft forever

So now I’m free to add/remove PCIe cards at will (this system gets used for a lot of testing random devices) and not worry about what that will do to my main NIC.

Migrating my PERSONAL SERVER from TrueNAS to Proxmox

Wed, 20 Sep 2023 01:00:05 -0300

Today I’m taking my 10 servers and hopefully working that list down to just 7! JUST SEVEN! So, driven by my desire to consolidate my critical services into one box so I can lab away with the rest of the boxes, I am taking the time to shut down some of the most critical servers in the house and re-home them, then disassemble the parts for the next project.

Come along with me on this adventure!

Video

My Servers

So anyway here’s the list of what I have now and their current jobs:

Iridium - TrueNAS SCALE - It’s just doing Samba, nothing else at this point. I kinda hate the TrueNAS dev’s highly restrictive attitude.
Minilab - my Ryzen 2400G mini-PC - It’s doing mission critical VMs and CTs like Home Assistant. It also has some Coral TPU cards for Frigate.
Terra - This is the Terramaster NAS from another video. It’s currently only used for filming videos, but it’s cheap and low-power.
Corona - Runs Frigate and was formerly a VM on Minilab, but I moved it to the ZimaBoard so I can do a test of different acceleration methods for an upcoming video.
Megalab - This is the PC on a box that you see in a lot of my vieos.
pve1,pve2,pve3 - This is my 3-node cluster from the cluster series
My firewall router running OPNsense - nothing will change here
Bigstor, my Proxmox Backup Server (also feat. LTO Tapes) - nothing will change here either

Today, I have a few tasks all happening at once, to minimize downtime:

Purge everything off Terra and remove the spinning drive pool and drives
Migrate all of the workloads on Minilab to Terra and bring them back up, especially home automation, with minimal downtime
Spin down the TrueNAS SCALE server and migrate those Samba datasets to a container in Proxmox on Minilab, similar to my Pretty Good NAS video.
Properly deal with file permissions when passing datasets into a container via bind mounts, since I glossed over permissions in the previous video.
Setup proper backups to my Proxmox Backup Server from the Cockpit+Samba container, in a way that is scalable and lets me configure the frequency and stuff for each dataset, and get the full benefit of file-based recovery from PBS.
Setup snapshots of the zfs dataset in Proxmox (like I had in TrueNAS), more for accidental deletion prevention than long term backups. These should be done extrmely frequently.
Setup Shadow Copy so I can access accidentally deleted files myself without going to the backup server

I’m not including separate sections in the blog on the pruging / migration since it was unremarkable. I shutdown the VM/CT on Minilab, ran a backup in the Proxmox UI to the PBS server, and restored the backup on Terra. It just worked.

Likewise, importing the zfs pool from TrueNAS to Proxmox was also fairly straightforward, I did a zpool import -f to get the name of the pool and then zpool import -f poolname to import it despite warnings that it was exported on a different system. All done, it imports automatically now. No need to even mention the names of the disks, zfs figures it all out.

Unprivilaged Container Bind Mount UID/GID Mapping

One of the challenges in dealing with unprivilaged LXC containers is that the UIDs/GIDs are mapped to 100000 in the host. This is a security feature, so the root user in the container doesn’t have root access if they are able to escape their container, but it’s also kinda a pain when sharing files between the host and container. So, to solve this, I created a user and group on host and on guest with the same uid (in my case, 1000) to access my shares. I can then map the Samba users in the container to this group to get permissions to the files.

So I added these lines to the LXC config (in my case /etc/pve/lxc/104.conf, replace 104 with your CT ID):

lxc.idmap: u 0 100000 1000
lxc.idmap: g 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: g 1000 1000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 1001 101001 64535

Just to walk through what this does:

There are IDs 0 through 65535 (total of 65536 entires) which must exist on the guest, for both User and Group
These lines configure how to map IDs in the host to IDs in the guest
The first line maps 0 on guest -> 100000 on host quantity 1000 sequential IDs, for Users
The second line does the same for Groups
Then we map user 1000 on guest -> 1000 on host quantity 1 sequential ID, again for User then Group
Finally we map user 1001 on guest -> 101001 on host quantity 64535 (65536 total - 1000 - 1 already mapped) for the rest of the IDs We need to make sure that no IDs are listed twice on the guest side, there are no gaps on the guest side, and the total of UIDs and GIDs is 65536 on the guest side. The CT will fail to start if you get this wrong.

In my case, I created a regular user + group (so not a system user) and it is 1000 on both sides, so I can map 1000 to 1000. If they are different, you’ll need to change them of course.

root:1000:1

This command allows the host user root to idmap {user, group} id 1000 quantity 1 sequential ID. Adjust as required for your setup.

After doing this, I did a recursive chown of the data directories to 1000:1000 on host and this also makes the guest’s user 1000 have access (since it’s idmapped).

To actually pass the directory into the container, I use pct like this:

pct set 100 -mp0 /data/video,mp=/mnt/video

where 100 is the ID of the container, mp0 is the mount point ID that Proxmox keeps track of, /data/video is the path on host, and /mnt/video is the path in the container.

Large Dataset Backups to Proxmox Backup Server

One challenge with Proxmox Backup Server is that it’s not particularly good at large host backups. It’s very optimied for VMs, where the qemu dirty bitmask is used to know which static blocks have changed and must be reuploaded, but for file-based backups it reads all of the files and uploads chunks which have changed. This ‘read all the files’ takes awhile. I also have different backup requirements for each of my datasets, and PBS has a requirement that all backups for a single VM/CT/Host are done together, since all of the differend disks end up as different entries in the same backup.

So, with all of these requirements, I wrote some systemd scripts to do my large dataset file backups. Instead of doing a single backup of all of the datasets, I name each backup ‘host’ after the dataset - so in PBS it shows up as host/video, host/media, … instead of host/iridium with separate datasets for video and media. This means I can backup each dataset on its own time, separately. I’m happy with this. I also structured all of my datasets so they are located at /mnt/, which means I can rely on this path in the backup scripts (name -> path translation means adding /mnt in front, no path lookups).

First step is to install the Proxmox Backup Client within the container. Proxmox provides a Debian repository for the backup client only, which you can add following their instructions. Then apt install proxmox-backup-client. Yay! Now create an API user and key and all that jazz and get ready to copy it into the backup script.

Here’s the systemd script I wrote to do a single backup. It’s instanced, so it has @ at the end, and you pass a parameter for the backup you want to do (i.e. pbs-client@video). It goes in /etc/systemd/system/pbs-client@.service. Make sure your backup user has read permissions to the data at least.

[Unit]
Description=Run Backups

[Service]
Type=oneshot
#Run as backup
User=backup
Group=backup
#Allow up to 15% CPU
CPUQuota=15%
#Backup settings
Environment=PBS_REPOSITORY=user@pbs@dns_name.lan:datastore
Environment=PBS_PASSWORD=api_key
Environment=PBS_FINGERPRINT="fingerprint"
#Run proxmox backup client for the passed directory
ExecStart=proxmox-backup-client backup %i.pxar:/mnt/%i --all-file-systems true --backup-id "%i"

[Install]
WantedBy=default.target

After a systemctl daemon-reload you are ready to do manual backups! Just systemctl start pbs-client@video and it will create a backup id ‘video’ with the file ‘video.pxar’ from the path ‘/mnt/video’. Since Cockpit is amazing, you can watch the progress by going to the Services tab, finding that specific service, and viewing the logs. Awesome!

Now we just need to time it to run regularly. Here’s a unit I use for that (/etc/systemd/system/pbs-video.timer):

[Unit]
Description=Backup Video Data
RefuseManualStart=no
RefuseManualStop=no

[Timer]
#Run at 4am EST / 09 UTC some days
OnCalendar=Mon,Wed,Fri *-*-* 09:00:00
Unit=pbs-client@video.service

[Install]
WantedBy=timers.target

Now enable the timer with systemctl enable pbs-video.timer and it will run on schedule. You can again monitor it from the Cockpit services tab.

Sanoid for Snapshots

Avoiding the issue of delegating zfs permissions into a container, I am installing Sanoid on the Proxmox host to take snapshots of the datasets used by the container. It’s a tool designed to take snapshots of zfs datasets basically, and it’s good at it. It’s also in the Debian repos, so we can just apt install sanoid and edit the config file. Since this is mostly for Shadow Copy, I’m using the ‘frequently’ snapshots due to naming convention quirks making it harder to use all of the different time steps in Shadow Copy.

Users can still mount their own snapshots by browsing to the .zfs folder in any directory, including within the container, and permissions are maintained for snapshots. So the container can now read snapshots of its own mounts, automatically.

Here’s the /etc/sanoid/sanoid.conf snippet:

# you can also handle datasets recursively in an atomic way without the possibility to override settings for child datasets.
[data/video]
        use_template = production
        recursive = zfs
[data/media]
        use_template = production
        recursive = zfs


#############################
# templates below this line #
#############################

# name your templates template_templatename. you can create your own, and use them in your module definitions above.

# Using a lot of frequently at 30min for Shadow Copy, since it isn't a fan of the differently named snapshots.
[template_production]
        frequently = 144
        frequent_period = 30
        hourly = 0
        daily = 30
        monthly = 3
        yearly = 0
        autosnap = yes
        autoprune = yes

Shadow Copy with Sanoid

In Cockpit File Sharing, I enabled the ‘Shadow Copy’ and ‘MacOS Share’ options, since I want Shadow copy and also want the shares to work well with my iphone and mac. Here are the ‘advanced options’ for each share:

shadow:snapdir = .zfs/snapshot
shadow:sort = desc
shadow:format = autosnap_%Y-%m-%d_%H:%M:%S_frequently
vfs objects = catia fruit streams_xattr shadow_copy2
fruit:encoding = native
fruit:metadata = stream
fruit:zero_file_id = yes
fruit:nfs_aces = no

Of note, the shadow:format means it will be taking Sanoid’s frequently snapshots as the source of Shadow Copy. This works well, I am pleased.

Mellanox NICs with VLAN-Aware Bridges on Linux

Thu, 14 Sep 2023 01:00:05 -0300

A Discord member of mine came to me with an interesting problem - enbling the VLAN-aware bridge in Proxmox would cause all network traffic on the physical card to stop, entirely. Definitely a frustrating issue, especially since the kernel logs made no sense.

The Problem

Here’s what he sent from dmesg:

[   32.732509] mlx5_core 0000:19:00.1: mlx5e_vport_context_update_vlans:179:(pid 13470): netdev vlans list size (4080) > (512) max vport list size, some vlans will be dropped
[   32.735782] mlx5_core 0000:19:00.1: mlx5e_vport_context_update_vlans:179:(pid 13470): netdev vlans list size (4081) > (512) max vport list size, some vlans will be dropped
[   32.739011] mlx5_core 0000:19:00.1: mlx5e_vport_context_update_vlans:179:(pid 13470): netdev vlans list size (4082) > (512) max vport list size, some vlans will be dropped
[   32.742247] mlx5_core 0000:19:00.1: mlx5e_vport_context_update_vlans:179:(pid 13470): netdev vlans list size (4083) > (512) max vport list size, some vlans will be dropped
[   32.745550] mlx5_core 0000:19:00.1: mlx5e_vport_context_update_vlans:179:(pid 13470): netdev vlans list size (4084) > (512) max vport list size, some vlans will be dropped
[   32.748835] mlx5_core 0000:19:00.1: mlx5e_vport_context_update_vlans:179:(pid 13470): netdev vlans list size (4085) > (512) max vport list size, some vlans will be dropped
[   32.751987] mlx5_core 0000:19:00.1: mlx5e_vport_context_update_vlans:179:(pid 13470): netdev vlans list size (4086) > (512) max vport list size, some vlans will be dropped
[   32.755209] mlx5_core 0000:19:00.1: mlx5e_vport_context_update_vlans:179:(pid 13470): netdev vlans list size (4087) > (512) max vport list size, some vlans will be dropped

So somehow the mlx5 driver is unhappy with how many VLANs he’s tagging (all of them), and only wants to pass 512. But even when he reduced that number, it still didn’t work. For reference, this is what a /etc/network/interfaces looks like with this configuration in Proxmox:

#Network adapter
auto enp6s0f1np1
iface enp6s0f1np1 inet manual

#Proxmox bridge
auto vmbr0
iface vmbr0 inet6 static
        address 2001:db8:beef:cafe::6969/64
        gateway 2001:db8:beef:cafe::420
        bridge-ports enp6s0f1np1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

Note the bridge-vlan-aware and bridge-vids tags. In this case, the bridge supports VLANs, and VIDs 2-4094 are tagged (so vlan 1 is untagged). He tried reducing this number to 255, still no dice.

What made this even more interesting is we happen to have the exact same card! A Mellanox Connect-X4 dual 25G, both Dell branded as well! So I hopped on my test setup to figure out what the heck was going on, since my setup is working perfectly fine.

The Solution

After searching the forums, we found two things:

Nvidia are total assholes of the Linux community, their support forums just said ‘oh it looks like you’re using the Linux kernel drivers, we can’t help unless you use our proprietary drivers’. Linus was right to hate them! (“Nvidia has been one of the worst trouble spots we’ve had with hardware manufacturers”).
Someone managed to fix it despite their non-help by enabling promiscuous mode.

So now that the problem is psuedo-solved, why does my setup work when his does not when we are using identical cards with the same kernel version (Linux 6.2.16-12-pve and Proxmox VE 8.0.4)? Maybe my dmesg will help:

[   17.708193] device bond0 entered promiscuous mode
[   68.300117] device enp6s0f1np1 entered promiscuous mode

I’m using a bond, and the bond sets its slaves into promiscuous mode. In my case, I’m using an active-backup bond between my 25G link and the 1G on the motherboard as a backup. He’s not using a bond, so no promiscuous mode, so Mellanox driver does weird things and can’t handle vlans properly.

Here’s the solution: ip link set dev enp6s0f1np1 promisc on We can add that to our /etc/network/interfaces like this:

#Network adapter
auto enp6s0f1np1
iface enp6s0f1np1 inet manual
        #Enable promiscuous mode on the slave interface
        post-up ip link set dev $IFACE promisc on

Even More Solution

While this got the traffic flowing again, he was still getting a bunch of vlans list size > 512 errors after expanding the bridge-vids back to 4094. So, yes it fixed the initial issue, but it’s still trying to offload all vlans and not succeeding. If you still get that error in dmesg, then maybe this additional fix will work for you (it wasn’t necessary for me, with the bond, so the bond appears to also disable hardware offload of vlans): ethtool -K enp6s0f1np1 rx-vlan-filter off. If you just want to see the status, you can use ethtool -k enp6s0f1np1 (note the little k to read, big K to write). The new /etc/network/interfaces would be:

#Network adapter
auto enp6s0f1np1
iface enp6s0f1np1 inet manual
        #Enable promiscuous mode on the slave interface
        post-up ip link set dev $IFACE promisc on
        #Disable RX VLAN filtering in hardware offload
        pre-up ethtool -K $IFACE rx-vlan-filter off

Hope the tip helps you!

Gitea: Easy Self-Hosted Git Repositories!

Thu, 07 Sep 2023 01:00:05 -0300

Video

Installation

I’m using an LXC container in Proxmox running Debian 12. You’re free to use any other Debian 12 system, and the instructions should still work. It’s not particularly resource intensive, but you can monitor it to see if you need to increase the RAM/CPU allocations. I also added a second mount point to /var/lib/gitea, which is where all of the Gitea data will be stored. This just makes it easier to put the data and OS on separate storage locations, restore the entire Gitea install on another system later, or back it up separately.

Here are the commands to setup Gitea. Make sure you go to the download site and get the link to te latest version. The download is the binary (it’s a Go static binary), so there’s nothing to unzip, and no file extension. You probably want linux-amd64.

#Install git
apt update && apt install git -y
#Get the correct download link for the latest version
wget https://dl.gitea.com/gitea/1.20.3/gitea-1.20.3-linux-amd64
#Move the binary to bin
mv gitea* /usr/local/bin/gitea
#Make executable
chmod +x /usr/local/bin/gitea
#Ensure it works
gitea --version
#Create the user/group for gitea to operate as
adduser --system --group --disabled-password --home /etc/gitea gitea
#Config directory was created by adduser
#Create directory structure (mountpoint should be /var/lib/gitea)
mkdir -p /var/lib/gitea/{custom,data,log}
chown -R gitea:gitea /var/lib/gitea/
chmod -R 750 /var/lib/gitea/
chown root:gitea /etc/gitea
chmod 770 /etc/gitea

After that, we need a Systemd Service: (/etc/systemd/system/gitea.service)

[Unit]
Description=Gitea (Git with a cup of tea)
After=syslog.target
After=network.target

[Service]
# Uncomment the next line if you have repos with lots of files and get a HTTP 500 error because of that
# LimitNOFILE=524288:524288
RestartSec=2s
Type=notify
User=gitea
Group=gitea
#The mount point we added to the container
WorkingDirectory=/var/lib/gitea
#Create directory in /run
RuntimeDirectory=gitea
ExecStart=/usr/local/bin/gitea web --config /etc/gitea/app.ini
Restart=always
Environment=USER=gitea HOME=/var/lib/gitea/data GITEA_WORK_DIR=/var/lib/gitea
WatchdogSec=30s
#Capabilities to bind to low-numbered ports
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

Then run it:

systemctl daemon-reload
systemctl enable --now gitea

Now you can access it via :3000 to do the intial setup.

Configure HTTPS (Self-Signed)

And finally, configure Gitea to use HTTPS and the usual ports (80/443) using a self-signed cert (or one you provide, old-school) by editing /etc/gitea/app.ini. I’ve provided a diff below, the +- indicates what lines to add and remove.

 [server]
+PROTOCOL=https
+REDIRECT_OTHER_PORT=true
+CERT_FILE = /etc/gitea/cert.pem
+KEY_FILE  = /etc/gitea/key.pem
 SSH_DOMAIN = gitea.palnet.net
 DOMAIN = gitea.palnet.net
-HTTP_PORT = 80
+HTTP_PORT = 443
 ROOT_URL = https://gitea.palnet.net/
 APP_DATA_PATH = /var/lib/gitea/data
 DISABLE_SSH = false

And then generate a self-signed certificate and restart the server:

#Cd to the gitea directory
cd /etc/gitea
#sign cert
gitea cert --host teapot.apalrd.net
#Give gitea user read permissions
chown root:gitea cert.pem key.pem
chmod 640 cert.pem key.pem
#Restart gitea
systemctl restart gitea

To temporarily ignore certificates in Git (for testing), you can use the option -c http.sslVerify=false to git.

Configure HTTPS (Let’s Encrypt)

To use Let’s Encrypt you need a few different options in /etc/gitea/app.ini:

 [server]
+PROTOCOL=https
+REDIRECT_OTHER_PORT=true
+ENABLE_ACME=true
+ACME_ACCEPTTOS=true
+ACME_DIRECTORY=https
+ACME_URL=https://acme-staging-v02.api.letsencrypt.org/directory
+ACME_EMAIL=adventure@apalrd.net
 SSH_DOMAIN = gitea.palnet.net
 DOMAIN = gitea.palnet.net
-HTTP_PORT = 80
+HTTP_PORT = 443
 ROOT_URL = https://gitea.palnet.net/
 APP_DATA_PATH = /var/lib/gitea/data
 DISABLE_SSH = false

I have the URL set to the let’s encrypt staging repository as an example, you can use the directory URL of your own private CA, or leave it out entirely to use the let’s encrypt production server, which is the default if you leave the option out entirely. And then of course restart:

#Restart gitea
systemctl restart gitea

If Gitea can’t get a cert from Let’s Encrypt it will crash and you will have to look at journactl -xeu gitea to figure it out. Very frustrating. So make sure the Let’s Encrypt challenges will work (port 80 + 443 are correctly allowed by your network firewall)

Should I use TAPE BACKUP in 2023? LTO-5 Drive with Proxmox Backup Server

Mon, 21 Aug 2023 00:00:05 -0300

As promised in a previous video about my Proxmox Backup Server, I have a Quantum LTO-5 tape drive that I’m going to try and use to implement a proper 3-tier backup strategy with offsite tapes.

I’m currently using Linode’s object storage for backing up my personal data (~200G), and not backing up the video files outside of the two existing copies (on the storage server and the PBS server). With the affordability of tapes, I can keep the video files and personal data off-site reliably.

MOVING My Website from Static Hosting to Caddy!

Fri, 28 Jul 2023 01:00:05 -0300

I’ve been using Linode’s object hosting for my website for ~2 years now, and it’s time for a change. I’m not unhappy with Linode, but object hosting isn’t for me any more and I’d like to move up to a virtual private server. Object hosting is a fantastic way to get started with a static website for a low cost, but I want better backend analytics and more control of the whole process, so I’m setting up my own VPS using Caddy. At the same time, I’m setting up a privacy-respecting analytics system (Goatcounter) to read the web server log files and tell me how many viewers are enjoying my website. So, come along on this adventure!

Video

Caddyfile

Caddyfile is a super simple syntax, easy to understand and use. Default options are very good for most people.

# The Caddyfile is an easy way to configure your Caddy web server.
www.apalrd.net {
        # Set this path to your site's directory.
        root * /var/apalrd-net/public

        # Enable the static file server.
        file_server

        # Allow gzip encoding
        encode gzip zstd

        # Use TLS
        tls "adventure@apalrd.net"

        #Log to the usual place
        log {
                output file "/run/access/access-www-apalrd-net.log" {
                        roll_keep_for 1d
                        roll_size 10MiB
                }
                format transform `{request>remote_ip} - {request>user_id} [{ts}] "{request>method} {request>uri} {request>proto}" {status} {size} "{request>headers>Referer>[0]}" "{request>headers>User-Agent>[0]}"` {
                        time_format "02/Jan/2006:15:04:05 -0700"
                }
        }
}

#Global redirects
apalrd.net:80 {
        redir https://www.apalrd.net{uri} permanent
}
apalrd.net:443 {
        redir https://www.apalrd.net{uri} permanent
        tls "adventure@apalrd.net"
}


#Stats server redirects to Goatcounter
stats.apalrd.net {
        reverse_proxy localhost:8081
        tls "adventure@apalrd.net"
        encode gzip zstd
}
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

Caddy Service Override

### Editing /etc/systemd/system/caddy.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Service]
#Add a directory 'access' for runtime, systemd will add it to /run and deal with permissions
RuntimeDirectory=access
#Replace Exec with the /usr/local/bin versions
ExecStart=
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --force

### Lines below this comment will be discarded

I need to reduce the lifetime of these log files so they don’t continue to accumulate on the system, since IP addresses are considered personally-identifying information by modern privacy standards, even if the US where I’m located doesn’t actually have any sort of protections for personal information. The log files need to include the IP address so Goatcounter can feed it into a geo-IP database to guess the country of the viewer, and after that they are discarded. Once the data makes it to the Goatcounter database it’s been sufficiently anonymized that it’s no longer sensitive.

I chose to create a directory in /run for caddy, which is surprisingly easy with Systemd. I’m always amazed at how much of Linux relies on systemd and how good it is at well really everything it touches. Since /run is a RAM disk on modern Linux systems, I don’t have to worry about data encryption at rest, and the backup system will skip over mount points by default so the entire /run disk won’t end up in backups. Caddy will prevent the logs from getting over 10MB, and Goatcounter should be watching for changes every few seconds.

Goatcounter Service

Now I just need a systemd service for Goatcounter itself, plus the import daemon which I made as a template to make my life easier when I end up adding that link shortener someday. Let me tell you, systemd units are wildly powerful. I also took a moment to appreciate the Goatcounter developer’s shared hate for Docker.

Goatcounter main service: /etc/systemd/system/goatcounter.service:

[Unit]
Description=Goatcounter statistics system
After=network-online.target

[Service]
User=www-admin
Group=www-data
ExecStart=/usr/local/bin/goatcounter serve -listen localhost:8081 -db sqlite3:///var/apalrd-stats/db/goatcounter.sqlite3 -tls none
Restart=always

[Install]
WantedBy=multi-user.target

Logging service: /etc/systemd/system/goatcounter-logs@.service:

[Unit]
Description=Goatcounter statistics system
After=network-online.target caddy.service

[Service]
#Obviously put in your own API key, you get it from your user account in Goatcounter
#Combined log format is Apache style, Common Log Format + Referrer + User-Agent
Environment=GOATCOUNTER_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
User=caddy
Group=caddy
ExecStart=/usr/local/bin/goatcounter \
	import \
	-follow \
	-format=combined \
	-site="https://stats.apalrd.net" \
	-exclude 'path:glob:/assets/*' \
	-exclude 'status:404' \
	-exclude redirect \
	-exclude 'path:glob:/img/*' \
	/run/access/access-%i.log

[Install]
WantedBy=multi-user.target

I’m storing access totals for each page on the site, plus a breakdown by operating system, browser, and country for the site as a whole. It’s free to view at stats.apalrd.net, so enjoy!

Web Rebuild Service and Timer

Last step is to make the website periodically check for updates and re-sync. I’ve chosen to have this happen daily. I wrote a systemd service file that regenerates the site when started, basically by doing a git pull followed by a hugo -v. Of course, to trigger this remotely over SSH I just need to tell systemctl to run this service. How easy!

[Unit]
Description=Rebuild Website Tasks

[Service]
Type=oneshot
#Run as www-admin which has permissions to this
User=www-admin
Group=www-data
WorkingDirectory=/var/apalrd-net/
#Run git-pull here
ExecStart=git pull
#Next run Hugo
ExecStart=hugo -v

[Install]
WantedBy=default.target

Running systemctl start webrebuild gives me this beautiful log in journalctl:

Jul 24 23:20:40 web-ash1 git[34589]: Already up to date.
Jul 24 23:20:40 web-ash1 hugo[34590]: Start building sites …
Jul 24 23:20:40 web-ash1 hugo[34590]: hugo v0.111.3+extended linux/amd64 BuildDate=2023-03-16T08:41:31Z VendorInfo=debian:0.111.3-1
Jul 24 23:20:40 web-ash1 hugo[34590]: INFO 2023/07/24 23:20:40 syncing static files to /
Jul 24 23:20:41 web-ash1 hugo[34590]:                    | EN
Jul 24 23:20:41 web-ash1 hugo[34590]: -------------------+------
Jul 24 23:20:41 web-ash1 hugo[34590]:   Pages            | 251
Jul 24 23:20:41 web-ash1 hugo[34590]:   Paginator pages  |  46
Jul 24 23:20:41 web-ash1 hugo[34590]:   Non-page files   | 254
Jul 24 23:20:41 web-ash1 hugo[34590]:   Static files     |  17
Jul 24 23:20:41 web-ash1 hugo[34590]:   Processed images |   0
Jul 24 23:20:41 web-ash1 hugo[34590]:   Aliases          |  62
Jul 24 23:20:41 web-ash1 hugo[34590]:   Sitemaps         |   1
Jul 24 23:20:41 web-ash1 hugo[34590]:   Cleaned          |   0
Jul 24 23:20:41 web-ash1 hugo[34590]: Total in 771 ms
Jul 24 23:20:41 web-ash1 systemd[1]: webrebuild.service: Deactivated successfully.

Now that the service works, I can add a timer unit to trigger it regularly.

[Unit]
Description=Rebuild website daily
RefuseManualStart=no
RefuseManualStop=no

[Timer]
#Run 180 seconds after boot for the first time
OnBootSec=180
#Run at 9am EST / 4am UTC daily
OnCalendar=*-*-* 04:00:00
Unit=webrebuild.service

[Install]
WantedBy=timers.target

End

I hope you enjoyed this overview of how my website is hosted in the background. Sanitized versions of the configs are all posted on my blog if you are looking to replicate this setup. Of course, feel free to reach out to my Discord if you’d like ot hang out with like minded people, and as always, I’ll see you on the next adventure!

Layer 4 vs Layer 7 Reverse Proxies: Using HAProxy to front Web Services (for IPv4 to v6 Transition)

Thu, 20 Jul 2023 01:00:05 -0300

A common challenge in web design and network architecture is grouping multiple web services in a single host, or behind a single IP address. This is especially true with IPv4 due to the scarcity of addresses. The solution to this is a reverse proxy or load balancer. Essentially, this takes connections from clients and dispatches them to the correct server based on the domain name or URL in the request. In this video, I’m going to explain what a layer 4 or layer 7 load balancer even is, and setup a layer 4 example using HAProxy. So come along on this adventure!

Video

Transport Layers of HTTP

So to understand load balancers and reverse proxies, let’s take a look at what a normal modern web session looks like:

Example: A web session inside a TLS tunnel inside a TCP socket

With HTTP/1.1, the HTTP commands are tunneled inside a TLS session, and the TLS session is itself inside a TCP socket. To setup the connection, the web browser sends out a TCP SYN and the 3-way handshake is done, and once the tunnel is open the client can send the Client Hello message to begin establishing the TLS session. Once that’s done, within the TLS session the HTTP commands can be sent and information can be exchanged.

In this case, TCP is our layer 4 transport protocol, establishing a long-lived tunnel out of layer 3 IP packets. Within the transport is our TLS sesssion and HTTP session, which are generally grouped together as the application layer, or layer 7.

Our proxy will sit somewhere in this chain and intercept messages so we can modify them or dispatch them to the correct back end server. In short, if we are interrupting the TLS session, we are proxying at layer 7, and if we are passing through the TLS messages unmodified we are proxying at layer 4. Since all of this happens at layer 4 and above, we can also use a proxy to change the layer 3 protocol from IPv4 to IPv6, meaning we can keep our internal network entirely IPv6 as long as the first proxy the client hits supports IPv4.

Example: Layer 4 TCP proxy

So with layer 4, TCP comes in, TCP goes out, and the only thing we can see is the Client Hello before everything is encrypted. Thankfully the Client Hello includes the SNI, or the domain name of the site, but not the full URL. This is just enough information for us to choose a server based on its domain, and let the server handle the TLS side of the transaction.

Example: Layer 7 HTTPS proxy

But with a layer 7 proxy we are intercepting the TLS sessions. At a minimum, our layer 7 proxy needs to have the TLS certificate and key so users can authenticate to us. We also have the opportunity to inspect the entire HTTP request, convert from HTTP/1.1 to HTTP/2 or HTTP/3, and possibly connect to the backend server via a separate TLS session or via unencrypted HTTP. Since we can inspect the entire URL in every query, we can redirect different paths to different servers, rewrite paths, rewrite responses, cache things, and basically change whatever we want. Even if we aren’t changing or caching anything, we can also use this architecture to move cryptography off of the application servers, helping spread the load around the whole system.

While either of these could be called a ’load-balancer’ or ‘reverse-proxy’, a layer 4 proxy would be more likely to be called a load balancer, while a layer 7 proxy would be more likely to be called a reverse proxy.

In case you’re confused on why this is a ‘reverse’ proxy, a normal or ‘forward’ proxy would act on the other side, between a group of clients and the internet. This can also be a layer 4 or layer 7 proxy, and is often used to give clients heavily filtered, monitored, or cached access to the internet.

HAProxy Setup

For my setup, I’m using a layer 4 TCP proxy for secure connections and a layer 7 HTTP proxy for insecure connections. I could also just have a blanket redirect in the proxy that tells all HTTP users to go to HTTPS, so I threw in the config for that bit too. My backends are all IPv6, so IPv6 clients can go straight to the web server while IPv4 clients all have to go through the proxy since they are all sharing the same v4 address.

Installing HAProxy

This one’s crazy simple. We just install it from apt!

apt update && apt install haproxy -y

Configuring HAProxy

Feel free to pick and choose the config snippets from the next section as you’re building your config. Some of these examples are mutually exclusive, so read the notes carefully! The file is /etc/haproxy/haproxy.cfg by the way.

Default Config

I left the default config from Debian alone, so here it is in case you’re curious. I added all of my config after this.

#Default global configuration from Debian upstream
global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

#Defaults, also from Debian upstream
defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

HTTP Config - Redirect to HTTPS

This is an HTTP config which redirects everything to HTTPS, so we don’t also need to proxy HTTP/1.1 over port 80

# Listen on port 80, layer 7 (HTTP)
# Redirect everything to https
# That leaves the client to reconnect properly,
# and means we don't need to proxy HTTP, just HTTPS
frontend www
        mode http
        bind :80
        http-request redirect scheme https

HTTP Config - Layer 7 Proxy

Here’s a version of the HTTP config which actually does proxy port 80, if you want to continue to support non-TLS connections and/or HTTP-01 ACME challenges (Caddy supports TLS-ALPN-01 so it can work over 443 without 80 proxied).

# For port 80, we can do layer 7 (HTTP)
# Since it's insecure, there's no reason to do layer 7
# since there are no certs or encryption to interrupt
frontend www
        mode http
        bind :80
        # We are building the name of the backend from the 'host'
        # field in the request plus the literal '_http'
        # See backends for an example of how to name them
        use_backend %[req.hdr(host),lower,word(1,:)]_http

# Backends for HTTP
backend test1.apalrd.net_http
        mode http
        server test1_http 2601:40e:69:69:0:0:0:feed:80
backend test2.apalrd.net_http
        mode http
        server test2_http 2601:40e:69:69:0:0:0:beef:80

HTTPS Config - Layer 4 SNI Proxy

And of course, the layer 4 SNI proxy for TLS works similarly, but at layer 4 instead of layer 7

# For port 443, we want to do a TCP proxy so
# we don't have to terminate the TLS session. 
# Since we are only doing this for IPv4 clients,
# We don't want the private keys both here (for v4)
# and on the real server (for v6).
frontend www-tls
        # Layer 4 (TCP) mode
        mode tcp
        # Use TCPlog mode instead of HTTPlog
        option tcplog
        # Listen on TCP 443 (HTTP/1.1 and HTTP/2)
        bind :443

        # Wait for SSL Hello before forwarding
        tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }

        # Select backends for each server
        # Similar method to above, but using '_tls' on the end
        use_backend %[req_ssl_sni,lower,word(1,:)]_tls

# Backends for TLS servers
backend test1.apalrd.net_tls
        mode tcp
        server test1_tls 2601:40e:69:69:0:0:0:feed:443
backend test2.apalrd.net_tls
        mode tcp
        server test2_tls 2601:40e:69:69:0:0:0:beef:443

Logging

HAProxy uses rsyslog by default to log, so we won’t get a log file unless we install that package too. So, let’s do that:

apt update && apt install rsyslog -y
systemctl enable --now rsyslog

Now you should find an /var/log/haproxy.log file for you to explore! Be aware that it may fill up and you should probably take care of that.

The Power of Zero-Trust Architecture: Building a Secure Internal Network with Nebula

Thu, 13 Jul 2023 01:00:05 -0300

Imagine if we could establish a level of trust that in our network, we can verify with certainty that a computer really is who it says it is. By bringing mutual authentication and trust into networking, we can better make security decisions on when connections should be allowed. This can enable our services to talk to each other securely over the global internet, and reduce the dependence on a a trusted perimeter. This mututal trust is the foundation of a zero-trust security model. In this video, I’m going to walk through the basics of zero-trust security, the choices I’ve made to implement it in my own network. One of those choies is Nebula, an open-source zero trust overlay network designed for highly scalable distributed networks. Come along on this adventure!

Video

This absolutely beautiful image comes from NASA and the James Webb Space Telescope. Go Science!

Zero Trust Architecture

If you’re into tech buzz words, you might have heard of a ‘zero trust architecture’. A lot of companies are certainly pushing products in the space which take different approaches, but to understand it let’s first look at an example network that is certainly not zero-trust.

Example: A standard firewall router

In this example, we consider the firewall to be our security plane. Users inside the network get access directly to services. In a lot of organizations, they are even running without encryption on internal services like this! I made a video on fixing that internal TLS cert issue, card up there. In general, we trust that devices belong on the network becuase they are within the security plane behind the firewall, but we probably also have some token of security like basic passwords to access systems.

But what happens when an attacker is able to compromise a single laptop? Now they can hunt every single service we have running, bang at SSH to see if you actually changed the password on your Raspberry Pi’s, see all of your unencrypted security cameras which you thought were local only, etc. And all from one compromised machine! If only there was a way we could limit the blast radius of a single machine down to just the services it actually needs to talk to, hopefully reducing the chance that an attacker is able to spread laterally through our network.

Example: Siloing services and clients using subnets

If we want better network security, we start to implement more subnets and VLANs. We are relying almost entirely on the firewall to filter what ports users should have access to, so we might need a whole lot of subnets to group everything by membership. This forces a ton of traffic to go through the firewall if we want really fine-grained firewall rules, which means we need a beefy firewall to handle all of the traffic across our network in one place.

Example: Some clients via a VPN

When we start asking for remote access, this gets even more complicated. We start opening holes in our security plane to allow remote users in, via VPN tunnels. Usually this is secure, but it expands our security wall even further. If we are using cloud services, we might be spinning up new cloud instances of routers just to tunnel traffic via a VPN, so everything is within our secure perimeter.

As an extreme example, we can connect every single device directly to the firewall, and force all traffic to go through it so we can filter it all. We might be able to do this on high end switches, but it’s an expensive proposition on a network of really any scale.

Example: Each node has a tiny firewall

So zero-trust brings a simple distributed solution. We get rid of the central firewall, and move it to every single node. Okay, maybe we don’t get rid of it entirely, but we stop asuming that the firewall is a perfect line of defense. Now, the key tenet of zero trust is that we mutually authenticate both sides of every single connection before exchanging any data. Since we’ve mutually authenticated each other, we can rely on this strong cryptographically verified identity to know what should have access to what, without bringing in physical network changes.

My Zero Trust Setup

Now, all of this sounds fantastic in concept. But, by moving the firewall to every node, we’ve created a bit of a configuration management hell for ourselves. How do we know what to trust? Who should be able to access what services?

An overview of my network architecture

This is the key question solved by zero-trust solutions, and each vendor or open source project will have a different answer. Here’s the solution I’m working with for my network currently:

For services to talk to other services, and for admin access to management UIs that aren’t for end-users, I’m using Nebula. This provides an overlay network with secure, authenticated access to anything at layer 3. That’s covered in this video.
For services which users need to access, I’m going to deploy mutual TLS. This will authenticate users devices via hardware certificates. This gives me confidence that a user is logging in from their own device or security key without needing to install any software on their device. I’m actually already using this, but I’m still working on a good workflow for enrolling users so this is a topic for a future video.
Of course, there are some cases where neither of these are a good choice, so I can continue to do segmentation in the traditional sense in parallel to the more modern approaches. Hopefully I can phase them out over time. In particular, I’m going to keep zero trust out of my security camera and home automation devices, as they are already struggling to implement IP at all. I’m also not going to use zero trust in the Ceph storage network, since it can’t afford to take the performance hit.

I’m not going to setup a whole-network remote access VPN such as Wireguard, Tailscale, or OpenVPN. Implemented properly, a zero trust solution can remove the need for site to site VPNs between homelabs, datacenters, and cloud providers and also obsolete SD-WAN solutions. I will still use these things to give me unfiltered internet access on the go or avoid geo restrictions, but not as a security mechanism.

What is Nebula?

So when you think of a VPN, you probably think of a point to point connection, between you and a cloud provider, you and your home network, or office, or something like that. Nebula creates individual VPN tunnels directly between nodes on demand, allowing anyone participating in your private mesh to access services. To properly deploy it, you don’t just open up your whole network to Nebula and bring back the perimeter firewall model, you install it individually on all of the hosts you want to communicate with.

Once everyone has Nebula installed, they don’t need any port forwarding to work. The Nebula transport works over both IPv4 and IPv6. Clients start with the public address or domain name of one or more Lighthouse nodes, and from the lighthouse they can discover all of the possible addresses of all other nodes, to establish secure tunnels on demand.

Nebula also integrates a role-based firewall at every single node, where we can filter packets coming in and out of the node based on the group membership of the other node in the tunnel. Since the other nodes identity and group membership is cryptographically verified, no central source of authentication truth is needed, and the network can function without trusting anything but the root certificate.

Example Network used in this tutorial

Standing Up a Network

Finally understand what the big deal is with zero trust and Nebula? Lets’s stand up a network! For this example, I’m going to use my laptop, a Proxmox server, and a lighthouse. The lighthouse is the only node that needs a public IP address or port forward, and I’m going to set it up in a cheap Nanode instance on Linode. Why Linode? It’s what I’m already using. Hetzner is cheaper.

Mapping Out the Network

Before we start signing certs, we need to give each node a fully qualified domain name and IP address on the overlay. Nebula currently only supports IPv4 within the overlay, so we’re going to set aside a nice large space for all of our hosts to get randomly assigned addresses. Since the IPv4 world is full of bad practices, I’ve decided to go ahead and continue that and chose the perfect prefix for my network: 42. As you know, it’s the answer to life, the universe, and everything, so hopefully nobody minds that my private network is 42.69/16. For individual node addresses I’m using a random number generator in most cases.

(In case you are actually thinking about using IPv4 squat space, you might want to read this article on the dangers of squat space)

Unlike normal IPv4, each node is a /32 and the network is fully routed, so there are no concerns about subnetting it down and losing addresses everywhere to the network and broadcast and gateway. We truly have 65k host addresses.

I also decided to place all of my nodes under the domain neb.apalrd.net, which lets me delegate the Nebula network as an authoritative name server for that subdomain. It also makes it clear that addresses are within the nebula network.

Install Nebula

First, we need to get the Nebula binaries for our operating system. It’s the same process for every OS, so you’ll need to do this on the laptop, server, lightouse, all of them.

#Copy the link from the lates release on Github: https://github.com/slackhq/nebula/releases/
wget https://github.com/slackhq/nebula/releases/download/v1.7.2/nebula-linux-amd64.tar.gz
#Untar it into /usr/local/bin so it's accessible (it's just two binaries, nebula and nebula-cert)
tar -xzf nebula-linux-amd64.tar.gz -C /usr/local/bin

If you’re in an LXC Container on Proxmox, you additionally need permissions to create your own tun devices, and need a privilaged LXC container. Add this to the container config, which will be /etc/pve/lxc/<vmid>.conf:

lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir

If you’re able to get permissions right on an unprivilaged container to use tun devices, let me know how and I can update the guide. I’ve already spent a long time on this lol

Setup the Certificate Authority

You can do this on any machine, even one that isn’t part of the Nebula network. You just need a way to copy the certs out later. They are really tiny, copy and paste works fine. You should NOT do this on the lighthouse, since that’s the most likely system to be compromised. I’m doing it on my laptop.

This one is way easier than the x509 stuff we did in OpenSSL! Do this on whatever node you want to keep your key on, it can be any node.

#Name it whatever you want
nebula-cert ca --name "ApalrdsAdventures2023"

Now keep ca.key very private, and come up with a way to distribute ca.crt to all nodes in the cluster. I use cat and copy and paste the contents. Ansible would also probably make a good choice for this. Also remember that the CA expires after 8760h0m0s, but you can override this by setting the -duration <time> flag with hours/mins/secs. Client certs cannot last loner than the CA cert.

Create and Sign Node Certificates

In this example, I’m creating and signing a cert for my laptop, my Proxmox VE cluster, and the lighthouse.

When you sign a Nebula cert, you are including group membership, hostname, and address in the cert. All of this is verified by other nodes, and is used later by DNS and the firewall, so get it right!

So anyway here’s my rapid fire commands to generate the certs I needed (obviously change this for your own network)

nebula-cert sign -name "lighthouse1.neb.apalrd.net" -ip "42.69.0.1/16"
nebula-cert sign -name "spaceship3.neb.apalrd.net" -ip "42.69.73.46/16" --groups "user,admin"
nebula-cert sign -name "pvering1.neb.apalrd.net" -ip "42.69.212.1/16" --groups "hypervisor,dev"
nebula-cert sign -name "pvering2.neb.apalrd.net" -ip "42.69.212.2/16" --groups "hypervisor,dev"
nebula-cert sign -name "pvering3.neb.apalrd.net" -ip "42.69.212.3/16" --groups "hypervisor,dev"
nebula-cert sign -name "pvering4.neb.apalrd.net" -ip "42.69.212.4/16" --groups "hypervisor,dev"
nebula-cert sign -name "pvering5.neb.apalrd.net" -ip "42.69.212.5/16" --groups "hypervisor,dev"

In the future once the network is stood up, it’s more secure to generate the private key on the device and transfer the public key to the certificate authority to sign it. For now, I’m generating them all and distributing them during the initial setup, but will switch to the more secure process once I’ve gotten the basics functional.

Also be very aware that your root key expires in a year, and your node keys by default expire a second before the root! So an automated process for rotating certs would be good. Something like Ansible, rotate them in your periodic update script or whatever. You can set the duration to anything from zero to the expiration of the CA’s cert using the -duration flag similar to above. This post is already too long to set that up too.

Anyway, the on-device keying process would be:

#Generate key on device
nebula-cert keygen -out-key laptop.key -out-pub laptop.pub

#Transfer laptop.pub to CA to be signed

#Sign cert on CA
nebula-cert sign -in-pub laptop.pub -name "spaceship3.neb.apalrd.net" -ip "42.69.73.46/16" --groups "user,admin"

#Transfer new crt back to device which generated the key

Setup the Lighthouse

For this example, I’m running it on a Debian 12-based Nanode at Linode, with a public IPv4 and IPv6, at the name ls1.apalrd.net in DNS (both A and quad-A records), plus an NS record at neb.apalrd.net which points back to the lighthouse so my Nebula network resolves via public DNS (how fun!). All of these will be gone by the time the video comes out.

Since we want DNS to work, you might need to disable systemd-resolved stub listener (which would have taken port 53) before we setup the lighthouse, and here’s how to do that on Debian. On some systems it might already be disabled, so if you get any errors you can safely ignore them here.

#Write out a conf file change for resolved
echo DNSStubListener=no >> /etc/systemd/resolved.conf
#Restart service
systemctl restart systemd-resolved

For this node, we need the binaries, crt and key, the ca.crt (but NOT key), a simple lighthouse config, and a systemd service. Rename the crt and key to host.crt and host.key, and put them along with ca.crt in /etc/nebula.

Here’s the config (/etc/nebula/config.yaml):

# PKI paths
pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/host.crt
  key: /etc/nebula/host.key

# Port settings
listen:
  #Default for this key is 0.0.0.0 which is v4-only and stupid
  host: '[::]'
  port: 4242

# No static hosts for lighthouses
static_host_map:

#Lighthouse settings
lighthouse:
  am_lighthouse: true
  # Enable serving DNS to anyone (Even our external IP)
  serve_dns: true
  dns:
    #Default for this key is 0.0.0.0 which is v4-only and stupid
    host: '[::]'
    port: 53

#Firewall settings
firewall:
  outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

  inbound:
    # Allow icmp between any nebula hosts
    - port: any
      proto: icmp
      host: any
    # Allow DNS incoming since we are serving DNS at this lighthouse
    - port: 53
      proto: udp
      group: any

And the systemd service (/etc/systemd/system/nebula.service):

[Unit]
Description=Nebula overlay networking tool
Wants=basic.target network-online.target nss-lookup.target time-sync.target
After=basic.target network.target network-online.target
Before=sshd.service

[Service]
SyslogIdentifier=nebula
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/nebula -config /etc/nebula/config.yml
Restart=always

[Install]
WantedBy=multi-user.target

And finally, the commands to start it up (assuming you copied the cert/config files into /etc/nebula):

#Reload the daemon files
systemctl daemon-reload
#Start and enable the service
systemctl enable --now nebula

Setup the Client on Linux

In this example, I’m using a Proxmox VE 8.0 system. Again, need to copy our host.crt, host.key, and ca.crt to /etc/nebula.

The config for this node (/etc/nebula/config.yml):

# PKI paths
pki:
  ca: /etc/nebula/ca.crt
  cert: /etc/nebula/host.crt
  key: /etc/nebula/host.key

# Need a static host map, using the DNS name of the lighthouse
static_host_map:
  #Put all of your lighthouses here
  '42.69.0.1': ['lighthouse1.palnet.net:4242']

#This is completely undocumented
#static_map is how to interpret static_host_map
#It defaults to ip4, trying to connect to the lighthouse
#using only ipv4. This sorta-kinda makes sense since the node
#knows its own public v6 already but not its public v4 (Via NAT)
#so connecting to the lighthouse via v4 lets it learn that
#For ipv6-only hosts, change to `ip6` instead
static_map:
  network: ip4

# Lighthouse config for clients
lighthouse:
  hosts:
    - '42.69.0.1'

#Listen
listen:
  #Default for this key is 0.0.0.0 which is v4-only and stupid
  host: '[::]'
  #Port of 0 means randomly choose, usually good for clients
  #Want to set to 4242 for relays and lighthouses
  port: 0

#Firewall settings
firewall:
  outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

  inbound:
    # Allow icmp between any nebula hosts
    - port: any
      proto: icmp
      host: any

For the Systemd service, I used the same one as the lighthouse, so the same file / commands apply here.

If you want to put the keys in the config file directly instead of on-disk (might be easier for laptops or something idk) you can do it like this:

pki:
  ca: |
    -----BEGIN NEBULA CERTIFICATE-----
    CkgKFlRoZSBPbmUtSG91ciBOZWJ1bGEgQ0Eo/pL7jAYwjq/7jAY6IDIi7yqkRV9F
    1+tozxvnHCmuuuwdArt7YbMMdCR4AYm/QAESQHBitbcetbJ06RQckqGi+hXJXd/U
    TXKEul4TxP4Qxmd7g+cHDE6oYZhRwup+1xg/Sv9bMg2E2/LNXKV3rNf1Yw8=
    -----END NEBULA CERTIFICATE-----

If you have multiple root certificates, you just add them together in one file and it will trust them all. If you have automation on your servers, it would be a good idea to rekey the root certificate periodically, and this can be used to keep old certs valid during the transition.

Relaying

Now this all works well, but what if my laptop roams to some public hotspot without IPv6? My servers only have IPv6, so they can’t peer with my laptop. The lighthouse can, since it has both v4 and v6, but that doesn’t help. So let’s go back to the configs for the lighthouse and server and enable peers of the server to relay through the lighthouse.

When relaying, the destination node must specify in its configuration which nodes are allowed to relay on its behalf, and the clients will use the list of allowed relays for a destination to try and connect if they are unable to connect directly. So if we are getting around NAT traversal that’s only a problem in one direction, or only care about sessions established from one side, we don’t need to add the relay to every single file. It’s still easy enough to add to everyone’s config.

We also need to specify in the node configuration if the node is allowed to be a relay, but relays don’t actually care who they are relaying for, just that they are allowed to relay. So, the lighthouse is going to be my relay and my Proxmox systems (which are v6-only) are going to get the lighthouse added as an available relay. If you have troublesome nodes behind many layers of NAT the setup for them would be similar.

Change to the lighthouse config (add this to the end):

#I can also be a relay
relay: 
  am_relay: true

Change to the Proxmox server config (again add to the end):

#I can use the lighthouse as a relay
relay:
  relays: 
   - 42.69.0.1
  am_relay: false
  use_relays: true

Now, the laptop can connect happily to the server, via the lighthouse.

Security

Let’s see if we can understand how amazing the group-based distributed firewall is, let’s try a few examples. All of these need to go under the firweall.inbound section as one list, don’t add a bunch of firewall sections to one file.

Allow Anyone to access Web Server

In this case, anyone can access port 443, so we are only using Nebula for secure tunneling and relying on the server for user authentication. Also useful for services that are network-wide, like DNS. In this case, we use the host field and it’s special value any. If we want to allow HTTP/3 (QUIC), we can set proto to any instead of tcp.

#Firewall settings
firewall:
  inbound:
    # Allow anyone to web server
    - port: 443
      proto: any
      host: any

Allow Admins to Web UI

Pretty basic rule, server-side we specify the port and groups which can access that port. In this case, Proxmox only does HTTP/1.1, so it’s TCP only.

#Firewall settings
firewall:
  inbound:
    # Allow admins to the web UI of Proxmox systems
    - port: 8006
      proto: tcp
      group: admin

Allow Admins OR Breakglass to Web UI

When we want to allow members of EITHER group, we write the rule twice. So, this would be the new rule:

#Firewall settings
firewall:
  inbound:
    # Allow admins to the web UI of Proxmox systems
    - port: 8006
      proto: tcp
      group: admin
    # Allow breakglass to the web UI of Proxmox systems
    - port: 8006
      proto: tcp
      group: breakglass

Allow Admins from US Region to Web UI

When we want members to have BOTH groups, we write the groups as a yaml list. So, the rule:

#Firewall settings
firewall:
  inbound:
    # Allow admins from the us region to the web UI of Proxmox systems
    - port: 8006
      proto: tcp
      groups:
        - admin
        - region-us

Allow a Port Range

Similar to the single-port, we can specify a range of ports.

#Firewall settings
firewall:
  inbound:
    # Allow admins to the VNC port range
    - port: 5900-5999
      proto: tcp
      group: admin

Links and Conclusion

Here’s some links which might help you if you need more reference information:

Using Proxmox METRICS In Your Homelab

Fri, 07 Jul 2023 07:00:05 -0300

As an engineer, I LOVE looking at DATA! So today, I’m setting up my Proxmox system to push data to InfluxDB, where I can view it using Grafana. With this setup, I can keep track of how many resources all of my homelab services are using, which really helps when trying to size VMs, hosts, containers, etc.

Video

Install InfluxDB 2.0

See here for the basic instructions and updated download links. We don’t have sudo, so here’s a rewriting of it without sudo:

# We need a few things for this
apt install gpg apt-transport-https -y

# Load GPG fingerprint and debian repo from Influx
# influxdata-archive_compat.key GPG fingerprint:
#     9D53 9D90 D332 8DC7 D6C8 D3B9 D8FF 8E1F 7DF8 B07E
wget -q https://repos.influxdata.com/influxdata-archive_compat.key
#Verify integrity of key download
echo '393e8779c89ac8d958f81f942f9ad7fb82a25e133faddaf92e15b16e6ac9ce4c influxdata-archive_compat.key' | sha256sum -c
#Dearmor key into trusted.gpg.d
cat influxdata-archive_compat.key | gpg --dearmor > /etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg
#Add repo to influxdata.list
echo 'deb [signed-by=/etc/apt/trusted.gpg.d/influxdata-archive_compat.gpg] https://repos.influxdata.com/debian stable main' > /etc/apt/sources.list.d/influxdata.list

#Actually apt update and install (or update)
apt update && apt install influxdb2 -y

#Start the servyce
systemctl enable --now influxdb

If you want to check it, run systemctl status influxdb.

Now login to the host at port 8086 and initialize, setting the admin password here

Install Grafana

See Here for the basic instructions from Grafana

#Install the signing key
wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key

#Add repository to grafana.list
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" > /etc/apt/sources.list.d/grafana.list

#apt update and install (or update)
apt update && apt install grafana -y

#Start the server
systemctl enable --now grafana-server

Now login at port 3000 with admin/admin and change the password

Example Dashboard

I used this dashboard for my example, but I’d love to see any customizations or dashboards you make!

Using TLS

Since it’s a good idea to use TLS certificates but both InfluxDB and Grafana don’t automatically configure it and don’t generate self-signed certificates, it’s a bit of .. fun to get that setup. Here’s the guides from both Influx and Grafana on setting up their software with TLS certificates:

Self-Hosted TRUST with your own Certificate Authority!

Wed, 14 Jun 2023 01:00:05 -0300

TRUST. It’s what certificates are all about. How do we know that we can trust a server? We verify that the server has a certificate, and that the certificate is signed by someone we trust. That can be a well-known third party like Let’s Encrypt, or our own certificate authority. In this video, I’m going to cover the basics of setting up a root private key and signing certificates using OpenSSL, and running a certificate authority server. As a bonus, I’m using a Yubikey to store the certiicate authorities private keys, so they can’t be compromised without stealing the physical dongle (they CAN however be used to generate leaf certificates if the certificate authority is compromised). So follow along for a fun journey into the basics of setting up your public key infrastructure!

Video

A Bit About PKI

Public key encryption provides a secure method of transmitting data over insecure networks. It allows for secure communication by using a pair of keys: a public key for encryption and a private key for decryption. TLS (Transport Layer Security) certificates, which are based on public key encryption, ensure the authenticity and integrity of data exchanged between a server and a client. They provide trust and verification, protecting against unauthorized access, data tampering, and eavesdropping, thus establishing secure and encrypted connections.

2-layer vs 3-layer CA

2-layer
1. Root CA public trusted by users, private used to sign servers
2. Leaf CA generated via ACME ~daily from servers
3-layer
1. Root CA public trusted by users, private kept entirely offline
2. Intermediate CA public unused, private used to sign servers
3. Leaf CA generated via ACME ~daily from servers

Advantages to 2-layer:

Simple to setup
Single private key can be kept in multiple places (backups, HSMs, …)

Advantages of 3-layer:

Root CA can issue a CRL which can revoke intermediate certificates (although we will not do that in this tutorial!)
Intermediate CA can be issued for less time than root, so you can manually renew intermediate CA periodically so if it’s lost the time for exposure isn’t as high
Root CA private key can be kept entirely offline, and reissue intermediate CA certs without keeping the private key in multiple palces

We will be setting up a 3-layer CA where the root keys are generated by OpenSSL and kept entirely offline (how you do that is up to you, this tutorial is already long enough) and the intermediate certificates are kepy on the Yubikey and used to sign server and user certificates.

Certificate Authority in OpenSSL

First, we are going to setup the certificate authority and generate our most precious private keys. To do this, we can use any Linux system with OpenSSL, such as Debian or Alpine. We also need the yubikey manager package installed. On Debian you can install this through apt with apt install yubikey-manager.

You do NOT need to use the same system as your eventual Certificate authority to generate these private keys! You can use an ephemeral system like a live image, as long as you can copy off the resulting certificates (public keys) for the CA and the root public key somewhere safe for later. For ease, I’m going to create a new directory in /root/ca on my eventual CA system to house the certificates and then delete them once all is done and backed up safe. Make sure you update any paths on your own system if you’re using a USB drive for your private keys, or copy them out later.

This would also be a good time to set your Yubikey PINs. The defaults are what an idiot would use on their luggage.

Setup the OpenSSL CA Config

We don’t need an intermediate CA config file since we are just generating the private key to be used by Smallstep, so certificates won’t be signed by OpenSSL using the intermedate key.

Put the config file in /root/ca/root.cnf.

# OpenSSL root CA configuration file.

[ ca ]
# `man ca`
default_ca = CA_root

[ CA_root ]
# Directory and file locations.
dir               = /root/ca
certs             = $dir/certs
crl_dir           = $dir/crl
new_certs_dir     = $dir/newcerts
database          = $dir/index.txt
serial            = $dir/serial
RANDFILE          = $dir/private/.rand

# The root key and root certificate.
# Match names with Smallstep naming convention
private_key       = $dir/root_ca_key
certificate       = $dir/root_ca.crt

# For certificate revocation lists.
crlnumber         = $dir/crlnumber
crl               = $dir/crl/ca.crl.pem
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 25202
preserve          = no
policy            = policy_strict

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
# See the POLICY FORMAT section of `man ca`.
countryName             = match
organizationName        = match
commonName              = supplied

[ req ]
# Options for the `req` tool (`man req`).
default_bits        = 4096
distinguished_name  = req_distinguished_name
string_mask         = utf8only

# SHA-1 is deprecated, so use SHA-2 instead.
default_md          = sha256

# Extension to add when the -x509 option is used.
x509_extensions     = v3_ca

[ req_distinguished_name ]
# See <https://en.wikipedia.org/wiki/Certificate_signing_request>.
commonName                      = Common Name
countryName                     = Country Name (2 letter code)
0.organizationName              = Organization Name

[ v3_ca ]
# Extensions for a typical CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA (`man x509v3_config`).
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:0
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

And also don’t forget to create those directories:

mkdir -p /root/ca /root/ca/certs /root/ca/crl /root/ca/newcerts /root/ca/private
touch /root/ca/index.txt
echo 1420 > serial

Generate Root Certificates

Keep this VERY SAFE, preferably offline. Compromising the root_ca_key means any of your servers (and users with mutual TLS) can be impersonated.

# Generate a private key (needs a passphrase, don't forget the passphrase)
openssl genrsa -aes256 -out /root/ca/root_ca_key 4096

# Sign a 69-year (nice) certificate, to be used by clients mostly
openssl req -config /root/ca/root.cnf -key /root/ca/root_ca_key -days 25202 -new -x509 -sha256 -extensions v3_ca -out /root/ca/root_ca.crt

Congrats, we have a private key and 69-year certificate for it! Now we need a short(er) lived intermediate certificate. For this example, I’m going to use 10 years, although in an enterprise environment you’ll probably want to go a lot shorter (2-5 years, or as often as you feel comfortable bringing out the root private key to offline-sign new certs and update your Yubikeys).

If you want to examine the key, you can use openssl x509 -noout -text -in /root/ca/root_ca.crt

Generate Intermediate Certificate

Similar to above, but we sign the public key with the root private key. We need to use a 2048 bit key for this to fit in the Yubikey’s PIV storage (which does not support 4096 bit keys with the PIV app, but does with PGP)

# Generate a private key (needs a passphrase)
openssl genrsa -aes256 -out /root/ca/intermediate_ca_key 2048
# Generate a certificate-signing-request (CSR) for the intermediate CA key
openssl req -config /root/ca/root.cnf -new -sha256 -key /root/ca/intermediate_ca_key -out /root/ca/intermediate_ca.csr.pem
# Sign the CSR with the root key
openssl ca -config /root/ca/root.cnf -keyfile /root/ca/root_ca_key -cert /root/ca/root_ca.crt -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -in /root/ca/intermediate_ca.csr.pem -out /root/ca/intermediate_ca.crt

If you want to examine the key, you can use openssl x509 -noout -text -in /root/ca/intermediate_ca.crt

Add Intermediate Key to Yubikey

In this phase, we take the intermediate CA key and import it to the Yubikey so we can use it. We also need to start the Yubikey service.

So, the commands:

# Instll Yubikey manager tools
apt install yubikey-manager

# Start Yubikey PCS service and enable it on boot
systemctl enable pcscd --now

# Add the intermediate CA keys in slot 9C
# You will need the passphrase for the intermediate private key
# There are other slots available, including all of the 'retired' slots if you want
# to use your Yubikey for other things and not overwrite slot 9C
ykman piv certificates import 9c /root/ca/intermediate_ca.crt
ykman piv keys import 9c /root/ca/intermediate_ca_key

If you are curious about the results, run ykman piv info

Install Smallstep

Setup Debian

Install Debian Bullseye (I know Bookworm just released), no GUI, with SSH although we can disable that later. Then login, make sure you are root - either login as root, or sudo su to get to a root prompt. Then cd /root so we can begin.

I’m using “tempest” as the name of my CA, located at tempest.palnet.net. DNS resolves to the IP address of my system locally, not externally, I don’t own that domain name. Make sure you update the names in any configuration / examples I list with your own names.

As usual, once you’re done installing run apt update and apt full-upgrade -y to make sure everything is up to date before continuing.

Install Smallstep CLI

Unfortunately their deb package doesn’t include Yubikey support, so we have to compile that from source, but we can install the CLI from deb packages. If you don’t want to use a Yubikey for your private keys, you can also instal the CA from deb packages and skip the whole compiling step below.

# Download step-cli from github (latest release is 0.24.2)
# We need to build step-ca from source to use Yubikey, but if you aren't using Yubikey you can go ahead
# and install the ca from deb packages as well.
#wget https://dl.smallstep.com/gh-release/certificates/gh-release-header/v0.24.2/step-ca_0.24.2_amd64.deb
wget https://dl.smallstep.com/gh-release/cli/gh-release-header/v0.24.4/step-cli_0.24.4_amd64.deb

# Install using apt
apt install ./*.deb -y
rm ./*.deb

Build Step-CA from Source

To build Step-CA, we need to install a recent version of the Go compiler, a few other tools and libraries we need to build Step-CA, to actually git clone and build step-ca, and install it. Here’s the process:

# Add Backports to sources.list
cat >> /etc/apt/sources.list << EOF

#Backports repository
deb http://deb.debian.org/debian/ bullseye-backports main
deb-src http://deb.debian.org/debian/ bullseye-backports main
EOF

# And apt-update
apt update

# Install golang (this is the same version in Bookworm, fyi, 1.20 is in Sid still)
# And Git, so we can clone stuff
# And also some other packages we need to compile Smallstep
apt install golang-1.19-go git libpcsclite-dev gcc make pkg-config curl -y

# Add Go 1.19 to PATH for the next operations
export PATH="/usr/lib/go-1.19/bin:$PATH"

# Clone the git repo, move to it, and checkout the latest release (as of this writing, 0.24.2, but check Github)
cd /root
git clone https://github.com/smallstep/certificates.git
cd certificates
git checkout v0.24.2

# Build process
# Once you execute these, go get a coffee and come back, or maybe drive to get a snack, it'll be awhile
make bootstrap
# I know this sounds odd but I promise 'install' also does 'build', since it depends on the binary
# GOFLAGS are clear so it does a CGO build, which is required for Yubikey support
make install GOFLAGS=""

# This tells the kernel that step-ca can bind to service ports
setcap CAP_NET_BIND_SERVICE=+eip /usr/bin/step-ca

Now check version: step version and step-ca version to make sure they both run. step-ca should show a release time/date of now the time you actually built it, which should be now() but in UTC and not your local timezone.

Setup Step-CA

Step-CA usually wants to do its own PKI, so we are going to let it generate a new set of private keys and sign them, then delete them. So, that’s fun. There aren’t any options to not generate keys.

Anyway, commands for this phase:

# Create a new user for step-ca process, and a home in etc for it
mkdir -p /etc/step
export STEPPATH=/etc/step
useradd step 
passwd -l step

# Run Step Init to create its folder structure and config
# Enter the password for your admin user ('provisioner')
step ca init --name="TEMPEST" --dns="tempest.palnet.net" --address=":443" --provisioner="apalrd" --deployment-type standalone

# Copy the certificates (but not the private keys) from the root location to /etc/step and own them to step
cp /root/ca/root_ca.crt /root/ca/intermediate_ca.crt /etc/step/certs/
chown -R step:step /etc/step/

Next up, we need to edit /etc/step/config/ca.json to configure our Yubikey instead of internal PKI for this Step instance. Specifically, replace the key directive which points to a private key with a bit about it being a yubikey and selecting the parameters for the key management system (KMS). The diff looks like this:

 	"root": "/etc/step/certs/root_ca.crt",
 	"federatedRoots": null,
 	"crt": "/etc/step/certs/intermediate_ca.crt",
-	"key": "/etc/step/secrets/intermediate_ca_key",
+   "key": "yubikey:slot-id=9c",
+   "kms": {
+            "type": "yubikey",
+            "pin": "123456"
+   },
 	"address": ":443",
 	"insecureAddress": "",

And now we can test it! sudo -u step step-ca /etc/step/config/ca.json, shouldn’t give any errors and should sit and wait for requests to come in.

Add SystemD Service

Since we presumably want this to run all the time, this should be a service, and systemd is the thing that does services.

So, write out this service file, or copy and paste this all into the terminal to do it for you:

# The service script
cat > /etc/systemd/system/step-ca.service  << EOF
[Unit]
Description=Smallstep Certificate Authority

[Service]
User=step
Group=step
Environment="STEPPATH=/etc/step"
ExecStart=/usr/bin/step-ca /etc/step/config/ca.json

[Install]
WantedBy=multi-user.target
EOF

# Reload daemons and start now
systemctl daemon-reload
systemctl enable --now step-ca

Enable ACME Challenges

Now that our step-ca is up and running, we can enable the ACME provisioner for it. We are, finally, almost ready to issue certificates to our infrastructure! For this, we also need the fingerprint, which the ca prints when it starts up (get it from systemctl status step-ca and look for X.509 Root Fingerprint).

And the commands for this step:

# Make sure STEPATH is still set, since we need it for this phase 
export STEPPATH=/etc/step
# Add the acme provisioner using our admin account
step ca provisioner add acme --type ACME --admin-name apalrd
# Restart the service
systemctl restart step-ca

Using your CA

Trust your Root

Here are some instructions on trusting your root certificate in Debian. Remember that the certificate is hosted by our ACME server, so we can just download it (although it’s certificate won’t yet be trusted, so we need to ignore certificate errors for now).

# As root, or prepend with sudo of course
wget --no-check-certificate https://tempest.palnet.net/roots.pem -O /usr/local/share/ca-certificates/tempest.crt
update-ca-certificates

You’ll need to do this for every single computer which you use to access your sites, or you’ll get a certificate error. Fun, right? But once you add the root certificate, then you can continue to add homelab services without needing to individually trust each one on each user’s system.

Using your CA in Caddy

Here’s an example Caddyfile which uses my local CA to issue a signed certificate. Also note that I first added the root certificate to the trust store on the local system, so I don’t need to separately tell Caddy to trust the root certificate.

#Global options
{
        #Our local ACME server
        acme_ca https://tempest.palnet.net/acme/acme/directory
}

#A single server which will get a TLS certificate automatically
ca-testsvr.palnet.net {
        #All of the options here are left as defaults
        #But just say hello world for now
        respond "Hello, World!"
}

References

Here are some guides I used when writing this script, they might also be useful to you.

Organize your Homelab Services with Dashy!

Thu, 01 Jun 2023 07:00:05 -0300

Do you have so many self-hosted services running in your homelab that it’s hard to keep track of them all? Do you hate typing the IP/port for each one? You could use DNS, but a nice dashboard would make it easier too! Today I’m setting that up with Dashy, a beautiful and easy to edit homelab dashboard tool. It’s not the lightest weight tool in the world, but the look is worth it for me.

Video

Install Script

Start with a Debian 11 (Bullseye) LXC container, and run this all as root. Feel free to go section by section if you want, or copy and paste the whole thing and yolo it.

#NodeJS 16
wget -O - https://deb.nodesource.com/setup_16.x | bash -
#apt update
apt update
#apt install stuff (nodejs version should track Nodesource now)
apt install -y nodejs git
#NPM install stuff (need a newer version of yarn than in apt)
npm install --global yarn

#Move to dashy dir, clone the git repo
cd /opt
git clone https://github.com/Lissy93/dashy.git
cd dashy

#Do the yarn
yarn
yarn build

#Create systemd service for Dashy
cat > /etc/systemd/system/dashy.service << EOF
[Unit]
Description=Dashy homelab dashboard
After=network-online.target

[Service]
Environment="PORT=80"
Environment="HOST=::"
ExecStart=/usr/bin/yarn --cwd=/opt/dashy start
Restart=always

[Install]
WantedBy=multi-user.target
EOF

#Create systemd service to rebuild dashy on config change
cat > /etc/systemd/system/dashy-rebuild.service << EOF
[Unit]
Description=Rebuild Dashy on Config Changes

[Service]
Type=oneshot
ExecStart=/usr/bin/yarn --cwd=/opt/dashy build
EOF

#Create systemd path to call service on config file change
cat > /etc/systemd/system/dashy-rebuild.path << EOF
[Unit]
Description=Monitor Dashy Config for Changes

[Path]
PathChanged=/opt/dashy/public/conf.yml

[Install]
WantedBy=multi-user.target
EOF

#Make systemd aware that we just changed stuff
systemctl daemon-reload

#And start our services!
systemctl enable --now dashy
systemctl enable --now dashy-rebuild.path

The ULTIMATE Guide to Fiber Optic Home Networking

Thu, 25 May 2023 01:00:05 -0300

Do you have a need to extend your home network around your property? Maybe you want reliable internet in the shed you turned into a work-from-home office, or your garage or workshop? Today I’m going to explain what you need to run fiber optic newtorking around your home and property on a budget, for high bandwidth and low latency networking. Fiber doesn’t have any issues with lightning or electrical potential changes between buildings, and can handle much higher bandwidth with higher reliability than wifi mesh or point to point systems.

It’s not as expensive as you think to run fiber in your network!

Video

Hardware Recommendations

This section has more detailed links to the products I reference in the video. Enjoy!

Some links to products may be affiliate links, which may earn a commission for me.

Fiber

For Fiber, there are a lot more options for singlemode than multimode. In general, ordering custom length patch cables is the easiest way to go. In general I have bought fiber from fs.com for my own projects, so that’s what I’m linking here. For all of these options, you can click ‘customize’ and have the right length made to order.

Networking fiber uses LC connectors with UPC polish, which is color coded blue (vs green for APC polish, used in PON fiber-to-the-home systems).

In general you should use riser rated cables indoors, there are very narrow use cases where you would need plenum rating (low smoke) in a normal house. In my locale it’s not required for low voltage cables at all in the residential code. If you are going underground in conduit the whole way, you can use indoor rated fiber (it’s considered a wet location, but not subject to damage), and if you are going outdoors you need to use a UV stable jacket (such as TPU) instead of PVC. If you need plenum rating, when customizing, change the jacket type from ‘PVC’ to ‘Plenum (OFNP)’

I don’t use armored cable, but I have done all of these runs after building the house and am more concerned with fishing the wiring easily through tough spaces. If you’re installing at construction time, it might be a good idea to use armored cable to protect from damage during construction. Remember, it’s effectively imposible to repair the fiber.

OM3 Multimode (Duplex)

Remember, you want aqua cables (if they are colored indicating type), not orange. If the ends are colored, they should be blue, not green.

OS2 Singlemode (Duplex)

OS2 should be colored yellow if colored to match type, and the ends should be blue if colored to match type.

OS2 Patch Cable Duplex LC
OS2 Patch Cable Armored Duplex LC
OS2 Industrial Armored TPU Cable - Make sure to select 2 fibers, LC UPC Duplex

OS3 Singlemode (Simplex / BiDi)

These are only for BiDi use, use duplex if you aren’t going to do BiDi.

OS2 Patch Cable Simplex LC
OS2 Industrial Armored TPU Cable - Make sure to select single fiber, LC UPC Simplex

Media Converters

These devices strictly convert from a transceiver to copper, nothing more, nothing less. They are cheap and you should use them if you already have wired Ethernet hardware at this end of the link you want to connect to. Any of these options can go at either end of the link, as long as they are running at the same speed.

1G SFP to RJ45 - TP-Link MC220L - I use this one, it runs at 1G SFP.
10G SFP+ to RJ45 Mult-Gig - I have not used this one myself, it requires 10G on the SFP+ side, and will negotiate 1/2.5/5/10G on the RJ45 side.

Network Cards

These go in your computer and accept an SFP/SFP+ module. You may need to do some configuration with ethtool / windows / … to set the link speed if you are not using the rated speed. Recommended for ‘servers’ especially, but also good for high bandwidth desktops and workstations. Also, be aware of the form factor - a lof of these cards come in OCP NIC and normal PCIe slot sizes, and you probably can’t use an OCP NIC in your regular desktop.

Intel X520-DA1/DA2 (10G capable) - only difference is DA1 = 1 port, DA2 = 2 port. Do NOT get T1/T2/.. versions, those are RJ45.
Mellanox Connect-X 3 (10G capable) - Make sure it’s Ethernet and 10G SFP+, they made CX3 cards with 10G QSFP and Infiniband variants. If you want to go for higher speeds in the future, the Connext-X 4 family can do 25G with SFP28, which is the next speed tier up above SFP+, and is also backwards compatible. If you are building a router, some people claim the newer Intel X710-DA is better, although I can’t say one way or another.

Network Switches and Routers

A ton of network switches have SFP or SFP+, here are just a few of my favorites. I use Mikrotik equipment, so that’s what I’ve listed here. Feel free to shop around as well, there are a lot of good regional distributors for this (i.e. Baltic Netoworks in the US, Getic in the EU) depending on your location.

Mikrotik RB260GS (1G capable) - 5 Copper, 1 SFP port
Mikrotik RB260GSP (1G capable) - Same as above, but with passive PoE out (if you don’t know about passive PoE, be warned).
Mikrotik hEX S Ethernet Router (1G capable) - Same ports as RB260, but a full router
Mikrotik hAP AC WiFi Router (1G capable) - Same as above, but with WiFi as well. You can also use it as a WiFi access point + switch, using the router you already have.
Mikrotik CRS328-24P-4S+RM (10G SFP+ capable) - The big 24 PoE + 4 SFP+ port switch that I use in my house

Transceivers

OM3 Multimode (Duplex)

Choose the same transceiver for both ends.

OS2 Singlemode (Duplex)

Choose the same transceiver for both ends.

OS3 Singlemode (Simplex / BiDi)

Choose a MATCHED PAIR here, with opposite transmit/receive wavelengths on each end

Example Setup Lists

These are some example setups I’ve setup for you to use to think about what you need for your project.

1G Cheap BiDi Setup

This is a simple gigabit BiDi setup that can be used to run Ethernet long distances, to outbuildings, etc. with a simple media converter to connect to your existing network and a switch at the far end.

Fiber (Indoors / In Conduit) - I recommend the 3mm jacket for increased robustness
Fiber (Outdoors / Direct Burial) - Fiber Count = Simplex, Connectors = LC UPC Simplex
Basic 1G SFP SIDE A
Basic 1G SFP SIDE B
Media Converter for one side
Mikrotik Switch for other side Total Cost is around $100 for indoor fiber or $200 for outdoor fiber, depending on the length.

My 10G Gaming Shed Setup

This is the setup I actually used in the video.

Since I already had the switch, I actually spent $98 on the rest of the parts, but the switch is an additional $489 at list price (obviously it can do a whole lot more than just this one run though).

The Homelab Swiss Army Knife: ZimaBoard

Thu, 18 May 2023 07:00:05 -0300

I’ve used a lot of different small form factor machines over the years, from the Raspberry Pi to used ebay thin clients. All of them are good at some things. But when Icewhale sent over their x86-based Zimaboard for me to take a look at, I’ve been impressed with the flexibility it has for me to test new software and hardware in a relatively cheap way. It’s not spectacular at any one thing, but it’s versatile enough that it’s a great foundation for so many of my projects.

So what’s good about the Zimaboard: -Reasonable price for an x86-based single board computer -Being x86 based is a huge advantage for running the usual homelab software -Enough eMMC for a full operating system, enough for many use cases without additional storage -Dual NICs are a huge benefit to a lot of projects -Low power 12V input -CasaOS is open-source and very easy to get started with if you want the basics for your homelab without a lot of fuss

A few negatives: -PCIe slot is in a bit of a weird position, slot covers don’t fit well and hit the case, I ended up taking them all off -Not enough USB ports if you want to actually use it as a desktop (or install and OS via a GUI) -The ZimaBoard itself is beautiful, but when you add cards and SSDs, it can look very hacked together.

Contents:

Video

lscpu

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Address sizes:                   39 bits physical, 48 bits virtual
Byte Order:                      Little Endian
CPU(s):                          4
On-line CPU(s) list:             0-3
Vendor ID:                       GenuineIntel
Model name:                      Intel(R) Celeron(R) CPU N3450 @ 1.10GHz
CPU family:                      6
Model:                           92
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
Stepping:                        9
CPU max MHz:                     2200.0000
CPU min MHz:                     800.0000
BogoMIPS:                        2188.80
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault cat_l2 ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust smep erms mpx rdt_a rdseed smap clflushopt intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts md_clear arch_capabilities
Virtualization:                  VT-x
L1d cache:                       96 KiB (4 instances)
L1i cache:                       128 KiB (4 instances)
L2 cache:                        2 MiB (2 instances)
NUMA node(s):                    1
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected

lspci

00:00.0 Host bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Host Bridge (rev 0b)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 500 (rev 0b) (prog-if 00 [VGA controller])
00:0e.0 Audio device: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Audio Cluster (rev 0b)
00:0f.0 Communication controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Trusted Execution Engine (rev 0b)
00:12.0 SATA controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SATA AHCI Controller (rev 0b) (prog-if 01 [AHCI 1.0])
00:13.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port A #1 (rev fb) (prog-if 00 [Normal decode])
00:14.0 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port B #1 (rev fb) (prog-if 00 [Normal decode])
00:14.1 PCI bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series PCI Express Port B #2 (rev fb) (prog-if 00 [Normal decode])
00:15.0 USB controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series USB xHCI (rev 0b) (prog-if 30 [XHCI])
00:1c.0 SD Host controller: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series eMMC Controller (rev 0b) (prog-if 01)
00:1f.0 ISA bridge: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series Low Pin Count Interface (rev 0b)
00:1f.1 SMBus: Intel Corporation Celeron N3350/Pentium N4200/Atom E3900 Series SMBus Controller (rev 0b)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

vainfo

vainfo: VA-API version: 1.14 (libva 2.12.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 22.3.1 ()
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSliceLP
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileJPEGBaseline           :	VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD

Setting Up my PROXMOX Backup Server!

Thu, 11 May 2023 07:00:05 -0300

I’ve gone from “no backups” to “raid is a backup” to “two zfs pools in one box”, and decided it’s finally time for a proper backup solution. So, I settled on Proxmox Backup Server! And today, I rebuild my HP Microserver Gen8 with 4x10T refurbished SAS drives, a new SAS controller card, and more! With this backup solution, I’m feeling a lot better about my data migration to Ceph.

Video

SAS Drive Formatting

Since these drives were refurbished they were formerly formatted for a hardware RAID controller and were giving me lots of protection errors in dmesg - specifically blk_update_request: protection error (and failing to read, but not write). Reading around I found the answer - format without hardware protection. The drives were likely previously used with a hardware RAID controller and had additional data integrity features enabled, something that ZFS doesn’t need. The command I used to do this is sg_format --format /dev/sdX and took about a full day per drive to complete (I ran them all in parallel, no I didn’t wait a week). This is different than a filesystem format, since it’s a SCSI command to request that the drive format itself with the sector size and layout requested (in this case, the manufacturer’s defaults).

I also ran SMART tests and they passed.

ZFS Pool Setup

I wanted to use a ZFS Special Device, which stores filesystem metadata on a separate drive. This can let me use a smaller faster drive to speed up directory listings and large metadata operations, which is important in PBS since it can potentially store up to 65k files per directory. The chunk-based deduplication system splits data up into files named by their hash, so it relies on the filesystem to store a ton of files.

I created the zfs pool via the GUI, using RAIDZ1 with the 4x10T drives (30T usable). Then I added the special device from the shell: zpool add <poolname> -o ashift=12 special mirror /dev/sde /dev/sdf -f

Next Steps

Going forward, I have a few plans for this system that I haven’t implemented yet:

Auto-start the server at backup time, and auto shut-down when backups and other tasks are done
LACP bond (2x1gig) when I have enough switch ports to do that
Add backup of CephFS directly on the PBS server (mount Ceph on PBS, run proxmox-backup-client to the CephFS mount, …)
TAPE BACKUP! I bought an LTO5 SAS drive to dip my toes in tape backup, to see if I like it before moving to an autoloader

Remove Proxmox Server Access over SERIAL! Cheaper Alternative to IPMI or KVMs

Thu, 27 Apr 2023 07:00:05 -0300

I have a rack-mounted KVM now, and it’s great, but I’m working on building out a new Proxmox cluster which might not have a graphics output at all on some of the nodes. So, I need a new remote access solution for them.

The new nodes I’m planning on building will all be based on used consumer hardware, so I’m limited by what would be available on normal mATX boards. This doesn’t include IPMI, and if I go with AMD-based CPUs, doesn’t include an iGPU either. So I either need to get cheap GPUs to add to the second PCIe slot (which I then can’t use for things like HBAs), or at least put a GPU in for troubleshooting. However, adding a GPU often moves PCIe devices, leading to issues where the network card on enp4 is now enp5 when the GPU is attahed, for example. So trouleshooting for real would be a nightmare with this approach.

But! There is an alternative. Despite being ancient, many boards still have a COM header connected to the chipset, which provides a basic serial port header (‘COM1’ in Windows) and is completely usable. So, my goal is to connect this COM header to a female RJ45 port on the back of the computer, so I can plug in a ‘Cisco-style’ terminal cable. These sorts of cables (RJ45 serial to USB) are readily available cheaply on the internet, and I can open a terminal emulator on my laptop to diagnose the worst issues that lock up the system.

Since some systems are rumored to not POST without a graphics card, I’m testing this approach today.

Video

Building the Cable

The Cisco/Juniper style RJ45 pinout is fairly simple if we exclude all of the flow control lines, so we just need 3 wires. I bought a pack of 2x5 IDC headers and a panel mount female RJ45 extension cable (which can be mounted in the case), cut the male end off, and crimped the resulting wires directly in the IDC header. Then I used a cheap USB to RJ45 serial cable and all worked great.

Here’s the pinout drawing for reference:

Proxmox Setup

Setup is really two steps - enabling kernel console via serial, and enabling a tty on that serial port. One of these requires editing the kernel cmdline (Which can vary a bit - see Proxmox’s docs for more info), and the other requires using systemd.

First, we need to add a new console argument to the kernel cmdline. Default for this is tty0, and we can add multiple arguments, so we should make sure both are specified. On my system I am using systemd-boot, so I edit the file /etc/kernel/cmdline and add:

console=tty0 console=ttyS0,115200n8

(ttyS0 should be ‘COM1’ in Windows, and should be the COM port on the motherboard. You can also use USB Serial here if you need to, or any other serial ports you have available)

Now, we just need to enable getty on our serial port so we have a proper login terminal:

systemctl enable serial-getty@ttyS0.service

And reboot, watching boot messages on the console happily

Of course, since Proxmox is Debian (with Ubuntu kernels), this should work on any of your other systems also Debian-based, as long as you are aware of the right place for the cmdline config (GRUB or Systemdboot).

Parts

Some links to products may be affiliate links, which may earn a commission for me.

Network KVM for ALL of my Servers!

Thu, 13 Apr 2023 07:00:05 -0300

Since I use a lot of repurposed computers as servers, I don’t have many with IPMI / remote access built in. Usually I can configure them remotely through SSH or their web UI, but sometimes things go wrong and it’s nice to have remote console access. Enter, PiKVM, a project to build a networked KVM switch with a Raspberry Pi. Unfortunately, I have a more than one server, and building a PiKVM for every one gets expensive, so I’m combining it with an 8 port rack mount KVM switch so I can remotely view and control all of the servers in my rack.

The KVM I’m using supports network control (but it’s not a network KVM, it can just switch inputs over the network) and I’ve configured that in this video.

Video

Networking Configuration

The network configuration goes in /etc/systemd/network/eth0.network. Make sure to run rw to make the filesystem writeable before editing it, and ro when you are done.

[Match]
Name=eth0

[Network]
DHCP=yes
DNSSEC=no
# Use DHCP for primary address
# Add a static address in a /31 for the TESmart KVM
Address=192.168.1.11/31


[DHCP]
# Use same IP by forcing to use MAC address for clientID
ClientIdentifier=mac
# https://github.com/pikvm/pikvm/issues/583
RouteMetric=10

PiKVM Configuration

The PiKVM configuration override goes in /etc/kvmd/override.yaml. Make sure to run rw to make the filesystem writeable before editing it, and ro when you are done. I’m using the default IP of the Tesmart and a /31 subnet to avoid changing it, but if you do change the IP you’d change it here as well.

kvmd:
    gpio:
        drivers:
            tes:
                type: tesmart
                host: 192.168.1.10
                port: 5000
        scheme:
            server0_led:
                driver: tes
                pin: 0
                mode: input
            server0_switch:
                driver: tes
                pin: 0
                mode: output
                switch: false
            server1_led:
                driver: tes
                pin: 1
                mode: input
            server1_switch:
                driver: tes
                pin: 1
                mode: output
                switch: false    
            server2_led:
                driver: tes
                pin: 2
                mode: input
            server2_switch:
                driver: tes
                pin: 2
                mode: output
                switch: false    
            server3_led:
                driver: tes
                pin: 3
                mode: input
            server3_switch:
                driver: tes
                pin: 3
                mode: output
                switch: false    
            server4_led:
                driver: tes
                pin: 4
                mode: input
            server4_switch:
                driver: tes
                pin: 4
                mode: output
                switch: false    
            server5_led:
                driver: tes
                pin: 5
                mode: input
            server5_switch:
                driver: tes
                pin: 5
                mode: output
                switch: false    
            server6_led:
                driver: tes
                pin: 6
                mode: input
            server6_switch:
                driver: tes
                pin: 6
                mode: output
                switch: false    
            server7_led:
                driver: tes
                pin: 7
                mode: input
            server7_switch:
                driver: tes
                pin: 7
                mode: output
                switch: false
        view:
            table:
                - ["TESMART Switch"]
                - []
                - ["#Terra NAS", server0_led, server0_switch|Switch]
                - ["#Megalab", server1_led, server1_switch|Switch]
                - ["#PVE1", server2_led, server2_switch|Switch]
                - ["#PVE2", server3_led, server3_switch|Switch]
                - ["#PVE3", server4_led, server4_switch|Switch]
                - ["#Minilab", server5_led, server5_switch|Switch]
                - ["#OPNsense", server6_led, server6_switch|Switch]
                - ["#Iridium",  server7_led, server7_switch|Switch]

Fully Routed Networks in Proxmox! Point-to-Point and Weird Cluster Configs Made Easy

Thu, 16 Mar 2023 07:00:05 -0300

Are you playing with Proxmox clustering, but want faster networking without paying for multi-gig switches? For small clusters, sometimes it can make sense to use fast point to point links between nodes. This could be in a small 2 or 3 node cluster, where you can use dual port 10 gig cards and direct attach cables without a switch. Maybe you’ve got a wacky 5 node cluster with quad port gigabit cards on each node, and don’t want to buy a 20 port switch and do link aggregation. Or maybe you want to be the crazy guy who uses intel NUCs with thunderbolt between them. Whatever your use case, this video will help you setup your fully routed cluster network properly.

This is accomplished by creating point to point links between each node in any topology you can think of, and allowing OSPFv3 to exchange route information across all of the links. Once we have configured OSPF on all of the relevant interfaces, the cluster route map will automatically be generated and updated if any links go down and the shortest path will be chosen based on link speeds and priorities.

Video

Routed Cluster Subnet

Our existing network has a subnet mask, something like 192.168.1.0/24 or 2000:db8::/64. To identify traffic on the cluster network, we are going to create a completely new subnet, in this example fd69:beef:cafe::/64. Only nodes in the cluster know about this second subnet, and traffic using this second subnet will go over our high speed backbone.

Out of this subnet, we will assign an address to each node. In my case, I decided to use 551, 552, and 553 for the nodes.

To setup the individual links, we just need to bring the interfaces up so a link-local address will be assigned automatically.

Routing Configuration

We are going to use a package called free range routing for this, which implements the OSPFv3 protocol. So, naturally, we need to install it. apt install frr -y.

Next we need to enable the OSPFv3 daemon in /etc/frr/daemons. We could start it now, but we should probably configure it first.

To configure FRR, edit /etc/frr/frr.conf. I’m going to show this on the first node, but you’ll need to do this on all of your nodes with the specific interfaces on that node.

Since Linux will route packets within the kernel (assuming routing is enabled), we can send packets destined to our cluster subnet on any interface and Linux will accept them. So, we don’t need to assign the address on the point to point links themselves, as long as they go across the right link. To make sure we can accept packets for our cluster network when any link is down, we are going to add our node address to the loopback interface. So, starting off the config file, we will add an address to the loopback interface. We will also add it to the zero-area and set it to passive since there’s no reason for OSPF to be active on this interface.

!
interface lo
        ipv6 address fd69:beef:cafe::551/128
        ipv6 ospf6 area 0.0.0.0
        ipv6 ospf6 passive

Next up, we can add the existing gigabit interface as a backup. If there is no fast path to the other node, we can fall back on the ‘public’ network. So, set up vmbr0 as a broadcast network like this. When setting cost you can choose any integer you want, the standard is to divide some reference bandwidth by the link speed, but the actual routing path used will be the path that results in the lowest cost across all links. In my example I set the backup cost to 100 and point to point cost to 10.

#Backup links via primary gigabit link (vmbr0)
#Cost for 1G assumptions (100 gig reference / 1 gig = 100 cost)
!
interface vmbr0
        ipv6 ospf6 area 0.0.0.0
        ipv6 ospf6 network broadcast
        ipv6 ospf6 cost 100

Now setup all of your point to point links. They can have a different cost if your interfaces have different bandwidths. Use the interface names here (use ip a to find them).

!
interface ens19
        ipv6 ospf6 area 0.0.0.0
        ipv6 ospf6 network point-to-point
        ipv6 ospf6 cost 10
!

Finally, let’s setup the router. Set the router ID to something unique for each node, it can be whatever you want.

#OSPF router settings (unique router ID required for each router)
!
router ospf6
        ospf6 router-id 0.5.5.1
        redistribute connected
        auto-cost reference-bandwidth 100000

And finally, we can restart frr so it will come up with the new config - systemctl restart frr.

Within a few seconds of starting frr, routes should start coming in. Try viewing them with ip -6 route, you should see something like fd69 something via fe80 dev ens19 proto ospf.

Full Example Config

# default to using syslog. /etc/rsyslog.d/45-frr.conf places the log in
# /var/log/frr/frr.log
#
# Note:
# FRR's configuration shell, vtysh, dynamically edits the live, in-memory
# configuration while FRR is running. When instructed, vtysh will persist the
# live configuration to this file, overwriting its contents. If you want to
# avoid this, you can edit this file manually before starting FRR, or instruct
# vtysh to write configuration to a different file.
log syslog informational

#Enable IPv6 forwarding since we are using IPv6
ipv6 forwarding

#Add our router's private address on lo (loopback)
#This address is a single address (/128) out of the subnet (/64)
#of our 'cluster' network, of which routes to individial /128s are
#distributed using OSPF
!
interface lo
        ipv6 address fd69:beef:cafe::551/128
        ipv6 ospf6 area 0.0.0.0
        ipv6 ospf6 passive
#Backup links via primary gigabit link (vmbr0)
#Cost for 1G assumptions (100 gig reference / 1 gig = 100 cost)
!
interface vmbr0
        ipv6 ospf6 area 0.0.0.0
        ipv6 ospf6 network broadcast
        ipv6 ospf6 cost 100
#Two p2p links ens19 and ens20
#Since we are using IPv6 we do not need to assign
#addresses on these links, relying on link-local addresses
#Cost for 10G assumptions (100 gig reference / 10 gig = 10 cost)
#Feel free to edit your cost as appropriate
#You can tweak these cost values to change the traffic flow
!
interface ens19
        ipv6 ospf6 area 0.0.0.0
        ipv6 ospf6 network point-to-point
        ipv6 ospf6 cost 10
!
interface ens20
        ipv6 ospf6 area 0.0.0.0
        ipv6 ospf6 network point-to-point
        ipv6 ospf6 cost 10
#OSPF router settings (unique router ID required for each router)
!
router ospf6
        ospf6 router-id 0.5.5.1
        redistribute connected
        auto-cost reference-bandwidth 100000

One WiFi, Multiple Networks! Segment your WiFi Network with Private Pre-Shared-Keys

Thu, 09 Mar 2023 07:00:05 -0300

Do you love segmenting your network into as many subnets and VLANs as possible? Do you have too many Wifi networks for all of your special flower IoT devices that can barely speak IP, let alone fend for themselves on the wild internet? You could use WPA EAP Enterprise Authentication, but good luck getting your smart toaster to log in. The solution I’m playing with is called Private Pre-Shared Keys, where each client can potentially have their own passphrase and VLAN assignment for the same SSID, and the client just has to support normal passphrase authentication.

For this video, I’m using a Mikrotik wAP AC with RouterOS 7.8. I’d like to try OpenWRT in the future, but as of the making of this video it’s not quite ready.

Video

Full Mikrotik Config Export

/interface bridge
add ingress-filtering=no name=bridge vlan-filtering=yes
/interface wireless
set [ find default-name=wlan1 ] disabled=no mode=ap-bridge ssid=\
    SpaceLasers wps-mode=disabled
set [ find default-name=wlan2 ] disabled=no mode=ap-bridge ssid=\
    BillWiTheScienceFi wps-mode=disabled
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa2-psk mode=dynamic-keys \
    radius-called-format=ssid radius-mac-authentication=yes \
    supplicant-identity=MikroTik tls-mode=dont-verify-certificate
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/interface bridge port
add bridge=bridge interface=ether1
add bridge=bridge interface=wlan1
add bridge=bridge interface=wlan2
/interface bridge vlan
add bridge=bridge tagged=ether1,wlan2,wlan1 vlan-ids=9
/ipv6 address
add address=2001:db8:beef:cafe:ba69:f4ff:fe78:5a22 eui-64=yes interface=bridge
/radius
add address=2001:db8:beef:cafe::420 service=wireless
/system identity
set name=WAP

Full FreeRADIUS Configuration

authorize File:

#MAC is a specific value, SSID has specific value
FC:D4:36:B3:00:33 Called-Station-ID == "BillWiTheScienceFi", Auth-Type := Reject

#MAC has specific value, also pass a unique PSK and VLAN ID
FC:D4:36:B3:00:33 Called-Station-ID == "SpaceLasers", Auth-Type := Accept
        Mikrotik-Wireless-Psk = "ClientKey",
        Mikrotik-Wireless-VLANID = 9,

#MAC matches by OUI FC:D4:36 (Motorola, my test phone)
DEFAULT User-Name =~ "^FD:D4:36", Auth-Type := Accept
        Mikrotik-Wireless-Psk = "Motorola"

#MAC is locally defined (starts with one of x2, 06, xA, xE))
DEFAULT User-Name =~ "^[\dA-F][26AE]", Auth-Type := Accept
        Mikrotik-Wireless-Psk = "RandomKey"

#MAC is a specific value, reject user
FC:D4:36:B3:00:33 Auth-Type := Reject

#Any other users (you don't have to accept them)
DEFAULT Auth-Type := Accept
        Mikrotik-Wireless-Forward = 1,
        Mikrotik-Wireless-Psk = "GenericKey"

Also, the clients.conf:

client lan {
        ipaddr = 2601:40e:8101:6820::/64
        secret = CorrectHorseBatteryStaple
}

Manage your Media Collection with Jellyfin! Install on Proxmox with Hardware Transcode

Thu, 23 Feb 2023 07:00:05 -0300

In the last video I introduced Linux Containers, today we’re going to supercharge that by seeing if we can get some graphics hardware into our container, and give our large blu-ray collection a new home. We’re going to cover a few more advanced Proxmox container features, such as privilaged containers, hardware pass-through, and Jellyfin setup and transcoding for Intel and AMD GPUs.

There are always hardware quirks with hardware transcoding, but I’ve worked through it with two examples - a modern Intel Jasper Lake Celeron (which requires the guc/huc firmware), and an AMD Radeon WS3100.

This video is part of the Ultimate Home Server Megaproject

Video

Welcome to the Editing Den! My Editing Workflow

Thu, 16 Feb 2023 07:00:05 -0300

Today, I’m going on a tour of my editing den, and the process I go through to take footage of stuff and turn it into a final video for you all. So come along on this behind the scenes adventure!

Video

Ingest

The first step in my editing workflow is ingest. I have footage from a variety of devices and SD cards, and I need to get it onto a reliable storage as quickly as possible.

With the equipment I have, I really have four types of files.

Footage from my Nikon camera is in h.264 format on SD cards
Audio from the DJI Mic is in uncompressed wav internal to the device
Footage from my iPhone is on the iPhone and needs to be copied over the network
Screen capture of tutorials is captured in h.264 using OBS, on my local SSD

For the camera and mic, ingesting the footage means plugging the card into my desktop and copying the files. Since the MOV and WAV files are both compatible with Resolve directly, I copy these files straight into the working folder on my NAS for this video.

For the iPhone, it’s recorded in h.265, which isn’t compatible with the free version of Davinci Resolve. I haven’t bought a license yet since I want to try out the Linux version first before locking a license down to a single computer. So, using the native Files app, I copy the files into a big ‘iphone’ folder, where I can transcode them. I use Handbrake to transcode them down to h.264, and the resulting files go into the video’s working folder.

At this point, all of my footage is relatively safe on my NAS, stored on a ZFS mirror. I still keep the original footage on the card for now, until the nightly backup runs and I’m sure the footage is definitely safe. If I were to have a drive failure or NAS failure before the nightly backup runs, I’d still have the originals on the SD card.

Workstatio and Peripherals

As for the workstaion itself, my editing beast is a first-gen AMD Ryzen Threadripper 1950x, and it’s great at rendering and transcoding as well as virtualization. I like to use an ultrawide as my main monitor, so I have a 29" 21:9 monitor paired with a 24" 16:9 side monitor. It’s plenty of real estate for a video timeline or immersive game.

While I’ll usually write scripts on my laptop, which can sit on the corner of the desk, I have a very clicky typing keyboard made by Unicomp, paired with an ergonomic vertical mouse by Logitech.

The whole environment is tucked neatly into a closet, with LED lighting that I’ve installed for both atmosphere and recording. These are controlled by a QuinLED board via Home Assistant, and can be controlled from the Stream Deck.

Screen Capture

Since most of my videos are software tutorials, they tend to include a lot of screen capture segments. For these, I film at my desk, and the workflow is a bit different from the other video segments.

Screen capture is done in OBS, the open broadcaster software.

During sessions, I rely heavily on my Stream Deck to control OBS. I usually use my side monitor with display scaling set to 150% to better accomodate viewers at lower resolutions or on smaller screens. Overlaying my face is done live in OBS, and I can turn it on and off with my Stream Deck. I use Edge as my web browser in all of my tutorials, and I reset it to defaults before each video.

After a recording session, program output gets copied to the working folder on the NAS as usual.

Editing Workflow

At this point, all of my footage is in the working folder on the NAS. I import it into Davinci Resolve directly, so I’m editing directly over the network and not copying anything back and forth.

Since I have 5 gigabit Ethernet between my workstation and my switch, and 10 gig from there to the NAS, and I edit at 1080p with 1080p footage, I haven’t found a need to transcode to a mezzanine codec such as ProRes or generate proxies. In general the ZFS ARC hit rate on my NAS is extremely high during editing sessions so the speed of the spinning drives isn’t a bottleneck. I’ll probably re-evaluate all of these choices if I upgrade to equipment that can do 4K, but for the types of content I produce that isn’t necessary right now.

While I usually don’t generate proxies, timelapse segments in particular do stutter in the preview window. Resolve makes it pretty easy to right-click on a clip and generate optimized media as needed, which is basically a proxy. This takes quite a bit of space, with the timelapse clips for this project taking over 12 gigs at half resolution! But it does make it way nicer to scrub through clips with speed changes. So maybe I’ll reconsider my decision not to render proxies after all.

I use a single project in Resolve for all of my videos, with bins for different categories and videos. I haven’t really seen any suggestion on one project vs multiple and I find it easier to be able to grab clips and bits from other videos. Experts will probably disagree on this choice, and I’d love it if you did to boost engagement.

When I have all of the footage imported, I start by combining multiple files of the same shot into multicam clips. Most of my clips are actually multicam clips, since I add the audio track from the mics as another camera angle, but sometimes I also have as many as 3 real cameras. Resolve makes it easy to sync by audio automatically, and like magic I have a multicam clip and can independently select the video and audio angle shot by shot.

Once all of the multicams are syncd, I categorize clips into A-roll, B-roll, and screencap and may rename clips in the editor based on what makes sense. This is all purely in Resolve, the original filenames are still kept on disk.

Finally, I can start dropping stuff on the timeline. Usually I work from start to finish in a single editing session, although sometimes I take a break if I feel like I need to shoot a follow-up or more content. I have a few videos that have been sitting half-finished for months now, so you’ll see those eventually.

Rendering

After editing is done, it’s time to render out the project. I use the YouTube preset and tweaked it a little. Youtube is going to end up transcoding from this down to all of the resolutions they support, so I’d like to use the highest reasonable bitrate to reduce the quality loss through that process. I save the file as an mp4 and click the button. Then, as the CPU goes to town and has fun, I wait for it to finish so I can upload it to Youtube. I name the final files with a major number and minor letter and keep all finished renders, even if they aren’t the final upload. For major changes, I’ll create a duplicate timeline for V2 and beyond.

I spent a WEEK without IPv4 to understand IPv6 transition mechanisms

Thu, 09 Feb 2023 07:00:05 -0300

The time has come to talk about something uncomfortable to a lot of you. You’ve been using legacy methods for far too long. It’s time to move to IPv6.

But, of course, there’s a lot more to IPv6 than ‘just’ switching everything over. A lot of systems in the world still haven’t adopted it after nearly 25 years, and although software support is virtually a requirement these days, that doesn’t mean it’s widely enabled. There are also still a lot of misconceptions from network administrators who are scared of or don’t properly understand IPv6, and I want to address all of that.

But, for me to describe to you the best setup for your networks going forward, I need to understand for myself how all of the IPv6 transition mechanisms and behaviors work. To understand where transition mechanisms fail, I’m spending a fully week with only IPv6 and reporting on what works and doesn’t.

At the start of this, I’d like to absolutely stress that the NAT you know and hate was not imagined when the Internet Protocol was created, and in many ways has introduced a huge amount of headache in routing. You should stop thinking of NAT as a security mechanism and think of it as the emergency address exhaustion prevention that it is. Likewise, CG-NAT is a dire emergency address exhaustion prevention mechnism that nobody really wants to deploy unless they absolutely must. Don’t blame your provider when they deploy CG-NAT, embrace IPv6 and global routing instead.

Video
Crash Course on IPv6
Advantages for the Homelab
Transition Mechanisms
- Dual Stack
- Stateless IP/ICMP Translation
- NAT64 - the method I’ve setup for this test
- 464XLAT - the method I ended up using on macOS/iOS
Lessons Learned

Video

Crash Course on IPv6

The biggest hurdle to implementing IPv6 on your own isn’t usually ISP support, router support, or client support. It’s the mental thought process behind un-learning the legacy methods of network design. With IPv6, we aren’t just copying the mistakes of IPv4 and adding a wider address space, we’re going back to how the internet protocol was intended to behave before we started running out of addresses, getting rid of network address translation, and generally adding a stronger focus on proper subnet design and routing.

So, here are a few quick notes that can are crucial in understanding IPv6:

Addresses are 128 bits long and written as 8 four-letter hex blocks separated by colons (i.e. fd69:beef:cafe:feed:face:6969:0420:0001)
Leading zeros can be omitted (i.e. 0420 can become 420, but not 42)
Groups of zeros can be omitted with two colons, but only once in an address (i.e. 2000:1::1, but not 2000::1::1 as that is ambiguous)
Network prefixes are (virtually) always 64 bits long, with a 64-bit client suffix, using CIDR notation (i.e. 2000::/3)
All addresses should be treated as if they are globally unique, even if they are only within our organization
You are encouraged to have as many addresses as you want on a single interface, for different purposes or scopes
Similarly, we can have multiple routers advertising prefixes on the same layer 2 domain, and this is also encouraged
We no longer need to centrally assign addresses via DHCP, since nodes can now assign themselves addresses in the vast 64-bit local client space
Since everything is globally routable and unique, we have no need to do network address translation or port forwarding

Advantages for the Homelab

A question I get over and over when I bring up IPv6 is ‘what advantages does this bring to my home lab network’. So, here are some reasons you should start using IPv6 within your own network:

If you are behind carrier-grade NAT on IPv4, your IPv6 connectivity will still be globally routable and can receive incoming connections such as VPNs or game servers. I’ve found that I can host servers on a mobile hotspot using IPv6
Peer-to-peer communications such as gaming usually have to deal with NAT traversal, but with IPv6 this is no longer an issue, especially for multiple gamers using the same connection
IPSec VPN is widely used but often performs poorly as it struggles to traverse NAT, so like gaming this is not an issue with IPv6
Since we always have a link-local address, we don’t need to assign addresses at all on point-to-point links or isolated networks
If you want to host services, you don’t need to use different ports or a reverse proxy to separate traffic out of the single port on the singlular WAN address
This same advantage applies to single servers, where you can assign multiple addresses for different services, potentially using the same port

Transition Mechanisms

The primary methods of transition are Dual-Stack, SIIT, NAT64, and 464XLAT, each of which increases in complexity and has different advantages.

Dual Stack

In most cases, you will end up deploying dual-stack networks. With this setup, both IPv4 and IPv6 are completely deployed across the entire network. Resources are accessible via one or the other IP version, all network segments must be assigned both an IPv4 and IPv6 subnet, and all routers must have routing tables for both IPv4 and IPv6. Clients may choose to access any resource via either IP version, likely receiving both A and AAAA records for destinations. As you can see, this is manageable for home networks of only one router but generally is difficult to scale. However, it’s the easiest way to deploy IPv6 in your home network or small organization.

Stateless IP/ICMP Translation

If you’d like to avoid duplicating routing tables and all of your routing configuration, you can transition fully to IPv6, and statelessly translate between IPv4 and IPv6 at the edge of your network. This method is known as ‘Stateless IP/ICMP Translation’ and is most suited for applications where each device which needs to access the public IPv4 internet has a public IPv4 address, such as datacenters. This way, the IPv4 public addresses can be statelessly translated 1:1 to IPv6 destinations and relayed to IPv6-only servers, allowing the datacenter to operate fully IPv6 internally while still providing access to public IPv4 clients. However, this method does not perform traditional source NAT / masquarade as would normally be used in IPv4 networks.

NAT64

Another mechanism is NAT64. This operates similarly to ’legacy’ IPv4 NAT, in that we have a pool of addresses which we are masquarading behind a single public address. However, in our case, the address pool is IPv6 and the public address is IPv4. So, the NAT64 server takes requests from native IPv6 clients with the destination IPv4 address encoded into the destination IPv6 address (possibly using the well-known prefix 64:ff9b::/96 and adding the 32-bit IPv4 on the end). It then performs both protocol, address, and port translation as it would in IPv4 NAT, and outgoing connections depart the NAT64 server appearing as a normal network behind a single public IPv4. The downside to this is that clients need to know that they should be connecting to IPv4 destinations. Like traditional NAT, this is a stateful transition and imposes the NAT64 gateway as a potential choke point on the network (as the NAT service always has been).

The most common mechanism used alongside NAT64 is DNS64. In this transition mechansim, the DNS sever is aware that it’s serving an IPv6-only network and the prefix in use for NAT64 translation. If a DNS server encounters a query which has no AAAA record but does have a valid A record, it will synthesize an AAAA record by combining the A record with the NAT64 prefix. Now, any client software which utilizes DNS will automatically connect to the NAT64 gateway for access to the IPV4 internet via IPv6.

464XLAT

In addition to DNS64, the final mechanism which can be deployed in a NAT64 network is known as ‘464XLAT’. Conceptually it’s nearly identical to a combination of NAT64 at the network level and SIIT at the individual device level (4->6 1:1 followed by 6->4 NAT), although different terminology is often used due to the different organizations which developed this. The NAT64 server at the edge of the network usually performs client NAT or CG-NAT, and is known as the ‘PLAT’ (Provider side transLATor). Each client then runs its own ‘CLAT’ (Client side transLATor), which self-assigns an IPv4 address in its networking stack as well as an IPv6 address to communicate with the PLAT. Client applications which send packets using IPv4 see the CLAT’s IPv4 as the default IPv4 route, and the CLAT then performs stateless 4->6 translation. This removes or reduces the need for DNS64, imposes similar network requirements for a NAT64 gateway, and allows compatible clients to perform IPv4->IPv6 translation if they need direct IPv4 communication. This is usually the method used within ISP networks, as their NAT64/PLAT entity would already be required as a CG-NAT gateway and they can implement the CLAT service on their modem/gateway to provide the appearance of end-to-end IPv4 to clients with no downsides.

On our own networks, using 464XLAT is mostly limited by client software support. iOS and macOS both have native CLAT functions which will automatically activate, but older versions of macOS and essentially every other OS currently in use do not automatically function. That said, it’s possible to deploy both 464XLAT and DNS64 on the same network easily, so clients without native 464XLAT support will fall back on DNS64 and only have issues with IPv4 literals in certain peer-to-peer software.

Lessons Learned

A summary of the lessons I learned in my week without IPv4:

IPv6 is absolutely ready for prime-time and has been for awhile
About half of the internet sites I rely on support IPv6 natively, so there needs to be more pressure on site admins and CDNs to support IPv6 natively
There seems to be a lack of drive (judging by forum posts) to enable IPv6 on internet services by admins, either because they don’t care to, or it’s more work to manage a public IPv4 and public IPv6 presence
Networks should be designed IPv6-first instead of IPv4-first, and this design approach largely solves most of the major issues
NAT64 should replace traditional NAT in network architectures and is essentially a drop-in replacement pending better software support by routers
DNS64 alone is a mostly usable transition mechanism and is probably ’enough’ for public Wifi or other well-managed networks where you know what services are important to you or don’t mind peer-to-peer IPv4 addresses failing
464XLAT is a solution with no user-visible downsides and is the way ISP networks should be deployed going forward, combined with CG-NAT
Apple has excellent IPv6 support on their devices, fully supporting automatic configuration of 464XLAT on devices with NAT64, and overall an excellent attitude to forcing IPv6 support from developers
Other operating systems are bit of hit or miss

Linux VM Templates in Proxmox on EASY MODE using Prebuilt Cloud Init Images!

Thu, 02 Feb 2023 07:00:05 -0300

Have you ever wanted a nice, easy way to create new VMs to play with using your favorite base distro, without doing a lot of work to configure basic settings like your account, networking, hostname, etc?

Cloud-Init can do all of that and more, but it’s designed more for big cloud providers and not the easiest thing to setup. But, what if we could take a generic cloud image, and use it with Proxmox’s built-in Cloud-Init automation, to provision easy bare VMs without having to build our own templates? That’s what I’ve done, and I’ve written a script to automate downloading these templates on new Proxmox systems from the major distros which provide them (Debian, Ubuntu, Fedora).

Of course, once cloud-init is installed and configured, there’s no reason we can’t clone a template, install software on it, and then clone the clone to have a newly-configured VM with more software installed. From here, you can build out your library of useful application templates for whatever you do regularly.

The Video

As always, here’s the video with the full instructions.

The Script

This script will download a bunch of premade cloud images from the most common Linux distributions and add them as new 900-numbered VM templates, import your SSH keys, and configure the VM how I like it as a template. Of course, feel free to change any of the settings in the script before you run it, and you’ll need to add your public key in a plain text file and set your username.

#!/bin/bash

#Create template
#args:
# vm_id
# vm_name
# file name in the current directory
function create_template() {
    #Print all of the configuration
    echo "Creating template $2 ($1)"

    #Create new VM 
    #Feel free to change any of these to your liking
    qm create $1 --name $2 --ostype l26 
    #Set networking to default bridge
    qm set $1 --net0 virtio,bridge=vmbr0
    #Set display to serial
    qm set $1 --serial0 socket --vga serial0
    #Set memory, cpu, type defaults
    #If you are in a cluster, you might need to change cpu type
    qm set $1 --memory 1024 --cores 4 --cpu host
    #Set boot device to new file
    qm set $1 --scsi0 ${storage}:0,import-from="$(pwd)/$3",discard=on
    #Set scsi hardware as default boot disk using virtio scsi single
    qm set $1 --boot order=scsi0 --scsihw virtio-scsi-single
    #Enable Qemu guest agent in case the guest has it available
    qm set $1 --agent enabled=1,fstrim_cloned_disks=1
    #Add cloud-init device
    qm set $1 --ide2 ${storage}:cloudinit
    #Set CI ip config
    #IP6 = auto means SLAAC (a reliable default with no bad effects on non-IPv6 networks)
    #IP = DHCP means what it says, so leave that out entirely on non-IPv4 networks to avoid DHCP delays
    qm set $1 --ipconfig0 "ip6=auto,ip=dhcp"
    #Import the ssh keyfile
    qm set $1 --sshkeys ${ssh_keyfile}
    #If you want to do password-based auth instaed
    #Then use this option and comment out the line above
    #qm set $1 --cipassword password
    #Add the user
    qm set $1 --ciuser ${username}
    #Resize the disk to 8G, a reasonable minimum. You can expand it more later.
    #If the disk is already bigger than 8G, this will fail, and that is okay.
    qm disk resize $1 scsi0 8G
    #Make it a template
    qm template $1

    #Remove file when done
    rm $3
}


#Path to your ssh authorized_keys file
#Alternatively, use /etc/pve/priv/authorized_keys if you are already authorized
#on the Proxmox system
export ssh_keyfile=/root/.ssh/id_rsa.pub
#Username to create on VM template
export username=apalrd

#Name of your storage
export storage=local-zfs

#The images that I've found premade
#Feel free to add your own

## Debian
#Buster (10) (really old at this point)
#wget "https://cloud.debian.org/images/cloud/buster/latest/debian-10-genericcloud-amd64.qcow2"
#create_template 900 "temp-debian-10" "debian-10-genericcloud-amd64.qcow2"
#Bullseye (11) (oldoldstable)
#wget "https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-genericcloud-amd64.qcow2"
#create_template 901 "temp-debian-11" "debian-11-genericcloud-amd64.qcow2" 
#Bookworm (12) (oldstable)
wget "https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2"
create_template 902 "temp-debian-12" "debian-12-genericcloud-amd64.qcow2"
#Trixie (13) (stable)
wget "https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2"
create_template 903 "temp-debian-13" "debian-13-genericcloud-amd64.qcow2"
#Sid (unstable)
wget "https://cloud.debian.org/images/cloud/sid/daily/latest/debian-sid-genericcloud-amd64-daily.qcow2"
create_template 909 "temp-debian-sid" "debian-sid-genericcloud-amd64-daily.qcow2" 

## Ubuntu
#20.04 (Focal Fossa) LTS (really old at this point)
#wget "https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-amd64.img"
#create_template 910 "temp-ubuntu-20-04" "ubuntu-20.04-server-cloudimg-amd64.img" 
#22.04 (Jammy Jellyfish) LTS
wget "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
create_template 911 "temp-ubuntu-22-04" "ubuntu-22.04-server-cloudimg-amd64.img" 
#24.04 (Noble Numbat) LTS
wget "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img"
create_template 912 "temp-ubuntu-24-04" "ubuntu-24.04-server-cloudimg-amd64.img"

## Fedora 41
wget https://mirror.accum.se/mirror/fedora/linux/releases/41/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-41-1.4.x86_64.qcow2
create_template 921 "temp-fedora-37" "Fedora-Cloud-Base-Generic-41-1.4.x86_64.qcow2"
## Fedora 42
wget https://mirror.accum.se/mirror/fedora/linux/releases/42/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-42-1.1.x86_64.qcow2
create_template 922 "temp-fedora-38" "Fedora-Cloud-Base-Generic-42-1.1.x86_64.qcow2"

## Rocky Linux
#Rocky 8 latest
wget "http://dl.rockylinux.org/pub/rocky/8/images/x86_64/Rocky-8-GenericCloud.latest.x86_64.qcow2"
create_template 930 "temp-rocky-8" "Rocky-8-GenericCloud.latest.x86_64.qcow2"
#Rocky 9 latest
wget "http://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud.latest.x86_64.qcow2"
create_template 931 "temp-rocky-9" "Rocky-9-GenericCloud.latest.x86_64.qcow2"

## Alpine Linux
#Alpine 3.22.0
wget "https://dl-cdn.alpinelinux.org/alpine/v3.22/releases/cloud/generic_alpine-3.22.0-x86_64-bios-cloudinit-r0.qcow2"
create_template 940 "temp-alpine-3.22" "generic_alpine-3.22.0-x86_64-bios-cloudinit-r0.qcow2"

## FreeBSD
#FreeBSD 14.2 RELEASE
#Despite the images being named CLOUDINIT, they do not actually use cloud-init
#the default account is freebsd password freebsd
wget "https://download.freebsd.org/releases/VM-IMAGES/14.2-RELEASE/amd64/Latest/FreeBSD-14.2-RELEASE-amd64-BASIC-CLOUDINIT.ufs.qcow2.xz"
xz -d -v "FreeBSD-14.2-RELEASE-amd64-BASIC-CLOUDINIT.ufs.qcow2.xz"
create_template 960 "temp-freebsd-14.2" "FreeBSD-14.2-RELEASE-amd64-BASIC-CLOUDINIT.ufs.qcow2"

Making Proxmox into a pretty good NAS

Thu, 26 Jan 2023 07:00:05 -0300

Continuing the series where apalrd teaches proxmox skills through meaningful applications, today we are setting up a proper fileserver on our Proxmox system using Linux Containers. I’ve chosen to use a lightweight Linux Container (LXC) for this, so we can share the host’s ZFS filesystem. To manage shares and users using a web UI, I’m installing Cockpit, as well as some additional modules from 45Drives to deal with Samba. This should provide a pretty easy to use storage interface, keep all of our storage contained in the host Proxmox system without adding another layer of filesystem or a virtual machine, and run well on lower end hardware such as the Terramaster unit I’m using.

This video is part of the Ultimate Home Server Megaproject

Video

Cockpit Setup

I started with Debian 11 (Bullseye), although 12 (Bookworm) has now released and you should use that instead.

In the video I described how to add the bullseye-backports repository to /etc/apt/sources.list, however, as of the release of Bookworm this is no longer necesary to do. We also no longer need to specify the repository during installation, as it’s part of the normal Bookworm repos.

So here’s the command to install Cockpit:

apt install --no-install-recommends cockpit -y

After Cockpit is installed, we can allow root access (temporarily, while we setup more users). To do that, edit /etc/cockpit/disallowed-users and comment out the entry root.

At this point, you can connect to cockpit at https://:9090 and login with your root credentials, and it should work.

Cockpit Modules

Next, install the three modules from 45Drives:

As of this last update, here are the commands to download and install the most recent versions. Make sure you update these with the latest versions before you download!

#Download Cockpit File Sharing
wget https://github.com/45Drives/cockpit-file-sharing/releases/download/v4.2.8/cockpit-file-sharing_4.2.8-1focal_all.deb
#Download Cockpit Navitator
wget https://github.com/45Drives/cockpit-navigator/releases/download/v0.5.10/cockpit-navigator_0.5.10-1focal_all.deb
#Download Cockpit Identities
wget https://github.com/45Drives/cockpit-identities/releases/download/v0.1.12/cockpit-identities_0.1.12-1focal_all.deb
#Install them
apt install ./*.deb
#It will complain about being unable to delete the deb files, so we will do that now
rm ./*.deb

FIXING my USB3 2.5Gbe network adapters on Linux / Proxmox!

Thu, 19 Jan 2023 07:00:05 -0300

Do you have a USB3 2.5Gbe network adapter that doesn’t work well in Linux? Well, the kernel has included the upgraded Realtek drivers for awhile now, but for some reason it needs additional udev rules to load correctly. It’s a pretty simple fix and results in dramatically improved bandwidth (full duplex ~1.5Gbps and half duplex ~1.9Gbps), and functions properly with the correct driver.

tl;dr if you’re on a Debian-based system (including Proxmox) create the udev file below and reboot, and it should work fine. Validate it by running dmesg | grep enx to see if it bound the right driver for your enxXXXXXXXXXXXX interface.

Video

Udev File

This file needs to go at /etc/udev/rules.d/50-usb-realtek-net.rules

And the file contents (from the Realtek driver download):

# This is used to change the default configuration of Realtek USB ethernet adapters

ACTION!="add", GOTO="usb_realtek_net_end"
SUBSYSTEM!="usb", GOTO="usb_realtek_net_end"
ENV{DEVTYPE}!="usb_device", GOTO="usb_realtek_net_end"

# Modify this to change the default value
ENV{REALTEK_MODE1}="1"
ENV{REALTEK_MODE2}="3"

# Realtek
ATTR{idVendor}=="0bda", ATTR{idProduct}=="815[2,3,5,6]", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="0bda", ATTR{idProduct}=="8053", ATTR{bcdDevice}=="e???", ATTR{bConfigurationValue}!="$env{REALTEK_MODE2}", ATTR{bConfigurationValue}="$env{REALTEK_MODE2}"

# Samsung
ATTR{idVendor}=="04e8", ATTR{idProduct}=="a101", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"

# Lenovo
ATTR{idVendor}=="17ef", ATTR{idProduct}=="304f", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="3052", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="3054", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="3057", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="3062", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="3069", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="3082", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="3098", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="7205", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="720a", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="720b", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="720c", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="7214", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="721e", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="8153", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="a359", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"
ATTR{idVendor}=="17ef", ATTR{idProduct}=="a387", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"

# TP-LINK
ATTR{idVendor}=="2357", ATTR{idProduct}=="0601", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"

# Nvidia
ATTR{idVendor}=="0955", ATTR{idProduct}=="09ff", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"

# LINKSYS
ATTR{idVendor}=="13b1", ATTR{idProduct}=="0041", ATTR{bConfigurationValue}!="$env{REALTEK_MODE1}", ATTR{bConfigurationValue}="$env{REALTEK_MODE1}"

LABEL="usb_realtek_net_end"

Take Control of your Smarthome with Home Assistant! Installation Tutorial on Proxmox

Thu, 12 Jan 2023 07:00:05 -0300

If you’re building your first home server (or following my Ultimate Home Server series!), the first app I recommend playing with is Home Assistant. Taking control of your home automation with free and open-source software is an excellent way to get out of the cloud-based walled gardens, and self-hosting an app like this is a great way to learn about self-hosting in general without the pressure of hosting something like your firewall.

In this video, I go over the process for creating a new virtual machine in Proxmox for Home Assistant, what settings you should choose, and why. I also include the steps to pass through USB hardware such as Zigbee and Z-Wave dongles to your virtual machine. Enjoy!

This video is part of the Ultimate Home Server Megaproject

Video

Click on the thumbnail to see the video

Terramaster NAS as low-cost Proxmox node? Teardown and SW Install!

Thu, 05 Jan 2023 07:00:05 -0300

I get asked a lot about what hardware I recommend for homelabs and home servers. It’s a very difficult question since it depends on what exactly you want to get out of your setup. But, whatever you choose, I’m starting a new series where I’ll setup all of the commonly requested home server software in a single box. Since I want to try this on both used and new hardware, here’s a low cost NAS you can buy brand-new and run your own software on it!

A few basics of this guy:

Terramaster F2-223 - F2 means 2-slot cube form factor, -2 is dual core, 23 is the ’new’ series (at least when this video was produced)
Intel Celeron N4505 dual core
Dual DDR4 sodimm slots, comes with 1x 4G stick and one easily accessible empty slot. I added an 8G stick to bring it up to 12G so I can do virtualization
Dual Intel I225-v3 2.5G NICs

A few upsides:

They are pretty cheap for the HW you get new
Mix of SATA and NVMe storage options, for bulk storage and high speed storage
x86 processor so it can run Linux based OSes well
Intel I225 2.5Gbe NIC is considered ‘good’ compared to Realtek chipsets, and widely compatible across OSes such as Linux and BSD

A few downsides:

The included software seems to want to install itself on your drive as the boot disk, but I was unable to get it to do so in a short amount of time so I gave up and installed Proxmox (which I was planning on doing anyway)
The internal USB seems to perpetually change some BIOS/UEFI settings back to default, removing it fixed all of the issues I had getting it to boot Proxmox properly
The second RAM stick is harder to get to, although I don’t think you realistically need more than 12G of RAM to go with this much CPU power. Maybe it would matter more for the quad-core version

This video is part of the Ultimate Home Server Megaproject

Video

Click on the thumbnail to watch the video:

Hardware Information

lscpu

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           156
Model name:                      Intel(R) Celeron(R) N4505 @ 2.00GHz
Stepping:                        0
CPU MHz:                         2899.952
CPU max MHz:                     2900.0000
CPU min MHz:                     800.0000
BogoMIPS:                        3993.60
Virtualization:                  VT-x
L1d cache:                       64 KiB
L1i cache:                       64 KiB
L2 cache:                        1.5 MiB
L3 cache:                        4 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Vulnerable: Clear CPU buffers attempted, no microcode; SMT disabled
Vulnerability Retbleed:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:             Vulnerable: No microcode
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acp
                                 i mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc art arch_perfmon peb
                                 s bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq
                                  dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe
                                  popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch cpuid_fault epb cat_l
                                 2 cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_a
                                 d fsgsbase tsc_adjust smep erms rdt_a rdseed smap clflushopt clwb intel_pt sha_ni xsave
                                 opt xsavec xgetbv1 xsaves split_lock_detect dtherm ida arat pln pts hwp hwp_notify hwp_
                                 act_window hwp_epp hwp_pkg_req umip waitpkg gfni rdpid movdiri movdir64b md_clear flush
                                 _l1d arch_capabilities

lspci

00:00.0 Host bridge: Intel Corporation Device 4e14
00:02.0 VGA compatible controller: Intel Corporation Device 4e55 (rev 01)
00:04.0 Signal processing controller: Intel Corporation Device 4e03
00:08.0 System peripheral: Intel Corporation Device 4e11
00:14.0 USB controller: Intel Corporation Device 4ded (rev 01)
00:14.2 RAM memory: Intel Corporation Device 4def (rev 01)
00:14.5 SD Host controller: Intel Corporation Device 4df8 (rev 01)
00:16.0 Communication controller: Intel Corporation Device 4de0 (rev 01)
00:17.0 SATA controller: Intel Corporation Device 4dd3 (rev 01)
00:1c.0 PCI bridge: Intel Corporation Device 4db8 (rev 01)
00:1c.1 PCI bridge: Intel Corporation Device 4db9 (rev 01)
00:1c.5 PCI bridge: Intel Corporation Device 4dbd (rev 01)
00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01)
00:1f.3 Audio device: Intel Corporation Device 4dc8 (rev 01)
00:1f.4 SMBus: Intel Corporation Device 4da3 (rev 01)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Device 4da4 (rev 01)
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961/SM963

vainfo

libva info: VA-API version 1.10.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_10
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.10 (libva 2.10.0)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 21.1.1 ()
vainfo: Supported profile and entrypoints
      VAProfileNone                   :	VAEntrypointVideoProc
      VAProfileNone                   :	VAEntrypointStats
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSliceLP
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSliceLP
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileJPEGBaseline           :	VAEntrypointEncPicture
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSliceLP
      VAProfileVP8Version0_3          :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointEncSliceLP
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointEncSliceLP
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile1            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileVP9Profile3            :	VAEntrypointVLD
      VAProfileHEVCMain422_10         :	VAEntrypointVLD
      VAProfileHEVCMain444            :	VAEntrypointVLD
      VAProfileHEVCMain444            :	VAEntrypointEncSliceLP
      VAProfileHEVCMain444_10         :	VAEntrypointVLD
      VAProfileHEVCMain444_10         :	VAEntrypointEncSliceLP

Ultimate Home Server Megaproject

Thu, 05 Jan 2023 08:00:00 +0000

In this project, I explore an all-in-one home server using low cost hardware, bringing together as many common home applications as possible in a single box.

Terramaster NAS as low-cost Proxmox node? Teardown and SW Install!

In the first video, I introduce the hardware for this project - a cheap Terramaster NAS! It combines two HDD bays and two NVMe slots in a very tiny and low power brick, with dual 2.5G ethernet too! I get it running with Proxmox, so future projects have a platform to build on.

Take Control of your Smarthome with Home Assistant! Installation Tutorial on Proxmox

In this video, I go over the process for creating a new virtual machien in Proxmox for Home Assistant, what settings you should choose, and why. I also include the steps to pass through USB hardware such as Zigbee and Z-Wave dongles to your virtual machine. Enjoy!

Making Proxmox into a pretty good NAS

Manage your Media Collection with Jellyfin! Install on Proxmox with Hardware Transcode

Migrating my PERSONAL SERVER from TrueNAS to Proxmox

Unleash your Home Cameras with FRIGATE Self-Hosted AI Video Recorder! Install on Proxmox LXC

All about POOLS | Proxmox + Ceph Hyperconverged Cluster fäncy Configurations for RBD

Thu, 29 Dec 2022 08:14:05 -0300

In this video, I expand on the last video of my hyper-converged Proxmox + Ceph cluster to create more custom pool layouts than Proxmox’s GUI allows. This includes setting the storage class (HDD / SSD / NVME), failure domain, and even erasure coding of pools. All of this is then setup as a storage location in Proxmox for RBD (RADOS Block Device), so we can store VM disks on it.

After all of this, I now have the flexibility to assign VM disks to HDDs or SSDs, and use erasure coding to get 66% storage efficiency instead of 33% (doubling my usable capacity for the same disks!). With more nodes and disks, I could improve both the storage efficiency and failure resilience of my cluster, but with only the small number of disks I have, I opted to go for a basic 2+1 erasure code.

This is part of the Hyper-Converged Cluster Megaproject.

Video

HDMI Distribution over your Home Network? Low-Cost HDMI Matrix using IP-Based Hardware

Thu, 08 Dec 2022 00:00:05 -0300

So, you want to send HDMI video around your house? Maybe you want to use your office computer on your living room TV without a proprietary streaming solution like AirPlay or Chromecast? Share a cable or satellite box between your living room and bedroom? Or you’re crazy like me and you want to put all of your computers into the basement, and connect to any of them from any desk in the house?

Traditional video distribution methods which support many-to-many configurations usually require expensive matrix switches, either for HDMI or HDBaseT. With lower cost IP-based equipment, we can use the network infrastructure we already have in our home networks to send HDMI video across the network, at the cost of video compression. If you can tolerate 1080P/60 video for your application, this is far cheaper than other alternatives in a many-to-many configuration.

I’ve tried running a thicc HDMI cable through the wall at my house. This will work, but over a limited distance (and the workable distance gets shorter with the higher bitrates of each HDMI spec). You can’t repair an HDMI cable realistically, so if you break it you’re going back in the wall or attic. I have a few places in my house with floating TVs (I absolutely hate cable cluttter) and running HDMI from the TV down to a media cabinet is as far as I’d go with a physical cable.

There are solutions like active optical cables and HDBaseT which are suitable for higher bitrate uncompressed applications like home theaters, but I’m primarily concerned with workstation tasks which aren’t as demanding of the video stream. However, these solutions might be right for you.

Video

Q and A

Will it do 4K?
- No, it does 1080p60
Will it do 144hz?
- No, it wont.
Does it support ultrawide?
- The transmitter and receiver negotiate resolutions separately
- The transmitter supports 800x600 up to 1920x1090 without knowing the display’s supported resolutions
- The receiver will separately try to negotiate 1920x1080 with the display, or lower if the display doesn’t support it
- When the transmitter and receiver are ’tuned’ together it will add black bars to match the two aspect ratios and scale the resolutions.
What type of encoding does it use? Can I open it in VLC?
- I do not know what encoding it uses but strongly suspect h.265
- I opened the multicast group/port in VLC as both UDP and RTP and VLC was not able to decode it. So, the encapsulation is unknown.
- I captured a pcap of the two boxes in use and have a link to it here.
Support for HDCP
- The TiVo negotiated HDCP 1.x successfully. I do not know if the device supports HDCP 2.x
- I can’t tell if any displays are negotiating HDCP, they won’t tell me in their menus
- I do not know if HDCP is being stripped or if HDCP is separately negotiated on each end.
What about NDI / other professional solutions?
- They exist
- They are more expensive
- NDI does not allow directly connecting sources and sinks without a vision mixer in the middle
- Other solutions are priced with ‘contact us’, meaning they are effectively not available to home users
Display Power Savings
- I noticed that the receiver will always display the no connection logo once the transmitter stops receiving a signal, so the display will never power off.
How are the compression artifacts?
- In general not bad.
- I watched some (CC licensed) movies side by side and had to scrub back and forth to spot differences
- In general it looks like the i-frame interval is relatively long, so sometimes scene changes in movies will cause some artifacting until the next i-frame. This would not happen on a computer or game where there aren’t major scene changes.
Does it work over WiFi extenders / Powerline extenders / MoCA coax?
- I don’t have any of these to test
- In theory it will work on anything that supports multicast and creates a layer 2 bridge (not layer 3 route)
What is the latency?
- 4 frames. See the video for more detailed info.
How does it handle multiple keyboards/mice?
- It sends separate data from each of them. Clicking one mouse and dragging the other won’t click + drag, it will take all of the mouse data from the moving mouse (release the click).
- Keyboards will send both keys but not mix scancodes, for example holding shift on one keyboard and pressing a letter on the other won’t result in a capital letter.
Why don’t they support PoE?
- I don’t see a reason to use PoE when they will always connect to a higher powered device anyway, like a TV or computer
What is the power consumption / can you really connect them to USB power?
- Measured ~1.5W on the receiver and ~2W on the transmitter, measuring the AC input with a kill-a-watt meter. Well within the 500ma limit of traditional USB power.

Reverse Engineering

My subscribers have been very interested in some light reverse-engineering of these, so if anyone wants to try, here’s the information I have to share:

Packet Captures

Display is established (ID #1), mouse movement and keyboard input on an Ubuntu jellyfish desktop - download here
Establishing a connection (ID #1), nothing connected, transmitter powered on and given time to settle, then receiver powered on and given time to settle - download here More will come.

Parts I Used

TESmart HKE12MMA20 KVM over IP (Newer Model, shown here)
TESmart hKE12MMA10 KVM over IP (Older Model) available on Amazon Some links to products may be affiliate links, which may earn a commission for me.

Import a Virtual Machine Template (OVA, VMDK, RAW, ...) into Proxmox!

Thu, 01 Dec 2022 10:00:00 +0000

Short tutorial this week, how to import a VM image you may get to use in Proxmox. I took a vacation for Thanksgiving, so back to the regularly scheduled madness soon. Enjoy!

The Basics: You need to use qm importvm (See the man page here if you want to reference it). The basic command is:

qm importvm <vmid> <source> <storage> <options>

Where <vmid> is the number of a VM which alraedy exists, <source> is the path to a file which qemu can convert (qcow2, vmdk,raw img), and <storage> is the name of a storage location in Proxmox. For options, they are only relevant if you are using file based storage, then you can pass --format with one of qcow2, vmdk, or raw to specify the file format to use. If you are using block based storage such as ZFS, LVM, … then this is ignored and unnecessary.

The Video:

Proxmox NETWORKING: VLANs, Bridges, and Bonds!

Thu, 17 Nov 2022 08:00:00 +0000

I’m sure many of you follow me because you use Proxmox. It’s been a staple of my content for some time now. So, while working on the next episode of the Ceph series, I thought it would be good to do a separate segment on networking. So, here you have it. The basics of VLANs, Bridges, and Bonds in Proxmox VE. I’m only covering the native Linux versions, not Open VSwitch and VXLAN. I’m sure I’ll get around to a video on that topic someday.

So, what are the most important things to know when choosing a network topology for Proxmox (or any virtualization environment)? TRAFFIC! Where is traffic going, and how much of it is going everywhere?

How much traffic is going to Proxmox itself? This includes the web UI and API (which should be minimal), but also SPICE sessions if you’re using SPICE for VDI.
How much traffic is going from Proxmox to your storage solutions? If you’re using NFS / SMB / iSCSI, it could be a lot. Are you keepign your storage network separated, either physically or virtually (VLANs)? Proxmox will need an IP address on any network you use to communicate with storage
How much traffic is going to your VMs? Do they need to be on specific VLANs?
Do any VMs do routing or need access to a VLAN trunk port? If so, should they get open access or restricted to certain VLANs? Do you want to expose each VLAN as a separate virtual network interface or trunk them over a single interface?
Do you require high availability at the network level, i.e. bonded failover? Do you want to use a slower 1G network when your 10G network fails, or just lose connectivity altogether?

Once you can answer these questions, you can proceed to decide how to arrange the physical interfaces you have (or are buying/adding) for the best performace for your use case.

In my test setup, I’m going to demonstrate bonding between identical (two Gigabit) and different (multi-gig + gigabit), and the concepts apply to 10G and faster networking as well.

And a good reference from Proxmox: Proxmox Networking Admin Guide

Video

Thin Client Addicts Anonymous - HP T530 Teardown/Review

Thu, 10 Nov 2022 00:32:05 -0300

I have a thin client addiction. You all have seen my 3x Dell Wyse 5060 Hyperconverged Cluster project. And you know that I bought a Dell Wyse 3040. But, I actually bought 3x 3040s, and someone sent me a Wyse 7010, and an HP T620 (yet to be reviewed). And now I bought another. An HP T530.

I’d consider this to be an excellent choice for anyone wanting to run Home Assistant, since it has enough power for, an upgradeable M.2 storage slot, and can handle a Coral TPU for Frigate using the M.2 WiFi slot (but not in the storage slot). It’s also a bit more modern than the Dell units, using DDR4 (only single channel though) and sporting a USB-C port and more USB 3.0 ports. That said, it’s a lower end processor and GPU than the 5060, if that’s a concern for you.

If you’d like to see my saved eBay search for T530’s, find it here (Affiliate link)

Video

Click on the thumbnail to view the video on Youtube

M.2 Compatibility

This unit has two M.2 slots - one E key (WiFi) and one M key (Storage). The M slot can fit up to 2280 size SSDs.

I found that the Storage slot electrically works for SATA only, not PCIe. Be aware of this when buying SSDs for this thin client.

I tested an Intel WiFi + Bluetooth card in the Wifi slot, and found that at least one PCIe lane and USB are working on that slot. I can’t confirm if the other PCIe bus, SDIO, etc. work on this slot.

Hardware Info

All of this was taken via an Xubuntu 20.04 Live ISO, so your kernel may be configured slightly differently

lscpu

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      21
Model:                           112
Model name:                      AMD Embedded G-Series GX-215JJ Radeon R2E
Stepping:                        0
Frequency boost:                 enabled
CPU MHz:                         1107.674
CPU max MHz:                     1500.0000
CPU min MHz:                     1100.0000
BogoMIPS:                        2994.56
Virtualization:                  AMD-V
L1d cache:                       64 KiB
L1i cache:                       128 KiB
L2 cache:                        2 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36
                                 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp 
                                 lm constant_tsc rep_good acc_power nopl nonstop_tsc cpuid extd_apicid 
                                 aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt 
                                 aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a 
                                 misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr 
                                 tbm perfctr_core perfctr_nb bpext ptsc mwaitx cpb hw_pstate ssbd vmmcall 
                                 fsgsbase bmi1 avx2 smep bmi2 xsaveopt arat npt lbrv svm_lock nrip_save 
                                 tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold 
                                 avic v_vmsave_vmload vgif overflow_recov

CPU does support aes-ni and AMD-V, but not SMT (hyper-threading). So, 2 cores, 2 threads.

lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) I/O Memory Management Unit
00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Stoney [Radeon R2/R3/R4/R5 Graphics] (rev 83)
00:01.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 15b3
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Host Bridge
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Processor Root Port
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Host Bridge
00:08.0 Encryption controller: Advanced Micro Devices, Inc. [AMD] Carrizo Platform Security Processor
00:09.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Carrizo Audio Dummy Host Bridge
00:09.2 Audio device: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 60h-6fh) Audio Controller
00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 20)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 4b)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 49)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 4b)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Stoney HT Configuration
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Stoney Address Maps
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Stoney DRAM Configuration
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Stoney Miscellaneous Configuration
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Stoney PM Configuration
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Stoney NB Performance Monitor
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

Realtek NIC, as expected for a device of this price range

vainfo

vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Mesa Gallium driver 21.2.6 for AMD STONEY (DRM 3.41.0, 5.13.0-30-generic, LLVM 12.0.0)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileNone                   :	VAEntrypointVideoProc

Some links to products may be affiliate links, which may earn a commission for me.

Waveshare UPS Battery Backup for your Raspberry Pi! With Data Logging to MQTT

Thu, 03 Nov 2022 00:00:00 +0000

As some of you may know, I have a Raspberry Pi which handles all of the radios for Home Assistant. It runs ZwaveJS2MQTT (using the WebSockets connection to Home Assistant, not MQTT), Zigbee2MQTT (using MQTT), and RTL-433 (also using MQTT). So, it’s fairly critical infrastructure. Usually, with critical infrastructure, I try to get it PoE powered so I don’t have to worry about power bricks and can get battery backup via the CRS328 network switch in the basement (which is on a battery of its own). But, unfortunately, the connection between my upstairs laundry room and basement network closet is via fiber, so no power being carried there. I need another solution, and not wanting a giant AC UPS I decided on a tiny Pi UPS from Waveshare.

It’s a hat for the Pi with two 18650 lithium batteries, a charger circuit, power monitor and balancing ICs, and an 8.4v to 5v buck converter to power the Pi itself. It comes with an 8.4v wall brick to power the whole thing, and is designed to be plugged in to wall power normally and transition to battery power without any interruption. But, that’s still not good enough for me. I want the data collected to get into Home Assistant. So, I wrote a custom Python script to push the power monitor data to MQTT, and made a video about it.

In a bit of a divergence from my other blog posts, I’m linking to my Github for the code and instructions. So, read the readme to install my code if you want to use it.

Video

Click on the thumbnail to watch the video on Youtube

Parts List

Waveshare Pi UPS (B) - Pogo Pins model is what I got - Amazon or Waveshare
Waveshare Pi UPS - female header hat is another model that I didn’t get - Amazon or Waveshare
Wavehare Pi UPS for Pi Zero - another model that I didn’t get but the code should work with hopefully - Amazon or Waveshare
My Software on Github

Backup Proxmox VE to the CLOUD! Backup Hook Scripts and S3

Thu, 27 Oct 2022 11:00:00 +0000

Proxmox has a pretty good backup scheduler, but it relies on the backup destination being mounted as a storage location. This implies that the backup destination needs to be a protocol that Proxmox supports - SMB (CIFS), NFS, … or Proxmox Backup Server. If you want to push your backups to a cloud service, you probably need something a bit more complicated. Thankfully, Proxmox’s backup scheduler thought about this and has a hook feature we can use for this purpose, and we can use any protocol supported on the Debian base system, including things such as FUSE or s3cmd.

In this project, I integrate S3 object storage with the Proxmox backup scheduler, copying resulting backups to a cloud storage service directly from the Proxmox system. The S3 protocol is ubiquitous among cloud object storage providers, so we aren’t tied to AWS - we can use any service such as Backblaze. In my case, I’m demonstrating this with Linode.

Video

Click on the thumbnail to view the video on Youtube

Proxmox VZDump hook scripting

Proxmox’s documentation mentions briefly that the --script argument can be passed to vzdump, the Proxmox backup utility. We can also add the line script <path_to_script> in the /etc/pve/jobs.conf file, and it will retain this script argument even if you modify the backup job from the GUI. You can’t add a script command from the GUI, but you need to initially create the job there.

Proxmox mentions a backup file (vzdump-hook-script.pl), but I was unable to find a copy easily. For your convenience, I’ve uploaded it to my site here for you to view. The tl;dr is that:

The first argument to the script is the phase, which is one of:
- job-start
- job-end
- job-abort
- backup-start
- backup-end
- backup-abort
- log-end
- pre-stop
- pre-restart
- post-restart
The ‘job’ type phases are called once for a job, while the rest are called for each backup in the job
The backup mode (stop/suspend/snapshot) and vmid are passed as arguments to the non-job types
Additional information is passed as environment variables for certain phases, including TARGET (the full path on the local system to the backup file), LOGFILE (the full path to the log file), and a few other useful attributes.

In our case, we are going to hook to the job-end, backup-end, and log-end phases and do the following:

For backup-end, copy the file TARGET to the S3 bucket and delete the original
For log-end, copy the file LOGFILE to the S3 bucket and delete the original
For job-end, check if there are any backups which should be expired and remove them. This is done only once, even if there are multiple VMs to backup in a single job.

S3 Tools

There are a few different open source options for general purpose file management using the S3 protocol. The most common are s3cmd and s3fs-fuse. The first provides a command line utility to perform actions on an S3 bucket (ls, put, get, delete, …) and the second implements a filesystem backed by an S3 bucket. Since the S3 protocol can only replace files (not modify parts of a file), s3fs-fuse can be quite slow depending on how files are written/read. Given the use of hook scripts here, I’ve decided to use s3cmd. It’s easily installed by running apt install s3cmd on Proxmox.

Once s3cmd is installed, we need to configure it by running s3cmd --configure. It will ask you a few questions, and the answers will depend on your cloud storage provider. They should have a guide on configuring S3 access with their service (i.e. Linode’s is here). I’ve walked through my own setup in the video.

Once we run configure, it will generate a file called ~/.s3cfg which includes the access key and endpoint URL. Since we ran this as root, it will be /root/.s3cfg on our Proxmox host. That’s fine, since backup jobs run as root.

If you’re interested in the details, feel free to read the s3cmd docs here.

Once you’ve verified that you have s3cmd installed and can s3cmd ls and s3cmd put to the target bucket, we can add the s3 put’s to our backup script.

Backup Expiration Script

I found a script to handle expiration of files in S3. I’ve copied it here for reference. I named it s3cleanup.sh on my system.

#!/bin/bash

# Usage: ./s3cleanup.sh "bucketname" "7 days"

s3cmd ls s3://$1 | grep " DIR " -v | while read -r line;
  do
    createDate=`echo $line|awk {'print $1" "$2'}`
    createDate=$(date -d "$createDate" "+%s")
    olderThan=$(date -d "$2 ago" "+%s")
    if [[ $createDate -le $olderThan ]];
      then
        fileName=`echo $line|awk {'print $4'}`
        if [ $fileName != "" ]
          then
            printf 'Deleting "%s"\n' $fileName
            s3cmd del "$fileName"
        fi
    fi
  done;

Don’t forget to chmod +x it once you are done.

Final Backup Script

Here is the final backup script which you can copy to your system. Again, don’t forget to chmod +x it.

#!/usr/bin/perl -w
# Hook script for vzdump to backup to an S3 bucket
# 2022 Andrew Palardy
# Based on example hook script from Proxmox

use strict;

# Define the name of your bucket here
my $bucket = "apalrd-proxmox";
# The S3 endpoint comes from the s3cmd --configure setup, it is not set here

# Define the number of days to retain backups in the bucket
# Only accepts days, doesn't check hours/minutes
my $retain = 30;


#Uncomment this to see the hook script with arguments (not required)
#print "HOOK: " . join (' ', @ARGV) . "\n";

#Get the phase from the first argument
my $phase = shift;

#For job-based phases
#Note that job-init was added in PVE 7.2 or 7.3 AFAIK
if (    $phase eq 'job-init'  ||
        $phase eq 'job-start' || 
        $phase eq 'job-end'   || 
        $phase eq 'job-abort') { 

        #Env variables available for job based arguments
        my $dumpdir = $ENV{DUMPDIR};

        my $storeid = $ENV{STOREID};

        #Uncomment this to print the environment variables for debugging
        #print "HOOK-ENV: dumpdir=$dumpdir;storeid=$storeid\n";

        #Call s3cleanup at job end
        if ($phase eq 'job-end') {
                system ("/root/s3cleanup.sh $bucket \"$retain days\"");
        }
#For backup-based phases
} elsif ($phase eq 'backup-start' || 
         $phase eq 'backup-end' ||
         $phase eq 'backup-abort' || 
         $phase eq 'log-end' || 
         $phase eq 'pre-stop' ||
         $phase eq 'pre-restart' ||
         $phase eq 'post-restart') {

        #Data available to backup-based phases
        my $mode = shift; # stop/suspend/snapshot

        my $vmid = shift;

        my $vmtype = $ENV{VMTYPE}; # lxc/qemu

        my $dumpdir = $ENV{DUMPDIR};

        my $storeid = $ENV{STOREID};

        my $hostname = $ENV{HOSTNAME};

        my $tarfile = $ENV{TARGET};

        my $logfile = $ENV{LOGFILE}; 

        #Uncomment this to print environment variables
        #print "HOOK-ENV: vmtype=$vmtype;dumpdir=$dumpdir;storeid=$storeid;hostname=$hostname;tarfile=$tarfile;logfile=$logfile\n";

        # During backup-end, copy the target file to S3 and delete the original on the system
        if ($phase eq 'backup-end') {
                #S3 put
                my $result = system ("s3cmd put $tarfile s3://$bucket/");
                #rm original
                system ("rm $tarfile");
                #Die of error returned
                if($result != 0) {
                        die "upload backup failed";
                }
        }

        # During log-end, copy the log file to S3 and delete the original on the system (same as target file)
        if ($phase eq 'log-end') {
                my $result = system ("s3cmd put $logfile s3://$bucket/");
                system ("rm $logfile");
                if($result != 0) {
                        die "upload logfile failed";
                }
        }
#Otherwise, phase is unknown
} else {

        die "got unknown phase '$phase'";

}

exit (0);

Sub-$100 Networked 433Mhz Receiver for Home Assistant

Thu, 20 Oct 2022 07:00:00 +0000

I previously wrote about my install of RTL-433 on a Raspberry Pi, running Raspberry Pi OS Buster. With the release of Bullseye, rtl-433 is now merged into the repository and doesn’t need to be compiled from source. So, I thought it would be a good time to revisit this project and make a video about it, this time using a cheap eBay thin client instead of a raspberry pi, and showcasing my setup a bit. As I’m using Home Assistant in my home automation system, anything that can publish to MQTT can be brought in, so buying fairly cheap off the shelf sensors which are rugged enough to survive for years outdoors is a great use case for me, even if there is virtually no security in the 433Mhz world. Overall, I was able to setup the receiver for under $100, which is pretty good as far as home automation products go.

Video

Click on the thumbnail to view the video on Youtube

Software Installation

First off, I chose Debian 11 (Bullseye) since it’s a pretty familiar OS to a lot of Linux users, it’s very easy to setup, and includes rtl-433 in the package repository now. So, after setting installing Debian on my thin client, I just need to install rtl-433, right?

Well yes, we do need to install it:

sudo apt install rtl-433

But once it’s installed, it doesn’t configure and launch itself as a service like we want. We can try out rtl433 by running it using sudo to get a feel for what options we might want for our service install. In my case, I use most of the default options and specify an MQTT server as the output:

rtl_433 -C si -F "mqtt://<server>:1883,user=<user>,pass=<pass>"

And now we need to make a systemd script so it can launch on boot and use the service logging functions of systemd.

sudo touch /etc/systemd/system/rtl433.service
sudo chmod 664 etc/systemd/system/rtl433.service
sudo nano /etc/systemd/system/rtl433.service

And the file contents (obviously change your MQTT configuration):

[Unit]
Description=rtl_433 SDR Receiver Daemon
After=network-online.target

[Service]
ExecStart=/usr/bin/rtl_433 -C si -F "mqtt://<broker>:1883,user=<user>,pass=<pass>"
Restart=always

[Install]
WantedBy=multi-user.target

If you installed via package like me, it will be in /usr/bin/rtl_433. If you compile it from source for any reason, it will be in /usr/local/bin/rtl_433.

And that’s basically it. Tell systemd to reload daemons since we changed it’s service files, then enable / start the service.

sudo systemctl daemon-reload
sudo systemctl start rtl433
sudo systemctl enable rtl433

Home Assistant Configuration

I’m using a relatively dated version of Home Assistant and I know there’s a lot of changes going on in the UI, but here’s what I have in my manual MQTT configuration:

# Outdoor temperature and humidity sensor (Acurite 609TXC)
- platform: mqtt
    name: "Outdoor Temperature"
    unique_id: "outdoor_temperature"
    state_topic: "rtl_433/zeus/devices/Acurite-609TXC/252/temperature_C"
    unit_of_measurement: "°C"
- platform: mqtt
    name: "Outdoor Humidity"
    unique_id: "outdoor_humidity"
    state_topic: "rtl_433/zeus/devices/Acurite-609TXC/252/humidity"
    unit_of_measurement: "%"

Parts and Software

RTL-433 Software
Dell Wyse 3040 Thin Client From Ebay
Nooelec NESDR Mini 2+ From Amazon or From Nooelec
Acurite 609TXC replacement sensor From Amazon or From Accurite (choose option 2) Some links to products may be affiliate links, which may earn a commission for me.

CNC Router Web Control Appliance

Thu, 13 Oct 2022 11:00:00 +0000

In this part of the 3018 Desktop Router project, I setup a permanent home for CNCjs on a Dell Wyse 3040 thin client. I’m running CNCjs as the CNC control software and G-code sender (the CNC’s grbl controller is actually doing the motion control). I’m using mjpg-streamer to add a USB webcam to the CNCjs web UI, with nearly no load on the CPU to encode. And I’ve setup a script to launch ffmpeg to record the mjpeg stream when g-code is started and stopped (also using nearly no load on the CPU to transcode). As a cherry on top, I’m running all of this with systemd services so everything autostarts properly on boot without the hacky cron suggestions of the CNCjs docs.

Any Linux single board computer will of course work, including a raspberry pi, but the 3040s are really cheap and tiny, and come with a case and storage for less than the cost of a bare Pi.

This video is the part of the CNC Router Megaproject.

Video

Click the thumbnail to view the video on Youtube

CNCjs

The actual web UI I’ve chosen is CNCjs, a node.js based web app for controlling CNC machines which use an open source control board based on grbl, Smoothieware, or TinyG. It’s actually just a fancy G-code sender, interpretation of the G-code and motion control is done by the router’s control board (which in many cases runs grbl).

Install

CNCjs is written in node.js and available through the node package manager. To install it though, we first must install node.js (package nodejs) and the node package manager (npm) - this will take a very long time:

sudo apt install nodejs npm

Once complete, we can use npm to install cncjs using node:

sudo npm install -g cncjs --unsafe-perms

CNCjs Test Run

To test cncjs, we should be able to just run it as the current user - try this command:

cncjs --allow-remote-access -p 8080

Then we can edit .cncrc to add allowRemoteAccess so we don’t need to pass it as an argument any more. See my final .cncrc at the end of this post for the configuration I ended up with.

To give us permissions to run on port 80 without root, we could delegate this capability to the cncjs binary (/usr/local/bin/cncjs, which is actually a symlink into /usr/local/lib/node_modules/). But, actually, we need to give this to the node.js binary, since that’s the process that ultimately will run and needs the permissions.

sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/bin/node

Also, to give permissions to use the serial port, we need to add our local user to the group dialout:

sudo usermod -a -G dialout discovery

CNCjs Systemd Service

This will make CNCjs run as a system service. We can do the usual stuff with systemctl, which is how services should be managed on distros which use systemd. So, let’s create the service file and edit it:

sudo touch /etc/systemd/system/cncjs.service
sudo chmod 664 /etc/systemd/system/cncjs.service
sudo nano /etc/systemd/system/cncjs.service

And now the service contents. Notably, running as our local user in the local user’s home directory.

[Unit]
Description=CNC Controller Web UI
After=network-online.target

[Service]
ExecStart=cncjs -p 80
User=discovery
WorkingDirectory=/home/discovery
Restart=always

[Install]
WantedBy=multi-user.target

Now, we can start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start cncjs
sudo systemctl enable cncjs

Webcam Streaming

We have a few pieces of software to install. Notably, v4l2-ctl and mjpg-streamer along with dependencies.

v4l2-ctl

If you want to see the capabilities of your camera or identify which one is which, you can use v4l2-ctl to do so. If you only have one camera and assume it supports MJPEG and is /dev/video0, you can skip this if you’d like.

Install it using apt:

sudo apt install v4l-utils

List devices - sudo v4l2ctl --list-devices

If you have more than one device (not /dev/video* nodes, actual devices) you can specify the device using -d /dev/videoX in the following commands. If you have multiple cameras, then do this for each.

Then list formats - sudo v4l2-ctl --list-formats - it should show MJPEG as an option. If it doesn’t, all will still work fine, but the CPU will encode the images as JPEG and it will put more load on the processor.

You can also do sudo v4l2-ctl --all to see everything. This should include the maximum resolution for the camera. You’ll need to note that for later.

mjpg-streamer

mjpg-streamer is a project which was originally designed to allow extremely low resource Linux systems to host camera streams, by relying entirly on the webcam hardware to JPEG-encode each image (resulting in a motion-JPEG stream). mjpg-streamer then only has to deal with shuffling bytes around to the HTTP clients. It’s been updated to support the Pi camera (if you want to use that), and can also encode to MJPEG if the camera doesn’t support it (at the cost of higher CPU usage). Since we need an HTTP MJPEG stream for the CNCjs camera widget, we can’t use h.264 here, so relying on the camera’s MJPEG codec is the lowest resource way to achieve this. The bitrate is fairly high (~35mbps for my setup) which will make the recordings bigger, but you can always transcode the recordings before keeping them.

Anyway, let’s install it. In this case, it’s not available from apt, so we are going to install dependencies through apt and then build it from source.

sudo apt install cmake libjpeg8-dev
git clone https://github.com/jacksonliam/mjpg-streamer
cd mjpg-streamer/mjpg-streamer-experimental
make
sudo make install

Webcam Test Run

mjpg-streamer has different modules to deal with input and output, so in this case we are using input_uvc which deals with v4l2 devices and output_http which provides an http webserver for streams and snapshots. You can pass arguments to both modules, in this case I’m passing a device and resolution (make sure to choose something your device supports!).

sudo mjpg_streamer -i "input_uvc.so -d /dev/video0 -r 1920x1080" -o "output_http.so -p 8080"

You can leave out the device option if you only have one camera. We need to use sudo to gain permissions to access the /dev/video devices, since we aren’t adding ourselves to the video group. Systemd will run the service as root by default anyway.

Webcam Systemd Service

First, create the script - you can replace webcamd with whatever you want, i.e. if you are using multiple cameras

sudo touch /etc/systemd/system/webcamd.service
sudo chmod 664 /etc/systemd/system/webcamd.service
sudo nano /etc/systemd/system/webcamd.service

And now the service contents. Make sure your devices supports the resolution you specify!

[Unit]
Description=Webcam Stream
After=network-online.target

[Service]
ExecStart=mjpg_streamer -i "input_uvc.so -d /dev/video0 -r 1920x1080" -o "output_http.so -p 8080"
Restart=always

[Install]
WantedBy=multi-user.target

Now, we can start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start webcamd
sudo systemctl enable webcamd

Recording

Finally, nearing the end of this big software dump. The final step is to start and stop recordings from CNCjs, preferably automatically, in a location we can access easily, and also with the ability to manually start and stop recordings.

ffmpeg

We will record the stream with ffmpeg. Depending on the GPU you are running on, you may be able to use libva hardware acceleration. In my setup, I am using mjpg passthrough, so the resulting file is a .mjpeg and is not transcoding to a more space efficient codec. This reduces CPU usage dramatically if libva doesn’t support both MJPEG decoding and h.264 encoding on your hardware. Not many programs can play back the .mjpeg file directly, but you can always transcode it to h.264 in ffmpeg or Handbrake for a more universal format (and you probably should to keep file sizes reasonable).

ffmpeg -i "http://localhost:8080/?action=stream" -codec copy /mnt/cnc/test1.mjpeg

Network Mount using Autofs

I’ve chosen to network mount my recordings directory using autofs. You can use a directory on the system if you have enough space. My 3040 thin client has 8GB of emmc, so it can’t really fit any recordings. Here’s the basic process to setup autofs for this use case:

Install autofs and the cifs (samba) client: sudo apt install autofs cifs-utils
Create a mount point: sudo mkdir /mnt/cnc
Edit /etc/auto.master using sudo and add the following line: /- /etc/auto.smb.shares --timeout 15 browse
Add a new share in that new file, again as root:

/mnt/cnc -fstype=cifs,rw,username=<user>,password=<password>,noperm ://server/share/directory

Then enable and start autofs

sudo systemctl enable autofs
sudo systemctl restart autofs

You should be able to see the network share - ls /mnt/cnc

Start/Stop Scripts

Now that we have a place to store the recordings and a command to record, we need to be able to spawn this from cncjs on command. Here are the scripts to do that with ~/rec-start.sh:

#!/bin/bash
#Date/time stamp for new recording
fname=$(date +%F_%H-%M-%S)
#Call ffmpeg with filename cnc_<date>.mjpeg in the background
ffmpeg -i "http://localhost:8080/?action=stream" -codec copy /mnt/cnc/cnc_$fname.mjpeg -nostats &
#End terminal session
disown

And to stop, we kill ffmpeg with ~/rec-stop.sh and it will clean up itself nicely:

#!/bin/bash
killall ffmpeg

Final CNCjs Configuration

Here is my final configuration file for CNCjs, in case you’d like to copy anything from it. All of this can be configured via the web UI if you like as well.

{
    "state": {
        "checkForUpdates": true,
        "controller": {
            "exception": {
                "ignoreErrors": false
            }
        }
    },
    "secret": "$2a$10$FBerJXHRoBaO0YD4nVZUM.",
    "allowRemoteAccess": true,
    "commands": [
        {
            "id": "727d8fa0-0790-44f0-a9bb-6fcb992fab2d",
            "mtime": 1665444504169,
            "enabled": true,
            "title": "Start Record",
            "commands": "/home/discovery/rec-start.sh"
        },
        {
            "id": "ec02236a-529d-4fba-a575-70fbce559222",
            "mtime": 1665444509575,
            "enabled": true,
            "title": "Stop Record",
            "commands": "/home/discovery/rec-stop.sh"
        }
    ],
    "events": [
        {
            "id": "80a0b85d-bcab-414a-9c22-0d8a25a83368",
            "mtime": 1665444515639,
            "enabled": true,
            "event": "gcode:start",
            "trigger": "system",
            "commands": "/home/discovery/rec-start.sh"
        },
        {
            "id": "811d3570-9200-4b9f-88db-38b518f870f6",
            "mtime": 1665444519136,
            "enabled": true,
            "event": "gcode:stop",
            "trigger": "system",
            "commands": "/home/discovery/rec-stop.sh"
        }
    ]
}

Parts and Links

Some of these may be affiliate links, which may earn a commission for me.

CNCjs software
Sainsmart Genmitsu 3018 PROver router:
- At Sainsmart
- At Amazon (Affiliate Link)
Webcam I am using
CNC tape that I tried and liked
Dell Wyse 3040 Thin Client Some links to products may be affiliate links, which may earn a commission for me.

New Toy! HP MicroServer Gen8

Thu, 06 Oct 2022 07:00:00 +0000

Today, I open a new gift to the homelab - an HP MicroServer Gen8. This little chonky cube is full of hard drives and not a whole lot else, making it a perfect test system for ZFS, TrueNAS, Proxmox VE and Proxmox Backup Server, etc. and I’m already planning the videos I want to make with it. So come along as I open it up and see roughly what’s inside, the specs, and use HP iLO (their proprietary IPMI) for the first time.

Video

There’s a video of my fun. Click on the thumbnail to watch it on Youtube.

Exterior

Here are some pictures of the exterior of the case. There aren’t many ports, dual gigabit NICs, two USB2, two USB3, VGA, and iLO’s dedicated Ethernet. The one PCIe half height expansion slot has been filled with an LSI SAS card.

Internals

Inside, there’s a handy SD card slot and USB port to plug in your boot media (if you don’t want to boot off the data drives). There’s also an extra SATA port that’s labeled ‘ODD’ (optical disk drive), although that isn’t installed on this model. This modified unit has the power cable for the DVD drive connected to a 2 port Molex to SATA power cable to power the extra 1TB 2.5" hard drives.

My Introduction to CNC - 3018 Desktop Router

Wed, 21 Sep 2022 00:00:00 +0000

My dad bought a Sainsmart Genmitsu 3018 PROver CNC router over a year ago, but never really set it up or learned how to use it. Since I enjoy the role of creating and maintaining tools and infrastructure, I decided to help him set it up and operate it to cut thin polycarbonate sheets. In this first episode, I hook the machine up to my laptop and try to cut out a representative model from 1.5mm polycarbonate sheet. It goes… okay… as you can see in the video.

I have big plans for this router in the future, including making it literally bigger (3040 upgrade), and setting up one of my absolute favorite Dell Wyse 3040 thin clients to run it standalone without a control computer. All of that’s in the future though.

This video is the first episode in the CNC Router Megaproject.

Video

Click the thumbnail to watch it on Youtube.

Parts and Software

Here are links to some of the hardware and software that make this project possible:

CNCjs - https://cnc.js.org/
FreeCAD - https://www.freecadweb.org/
grbl - https://github.com/gnea/grbl
Sainsmart Genmitsu 3018 PROver:
- At Sainsmart
- At Amazon (Affiliate Link) Some links to products may be affiliate links, which may earn a commission for me.

CNC Router Megaproject

Tue, 20 Sep 2022 11:00:00 +0000

In this project, I explore the setup, operation, and ’tricking out’ of a desktop 3018 class CNC router.

My Introduction to CNC - 3018 Desktop Router

In this video, I setup the machine and try to cut out a part using CNCjs. I quickly learn that it’s not the easiest thing in the world, makes a huge mess, and is loud. But I will continue on!

CNC Router Web Control Appliance

In this part of the 3018 Desktop Router project, I setup a permanent home for CNCjs on a Dell Wyse 3040 thin client. I’m running CNCjs as the CNC control software and G-code sender (the CNC’s grbl controller is actually doing the motion control). I’m using mjpg-streamer to add a USB webcam to the CNCjs web UI, with nearly no load on the CPU to encode. And I’ve setup a script to launch ffmpeg to record the mjpeg stream when g-code is started and stopped (also using nearly no load on the CPU to transcode). As a cherry on top, I’m running all of this with systemd services so everything autostarts properly on boot and recordings terminate if anything crashes and restarts.

A Viewer's Donation (Part 1) - Dell Wyse 7010

Mon, 16 May 2022 07:07:05 -0300

Casually tearing down the Dell Wyse 7010 (Zx0) that was kindly sent in by a viewer named Tom! Thanks Tom for making this happen. Tom also sent an HP thin client, but you guys only get one treat per video, and that one needed Torx bits that I didn’t have handy on the bench. tl;dr the CPU supports AMD-V (not that you really have enough RAM to think about virtualization, but you can expand it with ordinary DDR3 DIMMs), the GPU is kinda awful, and it has a Realtek NIC. I’d rather have the 3040 personally, it’s compatable in performance in a much smaller package. But, the 7010 is an older model which has upgradeable RAM and might be able to be found more cheaply.

Teardown Video

As usual, a teardown video of the hardware!

Hardware Info for the curious

All of this was taken via an Xubuntu 20.04 Live CD, so your kernel may be configured slightly differently

lscpu

xubuntu@xubuntu:~$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   36 bits physical, 48 bits virtual
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      20
Model:                           2
Model name:                      AMD G-T56N Processor
Stepping:                        0
CPU MHz:                         922.265
BogoMIPS:                        3292.78
Virtualization:                  AMD-V
L1d cache:                       64 KiB
L1i cache:                       64 KiB
L2 cache:                        1 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 c
                                 lflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 
                                 constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni m
                                 onitor ssse3 cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm ss
                                 e4a misalignsse 3dnowprefetch ibs skinit wdt hw_pstate vmmcall arat npt l
                                 brv svm_lock nrip_save pausefilter

Of note, it does support AMD-V if you want to do virtualization. AES is not in the list, so crypto offload for TLS is not available, making this a worse choice for a home server.

lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 14h Processor Root Complex
00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Wrestler [Radeon HD 6320]
00:01.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Wrestler HDMI Audio
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1)
00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 12h/14h Processor Function 0 (rev 43)
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 12h/14h Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 12h/14h Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 12h/14h Processor Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 12h/14h Processor Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 12h/14h Processor Function 6
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 12h/14h Processor Function 5
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 12h/14h Processor Function 7
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
05:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)

As you can see, it uses a Realtek Ethernet Controller

vainfo

libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/r600_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Mesa Gallium driver 21.2.6 for AMD PALM (DRM 2.50.0 / 5.13.0-30-generic, LLVM 12.0.0)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileNone                   :	VAEntrypointVideoProc

Significantly less capability than the other devices in codecs and there are no encoder slices available.

Net Booting the Proxmox VDI Client (feat. Alpine Linux)

Thu, 12 May 2022 07:00:05 -0300

This is a continuation of my previous article on the Net Booted Thin Client. The instructions got way too long, so I created a new article for the client setup. You need a functional server setup (TFTP, HTTP, iPXE) which I did in my previous post. I could use something like Linux Terminal Server Project, but that’s a bit overkill for this, and I wanted to learn Alpine anyway, so I’ve chosen to use Alpine Linux for the client operating system. It’s extremely basic.

Here, we’re gonna create an apkovl file. This is essentially a minimal set of the data on the system, so the system can be recreated. It includes all of the system configuration in /etc, any other directories you specify, and the /etc/apk/world file. When the APKOVL is loaded, it will install all of the apk packages in the world file, and apply the apkovl files on top of any defaults.

We are going to take the ISO image for Alpine Linux, boot it in a VM without a hard disk (live, no installation), and configure it to our liking as a thin client. Once we’re happy, we’re going to generate an apkovl file and save it to our http server, then delete the VM.

Here’s a table of contents. This assumes you have a functional netboot environment including TFTP, HTTP, and iPXE - such as what I setup in my previous article/video.

Video

Click on the thumbnail to watch the video for this!

Basic Alpine Environment

The basics of getting Alpine setup:

Login as root in the base system (no password by default)
Run setup-alpine, with the following selections:
1. Pick your keyboard layout
2. Setup hostname. I chose thinclient
3. Setup eth0 for DHCP with no additional configuration
4. Set a root password
5. Choose timezone (I chose America/Detroit)
6. No HTTP proxy information
7. Choose ‘1’ for the mirror (this is alpine’s primary mirror)
8. Choose none for SSH
9. Try boot media as a package repository (for now)
10. Disk = none (this will not install Alpine anywhere, just setup the system in RAM)
11. Where to store configs = none (this means it won’t have a place for lbu, but it’s okay)
12. APK cache directory = none (this means it has to go out to the internet to install anything)
Install nano - apk add nano
Edit /etc/apk/repositories to comment out the cdrom (first line) and uncomment the community repository (third line)
Setup the X11 server for graphics - setup-xorg-base
Install all the stuff we need for a basic kiosk via apk - apk add openbox xterm terminus-font font-noto
Add user - adduser vdi, follow prompts, then add to input and video groups addgroup vdi input and addgroup vdi video
If you want to be fun, you can edit /etc/motd with your own message, but nobody should see it.

Install PVE VDI Client

If you’re creating your own netbooted appliance based on this guide, you would install whatever software you want here (like Firefox or Remmina) using apk, set it up, and skip down to Configure Openbox to launch your kiosk software of choice graphically. Make sure all files you need for your appliance are stored in either your home directory (/home/vdi) or in /etc, otherwise they will be lost when we bundle the APK Overlay.

Install and configure PVE VDI Client in the simplest way possible. You’re free to add your own icons, etc. if you wish, but this uses the ones that Josh created already.

Install packages we need specific to the thin client - apk add python3 py3-pip py3-pyside2 virt-viewer git
Login as new user vdi (exit to logout)
Install pip packages - pip3 install proxmoxer PySimpleGUIQt requests
Upgrade PySimpleGUI to 0.35.0 since it doesn’t find pyside2 installed via apk - pip3 install PySimpleGUIQt==0.35.0 --no-deps
Clone PVE-VDIClient - git clone https://github.com/joshpatten/PVE-VDIClient.git
Make executable - chmod +x ~/PVE-VDIClient/vdiclient.py
Create config dir - mkdir -p ~/.config/VDIClient
Create config file - nano ~/.config/VDIClient/vdiclient.ini

And the contents:

[General]
#All of the settings in general can be customized if you want
title = apalrd VDI
theme = DarkAmber
icon = /home/vdi/PVE-VDIClient/vdiicon.ico
logo = /home/vdi/PVE-VDIClient/vdiclient.png

[Authentication]
auth_backend=pve
auth_totp=false
tls_verify=false

[Hosts]
#Replace this with your Proxmox host or DNS name
1.2.3.4=8006

Configure Openbox

Openbox is going to be the window manager here, so we need to configure it to start when X initializes, to launch the VDI client in a run loop, and to start X when the user vdi logs in

Set X to launch on login - echo 'exec startx' >> ~/.profile
Set openbox as default session - echo 'exec openbox-session' >> ~/.xinitrc
Copy Openbox default config - cp -r /etc/xdg/openbox ~/.config
Remove autostart file - rm ~/.config/openbox/autostart
New autostart file - nano ~/.config/openbox/autostart

And the contents:

while true
do
    ~/PVE-VDIClient/vdiclient.py
done

Test that the VDI Client Works

Run startx and ensure that the client starts correctly. If it doesn’t, right-click on the black background and launch xterm, then try running ~/PVE-VDIClient/vdiclient.py manually to see the terminal output.

Once you are done, right-click on the black background and launch xterm, then run killall xinit which should kick you back to the login screen. Try logging in again to make sure you are launched directly into the X session.

Autostart on Boot

Log back out of vdi (if you are currently in the X session, you have to do killall xinit)
Log in as root with password
Edit inittab to login as vdi automatically - nano /etc/inittab

#Find this line
tty1::respawn:/sbin/getty 38400 tty1
#Replace with this line
tty1::respawn:/bin/login -f vdi

Create APKOVL

Now we can generate the apkovl.tar.gz, still logged in as root:

Remove the .ash_history from vdi so our command history isn’t visible to anyone - rm -f /home/vdi/.ash_history
(Optional) Remove the packages we installed but only needed for setup so they aren’t usable in the final system - apk del git xterm nano - if you do this, you will have absolutely no way of debugging the setup once it netboots, so make sure it works at this point.
Include /home in the lbu package - lbu include /home
Package the system using lbu - lbu package thinclient.apkovl.tar.gz

Copy APKOVL to Netboot Server

Now we are going to temporarily stand up an HTTP server to offload our .tar.gz. Busybox includes an httpd server, but it’s not compiled in the busybox that Alpine includes by default. Thankfully, Alpine provides another busybox binary as a package that includes all options.

Install busybox-extras so we have the version with httpd - apk add busybox-extras
Get our IP address - ip a
Use busybox-extras to launch an httpd server here - busybox-extras httpd -f -v
Switch to the netboot server we set up in the previous episode, log in as root
Change to the www directory - cd /srv/www
Download the file - wget http://<temp client ip>/thinclient.apkovl.tar.gz

Testing

If all goes well, you should be able to boot your client, and as long as PXE is enabled higher than any other bootable devices in the boot order, it should launch into our thin client. You can even configure a VM in Proxmox with no disk and no DVD drive and it should netboot.

Not Every Project Works, And That's Okay (Multiseat USB Dongles)

Mon, 09 May 2022 08:49:05 -0300

Sometimes, projects don’t work. Today, I’m going to describe a bit about a few of them. Thank you for coming to my ted talk lol.

I’ve been working on Linux Multiseat for awhile now, it’s a topic that has fascinated me for over a decade now. But, getting it to actually work with cheap hardware has eluded me. So, here’s a bit of an overview of what I’ve learned so far.

The hardware I’ve tried:

WYSE Zero Client - Under $10 shipped, so it didn’t hurt to try. It uses a SIS USB VGA chip, which once had support in the Linux kernel, but a lot of the heavy lifting appears to be in the userspace xorg sis driver, and of course neither the kernel driver or xorg driver have been maintained in the past decade. Recompiling the kernel module and adding the USB IDs to the IDs which the driver supports got the kernel to load the correct module, but it provides a special device (not our favorite Linux framebuffer) and I wasn’t able to bind an xorg to it. Maybe with some work I could build a kernel space module which exposes a framebuffer instead of their awful junk, but it was done this way originally since the hardware has support for partial redrawing and hardware cursor (so the CPU doesn’t have to redraw the regions under the cursor as it moves), which is why it was originally part of the X server. If I get bored, maybe I’ll become a kernel developer I guess.
Pluggable USB2 DisplayLink dock - This one just came in the mail, and I haven’t even plugged it in yet, but this was designed to be used for multiseat way way back when, and Displaylink USB2 drivers for a framebuffer device are in the kernel and should work just fine. The hardware is a bit dated, with DVI outputs.
Random USB-C laptop dock - I tried several docks that I could borrow from friends and family to see how they respond. Most use USB-C DisplayPort, which is to be expected, so they rely entirely on the host GPU. This of course makes them cheaper to produce, but potentially at the expense of USB3 bandwidth. In lsusb they show up as a USB Billboard device, which basically means the USB endpoint exists purely to display a message on your computer saying it doesn’t support the dock. No data goes over the USB connection.
USB-C DisplayLink dock - One of the docks I tried supports DisplayLink over USB3. Unfortunately, DisplayLink has decided to go the Nvidia route and produce a proprietary Linux driver, and unlike Nvidia their Linux support is actually terrible. They claim it will support Ubuntu 20.04 LTS and don’t guarantee anything else, but even there it didn’t really do what I wanted. I was able to get display mirroring to work, but not multiseat. Maybe, with more time, I’d be able to fight with the driver enough to get multiseat to work. It’s really unfortunate too, since USB3 DisplayLink docks are really common and somewhat affordable now, at least compared to adding a new GPU like I did in my multiseat gaming video.

And of course, the video:

Making the $250 Proxmox HA Cluster Hyperconverged

Thu, 05 May 2022 08:14:05 -0300

I previously setup a Proxmox high availability cluster on my $35 Dell Wyse 5060 thin clients. Now, I’m improving this cluster to make it hyperconverged. It’s a huge buzzword in the industry now, and basically, it combines storage and compute in the same nodes, with each node having some compute and some storage, and clustering both the storage and compute. In traditional clustering you have a storage system (SAN) and compute system (virtualization cluster / kubernetes / …), so merging the SAN into the compute nodes means all of the nodes are identical and network traffic is, in aggregate, going from all nodes to all nodes without a bottleneck between the compute and SAN nodes.

This is part of the Hyper-Converged Cluster Megaproject.

Video

Here’s the video! Click on the thumbnail to view it

Cost Accounting

Since the $250 number is important in the Youtube title, here’s the cost breakdown:

I spent $35 each on the thin clients. The seller had a 10% discount for buying 2 or more, so I bought one at full price followed by two more at $31.50 each.
I spent $25 each on the 8G ram sticks to bring each node up to 12G of RAM
I spent $16 on each 128G flash drive for my hyperconverged storage. I should note that these guys are pretty darn slow comapred to spinning rust even.

Ceph Install

Step Zero, have a functional Proxmox cluster. You don’t need any storage or VMs, but at least have the nodes already physically connected, Proxmox installed, and joined.

Step One, go to one node and click the Ceph tab. It will tell you Ceph is not installed. Install the latest, as of the writing of this the latest is Pacific (16.2). You may have to hit enter when the console asks you if you’re sure you want to install, then wait for apt to do its magic.

Since this is our first node, we don’t have a Ceph configuration yet, so it will guide us through creating one.

Select your public and cluster (private) networks. I’m going to make a more detailed video on Ceph networking at some point hopefully, but the cluster network is used when Ceph communicates with itself (i.e. rebalancing, distributing copies of data between cluster members) and the public network is used when clients (including Proxmox’s RBD client) communicate with the Ceph cluster. Normally the cluster network will handle roughly twice as much traffic as the public network, since data replicated 3 times will be first sent over the public network to the first OSD, which will then send it to the other two OSDs via the cluster network. However, during any drive additions or removals, or when rebalancing is needed, the cluster network can potentially handle significantly more data. You can use a single network for both public and cluster if it is fast enough.

Also, select the node for the first monitor. I’m not sure why they bother asking, if you don’t have a cluster setup yet it must be the node you just installed Ceph onto, since you won’t have Ceph installed anywhere else.

Once you’ve finished the wizard on the first node, go to each other node, click Ceph, it’ll say Ceph is not installed, and install it. Once you finish with the apt install, it should say the cluster is already configured and not let you configure it again, so click to finish.

After everything is installed, you should have a configured cluster, with the monitor and manager installed on the node you chose first. Click on any node under Datacenter, go to Ceph -> Monitors, and create two more so there are 3 total in the cluster. If you have a bigger Proxmox cluster, spread them around a bit if possible, but you don’t need more than 3.

Ceph Manager and Dashboard

Proxmox doesn’t install Ceph’s dashboard by default, and they have their own dashboard which shows an overview of the cluster, but I like Ceph’s dashboard a bit better. Plus, it’s much easier to manage more complex pool arrangements through Ceph’s GUI than using the command line.

So, we need to install it. The dashboard runs as part of the manager, so you only need to install it on nodes which are going to run the manager. You don’t need many managers, I only setup 2 in this (they are active/passive, not active/active).

The setup steps, from the shell of the Proxmox system:

Install the manager package with apt install ceph-mgr-dashboard
Enable the dashboard module with ceph mgr module enable dashboard
Create a self-signed certificate with ceph dashboard create-self-signed-cert
Create a password for the new admin user and store it to a file. Ceph is actually picky about password rules here. echo MyPassword1 > password.txt
Create a new admin user in the Ceph dashboard with ceph dashboard ac-user-create <name> -i password.txt administrator - ‘administrator’ is the role that Ceph has by default, so this user can then create more users through the dashboard
Delete the password file - rm password.txt
Restart the manager or disable and re-enable the dashboard (ceph mgr module disable dashboard and ceph mgr module enable dashboard). I rebooted the node here. The documentation suggests this shouldn’t be required.

As of 2024, create-self-signed-cert is no longer functional, however, it will return an error message telling you exactly what OpenSSL commands to execute to generate a self-signed cert, so you can follow those instructions.

Object Storage Daemons

Ceph stores data a bit differently than a filesystem most of my viewers/readers are probably familiar with - the venerable ZFS.

In ZFS, you group disks into vdevs, where redundancy is handled at the vdev level. You can then designate certain vdevs with device classes, which are different from normal data disks. This includes L2ARC (second tier of cache), SLOG (dedicated fast space for the synchronous ZFS Intent Log), SPECIAL (separate devices to store metadata separately from file contents, to speed up access to directory listings and the like), and DDT (dedicated to storing deduplication tables). ZFS’s metadata stored on disk (or the special vdev) contains the vdev and sector on disk where the data is, so ZFS can start from the uberblock (which contains the pointer to the root of the metadata tree and is stored at several well-known locations on disk) and follow all of the vdev+sector addresses all the way down to individual blocks of files.

In Ceph, each disk is an OSD (Object Storage Daemon). The OSD should be backed by a single data disk, not hardware or software RAID and not a partition. Each data disk then has its own daemon which communicates with the Ceph monitor and other OSDs, so there’s an almost 1:1 correlation between the daemon and physical data disks. Reduncandy is done at a higher level, with Ceph computing which OSD should store data blocks. The Ceph system uses the ‘CRUSH Map’ to compute the location of data to the OSD level, so the client can communicate directly with the OSD hosting the data. However, within the OSD, the OSD needs to keep track of which sectors on its local disk it has put that data, and this goes in the database. The OSD can keep the database on the data disk, or we can optionally put the OSD’s database on a separate disk (guidance suggests allocating 2-4% the size of the disk depending on the workload) to speed up metadata access, similar to a SPECIAL devices in ZFS. Additionally, like ZFS, we can optionally give it a faster device to store synchronous writes to reduce the commit latency (although this does NOT improve throughput), known as the WAL (Write-Ahead Log). If you give the OSD a dedicated DB disk it will automatically use a small bit of that as the WAL, but if you don’t have space for a dedicated DB disk you can take a small partition of another disk to just have a dedicated WAL (or both, I guess).

Since redundancy is usually specified at the host level, it’s safe to assume that all disks within a single host could fail at the same time (unless you change the redundancy to be at the OSD level), therefore it’s safe to use a partition of a faster disk as the DB for multiple data disks. For example, with a 10TB spinning disk you would want ~400G of DB, so taking a 2TB NVMe SSD and partitioning it to use as the DB disk for 5x spinning disks would be reasonable. Failure of the DB would mean all 5 spinning disks would effectively be lost (since the data can’t be located on disk), but we already assumed host-level failure anyway. If you ARE using OSD level redundancy, then don’t use partitions for your DB disks.

The final note here is the device class. You can specify either HDD, SSD, or NVMe by default. This isn’t immediately useful, but later we can set rules specifying certain data pools must be located on a certain device class, so setting it correctly is always a good idea. This allows you to avoid creating multiple Ceph clusters in a mixed storage environment. For example, you could create a hyperconverged cluster with Proxmox, having a mix of NVMe and Sata SSD at each node, plus additional storage only nodes with a ton of HDDs. Your VM disks can be pushed to NVMe tiered with SSD, and CephFS can then use SSD tiered with HDD, all in the same cluster, like magic.

Now to actually set it up: First find the location of the disk (/dev/* path) and zap it: ceph-volume lvm zap /dev/sdX --destroy

Once the drive is zapped you can add it in Proxmox. Go to the node with the drive, go to Ceph -> OSD, click Create OSD, and it should find the unused disk.

Absolute Basics of Pools

I’m just covering the absolute basics of pools here, what you can create in Proxmox’s storage pool dialog. There are of course many other ways to configure pools, and I may produce more content in the future on Ceph.

Create a pool in Proxmox by going to Ceph -> Pools, click Create, enter the information. Of note:

Name it whatever you want. Proxmox will use this name as the name of it’s corresponding storage.
PG Autoscale should usually be on. This dynamically resizes the number of placement groups (PGs) based on how much data is in the pool.
Size of 3 means the pool will try to have 3 copies of all data. Min size of 2 means the pool will allow write operations to complete when only 2 of the copies are completed, and allow continued operation on the pool when at least 2 of the copies are intact. This means you can never get into a scenario where losing a host causes loss of any data (as long as Ceph has completed the write), as the write is not completed until the data is on at least 2 hosts, and you can continue operating with at least one host down.
The CRUSH rule (replicated_rule) is the default, we will get into how to configure this in figure episodes of the series. But the CRUSH rule says things like which disk type and what the failure domain is (defualt host redundancy, and default any disk type). Proxmox has no GUI to configure this.

Mikrotik SXT US with T-Mobile - LTE Backup for the Homelab?

Mon, 02 May 2022 09:00:05 -0300

I’ve wanted to play with multi-WAN routing setups for awhile. I thought about buying Starlink, but it’s a bit pricey as a purely backup solution, and I’m in a situation where my primary internet is very reliable for about the same price. So I settled for LTE backup. I looked around for awhile, trying to decide what the right solution is for me. I really wanted to pass through the IPv4 address and IPv6 prefix from the ISP through to my OPNsense router, since I don’t want to deal with triple-NAT and dual layer firewall. A lot of LTE hotspots, even those that support wired Ethernet, still act as firewall routers, and I didn’t want that hassle.

Oh, and also a rant on how we should stop encouraging people to continue using IPv4 and deploying IPv4-only networks because it’s easier. IPv4 is legacy and should be for backwards compatibility only when you are traversing the internet.

The Video

Feel free to watch my video on this topic! Click the thumbnail to view it I made a big oof in the video and used fast.t-mobile.net instead of fast.t-mobile.com. oops.

Slight Background on Mobile Networking, and IPv6 Rant

Back in the early days of cellular data, there was a lot of overhead in encapsulating IP frames within cellular data channels. The oldest digital cellular system in use, GSM, was originally designed to carry circuit switched voice traffic and had the option for circuit switched data (basically using the fixed-bandwidth voice channel as an extremely expensive modem). Packet traffic was eventually layered on top, and packets had to be carried across the T1/E1 network which was really not designed to carry packets. GSM used TDMA, ‘Time Division Multiple Access’, so each client had a certain time slot in the sequence to transmit a fixed amount of data.

Eventually 3G came, and it could do variable-bandwidth channels using its CDMA scheme. Both the worldwide UMTS standard and the US/CA CDMA2000 standard used Code Division Multiple Access, where transmissions between all clients and the tower were done at the same time and encoded using different orthogonal codes (Walsh/Hadamard codes). This basically added a ton of error correction code and then relied on the fact that the error correction encoding was different for each client to decipher the message intended for this specific client. The upside to this scheme was the variable bandwidth of the channel, so instead of transferring the client from the RACH channel (random access channel - mobile requests a control channel) to the low-bandwidth control channel to a different higher bandwidth data channel, the tower could change the channel bandwidth as needed. But the system was still primarily circuit-based, with better support for packets.

Now we have LTE. LTE is a purely packet based network. The entire system is based around IPv6. The RAN (radio access network) translates radio waves to/from IPv6 right at the tower - the eNodeB. From there, it’s all networking gear and specialized software.

The upside to this is we get native IPv6 subnets routed to the client device, and the use of IPv6-only mobile networks has really improved IPv6 uptake (and we desperately need to improve IPv6 uptake). The downside is we don’t have native IPv4, like at all. No dual-stack like the cable and fiber companies. You just don’t get IPv4.

The solution mobile operators have to deal with legacy IPv4 services is to use a technique called 464XLAT as well as CGNAT, and also DNS64 is an optional solution as well.

464XLAT is a way to rewrite the IP header of an IPv4 packet into an IPv6 one, using a special 96-bit prefix and filling the remainder with the IPv4 address. The mobile device (CLAT - client / customer side) will rewrite IPv4 packets into IPv6 ones, send them over the RAN and throught the mobile network, where they are terminated by the PLAT (provider-side), which is the gateway to the IPv4 world.

DNS64 is a method which can replace the CLAT by rewriting A records (IPv4) with AAAA records (IPv6) using the 96-bit xlat prefix when the site has no native AAAA records, so the application which requested access will then do a native IPv6 connction to the PLAT. For sites which have native AAAA records, the application can stay on IPv6 the entire time.

Due to IPv4 address exhaustion, at this point we really need to stop giving out public IPv4s to home internet users. If your game server can’t handle IPv6 the devs have only had two decades to fix it. So, anyway, mobile ISPs can’t allocate a public IPv4 to each device so it can talk to the legacy IPv4 world, so the PLAT will also do CGNAT - network address translation. In theory the devices should be assigned an IPv4 in the 100.64/10 prefix, but T-Mobile seems to be re-using the 6/8 prefix that belongs to the DoD, presumably because the 100.64/10 prefix hadn’t been formalized and they were already doing IPv4 CGNAT long before LTE and IPv6 for their GSM network dating back to the late 90s.

The Hardware

I ended up selecting the Mikrotik SXT US Kit, which includes a US band LTE modem, decently directional antenna, wired Ethernet with PoE, and RouterOS. RouterOS is …. dense with configuration options, and I’m not that good at managing it, but it’s supposed to be able to do a pure passthrough of the LTE modem to an interface without interfering with the network at all.

I was able to configure this in RouterOS eventually. I’m happy with the hardware so far. I stuck it in a window facing roughly the direction of my nearest T-mobile tower and it just worked. I’m sure if I was more on the fringes of cell service I would have needed an actual pole and to actually aim it.

RouterOS

I started with the default config that the SXT had. It really wanted to be a firewall router, but I gave it a static IP on my internal LAN network instead and turned off DHCP and firewalling. I then created a new VLAN interface in RouterOS for the LTE modem, and bridged the LTE modem to the VLAN using this command:

/interface lte apn
add apn=fast.t-mobile.com ip-type=ipv4-ipv6 passthrough-interface=LTEWAN

For some reason RouterOS defaulted to ipv4 only, no idea why. adding ip-type fixed that.

I then plugged it into my Mikrotik CRS328-24P-4S+ switch, it got 802.3af PoE, used the untagged network with its LAN IP for management, and has a tagged network for the internet connection back to OPNsense.

Here’s the full config I ended up with:

[admin@MikroTik] > export 
# apr/26/2022 20:06:56 by RouterOS 6.45.9
# software id = XXXX-XXXX
#
# model = RBSXTR
# serial number = XXXXXXXXX
/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment=defconf name=bridge
/interface vlan
add interface=ether1 name=LTEWAN vlan-id=2
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
add apn=fast.t-mobile.com ip-type=ipv4-ipv6 passthrough-interface=LTEWAN passthrough-mac=auto
/interface lte
set [ find ] apn-profiles=fast.t-mobile.com mac-address=xx:xx:xx:xx:xx:xx name=lte1
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile
set [ find default=yes ] html-directory=flash/hotspot
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=ether2
/interface list member
add comment=defconf interface=bridge list=LAN
add interface=lte1 list=WAN
/ip address
add address=172.27.0.9/20 comment=defconf interface=ether1 network=172.27.0.0
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf gateway=192.168.88.1 netmask=24
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=172.27.0.9 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" disabled=yes ipsec-policy=out,none out-interface-list=WAN
/system clock
set time-zone-name=America/Chicago

OPNsense

I was able to get DHCP (IPv4) and SLAAC (IPv6) working. T-Mobile doesn’t do IPv6 PD via DHCPv6, so SLAAC is what I get. It seems like the SLAAC prefix is stable as long as the modem is powered on, and it gets a new one when the modem reboots. This correlates with phone users reporting a stable prefix through tower handover and phone sleep unless they reboot the phone. Not totally unexpected. Unlike DHCPv6 there isn’t a native IPv6 way to request a static SLAAC prefix since the DUID is only used in DHCPv6 (although they could tie prefix to IMEI for a mobile-specific solution). The lack of a delegated prefix means we can either have exactly one IPv6 subnet in the LAN side (passing through the prefix we got from upstream) OR we can use exclusively DHCPv6 and smaller than /64 subnets internally. Android’s core devs are stupidly opposed to using DHCPv6 in Android, but every other OS supports it.

I tested with a temporary server instead of OPNsense and was able to successfully receive inbound connections to my IPv6 address (with the server connected directly to the LTE modem), although I’ve heard T-mobile blocks ICMP (ping) packets. HTTP inbound worked just fine. There’s sometimes a huge latency in initiating the connection (up to 1 sec for the TCP socket to connect). Presumably it has to wait for the mobile side to wake / check for notifications from the tower, and LTE is designed to conserve battery power so it has a relatively lengthy wake up interval compared to WiFi. Once the session is up it seems like latency isn’t bad at all.

Setting up a Netboot (PXE) Server for Alpine Linux

Wed, 27 Apr 2022 08:00:05 -0300

I’ve played with network booting before (and I even tried to netboot Windows, what a nightmare that was), but now I want to get serious about it. I want to netboot my VDI Clients. I’ve been working on a thin client series, and the next step is to get rid of the installation entirely. I could use something like Linux Terminal Server Project, but that’s a bit overkill for this, and I wanted to learn Alpine anyway, so I’ve chosen to use Alpine Linux for the client operating system. It’s extremely basic. However, even using Alpine is still a huge setup, so this article will focus on setting up a netboot server (also using Alpine Linux), to prepare for the next video where I create the Proxmox VDI Client using this netboot system.

Here’s a table of contents:

Video

Click on the thumbnail to watch the video for this!

The Basics

The boot process looks something like this:

BIOS/UEFI on victim machine decides it should network boot, either because that’s the highest priority boot order or it can’t find any other options.
BIOS/UEFI loads the PXE ROM from the network card and executes it.
PXE code does a DHCP DISCOVER, finds the DHCP server, and receives a next-server. It may also receive one or more boot filenames for UEFI/BIOS.
PXE code downloads the boot filename from the next-server address via TFTP and executes it. If filename is not provided, it first does a DHCP request to the address of next-server (instead of broadcast like normal) to get the filename. This method is called ‘Proxy DHCP’. We will be using that method today.
We’ve compiled a special version of iPXE with an embedded script, and that is the boot code being executed at this point. iPXE is a ‘better’ PXE ROM. Here we have the option of hardcoding the boot commands into an embedded script. The script also has the option of requesting more commands over HTTP, so you can compile a single version of iPXE and use the HTTP server along with CGI/PHP/… to decide what boot commands a specific client needs, since the HTTP request can include things like the MAC address formatted in the URL. We will not be doing this today, but it’s a future enhancement.
The script includes commands to download and load the kernel and initrd over HTTP instead of TFTP, so using iPXE saves us a ton of time over using TFTP and gives us more control over what arguments to pass to the kernel.
For Alpine we can add a modloop argument which specifies the HTTP address to download the modloop file. If you provide it, that file will contain all of the additional kernel modules that may be needed after the early boot process.
For Alpine, we can pass the URL of an Alpine mirror as a kernel command line argument, so the system has a working package manager by default. Normal systems would use the CD/USB IMG as the mirror until you configure networking, so this can get a fully functional working system without an image at all.
For Alpine, we can pass the URL to an APKOVL file which will be applied on top of the initramfs root image. This usually includes /etc but can include more. It also includes the /etc/apk/world file, which is the list of packages that should be installed. Essentially, given an apkovl file, Alpine will rebuild the installation that was saved previously, using a combination of downloading and installing packages from the mirror configured in the /etc/apk/repositories file and the configuration and user data in the APKOVL file.
Due to the modloop and apkovl, we have a completely running system using only 4 files on the server (vmlinuz, initramfs, modloop, apkovl) which will then configure itself exactly as we want. We can export the configuration of a running system using lbu to generate the apkovl file, and the other 3 files are provided by Alpine for netbooting. If we include /home in our apkovl, we can include all of the configuration we need for the thin client without having to mount anything off NFS.
(Optional) If we are doing this in a big organization, we shouldn’t hammer the Alpine package servers every time a thin client boots, so you should setup your own local mirror. That’s beyond the scope of this tutorial. Alpine provides a public rsync server for mirrors to synchronize from, and you can configure your mirror to only mirror specific Alpine versions and architectures to save space and pass requests to the primary mirrors otherwise.

Netboot Server Setup

For the server, we can choose whatever OS we want. But, I’m into Alpine now, so I’m going to use Alpine for the server as well. I’m installing it on Proxmox VE as a VM. I used the full extended ISO, but I don’t think that’s necessary. I ran setup-alpine to install it to disk using sys mode, which is the full system installed on disk and not running from RAM.

Alpine is pretty darn minimal. We need a text editor, it doesn’t come with nano.

#Nano is a text editor that's useful
apk add nano

Now that we have nano we can add the community repository (nano /etc/apk/repositories) and uncomment the line ending in community, but not the ones with edge in them.

And finally, enable root SSH login for now so we can setup the system (disable this once you are done): nano /etc/ssh/sshd_config, find the line starting with #PermitRootLogin and change the line to PermitRootLogin yes. Restart the server using rc-service sshd restart. From now on you can use ssh for the rest of the commands.

Setup HTTP Server

Going to use nginx here, since we need PHP support in the future. You could also use lighttpd.

#Install it
apk add nginx
#Add www user for nginx to run as
adduser -D -g 'www' www
mkdir /srv/www
chown -R www:www /var/lib/nginx
chown -R www:www /srv/www
#Backup old nginx config since we will rewrite it
mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak

Now we need to create an nginx config as simple as possible, since the default one will 404 on everything. So nano /etc/nginx/nginx.conf and paste this in:

user www;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx/nginx.pid;

events {
    worker_connections 1024;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    sendfile on;
    access_log /var/log/nginx/access.log;
    keepalive_timeout 3000;
    server {
        listen 80;
        root /srv/www;
        index index.html index.htm;
        server_name localhost;
        client_max_body_size 32m;
        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
            root /var/lib/nginx/html;
        }
    }
}

And finally configure it to start on boot:

#Start on boot
rc-update add nginx
#Start now
rc-service nginx start

Download Alpine Netboot

Alpine provides netboot images in a .tar.gz format. We need to put these in our /srv/www directory.

Go to the Alpine download page and copy the link to the netboot x86_64. As of the writing of this, the file is called alpine-netboot-3.15.4-x86_64.tar.gz.

Now download the file:

#Move to web directory
cd /srv/www
#Download alpine link
wget <paste the link here>
#Untar it
tar -xzf alpine*
#Delete the tar
rm alpine-netboot*
#Files are now in boot folder

There are two versions of the files, virt and lts. Virt is stripped down to only include the drivers needed to run on common hypervisors, while lts should include more driver support.

Setup TFTP Server

We’re gonna need a TFTP server for the ipxe image, since PXE normally requires TFTP to be used. Let’s get that setup now.

#Install it
apk add tftp-hpa
#Add it to run as a service
rc-update add in.tftpd
#Start service now
rc-service in.tftpd start
#Default directory is /var/tftpboot/

Since tftp-hpa creates a chroot for itself, it can’t follow symlinks up the directory tree, so we will need to copy our files into /var/tftpboot/ instead of symlinking them. We will need to be aware of this when we install iPXE later.

Download & Configure iPXE

Now that we have a functional server, we need to write our embedded ipxe script to download vmlinuz and friends over HTTP, then compile this into an ipxe binary and put it on our tftp server.

First we need to install git then clone the ipxe repo:

#Install git, make, gcc, perl, all the stuff ipxe will need
apk add git make binutils mtools perl xz-dev libc-dev gcc
#Going to put ipxe in the srv folder
cd /srv
#Clone ipxe
git clone https://github.com/ipxe/ipxe.git
#cd into it
cd ipxe/src

Now we can create an embedded script (nano netboot.ipxe):

#!ipxe

#Init networking
dhcp

#Networking info we got from the DHCP server
echo next-server is ${next-server}
echo filaneme is ${filename}
echo MAC address is ${net0/mac}
echo IP address is ${ip}

#Set flavor to lts
set flavor lts
echo flavor is ${flavor}

#Set command line 
set cmdline modules=loop,squashfs quiet
echo cmdline is ${cmdline}

#Server address
set server http://${next-server}
echo server is ${server}

#Kernel file
set vmlinuz ${server}/boot/vmlinuz-${flavor}
echo vmlinuz is ${vmlinuz}
set initramfs ${server}/boot/initramfs-${flavor}
echo initramfs is ${initramfs}

#Modloop file
set modloop ${server}/boot/modloop-${flavor}
echo modloop is ${modloop}

#Repository for apk
#Update this if you'd like a newer version of Alpine
#Alternatively, set branch to edge for the absolutel latest
set mirror http://dl-cdn.alpinelinux.org/alpine
set branch v3.15
set repo ${mirror}/${branch}/main
echo repo is ${repo}

#apkovl file - set this if you want to apply
#an apkovl file to configure the Alpne instance
set apkovl ${server}/thinclient.apkovl.tar.gz
echo apkovl is ${apkovl}

#Uncomment this if you want to see the information before continuing
#prompt Press any key to continue

#Kernel, initrd
#For EFI, we need to tell the kernel the initrd filename. For BIOS it doens't hurt to leave the initrd argument.
#If you want to use Alpine bare, use this line:
#kernel ${vmlinuz} ${cmdline} alpine_repo=${repo} modloop=${modloop} initrd=initramfs-${flavor}
#If you want to use Alpine with an apkovl, use this line:
kernel ${vmlinuz} ${cmdline} modloop=${modloop} apkovl=${apkovl} initrd=initramfs-${flavor}
initrd ${initramfs}

#Boot
boot

#Pause if errors
prompt Some error occurred, press any key to continue

Then we build ipxe (make a script nano build.sh then chmod +x build.sh to automate this):

#Build BIOS version (x86 but should boot into x64 environment)
make bin-i386-pcbios/undionly.kpxe EMBED=netboot.ipxe
#Build EFI version (x86)
make bin-x86_64-efi/ipxe.efi EMBED=netboot.ipxe
#Copy files to tftp root
cp bin-i386-pcbios/undionly.kpxe /var/tftpboot/
cp bin-x86_64-efi/ipxe.efi /var/tftpboot/ipxe64.efi
#The APKOVL we are using is for x64, so not building ipxe32.efi right now
#Also not building arm variants for this project

After this, we should have the binaries in our tftp directory and can enable netbooting in our DHCP server

GCC 12 Resolution

Error-checking in GCC 12 results in many compile errors for iPXE, as referenced by this issue. Until the issue is resolved on their end, use this script instead to treat errors as warnings:

#Build BIOS version (x86 but should boot into x64 environment)
NO_WERROR=1 make bin-i386-pcbios/undionly.kpxe EMBED=netboot.ipxe
#Build EFI version (x86)
NO_WERROR=1 make bin-x86_64-efi/ipxe.efi EMBED=netboot.ipxe
#Copy files to tftp root
cp bin-i386-pcbios/undionly.kpxe /var/tftpboot/
cp bin-x86_64-efi/ipxe.efi /var/tftpboot/ipxe64.efi
#The APKOVL we are using is for x64, so not building ipxe32.efi right now
#Also not building arm variants for this project

Configure Netboot in OPNsense

I use OPNsense as my firewall, so here’s how I added the next-server and filenames to OPNsense’s DHCP server.

Testing

2 gamers, 1 cpu, NO Virtualization!

Thu, 21 Apr 2022 12:36:00 +0000

I’ve been using Linux for a long time. The first distribution I ever installed was Ubuntu 6.06 ‘Dapper Drake’, on which I ran a phpBB server for my school friends. Security was poor. But shortly after, as I learned about Linux desktops (and first daily-drove a Linux laptop around 2012), I learned about ‘Multiseat’. The Wikipedia article has a cool picture of a very 90s looking desktop tower with 4x CRT monitors, and that image was published to Wikipedia in 2005, and I saw it in middle/high school years. This multi-seat configuration stuck with me, and for years I’ve wanted to set it up myself. Finally, in 2022, I’m ready to set this up, and I’m going all in - ONE computer, TWO users, and support for both OpenGL and Vulkan gaming!

This project is part of the Multiuser, Multidesktop, and Multiseat Megaproject.

What Is Multiseat?

Multiseat is a system that allows Linux systems to operate with multiple users at the same time. Linux has been inherently a multi-user system essentially for its entire life, but normally there is one graphical session plus more than one terminal session (i.e. by SSH). It’s also possible to create multiple graphical sessions using a remote access protocol like I did in my XRDP Video. But, with multiseat, we take this to the extreme - we hook up multiple monitiors, keyboards, mide, peripherals to one system and partition them up so each user can use their own hardware, sharing the computational resources of one computer amongst all of the users who are physically seated within cable length of the computer.

Currently, to get full GPU acceleration (OpenGL + Vulkan), we need to dedicate one GPU to each user. This is due to security concerns in the kernel devs about the potential for poor information security between multiple users, as graphical applications are given direct access to GPU VRAM to improve performance. If you don’t care about hardware acceleration, stay tuned for a future video where I omit the gaming and 3D requirement and build a many-user productivity box for poor cube farm workers to use basic desktop things.

This is a super long blog, so feel free to jump down to any specific segments

Video

Click the thumbnail for to view the video on Youtube

Hardware and BIOS Configuration

The hardware I’m using for this test is:

AMD Ryzen 5 3400G APU - Zen+ architecture, Raven Ridge + Vega graphics (shows up as ‘Raven2’ in radeontop)
AMD Radeon RX580 GPU - Polaris architecture (shows up as ‘Polaris10’ in radeontop)
ASRock B450M Pro4 AM4 motherboard (it was low cost when I bought it)
32G DDR4 (2x16G sticks)

I specifically chose this test hardware for this project since both the APU and GPU use the same driver - the open source amdgpu driver. In theory it should be possible to mix AMD and Intel graphics (since both use Mesa), but as far as I can tell it is NOT possible to mix Nvidia and other GPUs using the proprietary drivers.

I played a bit in the BIOS to make sure that both GPUs show up correctly. I found that I need to set the primary GPU in the BIOS to the internal GPU, otherwise it disables the Vega GPU entirely. This could be a quirk of the BIOS on this board. I cannot update past the current BIOS version as later versions dropped support for Summit, Raven, and Pinnacle Ridge CPUs (og Zen and Zen+). Knowing which GPU is ‘primary’ (which should be the one that GRUB and the BIOS show up on) is going to be very important later.

Software Setup

I chose to install Ubuntu 22.04 ‘Jammy Jellyfish’, which is currently in beta as of the writing and filming of this video but should be released completely by the time this blog and video drop. I know that Linux distros are extremely controversial topic, but Ubuntu has excellent hardware support, 22.04 uses the Linux 5.15 kernel which is more recent than Debian ships, and the environment should be fairly well tested as it’s the basis of other very popular desktops such as Pop!OS. I’m intentionally using AMD GPUs to avoid fighting with the Nvidia proprietary drivers, and I’ve found the open source amdgpu driver to be quite excellent.

During the install, I selected ZFS on root (although you can choose whatever you want of coruse), and to install third-party codecs.

After installing, I did some basic housekeeping (sudo apt update && sudo apt upgrade -y), and installed OpenSSH server so I can access the system if I royally mess up later. In case you need to hit the display manager over the head later, you can run sudo systemctl restart display-manager and it should restart gdm3, logging out everyone and starting back from the login screens.

Switch from Wayland to X11

I found that, while it’s possible to configure multiseat on Wayland, Vulkan did not work properly with multiple GPUs. I chose to revert back to X11 to deal with this. It’s a fairly simple configuration change. For Ubuntu 22.04 with GNOME, edit /etc/gdm3/custom.conf and find the line #WaylandEnable=false and uncomment it. Once you reboot, it should be using X11 entirely.

Initial Benchmarking

Before I split into multiple seats, I’m going to run with just the iGPU, followed with just the dGPU, and benchmark the system. I’m using Unigine Heaven Benchmark for OpenGL, and Portal 2 for Vulkan. Both are fairly dated titles at this point, but I’m honestly not a huge gamer and this game has good Vulkan support on Linux. I just want to make sure performance later is within the same range as it was before I added two seats. I do this by enabling just the iGPU, then just the dGPU, then switching back to both of them. You don’t need to do this, it’s just nice to confirm that we aren’t really losing GPU performance at all.

Configure Seats

Here, we do all of the hard work with configuring the seats with loginctl. Systemd supports the concept of multi-seat natively, and the most popular desktop environments should properly listen to the seat tags and assign devices correctly.

First, we can list which seats currently exist:

sudo loginctl list-seats

It should tell you seat0 and nothing else. We are going to name our new seat seat1.

If you’re curious about the hardware currently on your system, use seat-status:

sudo loginctl seat-status seat0

I’ve found it’s handy to save this to a file for later reference:

sudo loginctl seat-status seat0 > seat_hardware.txt

By default, all hardware is attached to seat0. By using loginctl, we can create rules that force specific hardware to be assigned to a different seat. Hardware can be dynamically reallocated using loginctl, and it will unassign hardware from one seat and move it to the other. A seat must have a “Master of Seat” (shown in seat-status as [MASTER]), which is a graphics card. Once a master of seat exists, a login prompt will be spawned for it, and once you add a keyboard/mouse, you can login using it.

In order to attach hardware to a seat, we need the entire sys path (as shown in seat-status) and to call attach:

sudo loginctl attach seat1 /sys/<path to device>

Once we’ve saved the current seat status as a text file, we can comb through it and decide what we want to attach to the second seat. Then, we can copy paths out of the file so we don’t have to type them.

Graphics, Sound

In my case, I have 2 graphics and 3 sound devices. Each GPU has sound, plus the on-board chipset audio. I’m using the HDMI audio output for both seats, so each graphics and its associated sound device is passed through. I left the chipset audio assigned to the initial seat, so only that one will have analog microphone. If you want the second seat to have a microphone input, you’ll need a USB audio device for it.

Figure out the PCIe addressees of both of your graphics cards using lspci. Look for VGA Compatible devices, and their associated sound devices.

In my case, the Raven GPU is 07:00.0 and the Polaris GPU is 01:00.00 (with the sound device at *:00.01 for both)

USB Ports

For USB ports, you need to forward them by USB bus, which is a combination of the PCIe root device and the port off the root complex. For most systems the USB root complex is part of the CPU itself or the chipset, possibly with USB hubs on the motherboard to split USB3 root ports into multiple USB2 ports. You will need to go through using lsusb with a known USB device to see which ports on your motherboard are associated with which USB bus, and move devices around until the devices for multiple seats aren’t sharing the same bus. I recommend getting a USB3 hub for each seat to make it simpler to deal with this, since you can find the USB bus of a specific USB3 port and easily pass through all devices downstream.

Once you’ve done that, go back to your seat hardware file and find the entry for the USB bus you want, then loginctl attach it to the new seat. Changes should take effect immediately.

Removing Multiseat

If you want to completely revert, run sudo loginctl flush-devices to flush the seat config, which should take you back to the seat0 configuration you started with at the beginning.

The settings should be retained through reboots, so you shouldn’t need to call loginctl again after everything is setup unless you want to modify hardware assignments.

Small Proxmox Cluster Tips and Tricks, and QDevices

Thu, 14 Apr 2022 09:15:05 -0300

So you want to turn your one or two Proxmox nodes into a small Proxmox cluster? What do you need to know about quorum, especially as it pertains to really small clusters of 2 and 3 nodes? In this video, I go over the different ways to stabilize a 2-node cluster for high availability, the storage requirements for high available VMs, installing a QDevice using a Raspberry Pi, and if you even need one. I also talk about how a QDevice might not help you as much as you think in a 3 node cluster.

This is part of the Hyper-Converged Cluster Megaproject.

Video

Most of the content is in video form. If you’d like to reference any commands to install the QDevice, scroll down below.

Installing a QDevice on a Rasperry Pi

Here are the instructions for installing a QDevice, in case you would like to copy commands.

First, for Proxmox to be able to configure the Pi, it will log in over SSH with the root password. So, we need to set a root password, and temporarily enable SSH login as root with a password.

To set the password, use sudo passwd root. Then, edit /etc/ssh/sshd_config until you find the key #PermitRootLogin and change that line to PermitRootLogin yes. Restart the ssh server using sudo systemctl restart sshd.

Next, we need to install the QDevice daemon. Make sure the Pi is up to date with sudo apt update && sudo apt upgrade -y, then install the daemon itself:

sudo apt install corosync-qnetd corosync-qdevice

At this point, the Pi is ready to go, now move over to Proxmox.

We now need to install QDevice support onto all nodes of the cluster. From the Shell tab for each node, first run package updates (apt update, you may need to setup the pve-no-subscription repo if you haven’t already) then install qdevice with apt install corosync-qdevice. Repeat for each node in your cluster.

Finally, we can add the QDevice to the cluster. From any node in the cluster, run the pvecm command to add your node. It needs the IP address of the node here. -f forces it to overwrite any existing configuration.

pvecm qdevice setup <ip> -f

It will then ask you to accept the SSH fingerprint and for the root password of the raspberry pi. Then, it should do its magic, and the QDevice will be setup.

The QDevice won’t show up anywhere in the GUI. To check that it’s working, use pvecm status and it will display the QDevice under Membership Information, and under Votequorum Information it will show the total number of votes and the number required for quorum. The QDevice flag should also be listed under Flags.

Installing Proxmox VE 7 on Debian Bullseye

Mon, 11 Apr 2022 00:31:05 -0300

I recently tried to install Proxmox on my Dell Wyse 3040 thin client so it can act as a cluster member, since it’s arguably easier to setup a full Proxmox install on x64 based devices than it is to setup a QDevice. Proxmox has a great installer, you have the full GUI to manage the node (not that you need to do much with it as a QDevice), and you have the option of setting it up for other Proxmox services and containers even if it’s really not very capable. So, I’m committed to installing Proxmox on my 3040. But then the problems came. The Proxmox installer was unhappy with the eMMC storage, and wasn’t able to create a partition table on it. Well, I guess I’m diving down the more difficult path of installing Debian, and then installing Proxmox on top of it.

In addition to my use case, it’s not uncommon to install Proxmox on top of Debian if you want a specific root filesystem layout, since Proxmox will not install without LVM (or ZFS). In my case, I’m going to install ext4 on root with no LVM, since the disk is so small. Proxmox has a guide on instlaling Proxmox 6 on top of Debian Buster, but there are a few minor changes to be aware of with the more recent version. Not a big deal though, the guide is still mostly accurate and you should be able to follow it.

Video

Video form of this, if you prefer to follow along there:

Install Debian

First we need to install Debian. I started with the Debian 11 (Bullseye) netinst image, ran through it, deseleted the desktop, and selected SSH server. From there, it’s pretty much a standard Debian install. Of course, if you’re going through the trouble of installing Proxmox on Debian instead of ’natively’, you probably want to use a custom partition layout on the boot partition, and here is where you’d configure that.

Networking

Before installing Proxmox we need to make sure we have a static IP address, and the hostname resolves back to our public static IP address by setting it in the hosts file.

First, let’s set that static address by editing /etc/network/interfaces: Find the line for your adapter (mine is enp1s0) and modify it from dhcp to static and add the static IP. Here’s what mine looks like:

iface enp1s0 inet static
address 172.27.1.154
netmask 255.255.240.0
gateway 172.27.0.1

After that, we need to add this static address to the hosts file by editing /etc/hosts: By default, Debian will map 127.0.1.1 to your hostname and 127.0.0.1 to localhost. We need to change the second entry to point to our network-accessible IP instead. This is what mine looks like:

127.0.0.1   localhost.localdomain localhost
172.27.1.154 pve3040.palnet.net pve3040

# The following lines are desirable for IPv6- capable hosts
::1     localhost   ip6-localhost   ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

You should only need to change the first two lines here.

Once you’re done, do ifdown enp1s0 followed by ifup enp1s0 (obviously put in your own interface name) so the new IP address takes effect. Be careful if you’re doing this over SSH! You can use ip a to make sure the new IP is being used.

Add Proxmox Repository

Create a new file (nano /etc/apt/sources.list.d/pve-install-repo.list) and paste in the following contents for PVE 7 on Bullseye:

deb [arch=amd64] http://download.proxmox.com/debian/pve bullseye pve-no-subscription

Now download the Proxmox GPG key that the repository is signed with. The file naming convention has changed, so you can’t just go from proxmox-ve-release-6.x.gpg to proxmox-ve-release-7.x.gpg like the Proxmox guide would let you believe. I believe this is because the same key now signs the PBS releases as well as PVE.

wget http://download.proxmox.com/debian/proxmox-release-bullseye.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bullseye.gpg

APT Update, Upgrade, Install Proxmox

Now do the usual apt update and apt full-upgrade to prepare. This should also download package lists from PVE’s repo.

Now we can finally install Proxmox:

apt install proxmox-ve postfix open-iscsi

Postfix will ask you some questions, choose Local Only unless you know you have a better answer.

Proxmox also suggests we remove os-prober:

sudo apt remove os-prober

Now reboot, and DONE!

reboot now

PVE-VDIClient - A Python Graphical VDI Client for Proxmox

Thu, 07 Apr 2022 08:27:05 -0300

For my last trick, I setup a multi-user thin client where each user account was connected to a specific virtual machine. This is great, but if you have a lot of thin clients you might not want to create a ton of VMs and might instead want each user in the system to have one or more VMs. Well, Josh Patten has written a Python-based GUI to select thin clients which you have access to, and today we are going to turn that into an appliance on both a Raspberry Pi and an actual x86 based Thin Client (running Debian).

This article is part of the Thin Client Series

This article was written with Raspberry Pi OS (32-bit) Bullseye release 2022-01-28 and Debian Bullseye release 11.2.0.

Sections

This page is really long, so here are pointers to individual sections

Video

Of course, I have a video to go along with this topic. Click the thumbnail to watch it on Youtube.

Setup for Raspberry Pi

I am starting with Raspberry Pi OS Bullseye (32-bit). I struggled with this for a bit before I realized that the Pi OS packages for Qt are just a mess, so we have to install some things through apt and some through pip to get what we want. Follow along below if you’d like, or copy and paste the whole thing.

#First up, as always, make sure the system is up to date before we even start:
sudo apt update && sudo apt upgrade -y

#Now, the dependencies we need via apt
#Pyside2 doesn't have pip packages for armv7, but apt does
#Pip won't find it though
sudo apt install python3-pip python3-pyside2* virt-viewer git -y

#And finally, the dependencies via pip
#This will install PySimpleGUIQt version 0.30.0, the last one wihch
#didn't require PySide2, since it won't find a PySide2 package for armv7
#But it'll also install all of the other deps that PySimpleGUIQt has
pip3 install proxmoxer PySimpleGUIQt

#Force Pip to upgrade to the latest verion even though it can't find PySide2 dep
#PySide2 was installed via apt, so it will find it at run time
pip3 install --upgrade PySimpleGuiQt==0.35.0 --no-deps

#Now we can clone Josh's repo and cd into it
git clone https://github.com/joshpatten/PVE-VDIClient.git
cd ./PVE-VDIClient

#Make it executable
chmod +x vdiclient.py

Now continue along to Client Setup!

Setup for Debian

I am starting with Debian Bullseye on an x64 system. I installed Debian with desktop support using the LXDE desktop environment, so it’s as similar as possible to the Pi environment. During install it will ask you if you want a desktop, so deselect GNOME and select LXDE. I also selected SSH server here so I don’t have to install it myself later.

Everything installs and runs just fine on Debian, no quirks at all.

#First up, as always, make sure the system is up to date before we even start:
sudo apt update && sudo apt upgrade -y

#Now, the dependencies we need via apt:
sudo apt install python3-pip virt-viewer git -y

#And finally, the dependencies via pip
#On Debian pip will install PySide2 just fine if you're on x64
pip3 install proxmoxer PySimpleGUIQt

#Now we can clone Josh's repo and cd into it
git clone https://github.com/joshpatten/PVE-VDIClient.git
cd ./PVE-VDIClient

#Make it executable
chmod +x vdiclient.py

Client Setup

Now we need a configuration file, which I’ll put in /etc/vdiclient to simplify things:

#Create the directory
sudo mkdir -p /etc/vdiclient

#Create the config file
sudo nano /etc/vdiclient/vdiclient.ini

And the contents:

[General]
title = apalrd VDI
icon=vdiicon.ico
logo=vdilogo.png
kiosk=false

[Authentication]
auth_backend=pve
auth_totp=false
tls_verify=false

[Hosts]
<Proxmox IP or DNS name> =8006

There’s a lot of fun you can have with configuration here, including picking a theme from PySimpleGUIQt’s theme set, choosing your own logo, and your own icon. Make sure you either copy the images to the path you call the script from (the user’s home directory later down in my example), or put them in /etc/vdiclient and use absolute paths in the ini file. Otherwise, the client won’t find the images.

Optionally if you want to share this between users, you can copy the files to system wide directories, although I did not do that for this appliance setup where there are no local other local users.

#Copy vdiclient.py to the local bin folder
sudo cp vdiclient.py /usr/local/bin/vdiclient
#(Optional) Copy images too, unless you're using your own
sudo cp vdiclient.png /etc/vdiclient/
sudo cp vdiicon.ico /etc/vdiclient/

(Optional) Configuring VDI Client on Boot

Here we are going to modify our user so they launch into the VDI client on boot.

Basically, we are doing similar steps to the graphical thin client chooser except we are logging in to the user by default, so LXDE remains installed and we can get it back if we ever need to.

Before you do this on the Pi, setup your default audio device using the audio icon in the GUI. This should change the audio preferences for your.config which should make audio work correctly. On Debian it should pick the correct audio device by default.

First we are going to remove the system autostart file and replace it with a blank one, so LXDE doesn’t autostart itself:

For Debian:

#Go to directory
cd /etc/xdg/lxsession/LXDE
#Backup original
sudo mv autostart autostart.bak
#Create empty autostart file
sudo touch autostart

For Pi OS:

#Go to directory
cd /etc/xdg/lxsession/LXDE-pi
#Backup original
sudo mv autostart autostart.bak
#Create empty autostart file
sudo touch autostart

Next, we are going to create an autostart file in our user’s home directory, backing up and replacing any existing one For Debian:

#Make sure directory exists if it doesn't already
mkdir -p ~/.config/lxsession/LXDE
#Change to it
cd ~/.config/lxsession/LXDE
#Backup file if it exists (this may error if it doesn't exist, that's fine)
mv autostart autostart.bak
#Create new file
nano autostart

For Pi OS:

#Make sure directory exists if it doesn't already
mkdir -p ~/.config/lxsession/LXDE-pi
#Change to it
cd ~/.config/lxsession/LXDE-pi
#Backup file if it exists (this may error if it doesn't exist, that's fine)
mv autostart autostart.bak
#Create new file
nano autostart

The contents of that file:

@/usr/bin/bash /home/<user>/thinclient

And now we need to actually write that file (nano ~/thinclient):

#cd into the PVE-VDIClient directory so local paths to images work
cd ~/PVE-VDIClient
#Run loop for thin client so user can't close it
while true
do
    /usr/bin/python3 ~/PVE-VDIClient/vdiclient.py
done

At this point it automatically runs the VDI client when the user logs in, but we need it to log in the user by default.

For Raspberry Pi OS, run sudo raspi-config and select boot mode to Desktop Autologin, and that’s it. raspi-config handles the rest for you.

For Debian, we need to edit /etc/lightdm/lightdm.conf to enable autologin for our user. Find the line which has #autologin-user= and uncomment it, adding your own username (i.e. autologin-user=vdiuser).

Getting your desktop back

If you make a terrible mistake and want your desktop back to debug it, just move the autostart files back and reboot:

For Debian:

cd ~/.config/lxsession/LXDE
mv autostart autostart.new
mv autostart.bak autostart
cd /etc/xdg/lxsession/LXDE
mv autostart autostart.new
mv autostart.bak autostart

For Pi OS:

cd ~/.config/lxsession/LXDE-pi
mv autostart autostart.new
mv autostart.bak autostart
cd /etc/xdg/lxsession/LXDE-pi
mv autostart autostart.new
mv autostart.bak autostart

And of course you can do the inverse (autostart -> .bak and .new -> autostart) once you’re done using the desktop

Conclusions

I’m happy with the progress this project is making. We’re getting closer and closer to the goal of being able to dynamically spin up new VM clones as users request them, like a commercial VDI solution. This new python tool by Josh is a great step in that direction, and hopefully you are inspired to help him out by supporting it. Certainly more episodes on this subject from me are coming in the future!

Buying More Thin Clients for more 'fun'?

Mon, 04 Apr 2022 00:32:05 -0300

After really enjoying my trio of Dell Wyse 5060 Thin Clients, I bought another cheap one to see how it compares, and hopefully to give advice to the many commenters. The 5060 was a great value for $35 but it’s hard to find at that price normally, while the 3040 is always available for that price, physically much smaller, and also worse on paper. It has a quad-core Intel CPU, 2G of RAM, and was advertised as having an 8G SSD but mine is actually 16G. It’s eMMC instead of SATA, soldered on board, and the RAM is soldered as well (the case is tiny!). I opened it up and replaced the power connector with a short USB cord instead of trying to buy a power cord.

tl;dr I was able to run Home Assistant on this easily, and I’d recommend it if you can’t get a Pi 4 (which is basically a given considering the lead times in 2022 are a year or more). This model is a step down from the 5060, but it’s still a good value for the price of the lowest Raspberry Pi models.

Teardown Video and USB Power Mod

I made a video showing the hardware inside my thin client, if you’re interested in seeing the teardown, USB power mod, or watching the BIOS configuration. I also prove that Home Assistant does install easily. xClick the thumbnail below to watch it.

I also did a USB power mod to this, since it didn’t come with a power supply and needs 5V/3A. I initially got a 3’ USB cord and it functioned but it would occasionally reboot, especially under high CPU+GPU load. I cut the cord down to 18" to reduce the voltage drop along the cord and haven’t had any boot issues since then. It’s a cheap way to get this thing up and running, since USB bricks are really cheap.

How to Install your own Software

The 3040 has a pretty terrible BIOS/UEFI. The revision I have does not support legacy boot at all, so you can only boot OSes that support UEFI. It also doesn’t support configuration of the boot file path, so the boot file must be in the fallback path. This usually means you need to configure GRUB for ‘removable media’ boot. Xubuntu and Home Assistant both did this by default, but Debian didn’t, so I had to boot back into rescue mode and re-install GRUB at the removable media path for it to boot correctly.

Hardware Info for the curious

All of this was taken via an Xubuntu 20.04 Live CD, so your kernel may be configured slightly differently

lscpu

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   36 bits physical, 48 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           76
Model name:                      Intel(R) Atom(TM) x5-Z8350  CPU @ 1.44GHz
Stepping:                        4
CPU MHz:                         534.804
CPU max MHz:                     1920.0000
CPU min MHz:                     480.0000
BogoMIPS:                        2880.00
Virtualization:                  VT-x
L1d cache:                       96 KiB
L1i cache:                       128 KiB
L2 cache:                        2 MiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user
                                  pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB condit
                                 ional, IBRS_FW, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtr
                                 r pge mca cmov pat pse36 clflush dts acpi mmx f
                                 xsr sse sse2 ss ht tm pbe syscall nx rdtscp lm 
                                 constant_tsc arch_perfmon pebs bts rep_good nop
                                 l xtopology tsc_reliable nonstop_tsc cpuid aper
                                 fmperf tsc_known_freq pni pclmulqdq dtes64 moni
                                 tor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse
                                 4_1 sse4_2 movbe popcnt tsc_deadline_timer aes 
                                 rdrand lahf_lm 3dnowprefetch epb pti ibrs ibpb 
                                 stibp tpr_shadow vnmi flexpriority ept vpid tsc
                                 _adjust smep erms dtherm ida arat md_clear

Of note, it does support VT-x if you want to do virtualization, and AES-NI if you want to do anything involving crypto offload (VPN, HTTPS, …)

lspci

00:00.0 Host bridge: Intel Corporation Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series SoC Transaction Register (rev 36)
00:02.0 VGA compatible controller: Intel Corporation Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Integrated Graphics Controller (rev 36)
00:0b.0 Signal processing controller: Intel Corporation Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series Power Management Controller (rev 36)
00:11.0 SD Host controller: Intel Corporation Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series SDIO Controller (rev 36)
00:14.0 USB controller: Intel Corporation Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series USB xHCI Controller (rev 36)
00:1a.0 Encryption controller: Intel Corporation Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series Trusted Execution Engine (rev 36)
00:1c.0 PCI bridge: Intel Corporation Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series PCI Express Port #1 (rev 36)
00:1f.0 ISA bridge: Intel Corporation Atom/Celeron/Pentium Processor x5-E8000/J3xxx/N3xxx Series PCU (rev 36)
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

As you can see, it uses a Realtek Ethernet Controller

vainfo

libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva error: /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so init failed
libva info: va_openDriver() returns 1
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/i965_drv_video.so
libva info: Found init function __vaDriverInit_1_6
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Intel i965 driver for Intel(R) CherryView - 2.4.0
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Simple            :	VAEntrypointEncSlice
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileH264MultiviewHigh      :	VAEntrypointVLD
      VAProfileH264MultiviewHigh      :	VAEntrypointEncSlice
      VAProfileH264StereoHigh         :	VAEntrypointVLD
      VAProfileH264StereoHigh         :	VAEntrypointEncSlice
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileNone                   :	VAEntrypointVideoProc
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileJPEGBaseline           :	VAEntrypointEncPicture
      VAProfileVP8Version0_3          :	VAEntrypointVLD
      VAProfileVP8Version0_3          :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointVLD

Hardware encode/decode is certainly usable for H264, although it’s not nearly as capable as the 5060 I reviewed previously

A Modern Linux Graphical Terminal Server

Thu, 31 Mar 2022 08:00:05 -0300

I’ve made many videos on Thin Clients before, all of them relying on Proxmox and the SPICE protocol. This works well when you control both the client and the hypervisor, and allows a lot of flexibility in the guest OS at the expense of flexibility at the client. If you want to rely on a remote access / Bring-Your-Own-Device type solution, you probably care more about solid multi-platform client support than flexibility in mixing VM OSes and running with no software installation on the VM. To this end, I’ve setup a modern Linux terminal server, which can be used to allow many clients to simultaneously connect to their own Linux desktops remotely, from nearly any device OS in common use today.

This project is part of the Multiuser, Multidesktop, and Multiseat Megaproject.

This is a very long article, filled with many commands to copy and paste along with the reasoning behind them. Feel free to jump around to the sections relevant to you using the links below.

Video

This project has a video! Click on the thumbnail to watch it.

Choosing a Distribution

I’ve chosen to base this install off xubuntu 21.10 Impish Indri. I chose xubuntu becasue it’s lighter weight than the GNOME or KDE desktops, and I’m hoping the reduced fluff will improve memory usage on systems with a large number of clients. I also enjoy XFCE as a generally simple and easy to use desktop environment. As to the release, I chose 21.10 as it’s the most recent release as of this writing, although support ends in just a few months as the 22.04 LTS drops very soon.

Even though I’m setting up a terminal server, I installed the Desktop version, since I’m going to need the full desktop environment and all of the user applications anyway. This also means the server’s physical console is now graphical, which isn’t really a huge deal, but you cannot log in to the physical console and virtual session at the same time, and the remote user cannot kick the physical console user.

Instead of using a hypervisor as I usually do, I’ve installed this on a bare metal server, since I’m trying to load it up heavily with clients. My test platform is an AMD Ryzen 3400G (4c/8t) APU with 32G of RAM and a 500G SATA SSD.

After installation I ran the usual software updates (sudo apt update && sudo apt upgrade) and installed OpenSSH server so I can comfortably access it remotely (sudo apt install openssh-server).

Install XRDP

First we need to install xrdp itself. Ubuntu has a package for it, so it’s pretty simple: sudo apt install xrdp At this point, you should be able to connect to the system using an existing user and get the XFCE desktop we know and love. However, you’ll notice a decently big problem - there is no sound.

XRDP relies on an internal PulseAudio API which requires it to be compiled for the specific version of PulseAudio on the system, so we have to build the PulseAudio plugin ourselves. XRDP has a great guide on this here.

So, here’s all of the commands from their guide compressed into one. It’ll ask for your sudo password and for you to confirm a few things.

#Install packages and make sure git is installed
sudo apt install build-essential dpkg-dev libpulse-dev git autoconf libtool -y
#Change to home directory for git checkout
cd ~
#Clone repository
git clone https://github.com/neutrinolabs/pulseaudio-module-xrdp.git
#cd into repository
cd pulseaudio-module-xrdp
#Run the script they provide to get started
scripts/install_pulseaudio_sources_apt_wrapper.sh
#Configure and make
./bootstrap && ./configure PULSE_DIR=~/pulseaudio.src
make
#Install
sudo make install

RDP Security Tips

RDP with modern security settings to prevent protocol downgrade attacks, is generally considered to be a secure protocol on its own. User traffic is encrypted via TLS and user sessions are secure from eavesdropping and information leakage. However, the RDP server is not so lucky. Since RDP runs on a well-known port number, it’s not uncommon for bots to scan the entire internet for open RDP servers and try to connect with common Windows usernames and passwords. So not only is the protocol only as secure as the user’s password, but in many cases the RDP server will spawn a new remote session and then allow graphical login via the remote session, so bots will increase your CPU load by spawning a bunch of login screens.

For some sense of scale, shodan.io currently lists 3.8 million public facing RDP servers, and lists the operating system, FQDL, and certificate info for many of them. You WILL be on this list if you open your server up to the internet.

Given all of this, I’d recommend against opening your RDP server directly to the internet. The xrdp developers have recently merged changes to log data specifically so fail2ban can block repeat abusers (as is common with SSH), but that release isn’t yet in Ubuntu and isn’t scheduled to be in the 22.04 LTS either. And even with fail2ban, you’re still letting an attacker connect to the server, start a graphical session, and then be disconnected for authentication failure, something which uses a lot more resources with xrdp than it does with SSH.

Using a proper remote-access VPN solution to your home/business network or a cloud relay point is good security practice anyway, and I’d feel comfortable leaving RDP exposed to users within my private home/business network without further protection.

Even with protocol level protections, each users on the system is still a user on the same system, so it’s possible for them to interact with the system and other users in a potentially undesirable way. Here are a few options you have to restrict user access a bit, even though they aren’t a comprehensive security guide.

Restrict Users to Remote Access Group

By default, sesman (xrdp session manager) will restrict access to only users in the group tsusers, but only if the group exists. Since the package doesn’t create it, it won’t exist, and any account can login. Additionally, /etc/xrdp/sesman.ini has an option to disable root login, separately from the tsusers group.

It’s up to you to decide if you want to limit access or allow any user on the system to connect. Depending on how exposed this server is, allowing users with sudo permissions may be more dangerous than you’d like. This all ties into the security concerns you have for the system. You should absolutely edit sesman.ini to disable root access though.

So, if you want to restrict user access, let’s create this group and add our VDI users to it:

sudo groupadd -r tsusers

To add a user, it’s pretty simple also:

sudo gpasswd -a <username> tsusers

Limiting Resources Per User

This section is optional. If you want to prevent a single user from hogging all of the system resources, then you can use Linux control groups to achieve this. Depending on how you set your limits, this won’t stop multiple users from really bogging down the system, so be aware of how many resources you actually want to give each user, but at the same time give them enough to do what they need successfully. If you occasionally have 10 users but normally have 1-2, setting the limits to 1/3 of the system capacity would let the normal users perform more processor intensive work (such as compiling) without being artificially limited to a fraction of an otherwise unused CPU.

Systemd has a reference on resource control for user slices here.

Systemd has a folder structure where we can limit resources per user - they need to go in /etc/systemd/system/user-1000.slice.d for user id 1000. However, we can create defaults by creating a .conf file in the folder folder /etc/systemd/system/user-.slice.d/*.conf. Systemd recognizes the user- as a default path if there is no user-specific file or folder.

So, we will create this default user slice configuration folder and a configuration file inside of it:

sudo mkdir -p /etc/systemd/system/user-.slice.d
sudo nano /etc/systemd/system/user-.slice.d/50-vdiusers.conf

And the contents - CPUQuota is a % relative to a single thread, so if you have a 4c/8t CPU, the maximum would be 800%. In this case, I’ve limited it to a two threads. MemoryMax can be specified in M or G, and the OOM Killer will come and reap processes from the user when they exceed their limits as if the system only had that much memory and could not swap. There’s also a MemoryHigh option you can use which will start slowing down processes when the memory reaches above the threshold, if you’d like to set that to something less than MemoryMax.

[Slice]
CPUAccounting=on
CPUQuota=200%
MemoryAccounting=on
MemoryMax=2G

If you want to read the full systemd documentation on resource control, you can read it here. There are a lot of options there if you really want fine-grained control of resource usage by your users.

Since XRDP runs as its own user as a system service, it’s not part of the user pool, it’s part of the service pool. This means that xdrp’s actions of performing graphical compression are not counted to the user’s quotas, but they can add up, especially if the user is doing a lot of work with motion video or graphics.

Color Managed Device Error

This one is intermittent. On Ubuntu 21.10, we can fix this by adding a PolKit configuration file to allow users to change their own color profiles. Note that the version of polkit matters here, version 0.105 is shipped with Ubuntu 21.10 and requires pkla files.

sudo nano /etc/polkit-1/localauthority/50-local.d/45-allow-colord.pkla

[Allow Colord all Users]
Identity=unix-user:*
Action=org.freedesktop.color-manager.create-device;org.freedesktop.color-manager.create-profile;org.freedesktop.color-manager.delete-device;org.freedesktop.color-manager.delete-profile;org.freedesktop.color-manager.modify-device;org.freedesktop.color-manager.modify-profile
ResultAny=no
ResultInactive=no
ResultActive=yes

For various reasons, Ubuntu has stuck with backporting security fixes to an ancient version of polkit instead of upgrading to a more modern release that uses the javascript configuration method, as they do not want to depend on mozjs (Spidermonkey) for security critical functions. Since polkit has just switched to a proper tiny embedded javascript engine, it’s very likely that all of the rules syntax above is going to become obsolete within a few Ubuntu releases.s

Conclusions

I chose RDP over a VNC-based solution as the protocol is extremely well standardized and has very wide client support, including clients available for the usual Windows/macOS/Linux, but also iOS, iPadOS, Android, Android TV, and even Samsung’s Tizen OS for smart TVs. It’s also extremely easy to get a basic setup working. Performance in terms of number of users on a single system is good, since we aren’t relying on virtualization at all, and all users are able to efficiently share system resources.

This setup is good if you want to:

Centralize computing / storage in a bring-your-own device fashion
Get full desktop functionality out of an otherwise limited operating system (i.e. Android, iOS, Smart TVs)
Connect back to your Linux desktop while away from home, which also keeps sensitive data off your mobile devices while traveling in case they are lost / stolen / forced to be unlocked by customs and border patrol
Less system overhead and resource utilization than VMs
Hardware GPU acceleration is available to all users from a single GPU for transcoding (but not OpenGL AFAIK)

It’s not a great solution if you want:

Windows (Microsoft offers this for $$$)
Complete multi-user and entire filesystem isolation for each session
Ephemeral clones of the entire system (VM), cleared after each user

Multiuser, Multidesktop, and Multiseat Megaproject

Thu, 31 Mar 2022 11:00:00 +0000

In this project, I explore thin client software, and set up a Raspberry Pi to act as a thin client. This is a multi-part project, follow along below for each sub-project.

A Modern Linux Graphical Terminal Server

For the first project, I setup a remote desktop terminal server, allowing many users to connect to a single server and operate indepednent graphical sessions. My test bench was able to handle a dozen simulatneous users, and a proper server should be able to handle many more. Inspired by an email sent to me, my goal of this project is to create a low cost solution for schools in developing regions to be able to teach programming and other things that require a computer desktop, while using the hardware available (tablets, TVs, …) as the user interface to the desktop. But it’s widely applicable to any organization using Linux, and can be expanded into a cluster and shared storage system to allow all users to do their work with centralized compute, keeping data securely at the server.

2 gamers, 1 cpu, NO Virtualization!

The second project in the series was a ton of fun to research and produce. I was able to use Linux Multiseat to operate a single Ryzen system with two graphics cards (one integrated, one dedicated) to allow two gamers to play native Linux games with full OpenGL and Vulkan support on the same computer. Unlike other people before me, I did not rely on virtualization or hardware passthrough, but this is a purely multi-user Linux solution running on a single kernel.

Low-Cost Garage Door Automation with Home Assistant and Sonoff

Thu, 24 Mar 2022 07:43:00 +0000

Of all of the doors in a normal US McMansion, the garage doors are the biggest, and are almost always motorized. This means they are an easy target for automation, since most of the hardware is already there, we just need to bridge it to the virtual space. The cheapest way to do this is to use a door contact switch and dry contact relay which are compatible with Home Assistant, and some YAML magic to bridge them together.

This is a pretty long one, so here’s some links to different sections:

Video

This article has a corresponding video! Watch it here

Sensing

You can use regular magnetic door contacts with garage doors, but they also make more robust garage door contacts if you’re concerned with durability. I ended up using an Aqara Zigbee sensor, and I bought a pack of them for some other doors in the house. The are pretty functional for the cost. I attached them near the top corner of the garage door, and they can sense when the door is all the way closed. Make sure the magnet and sensor are close enough to each other, one of my 3 doors was not close enough and only intermittently detected the door as closed.

Unboxing the sensor - packaging is kinda wasteful, but the sensor isn't too ugly

Door sensor installed next to top of the door

Control

It’s critical for the Control side that you get a smart relay with a dry contact. A lot of smart relays are mains or DC powered, and output mains or DC from the relay. This won’t work for a garage door opener, where you need to bridge two contacts and not connect them to anything else. The Sonoff Mini will NOT work, but the Shelly 1 will, and also has a switch input if you want a wired door switch, so it’s a great option for a single door.

In my case, I have 3 doors to control. So, the Sonoff 4CH PRO R3 works for me for less cost total than the Shelly’s - it has 4 separate relays, each of which is electrically isolated from each other (only isolated on the PRO!). I flashed it with Tasmota (see my Tasmota Day for how), which gives me full control of the device with purely local integration over MQTT.

Once I had Tasmota flashed and the Sonoff connected to my network, it’s time to configure it. Start by finding the IP address (I look on my router’s DHCP leases for new Tasmota devices but Tasmotizer has a ‘Get IP’ button that works too), and connect to it over HTTP. The 4CH has four bit ‘ON’/‘OFF’ labels as well as 4 Toggle buttons, as well as the usual Tasmota configuration buttons. Click ‘Configuration’, then ‘Configure MQTT’, and enter the broker information. I don’t like how Tasmota pollutes the root namespace in MQTT with tele, cmnd, etc. so I change the default topic structure to be ‘raw/%topic%/%prefix%/, which ends up with a topic like raw/tasmota_123ABC/stat/POWER1. I set this structure for my Tasmota devices when I was newer to MQTT, but now I would have chosen ’tasmota/%topic%/%prefix%/’, so tasmota devices get their own folder in the namepace, followed by each individual device. This is more of a personal opinion on how to structure MQTT topics. Save configuration here, and it should restart automatically.

Another configuration which is optional is to change the name. It doesn’t need a name, but if you set it, it will show the name on the main screen and in the HTML title. To set this, go to Configuration -> Configure Other and set Device Name to whatever is meaningful to you. If you have a bunch of Tasmota devices, having the name listed on the top of the page can really help you ensure that you’re connected to the correct one. You can also name the individual outputs, and naming them now will make things easier in Home Assistant later (using the Tasmota integration).

For the outputs we are using for garage doors, we want the outputs to turn off automatically (so sending an ‘ON’ causes the output to turn on, wait a second, then turn back off). This can be done by setting the parameter PulseTimeX, where X is the relay number (and X can be omitted entirely for single-relay modules like the Shelly). From the main screen, click Console, and type in the following commands to change the values. In my case, I am using 3 relays for my 3 doors, so I only set the pulse timer for those 3 relays. Tasmota Documentation on PulseTime. The number changes meaning at different scales, for numbers less than 111, it means 0.1 seconds, so 15 equals 1.5 seconds.

PulseTime1 15
PulseTime2 15
PulseTime3 15

Tasmota should echo the new value of the setting in JSON, such as {"Pulsetime":15}

Finally, after setting up the Sonoff, add it using the Tasmota integration in Home Assistant. Rename the devices so they make sense, the usual Home Assistant stuff.

Template Cover

Here’s the Template Cover I created for one of the garage doors.

First, make sure you have a covers section in your configuration.yaml:

cover: !include covers.yaml

Then I put my template covers in covers.yaml:

  - platform: template
    covers:
      garage_door_south:
        device_class: garage
        unique_id: 'garage_door_south'
        friendly_name: "Garage Door South"
        value_template: "{{ is_state('binary_sensor.garage_door_south_contact','on') }}"
        open_cover:
          - condition: state
            entity_id: binary_sensor.garage_door_south_contact
            state: "off"
          - service: switch.turn_on
            target:
              entity_id: switch.garage_door_opener_south
        close_cover:
          - condition: state
            entity_id: binary_sensor.garage_door_south_contact
            state: "on"
          - service: switch.turn_on
            target:
              entity_id: switch.garage_door_opener_south
        stop_cover:
          service: switch.turn_on
          target:
            entity_id: switch.garage_door_opener_south
        icon_template: >-
          {% if is_state('binary_sensor.garage_door_south','on') %}
            mdi:garage-open
          {% else %}
            mdi:garage
          {% endif %}

Project Files and Parts List

Here are all of the files and parts required to replicate this project. Since this project has no custom parts, there are no files, just things you can buy and open source software.

Tasmota software page
Tasmota Sonoff 4CH PRO R3 configuration
Tasmotizer flashing tool
Aqara Door Switch (affiliate link)
Sonoff 4CHPRO
Shelly 1 (Single Door Option) Some links to products may be affiliate links, which may earn a commission for me.

Studio Upgrade - New Mic!

Mon, 21 Mar 2022 08:27:05 -0300

Today I got my first upgrade to the studio setup: A new microphone! I chose a Rode Videomic Go II, which has USB-C and 3.5mm outputs, and they have a USB-C to Lightning cable available to connect to my phone. Since I use my iPhone for filming (often my older iPhone SE), this was important to me, but the ability to connect digitally to my desktop for recording screen capture with high quality audio was also important, so I didn’t want to get the smaller mics that clip directly on to the phone’s lightning port. The mic has a ‘cold shoe’ style mount, and since I don’t have any equipment with that, I 3D printed a bracket to hold the mic on top of my monitor that can also act as a kick-stand to hold the mic on a table.

Anyway here’s the video to go along with this blog:

I also 3D printed a mount for the mic, and you can download them below. As usual, CAD files are licensed CC-BY-SA. You’ll need a 1" #8-23 UNF fastener and nylock nut to attach the foot to the base.

Base Model (amf format)
Foot Model (amf format)
PrusaSlicer 3MF file
FreeCAD Original No idea if the nearest metric equivalent (M5) will fit in the nut pocket, the struggles of living in a country with backwards units.

Setting up a Proxmox HA Cluster

Thu, 17 Mar 2022 09:00:05 -0300

I’ve been playing with the Dell Wyse 5060 I bought before and I …. might have bought a few more. I was so impressed by the performance for $35, the CPU and GPU were very functional, and the system had enough SSD storage and RAM to do actual real work. So, considering people have tried to run Proxmox on the Raspberry Pi 4, these things must be adequate for a Proxmox HA cluster. Proxmox has a ton of clustering features, so using these cheap thin clients I was able to play with Proxmox’s HA and migration features. A fun bit of experimentation for sure.

This is part of the Hyper-Converged Cluster Megaproject.

The Video

Here’s the video that goes along with this. I don’t really have much to add here, I basically just run through the Proxmox cluster setup and demonstrate some features of Proxmox’s cluster features.

Hyper-Converged Cluster Megaproject

Thu, 17 Mar 2022 11:00:00 +0000

In this project, I explore using low cost thin clients as cluster nodes, the fundamentals of Proxmox clustering, redundant storage, and hyper-converged infrastructure using Proxmox and Ceph.

Setting up a Proxmox HA Cluster

In the first video, I take the Dell Wyse 5060 I bought before and … bought 2 more. Once I had 3, I built a complete high availability cluster using them, demonstrating the very basics of Proxmox clustering, high availability resources, and how Proxmox handles failure.

Small Proxmox Cluster Tips and Tricks, and QDevices

In this video, I walk through the nuances of 2 and 3 node Proxmox clusters, maintaining quorum, and installing a QDevice if you need one.

Making the $250 Proxmox HA Cluster Hyperconverged

In this video, I add additional RAM and storage to each node, and turn it into a hyper-converged cluster using Ceph. Ceph is the filesystem of BIG DATA, scale out solutions, so I’m excited to learn it. But Ceph is also a BIG topic, so in this episode I just focused on setting it up within Proxmox and setting up a basic replicated pool for RBD (RADOS Block Device) storage of virtual machines, all features that can be doen through the Proxmox GUI. I’ll dive into deeper Ceph topics in the future.

All about POOLS | Proxmox + Ceph Hyperconverged Cluster fäncy Configurations for RBD

Orders of Magnitude

Tue, 15 Mar 2022 00:10:05 -0300

When I reached 100 subscribers, I was very excited and posted about it here. But now, I’ve crossed 1000. After reaching 100, I was excited for the next hundred, but it came by so fast that within a few weeks I blew by 200,300,400… and was at 1000. The next goal becomes not a doubling, but the next order of magnitude. Exponential growth. I don’t expect to reach 100k (two more orders of magnitude), but maybe I’ll eventually reach the next order of magnitude.

On the flip side, every subscriber is more meaningful with lower numbers. Going from zero to one, one to ten, those individual users all meant something individually. Now, it’s more of a trend, tracking what my viewers like in aggregate, how videos respond with hundreds of views.

Just some late night thoughts on statistics and growth.

Also, a very cool video by Charles and Ray Eames titled ‘powers of ten’, for your curiosity.

Choose a Thin Client Session Graphically

Thu, 10 Mar 2022 08:27:05 -0300

In previous posts, I’ve been building up a thin client / VDI infrastructure based on Proxmox hosted virtual machines, using the SPICE protocol. This has gone well. However, the current setup basically launches the computer into a purely thin client mode, where it’s hardcoded to a specific VM and can do nothing else. There has been some interest in making some kind of launcher to select VMs to log in to, and I decided to find a solution for this. I started on the latest Raspberry Pi OS, but I found that they’ve hacked at their display manager / desktop environment enough that it’s not exactly like any other distribution, so in this blog post I’m going to try and implement this using both Raspberry Pi OS and Debian with LXDE installed. This is as close as I can get to the same environment as the Pi, so the instructions are similar.

This article is part of the Thin Client Series

This article was written with Raspberry Pi OS (32-bit) Bullseye release 2022-01-28 and Debian Bullseye release 11.2.0.

Sections

This page is really long, so here are pointers to individual sections

Video

Of course, I have a video to go along with this topic. Click the thumbnail to watch it on Youtube.

Setup for Raspberry Pi

Getting started, we need to use Raspberry Pi Imager to image an SD card with the latest version of Raspberry Pi OS. This time, we are starting from the full version, not lite, so we get the full graphical environment.

Once it’s installed, boot it up, and go through the ‘Welcome to Raspberry Pi’ wizard, which sets your keyboard layout, timezone, etc, and let it run updates. It might take awhile.

Now, we have a few more configuration things to do. Open a terminal, and run sudo raspi-config. We need to configure the following things:

System Options -> Boot / Auto Login, set to Desktop instead of Desktop Autologin (we want to require a login)
Interface Options -> SSH and enable it (we will use it to configure the Virtual Session users)

Now reboot and continue with the software install below.

Setup for Debian

I started with the Debian Netinst image and selected ‘Install’ (NOT ‘Graphical Install’), so we will end up with a console environment and need to install the graphical bits later. Important bits for the installer:

Do not set a root password, so Debian installs sudo
The new user you create will be the administrative user (with sudo permissions), so choose something good, not something you’re going to want to use later as a virtual session user
Choose Guided -> Use Entire Disk for partition type
Choose ‘All files in one partition’

Eventually, it will ask you if you would like to install a desktop environment.

Deselect GNOME
Select LXDE
Select SSH server (so we can debug like we could on the Pi)

We have one last configuration bit to modify: editing the lightdm configuration to show the user list. Edit it with sudo nano /etc/lightdm/lightdm.conf. Scroll down until you find #greeter-hide-users=false and remove the #, so the user list is not hidden.

Installing Dependencies

We already have a working graphical environment, so we need to update (just to be safe) and install remote-viewer (part of the virt-viewer package), and then create our thin client script. We also need curl, which isn’t installed by default on Debian but is on Raspberry Pi OS.

sudo apt update
sudo apt upgrade
sudo apt install virt-viewer curl

Thin Client Script

We’re using basically the same script as the previous iteration of this project, except this time we are sharing the script across users and passing in the VM ID as an argument. In addition, at the end of the script, we are killing lxsession which effectively logs us out, so instead of the user being stuck in kiosk mode forever, any attempt to exit the thin client results in being sent back to the login screen. Since both Raspberry Pi OS and Debian are using LXDE, the same script works for both.

Important note here. This script will ask the Proxmox API to return a spiceproxy file. The spiceproxy file, by default, will contain the DNS name of the Proxmox node currently running the VM (the name you set in the Proxmox installer). If that name does not resolve correctly on your network, you will need to add an argument to curl to force the spiceproxy file to use the specified proxy address instead (which is $PROXY, likely the IP address). I’ve left a line commented out that shows what the alternate command would be, so replace the uncommented curl command with the commented one if you need the proxy argument.

sudo nano /usr/local/bin/thinclient

#!/bin/bash
set -e

# Set auth options
PASSWORD='vdiuser'
USERNAME='vdiuser@pve'

# Set VM ID from the first and only argument
VMID="$1"

# Set Node
# This must either be a DNS address or name of the node in the cluster
NODE="pvehost"

# Proxy equals node if node is a DNS address
# Otherwise, you need to set the IP address of the node here
PROXY="$NODE"

#The rest of the script originated from a Proxmox example

#Strip the DNS name to get the node name
NODE="${NODE%%\.*}"

#Authenticate to the API and get a ticket and CSRF token
DATA="$(curl -f -s -S -k --data-urlencode "username=$USERNAME" --data-urlencode "password=$PASSWORD" "https://$PROXY:8006/api2/json/access/ticket")"

echo "AUTH OK"

#Extract the ticket an CSRF token from the returned data
TICKET="${DATA//\"/}"
TICKET="${TICKET##*ticket:}"
TICKET="${TICKET%%,*}"
TICKET="${TICKET%%\}*}"

CSRF="${DATA//\"/}"
CSRF="${CSRF##*CSRFPreventionToken:}"
CSRF="${CSRF%%,*}"
CSRF="${CSRF%%\}*}"

#Request a SPICE config file from the API
#Note that I've removed the proxy argument
#This results in Proxmox pointing remote-viewer to the node that is currently running the VM,
#instead of the node that we specified with PROXY. Only useful in clustered scenarios,
#but it doesn't hurt to leave it out.
#I left the other command commented out, so you can replace the first curl with the second if you need the argument
curl -f -s -S -k -b "PVEAuthCookie=$TICKET" -H "CSRFPreventionToken: $CSRF" "https://$PROXY:8006/api2/spiceconfig/nodes/$NODE/qemu/$VMID/spiceproxy" -X POST > spiceproxy
#curl -f -s -S -k -b "PVEAuthCookie=$TICKET" -H "CSRFPreventionToken: $CSRF" "https://$PROXY:8006/api2/spiceconfig/nodes/$NODE/qemu/$VMID/spiceproxy" -X POST -d "proxy=$PROXY" > spiceproxy


#Launch remote-viewer with spiceproxy file, in full screen mode
#You can add USB passthrough options here if you'd like
#Not calling via exec, so the script continues after remote-viewer exits
remote-viewer -f spiceproxy

#Kill lxsession
#This is how LXDE is designed to logout, it's not a hack lol
killall lxsession

And of course, don’t forget to make it executable

sudo chmod +x /usr/local/bin/thinclient

Creating Virtual Session Users

Since we are going to abuse the login system to select which thin client to launch, we need to create a new user on the local system for each VM we want to be directed to.

To do this, we use sudo adduser <username> It will ask you for a password, and you must give the account a password. If you don’t want users to require a password, we will get there later. When it asks you for user information, fill in the ‘Full Name’ field with the description of the VM. This field is what’s displayed in the login box, so you could set the full name to something like Windows 10 w/ GPU so users know what VM they are logging into.

To allow users to login without specifying a password, we’re going to modify the PAM configuration to automatically succeed if the user is part of the nopasswdlogin group. To do this, we need to modify /etc/pam.d/lightdm and add the following line right after #%PAM-1.0 at the top of the file:

auth    sufficient  pam_succeed_if.so user ingroup nopasswdlogin

And of course we need to actually create that group

sudo groupadd -r nopasswdlogin

Now, for each no-password virtual session user, add them to the group nopasswdlogin

sudo gpasswd -a <username> nopasswdlogin

Configuring LXDE Autostart

This is the step where we break everything and then fix it. Make sure SSH works so you can correct anything broken.

In short, when LXDE starts, it runs the autostart file for the system, followed by the autostart file for the user. The autostart file for the system includes things like a program which displays desktop icons (pcmanfm), and a program which displays the menu toolbar with useful buttons and widgets (lxpanel), and also the screensaver (xscreensaver). For virtual session users, we aren’t really a fan of any of these programs, since thin client shouldn’t have access to the underlying Linux system. However, we still need to retain a way to launch these programs when a local user logs in.

In order to achieve this, we need to completely empty the system autostart file and put all of the usual system-wide LXDE utilities (pcmanfm, lxpanel, xscreensaver) into the user’s autostart file, but only for users which have local control. This means any new users you create will be trapped with a desktop background and no way to do anything until you create a user autostart file, so be careful with this process.

To do this, log in as the user you want to retain local access (probably Pi?), backup the autostart file in case we need it later, and then copy it as the local users autostart file.

Commands for Raspberry Pi:

#Backup old system autostart file
sudo mv /etc/xdg/lxsession/LXDE-pi/autostart /etc/xdg/lxsession/LXDE-pi/autostart.bak
#Create new empty system autostart file
sudo touch /etc/xdg/lxsession/LXDE-pi/autostart
#Create configuration directory and parents for the local user
mkdir -p ~/.config/lxsession/LXDE-pi
#Copy the original system autostart as our new user autostart
cp /etc/xdg/lxsession/LXDE-pi/autostart.bak ~/.config/lxsession/LXDE-pi/autostart

Commands for Debian:

#Backup old system autostart file
sudo mv /etc/xdg/lxsession/LXDE/autostart /etc/xdg/lxsession/LXDE/autostart.bak
#Create new empty system autostart file
sudo touch /etc/xdg/lxsession/LXDE/autostart
#Create configuration directory and parents for the local user
mkdir -p ~/.config/lxsession/LXDE
#Copy the original system autostart as our new user autostart
cp /etc/xdg/lxsession/LXDE/autostart.bak ~/.config/lxsession/LXDE/autostart

Now, one at a time, we are going to create autostart files for each thin client user. Since you need to run these commands as the user in question, it’s easiest if you login over ssh as the virtual session user. Repeat this step for each virtual session user you have.

Commands for Raspberry Pi:

#Create folder where the autostart file goes
mkdir -p ~/.config/lxsession/LXDE-pi
#Create autostart file and edit it
nano ~/.config/lxsession/LXDE-pi/autostart

Commands for Debian:

#Create folder where the autostart file goes
mkdir -p ~/.config/lxsession/LXDE
#Create autostart file and edit it
nano ~/.config/lxsession/LXDE/autostart

Contents of the file:

@/usr/bin/bash /usr/local/bin/thinclient 100

Replace 100 with the VM ID you want this user to be launch. This file is not a shell script and is not executed as such, so we need to call bash (with the full path!), and pass it an argument for the script to run and arguments to further pass to the script.

Conclusions

This is a great solution if you don’t want the thin client to be tied to a specific VM. Ideally we’d be able to dynamically clone VMs as commercial VDI solutions do, but at this point, each virtual session user is tied to a specific VM ID in Proxmox.

Thin Client Megaproject

Wed, 09 Mar 2022 11:00:00 +0000

In this project, I explore thin client software, and set up a Raspberry Pi to act as a thin client. This is a multi-part project, follow along below for each sub-project.

Raspberry Pi SPICE Thin Client

The first project in this series creates a Raspberry Pi Thin Client which is permanently bound to a specific VM in Proxmox, and boots directly into the thin client session. You would use this when you have a 1:1 relationship between clients and VMs, such as a computer lab. When the user sits down at the client, we’re trying hard to keep them from doing anything at all on the local system, and channeling their work over the network to be performed by the virtual machine.

Adding USB Redirection to the Thin Client

The next project in this series explores USB redirection, and passing USB devices such as flash drives from the client to the guest virtual machine. This wasn’t completely successful, as auto redirection didn’t work properly, but hopefully this gives you a bit of a guide on making this work as you expect.

In this project, I add a login screen, allowing users to log in to the local system, with certain users automatically launching thin client sessions instead of a desktop environment. This project is more useful when you want to give users access to one of many virtual environments, possibly with password protection for some. We still don’t dynamically allocate VMs or anything like that, but we’re working closer to the ideal scenario of a ‘virtual computer lab’ where you can choose the type of machine you need when you log in.

PVE-VDIClient - A Python Graphical VDI Client for Proxmox

For the previous episode (see above), I setup a multi-user thin client where each user account was connected to a specific virtual machine. This is great, but if you have a lot of thin clients you might not want to create a ton of VMs and might instead want each user in the system to have one or more VMs. Well, Josh Patten has written a Python-based GUI to select thin clients which you have access to, and in this episode I turn his project into a Raspberry Pi (or Debian) appliance. This is even closer to my goal of a virtual computer lab, even if the administrator has to individually allocate VMs for users.

Setting up a Netboot (PXE) Server for Alpine Linux

While not strictly a thin client, this video is important since I’m going to use this netboot server going forward for my netbooted thin client projects. So, stay tuned for the second part of this video where I netboot PVE-VDIClient!

USB Pass Through for the Raspberry Pi Thin Client

Thu, 03 Mar 2022 21:27:05 -0300

As promised in my previous blog post on the topic, the SPICE protocol is capable of USB redirection. I didn’t dive into it for that post, but now the future is here and I’ve added USB redirection to my Raspberry Pi thin client. I’ve tested it with USB flash drives, webcams, and pretty much all of the USB devices I could find. And I’m here to tell you how it all works.

The Video

This blog post has a corresponding video. Click on the thumbnail below to watch it. For the commands and scripts used, continue down below.

Adding USB Ports in Proxmox

This one is pretty simple. In the VM’s Hardware panel, you just add USB ports, and select ‘SPICE’. You can add as many as you want. Each USB port you add can be used to pass through one USB device from the client, so add plenty of them. You need to restart the VM when you change the number of USB ports, but you can remap the USB ports you’ve already created while the machine is running (including changing them from SPICE to pass-through of host devices and back).

You can choose either USB2 or USB3 for this. In Proxmox, the underlying emulator (qemu) is emulating a USB host chipset and then attaching all of the USB ports to it, so choosing USB2 or USB3 changes the model fo the emulated chipset. It’s been recommended that the USB2 chipset is more reliable than USB3, and in our use case, it’s unlikely we will ever be able to exceed USB2 bandwidth in a thin client setup anyway (or that we’d want our users to tie up 1G+ of network bandwidth).

Configuring Remote Viewer

At this point, I edited the thinclient.sh file I created in the previous topic to switch from the locked down kiosk mode to simple full screen mode. you do this by changing -k --kiosk-quit on-disconnect to -f in the line which calls remote-viewer. This lets you use the UI of remote viewer by moving your mouse to the top of the screen.

Once you restart remote viewer so you have access to the UI (either reboot, or run killall xinit over SSH and then startx -- on the local terminal), you can pick USB devices to redirect from the USB button in the menu at the top. If you don’t have a USB button on the menu, then remote-viewer thinks there are no SPICE USB ports configured on the server, so check your Proxmox config and make sure you reboot if necessary to add more ports.

Of course, when passing through hardware, make sure you don’t pass through something critical to the system. Don’t pass through your actual keyboard and mouse, since you’ll lose the ability to even close the USB redireciton window.

Raspberry Pi Internal USB peripherals

On many models of the Raspberry Pi, devices are connected to the internal USB bus via a USB hub chip. On the Model A, the internal USB root is directly connected to the single USB port. But, on all Model B variants except the Pi 4, the single USB 2 root is connected to a USB hub and USB Ethernet chip. So, if we pass through absolutely everything, then the Ethernet chip will get passed through, and the Pi will lose ethernet. Not a great experience. So, we need to figure out what the USB vendor/product IDs of the Ethernet chip are so we can not pass them through automatically. To do this, use lsusb which should show a list of connected USB devices. In my case, I get something like this for a Pi 1 with nothing connected:

Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

In this case, the Fast Ethernet Adapter at 0424:ec00 is the device we are concerned about. Make sure you do this on the same model of Pi, since each Pi model uses slightly different hardware, especially the 3, 3+, and 4.

SPICE Client USB Redirection Rules

If you want to automatically connect some devices when the thin client starts, or automtaically while it runs, there are command line options for this. However, I was unable to get auto redirect to work in my setup once the thin client was already running - auto connect on start worked fine, and the USB redirection menu also worked fine, but plugging in a device and automatically passing it through didn’t happen.

The manual page is here for reference. It’s essentially all of the information available on the topic.

The key to this is that the rules (on-connect and auto-redirect) are written as a list which follows the following format:

<class>,<vendor>,<product>,<version>,<allow>

You then chain together rules, which are processed in order, like this

rule1|rule2|rule3

The example given is to block USB human interface devices and pass everything else, which is this:

0x03,-1,-1,-1,0|-1,-1,-1,-1,1

If you’re crafting rules, a handy reference is the USB device class list.

In my case, I wanto automatically redirect everything but USB HID devices and the USB Fast Ethernet controller on my Pi, so I wrote this rule:

0x03,-1,-1,-1,0|-1,0x0424,0xec00,-1,0|-1,-1,-1,-1,1

Auto Redirect Script Changes

I added a few new lines to thinclient.sh from the previous article. Here’s the new last few lines of the file:

#Define USB redirection rule
USBREDIR="0x03,-1,-1,-1,0|-1,0x0424,0xec00,-1,0|-1,-1,-1,-1,1"

#Call remote-viewer in fullscreen with redirection on connect and auto redirection
exec remote-viewer -f --spice-usbredir-auto-redirect-filter="$USBREDIR" --spice-usbredir-redirect-on-connect="$USBREDIR" spiceproxy

With this script, USB devices plugged in to the Pi will redirect when the Pi starts, although I wasn’t able to get them to redirect automatically in this environment.

Conclusions

This is a very useful feature that really sets SPICE apart from a lot of other open source or free VDI protocols. Some VNC servers/clients implement USB on their own (outside of the standard), but as far as I know, the Proxmox VNC server does not support these. Windows Remote Desktop Protocol (RDP) supports both high level redirection (of devices like USB drives and audio with the driver running on the client) as well as low level USB redirection, but it’s disabled by default and requires editing the group policies to allow it. RDP also doesn’t work natively with Linux guests, where SPICE works with all guests (even as far back as Windows XP).

My First 100 Subscribers!

Tue, 01 Mar 2022 15:03:05 -0300

As of today, my Youtube Channel has reached 100 subscribers! It’s really exciting to me, slowly starting to build a brand and produce content that I enjoy. I’ve enjoyed documenting my hobbies and projects on this website for almost a year now, and making videos is a fun way for me to expand that.

Thank you to everyone who’s enjoyed and commented on my content so far, it’s great to see that others are enjoying it too.

I don’t have any analytics on this site at all, and I hope you appreciate the desire to respect your data privacy, but I do know how much bandwidth I use per month, and last month, I passed more than 1GB for the site. I don’t get any decimals, but based on how small the site is on disk (the entire site, including static content, is currently 54MB), crossing above 1GB of traffic means someone must be reading it.

Buying a used Thin Client for 'Fun'

Tue, 01 Mar 2022 13:00:05 -0300

I recently was browsing eBay for used computer parts (as one does) and I found how cheap some used thin clients are. These are mini desktops with minimal IO which are designed to act as a modern terminal for modern VDI setups, something like Citrix or Microsoft Remote Desktop based enterprise systems. The thin client itself just has to deal with the local displays, keyboard, mouse, and local USB ports, the actual computation is done on a server somewhere else. This is all well and good, but a lot of these thin clients run Linux themselves, and we can repurpose them to run our own Linux environment for whatever we want. They’re also dirt cheap, presumably since the specs are pretty terrible if you want to use it as an actual desktop, but thankfully that’s not what I want to run at all

The Specs

The model I ended up with is A Dell Wyse 5060, which features an AMD Puma based quad core APU, 4GB of RAM, and a 16GB SATA SSD. It also includes a power supply, a somewhat rare find for it’s $35 shipped price. It’s very important to look at the specs of what you’re getting, since I found a ton of them with 2G of RAM and no SSD or an even smaller SSD. Expect none of the parts to be upgradeable, and 2G is a bit low these days. Many also have dual-core Atom CPUs, which use less power than the AMD Puma, but the Puma is functional as a desktop CPU in Linux.

Basically, this is a step up from a Raspberry Pi, but it costs less and you can buy it today. Consider it as an alterative for Raspberry Pi projects that don’t need IO or can benefit from gigabit, dual DisplayPort, or USB 3.

How to Install your own Software

By default, the BIOS is configured for legacy (non-UEFI) boot, and USB boot is disabled, so we need to fix both of those things. To enter the BIOS, press the DEL key while booting, and enter the BIOS password Fireport (capital F). From there, go to the Advanced tab and enable boot from USB and also set boot mode to Both. Then, go to the boot order tab and make the USB HDD higher than SATA 0 (by moving SATA down or USB up, using the +- keys). Save and exit. This should allow you to boot into a bootable USB drive and install whatever you want

Teardown Video

I made a video showing the hardware inside my thin client, if you’re interested in seeing a teardown, or watching the BIOS configuration. Click the thumbnail below to watch it.

My Plans

I bought this to use in my new studio space as a screen recorder. Basically, I’d have a USB HDMI capture device connected, so I can record directly to a network drive. I was able to set this up really easily by installing Xubuntu and OBS Studio, and the Puma CPU’s integrated graphics enabled transcoding seemed to work well enough to handle 1080p30 without a major sweat from the thin client. The CPU also has enough power to handle x264 software encoding, but it uses almost all of the CPU to do so.

Hardware Info for the curious

All of this was taken via an Xubuntu 20.04 Live CD, so your kernel may be configured slightly differently

lscpu

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   40 bits physical, 48 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      22
Model:                           48
Model name:                      AMD GX-424CC SOC with Radeon(TM) R5E Graphics
Stepping:                        1
CPU MHz:                         1000.000
CPU max MHz:                     2400.0000
CPU min MHz:                     1000.0000
BogoMIPS:                        4791.14
Virtualization:                  AMD-V
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        2 MiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled v
                                 ia prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user
                                  pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, STIBP disabled,
                                  RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtr
                                 r pge mca cmov pat pse36 clflush mmx fxsr sse s
                                 se2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtsc
                                 p lm constant_tsc rep_good acc_power nopl nonst
                                 op_tsc cpuid extd_apicid aperfmperf pni pclmulq
                                 dq monitor ssse3 cx16 sse4_1 sse4_2 movbe popcn
                                 t aes xsave avx f16c lahf_lm cmp_legacy svm ext
                                 apic cr8_legacy abm sse4a misalignsse 3dnowpref
                                 etch osvw ibs skinit wdt topoext perfctr_nb bpe
                                 xt ptsc perfctr_llc hw_pstate ssbd vmmcall bmi1
                                  xsaveopt arat npt lbrv svm_lock nrip_save tsc_
                                 scale flushbyasid decodeassists pausefilter pft
                                 hreshold overflow_recov

Of note, it does support AMD-V if you want to do virtualization, and AES-NI if you want to do anything involving crypto offload (VPN, HTTPS, …)

lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Root Complex
00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Mullins [Radeon R4/R5 Graphics] (rev 01)
00:01.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Kabini HDMI/DP Audio
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Host Bridge
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:02.5 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1
00:08.0 Encryption controller: Advanced Micro Devices, Inc. [AMD] Kabini/Mullins PSP-Platform Security Processor
00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 11)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 39)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 39)
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 39)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 42)
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller (rev 02)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 16h (Models 30h-3fh) Processor Function 5
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

As you can see, it uses a Realtek Ethernet Controller

vainfo

error: XDG_RUNTIME_DIR not set in the environment.
libva info: VA-API version 1.7.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_7
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.7 (libva 2.6.0)
vainfo: Driver version: Mesa Gallium driver 21.0.3 for AMD KABINI (DRM 2.50.0, 5.11.0-27-generic, LLVM 12.0.0)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            :	VAEntrypointVLD
      VAProfileMPEG2Main              :	VAEntrypointVLD
      VAProfileVC1Simple              :	VAEntrypointVLD
      VAProfileVC1Main                :	VAEntrypointVLD
      VAProfileVC1Advanced            :	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileNone                   :	VAEntrypointVideoProc

Hardware encode/decode is certainly usable for H264

Backup System Build

Sat, 26 Feb 2022 11:00:00 +0000

As I outlined in my first blog post on the topic, I want to build a new backup server, and I want to explore the different options I have. This project included a lot of testing, and will eventually culminate in actually building and setting up a proper backup system. It’s definitely an important and often overlooked part of a homelab, or even small business networks. Ideally, I can also get a functional offsite backup working, but that might be a future project.

Small Scale Tests

I’d like to back up bulk data stored in TrueNAS, as well as virtual machines and their associated storage stored in Proxmox. Ideally, the backup server will contain mostly lower cost spinning rust, store the complete contents of both the TrueNAS and Proxmox servers, and manage the long term archival and offsite backup. I have tried the software below, each has its own longer blog post on the topic and a short summary is listed here (More may be added in the future as I test more):

I have a few decisions I can make on the primary server side which affect the backup options in the backup server, such as:

Proxmox could use shared storage on a TrueNAS system, so backups are primarily done by TrueNAS instead of Proxmox. I could also use ZFS over iSCSI for this instead of NFS or CIFS so the images are zvols in TrueNAS instead of files on a dataset.
Proxmox in an cluster requires all of the backup and storage to be defined entirely in Proxmox, there is no way to reliably pull ZFS snapshots from Proxmox servers to TrueNAS in a cluster since the VM could potentially migrate from host to host.

TrueNAS

TrueNAS is a great solution for network storage, but is it also a good backup server?

Data Transfer to Backup Server

For TrueNAS -> TrueNAS, the feature is built in and works great. ZFS datasets and their snapshots are easily replicated from one system to another. All is golden.

For Proxmox -> TrueNAS, there are two options and neither are perfect. Proxmox supports a snapshot export, which makes a clone of the VM disk and compresses it, along with an archive of the VM configuration. This can optionally be pushed to remote storage (Even if the VM disk is on local storage). The downside to this is the TrueNAS backup server will end up with a dataset full of files, each of which are compressed VM disks. I can either deduplicate this dataset and hope it can recover some of this massive space penalty, leave the VM disks uncompressed and hope deduplication works better on uncompressed data, or waste a ton of space with the compressed images as files. The other option is the path I went down in my “Can TrueNAS backup a Proxmox host using ZFS replication?” blog post - create native ZFS snapshots of the zvol backing the VM, and pull them from PVE using TrueNAS’s builtin replication function. This works great for the VM disks, we get a history of snapshots natively in ZFS without the overhead of storing image files cloned from the zvol on a dataset, and incremental transfer of the backups. But, we don’t get the configuration file of the VM, and would either need to separately back them up or recreate them to restore.

Offsite Backup Options

The options you have here are to replicate to another offsite ZFS system, or to a supported cloud storage provider. Most people would probably use an S3 compatible provider or Backblaze B2, neither of which are particularly cheap if you are storing a lot of data, but the software support in TrueNAS is good if you are using a compatible service. While I didn’t make a blog post about it, I did make a quick video about using Linode Object Storage with TrueNAS, although I only pay for this for my most critical data.

Summary

In summary, TrueNAS is an excellent backup server when the primary server is also using ZFS and send/recv can be used to transfer snapshots. It’s less excellent when the primary server isn’t as cooperative or doesn’t run ZFS at all. If you are using shared storage on TrueNAS to host your PVE VMs anyway, then you can manage backups entirely in TrueNAS aside from the Proxmox configuration. If you are using local ZFS storage, then the method I tried in my blog post will work. If you are using anything else, you’re stuck with Proxmox compressing disk images and dumping them on a NFS/CIFS share on the backup server.

Proxmox Backup Server (PBS)

Proxmox Backup Server is a new offering from Proxmox, and designed for backup and archival storage of Proxmox VMs, CTs, and file storage (‘hosts’). It internally stores the data using its own archival / chunk format with deduplication, and client-side incremental sending, even in cases where the client’s underlying filesystem doesn’t support snapshots. It also has support for tape archival and tape loader management.

Data Transfer to Backup Server

For PVE -> PBS, the process is entirely automated in PVE. Proxmox supports scheduled backups to PBS, incremental backups, deduplication on the PBS side, and also live restores! The live restore function starts the VM as soon as it starts copying data, and transfers blocks from the server as the VM requests them. This means the VM is up and running immediately, even if it’s running slowly until all of the blocks have been read.

For TrueNAS -> PBS, the process requires us to run TrueNAS SCALE (Debian based) and run the Proxmox Backup Client through the command line via a cron job. I went through an exercise to script this backup in my blog post “Can TrueNAS backup to Proxmox Backup Server?”, where I created a shell script to backup ZFS datasets in TrueNAS to the Proxmox Backup Server. Since the backup isn’t done with zfs send/recv, the ZFS attributes and dataset properties aren’t copied as they are in TrueNAS -> TrueNAS. This means we are backing up the files, their contents, and their UNIX permissions, but not the associated ZFS dataset attributes (quotas, special properties, and dataset layout itself). We can separately backup TrueNAS’s configuration database (it’s in /data on the boot pool and copied to the system dataset nightly) as well as any ssh keys we might need to rebuilt the server, but this is something that a TrueNAS -> TrueNAS backup wouldn’t include either.

Offsite Backup Options

PBS offers two options to replicate backups - replication to another PBS server (a ‘remote’), and tape backups. The tape backup management can deal with autoloaders, scheduled backups to tape, and multiple sets of tapes. It seems fairly fully featured, although I haven’t personally tried using it.

Summary

PBS is the first choice for backing up a Proxmox cluster. It’s also capable of backing up Linux filesystems, but the client currently only supports x64 Linux and is only precompiled for Debian and Ubuntu. This is fine for use with TrueNAS SCALE (which is Debian based), but the backup client currently doesn’t compile for Raspberry Pi. Work is in progress to expand the client support to the Pi and it looks like a patch set is in place to build on AArch64 (ARMv8), and several other distros are beginning to compile it on for x64 for their own package repositories. The restore features with Proxmox are excellent (including live-restoring a VM!), and the Linux client can mount a backup as a filesystem to allow file recovery in a simple way. The tape archival support seems full-featured and easy to understand, especially compared to the other open source tape solutions I’ve read the documentation for.

Can TrueNAS backup a Proxmox host using ZFS replication?

Fri, 25 Feb 2022 17:20:05 -0300

As part of my series exploring backup options, I’m exploring the options for pulling a backup of a Proxmox Virtual Environment (PVE) host to TrueNAS SCALE server. In this case, PVE host has local ZFS storage, and the TrueNAS system is acting as the backup server. Ideally, PVE would snapshot in ZFS and we could sync those snapshots with a TrueNAS Data Replication task, but PVE doesn’t use the ZFS snapshot features by default. This complicates our setup somewhat. This test is part of the Backup Software Project, click there for other related projects.

Video

There’s a video for this article! Click the thumbnail below to watch it.

Proxmox Internal Backup Scheduler

Proxmox’s internal backup system relies on a qemu feature to export the VM disk coherently. When using the snapshot feature in the backup command, it will send a command to the qemu guest agent to tell the guest to synchronize disk activity (so ideally there aren’t in-progress writes on the VM disk), snapshot the ZFS zvol, then dump the contents of the snapshot to a compressed img file and delete the snapshot. This means that, even though the image is stored on a zvol in ZFS, we end up with a compressed img file in a dataset in ZFS, instead of a snapshot of the zvol in ZFS. Not ideal. Proxmox DOES create ZFS snapshots for the ‘snapshot’ feature (not the ‘backup’ method ‘snapshot’), but there doesn’t seem to be a way to schedule snapshots like you can schedule backups. But, I’d like ZFS snapshots to be created so TrueNAS can pull from them with zfs send/recv.

Using ZFS Snapshots on Proxmox

While the GUI won’t help us, there’s no reason we can’t execute the same commands to sync the VM and then take a zfs snapshot. In fact, someone has already written a python script to do this. So, I’m going to use his script. I first tested the script standalone and confirmed that it does in fact snapshot the zvol, then added the hourly/daily/monthly cronjobs and ensured those worked as well. Thank you to apprell for this.

Pulling the snapshots from TrueNAS

Now that we have PVE creating ZFS snapshots instead of files in the backup storage location, we can tell TrueNAS to pull them over SSH (using zfs send/recv). TrueNAS automates this almost entirely, which is super handy.

The first step is to create an SSH key pair which can be used to log in to the PVE system from the TrueNAS system. To do this, on TrueNAS, go in to Credentials -> Backup Credentials -> SSH Connections and click Add. Set the connection to Manual since this isn’t a remote TrueNAS system, add the host, port, username, and generate a new private key. Then, you can discover the remote host key. Save. Now, go to SSH Keypairs on the same screen, open the new keypair you generated, and copy the public key. I changed the name (truenas.local) for the IP address of the system, then copied the key.

We need to enter this key in PVE. Open the shell in the PVE web gui and run nano /etc/pve/priv/authorized_keys. Add the new key on a new line to this file and save it. It should look like the lines before it.

Now, we can add the replication task in TrueNAS, to pull the snapshots which are automatically being taken on PVE and copy them to the ZFS system on TrueNAS. To create this, go to Data Protection -> Replication Tasks -> Add in TrueNAS, and follow the wizard. Here are the important fields:

‘Load Previous Replication Task’ none
Source Location = On A Different System
SSH Connection = select the one you created earlier
Source = rpool/data (the zfs parent where PVE creates all of the VMs / CTs)
Check ‘Recursive’ under Source to copy all of the VMs / CTs
Destination = a new dataset under an existing dataset on your existing pool - type the name in, it will create a new dataset with the correct settings for you
Naming Convention = Snapshot Name Regular Expression
Regular Expression = .* (meaning all)
Task Name = something to help you remember
Setup the schedule of your choosing (at least as fast as snapshots are being taken, if no new snapshots exist it will copy nothing)
Deletion = Same as Source (the destination will delete a snapshot that no longer exists on the source)

One quirk I found is that I tried to replicate the rpool/data dataset in PVE (the parent dataset), after I had created snapshots of the VMs using the autosnap script above. However, TrueNAS refused to replicate recursively from rpool/data since it found no snapshots of rpool/data itself. To fix this, I created a single snapshot of rpool/data (zfs snapshot rpool/data@truenas) and found that TrueNAS was able to successfully copy the snapshots for the child datasets/zvols, even when they were newer than my manually created data snapshot. Since ZFS snapshots only affect the snapshotted item (not children), this snapshot of an empty dataset shouldn’t contain any actual data and thus I can leave it around forever just to make TrueNAS happy.

Restore Process

This backup method is functional, but incomplete. It only backs up the VM / CT disks, not the PVE configuration and qemu config file. So, the minimum process to restore a VM to a brand new Proxmox system is:

Create a new VM and configure it, then delete the hard drive. The configuration is lost with the ZFS backup, so you’ll need to recreate the entire VM configuration. Alternatively, you could restore an older Proxmox vzdump backup and then replace the hard drive with a newer copy from TrueNAS’s zfs snapshots, which gets you the VM configuration at the older point when the backup was taken and the VM disk at the newer point when the snapshot was taken.
Create a new replication task to push from TrueNAS to the new Proxmox system, including only the VM disk you want to restore, with the destination of rpool/data/vm-xxx-disk-y, where xxx is the VM ID from Proxmox that you are restoring to (which can be different from the existing name) and where y is the disk number within Proxmox (usually 0).
Force Proxmox to rescan the storage using qm rescan
You will find an ‘unused disk’ magically appears in the VM which matches the zvol you named in step 2, and you can add it as a hard disk and configure it

Conclusions

This method is functional, but incomplete.

It backs up the VM disks and CT datasets as snapshots in ZFS, and replicates the ZFS snapshots to the TrueNAS host. However, we don’t backup the Proxmox configuration metadata (including the qemu configuration file), so we would need to recreate the VM manually and replace the virtual drive with the archived zvol to retore it. You could mix this feature with Proxmox side backups, pushing compressed image files to TrueNAS over NFS/SMB, so you have periodic backups of the configuration in addition to more frequent snapshots of the disk images, but then you have two different types of backups managed in two places (pushed from Proxmox and pulled from TrueNAS) to deal with.

Like with the TrueNAS -> PBS setup, we had to do some minor console work to get this set up, as the feature isn’t accessible from the GUI. In theory it should work with TrueNAS CORE or SCALE (although I only tested this on SCALE).

How does it handle HA / clustered setups? You need to create a separate ssh connection for each node in the system, and a separate replication task for each node as well. Backups from each node will be in separate hierarchies on TrueNAS, and if VMs are moved between cluster nodes, the zvols on the old node will be orphaned in TrueNAS and need to be manually deleted. Likewise, if you outright delete a VM, the zvols will be orphaned in TrueNAS and also need to be manually deleted. So it could be worse, but it’s not a perfectly smooth experience.

Network Mounting OctoPrint Data from a NAS

Wed, 16 Feb 2022 09:27:05 -0300

Previously, I described my ‘Ultimate’ OctoPrint setup, and part of that setup process including remounting a lot of OctoPrint folders to locations on my NAS. This setup worked well until I added OctoLapse, and wanted to backup folders not part of the folder path configuration in OctoPrint. To solve this, I used a different approach entirely, using symbolic links instead of a bunch of network mounts to cleanly and easily relocate OctoPrint data to network storage. In this post, I’ll go through this process, without the length of my previous project (which included setup of all of my plugins, building the LED hardware, etc in addition to the network mount blerb).

Video

The corresponding video for this tutorial is below. Click the thumbnail to view it on Youtube.

Installing Software

I started with OctoPi, the Raspberry Pi distribution for OctoPrint. I imaged it on an SD card, booted up the Pi, and ran through the OctoPrint setup and updates. After that, I SSH into the Pi (default username is pi, password raspberry) to configure the netowrk mounts.

The required software to install is autofs. We should do an apt update and apt upgrade while we are at it, since the OctoPi distribution is not updated regularly and instead relies on OctoPrint to keep itself (but not necessarily the OS) up to date. At the current point in time OctoPi is still built on Raspberry Pi OS Buster instead of Bullseye, and you will get a warning that the repository has gone from ‘stable’ to ‘oldstable’. Accept the warning and continue.

#Just to be safe, run upgrade and update to make sure we are up to date
sudo apt update
sudo apt upgrade
#Install autofs
sudo apt install autofs
#Check to make sure it runs at boot, the install should have already done this.
sudo systemctl enable autofs

Configure Autofs Mounts

Since I wrote the last blog on this topic, I’ve found that it’s easeiest to mount a single directory from the server, and use symbolic links to point to the sub directories. So, the autofs configuration is pretty simple.

First, edit /etc/auto.master and add the following line, which points to a yet to be created shares file, and says any shares within that file are mounted relative to /:

/- /etc/auto.smb.shares --timeout 15 browse

Second, create the file (sudo nano /etc/auto.smb.shares) and add the new share we want to auto mount (copy this line and modify it as necessary):

/mnt/printer -fstype=cifs,rw,username=<user>,password=<password>,noperm,dir_mode=0777,file_mode=0777 ://server/share/directory

Key details here:

/mnt/printer is the location we are mounting the share on the local system
Replace <user> and <password> with the authentication information you’ve setup on your NAS for this share
noperm, dir_mode, file_mode mean that permissions are ignored on the local system. The NAS still enforces file permissions on its side. I found these options are needed for Octolapse to work correctly, since it often tries to chmod files in the timelapse folder for no apparent reason, and without the local system ignoring permissions, Octolapse will often fail to chmod files as it does not own them.
Obviously, server and share are dependent on your NAS. IP addresses and DNS names also work here for server.

Finally, create the directory at the mount point:

sudo mkdir /mnt/printer

And restart autofs so it reloads the configuration files

sudo systemctl restart autofs

Relocating OctoPrint Directories to the Network

I’ve setup a subsection with the commands for each folder. You can copy/paste whichever sets you need. Run these as user pi. If you are running on a new installation or otherwise don’t want to copy out the existing data, you can omit the rsync step, which copies data out of the old location before deleting it.

Also, you should not share the same printer directory between multiple printers. You should create a different folder on your NAS for each printer, and then create the folder structure underneath that mount point.

Watched

#Create new directory
mkdir /mnt/printer/watched
#Go to location
cd ~/.octoprint
#Delete old directory, watched should be empty already
rm -r watched
#Make link to new directory
ln -s /mnt/printer/watched

Timelapse and Timelapse Temp

This is a good candidate for network mounting.

#Create new directories
mkdir /mnt/printer/timelapse
mkdir /mnt/printer/timelase/tmp
#Go to location
cd ~/.octoprint
#Copy data to new location
rsync -r --progress timelapse/ /mnt/printer/timelapse
#Delete old directory
rm -r timelapse
#Make link to new directory
ln -s /mnt/printer/timelapse

Uploads

If you network mount this directory, do not edit it, let OctoPrint manage it on its own.

#Create new directory
mkdir /mnt/printer/uploads
#Go to location
cd ~/.octoprint
#Copy data to new location
rsync -r uploads/ /mnt/printer/uploads
#Delete old directory
rm -r uploads
#Make link to new directory
ln -s /mnt/printer/uploads

Logs

I’d generally recommend against relocating this folder to the network share, since you won’t get logs at all if the network share is inaccessible (autofs makes the directory inaccessible if it fails to mount). If you really want to reduce SD wear, you could mount the logs directory as a tmpfs or use folder2ram (which creates a tmpfs which is read from the SD card on boot, and written back on clean shutdown). It all depends on how you want to store your logs and what failure conditions you are most concerned about.

#Create new directory
mkdir /mnt/printer/logs
#Go to location
cd ~/.octoprint
#Copy data to new location
rsync -r logs/ /mnt/printer/logs
#Delete old directory
rm -r logs
#Make link to new directory
ln -s /mnt/printer/logs

Backup

You should absolutely mount this folder on the network, to keep your backups safe from SD card failure.

#Create new directory
mkdir /mnt/printer/backup
#Go to location
cd ~/.octoprint/data
#Copy data to new location
rsync -r --progress backup/ /mnt/printer/backup
#Delete old directory
rm -r backup
#Make link to new directory
ln -s /mnt/printer/backup

Octolapse Temp

Note that Octolapse timelapses still go in the timelapse folder after they are rendered, which you can redirect using the normal timelapse commands above. However, the temp folder is in a different location.

#tmp was already created above, no need to create another
#no need to copy tmp either, since it's temporary
cd ~/.octoprint/data/octolapse
#Delete old directory
rm -r tmp
#Make link to new directory
ln -s /mnt/printer/timelapse/tmp

Proxmox Clustering with 2 Nodes

Sun, 13 Feb 2022 00:01:05 -0300

I’m experimenting with Proxmox Virtual Environment (PVE), the same hypervisor I run on my Minilab. It supports clustering and high availability, and I’d like to implement the cluster option. Clustering without HA allows multiple nodes to be managed from a single user interface, and for VMs to be offline migrated between nodes. This sounds pretty useful for me, even without the high availability features like live migration. However, any cluster relies on a node voting scheme which requires agreement (quorum) from all of the nodes, and the cluster won’t function without quorum being met. With only two nodes in a cluster, it’s impossible to achieve quorum and no VMs will start. The usual solution to this is to add a new device specifically to vote (a ‘QDevice’) running on another server or a Raspberry Pi, but it’s also possible to give one node an extra vote to break the ties.

My minilab uses relatively little power (it’s fed by a ~90W DC power brick) and is self contained with local storage, and it’s adequate for many of my continuously running services, but I still have to rely on VirtualBox on my workstation for testing a lot of VMs at once. Ideally, I could setup a higher performance node for testing and power it down when I don’t need it, with the ability to migrate testing VMs to the production mini-server. In this use case, I can be fairly certain that the minilab will always be running but the testing server might not be, so giving the minilab an extra vote means it can maintain quorum on its own, even if the ‘megalab’ is shut down. This would allow me to operate without a seprate Qdevice and still cluster the two nodes together.

Quick Video

This video is a really quick tutorial which focuses on how to do it, and not the background behind why.

The Proxmox Cluster Filesystem

Proxmox stores the vast majority of configuration files in the Proxmox Cluster Filesystem (pmxcfs). Configuration for VMs and containers are stored in the cluster filesystem, along with configuration for each node and the cluster itself. Corosync is used to replicate the cluster filesystem across all of the nodes in the cluster, and deal with distributed locking of cluster configuration files. The cluster filesystem is used even for single-node Proxmox installations, but without synchronization across the cluster.

The cluster filesystem is mounted at /etc/pve, so files in this path can be edited by any node and synchronized automagically. The cluster configuration file itself is located at /etc/pve/corosync.conf. We need to edit this configuration file to change the quorum settings, but we need to be very careful not to mess anything up since corosync will propagate any change we make to all nodes immediately.

Editing the Configuration File

Proxmox has some warnings on editing the corosync file - see here. Specifically, they recommend creating a copy of the file when editing it, so accidental saving of the unfinished file doesn’t cause problems. So, we will do that.

cp /etc/pve/corosync.conf /etc/pve/corosync.new.conf

Now, we can edit the corosync config file in nano to give minilab two quorum votes, so it can run on its own. nano /etc/pve/corosync.new.conf

First, in the nodelist section, increase quorum votes to 2 for the node that we want to run on its own:

nodelist {
    node {
        name: minilab
        nodeid: 1
        quorum votes: 2
        ring0 addr: 10.0.0.3
    }
    node {
        name: megalab
        nodeid: 2
        quorum votes: 1
        ring0 addr: 10.0.0.4
    }
}

Increment the config version key in the totem section by 1 (in this example, from 2 to 3):

totem {
    cluster name: cluster
    config version: 3
    interface {
        linknumber: 0
    }
    ip version: ipv4-6
    link mode: passive
    secauth: on
    version: 2
}

And finally, copy the modified file back to the original:

mv /etc/pve/corosync.conf /etc/pve/corosync.conf.bak
mv /etc/pve/corosync.new.conf /etc/pve/corosync.conf

If you really mess things up, it’s possible that the nodes won’t quorate. If this happens, the entire Proxmox cluster filesystem becomes read-only. You can temporarily reduce the required quorum to 1 to continue: pvecm expected 1.

Conclusions

For homelabs which don’t have enough nodes to maintain quorum, or which don’t run all of the nodes all the time, increasing the votes for the continuously running servers can help keep the cluster online without adding additional QDevices and depending on those to be online.

Backing Up TrueNAS Datasets to Proxmox Backup Server

Thu, 03 Feb 2022 13:27:05 -0300

As part of my series exploring backup options, I’d like to see if I can use Proxmox Backup Server to archive both datasets and zvols of a TrueNAS SCALE server. Why would you want to do this? In my case, I’m trying to choose the best starting point for my new backup server, and one potential option is to use Proxmox Backup Server (PBS), but I’d like to store data outside of the Proxmox Virtual Environment (PVE) ecosystem. So, I need to make sure I can setup backups from TrueNAS to PBS in a way that makes sense and isn’t too difficult to manage. Based on how both PVE and TrueNAS handle backups, I’m not going to get everything I want out of either TrueNAS or PBS on the backup server, but I’d like to know how much manual scripting I’ll have to do with whichever option I choose. This test is part of the Backup Software Project, click there for other related projects.

The Video

Here’s a video of the process. Feel free to watch it before or after you read the rest of the post. Click the thumbnail to go to Youtube.

What Is Proxmox Backup Server?

Read their description here. Basically, it’s a backup system designed to integrate with Proxmox Virtual Environment (PVE), a hypervisor. However, it really manages 3 types of backups - virtual machines, containers, and hosts. The host backup option is designed to backup the entire filesystem of a host system by using a backup client on the system, proxmox backup client. With the host backup method, PBS can archive the entire contents of a file system, and the user can view the backup archive and download individual files or mount the entire archive as a read-only filesystem. This is pretty handy, and it’s the method I’m going to use to try to backup the TrueNAS storage server.

What are the downsides?

Well, TrueNAS has no integration for this, so I’m on my own with a shell script and cron job to ensure it runs. PBS deals with deduplication, snapshot management, etc. entirely on its own, so I could just run the cron job often and let PBS deal with the rapid influx of data. Another downside is that the PBS client only runs on Debian or Ubuntu Linux, which limits me to using TrueNAS SCALE instead of CORE (the traditional FreeBSD based version).

Another downside is that PBS doesn’t include mount points in the backup (so backing up /mnt on a TrueNAS system won’t backup anything, since all of the ZFS datasets are separate mount points). However, we can get around this using the --all-file-systems flag, if we actually want to backup all of the datasets as one folder, but this merges all of the ZFS datasets as subfolders instead of as separate pxar archives. You have the tradeoff between explicitly listing each dataset in the shell script (with the possibility of forgetting some and not backing them up) vs backing up all file systems into a single archive.

PBS is also unaware of ZFS on the client side, so it’s reading the live filesystem intead of a coherent snapshot and copying that. TrueNAS’s native duplication tasks work by taking a snapshot in ZFS and then copying that using the ZFS native send/recv methods, and none of that is done with PBS.

So, for a disaster recovery type backup, this seems like a functional solution, but ZFS snapshots are still much more convenient to deal with user-induced file deletion, ransomware, etc. Depending on how badly you want the PBS integration in PVE and number of datasets being backed up in PVE vs TrueNAS in your environment, it might make sense to deal with these limitations and continue using PBS to backup everything.

How do I set it up?

First, I need to install the proxmox backup client on the TrueNAS system. They only have install guides and binaries for Debian and Ubuntu, and we can use this on TrueNAS SCALE (which is based on Debian Bullseye as of this writing). So, you add the Proxmox public key and sources.list entry, run apt-get update, then apt-get install proxmox-backup-client. Fairly simple.

After this, I need to write a backup script for TrueNAS to call to initiate the backup. I created a new dataset on TrueNAS specifically to hold the backup script, which is itself part of the dataset which will be backed up. This way, the script itself can be easily restored along with the data. Everything is self-contained, including setting the environment variables for the API key / secret and host of the PBS server. Here’s a version of the script I use to do the backup

#Authentication and host information for PBS server
export PBS_REPOSITORY=<apikey>@pbs@<IP or hostname>:<datastore>
export PBS_PASSWORD=<apisecret>

# Backup Specifications
SPEC=""

#Backup dataset media on pool tank
SPEC="$SPEC tank.pxar:/mnt/tank/media"

#Backup dataset backup on pool tank
SPEC="$SPEC backup.pxar:/mnt/tank/backup"

#Backup a zvol
#I don't use this personally, so I can't test it, but this is the syntax for the backup client
#Don't try to back up a zvol while it is mounted / in use
SPEC="$SPEC zvol.img:/dev/zvol/tank/volume"

#Perform backup
#You may add your own client-side encryption if you wish
#By default, the client hostname is used as the backup id in PBS
#You can optionally specify your own using --backup-id <name>
#If you want each pool to have a unique backup ID, you'll need to call the client
#one time for each backup ID, each with a different spec.
echo SPEC is $SPEC
proxmox-backup-client backup $SPEC --all-file-systems true

Finally, I need to tell TrueNAS to run the script routinely. PBS will deduplicate at the client level (so it won’t send the entire dataset over the network if nothing has changed), so it’s safe to call this essentially as often as you want and let PBS deal with pruning on its own.

To do this, I went to System Settings -> Advanced and added a Cron Job. The job calls the backup.sh script hourly as root. Root should have the required permissions to read all of the files in the zfs pool, so it should be able to archive them as well. We could also create a new user for this job, and give them read-only permissions to the entire dataset.

Restore Process

The process for restoring a file backup, after rebuilding the TrueNAS system and assuming the PBS system is still fine, is to do the following:

Recreate the datasets and permissions (the UNIX permissions of files and folders are retained by PBS, but the ZFS datasets permissions are not).
Make a new mount point in /mnt to mount the backup using the backup client mkdir /mnt/backup
Mount the pxar archive using proxmox backup client’s FUSE abilities proxmox-backup-client mount host/<hostname>/<time> <archive>.pxar /mnt/backup
rsync the contents of the FUSE-mounted backup to the new ZFS pool

Conclusions

This method is good for backing up data, but we lose the ZFS attributes and ACLs for the datasets and will need to recreate them before we copy files off the backup.

It seems like running backup jobs from TrueNAS to PBS is functional with relatively minor console work, if you just want the contents of the filesystem and not the ZFS snapshots to be archived. It’s not as clean as the replication tasks within TrueNAS, but backing up a TrueNAS dataset to a PBS server seems like a functional solution. You have to use TrueNAS SCALE at this point, since the PBS client only runs on Linux, but the TrueNAS community seems to be moving to SCALE over CORE anyway. It’s not a bad solution

My History of Storage, Preparing for a new Backup Server

Mon, 31 Jan 2022 17:32:22 -0300

I’m working on the next revision of my homelab backend. Currently I rely on an Ubuntu server with ZFS on Linux for file storage over Samba, and a separate Proxmox Virtualization Environment (PVE) server (the Minilab) with local LVM storage for virtualization. Ideally, I’d like to add a backup server to the mix, with its own storage, that can handle both the Proxmox server and Samba shares. However, the big choice ahead is what software to use for backups - TrueNAS (CORE or SCALE), or the newer Proxmox Backup Server (PBS)? Both have their advantages and disadvantages, and I’d like to explore them before deploying one as my new backup server.

My Life in Storage, A History

When I started building my network, I relied on the original Raspberry Pi’s, each with a USB hard drive and each storing some files. I’d essentially manually distribute files across the Pi’s, and remember where data was stored. This was pretty terrible for my own data, but a lot of this data was media which would be indexed by XBMC (later KODI) so I didn’t really need to remember which drive it was on. I had multiple Pi’s since the original Pi had a pretty terrible processor and 100Mbit Ethernet, so using more than one meant it was possible to (on average, depending on where the files were located) access data at a reasonable speed from multiple clients. This was not a great solution, but it worked pretty well for me at the time.

Eventually I upgraded to FreeNAS, and built a new server for this purpose. I used an AMD Bulldozer based APU on an ITX board, using a pair of USB sticks to boot (at the time this was somewhat common for FreeNAS), onboard SATA for the drives, and a PicoPSU for power. This worked great for file sharing, and I was happily able to migrate all of my data from the old USB drives to the new ZFS pool. The downside to this setup was the lack of virtualization support. Back when I built this it wasn’t a huge deal, but over the years I’d try to get more into virtualization and FreeNAS (built on FreeBSD) just wasn’t great at it. What it was good at were jails (the BSD equivalent and predecessor to LXC containers), so as long as I could run software on FreeBSD, I could run it in a jail. I had a MySQL server for KODI, as well as some energy monitoring software for the first one-wire network, but it was always a struggle for me due to my inexperience with BSD at the time.

Eventually all of my USB sticks died and I got super sick of dealing with USB sticks in general (even ‘brand name’ ones can be absolutely awful quality, even for a read-only boot drive). Initially, I wanted to run ZFS on Root on Ubuntu to avoid the need for a separate boot drive and bring me back to a more familiar Linux environment (also with better software and driver support), but Ubuntu support just wasn’t quite ready for this, so I ended up getting a cheap SSD to run Ubuntu on EXT4 and then mount the ZFS pool. This had the side effect of limiting expansion of my ZFS pool, since I no longer had enough free SATA ports to add another mirror vdev. I setup this configuration, using LXC containers for my virtual environments, and it worked well. The Bulldozer processor really couldn’t handle proper virtualization anyway, and all of the software I wanted to virtualize ran in Linux, so containerization was great. I upgraded to a 10G NIC rather painlessly, and was pretty happy. I also eventually replaced the Bulldozer system with a new AMD Ryzen 3400G, so I could use the GPU for hardware acceleration of ffmpeg with Zoneminder. Being in a container, hardware passthrough is really easy (at least compared to PCIe passthrough with VMs) and all of this worked great.

After this, I started using my mini desktop (Asrock A300 Deskmini with Ryzen 2400G) as an XCP-NG and later Proxmox host, with a single 500G SSD for both the OS and VM storage (using LVM-Thin). This brings us to the present day, where I need to backup both the Ubuntu server (or its replacement) and the PVE host.

My Backup Needs and Options

Currently, my ZFS mirror is essentially my only backup. I regularly scrub it, but the data on it isn’t all stored somewhere else. That said, a large portion of the data stored on the pool is either nearly worthless shortly after it’s created (such as the security camera recordings from Frigate) or is using the ZFS pool as a backup location (i.e. all of my 3D printers, HomeAssistant, etc. push backups to the ZFS pool), so the data is still stored in the original working location. I’d certainly be able to recover from a failure of the storage server without much loss, but it’s not a friendly idea. At the same time, I’m kinda sick of dealing with command line Ubuntu for managing my Samba users and shares. So, upgrades are needed again.

What data do I have?

‘Bulk’ data in the Samba shares, including project files, raw video for editing, etc, some of this is very large and some is very important, but it’s mixed
Media, which was previously used with KODI and may be used with something like Plex or Jellyfin in the future (although realistically streaming services have improved a lot in recent years)
Backups of devices on the network such as HomeAssistant, and Raspberry Pi’s which push data over Samba periodically
Virtual machine drives from Proxmox

For me, the ideal setup would be to use TrueNAS (which is great at user management for sharing) for storage and Proxmox (which is great at virtualization), each running on their own hardware and with their own local ZFS pool doing what they do best. The challenge now is which to use for the single backup server?

TrueNAS has great features to push/pull from other ZFS systems using zfs senc/recv, as well as cloud sync features such as S3 which might be useful for me for very critical data. In theory, I might be able to pull snapshots from a PVE server which is using ZFS, but that’s not a feature built in to PVE.
Proxmox has PBS, which integrates well with PVE to backup virtual disks as well as filesystems and deal with archival, including tape backup. I’m particularly interested in tape backup although I know it’s not cost effective for me, but it’s cool and PBS seems to have a good solution for dealing with tapes.
TrueNAS as a backup server can also be a storage host on its own, allowing devices which support NFS/SMB to push their own backups to the backup server directly (i.e. Home Assistant can push a configuration backup to an SMB share, and this could be on the backup server instead of the working data server). PBS has no option to operate this way.
PVE doesn’t natively support delta backups when using local storage without using PBS, although I could store the VM disks in TrueNAS and mount them over NFS so I can do snapshots on the TrueNAS side, or use TrueNAS to pull snapshots from the PVE server.
PBS doesn’t support pull-style backups at all, it relies on the PBS client to push data to the backup server, so I’d need to run the PBS client on TrueNAS SCALE to use PBS as my backup system.
TrueNAS backup works mostly by using ZFS send/recv, so I’d need a completely separate solution to handle tape backup if I ever want to go there.
PBS doesn’t handle cloud/S3 backup like TrueNAS does, although I don’t plan on archiving a lot of data to the cloud, having the ability to back up a critical data share would be nice. I could of course do this entirely on the TrueNAS instance which stores the working data instead of the archival instance.

To try and answer these questions, I’m going to try and setup TrueNAS, PVE, and PBS all in virtual machines (PVE doesn’t run well in nested virtualization unfortunately) and try to see which set of interactions works best for data storage, PVE, and backups. I don’t want to virtualize TrueNAS given how much I work directly from the file shares, and I also don’t really want PVE to rely on remote storage, but those might be compromises I’ll have to deal with to get a single working backup solution.

Which backup solution would you use, and why?

Backing Up OPNsense to a Samba share using Node-Red

Sat, 22 Jan 2022 01:44:05 -0400

As I’ve posted about in other blogs, I use OPNsense as the firewall for my home network. It has an inbuilt method of backing up its configuration to the cloud, but I’d like to avoid that and back up locally. Unfortunately, there isn’t a plugin in the repository to have the firewall push a backup to a samba share, so I need to run code somewhere to pull the configuration from the firewall and store it on the storage server (where the backup policies will take care of it). I use Node-red for this, since I already have a lot of flows to pull data out of OPNsense to log in InfluxDB.

OPNsense Configuration

It has a fairly robust method of storing configuration in an XML file, so backing up the XML file alone is usually sufficient to restore the configuration in a disaster recovery scenario or after a bad update and total reinstall. In fact, you can migrate between hardware without too much trouble as well, although some of the interface configuration is specific to which Ethernet drivers are in use. It’s technically possible to have configuration outside of the XML file if you write custom options for Unbound or some other services, so if you use those custom config files, beware.

I quickly found a shell script which can do the backup on the OPNsense Forum, and it relies on the os-api-backup plugin (which exposes an API endpoint to download the whole configuration). See Here for the Shell Script. I will use the same os-api-backup plugin, but do the backup through Node Red instead.

Node-Red Samba File Access

To access the filesystem via Samba, I’m using the node-red-contrib-smb plugin. It seems to work adequately, although there are a lot of caveats to this (you must do \server\share with no subsequent paths, for example). It also doesn’t always work correctly without Windows-style paths, which must be escaped of course. So, beware of the frustration.

The Flow

Anyway, here’s my node-red flow to pull the backup from OPNsense’s API, write it to a file, read the contents of the directory it’s in for autobackup files, and trim the autobackups to keep the most recent only. It won’t touch other files in the folder.

As always, I recommend creating a new samba user on your NAS for this client, and give it permissions to only its backup folder. How you do that is up to you.

As always, my work is licensed CC-BY-SA unless otherwise specified. Link to the flow

My Start in Twitch streaming

Sun, 16 Jan 2022 01:42:05 -0900

I’ve decided to start Twitch streaming my evening gaming sessions. I usually spend a small amount of my evening playing games anyway, so broadcasting it to the world isn’t a big change to my routine, and hopefully I can improve my content creation skills at the same time. I play almost exclusively single-player games, particularly first person shooter, puzzle, and simulation games. I’m slowly building up my streaming stup and it should help with video creation too, which is something I’m excited about. I’ve already used OBS a lot to stream robotics competitions, so it’s not new to me, but Twitch is new, and so is being a personality.

My very first stream was two weeks ago (2022-01-05), where I streamed the game INFRA, a game I’ve never played before and only watched a very short let’s play segment on. I already have a low quality webcam (Logitech C270) that I bought to use with my 3D printers (but quickly found the Ethernet cameras far superior for that use case), and had it mounted above my desk facing my keyboard so I could record build videos on my desk, which is a bit of a weird angle to stream. But, I was eager to get into the game and play, and streaming was more of an afterthought, so I setup a view in OBS with the game and the camera, and went at it. It’s a bit weird since I play on an ultrawide (21:9) monitor, and the stream is normal width (16:9), so there are black bars in the top and bottom. I’ll decide later how to handle this.

My Better Setup

After playing INFRA on stream, I decided to get a slightly better setup. I ordered the parts, waited for them to ship, and continued playing INFRA with a progressively looking better OBS setup in the meantime. I replaced the black bars to fit the ultrawide game with an orange background to match my website, with the stream name and my cameras below the 21:9 game screen. I think this fits fairly well, the game isn’t covered by anything (so I don’t have to rearrange the screen depending on where the HUD is in-game), and I like the aesthetic.

The parts I settled on are two Logitech C270 webcams (720p30 and dirt cheap at $25 each), along with one Logitech C922 Pro HD (1080p60 with autofocus and a few other nice features). I also got a Stream Deck, which is useful for more than just streaming (i.e. I can use it in Davinci Resolve, and I already paired it with Home Assistant using the open source plugin to control my room lights). I intended to use the C922 for my face, but I realized that the game is really front and center and the tiny face view quality isn’t super important, so I have a C270 looking down and C270 facing me. I’ll reserve the C922 for VR streaming, which I haven’t started to setup yet.

The Future

I’m enjoying streaming casually at this point with the two cheap cameras, and the stream deck is definitely not a requirement but it is helpful. Eventually, I’d like to expand into VR streaming, and depending on the game, there are lot of different camera layouts I could use. For a fixed location game like Beat Saber, a single camera with Mixed Reality works well. For a more open world game like Half-Life Alyx or Boneworks, it might not work as well, but I’ll play around and see. I’ll be sure to post more here when I get my VR streaming setup dialed in.

My Channel

Link to my Twitch channel

Using a Raspberry Pi as a Thin Client for Proxmox VMs

Wed, 12 Jan 2022 09:27:05 -0300

Virtual Desktop Infrastructure (VDI) is quite a buzz-word now in enterprise computing, and it’s something I’d like to experiment more with in my homelab. Essentially, it’s a new way to describe old school terminal servers, but with modern features and marketing. The primary difference is that VDI normally implies that each ‘seat’ is a virtual machine and has some resources associated with it, as opposed to a terminal session running on a shared server. By using VDI, an admin can centralize all of the compute resources and the end devices only need to provide an interface (video / keyboard / mouse), and also guarantee resources such as RAM or GPU to the virtual desktop (something a terminal server does not do). This means the end devices can be significantly cheaper, since they aren’t doing much real work, although they now have to deal with a video stream of the virtual desktop.

In my specific use case, I would like to use a Raspberry Pi attached to the back of the monitor as a general purpose PC in the kitchen. I could just use the Pi itself, or a more expensive device like a NUC, but I already have a Raspberry Pi B+ and a perfectly useful server, so putting compute resources on the server would be ideal for me. Plus, I’d like to expand my knowledge of the different methods for VDI over the next few months, and this is a good start.

My goals for the experiment:

Use a Raspberry Pi B+ as a usable general purpose web browsing desktop
Host the general purpose desktop on my lab server (Proxmox)
Boot the Raspberry Pi directly into the server session without needing to log in to the Pi or launch the session.
Must support both Linux and Windows targets

Since this is a really long post, here are some bookmarks to help you navigate:

Video Form

I produced a video which covers some of these topics, click on the thumbnail below to watch it. While recording the video, I was able to find a Raspberry Pi 2 Model B. The article was written with an original Raspberry Pi, and the Pi 2’s performance is far more usable for general desktop usage, but not video playback.

Setup Linux VM for testing

I’ll use Ubuntu 21.10 Desktop for this, since it’s the latest version of Ubuntu as of this writing. I created a new VM in Proxmox with pretty minimal hardware:

4 CPUs
2GB RAM
32GB SCSI disk - emulated SSD, enable discard (TRIM support)
Graphics as SPICE
Add a device for audio, using ich9-intel-hda driver, with a SPICE backend

After creating the VM, I booted it to run the installation. Since graphics are SPICE, I need to use a SPICE client on my workstation (instead of the usual noVNC web app). I installed virt-viewer from virt-manager for Windows, and it includes remote-viewer which is designed for this purpose. I selected a normal desktop for Ubuntu and let it run. I had absolutely no issues with SPICE integration on my workstation, the Ubuntu VM integrated the mouse seamlessly with the host and the keyboard worked on the guest when the moust was over the host, as expected.

Setup Windows VM for testing

I also wanted to prove that this approach to VDI works with Windows, so I installed Windows 10 64-bit in a VM as well. It’s not as happy with little RAM as Linux is, so it gets 4GB. My Proxmox test host isn’t exactly well endowed with memory. The rest of the setup is the same, emulated SSD, enable TRIM, graphics and sound as SPICE.

Windows installed fine with SPICE, but once it was installed and rebooted, the SPICE keyboard and mouse didn’t work. I stopped the VM, switched back to default (VGA emulation) and rebooted it. Now using noVNC, they worked again. With this, I was able to get Windows to boot, let it deal with a ton of updates (as is Windows), and then it made me select my region and keyboard layout (all things Ubuntu did while it was copying files), and then it ‘had some important setup to do’ and left me waiting again. What a pain. Windows also really didn’t want me to setup an offline account, telling me what a ’limited experience’ it would be. I perservered and got my offline account on my test VM. The whole setup experience is maddening compared to Linux. Plus, Microsoft Edge demanded a setup wizard of its own. I don’t know how people live with this.

After all of this, I installed the qemu guest drivers and SPICE guest drivers and restarted the VM, re-enabling SPICE.

Setting Up Proxmox Authentication

We’re eventually going to need to authenticate with the Proxmox API to download the temporary proxy authentication token for SPICE. This requires hardcoding a username/password into the shell script which will launch on boot on the Pi. So, to prevent hardcoding our Proxmox admin password into the Pi’s shell script, we can create a new user just for this purpose and only give it permissions to view the console of our VDI servers. I’m using a single Proxmox host with built-in authentication, so if you’re using a more complex authentication method like AD or LDAP, you’ll have to figure this out on your own.

First, I created a new role (‘VDIViewer’) which only has access to VM.Console. This means our user can only view the console and nothing else. Viewing the console is quite powerful for the VM, but has no access to the host. You might also want to give this group VM.CDROM and VM.PowerMgmt permissions so they can insert their own virtual CDs and restart their VM when it dies or if they accidentally power it off, but that’s beyond the scope of this example.

Next, I created a new user in the pve (Proxmox VE Authentication Server) realm, and gave it a password.

Finally, I go to the VM(s) I want to let it access, scroll down to Permissions, and add the user to the VM with role VDIViewer.

If I had a larger number of VMs or a pool of VMs and VDI users, I could also make a user group and pool, add all of the VMs to the pool, and add the user group with the role VDIViewer. Since I only have one of each, just adding the user directly to the VM is easiest.

Setup the Raspberry Pi

Raspberry Pi OS Lite Setup

All I had free at this time was a Raspberry Pi Model B+ rev 1.2 (the original single core Pi, upgraded to the 40-pin header). Hopefully it can handle SPICE. I downloaded and installed the lite version of Raspberry Pi OS Bullseye (NOT desktop) and imaged a new SD card. I also enabled SSH by creating an empty file named ‘ssh’ in the boot partition. After this, the Pi booted up and was ready for me to start.

As with any new Pi, we need to run raspi-config and do all of the usual new Pi setup

sudo raspi-config

The important options here are

System Options -> Hostname, set it to something unique (I used ‘vdiclient’ for this example)
System Options -> Password, change it from ‘raspberry’
System Options -> Boot / Auto Login, set to Console Autologin
System Options -> Network at Boot, set to Yes so it won’t login until network is available (since we kinda need that to be a thin client)
Localization Options -> Timezone, set to the right time zone

Once finished, reboot like it asks. It should now auto-login on the physical console. If you need to use Wifi, you can also configure that here. I use wired Ethernet.

And, of course, we need to do updates!

sudo apt update
sudo apt upgrade

Minimal GUI Dependencies

Since I want to minimize the amount of software I install on this poor Pi, I’ve skipped the desktop environment and just have a console. However, I still need a little tiny bit of graphical environment to run the SPICE client. So, I’m going to install an X server and window manager, but skip the desktop environment and login manager. This means I will not have a graphical desktop (no desktop environment), and I will need to launch in to the graphical environment from an already logged in terminal (no login manager), but I can still launch programs which expect a working X session.

So, now I need to install all of this

#Install the X server and Openbox window manager
sudo apt install xserver-xorg x11-xserver-utils xinit openbox

We must be patient, but apt will take care of us. Then, we can continue on.

SPICE Client for the Pi

I now need to install the SPICE client and ideally test it out before setting it up to auto-run on boot. The SPICE client is ‘remote-viewer’, part of the ‘virt-viewer’ package, which itself is part of the ‘virt-manager’ project. In this case, we can install it from the Debian repo:

sudo apt install virt-viewer

Again, more patience. Unfortunately, we can’t test it yet since we don’t have a functional graphical environment, and we also don’t have a SPICE server to connect to.

Proxmox SPICE Proxy

After the viewer is installed, we need to get a configuration file for the SPICE client. Proxmox generates these, but it uses a temporary auth token with a limited lifetime, so we need to download a new configuration file each time we launch the remote viewer. Proxmox has a script available for this which we will use and modify. We could just run the script as-is, but without an x environment running at this point, it won’t work. However, I did test the script as-is on a graphical version of Raspbian and it did work fine, so I trust it.

I made a modified version of this script which hardcodes everything that we need. We just need to call this new script when the graphical environment is ready.

nano thinclient.sh

#!/bin/bash
set -e

# Set auth options
PASSWORD='vdiuser'
USERNAME='vdiuser@pve'

# Set VM ID
VMID="100"

# Set Node
# This must either be a DNS address or name of the node in the cluster
NODE="pvehost"

# Proxy equals node if node is a DNS address
# Otherwise, you need to set the IP address of the node here
PROXY="$NODE"

#The rest of the script from Proxmox
NODE="${NODE%%\.*}"

DATA="$(curl -f -s -S -k --data-urlencode "username=$USERNAME" --data-urlencode "password=$PASSWORD" "https://$PROXY:8006/api2/json/access/ticket")"

echo "AUTH OK"

TICKET="${DATA//\"/}"
TICKET="${TICKET##*ticket:}"
TICKET="${TICKET%%,*}"
TICKET="${TICKET%%\}*}"

CSRF="${DATA//\"/}"
CSRF="${CSRF##*CSRFPreventionToken:}"
CSRF="${CSRF%%,*}"
CSRF="${CSRF%%\}*}"

curl -f -s -S -k -b "PVEAuthCookie=$TICKET" -H "CSRFPreventionToken: $CSRF" "https://$PROXY:8006/api2/spiceconfig/nodes/$NODE/qemu/$VMID/spiceproxy" -d "proxy=$PROXY" > spiceproxy

#Launch remote-viewer with spiceproxy file, in kiosk mode, quit on disconnect
#The run loop will get a new ticket and launch us again if we disconnect
exec remote-viewer -k --kiosk-quit on-disconnect spiceproxy

And of course, don’t forget to make it executable

chmod +x thinclient.sh

At this point, we are almost ready to test, but we still can’t just launch the script without a running X server, so we need to configure Openbox (our window manager)

Openbox Window Manager

Openbox is the window manager I’ve installed, so we need to create a startup script for it to run which will launch our one graphical program.

sudo nano /etc/xdg/openbox/autostart

Replace all of the contents with the new startup script:

#Allow exit of X server with ctrl+alt+backspace
#If you don't want to let the user terminate/restart, leave this out
#You can always `killall xinit` via SSH to return to a terminal
setxkbmap -option terminate:ctrl_alt_bksp

#Start the shell script we already wrote in our home directory
#Runloop restarts the thin client (new access token, new config file)
#if the session is terminated (i.e the VM is inaccessible or restarts)
#User will see a black screen with a cursor during this process
while true
do
    ~/thinclient.sh
done

And, the final moment we’ve all been waiting for, from the physical terminal (not the SSH one), start the X server:

startx --

Start On Boot

The final task is to make startx run on boot (specifically, when our console user logs in to tty1). This one is pretty simple. We need to edit the bash profile to run startx on the first console only

nano .bash_profile

[[ -z $DISPLAY && $XDG_VTNR -eq 1 ]] && startx --

Try rebooting to see if it works

sudo reboot

After all this, we have a working SPICE thin client running on a Raspberry Pi, with functional video, keyboard, mouse, and speakers.

Conclusions

I tested this with both my Ubuntu VM and Windows 10 VM and it worked correctly in both cases. The Pi 1 is far too slow to be usable with Windows 10, stuttering through all of the menu animations. Ubuntu was better, but still not a daily driver experience. It’s still better than using the GUI on the Pi 1 itself. I’d imagine a newer Pi would work much better, but I’ve used all of my good ones for projects and am waiting on a few on backorder, so more testing will come in the future once I can get ahold of more Pi’s.

While filming the video for this blog post, I was able to swap out a Pi 2 from another project and test the setup with the quad-core version. It’s still not nearly as powerful as the Pi 4, but it makes general desktop usage completely usable on both Windows 10 and Ubuntu, although video playback and animations still struggle. Still, depending on your application, a Pi 3 or Pi 4 for each node in a computer lab or a minimal setup for kids homework is very low cost and perfectly functional. The SPICE protocol on capable client hardware is capable of really anything except low latency gaming, so the real limitation is in the Pi 1 and 2’s limited CPU/GPU power to deal with the screen video stream.

I found that, with the Pi 2, the Ubuntu desktop experience was excellent aside from streaming video, but Windows desktop animations and especially the login screen would occasionally cause major stuttering in display updates. It’s also very possible that this is a Windows issue due to running without hardware graphics acceleration, and not a problem with the thin client at all.

I didn’t mention this in either the article or the video, but the SPICE protocol is also capable of USB forwarding, although there are some setup hoops on the Windows client. I didn’t test this yet, so it’s a topic for a future project. Also, this setup should support audio, but the minimal Raspbian installation on the Pi doesn’t, so audio on the Pi is also a topic for a future project.

Logging Gateway Statistics from OPNsense to InfluxDB using Node-Red

Thu, 06 Jan 2022 01:42:05 -0900

I log data from my OPNsense firewall using Telegfaf, but there are some statistics in OPNsense which I’d like to keep track of which aren’t available to be pushed, but are accessible via The API. In particular, I want to keep track of the gateway statistics.

Gateway stats come from continuously pinging the gateway, and are used to determine if a gateway is available for routing. Normally, gateway monitoring is used with multi-WAN setups to remove a gateway from a load-balancing group, or fail over to the next tier in a failover group (or both). However, even with a single WAN gateway, gateway monitoring can be useful to keep statistics on the reliability of the ISP, or notify on failure. I’ve found that my cable modem periodically locks up and rebooting it fixes the problem, so looking for the gateway error rate and gateway status is a good indication that the modem should be rebooted. This flow handles any number of gateways, so you can still use it to keep track of multiple WAN or outgoing VPN gateways, I just don’t need it for that.

I initially set this up before I was using Home Assistant, so I implemented it entirely in node-red and push the results to InfluxDB. While it might be possible to implement this in Home Assistant using the RESTful sensor, OPNsense returns the data as an array for all gateways and parsing this data is far easier in Javascript than Jinja. By pushing the data to InfluxDB first, you can query it in Home Assistant if you want to use it for automations.

The API endpoint that I use is /api/routes/gateway/status, and it returns an array with the status of each gateway, containing the following information which I capture:

Name of the gateway (from OPNsense)
Address of the gateway
Status, either ’none’ (no errors), ’loss’, or ‘down’
Delay (ms) ping to the gateway
Std. Dev (ms) of the ping to the gateway

To use the flow, you should create an API user in OPNsense. It does not require any plugins in OPNsense, but does require node-red-contrib-influxdb in node-red. As always, my work is licensed CC-BY-SA unless otherwise specified. Link to the flow

Selecting Policy Routes in OPNsense from Home Assistant

Sat, 01 Jan 2022 13:44:05 -0400

Policy-based routing has a lot of applications in a home lab or home network. It can be used to change the route taken based on the source or destination, and this can be used in multi-WAN or VPN applications to selectively choose traffic to send over one WAN or which traffic to route over a VPN. It can also be used to selectively drop traffic instead of routing it. Having the ability to change some of these routes from Home Assistant makes it possible to control some of these functions from the HA app or via automations. One possible use case is to drop traffic from a client entirely when selected, as a rudimentary parental filter. Another, more complicated example of this is routing a media player through a VPN service to avoid geo-restrictions - the ability to enable/disable the VPN tunnel or change which VPN interface is in use, for a specific client, is something that might be useful to allow users to control via the HA app. While I use VPNs in my configuration, all of this would apply equally well to multi-WAN routing.

As a bit of a warning, everything here assumes the device you are controlling has either a known IP address (v4/v6) or a known MAC, and the device does not intentionally change its IPs or MAC to avoid the firewall rules. For MAC based entries, OPNsense will periodically scan the ARP/NDP cache for all known IP addresses and use those in the firewall rules, so it’s possible for traffic to ’leak’ if the host in question changes its IPv6 (e.g. due to using a temporary privacy address). However, relying on the NDP cache is the only way to identify IPv6 SLAAC client addresses.

IPv6 Quirks

For IPv6, you also have to deal with the potential for devices to have a globally routable IPv6 address using the prefix from the wrong interface (assuming you are getting a prefix delegation from each of your upstream WAN/VPN interfaces). This probably means you will need to rely on NPT (network prefix translation) for all but the primary WAN, or NAT66 if you are using a VPN interface which provides a /128 address instead of a prefix. Clients expect to see a 2000::/3 address as ‘global’, so using a ULA range for your network and prefix translating or NATing all of the interfaces will result in many clients preferring IPv4 over IPv6 since they don’t believe an FC00::/7 address is routable to the Internet. Hence, you need to pick one of the interfaces which has a global prefix and use its prefix for your internal network, then translate or NAT all of the other interfaces to that prefix. Not great, but for homelabs / home networks where we don’t buy our own address space and peer using BGP, it’s the best we can do.

Setting Up OPNsense for API Control

OPNsense has an API which we can use. They have some documentation on using the API, and large tables of the API endpoints. In particular, we are interested in the Firewall API. However, there isn’t a good way to control firewall rules via the API. The best we have is control of aliases. Thankfully, this is very powerful, we can (manually, not via API) write a rule which forwards traffic in a named alias over the VPN route, and then selectively add/remove the host in question from the alias in response to the command from Home Assistant. If the host isn’t in the alias, then the traffic won’t hit the VPN forwarding rule and will instead fall through to other rules or the default gateway. Great!

The basic logic of the aliases I am using is as follows:

Alias is created with the name of both the device and the destination. Alias contains the MAC address of the device.
Alias is created with the name of the destination only. Alias contains all of the other aliases which could be routed to this destination.
Firewall rules (IPv4/IPv6) are created for each destination, reading the alias for their respective destination
API calls will enable/disable each alias created in #1, causing the alias in #2 and rule in #3 to include or not include the IPs corresponding to the MAC of the device.

Now for a graphical tutorial on how to set the OPNsense side of all of this:

Create an API user and add it to your secrets.yaml file

Follow the directions in OPNsense’s Using The API docs to create a new user for the API, and generate API keys for it. Add the API key and secret as two entries in your home assistant secrets.yaml.

Setup the Device MAC Alias

Create a new alias, type is MAC, give it the MAC address of the device in question. OPNsense will periodically scan the ARP (IPv4) and NDP (IPv6) caches to determine what the most recent IP addresses of the device are. Once this is created, we can use the name of this alias later to reference this device. Since we enable/disable this alias to control routing, we need one of these for each destination, all with the same contents.

Setup the Destination alias

Create a new alias, type is Host(s), enter a descrpition, and add all of the aliases of hosts which can be routed to this destination. This will be referenced in the firewall rules later.

Setup the Destination routing rules

Create two new firewall rules, one for IPv4 and one for IPv6, which route any traffic from the destination alias to the gateway of the destination interface. We need two rules here since IPv4 and IPv6 traffic will have separate gateways. All of this assumes you already have your desired outbound gateways setup and working properly on its own. I have an IPv4 example below, but you’d need one for each to select the correct IPv4/IPv6 gateway.

If you are dropping traffic instead of routing it, you can create a single rule for both IPv4 and IPv6, and set it to Block instead of Pass.

Home Assistant Configuration

Now we need to configure Home Assistant to enable/disable our aliases.

Determine the UUID of the aliases we created earlier

The easiest way to do this is to backup the configuration in OPNsense, open the resulting XML file in a text editor, and search for the name you gave the alias. You should find an alias and it’s corresponding UUID nearby. We will need the UUID for all of the device->destination aliases for our automations.

Create RESTful Commands in Home Assistant

In our case, we need to access two API endpoints - one to enable/disable the alias, and another to reconfigure (which also updates firewall rules if required). The first command takes the UUID and boolean as parameters, the second takes no parameters.

Add this wherever you put your rest_commands section, either in configuration.yaml or a file you’ve included below it:

#Rest Commands
rest_command:
    firewall_alias_reconfigure:
        url: "https://192.168.1.1/api/firewall/alias/reconfigure"
        username: !secret firewall_key
        password: !secret firweall_secret
        method: "post"
        verify_ssl: false
    firwall_alias_enable:
        url: >
            {% if enable %}
            https://192.168.1.1/api/firewall/alias/toggleItem/{{uuid}}/1
            {% else %}
            https://192.168.1.1/api/firewall/alias/toggleItem/{{uuid}}/0
            {% endif %}            
        username: !secret firewall_key
        password: !secret firweall_secret        
        method: "post"
        verify_ssl: false

Create Input Boolean and associated Automation

Finally, we can create an Input Boolean to select if my laptop should be tunneled over the VPN or not, and an Automation which runs when the Input Boolean changes. The Automation needs to call the enable service with the uuid of the rule we want to enable, the new value of the Input Boolean, and then call the reconfigure service.

And, the automation:

alias: Laptop VPN Route
description: ''
trigger:
  - platform: state
    entity_id: input_boolean.laptop_vpn_route
condition: []
action:
  - choose:
      - conditions:
          - condition: state
            entity_id: input_boolean.laptop_vpn_route
            state: 'on'
        sequence:
          - service: rest_command.firwall_alias_enable
            data:
              uuid: 3ae2128e-9990-40a2-841e-8b5b6f7b4c92
              enable: true
    default:
      - service: rest_command.firwall_alias_enable
        data:
          uuid: 3ae2128e-9990-40a2-841e-8b5b6f7b4c92
          enable: false
  - service: rest_command.firewall_alias_reconfigure
mode: restart

What About DNS?

All of this routes the traffic using policy routing, but DNS isn’t traffic being routed through the firewall, it’s traffic that ends at the DNS resolver (Unbound) whithin the firewall. So, we end up with a less than ideal situation where everything on the network is still using the same DNS resolver, even if the traffic takes a different path.

There are a few ways to handle this:

Use Unbound as a DNS forwarder using DNS over TLS (DoT) and rely entirely on that (so all DNS goes directly over the default gateway, but via TLS to a provider like Quad9). This means the DNS requests don’t go to the VPN provider at all, but are still separately protected using TLS on their trip to Quad9.
Use Unbound or Dnsmasq as a forwarder, and rely entirely on the DNS servers provided by the VPN provider, so all clients use the VPN for DNS via the forwarder.

In addition, you should create a rule ahead of the policy routing rules which specifies that traffic where the destination is ‘This Firewall’ pass, and to ‘Apply action immediately on match’ to prevent it from trying to route DNS traffic incorrectly. OPNsense already does this for DHCP, but not for DNS.

Expanding This to Other Applications

So, now that we can enable/disable firewall rules, here are a few ideas you can take away from this:

Selectively block a list of devices from accessing the internet, by enabling/disabling the alias used by a block rule, using any logic you want (parental control type rules)
Manually select to use a secondary WAN interface in a similar way to selecting the VPN interface
Use an input select instead of input boolean to choose one of many routing options

Christmas Maze Boxes

Fri, 24 Dec 2021 18:06:32 -0400

This Christmas, I printed a whole series of maze boxes as gifts for family members. After finding a gift maze model on Thingiverse, I felt like there must be a better solution to programatically generate mazes of varying difficulty, and so each person who gets one has a unique experience. Before writing such a program myself, I checked to see if it had already been done, and sure enough it had. In this case, Adrian Kennard had written a C program to generate puzzle mazes, and hosts an interactive version on his website. His tool generates OpenSCAD code, which can then be used to generate a surface model to 3D print.

The Workflow

My workflow for this requires a few tools, all open source:

Adrian’s Interactive Puzzle Box Generator to generate the maze
OpenSCAD to generate a surface model from the solid body
PrusaSlicer (or a slicer compatible with your printer) to generate G-code for the printer

Adrian's Web Version, Click to Visit

Generating the Puzzles

Adrian’s puzzle box generator is used to generate an OpenSCAD model of the maze, including all of the parts. I used mostly default parameters, but there are a few I changed:

m (total parts to make) = 2 (outer + inner) for most people, or 3 (outer + middle + inner) for people I really wanted to challenge - printing fewer parts drastically reduced my workload later on, since printing the default 4 parts in separate colors for each person would be a huge pain. That said, I did one maze with the total parts set to 4 and it was a very fun challenge for the recipient.
c (core diameter for content) = 15mm - I found that 10mm default was too small for big American fingers to fit inside to get the goodies out
h (core height for content) = 75mm - I wanted to be able to fit $1 bills inside, so it needed to be big enough for USD
X (maze complexity) = varied from +-10 uniquely for each person - I tailored the difficulty to what I thought they could handle. Most family got a difficulty of 0 (average), close friends got 5-6 (I want to challenge them). I didn’t make anyone’s more difficult than +6, and I had to solve one or two of them for reasons I’ll describe later and the +6 was quite at least a half hour challenge for me.
s (number of outer sides) = 6 - The default is 7, but the default is to have a maze with 3 nubs (i.e. the maze is triplicated around the perimeter of the shape). Since 3 is not divisible by 7 (both are prime), this means there is only one way to assemble the maze where the septagons will line up when finished, and two ways to do it wrong. This caused me to assemble one of the first mazes incorrectly, and have to solve it completely, rotate it one nub over, and close it, etc. until I realized that I should change the number of outer sides to 6 for future mazes so there’s no way to assemble it without the hexagons lining up.
S (text on sides) = the person’s name
T (text scaling) reduced from 1 if their name didn’t fit
O (text outset / embossed) set randomly, so some are embossed and some are raised

Rendering in OpenSCAD

Once the scad files were generated for every person’s unique maze, I had to render them to a format which could be sliced. OpenSCAD is a tool which uses code to generate solid bodies. OpenSCAD is a sort of scripting / programming language, so it’s very easy to generate parametric solid models using the code. The tool itself then renders the solid model, and can convert it to other solid and surface formats. Once you open the scad file from the puzzle box generator in OpenSCAD, you need to render it (Design -> Render), and then export the render (File -> Export -> Export as) in the format of your choice (AMF/3MF are both good choices).

Slicing in PrusaSlicer

I have a Prusa printer, so I use PrusaSlicer to generate the gcode for the printer. However, the process so far has led to a single file which contains multiple parts, and I’d like to print each part separately (without an MMU). This means I need to split the part into multiple objects, and then generate a gcode file for each one separately.

It’s very easy to do in PrusaSlicer, thankfully. Right click on the object and click Split -> To Objects. This will take the multi-object single object and turn it into multiple objects. Now, I just disable them one at a time and generate gcode with one part of the print visible. Since I’m planning on printing these in a huge variety of colors, I didn’t bother trying to match up all of the prints of one color onto the bed, I just printed each part individually. So, now I have around 50 gcode files to print, each one taking under 2 hours (with the 0.3mm DRAFT profile, since it’s perfectly adequate for the level of detail in these mazes).

Final Assembly

After a print was done, I put it in a bag with the recipient’s name on it. I kept all of these in a bin and would print 1-4 pieces per day, depending on how I was feeling and if I had anything else to run on the printer, batching them by color. It took roughly a month to get them all done, but of course the printer wasn’t running full time, I printed other things, and I was also sick for a week and didn’t print anything.

For each maze, I had to stuff the maze with a $1 bill and a note, then solve the maze in reverse to assemble it. Thankfully, when you have the puzzle box open, you can examine the maze and find the correct path to close it, whereas when solving it without knowledge of the whole maze is far more difficult. Unfortunately, there were a few times where I had to solve the maze to open it again, due to the septagon/hexagon problem described above, but this was fairly rare. However, I definitely have an appreciation for the difficulty of solving this, especially if you aren’t expecting it and aren’t entirely sure how it works.

Holiday Lighting with Sonoff, Tasmota, and Home Assistant

Sat, 18 Dec 2021 11:00:00 +0000

Every year, the exterior holiday lighting gets reluctantly set up. It’s part of McMansion life, a requirement to appear as though you’ve made an attempt to decorate for the season. Between the colored optical projectors, to strings of lights haphazardly strung around the front porch, it all ends up needing to be plugged in and turned on/off. For years, the solution to this was a mechanical timer, with on and off markers which could be inserted around the ring to turn the timer on and off at the right time. This was all fine and good, but with little precisoin, no adjustability for the seasons or special events, and the inevitable time error if there is ever a power outage. But! No more! We can fix all of this with Home Assistant.

Sonoff Smart Plugs

Any smart plug will work for this, as long as it works with Home Assistant. I have a Sonoff S26 which is flashed with Tasmota (see Tasmota Day for instructions), and this allows me to keep all of the control on my local network (no cloud dependencies) and trust in the integrity of the firmware. Tasmota speaks MQTT natively, and I will add it as an MQTT switch in my switch.yaml file. Note that there is a Tasmota integration which you can use directly, but my Sonoff’s are still setup with the MQTT topic structure from when I used Node-Red entirely, so I’m doing it this way.

MQTT config:

  #Holiday lighting switch via Sonoff relay with Tasmota
  - platform: mqtt
    name: Holiday Lights
    unique_id: holiday_lights
    #Command to switch
    command_topic: "raw/tasmota_B70C5D/cmnd/POWER"
    payload_on: "ON"
    payload_off: "OFF"
    retain: true
    #Availability of switch
    availability:
      - topic: "raw/tasmota_B70C5D/tele/LWT"
        payload_available: "Online"
        payload_not_available: "Offline"
    #Status of switch
    state_topic: "raw/tasmota_B70C5D/stat/POWER"
    state_on: "ON"
    state_off: "OFF"

Sun Based Timers

I started with a simple sun based timer. I used a single automation instead of an on + off automation to reduce clutter in my Automations page, but it has a trigger for each event instead of a delay to turn the lights off (which could fail due to server restarts duringt the delay time). I decided that 30 minutes after the sun is below the horizon was a good time to turn the lights on, and 4 hours after that is a good time to turn them off. As the seasons progress, this will gradually turn the lights on earlier. Because Daylight Savings Time is the the worst, it’s basically dark out the entire time I’m at home, so I need many hours of holiday lights to cheer me up.

If I have an event, I can always turn off the automation and manually control the lights. If I setup a more complex lighting system in the future, you followers will be the first to know.

Here is the automation I setup, in YAML form. You should be able to copy it in as YAML and then switch back to the visual editor without issues. I like using a single automation with multiple triggers for both on and off, since it reduces the visual clutter in the automations editor, but really this could have been done with two simple automations instead.

alias: Holiday Light Timer
description: ''
trigger:
  - platform: sun
    event: sunset
    id: 'On'
    offset: '+30:00'
  - platform: sun
    event: sunset
    id: 'Off'
    offset: '+4:30:00'
condition: []
action:
  - choose:
      - conditions:
          - condition: trigger
            id: 'On'
        sequence:
          - service: switch.turn_on
            target:
              entity_id: switch.holiday_lights
    default:
      - service: switch.turn_off
        target:
          entity_id: switch.holiday_lights
mode: single

Camera Bracket for the CR-10 MAX

Mon, 13 Dec 2021 13:00:00 +0000

As I recently posted about, I got my CR-10 MAX working and did some basic upgrades to it. After that, it was printing pretty well, so I decided it needed a camera so I could start trusting it with longer unattended prints. I still don’t have power control of the printer, so I won’t let it run when I’m not home, but now I feel comfortable leaving the basement knowing I can check to see if it’s making a mess of itself.

I used the same camera as my Prusa, because I had already bought a second one and it works well enough for what I’m doing. It’s PoE, and with two Raspberry Pi’s and two PoE cameras, I had to get a little Netgear 5 port PoE switch for my 3D printer table since there are only two jacks on the wall.

Mounting Bracket

I’m not making an enclosure for the CR-10, so I need to mount the camera directly to the printer. Since the CR-10 is made of T-slot extrusion, I decided to make a part that fits into the slot snugly and mounts the camera. The CAD of this part is pretty simple. I found a dimensional drawing of the 2020 style extrusion used by the printer (so named for it’s 20mm x 20mm dimension), and replicated the features of the slot as closely as possible, then printed a test piece. As expected, it didn’t fit in the slot, so I reduced the dimensions down slightly until it was a very snug fit. The top of the bracket has a hole which is a bit larger than the tap size for a #8-32 UNF fastener, and I tighten the fastener through the camera bracket without using a nut (purely threading into the plastic). This works well enough for the little camera on the little bracket.

The Final Result

The Project Files and Parts List

Here are all of the files and parts required to replicate this small project. As usual, all design files are licensed Creative Commons CC-BY-SA unless otherwise noted.

Printer Mount Bracket (AMF Format)
Printer Mount Bracket (FreeCAD Original)
JIENUO 5MP Mini Panoramic Camera POE (affiliate link) Some links to products may be affiliate links, which may earn a commission for me.

Getting my CR-10 MAX Printing Again

Tue, 07 Dec 2021 19:11:00 +0000

I’ve had a Creality CR-10 MAX for about a year and a half at this point. I bought it on sale in early 2020 when I had big project ideas that wouldn’t fit on my workhorse Prusa i3 MK3S, so it’s fair to say I bought it for the large size for the price point and not the build quality or feature set. However, I’ve never really liked it, so now that the Prusa is working really well with Octoprint, the box, the cameras, and all of the other projects I’ve documented here, it’s time to move on to the CR-10 and get that one working well too.

While I’ve had the CR-10 for awhile, I’ve only used it for prints where I was running things in bulk and needed the larger bed to make duplicates at the same time. The stock bed on the CR-10 is a rough powedercoated aluminum, which is extremely sticky for prints, leading to usually good first layer adhesion but a really hard time getting prints off. I really miss the flexible PEI bed of the Prusa. The touchscreen is a nice feature compared to the scroll wheel and character LCD of the Prusa and lower-end Creality machines, but it’s not very intuitive and doesn’t have access to nearly as many settings as the Prusa menu. The translation can also be rough sometimes. Finally, the machine is also limited to about the same flow rates as the Prusa, so there is no speed advantage to the printer, hence I will always prefer the Prusa for prints which don’t need the larger bed. Thanks to all of these things, I really only use the CR-10 once every few months.

Naming the printer

The current Prusa printer is named ‘Columbia’, after the first Space Shuttle to fly to space (OV-102). This follows with the space-themed naming of many of my other servers and devices. I normally use manned vehicles for ‘manned’ computers (desktops and laptops), and while the Space Shuttle is a manned vehicle and a 3D printer really isn’t, the Space Shuttle is more notable for being the first space-based construction equipment, building the ISS, servicing the Hubble Space Telescope, etc.. and as such, it’s name is fitting for a 3D printer.

Following Columbia, the second Shuttle to fly to space was Challenger. Hence, the CR-10 is named Challenger.

I’m aware that both of these shuttles were destroyed in accidents, but they were the first two, so this is what I’m sticking with.

New PEI Magnetic Bed

To fix my first gripe, I bought a flexible magnetic PEI bed from Fulament. I bought the Fula-Flex 2.0 in 470x470mm size, as recommended for the CR-10 MAX (it’s a big boi). I had to trim the magnetic sheet slightly around metal clips which hold the aluminum bed to the bed structure, but assembly was very easy. Once I had the magnetic sheet on, the PEI sheet snapped in place easily (it’s way way more magnetic than the Prusa bed… it’s difficult to pull it off). Overall, I’m happy with the bed. But I still really struggled to get the printer leveled.

The CR-10 MAX includes a BLtouch Z-height sensor standard, while most Creality printers rely on hopes and dreams (or manual Z-leveling) to get a decent first layer height. The BLtouch is fairly similar in concept to Prusa’s PINDA probe, except it physically touches the bed with a little needle that pops out when probing. You still need to dial in a Z offset for the machine to account for the height difference between the probe and nozzle, but this process was very painful with Creality’s instructions. I’ve always settled for adjusting the bed knobs to get the bed leveled side to side, something that shouldn’t be necessary with multi-point calibration.

New Firmware

The next step in my journey is to find a better firmware build for the CR-10 MAX. A vast majority of consumer 3D printers use Marlin firmware, which is open-source and requires distribution of the source code changes, which Creality has done. In theory, I could take this, reconfigure it as I wish, compile it, and program the mainboard. However, the CR-10 MAX is a bit special in that it also has a graphic LCD which is separate from the mainboard. Creality has made the graphic files for this available too, so you can modify the screen graphics if you make corresponding changes to the firmware.

I looked around for the best CR-10 alternative firmware and ended up deciding on Tiny Machines 3D’s firmware. Unfortunately their site is a bit of a mess with guides hosted on Google Docs (ouch), but I was able to flash both their screen files (DWIN_SET) and mainboard hex file and found that the printer was slightly more pleasant to use. The Tiny Machines firmware has much better instructions for bed leveling, what to add to the start G-code to actually use bed leveling data, and does a larger grid for bed leveling (5x5 instead of 4x4). After working through the full bed leveling guide from them (something the CR-10 didn’t really come with), I found that the printer worked much better than before. I think the key is that my startup G-code was missing the M420 instruction, which tells Marlin to actually use stored bed leveling data (after G28 machine homing, bed leveling is not used until G29 commands bed leveling or M420 commands use of stored bed leveling data). Since the bed is never perfectly level, and especially the giant bed of the CR-10 MAX, doing the 5x5 leveling calibration and then actually using the data is important. The instructions were also very clear in how to do the ‘AUX leveling’, something that was not super clear in the Creality instructions but is actually very important to getting a good nozzle offset for the BLtouch probe.

The new firmware also makes a lot less noise, resorting to basic hissing instead of all of the low quality sounds the Creality firmware would play. Neither sound is particularly pleasant, but the hisses are shorter.

More Octoprint

Of course, I need another Raspberry Pi for Octoprint for this printer. Since the Raspberry Pi 4 has been backordered for months (I have an order in…), I decided to start with a Pi 2 that I already owned and will see how it goes. I know the Pi Zero is strongly not recommended for OctoPrint, and the Pi 2 has a step up in processor (4 cores vs 1 core), although it’s not as powerful as the Pi 3 and nowhere close to the Pi 4. I got a basic install setup, mounted everything as an autofs network share (this time using symbolic links in the ~/.octoprint folder instead of remapping the folders within Octoprint), configured the printer volume and such, and had it working pretty quickly.

There are some quirks. The CR-10 doesn’t have isolation on the USB port (at all, apparently), so the LCD will turn on and try to boot up just by plugging in USB to the Pi with the main power to the printer off. So, once I start turning the machine one and off automatically, I’ll have to do something about that.

Future Improvements

Now that the printer is working as it should, I have a few enhancements planned:

A case for the Pi
Sonoff S31 power control and monitoring
Updating all of my Home Assistant notifications to correctly work for both printers
New camera mount for the wide-angle camera
New NozzleCam for the CR-10, which will need a much more clever mount design than the Prusa did
LED lighting down the diagnoal bracing of the printer, since I’m not building an enclosure for this one

Introducing StorageTags

Fri, 03 Dec 2021 08:07:32 -0400

As you might recall from my last blog post, I’ve been playing with workshop organization using fiducial markers to locate storage boxes. I’ve started to move past the proof of concept code phase into the slightly functional open-source project phase, and I created a repository and developed some working code. The project is just barely functional at this phase, but it has the ability to connect to cameras over RTSP, skip images to reduce the detection framerate, decode them using one of the AprilTag types, and publish the resulting detection data to MQTT. Conceptually, this is all that’s required if you just want to get a bunch of tag ID and coordinate data from camera(s) into MQTT to deal with yourself.

I’m not a great frontend developer (I have basically no experience beyond HTML), so some assistance in that portion of the project would be great. I’m slowly churning through the backend (REST APIs, database code, …) but the UI is nonexistant. My goal is to eventually be able to view a list of each box in the system, sort by name / category / shelf, and find its location (or lack of known location).

Until then, feel free to follow my progress on both the Project Page as well as the Repository.

Also, if you’re interested in contributing but aren’t a programmer, I’m open to suggestions to new names, a graphical theme, and logo for the project as well.

Fiducial Markers for Physical Storage Tracking

Thu, 25 Nov 2021 08:06:32 -0400

Previously, I saw (and can no longer find the source) a vlog about a maker who stored all of his projects and stock of parts in bins over his long work bench. All of the bins were labeled with some sort of code, and a security camera in his office would periodically scan for codes in the image and keep track of the location of each bin. With this system, he could put any bin in any empty spot on the shelf and easily find it later, or find that it was not on the shelf, from the comfort of his desk computer. I sought to recreate this, for my own maker space.

I tried a few different approaches involving QR codes and fiducial marking, and at this point I’ve settled on AprilTags based on testing I’ve performed (See this blog post for the testing). This project page will catalogue all of the work I’ve done to make this a reality.

Initial Missteps with QR and 2D Barcodes

My naive approach was to try and encode the name of the box in a QR code, then have the camera read the code and publish the location of the box to MQTT. From there, I could centrlally process all of the MQTT topics from all of the cameras (gotta think about scalability), and make a simple inventory of each shelf and reverse inventory of each box, listing the shelf and position. If I could encode the name of the box (or something short enough for me to encode and long enough to recognize), I wouldn’t need any sort of database to correlate box IDs to metadata.

I initially started by researching the greater space of 2D matrix barcodes. The most commonly known is the QR code (‘Quick Response’), which is widely used to encode all kinds of information to be read primarily by mobile phones. Other codes I investigated include the microQR, Aztec code, and Data Matrix codes. I used a generator to generate codes in a variety of types for a 10 character string, and printed them out in two sizes - 0.9" square and 2.5" square. 0.9" square happens to be the largest size I can print on my label printer, and 2.5" seemed about right to use premade address label stickers which are easy to print.

I taped this paper on the wall above my desk, and tried a few different cameras before analyzing the images using OpenCV. This particular image was taken by the same model of camera as my 3D Printer Wide Angle Camera, which was a candidate to become the makerspace inventory camera.

This approach did not go well. With all images the same dimensions, the number of pixels in the image has a strong impact on the readability from the same distance. The MicroQR is only 13x13, while the Aztec is 15x15 and the regular QR is 21x21. This seems small as far as QR codes (and a 10 character string is really very little data), so it just gets worse from here. I was never able to decode any of the 0.9" codes with any camera at any reasonable distance, so I gave up on printing a barcode as part of the 24mm label maker label.

With the larger images, the MicroQR seemed promising visually, but I couldn’t find any open source library to decode it, so we will never know how well it would have performed. I found websites able to decode the Aztec code, but again no open source libraries were able to decode it. The websites seemed to be able to adequately decode the ‘mid’ depth image (shown above), which is actually only about 3’ since the camera has such a wide angle, but without being able to integrate it into my own code the Aztecs approach was another dead end.

That left the QR code that we started with. There are plenty of QR decoders available with OpenCV bindings (including OpenCV’s own built-in decoder, Zbar, and Wechat’s CNN based decoder). Of these, Zbar was the most difficult to install (partially because the project has been abandoned for around a decade and never compiled on macOS) but very notably outperformed the other two, and was also the only one to successfully return an array of codes when multiple codes were in view (the rest just picked the ‘best’ one). Since I really need all of the codes in view, that was very important.

Despite all of this success with Zbar, I was very unimpressed with how close the camera had to be to get working detection. So, I need a new system with lower resolution tags that the camera can resolve from further away.

Fiducial Markers

Wikipedia has a good introduction on fiducial markers for those unfamiliar. Essentially, these are specifically designed patterns which encode a very small amount of information in a way that is designed for machine vision to unambiguously identify the marker. Normally they are used to identify the position and orientation of the marker to locate the image or the marker in space (given a known position of either the marker or the camera), and any coding within the market is just to identify the marker out of a small set in case there are multiple fiducuials in view at once. The advantage of this is the data encoded is extremely low, and thus the images are extremely low resolution, and when properly scaled up they are easily identified by cameras at much further distances. Wikipedia has a comparison between a few fiducial marker packages, and based on the recent development and open-source nature I selected AprilTags as my next path of approach. I wrote a blog post on my initial experimentation with this.

Introducing StorageTags

I decided to create a proper software solution for this. I thought through the architecture and settled on some design goals:

Software should run on a single application server (instead of on the camera / Raspberry Pi)
Cameras should use standard video protocols such as RTSP or HTTP JPEG requests, and converting webcams, PiCams, or capture cards to RTSP/HTTP is not within the scope of this project
Application server should retrieve and process images from all cameras
Application server should maintain state of entire system
Scalability is not believed to exceed the capacity of a single server (as framerates are low and video processing scales well across threads), so separating the processing from state management is not a requirement at this time
Software should have a purely web-based UI (mobile friendly preferred) for users to view the status, box locations, manage boxes, and add/reassign boxes
Software should have REST and MQTT API to retrieve/publish state for use by other applications. Beyond associating tag IDs with boxes, we will not deal with the contents or inventory within the box

I chose to name the project ‘StorageTags’, although I’m not sure if it’s the most unique name, and I reserve the right to change it in the future.

I explain a bit more about the creation of the project on this blog post.

Next Steps

I’ll continue to update this page as I continue to develop the software. I’ve started to move out of the experimentation phase, but the software is far from finished

Playing with Fiducial Markers and AprilTags

Thu, 25 Nov 2021 08:06:32 -0400

Recently, I have been organizing all of the parts and pieces of my partially finished projects so I can go back to projects more easily. This started as a simple goal - keep all of my work in boxes, keep the boxes labeled, and keep the boxes in a place where they are easy to get to. I have mountains of leftover parts I’ve accumulated over the years, and I really need a better solution.

The first solution was simple - buy a bunch of smaller boxes. I primarily had 18 gallon storage totes and shelving designed to hold them, but they are way too big for any project and end up filled with everything. I decided on smaller 7 quart (sorry metric friends) bins with latching lids, as the perfect size for most of the electronics projects I normally work on.

Of course, I could just use my label maker to print labels for each bin, and manually search for them on a shelf. But, I could also take a much cooler approach and design something to track all of the bins for me. Is it more useful than manually reading all of the labels on the shelf? Maybe not, but sometimes the fun of the project is in building it more than using it.

Enter my new project - Storage Tracker

As a result of this, my wall is now plastered with grids of fiducial markers, reminiscent of the early Virtual Reality demos at VALVe where the entire room had giant block codes scattered all over.

Test image from Dahua 4MP (2560x1440) security camera

Test image from Wide Angle 5MP (2560x1920) security camera at a low angle

Initial Findings

I have 4 types of AprilTags here - I printed a sheet of 30x tags for each type (tag16h5, tag25h9, tag36h11, tagStandard52h13). These are roughly 1.5" square not including the keep out area, so these are smaller than the QR codes I tested earlier. I started with only the first 3 types (which are 4x4, 5x5, and 6x6 pixels active area plus a 1px solid border), which support up to 30, 35, and 998 unique codes respectively. I’d ideally like to use the smallest number of pixels possible for the number of boxes I have. Since I’m likely to expand past 30 and 35 boxes, the tag36h11 seems like the minimum for me, but maybe someday one of you will use my (eventual) software to manage a much larger storage system and the tagStandard52h13 with its 57,714 total tags will required for you. Another approach is to use two different tags (i.e. tag16h5 + tag25h9, which would be 30 * 35 or 1050 possibilities), since two tags side by side might fit better on a box than a single larger tag. Also, especially with the smallest tag16h5, the detector often detects blurred versions of larger tags (hence finding ~35 tags when there are only 30 in the image). Ideally you’d stick to only one type of tag in your entire inventory to simplify this. All of this must be considered when choosing the tag system to use for your storage tracker.

I accidentally printed the tagStandard52h13 incorrectly, so the printer cropped off the right most row of pixels, so 6 of the tags shouldn’t read correctly. It’s stil much harder for that tag type to be detected than the other 3 tag types at the distance I’d like to place my camera.

The Test Code

At this point, I’m taking still images from the cameras and running them through a python script to process them and export a new modified image. I’m running this on Ubuntu, so no guarantees about Windows or macOS. You need to install AprilTags and compile it with all of the python options working (it will let you know if configure can’t find python modules it needs, and thus won’t build the python bindings). You also need the pip module opencv-contrib-python to get the OpenCV python bindings.

import cv2
import numpy as np
from apriltag import apriltag

#Draw function
def display(im, detections, color):
    for detect in detections:
        #Extract points from detection data
        points = detect['lb-rb-rt-lt']
        n = len(points)
        # Draw the four lines making up the points
        for j in range(0,n):
            p1 = tuple([int(cell) for cell in points[j]])
            p2 = tuple([int(cell) for cell in points[(j+1)%n]])
            cv2.line(im, p1, p2, color, 3)

#Read in the file as both color (for display) and grayscale (for analysis)
fname = "tags.jpg"
imagecolor = cv2.imread(fname)
image = cv2.imread(fname,cv2.IMREAD_GRAYSCALE)

#Create a detector for each tag type
detector1 = apriltag("tag16h5")
detector2 = apriltag("tag25h9")
detector3 = apriltag("tag36h11")
detector4 = apriltag("tagStandard52h13")

#Run each detector on the grayscale image - comment out the ones you don't expect
detections1 = detector1.detect(image)
detections2 = detector2.detect(image)
detections3 = detector3.detect(image)
detections4 = detector4.detect(image)

#Print the total number of detections - only expect 24 of 52h13 since that one got messed up by the printer
print("Total ",len(detections1)," detections tag16h5 (expect 30)")
print("Total ",len(detections2)," detections tag25h9 (expect 30)")
print("Total ",len(detections3)," detections tag36h11 (expect 30)")
print("Total ",len(detections4)," detections tagStandard52h13 (expect 30, or 24)")

#Add overlays for each of the detection lists in a different color
display(imagecolor,detections1,(255,0,0))
display(imagecolor,detections2,(0,255,0))
display(imagecolor,detections3,(0,0,255))
display(imagecolor,detections4,(255,0,128))

#Write the output file back 
cv2.imwrite("output.jpg",imagecolor)

My goal going foward is to make blog post updates like this instead of massive project pages, since I have a lot of projects that I work on a lot of and keeping this documentation thorough and clearly written helps me keep track of everything I tried.

Setting Up my Creality CR-10 MAX

Tue, 23 Nov 2021 08:06:32 -0400

In this project, I’m going to setup my Creality CR-10 MAX in the same way that I’ve setup my Prusa i3 MK3S. I’ll continue to update this page as I make progress, instead of splitting the project into multiple parts like I did for the Prusa. My goals for the project are as follows:

OctoPi install, printer profile in OctoPrint, remote mount filesystem, and MQTT - Completed Here
PrusaSlicer profile for the CR-10 MAX - Completed, Not Published
Power control and monitoring - Not Started
HomeAssistant integration, including better notifications including which printer is done (currently my notifications assume ‘columbia’ is done) - In Progress
LED status indicators - Not Started
Camera mount for the wide angle camera - Completed Here
Camera mount for the nozzle camera - Not Started
Dealing with the lack of USB power/isolation of the Creality printers (The printer controller and LCD will stay on powered by the USB port) - Not Started

Most notably, I will not be building a box for this printer at this point in time.

The Project Files and Parts List

Here are all of the files and parts required to replicate this project. All of my design files are licensed Creative Commons CC-BY-SA.

Creality CR-10 MAX Printer
Raspberry Pi Model 3 B - I like Adafruit, but the Pi’s are often out of stock, so shop around
Fula-Flex 2.0
Tiny Machines 3D’s Marlin Firmware for CR-10 MAX

Bitscope Server as a Raspberry Pi Systemd Service

Thu, 18 Nov 2021 10:31:00 +0000

Bitscope makes some low-cost USB logic analyers and low bandwidth oscilloscopes. Since I am a computer engineer and primarily work in the digital domain, this is adequate for a lot of my projects, and I’ve had a Bitscope Mini for a few years now. I think there are better options available on the market now that weren’t back when I bought it, but I’m sticking with the tools I already own. To reduce workspace cable clutter with my laptop on the bench, I decided to use a Raspberry Pi to run Bitscope Server. They have a server version of the sofwtare with builds for armv7 linux, but it doesn’t set itself up as a linux systemd service and doesn’t run on boot, and is honestly kinda terrible. Today, I’m going to go through the whole setup process to get it working on a Pi 2B.

Prepare the Pi

Start by flashing an SD card with the latest Raspberry Pi OS, Lite version (no desktop) is fine. I use Etcher for this. After Etcher is done, remove and re-insert the card and a partition called BOOT should show up, as well as a partition that your computer may not be able to read (DON’T let Windows format it, just leave it alone). Create a new file in BOOT called ‘ssh’ with no extension. It doesn’t need any contents, it just has to be there to tell the OS it should enable the SSH server for us (so we don’t need to connect a display and keyboard to get started).

Next we do the usual to get a Pi ready. Find the IP address (I look at my router’s DHCP leases, but it might also show up as raspberrypi.local) and SSH into it. Default username is pi, password is raspberry. We have a few things to do as general housekeeping:

Change the default password
Change the time zone
Change the hostname

All of these are cone with raspi-config

sudo raspi-config

Password and Hostname are both under the first menu, System Options. Set Password to anything you’ll remember. Then go back to the System Options menu to change the hostname. You need to change it from ‘raspberrypi’ so you don’t have issues in the future with a bunch of raspberry pi’s all trying to call themselves raspberrypi.local, preventing you from finding the right one. It’s also good practice to avoid hostname collisions on a network, although it isn’t strictly required. I named mine ‘bitscope’. If you want to enable WiFi on your Pi (I’m using wired Ethernet), you can also set it under System Options

The final change is the time zone. Raspberry Pi OS defaults to UTC time. This isn’t necessarily a problem unless you have an app which depends on the local time, but it’s good practice to change it anyway in case you install something that does care about local time. This option is under Localization. First, choose your geographic area, then any city on the list in the same time zone as you. Once you are done, click Finish, and let it reboot for you.

Install Bitscope Server

Bitscope provides a small help page on Bitscope Server, but it isn’t much. You have to go to the downloads page to get the right .deb file from the Downloads Page. It’s odd that they have no binaries for Generic Linux, but they have i386 and amd64 for Raspbian Debian.. but we need Raspbian Debian and ARMv7. No ARMv8, sorry. Before you go any further, SSH into the Pi again to be ready to download the image.

Once you’re there, check the box for Bitscope Server and click Download. It’ll take you to a page with the download link to the deb. Right click on the link, and copy it. Now, go back to the SSH session and type ‘wget ’ and then paste the link you copied. In my case, I ran this command:

wget http://bitscope.com/download/files/bitscope-server_1.0.FK26A_armhf.deb

Now we need to manually install the deb using dpkg. Adjust this command if your filename is different:

sudo dpkg --install bitscope-server_1.0.FK26A_armhf.deb

It should install. If it gives you errors, you either got the wrong architecture (make sure it’s not i386 or amd64), or you are running a 64-bit OS (ARMv8). The good news is that, even on Pi’s which support ARMv8, Raspberry Pi OS is currently (as of 2021) booting in 32-bit mode by default on all Pi’s.

Finally, we can try to run the server. It outputs nothing unless you add -v for verbose output.

bitscope-server -v

If we run the help, it’s clearly not really designed to run on UNIX based systems at all. It has no systemd service out of the box, it has the option to daemonize (which just makes it disappear from the terminal, we still need to launch and control it), and the install/uninstall options are Windows only (so why even include them in the help on UNIX?).

Writing a Systemd service for Bitscope Server

Systemd shouldn’t be intimidating, but I see a lot of hacks around the internet for getting things to start on boot. By writing a service, systemd will take care of all of that for us. Referring to my cheat-sheet on systemd service files, I first find where the binary is

which bitscope-server

And it tells us

/usr/bin/bitscope-server

So now we can create the service file and edit it:

sudo touch /etc/systemd/system/bitscope.service
sudo chmod 664 /etc/systemd/system/bitscope.service
sudo nano /etc/systemd/system/bitscope.service

The contents of the file:

[Unit]
Description=Bitscope Server
After=network-online.target

[Service]
ExecStart=/usr/bin/bitscope-server
StandardOutput=null
Restart=always

[Install]
WantedBy=multi-user.target

Save and exit, then reload the daemons, start the service, and enable it to auto-start

sudo systemctl daemon-reload
sudo systemctl start bitscope
sudo systemctl enable bitscope

And we can view the status

sudo systemctl status bitscope

This bitscope-server command is really poorly behaved for a server process. Without -v, it outputs nothing at all describing what it’s doing, and with or without -v, it will continuously read stdin checking for ENTER to quit, and then it asks for a y or n to quit. When run as a systemd service, stdin is null, and it still reads from it anyway and interprets this failure as an ENTER, so it asks over and over and over to type Y or N to quit. The best solution to this is to set the standard output also null, so it doesn’t fill any log files with garbage. You can still check the status and it will report any errors that systemd sees, as well as the status of the process and if it quit on its own.

Conclusions

The Bitscope software is a mess. The list of platforms they offer downloads for really terrifies me as to their software development process. They have Windows x86 and x64 (makes sense), macOS x86 only (wtf?), Raspberry Pi Raspbian x86, x64, and ARMv7, and generic Linux x64 (Not sure who’s running Raspbian on x86 or x64). Their current release of MacOS software is still 32-bit only (which Apple deprecated in 2012 and removed support for entirely in 2018), because they wrote their entire MacOS system using the Carbon API, which dates back to the early days of OS X as a compatibility layer for Mac OS 9 apps and was never intended to be used for native OS X apps. It was also deprecated in 2007 and support was removed entirely in 2018, so no modern macs can run their software.

In addition to all of the weird platform problems, the server software is really designed to run as a terminal program and as you can see by the continuous reading for ‘press ENTER to quit’, its very poorly behaved on Linux. Maybe it does better in a Windows environment, but I don’t expect anyone is going to want to deal with the overhead of Windows for a little scope server.

Upgrading my Zigbee network with the zig-a-zig-ah! (zzh!)

Fri, 12 Nov 2021 11:00:00 +0000

I currently have a TI CC2531 based Controller for Zigbee2MQTT, and the performance is poor. Based on how much my Zigbee network is expanding with the really low cost of Zigbee hardware (despite my preference for Z-wave for reliability when it counts), I’m upgrading my Controller to the zzh! from electrolama. The new controller is based on the CC2652 and has a decent external antenna, and should hopefully improve my awful Zigbee reception and reliance on IKEA repeaters to get any signal at all.

As you can see in the ‘Before’ network diagram, most of my devices don’t want to connect directly to the controller, and those that do get very low link quality index. The laundry_temp_humid_sensor is physically located in the same room only a few feet away from the CC2531 and only gets an lqi of 68, while the andrew_bed_motion is only one wall over (~10 feet and 1 wall in between) and gets an lqi of 5. That’s really terrible. The first repeater (bed_andrew_repeater) is actually plugged in to the same power strip as the CC2531’s raspberry pi and is about 3 feet away, so the poor antenna on the CC2531 is obvious when we see how many devices want that repeater instead of the controller.

Back when I bought my IKEA Blinds, I also bought the CC2531 pre-flashed with controller firmware to use in my Zigbee network, but due to life getting in the way I didn’t end up setting up that network until a few months ago. Back then, the CC2531 was the recommended dongle from Zigbee2MQTT. In the time since, the CC2531 became less recommended compared to the CC2652, and now it’s listed as NOT recommended (but still supported). The zzh! helps support the Zigbee2MQTT developer somewhat, and I felt comfortable selecting that dongle for my network. I was fully aware that I could possibly have to re-pair everything, although migrating from the CC2531 to CC2652 sometimes does not require re-pairing if all goes well.

The first step in the transition is to flash my new CC2652-based zzh! with the correct firmware. The firmware primarily used with Zigbee2MQTT and open-hardware dongles is the open-source Z-Stack-firmware. I then followed Electrolama’s Guide to flash the dongle. Not wanting to deal with a Windows-only flashing utility from TI, I chose this option (with the open-source Python utility) despite the Windows version having a GUI.

Next step is to backup the configuration. I ran the backup script manually, then renamed the file it generated so that particular backup will persist forever in case I need to roll back. After that, I shut down the Pi, replaced the USB dongle, and booted it back up.

When I rebooted, I had to make a few configuration changes to zigbee2mqtt:

The new dongle was at /dev/ttyUSB0 instead of /dev/ttyACM0
The new dongle requires ‘rtscts: false’ in the advanced section of the yaml

After making those changes (with the service stopped, since it will overwirite configuration.yaml on its own), I started the service and it seemed to work fine. Devices in my network re-appeared as available in Home Assistant. I then ran a network map and got… odd results. No device links, just a cloud of devices. I went through the house and found out that most of my devices appeared to be working (motion lights followed me, etc.) but they still don’t show up in the link map.

I did a new backup and gave it a day to get over itself. Zigbee lacks a way to force neighbor updates like Z-wave’s ‘Heal’ function, since you basically just have to hope the devices discover new neighbors and links when they wake up and try to communicate. I had a few stubborn devices - two Aqara door switches on my south and center garage doors (but not the north!), and the basement temperature + humidity sensor. I eventually gave up and re-paired them, which was a bit of a hassle.

Was the upgrade worth it? The CC2652 supports 50 direct children, while the CC2531 only supported 20. Of course, you can have more than that if your nodes are connected via routers. The devices which are now direct children of the CC2652 seem to have better signal strength than before, and many of them are significantly further from the controller than the children of the CC2531. Overall, I’d say you should definitely start with a CC2652 or upgrade as soon as possible, so you don’t start running into the limitations of the CC2531 when you have a larger network that’s more hassle to reconfigure.

WLED Lighting for my Desk

Mon, 08 Nov 2021 11:06:32 -0400

Today I finished working on new lighting for my closet office.

Last year I renovated my bedroom, to fit my computer desk better. I don’t need a massive closet and traditional American McMansions require a large closet space with plenty of wasted space in every single bedroom, so I decided to take the closet doors off to expand my bedroom and put my desk into the closet. After some network and power wiring work, I had a pretty comfortable expansion of my desk space and cleared the center of my room for VR gaming.

The Story

That was last year. The downside I’ve found is that the closet doesn’t have a light. Sometimes this isn’t a problem, but if I want to use my desk for things other than computer work, I really need better lighting. So, I decided to use individually addressable RGBW LED tape. RGBW tape is similar to RGB tape, but with an extra white channel with a more broad spectrum light, to improve the CRI of the white light. An expert on this topic is Quindor of QuinLED. So, I followed his site and watched some of his videos to decide exactly what I want.

Since my closet door opening isn’t that big, I only need 1.5m of tape. Since I’m running a short tape, I don’t need higher voltages, and the voltage drop across a 5V tape this short with the AC power supply a few feet away is reasonable, so I went with a 5V LED for simplicity (and I had some fairly beefy 5V AC adapters already). The LED tape came with a connector, and I soldered that to 5’ of wire to mount the QuinLED down near the ground. I got the IP30 LEDs since they dissipate heat better with no enclosure or potting, and are easier to work with. I used the built-in tape to secure 1.5m of the strip to the back of the closet door, so it faces the angled ceiling of the closet to bounce light at my desk. I taped the wire to the wall around the back of the closet, and bundled it with wires I was already running up the closet wall (Ethernet and HDMI to my TV). I connected the QuinLED (wired Ethernet model), and a 5V 10A power supply I already had. It’s a bit overkill, with the full RGB white + dedicated white at full brightness, the strip is estimated to draw around 4.9A.

I found the WLED on the network by looking at DHCP leases in my router, connected to it, and set it up for my strip. I’m using 1.5m of 60 LED/m strip, which is 90 LEDs. They are RGBW (with a natural white channel), using SK6812 family of LEDs. I ran through all of the colors to make sure the configuration was correct, and was amazed at how easy to use the WLED software is. Home Assistant found the WLED through Discovery, and I added it and configured it to be part of my bedroom. It exposes most of the WLED functionality through the HA integration, which is excellent.

Results

Click on this image for an animation:

Project Files and Parts List

I didn’t create any of the software for this project. Here are links to the hardware and software developers who made this possible.

LED Tape - I bought SK6812, RGBNW (Natural White), 5m, 60 LED/m, IP30 - AliExpress Affiliate Link
QuinLED DigUno - I bought it from DrZZs with the Ethernet add-on board - Quindor’s Site or DrZZs Site
WLED Firmware - pre-installed if you buy the QuinLED above - Github Some links to products may be affiliate links, which may earn a commission for me.

Laundry Finished Notification

Tue, 02 Nov 2021 11:00:00 +0000

Often, when I’m focused on a task or project, I absolutely can’t think about anything else. This means that other tasks often get left unattended, possibly for hours. To reduce this, I’ve implemented a system in Home Assistant to notify me when either the washer or dryer are finished running. It’s a pretty simple automation, but it’s this kind of useful life-improvements that really make home automation worthwhile.

The Hardware

I have a gas dryer (not ideal, but I’ll get a heat pump in the next house) and normal electric washer. Both of them share a single power outlet on a 20A circuit, all fairly standard in the US. I do have power monitoring at the breaker panel via the Energy Monitor Project, but that includes power consumed by the washer, dryer, as well as some networking equipment which resides in the laundry room (including the Zigbee master, Z-Wave master, and 433Mhz receiver, a network switch, an audio amplifier for the deck speakers, and two Raspberry Pi’s). The washer and dryer use quite a bit of power, but I still need better resolution than this.

Thankfully, they both use less than 10A and plug into standard US 120V wall plugs, so I can just get two Sonoff S31’s and call it a day (I had one leftover from Tasmota Day and had to get a second). I then added them to Home Assistant, including all of the power monitoring channels, configured them to turn on automatically (PowerOnState 1), and send telemetry more often (TelePeriod 20). Once that was all done, I ran a cycle on each to establish how much power each consumed.

Unfortunately, the washer has a portion of the cycle wher it uses very little power, as it fills initially and again during the rinse cycle. I need to set the threshold very carefully to detect when it’s actually off (and just drawing standby power) versus when only the water valve is on. It seems like the water valve is on at 8W and the control power alone is 2W, so a threshold of 5W and a minimum time of 2-5 minutes is probably adequate. The dryer is much easier, it uses a lot of power for the entire cycle, so the threshold is more agressive at 10W for 1 minute.

The Notifier

Notifications in Home Assistant are really easy. You call a service on the notifier entity, which in this case is an iphone.

If you just want to copy it, here’s the yaml for the whole automation:

alias: Laundry Finished Notification
description: Notify when laundry is finished
trigger:
  - platform: numeric_state
    entity_id: sensor.laundry_dryer_power_w
    id: Dryer
    below: '10'
    for:
      hours: 0
      minutes: 1
      seconds: 0
      milliseconds: 0
  - platform: numeric_state
    entity_id: sensor.laundry_washer_power_w
    id: Washer
    below: '5'
    for:
      hours: 0
      minutes: 5
      seconds: 0
      milliseconds: 0
condition: []
action:
  - service: notify.notify
    data:
      title: Laundry Complete
      message: '{{ trigger.id }} Has Finished'
mode: single

You can add as many triggers as you want, it just passes ‘{Trigger ID} Has Finished’ as the notification value. So, set the Trigger ID to the text you want it to say when it notifies you. You can also specify a single device instead of notify.notify if you want to alert just one phone.

3D Printer Nozzle Camera

Mon, 25 Oct 2021 20:00:00 +0000

After buying some cheap USB boroscope cameras to use as an actual boroscope for home renovation projects, I decided to buy one with a flexible cable to mount near the nozzle of my Prusa i3 MK3s to get a time-lapse of the nozzle during the printing process. Watching the first layer closely during a print can show failures early, since most failures are due to bed adhesion or other first-layer problems. Keeping the camera out of the print volume while still getting a good view of the nozzle and print was challenging, but I ended up with a cool solution that I’m really proud of, and the view is fantastic.

Camera Mounting

I started by holding the camera up to the printer and measuring where I wanted to mount it. The Prusa i3 MK3S has a bracket which goes off the back of the extruder assembly which anchors the flexible cables to the extruder. The cables from the extruder motor, PINDA probe, filament sensor, and fans run over the top of the anchor, and the cables from the hotend heater and thermistor run over the bottom of the anchor. I decided to make a wedge to sit against the bottom of the anchor, allowing me to use the well designed zip tie holes to securely mount the camera. The hotend wires can move along the side of the anchor, leaving space for the camera. I held up the camera in place with the camera view open on my laptop and decided the slope I want (1/4" over the length of the camera), then measured the dimensions of the camera with calipers to make a mount with a half circle for the camera to sit in.

Bracket model in PrusaSlicer

After running the tiny part on the printer, I decided to tape the camera to the bracket, since the camera has a smooth metal finish and zip ties would not securely hold it against rotation. To align the camera, I set the bracket on the table (upside-down), placed a square object in front of it, and rotated the camera around by its cable until the square object was aligned to the image upside-down (since the bracket will be flipped when mounted on the printer). I then wrapped the camera and bracket in masking tape, careful to maintain alignment.

Camera and Bracket zip-tied to Prusa's cable anchor

Hotend cables out of the way

View port of camera (you have to read further for the camera's view)

Using OctoLapse to record two cameras

Since I now have two cameras, I’d like to use Octolapse to record both of them. I installed OctoLapse through the OctoPrint plugin manager. I’m starting from the config I’ve been building up in the OctoPi Project and 3D Printer Power Project.

Once it was installed, I setup two new cameras for the Ethernet camera and the built-in OctoPi camera, which is now the nozzle camera. I renamed the default camera to NozzleCam, and created an entirely new camera for the Ethernet Cam. I didn’t bother setting the stream address, it’s only used in the web UI and I usually use Home Assistant to watch the camera anyways.

OctoLapse doesn’t use the default timelapse temp folder to store its temporary files, which leads to temporary files being stored on the SD card of the Pi. This is not ideal, so we need to add another autofs mount for the folder /home/pi/.octoprint/data/octolapse so it can keep its stuff on the network. The temp data doesn’t really need to be on the network, but the NAS can handle write wear much better than the SD card of the Pi. OctoLapse was very angry mounting the entire octolapse folder on the network, since it seems to try to chmod files after creating them (even though this is unnecessary, it doesn’t have permissions to chmod a folder it doesn’t own), so I ended up mounting the /mnt/printer (the existing printer mount) with noperm (so Octolapse can chmod files it doesn’t own and the permissions will be ignored) and created a symbolic link from the octolapse temp folder to a folder in /mnt/printer. I also forwarded the timelapse folder with a similar method. Overall, Octoprint is not happy with the autofs mounts, and keeps trying to revert to default folder paths, so maybe it’s a better approach to symlink them anyway instead of using the settings file to change them to the paths I want.

ln -s /mnt/printer/octolapse/tmp ~/.octoprint/data/octolapse/tmp
ln -s /mnt/printer/timelapse ~/.octoprint/timelapse

Other Quirks

The camera has built-in lighting, and there’s a small box midline in the cable with a knob to adjust the LED intensity. However, there’s no way to turn it off, or control it from the Pi. My solution to this is to disable USB power when the printer is turned off, which opens a whole can of worms. I wrote a short blog post about my solution here.

Essentially, uhubctl works great to turn on/off the power to the LEDs, but it also turns off power to the camera itself, so the webcamd streamer needs to be restarted when the camera returns. It also cuts power to the printer, which isn’t powered over USB, but powers its USB isolation from the USB side, so the printer serial port drops out as well. When I re-enable power, it takes a few seconds for the USB UART for the printer to return as well, so I extended the connection delay (from when the power is turned on the printer and USB to when the printer connection is attempted) to 15 seconds, and this seems to work adequately.

Result

Click on the thumbnail to view the full video on Youtube

The Project Files and Parts List

Here are all of the files and parts required to replicate this project. All of my design files are licensed Creative Commons CC-BY-SA. As this project uses files created by other authors as examples, please read their licenses as well.

Camera I Used
Wedge Mount Design Files (AMF Format)
Wedge Mount Design Files (FreeCAD Originals)
3DBenchy Test Print
Flexy Rex Test Print
Low Poly Fox Test Print Some links to products may be affiliate links, which may earn a commission for me.

My First YouTube Video! And NozzleCam Project

Mon, 25 Oct 2021 20:00:00 +0000

Today I just published a project I’ve been working on for awhile - my 3D printer nozzle camera - and it’s the first project I’ve made a corresponding video for! I built this nozzle camera mount to carefully watch the first layer of 3D prints, since most failures happen early on due to poor first layer adhesion or related issues. Keeping the camera out of the print volume while still getting a good view of the nozzle and print was challenging, but I ended up with a cool solution that I’m really proud of, and the view is fantastic. I’m super excited about the video release, and want to share it here on the blog as well! I hope you enjoy the video. Here it is. Click on the thumbnail to view it on YouTube.

How Much Power does a 3D Printer Consume?

Sun, 24 Oct 2021 12:00:00 +0000

I have a Prusa i3 MK3S 3D printer, and I use it for many projects I post here. I want to know roughly how much power it consumes while printing, to get a feel for both the peak power consumption of the printer (to, say, size an off-grid power system) and energy consumed for an average print. I know the cost of filament is pretty low for most projects, but I don’t know the cost of energy (or wear on the printer). This project seeks to remedy that, using a Sonoff S31 power monitoring smart plug flashed with Tasmota firmware. I also set up this smart plug to be controlled by OctoPrint as an added bonus.

Flashing and Configuring Tasmota

I flashed my Sonoff S31 on my Tasmota Day. By the end of the ‘day’, I had a number of S31’s which had Tasmota flashed, and which were configured with the correct WiFi network and module type, but no other configuration.

The first step is to configure the MQTT broker. I am using the same broker as Home Assistant, although in this case commands will come from Octoprint instead of HA. Start by finding the IP address (I look on my router’s DHCP leases for new Tasmota devices), and connect to it over HTTP. It should show up with a big ‘ON’ or ‘OFF’ as well as a bunch of power measurement numbers, a toggle button, and configuration buttons. Click ‘Configuration’, then ‘Configure MQTT’, and enter the broker information. I don’t like how Tasmota pollutes the root namespace in MQTT with tele, cmnd, etc. so I change the default topic structure to be ‘raw/%topic%/%prefix%/, which ends up with a topic like raw/tasmota_123ABC/stat/POWER. I set this structure for my Tasmota devices when I was newer to MQTT, but now I would have chosen ’tasmota/%topic%/%prefix%/’, so tasmota devices get their own folder in the namepace, followed by each individual device. This is more of a personal opinion on how to structure MQTT topics. Save configuration here.

Next, we need to change the telemetry period. By default, Tasmota on the S31 will send power readings every 300 seconds. I’d like more data, and the backend can deal with all that data, so I bumped it up to 30 seconds. From the main screen, click Console. Now, we need to change the value ‘TelePeriod’ to the number of seconds we would like

TelePeriod 30

It should return a JSON string of the new value, like ‘{“TelePeriod”:30}’. If it still says 300, you didn’t type it right. No equals, no comma, just the command and value.

Setting Up OctoPrint Plugins

We have a bit of a dance in OctoPrint with plugins and subplugins. In the end, we need all of these plugins installed:

MQTT
HomeAssistant Discovery
PSU Control
PSU Control - MQTT
MQTT for PSU Control

Some of these will already be installed if you started from my OctoPi Project. If you followed that guide, skip down to PSU Control, the first two plugins are configured identically.

MQTT Plugin

Enable and configure this plugin first, by itself. This way, we can configure all of the topic prefixes before HomeAssistant Discovery messes with them. Setup your broker, topics (I used octoprint/<printer name>/ so I can add more printers later), and leave the rest of the settings as defaults.

HomeAssistant Discovery Plugin

Enable and configure this plugin second. There is nothing to configure here, other than Device Name. I suggest setting it to the name of your printer, if you have more than one printer, so they won’t conflict. It should automatically generate a Node ID.

Now, you should see a new Device in Home Assistant. Feel free to do any configuration you’d like there. It’s not super important for the purposes of this project, and it’s the same as my OctoPi Project.

PSU Control, PSU Control MQTT, and MQTT for PSU Control

The naming of all of these is a bit odd. PSU Control is a plugin which allows OctoPrint to control the power supply of your printer. It relies on sub-plugins to provide different interfaces to PSU control switches, and provides a UI and common startup/shutdown settings for all of them. In our case, the sub-plugin we are using is PSU Control - MQTT which sends the PSU on/off messages and returns status via MQTT. The third plugin, MQTT for PSU Control, is not a sub-plugin which provides a PSU interface, but instead provides access to the PSU Control switch via Home Assistant over MQTT.

PSU Control

For PSU Control, set the Switching Method to Plugin, and select PSU Control - MQTT as the plugin. For Sensing Method, again select Plugin, and again select PSU Control - MQTT. I set the polling interval to 1 second. For Power On Options, I added a 15 second Post On Delay to allow the printer to boot up, and enabled Connect when powered on. Finally, I set Power Off Options to turn off after idle for 30 minutes, and to disconnect from the printer.

PSU Control - MQTT

For PSU Control - MQTT, we need to set the topics of our Tasmota smart switch. Since I changed the default topics, my command topic is ‘raw/tasmota_049BED/cmnd/POWER’ and my status topic is ‘raw/tasmota_049BED/stat/POWER’, with ON and OFF being the payloads for both topics. Tasmota publishes state reliably, so no need to poll for state and we can disable the ‘actively query for switch state’ option.

MQTT for PSU Control

For MQTT for PSU Control, first go into the HomeAssistant Discovery plugin and copy the Node ID. Then, go into the configuration page for MQTT for PSU Control and paste this Node ID under the HomeAssistant automatic MQTT discovery heading. This will add the PSU Control switch to the existing Device created by the HomeAssistant Discovery plugin. Check ‘activate HomeAssistant support’ and ‘add switch entity to existing device’, and change the Switch Name if you wish. Device Name will be inherited from HomeAssistant Discovery, since they share the same node ID and we are adding the switch to an existing device.

Adding Power Monitoring to Home Assistant

Since I’m controlling the power relay through OctoPrint, I don’t want to add it directly as a switch in Home Assistant. The PSU control integrates with Home Assistant already, so the switch is already exposed indirectly through the OctoPrint device. I just want to capture the sensor data, since OctoPrint’s plugins don’t do that.

All of the sensors are numeric and exposed by Tasmota in a single JSON-encoded message, so all of them become sensor entities in Home Assistant. Therefore, we edit sensor.yaml and add each signal as a sensor. Also note that I use a different topic path than the Tasmota default, so you may need to adjust the paths (use MQTT Explorer if you need to find the paths).

#3D Printer Power consumption data
- platform: mqtt
  name: "Columbia Power [W]"
  unique_id: "columbia_power_w"
  state_topic: "raw/tasmota_049BED/tele/SENSOR"
  value_template: "{{value_json['ENERGY']['Power']}}"
  unit_of_measurement: "W"
  device_class: "power"
  availability:
    - topic: "raw/tasmota_049BED/tele/LWT"
      payload_not_available: "Offline"
      payload_available: "Online"
- platform: mqtt
  name: "Columbia Voltage"
  unique_id: "columbia_power_v"
  state_topic: "raw/tasmota_049BED/tele/SENSOR"
  value_template: "{{value_json['ENERGY']['Voltage']}}"
  unit_of_measurement: "V" 
  device_class: "voltage"
  availability:
    - topic: "raw/tasmota_049BED/tele/LWT"
      payload_not_available: "Offline"
      payload_available: "Online"
- platform: mqtt
  name: "Columbia Current"
  unique_id: "columbia_power_i"
  state_topic: "raw/tasmota_049BED/tele/SENSOR"
  value_template: "{{value_json['ENERGY']['Current']}}"
  unit_of_measurement: "A" 
  device_class: "current"
  availability:
    - topic: "raw/tasmota_049BED/tele/LWT"
      payload_not_available: "Offline"
      payload_available: "Online"
- platform: mqtt
  name: "Columbia Energy [kWh]"
  unique_id: "columbia_power_kwh"
  state_topic: "raw/tasmota_049BED/tele/SENSOR"
  value_template: "{{value_json['ENERGY']['Total']}}"
  unit_of_measurement: "kWh" 
  device_class: "energy"
  state_class: "total_increasing"
  availability:
    - topic: "raw/tasmota_049BED/tele/LWT"
      payload_not_available: "Offline"
      payload_available: "Online"
- platform: mqtt
  name: "Columbia Power Factor"
  unique_id: "columbia_power_fac"
  state_topic: "raw/tasmota_049BED/tele/SENSOR"
  value_template: "{{value_json['ENERGY']['Factor']}}"
  unit_of_measurement: "%"
  device_class: "power_factor"
  availability:
    - topic: "raw/tasmota_049BED/tele/LWT"
      payload_not_available: "Offline"
      payload_available: "Online"

The Print

I printed a 3DBenchy for the purposes of this power test. The printer peaked at 245W during the warmup phase, averaged 92W during the print phase, and consumed just under 0.1 kWh during the roughly 1 hour print job. Home Assistant doesn’t show very many decimals and the numbers are pretty small, so the precision on that 0.1 kWh is probably +-10%. The idle consumption is also not negligible, at 10W. Leaving it on all the time would be roughly 7.2 kWh/month, which would cost around $1 (at my energy prices in 2021), purely for idling. The print doesn’t cost much in terms of energy, coming in at an energy cost of 1.3 cents (again at my energy prices in 2021). Overall, the energy expense of a single hobby 3D printer is not high enough to worry about.

Lovelace Graph of 3DBenchy Print Data

The Project Files and Parts List

Here are all of the parts required to replicate this project. There are no design files for this project.

Flexy Rex Kid-Friendly 3D Printing

Fri, 22 Oct 2021 12:00:00 +0000

Flexy Rex is a 3D print that’s been circling the internet for awhile. See below for the links to all of the variants that have been created over the years.

Anyway, I printed this is one of the test prints with my upcoming Nozzle Cam project. My cousin was visiting from out of state along with her young son, so I let him print a Flexy Rex, watch the printer (and the camera feed), and take it off the bed when it was done. Enjoy this soothing timelapse until I finish documenting the whole project. Since the video is on YouTube, I’ve linked it instead of embedding in case you’re concerned with the privacy implications of using Google’s services.

Project Files

I did not create these files. The following links are to all of the known original authors, from newest to oldest. All files are licensed CC-BY-SA.

Tasmota Flashing Day

Tue, 19 Oct 2021 12:23:00 +0000

Today I spent the day flashing Tasmota on a variety of Sonoff devices, for use in future projects. I took pictures of the process, so you can follow along with all of the fun bits of playing with repurposed electronic hardware.

I have projects in mind for some of these, but some are still ’extra’ (nothing is really ever ’extra’, it will always be used eventually). I have two Sonoff S31 (US smart plug with power monitoring), two Sonoff S26 (US version of the low cost S20 smart plug), and two Sonoff 4CHPRO R3 (4 channel relay, the Pro version with isolated relays). I have plans for one of each, but when you’re paying for shipping, why not buy two of each?

What You Need

All Sonoff modules which support Tasmota use the ESP8266 family of chips (often the ESP8285, which has built-in flash memory). The ESP chips all include a built-in serial bootloader which is activated by connecting GPIO 0 to ground while powering on the chip. Most Sonoff smart plugs use GPIO 0 for the button on the device, so pressing it while powering on is an easy way to get into the bootloader. Most of them also have pads available for 3V3, GND, TX, and RX, and we just need to find them, solder pins or wires to them, and connect a USB to Serial adapter. I am using an Adafruit 3V USB to Serial cable with USB-C, along with male to female 0.1" header jumpers to break out the 6 pin header on the cable to individual pins. I’ve used the same color leads as the corresponding wire in the Adafruit cable, where Red = 3V3, Brown = Gnd, Orange = TX (Sonoff RX), and Yellow = RX (Sonoff TX).

The software we will be flashing is Tasmota, which was originally written as an alternative firmware for Sonoffs specifically but now supports a wide range of ESP based devices. I am flashing a pre-built standard binary, which includes all of the functionality I need for my Sonff’s. ESPHome is another popular option, but for stock Sonoff devices, Tasmota has pre-built module configurations. ESPHome would probably be a better choice if you are building your own devices.

The third piece of the puzzle is Tasmotizer, a simple tool which will download the Tasmota image, flash it, and let you configure common settings (WiFi, MQTT broker, Module Type) and push them over the serial console so you don’t have to go through the process of connecting to the temporary WiFi network to configure Tasmota for the first time.

Sonoff S31

This was the first plug I’ve used from Sonoff. I bought it because of the power monitoring, but the form factor fits a lot better with US plugs than the S26 (since US plugs are usually stacked vertically instead of horizontally like most other countries). For the small price difference, power monitoring is also a really nice feature to have, especially when paired with logic in Home Assistant, or even just as a cheap Wifi appliance power monitor. The S31 has 6 pads with 0.1" spacing already, two of them are used for the second serial link to the power monitoring chip and are not used. The process was straightforward and Tasmota already has a module type for S31 which works out of the box with power monitoring. Tasmota Page on the Sonoff S31

Everything you need except the USB-Serial cable

Pry the side cover off

Remove the linear trim pieces and remove the screws behind them

Pull out the module to reveal the ESP and pads

Solder pin header to pads, two pins are not used and I skipped soldering them

Connect to USB-Serial cable and Tasmotize

Sonoff S26

This one was the most difficult since the pads aren’t in a row (they are arranged in two sets of two), they are closer together than the usual 0.1" pitch, and the power and ground pads are very close to the power PCB. I managed to tear a pad off the first one I tried, and threw it away instead of fixing it (they are cheap). I also managed to tear a pad off trying to remove my pins on the second one, but it’s already flashed so it survived. GPIO 0 is again connected to the single button, so hold the button while plugging in the 3V3 pin to get into the bootloader, then flash in Tasmotizer. I set it up as a ‘Sonoff S2x’ module, since the S20 family are all the same software with variants for different country plugs. Tasmota Page on the Sonoff S26.

Ready for disassembly

TX/RX on the left, GND/3V3 on the right, tiny pads, I'm not a soldering expert

Despite ugly soldering it still flashed fine

Sonoff 4CHPRO R3

This was the easiest. After taking the case off, I found that it has plated through-holes specifically for the 3V3, RX, TX, and GND pins, which is what we need for flashing. I used long pin headers so they stood up from the board with room to solder without removing the Sonoff from the case lower housing, and even with the long pins the upper housing still fits, so I don’t even need to remove my pins. Unfortunately, since this model uses an additional microcontroller for IO expansion, none of the buttons map to GPIO 0 (which is what launches the bootloader on ESP chips). It’s tied to 3V3 through a resistor (R21). So, I need to poke the resistor with a test lead to get it into bootload mode. Once it’s powered on with GPIO 0 grounded it enters the bootloader, and Tasmotizer is able to flash it without issues. Set the module type to Sonoff 4CH and it works exactly as expected. Tasmota Page on Sonoff 4CH PRO.

They clearly want us to flash it with these through holes

But we still need a lead to ground R21 to enter bootload

Controlling USB Camera Lights on Raspberry Pi with USB Hub Power Control

Mon, 18 Oct 2021 18:06:32 -0400

I recently got a camera to use with my 3D printer which includes integrated LED illumination. It has a physical switch to control the lights, but I want to turn them on and off, either from OctoPrint, or from Home Assistant, based on the printer status. Wanting the simplest solution possible, I’m going to turn off power to the Pi’s USB hub when I don’t need the camera lighting. This has a side effect of also disabling the camera itself due to lack of power, which I will also need to deal with, but it doesn’t disable the USB data lines, so the 3D printer (which has its own power supply) might not be affected, depending on the printer model.

Basics of USB Hub Control

The key to this project is the uhubctl command, which allows us to control the power state of USB hub ports from the command line. It specifically supports the built-in hub(s) on the Raspberry Pi B+, 2B, 3B, 3B+, and 4B, although the commands vary slightly based on each model. In my case, I am using a Raspberry Pi 3B, but you can change the commands if you have a 3B+ or 4B (B+, 2B, 3B are all the same). For the Pi, all of the ports share a single power output, so we can’t control power individually. Other hubs don’t have this limitation, so if this is a deal breaker for your setup, you should get another hub on the supported list to control your outputs, OR use any old hub to power the devices which still need power (since uhubctl doesn’t actually disable communication on the ports, just turn off the power to them).

I’m starting with a completely bare OctoPi install in this case, but Raspbian / Raspberry Pi OS should be similar. I first need to install uhubctl, which is packaged with Debian (and thus Raspbian). SSH in and install it with apt.

sudo apt install uhubctl

Next, we need to give uhubctl permission to operate on the hub, or use sudo all the time. To start, we can test that it works at all using sudo.

sudo uhubctl

This should return a list of all of the hubs and ports. For my Pi, there is one hub (1-1) and it has 5 ports. Port 1 controls the WiFi and Ethernet, port 2-5 are the user-accessible USB ports. Controlling the power to port 2 controls all of the user ports. We can try toggling it with sudo:

#Turn off hub 1-1 port 2 (all external ports on the Pi)
sudo uhubctl -l 1-1 -p 2 -a 0
#Turn on hub 1-1 port 2 (all external ports on the Pi)
sudo uhubctl -l 1-1 -p 2 -a 1

In my case, the lights on my camera turned off (well, the whole camera turned off), and then came back on. Since we toggled power to the camera in addition to just the lights, the OctoPi camera streamer isn’t very happy with us, so we need to restart it

#The OctoPi webcam service is named webcamd
sudo systemctl restart webcamd

At this point, if we just wanted hub control, we could give user pi permission to control the hubs via udev rules. uhubctl gives us instructions for doing this. However, since we will always need sudo to restart webcamd, the script / tool we use to do this needs to have sudo permissions anyway.

Control using Node-Red

If you only want to control the USB hub to come on/off with the printer, you can use the OctoPrint PSU Control plugin to do this. However, I’m already using the PSU Control plugin to control my PSU, using a Sonoff S31 over MQTT (See Project), so I can’t use that plugin again. I decided to install Node-Red on the Pi, give it full passwordless sudo permissions (dangerous, I know), and have it subscribe to the same MQTT topic used to control the Sonoff, and respond to ON/OFF messages with the corresponding uhubctl commands.

I have another quirk in my setup, which you are also likely to encounter if your printer has a well-designed controller. Some printers have isolation between the USB port and the printer’s internal power supply, and rely on USB power to supply the USB-serial converter and/or USB isolator. In my case, my Prusa i3 MK3S has a USB-UART chip which is powered by the USB power, followed by a digital isolator for the serial lines which is also powered by USB power. So, at a minimum, I need to keep the USB bus power enabled as long as I want to connect to my printer. Since I am having the USB bus power directly follow the AC power to the printer, this should be perfect, since the PSU Control plugin can be configured to wait after it confirms AC power is on before it connects to the printer and it disconnects from the printer before it poweres off.

I installed node-red using the apt package, but the node-red website does have a script to install it on your own (and install a more recent version). Since I’m not particularly concerned about the version, I used the one in the distro repository.

sudo apt update
sudo apt install nodered

After that, I needed to allow no-password sudo access. Node-red runs as user pi, who already has sudo permissions, but would need to re-type their own password to use sudo. Since node-red can’t do that, I’m going to give user pi no-password sudo access, to allow node-red to execute systemctl to restart the webcam server and also control uhubctl without modifying udev rules. We could get around the uhubctl issue, but systectl must always run as root, so we still need to give root access to node-red anyway. To do this, we need to edit the sudoers file, and this must be done using visudo:

sudo visudo
#If it asks which editor you want, Nano is the easiest to use

#Find the following line
%sudo ALL=(ALL:ALL) ALL
#Replace with this line
%sudo ALL=(ALL:ALL) NOPASSWD: ALL

Now, setup a node-red flow to listen on the command topic and respond by executing the system commands to turn on/off USB power. Click Here for my Flow. You will need to setup the MQTT broker and topic to match your setup, I have examples listed.

The Bare Minimum Custom Systemd Service

Sun, 17 Oct 2021 10:00:00 +0000

Occasionally, I find myself wanting to start something custom as a systemd service, so it starts at boot. There’s a whole wealth of information on how to properly write systemd services, but I just want the bare minimum to get my command executed on boot and running on its own. Hence, here is the most basic systemd service guide. Feel free to read the systemd docs (systemd.service, systemd.unit, systemd.exec) for more info on what can go in the service file if you want to get fancy.

Start by determining the full path to your service. If it’s on the PATH, you can do this with which. If you wrote it and it doesn’t install itself in /usr/local/bin like a good program, it might be in your home folder. If it’s a script and needs an interpreter, you need the full path to the interpreter followed by the full path to the script

Second, create a systemd service file in the right place and edit it.

sudo touch /etc/systemd/system/myservice.service
sudo chmod 664 /etc/systemd/system/myservice.service
sudo nano /etc/systemd/system/myservice.service

Now, write out the shortest possible configuration file including the full path and command line to execute your service

[Unit]
Description=A Cool Service, Wow!
After=network-online.target

[Service]
ExecStart=/usr/bin/python3 /home/pi/myservice/main.py -a -b -c -d
Restart=always

[Install]
WantedBy=multi-user.target

Finally, tell systemd to reload it’s daemon files (since we modified them), then start the service (launch it now), then enable the service (set it to launch automatically on boot).

sudo systemctl daemon-reload
sudo systemctl start myservice
sudo systemctl enable myservice

You can also view the latest bit of the log to check for any errors

sudo systemctl status myservice

The End! Systemd is quite easy to work with, and using it to launch your scripts at startup is handy.

Networked DVD Ripping with Raspberry Pi and iSCSI

Thu, 14 Oct 2021 12:00:00 +0000

I’ve been working up to a better virtualization and storage setup for my homelab for awhile now. One part of this is cataloguing my media and expanding the virtual side of the media library. I have a legacy collection of DVDs and BDs which I’d like to import, and that means I need to rip them from disk. The decryption and transcoding process requires a decent CPU. The demand for high performance leads me to want to run this in a virtual machine (where it can get low priority access to a wealth of compute resources), but the need for a physical disk drive also makes me not want to walk down to the basement every time a disk is done to change disks. Thus, I want a networked disk drive for the virtual machine to read from.

iSCSI Enters The Chat

After my previous experiments with iSCSI trying to network boot Windows diskless, I’ve grown to have a stronger appreciation for the benefits of iSCSI in the right use case. SMB (CIFS) is still the go-to choice for file sharing across networks, with server-enforced file permissions and wide OS support (basically, the only protocol Windows will speak), but sometimes it’s the wrong use case. I could setup a SMB share for a USB drive on a Pi, but then I can only read the mounted filesystem remotely, not the entire block volume, I can’t read bolume header information, and I can’t control the disk drive (i.e. eject it when complete). So, it’s not ideal.

Basically all modern storage devices in modern computers use only a few protocols, even with a wide variety of physical connection mediums. Quite a few of them end up back at the SCSI command set. The USB 3.0 spec has adopted SCSI over USB (‘USB Attached SCSI’) for mass storage devices, replacing the old USB mass storage command set, and allows this to be used over USB 2.0 as well. This means, if I can get a DVD/BD drive that supports UAS, it should show up as a SCSI device in Linux and thus I can route it over iSCSI to other devices on the network.

But, what is iSCSI? Simply put, iSCSI is a way to tunnel SCSI commands over a network. SCSI being a protocol to perform block transfers to block IO devices, this means that we can tunnel a block device, instead of a filesystem as we would with a protocol like SMB. The host side (‘initiator’ in iSCSI terms) is responsible for dealing with drive partitioning and formatting, and interpreting the blocks as a filesystem. The server side (’target’ in iSCSI terms) can choose to emulate the block device or pass the commands through to a physical device, or a partition on a physical device. In my case, I’m going to pass through the physical USB SCSI device, meaning the initiator will have full control of it (including all of the extra bits like drive eject).

The downside to using iSCSI over SMB or NFS is, since it’s a block device with the filesystem being managed by the initiator, we need to treat each target like a physical drive - we can’t have two clients sending commands to the same physical drive (normally). So, only one computer at a time can connect to each iSCSI target, and they have exclusive control of the device. It’s like the SCSI device is physically installed in the initiator’s machine. That’s exactly what I want with the DVD/BD drive, but this is not a protocol you should go to looking for general purpose file sharing.

Setting Up the Target

To get started with iSCSI, we need a target and an initiator. The target is were the block devices are stored. Normally, iSCSI is used on SANs, and an iSCSI target may be something like a giant disk shelf / JBOD or large storage server providing virtual disk drives for VMs. But, here, we just need a simple Raspberry Pi. I used a Raspberry Pi 2, since it’s the best I had given the current Pi 3 and 4 shortages and how many older Pi’s I have laying around. It doesn’t support USB 3.0, but my DVD drive (already owned) does, so it should use the new UAS instead of USB Mass Storage protocol. I had to use an external powered hub, since the drive would cause power issues with the Pi every time it tried to spin up.

I first validated that the drive was actually showing up correctly. Before plugging in the USB device, I got a full list of what is in /dev already, then I plugged in the device, then I diff’d the two.

sudo ls /dev > before.txt
#Plug in your drive here
sudo ls /dev > after.txt
diff before.txt after.txt

You should see an sgX and srX device, and a bunch of other devices (cdrom, cdrw, dvd, dvrw, …) depending on the specs of your drive. The sgX device is the one we need here, it’s the raw SCSI device. If you look closely (ls -l /dev), you will see that all of the ‘bunch of other devices’ are actually just links to the srX device, which is the disk device.

I’m going to setup the target similar to my diskless Windows install, but instead use direct SCSI passthrough to a physical device. First, install the iSCSI target:

sudo apt install tgt

Next, create a config file for the target

sudo nano /etc/tgt/conf.d/targets.conf

The contents are a bit more complex than last time

<target lun.2021-10.net.apalrd:vdvd>
    backing-store /dev/sg0
    bs-type sg
    device-type pt
</target>

In short, we named the target ’lun.2021-10.net.apalrd:vdvd’, it’s backed by the physical SCSI device /dev/sg0 (which is the ‘raw’ SCSI device for my drive), the backing store type is ‘sg’ (‘raw scsi device’, the prefix sgX in dev), and the device type is passthrough.

Then, we restart and check that it’s working

sudo systemctl restart tgt
sudo tgtadm --op show --mode target

If there are any errors in the first step, use ‘sudo systemctl status tgt’ to see what they are. If it returns with no errors, then tgtadm should show one iSCSI target with two LUNs, one the controller and the other backed by the physical disk drive’s SCSI device.

Setting Up the Initiator on Windows

Initially, I wanted to see if this worked at all, so I setup the initator on my Windows workstation and tried to rip a dvd using MakeMKV.

To use iSCSI in Windows 10, open the start menu and search for ‘iSCSI Initiator’. It will ask you if you want it to run as a service, click yes. Then, you can enter the IP address of the target and discover the LUNs it exposes, click one, and connect to it. If you go to Windows Explorer, it should show a new DVD drive, and it should be usable (although possibly very slow).

Windows iSCSI Initiator

DVD Playback over iSCSI

However, Windows seems to really struggle to use the DVD drive. It’s very slow. Handbrake was unable to rip the DVD at all, and MakeMKV was successful but at a rate of under a megabit per second. VLC was able to load the menu and navigate the menu, but couldn’t read fast enough to play back the content. I’m not sure what the issues is, but something about the setup is just not ideal.

MakeMKV works, but is very slow

(Note: I did try connecting the same DVD drive to a USB 2.0 port on the Windows workstation, and it did play, although it wasn’t super fast either, but it was at least fast enough to play back the DVD without any buffering)

I then moved on to Linux, hoping for a better experience.

Setting Up the Initiator on Linux

First, we need a machine to run this, running Linux, and preferably with enough resources to handle decoding/encoding in Handbrake. I chose to install Ubuntu 20.04 LTS Server on a virtual machine which I gave 8 vCPUs to. Since ripping / transcoding is a background task, I recommend allocating a lot of resources at a low priority to this VM, since the automatic ripping machine can queue jobs for transcoding (since reading the disk is much less intense than transcoding it), so you could pop in a complete box set of a season of your favorite show, let it read the disks as fast as possible, and slowly work through the transcoding in the background without bogging down your other VMs if they are busy. If you install your VM from a virtual CD drive, make sure you detatch the drive entirely (not just remove the virtual disk) once you are done, otherwise, ARM will try to make use of that device for ripping (and that’s not what you want).

Now we must setup the iSCSI target:

#Not needed on Ubuntu 20.04, may be needed on your distro
sudo apt install open-iscsi
#Discover target
sudo iscsiadm -m discovery -t sendtargets -p <IP of your iSCSI Target>
#Login to target (no authentication)
sudo iscsiadm -m node --login
#Check that new devices showed up in /dev - should see a new sgX, srX, cdrom, ...
sudo ls -l /dev
#Save settings persistently, auto-start iscsi initiator on boot
sudo iscsiadm -m node --op update -n node.conn[0].startup -v automatic
sudo iscsiadm -m node --op update -n node.startup -v automatic
sudo systemctl enable open-iscsi
sudo systemctl enable iscsid

Then reboot to make sure the initiator came up and you still have /dev/sgX, /dev/srX, /dev/cdrom, etc. on your initator.

Automatic Ripping Machine

For this project I decided to try and use an open source project called ‘Automatic Ripping Machine (ARM)’, which seems to be designed to do exactly what I want. I of course didn’t bother trying it in a setup with real working hardware, and jumped straight into getting it to run over iSCSI. Since I’m using Ubuntu 20.04 which is the OS they are testing on, they have a script to install on a bare environment. They note that the script has no error handling and is only for environments where you already know it will work, but I am using the exact version of Ubuntu so I ran the script (after reading it). It essentially follows a near identical path to the Ubuntu instructions (installing a ton of stuff with apt, cloning the repo, setting up udev, …).

After installing it and setting up a user, I popped a DVD into the (remote) drive. It took a while, and the drive made some noises, and eventually the disk showed up as active in the ARM web UI. It couldn’t identify the title, so it gave me 60 seconds to edit it before it started the rip. Unfortunately, it then tried to rip the DVD with handbrake-cli, and handbrake did not like the disk and failed. I tried several DVDs, getting the same result with each. So, given my results with Handbrake on Windows, it seems like it does not like the remote iSCSI drive. In addition, on the VM’s display, I was seeing a lot of IO device read errors, indicating a problem reading from the iSCSI device.

I tried to rip the disk as data instead of DVD, but that gave me a dmesg error about reading scrambled sectors without authentication (since it is a video DVD), and caused dd to exit without copying the DVD. Probably as expected. Again, I also got a lot of IO device read errors from the kernel.

Manual Ripping

My final try on the Linux machine was to use makemkvcon (the console version) to rip the DVD. ARM installed this, so no install guide here. This worked on Windows, but was very slow. I logged in to SSH as the arm user (which is what ARM runs as, and has permissions to access the drive), and told it to rip the first title of the DVD (usually that’s the movie) to the home directory

makemkvcon mkv dev:/dev/sr0 0 ~

It found the disk, read the titles, and started working. It doesn’t have any sort of progress bar, and it pre-allocates the output file so I can’t look at the file size as an indication of how far along it is, but it seemed to be doing fine. There were no kernel errors at all. Networking about 30-40Mbps consistently from the iSCSI target device, which should mean the file should rip in under an hour (and also 50x faster than on Windows). This is not super fast, but it’s faster than the DVD would play back normally, so it’s reasonable to say this speed limit could be due to the DVD drive. I didn’t buy it for this project, it’s something I’ve had for years to install legacy software, so it might just be slow. Some drives are also known to slow down to only a bit faster than needed for playback when reading movies, for… reasons? As expected by the data rate, the command finished in just under a half hour, giving me an MKV file which played back correctly, with chapter markers and everything. It’s still in MPEG2 format as it was on disk, so it’s huge, but transcoding is something Handbrake can do without needing to access the device.

I also tried running the iSCSI target on a more powerful machine (a dual-core Atom, with USB3 and Gigabit Ethernet), using Ubuntu 20.04 Live Server, and got the same results. This means it’s not (largely) an issue with the Raspberry Pi, but there could be issues in tgt (since I used it for both tests).

As a final test, I installed MakeMKV on the Ubuntu Live Server (the dual-core Atom), and had it extract a smaller 2 minute title from the same disk (since live CDs put the root FS on a RAM disk, it can’t fit the full title). Based on the time it took to extract the 2 hour 30 minute title over iSCSI, I estimated it should take about 25 seconds for the smaller title, and the time was fairly close, so the speed is a limitation of the drive, and not the network or iSCSI overhead (at least with the Linux initiator, and using MakeMKV)

Conclusions

Does this work? Yes. MakeMKV is able to use the iSCSI device on Linux at (what I think) is about the same speed as it would on a physical device. However, not every program was able to access the device, and a lot resulted in transfer errors. I think iSCSI is a valid approach to this problem, but I need a better software stack to implement it (possibly using MakeMKV exclusively to deal with the disk, since it seems to have no problems with iSCSI on Linux). I am also pretty impressed by how well the Pi worked out as an iSCSI target, despite the low power and data throughput.

It’s possible I could abandon ARM and do it on my own. I don’t particularly need all of the features of ARM, and it seems to have gained an absolute ton of feature creep over the years (just look at all the options for notifications, IFTTT, etc.) but the core ripping code is still a pretty simple python script. I could also avoid iSCSI and run MakeMKV on the Pi directly, dumping the ripped file to a network share which the transcode VM scans, but I’d need to keep the temporary files on the network as well to avoid SD card wear. Maybe I just need to fork ARM and modify it to use MakeMKV for DVDs in addition to BDs, and Handbrake for transcoding the resulting file.

Whatever method I go down, I’ll be sure to document it in the future.

The Ultimate OctoPi Setup

Mon, 04 Oct 2021 12:00:00 +0000

In this project, I setup a proper OctoPrint server for my 3D printer, and integrate it into the enclosure I already built. I also add some RGB flair to make it look nice, and set it up to integrate with Home Assistant. I’m very pleased with the results, so follow along for how I set it up.

Building the Circuit

Since I want to use WS2812 LED strips to show the printing status, I need a small circuit. I decided to buy an Adafruit PermaProto with the Raspberry Pi header already included to simplify this process. Since the Pi needs 5V and so do the LED strips, I got a nice 5V 4A power supply brick to feed power in to the board, which will feed on to the LED strips and into the header to power the Pi and USB devices.

I also included a header for a BME280 to measure the temperature and humidity in the enclosure, but I haven’t finished that part yet, so I’ll update this project page when I do.

The 5V power comes in with a female barrel jack and gets connected to the 5V rails on the proto board (which are already connected internally to the Pi’s 5V pins on the header). I have a 4700uF 10V cap here to filter any transients from the LEDs, and because it’s a big cap and bigger is always better. I have a 5V 2-pin header for a fan (to cool the Pi’s CPU), and two 3-pin headers for the two LED strips. Why are there both male and female headers? I built the whole setup with male headers and then realized I couldn’t have male on both end since I bought pre-made 3 pin ‘PWM’ cables (used for hobby servos), so I added the females and left the males.

The signal to the LEDs needs to be buffered, both to protect the Pi’s GPIO pin and to shift the voltage to 5V for the LEDs. I used a 74AHCT125 for this. See the datasheet for the pinout. I connected 5V and GND to VCC and GND, and used two of the buffers for the two strips (in case one shorts out or something). The Pi plugin is only providing a single output on the MOSI pin, so I wired that to both 1A and 2A. I jumpered both 1OE and 2OE to ground to keep the output enabled. The Adafruit guide doesn’t show this, but I found that the enable would float since I had signal wires really close to the OE pins. The resulting 1Y and 2Y outputs go to the signal pins on the two female headers, which go to the LEDs.

On the LEDs, I soldered a 3-pin male 0.1" header, and bought standard hobby servo ‘PWM cables’ which come pre-made in plenty of lengths and of adequate wire gauge to power the 30 LEDs per strip.

Mounting the LEDs

I wanted to mount the LEDs as a bar graph, to show the print progress. I thought about using LED rings, but mounting anything to the door of the enclosure would be more of a cable management challenge. I settled on making right-angle channels to hold the LEDs, bolting them together to make longer lengths than would fit on my printer, and using them to increase the rigidity of the enclosure.

The LEDs I chose are 60 LED/m, WS2812 compatible RGB LEDs. I cut the strip in half, so there are 30 LEDs on each side, wired in parallel, so the bar graphs will show the same things. After printing 6 brackets (each bracket holds 10 LEDs), I assembled the two 30-LED sticks and tested them with a rainbow pattern.

I then mounted them to the enclosure with the same #8-32 fasteners I originally used to build the enclosure. Then, I took a beauty shot with the same test pattern, but the supervisor wanted to be involved.

Setting Up the Pi

The star of the show here (other than the printer obviously) is a Raspberry Pi 3 Model B. The Pi 4 is available, but the Pi 3 has a decent processor and was easier to find in stock at the time of this writing. I’d highly recommend a Pi 3 or Pi 4, as the CPU performance of these is far better than the older Pi’s.

There’s already a distribution of OctoPrint for the Raspberry Pi, named OctoPi. I downloaded the latest image and burned it to an SD card using Etcher. I’m not using WiFi, so I didn’t have to do anything to setup the wireless credentials, but if you are reliant on WiFi, you’ll need to edit a file in the boot partition (Which is the only partition that shows up on non-Linux computers) called octopi-wpa-supplicant.txt to configure WiFi.

Once I powered it on, I had to do some basic housekeeping tasks to get it ready for prime time:

Find the IP address. It responds to mDNS/Bonjour under the name octopi.local, but I went to my router and found the IP address it got over DHCP.
Log in via SSH, and run raspi config. There are 3 things we need to set here - the hostname (change it from octopi so you can have more than one octopi in the future), the local Linux user password (default is ‘raspberry’), and the timezone. Once that is done, reboot
```
 sudo raspi config
 <follow the menu for configuration>
 sudo shutdown -r now
```
Connect to the OctoPrint instance, using the https://IP (note that the certificate is self-signed so it might complain a bit)
Setup the printer - for the Prusa i3 MK3s, the build volume is 210x210x250
Set a custom bounding box with the Y limit at -20, since the Prusa purge line is outside of the build volume and the OctoPrint will complain if the g-code tries to take the printer out of bounds
Create a folder on your NAS to store the data from OctoPrint, so we can avoid writing to the SD card where possible. Ideally all of the g-code, timelapes, etc. all goes to the NAS.

Install autofs so we can mount the NAS on the Pi

 sudo apt-get install autofs
 sudo shutdown -r now

Log in via ssh again and setup the autofs monut. I have a guide on this. I mounted the folder on my NAS for this printer at /mnt/printer with full permissions to that directory for all users. I used the noperm option to disable client-side permissions to the share (so permissions are enforced only server-side), due to issues with OctoPrint and especially OctoLapse trying to modify file permissions on their own. You should only do this if you know you can manage permissions server-side, such as creating a dedicated user on the NAS with permissions to access the dataset for this printer.
(Optional) - You can also add an additional mount at /home/pi/.octoprint/data/backup/ which is where the Backup plugin will store backup zips. There’s no way to redirect the backups folder from within the config.yaml, so you’d need to make a new autofs mount here. In my case, I have a different NAS dataset for backups anyway, so it made sense for me to mount the backups folder separately. I don’t use scheduled backups, but I do manually create a backup (without timelapses or uploads) when I change settings.
Point all of the folders in OctoPrint to subfolders of this mount. Note that you need a dedicated folder on the NAS for each OctoPrint install, since it doesn’t like other programs touching it’s files. It will move files out of the Watched folder and into the Uploads folder automatically, and you should not touch files in the Uploads folder without going through the Octoprint GUI. Mounting these folders from a NAS is really done to prevent wear and usage on the Pi’s SD Card, not to give you better access (other than the Watched and Timelapse folders, where you can copy in/out directly). I found that OctoPrint seems to reset all of the folders to the defaults if the timelapse temp folder is mounted remotely (I’m not sure the triggering condition, but it won’t let you set a remote folder for timelapse temp and setting it from the yaml file will cause it to reset all of the folder settings if you view them from the web UI). OctoLapse does not use that temp folder, only the built-in timelapse, so this might not be an issue for you. You can edit the folder paths from the command line a bit faster than from the GUI:
```
sudo systemctl stop octoprint
mkdir /mnt/printer/logs
mkdir /mnt/printer/timelapse
mkdir /mnt/printer/uploads
mkdir /mnt/printer/watched
nano ~/.octoprint/config.yaml

Replace the folder section with this:
folder:
    logs: /mnt/printer/logs
    timelapse: /mnt/printer/timelapse
    uploads: /mnt/printer/uploads
    watched: /mnt/printer/watched

exit (and save)
sudo systemctl start octoprint
```
In the Folders menu in Octoprint, check ‘Actively poll watch folder’. I found this works better when the watched folder is network mounted.
Go into the GUI and install plugins. I installed all of these:
```
MQTT
HomeAssistant Discovery
WS281x LED Status
```
After they all install, we need to configure them. Go into Plugins and disable the new ones except for MQTT, then restart. We will tackle MQTT first.
Go to Settings, scroll down to MQTT, configure it for your broker information. You should also change the default path, since it assumes you’ll only have one octoprint. I made my path octoprint/columbia/ since my printer is named Columbia, after the first Space Shuttle. Save settings.
Use MQTT Explorer to verify that data is being published in the right place. It should have at least some information under your chosen path.
Go to Settings, Plugins, and enable HomeAssistant Discovery. Save, and restart OctoPrint when asked.
Optionally, go to Settings, HomeAssistant Discovery, and change the Device Name if you have more than one printer, so they don’t collide in Home Assistant
Entities should show up in Home Assistant automatically. Go there and check if it found them.
Go to Settings, Plugins, and enable WS281x LED Status. Save, and restart OctoPrint when asked.
A menu should popup asking you to configure the plugin. It will guide you through the setup process. Once that is done and the hardware SPI is working, you can set the length of your LED string (in my case 30) and test the LEDs.
Once everything is working in OctoPrint, create a backup. I excluded timelapses and uploads from my backups since they are already on the NAS. This will generate a zip file with all of your Octoprint configuration data which you can use in case the Pi’s SD card fails in the future or you have to re-install for any reason.

Setting Up Home Assistant

Since the HomeAssistant Discovery plugin did most of the hard work here, I just had to go in and rename the Device and set the Area for it. Once that was done, I made a simple dashboard to view the 3D printer progress (including my PoE camera). I attached the Lovelace dashboard as a yaml file in the Project Files below. It requires the auto entities card, which you can install through HACS.

If you are using the default OctoPi webcam function, here’s the yaml for adding the camera to Home Assistant. I have mine in camera.yaml, with a pointer in configuration.yaml (camera: !include camera.yaml) since I have quite a few generic cameras. If you don’t have a camera.yaml, you can put this in the camera: section of your configuration.yaml.

#Printer Webcam using OctoPi defaults
- platform: generic
  name: "Printer Webcam"
  still_image_url: 'http://<IP>/webcam/?action=snapshot'
  stream_source: 'http://<IP>/webcam/?action=stream'

Testing It

With everything hooked up, I was ready to test my first print. I used the 3D Benchy, a model used to test 3D printers for their capabilities (overhangs, curves, etc.). I was in the mood for speed, so I used the 0.3mm DRAFT profile.

OctoPrint UI while beginning to print

LEDs while heating to begin a print

LEDs showing print progress

LEDs showing cooldown after print

Future Ehnahcements

I’ll continue updating this project page in the future with some future enhancements. Specifically, I’d like to:

Print a custom case to mount the Pi to the enclosure
Add the Enclosure plugin or something like it for enclosure ambient temp+humid sensing (and hopefully get this data back over MQTT)
Add The Spaghetti Detective, including setting up a local server

The Project Files and Parts List

Here are all of the files and parts required to replicate this project. As usual, all design files are licensed Creative Commons CC-BY-SA unless otherwise noted.

Raspberry Pi 3 Model B+ - I like Adafruit, but there are many other resellers.
WS2812 or equivalent RGB LED strip - I like Adafruit, but there are many other resellers
PermaProto Pi Hat Kit
74AHCT124 Quad Buffer / Level Shifter
Female 0.1" Pin Headers
Male 0.1" Pin Headers
3 pin ‘PWM’ cables for hobby servos
2.1MM DC barrel jack
OctoPrint Website (for reference)
OctoPi Website (for reference)
WS281x LED Status Plugin
MQTT Plugin
HomeAssistant Discovery Plugin
Thingiverse Page
PrusaPrinters Page
HomeAssistant Auto Entities Card
HomeAssistant Lovelace Dashboard for my printer

A Little LED Teaser

Sun, 03 Oct 2021 11:00:00 +0000

I’ve been working on a new OctoPrint system for my 3D printer, and as part of this project I made some nice flashy individually addressable LEDs for the sides of the enclosure to show the print status. The full project page is coming soon, but I just had to give you guys a sneak peek at the LEDs. These are WS2812 style individually addressible LEDs, controlled by an OctoPrint plugin on the Raspberry Pi, along with a perf board PCB I soldered to power and level-shift them.

Experimenting with Kitchen Lighting Automation

Sat, 02 Oct 2021 11:00:00 +0000

I bought a ton of new Z-wave switches and dimmers, and I’m still trying to find the best user interface for automated lighting given the equipment I have, as well as what would be ideal in a new built house. Today, I tried a few automations for my kitchen lighting, which being the center of activity in the house, are the most noticed by everyone, and must work properly at all times.

First things first, what I’m working with to start. The kitchen has 4 can lights, 3 island pendants, and 2 strings of color changing LEDs fixed to blue on top of the cabinets. The kitchen has an open entry from the great room, stairs going to the second floor, short path to the basement stairs and front TV room, a door to the mud room (which is always left open), and a door to the back deck. Electrically, there is a 3-way switch for the can and pendant lights together (no separate control unfortunately), with the passage to the front TV room lackng a switch (which has been an annoyance for a long time). The color changing LEDs are wired to a single switch next to the stove, as the wiring was intended for under-cabinet lights.

The first place to start with any automation is sensing. In this case, I have a mix of motion sensors and door contacts which I’ve chosen for this project. The mud room already has a motion sensor, and I’m going to trigger the kitchen lights when the motion sensor in the mud room is triggered, since it’s likely the kitchen is the next destination. I also already have a door contact (from my Xfinity Sensors project on the deck door, and I’d like to turn the kitchen light on when the deck door is opened or closed (when the state changes, not when the state is open). I also decided to relocate the Inovelli 4-in-1 motion sensor from my Smart Bathroom to the kitchen, since the bathroom was doing fine with cheaper sensors and the Inovelli motion sensor has a sensing pattern that wasn’t very good in the tiny bathroom.

The next step is control. I chose the Inovelli Red Series Dimmer to control the kitchen lights, and was forced to mount it at the bottom of the stairs since that’s the 3-way switch where power enters the circuit. Ideally I could put it on the other side of the kitchen, but I’ll do some magic later to make both switches work. For the over-cabinet lights, I chose the Inovelli Red Series Switch. Because the wiring in the kitchen is bizarre, these lights are actually fed by the mud room power circuit, and are located next to the dumb 3-way switch for the main lights. This means I can use multi-taps of the over-cabinet switch to control the main light switch, or the dumb 3-way switch.

Finally, we have to bridge the two togehter with logic. In my case, there are a few rules for the logic:

Lights should default to a dim level at night, but only well after dark, and should revert to full brightness by sunrise
Lights should turn on automatically if motion is detected in the kitchen, mud room, or the deck door is opened/closed
Lights should have a way to be forced to stay on automatically at full brightness regardless of other commands
Lights should have a way to revert to automatic control
Lights should indicate to the user if they are in forced on mode
Lights should automatically exit forced on mode after several hours if they are not turned off manually
Lights should exit forced on mode if they are turned off manually by either the main or slave 3-way switch
Lights should have a way to be forced on from the other side of the room as intuitively as possible

I ended up breaking this down into a few key bits of automation in Home Assistant:

Group containing the binary sensors which trigger the automated lights
Input boolean containing the state of the forced on override
Automation to turn the lights on automatically
Automation to handle changes to the forced on override state
Automation to handle multi-presses of the button to change the forced override state
Automation at sunrise/sunset to change the default levels of the switch

The code for each of these is below.

1. Group for Sensors

In templates.yaml, I have a template which turns on a binary_sensor whenever the door state changes (and off automatically later). Since the template uses now() it is processed every minute, so the binary_sensor will stay on between 10 and 70 seconds depending on when the processing occurs.

- binary_sensor:
#Indicate if any door switches have **changed** recently (for motion lights) 
  - name: "Kitchen Door Deck Contact Changed"
    unique_id: "kitchen_door_deck_contact_changed"
    state: "{{ (as_timestamp(now()) - as_timestamp(states.binary_sensor.kitchen_door_deck_contact.last_changed)) | float < 10 }}"

In groups.yaml I have the group for the sensors:

#Triggers for kitchen lights
kitchen_light_motion:
  name: Kitchen Light Motion
  entities:
  #Motion sensors in the kitchen
    - binary_sensor.kitchen_motion_occupancy
  #Other sensors which should trigger kitchen lights
    - binary_sensor.mud_room_motion_occupancy
    - binary_sensor.kitchen_door_deck_contact_changed

2. Input Boolean

Just go to Helpers and create an Input Boolean. Mine is named input_boolean.kitchen_light_override

3. Automation to turn on the lights automatically

I can’t use the Blueprint that ships with HA to do this since I’m using a Group instead of a binary_sensor, so I implemented the equivalent manually.

alias: Kitchen Lights Motion
description: Turn on Kitchen Lights based on Motion
trigger:
  - platform: state
    entity_id: group.kitchen_light_motion
    to: 'on'
condition: []
action:
  - service: light.turn_on
    target:
      entity_id: light.kitchen_lights
  - wait_for_trigger:
      - platform: state
        entity_id: group.kitchen_light_motion
        to: 'off'
  - delay:
      hours: 0
      minutes: 4
      seconds: 0
      milliseconds: 0
  - service: light.turn_off
    target:
      entity_id: light.kitchen_lights
mode: restart

You can create a new automation in the GUI and paste this in, and then switch back to the visual editor if you want. Since the mode is ‘restart’, any new motion detected during the delay period will restart the automation and turn the light on again, waiting for the motion sensor to go off again, and the final turn off won’t occur until the delay is able to complete.

4. Automation to handle changes to Override State

This automation runs whenever the input_boolean changes and responds by setting the lights appropriately. I used a choose to implement the turned on and turned off logic, instead of two automations. It could have been done with two shorter automations instead.

alias: Kitchen Light Update Forced On
description: Update the state of Kitchen Lights when force-on is changed
trigger:
  - platform: state
    entity_id: input_boolean.kitchen_light_override
condition: []
action:
  - choose:
      - conditions:
          - condition: state
            entity_id: input_boolean.kitchen_light_override
            state: 'on'
        sequence:
          - service: automation.turn_off
            data:
              stop_actions: true
            target:
              entity_id: automation.kitchen_lights_motion
          - service: light.turn_on
            target:
              entity_id: light.kitchen_lights
            data:
              brightness_pct: 100
          - service: zwave_js.set_config_parameter
            data:
              parameter: '13'
              value: '85'
            target:
              entity_id: light.kitchen_lights
    default:
      - service: automation.turn_on
        target:
          entity_id: automation.kitchen_lights_motion
      - service: light.turn_off
        target:
          entity_id: light.kitchen_lights
      - service: zwave_js.set_config_parameter
        data:
          parameter: '13'
          value: '170'
        target:
          entity_id: light.kitchen_lights
mode: restart

In short, the automation does the following:

Check if the input boolean is currently on or off, and either run the on or off logic (the ‘choose’)
If the override is on:
- Turn off the motion sensor automation and do not let it complete (‘stop_actions: true’) so it does not turn off the light
- Turn on the light and force the brightness to 100%
- Write the parameter on the switch which sets the LED color, and set it to Green (this is Inovelli-specific) to indicate the light is under forced control
If the override is off:
- Turn the motion sensor automation back on
- Turn the light back off
- Write the parameter on the switch which sets the LED color, and set it to Blue (this is Inovelli-specific) to indicate the light is under automatic control

5. Automation to handle multi-presses

Again, I wrote this as a single automation with multiple event triggers. Inovelli switches support up to 5x taps on the up and down button. By default, a single tap is bound to ‘on’ or ‘off’, and the other taps are not bound to anything, but they still generate events of type ‘zwave_js_value_notification’. I capture these events and run logic based on them. I also check if the light is turned off, instead of capturing the single-press off event, so I can correctly capture any events which turn the light off (such as the remote 3-way turning the light off, which doesn’t generate any press events).

For the Inovelli switches, a property key name of ‘001’ is the down button and ‘002’ is the up button. Value will be ‘KeyPressed’ or ‘KeyPressedNx’ where N is 2,3,4,5 for multi-taps. This event relies on the Z-wave node ID, in my case 23 is the kitchen dimmer and 21 is the kitchen over-cabinet switch.

alias: Kitchen Light Double-Tap Handler
description: Double-tap the kitchen light for it to stay on full brightness
trigger:
  - platform: event
    event_type: zwave_js_value_notification
    event_data:
      node_id: 23
      property_key_name: '002'
      value: KeyPressed2x
    id: KeyPressOn
  - platform: event
    event_type: zwave_js_value_notification
    event_data:
      node_id: 23
      property_key_name: '001'
      value: KeyPressed2x
    id: KeyPressOff
  - platform: state
   entity_id: light.kitchen_lights
   id: KeyPressOffSingle
    to: 'off'
  - platform: event
    event_type: zwave_js_value_notification
    event_data:
      node_id: 21
      property_key_name: '001'
      value: KeyPressed2x
    id: KeyPressOffAccent
  - platform: event
    event_type: zwave_js_value_notification
    id: KeyPressOnAccent
    event_data:
      node_id: 21
      property_key_name: '002'
      value: KeyPressed2x
condition: []
action:
  - choose:
      - conditions:
          - condition: or
            conditions:
             - condition: trigger
                id: KeyPressOn
              - condition: trigger
                id: KeyPressOnAccent
        sequence:
          - service: input_boolean.turn_on
            target:
              entity_id: input_boolean.kitchen_light_override
          - delay:
              hours: 4
              minutes: 0
              seconds: 0
              milliseconds: 0
          - service: input_boolean.turn_off
            target:
              entity_id: input_boolean.kitchen_light_override
      - conditions:
          - condition: trigger
            id: KeyPressOffSingle
          - condition: state
            entity_id: input_boolean.kitchen_light_override
            state: 'on'
        sequence:
          - service: input_boolean.turn_off
            target:
              entity_id: input_boolean.kitchen_light_override
    default:
      - service: input_boolean.turn_off
        target:
          entity_id: input_boolean.kitchen_light_override
mode: restart

The automation first sets up 5 triggers which are named for the future. Double-tap up on either the kitchen dimmer or overcabinet switch, and change of state of the light to off (by any means).

If a double-tap up is pressed (either switch), the light override is turned on, and then after 4 hours it is turned back off. Since the mode is restart, any other events will stop this delay.

If the light is turned off for any reason and the override was already set on, the override is turned off to return to automatic control.

If a double-tap down is pressed (either switch), the default case runs and the override is turned off.

6. Sunrise and Sunset

I already have two very long sunrise and sunset automations which set up the lighting brightness and enable/disable automations, so I wrote parameters 9 and 10 (default level and default level Z-wave respectively) on the dimmer in these automations. These parameters are Inovelli specific, and even specific to the dimmer (for the LZW36 fan+light model, the parameters are 12 and 13).

After all of this, I’m fairly satisfied and might expand this UI concept (green means forced, blue means automatic) to the rest of my automated lighting. The RGB indicators on Inovelli switches really make this nice.

Integrating Security Camera Motion Detection with Lighting Control

Fri, 01 Oct 2021 01:00:00 +0000

As I expand the reach of Home Assistant, I continuously try to build automations that make life generally easier for the users of the home. To me, automation isn’t about being able to control anything from my phone - in fact, the less I have to get my phone out, the better. I will still enjoy tracking history entries and status of nodes with both the web UI and app, but I shouldn’t have to, the house should just work. With all of this, my next goal is to merge automated lighting with security cameras, to improve the user experience when coming home at night. This is an all-encompasing goal that includes motion sensing lighting with PIR sensors, door switches, and of course Frigate’s AI person detection.

We’ll Leave The Light On For You

Without any way to control exterior lighting from outside the house, this is a very common occurance when someone is out late. If someone is coming home and parking in the driveway, they will be wandering around in the dark trying to find the back door, or if they open the garage door, dealing with the dim light of a single bulb in an overhead door motor to see where they are going. It’s not a great user experience. The age-old solution to this is to leave the exterior lights on all the time, so anyone arriving after dark can see where they are going and find the back door. Even when they do find the back door, they enter the kitchen, and need to find the switch for that light (which is not near the exterior door), and the process continues with lights being turned on and off until they make it to their bedroom.

I’m approaching this in a multi-pronged way. I need more sensor information to make informed decisions about lighting, and I need more control of the lights. I’ve started with the following goals for automation:

The back deck exterior lights and deck rope lights shall turn on when persons are detected on the deck or in the driveway next to the garage between sunset and sunrise
The front and side exterior lights shall turn on when persons are detected on the deck, in the driveway next to the garage, in the front yard, or on the front porch, between sunset and sunrise
The front and side exterior lights shall turn on when any garage overhead door is opened, between sunset and sunrise
The garage lights shall turn on when any garage overhead door is opened, between sunset and sunrise
The mud room lights shall turn on when the garage to mud room door is opened or when motion is detected, at all times
The kitchen lights shall turn on when the deck door is opened, motion is detected in the kitchen, or motion is detected in the mud room, between sunset and sunrise, and automatically dim based on the time of day if on.
The second floor stair lights shall turn on when motion is detected, between sunset and sunrise, and dim based on the time of day if on.

Detecting if a door contact recently changed

One key here is that I don’t want the lights to stay one when the door is open, I want them to turn on when the door is opened and stay on for a few minutes. Since closing the door usually follows, but closing the door also indicates a person is in the vicinity of the door, I’m okay triggering when the door opens or closes, but not when it remains in the opened state. To do this, I wrote a template sensor for each door switch which triggers if the value has changed in the last 10 seconds. Since triggers are evaluated whenever a corresponding entity changes, but triggers containing now() are also evaluated every 60 seconds, this means the resulting binary sensor will stay true for between 10 and 70 seconds depending on when the condition is re-evaluated. This is acceptable to me. The following needs to go in your templates.yaml:

#Indicate if any door switches have **changed** recently (for motion lights) 
- binary_sensors:
    - name: "Kitchen Door Deck Contact Changed"
        unique_id: "kitchen_door_deck_contact_changed"
        state: "{{ (as_timestamp(now()) - as_timestamp(states.binary_sensor.kitchen_door_deck_contact.last_changed)) | float < 10 }}" 
    - name: "Garage Door Entry Contact Changed"
        unique_id: "garage_door_entry_contact_changed"
        state: "{{ (as_timestamp(now()) - as_timestamp(states.binary_sensor.garage_door_entry_contact.last_changed)) | float < 10 }}"       
    - name: "Front Door Contact Changed"
        unique_id: "front_door_contact_changed"
        state: "{{ (as_timestamp(now()) - as_timestamp(states.binary_sensor.front_door_contact.last_changed)) | float < 10 }}"

If you don’t have a templates.yaml, see if you have a templates in your configuration.yaml and add them there. If not, make a new templates.yaml and point to it from configuration.yaml like this:

template: !include templates.yaml

Setting Up Motion Groups

The easiest way to implement this in Home Assistant is to create groups for each of these events (groups OR the entities they contain by default), and use that group as the trigger for the motion sensing lights automation. In my case, I started with a group for the back deck lights based on person detection from the 3 relevant cameras, and a group for the mud room lights based on the door and motion sensors. These go in groups.yaml:

#Motion sensors which contribute to Deck Lights
deck_light_motion:
    name: Deck Light Motion
    entities:
        - binary_sensor.back_door_person_motion
        - binary_sensor.parking_person_motion
        - binary_sensor.driveway_person_motion
        - binary_sensor.kitchen_door_deck_contact_changed

#Sensors which contribute to mud room lights
mud_room_light_motion:
    name: Mud Room Light Motion
    entities:
        - binary_sensor.garage_door_entry_contact_changed
        - binary_sensor.mud_room_motion_occupancy

Obviously, more motion groups exist for all of the scenarios above, but this is an example.

Setting Up Lighting Automations

Although there is already a blueprint for motion-activated lights, it’s very picky about what it accepts. In my case, the issue is it will only accept a Light entity, not a Switch entity, so I can’t use it since I use Switches (Inovelli Red) for my exterior lights. I could use it for my interior lights which have Dimmers (and are Light entities), but it’s not a terribly hard automation to write manually.

So, the automation goes something like this (using the GUI):

Mode - Restart - means it will start the timer over when new motion is detected before the lights have turned off automatically
Triggers - State - (group name), to on
Conditions - none
Actions:
- Turn on light (using either Call Service or Device)
- Wait for Trigger - State - (group name), to off, no timeout
- Delay - the time you want the lights to remain on after motion is not detected
- Turn off light (using either Call Service or Device)

Copy that automation for every case where you want to use it.

Setting Up Sunrise/Sunset Actions

I setup two automations, one which runs at Sunrise + 30min, and one which runs at Sunset -30min, to setup the house for day and night time activities. These two automations are exactly the opposite of each other, and basically do the following:

Turn on/off all of the Automations which are responsible for daytime only motion sensing (bedrooms)
Turn on/off all of the Automations which are responsible for nightitme only motion sensing (garage, deck, kitchen)
Set parameters on the dimmers for the default dim level, to either 100% (daytime) or the night time % selected for that room, duplicated for both the local on and Z-wave on parameters - for my Inovelli dimmers, this is parameters 9 and 10 respectively (parameter 12 and 13 instead for Inovelli LZW-36 dual fan light dimmer))

Results

So far, I’ve been running this for a few weeks with a few different lighting groups - one for the back deck lights and one for the front porch lights. So far, feedback has been excellent. The front porch and garage exterior lights come on when cars or people are detected in the driveway, so they come on quickly as someone arrives. The back deck lights come on when people are detected on the deck or in the driveway, so they come on with slightly more filtering compared to the porch lights. There are relatively minor issues in dealing with what happens when the automation is turned on when the lights should be on - I need a bit more logic to decide if it should turn the lights on immediately when the automation becomes active. The interior kitchen lights are also starting to be automated, more on that in the future.

Temperature and Humidity Sensor Showdown

Mon, 27 Sep 2021 01:00:00 +0000

When I started my bathroom automation journey, I used the Inovelli 4-in-1 Motion Sensor (LZW60) which had sensors for motion, temperature, humidity, and ambient light level in my Smart Bathroom Project. I was happy with the automation, but wanted to try out some cheaper sensors to see if they were adequate for the other bathrooms in my house. I decided to try the Aqara (Xiaomi) Temperature and Humidity Sensor and the Sonoff SNZB-02 Temperature and Humidity Sensor as cheap alternatives for temperature and humidity. I bought several of each sensor on Aliexpress, and set them up in my bathroom to compare data.

Unboxing

The unboxing experience for each sensor was very different.

The Sonoff sensor comes in a box barely bigger than the sensor itself, along with a square of double sided foam tape to mount the sensor, and a small paper manual. The packaging is efficient and functional. You must pry off the back of the sensor using a small flat blade screwdriver (I was worried I’d break it) and remove a piece of paper covering the battery terminals, then it’s ready for pairing. To pair, you can hold the button on the sensor with your finger for 5 seconds or so and it will start the process.

The Aqara sensor comes in a much larger box. The sensor itself is fairly small, but it’s sitting in a white paper board stand which is actually entirely empty inside. The sensor comes with a pre-installed foam tape ring, and they include an extra one in the box, along with a small paper manual. The box could have easily been half the size or less and still fit everything inside. I mention this because I find the large white interior box very wasteful, and it adds up when you buy dozens of sensors per house. The inclusion of an extra foam tape ring was a nice touch though. I was unable to remove the battery cover easily with the screwdrivers I had on hand, it seems to be designed to use a really wide blade (such as a butter knife). However, there is no paper covering the battery terminals, so it’s ready for pairing out of the box. Unlike the Sonoff sensor, the button is not readily accessible, and must be poked through a hole using a bent papercip or similar tool.

The Aqara sensor is significantly smaller than the Sonoff sensor, although neither is particularly large. Both would be relatively unobtrusive mounted on the ceiling of a room.

Test Setup

Both sensors were mounted on the ceiling in between the shower and the bathroom fan, with the ventilation holes in both sensors facing the shower. This gives these sensors a significant advantage over the Inovelli sensor, which is mounted on the opposite corner of the room for good motion view.

Both sensors were paired in Zigbee2MQTT through the same repeater, and added to Home Assistant. Both had approximately 70 LQI (link quality). Both were left with all default settigns. Both sensors report all of the same data (Battery, Voltage, Temperature, Humidity, Linkquality), with the Aqara sensor also reporting pressure. I did not include pressure data or long term battery life in this comparison, only temperature and humidity reporting. The Inovelli sensor is paired with ZWaveJS2MQTT and has the reporting interval set to 1 minute (not the default), which is how it was already configured from my Smart Bathroom Project.

The Test

All three sensors ran for a week to compare results. During this time, control of the bath fan was still done by the Inovelli sensor, with the other sensors only used for logging. Data from a single shower is shown below, for analysis.

A few conclusions can be made from this data:

All of the sensors report the same temperature steady-state, before the shower.
The Sonoff sensor reports slightly higher humidity than the other two sensors at all times, but this is within the tolerance of all of the sensors.
As Zigbee2MQTT’s documentation warns, the Sonoff sensor has a very slow reporting interval for temperature, but a fairly quick reporting interval for humidity (which seems odd). This is confirmed with this data, where the Sonoff sensor reports no temperature changes for over a half hour, but continues reporting humidity changes.
The Aqara (Xiaomi) sensor also does some stair-stepping in reporting with both temperature and humidity, but the response seems closer for both signals than the Sonoff, and is fast enough for bathroom humidity control.
Neither sensor is fast enough for closed-loop temperature control.
Due to their location much closer to the shower, the Sonoff and Aqara (Xiaomi) sensors both report humidity much faster and reach much higher numbers than the Inovelli motion sensor on the far side of the fan from the shower. This also causes the humidity to decay much slower from these sensors than the Inovelli, as air enters the bathroom near the Inovelli motion sensor and exits through the fan, while the shower area itself is essentially not ventilated.
While not part of the showdown, I also included a Sonoff SNZB-03 motion sensor mounted between the two temperature + humidity sensors, and found that it performed better than the Inovelli at detecting motion in the bathroom. This is due to the very small size of the bathroom and the detection pattern of the Inovelli being designed for wall mounting (with a half-cone shape), while the Sonoff SNZB-03 has a full cone shape for ceiling mounting. In the tiny bathroom, the ceiling mount sensor is a better choice as it can ‘see’ into the shower and not turn the lights off while showering as the Inovelli would. I don’t believe this is a fault of the Inovelli sensor, but you do have to be aware of the detection pattern when selecting a sensor for wall vs ceiling mounting.

The Parts List

Here are all of the files and parts required to replicate this project.

A Quick 3D Printer Box Update, and a Raspberry Pi Case

Fri, 24 Sep 2021 13:06:32 -0400

I finished the box! At least I finished it enough to start using it. It still needs OctoPrint, lighting, and some RGB LEDs (doesn’t everything these days?). But, it’s usable now. Check out the Project Page for the full story.

The New Box

In addition, I’ve started designing a Raspberry Pi case for my OctoPrint Pi. First revisions are shown below. I thought I’d start with this case and re-print it as I add things to the enclosure, but I think I’m going to go caseless for now and work up to what I want the final design to be. This is the first Pi case I’ve printed, and I’ve learned a bit about fitment for the Pi along the way. I designed around the Rasbperry Pi 3, not the 4, since that’s what I’m using for my printer.

Case, With Pi

Case, Closed

The biggest issues are:

The USB ports do not clear the enclosure, they need another mm or two. I should have accounted for the flange on the ports more carefully. The opening for the ports needs to be wider
The pegs which the Pi sits on are quite a snug fit, and I don’t know if I’d be able to get the Pi back out without breaking it. I need to loosen these up. I can secure the Pi with hot glue if necessary. The holes are spec at 2.75mm, my pegs are 2.6mm, I’m going to go down to ~2.25mm ish.
The analog audio jack does not clear. I can either move the Pi up by a bit, or make the walls thinner to leave a bit more clearance. I don’t really need 5mm thick walls, so reducing these to 3.5mm should leave enough clearance for the audio jack. I don’t actually use the audio jack, so it doesn’t have a hole through the enclosure
The hexagonal hole for the power port is designed to avoid printing a square hole, which will cave in a bit without support, but I realized later that I can make it slightly larger to clear both the micro-USB on the Pi 3 and the USB-C on the Pi 4, making the case universal.
I printed the lid with a very small undersize between the wings, which are supposed to friction fit to the case. Depsite making this a fraction of a mm, the case is very loose, and I should print it exactly to size next time.

USB Connector Interference

In the future, I plan on expanding my OctoPrint setup and printing a new case dedicated to whatever electronics I come up with. Some things I’d like to include:

Large 5V power supply for LEDs and the Pi
Either all WS2812 type individually addressable LEDs, or MOSFETs to control the enclosure white LEDs plus WS2812 status indicators
Temperature and humidity sensor in the enclosure
Fan for the Pi and MOSFETs
Case design to support Pi 3 or Pi 4 with no mods (although the HDMI ports won’t be accessible in either case, since you don’t need them with OctoPi)

Follow along for more updates on this and other projects!

Andrew

Diskless Windows Desktop using PXE with a Linux backend

Tue, 21 Sep 2021 01:00:00 +0000

Inspired by recent a recent video on the basics of PXE booting by ‘Tall Paul Tech’ (formerly known as CWNE88), as well as a comment by Linus of Linus Tech Tips that his new home server could ’network boot everything in his house’, I wondered how easy it would be to network boot everything in my house. In an ideal world, this would solve a lot of problems regarding managing backups of the drives - by simply not having drives at any client, they can all be managed and backed up centrally by the server. I’ve already come to love the flexiblity in separating storage from compute in a virtualization environment, where I can use a ZFS backend to deal with snapshots, replication, and file integrity checking, and I’ve upgraded the network speed for my workstation so I can work on files directly off the network using mapped network drives, but this still leaves my primary boot disk vulnerable, and while my files are safe, my installed programs are not. I could certainly recover from a boot disk failure without data loss, but it would take a while to get everything re-installed again. So, if I could keep the boot disk on the storage backend, it would gain the integrity protection and snapshots that the data has. Being that Linus will only be concerned with gaming, it’s probably important to be able to boot Windows, as unfortunate as that sounds going in to this project.

The Basic Design

In short, we are going to rely on PXE to get the system off the ground, and then load iPXE (a better PXE) using the ‘chainloading’ method (where traditional PXE is used to load the improved open-source PXE), which we will then use to boot off an iSCSI target containing a Windows disk. If we had Linux clients, we could rely on NFS instead of iSCSI for the root filesystem, and we would curse a whole lot less.

So, there are a few components needed here

The DHCP server, which at a minimum has to point to the PXE server (the ’next-server’ directive as well as the boot file directive).
The TFTP server, which stores the files used by PXE to load iPXE
The HTTP server, which stores additional files used by iPXE to load the operating system, although not required after the bootable iSCSI drive is configured. This could also be expanded to serve custom iPXE scripts to each client based on their MAC address or other unique identifiers.
The Samba server, which stores the Windows installation media, not required after installation
The iSCSI target, which stores the virtual block device used to store the Windows C drive

To implement this project, for testing, I’ve created an Ubuntu 20.04 VM on Minilab to act as the server for TFTP + HTTP + Samba + iSCSI, and I have an old amd64 machine which will act as the client. It physically has no hard drive installed. It’s an old Atom processor in an ITX embedded motherboard, but it should be enough to prove that this method works without buying any hardware.

Setting Up the DHCP server

This one is pretty simple. I use OPNsense as my DHCP sever for my existing network, so I just had to go down to Network Booting, Enable Network Booting, and set next-server IP and boot file name. I set the DHCPv4 and DHCPv6 server to have a static reservation for the Ubuntu VM, so it will have a consistent IP without setting a static IP inside the VM, and set next-server to that IP. So, clients will now look to the Ubuntu server for their PXE boot files. I just need enough configuration in the DHCP server to get the dumb PXE clients to load iPXE which is compiled with an embedded script pointing it to the HTTP server, where it can go for the rest of its configuration.

What happens at this point? If I power on the test system with next-server set, and it tries, but there is no TFTP server for it to find. In this case, we want to boot iPXE, which is a better PXE environment than the default, and will hopefully let us boot from iSCSI.

Setting Up the TFTP server

This one is also pretty simple. In my Ubuntu VM, I installed it from apt and that was that.

sudo apt-get install tftpd-hpa

By default it places the data at /srv/tftp, but the config file is located at /etc/default/tftpd-hpa if you’d like to change that. I’m fine with the default location.

I added a blank file called undionly.kpxe to the tftp folder and set that as the boot file name for BIOS in the DHCP settings (the name will become relevant later). I booted my test system (which is old enough to use legacy BIOS) and it attempted to load the file but couldn’t boot it, since it’s just a blank file.

Setting Up the iSCSI Target

If you’re doing this for a ‘production’ environment, you should use something like TrueNAS and setup your iSCSI target there. I’m not an iSCSI expert by any means, but I got this to work. The whole point of this project is to try new things.

First we need to install the iscsi target on Ubuntu:

sudo apt-get install tgt

Then we need to setup an iSCSI target.

sudo nano /etc/tgt/conf.d/target01.conf

And the contents:

<target iqn.2021-09.net.apalrd:win10>
    backing-store /srv/disks/win10.img
</target>

And we need a blank image file:

fallocate -l 20G /srv/disks/win10.img

Restart the server

sudo systemctl restart tgt

Look at the status

sudo tgtadm --mode target --op show

So now we have an iSCSI target which we can use to store the Windows C drive, but nothing is installed on the disk image.

Setting up iPXE

We would like to use iPXE to get the process rolling. So, cd to a place you can keep a git repo, and clone it:

git clone git://git.ipxe.org/ipxe.git

Follow the instructions here to make sure you have the dependencies installed so you can build

Then we need to create an embedded file so it will run a script when it boots:

cd ipxe/src
nano boot.ipxe

And the contents:

#!ipxe
#Setup networking
echo Setup Networking
dhcp
#Boot from SAN
echo Boot from SAN
sanboot iscsi:<IP>:::<LUN>:iqn.2021-09.net.apalrd:win10

Although all of the examples for ipxe show LUN as blank (iscsi::::iqn…), I found that I needed to set LUN to 1 with the default configuration of tgt. It seems like LUN 0 is always a Controller, with LUN 1 being the Disk. Not sure why exactly this is, but tgtadm shows the LUN numbers for each target correctly. Additionally, I had trouble with Windows giving me an ‘INACCESSIBLE BOOT DEVICE’ error, which I’ll get to in a bit.

Then we can build ipxe with our embedded script:

make bin/undionly.kpxe EMBED=boot.ipxe

Then finally we copy that into our tftp folder so clients can find it

cp ./bin/undionly.kpxe /src/tftp

Setting up Apache2

Since TFTP is pretty ugly, I installed Apache2 to serve files over HTTP for the installation process. This isn’t used after install, but it could be used along with PHP / some scripts to send more complex commands to iPXE once it’s been loaded via TFTP. There are better guides than mine out there, so just setup a basic apache2 with the http root in /srv/html with full read access to anyone.

Setting up Samba

Again, we need Samba for the install process so I installed it with no permissions and mounting a share ‘server’ at location /srv. This also lets me copy files to the http root, tftp root, and iscsi disks folder from my Windows workstation as needed.

sudo apt-get install samba
sudo nano /etc/samba/smb.conf

Add the following lines to the end:

[server]
  comment = Server directory
  path = /srv
  browseable = yes
  guest ok = yes
  read only = no
  create mask = 0777
acl allow execute always = True

Then restart Samba

sudo systemctl restart smbd nmbd

Installing Windows

I first tried to install Windows in a VM, then copy that VM to the win10.img disk that’s being shared over iSCSI. It was not properly configured for iSCSI and gave me the ‘INACCESSABLE BOOT DEVICE’ error. So, I decided to install it over the network.

I chose to go with the ‘wimboot’ method, booting WinPE (windows preboot execution environment) according to this guide. I modified my boot.ipxe temporarily while installing, created the folder structure using copype, and moved it to my HTTPS directory. Since I was embedding the script instead of loading it from HTTP, I modified all of the paths to include http://<IP>/ in front of them. The resulting script is:

#!ipxe
#Start networking
echo Start Networking
dhcp
#Set web root
set webroot http://<IP>
echo Webroot is ${webroot}
#Set architecture
cpuid --ext 29 && set arch amd64 || set arch x86
echo ARCH is ${arch}
#Sanhook the Windows drive
echo Attaching iSCSI drive
sanhook iscsi:<IP>:::1:iqn.2012-09.net.apalrd:win10
#Load wimboot
echo Loading Wimboot
kernel ${webroot}/wimboot
initrd ${webroot}/${arch}/media/Boot/BCD        BCD
initrd ${webroot}/${arch}/media/Boot/boot.sdi   boot.sdi
initrd ${webroot}/${arch}/media/sources/boot.wim    boot.wim
boot

Of course, you need to make ipxe again and copy the new undionly.kpxe to the tftp root.

Now we need a Windows install to install from. Download the Microsoft media creation tool and use it to create an ISO image. I then unzipped the files on my Windows computer using 7-zip, dropping them in the ISO folder.

Now boot the client. It should boot into a Windows graphical command prompt. We now need to connect over Samba to the location of the Windows ISO and run the installer. Be warned, it’s a very slow process, don’t expect anything to happen instantly.

net use \\<ServerIP>\server
\\<ServerIP>\server\html\iso\sources\setup.exe

Since we told Samba earlier that executing anything is fine (who needs security when you’re prototyping), we can execute the installer. We now get a normal-ish Windows GUI installer, loaded over SMB, from the WinPE environment we network booted over HTTP, from the iPXE environment we loaded over PXE, with no physical storage or disks at all on the client. Not bad.

Unfortunately, this win was short lived, as I couldn’t get the installer to partition the iSCSI drive. It found it, and it was blank, but clicking ’new’ would let me go through the process of creating the partition, and then it would hang when actually creating it. Unfortunately, this is the end of my adventure, as I’m sick of dealing with Windows 10. I’ve read (relatively few) guides of this working correctly in Windows 7, but it seems to be a fairly unpopular solution in the modern era.

Conclusions

Is this a good idea in a homelab in 2021? Probably not the best idea. There are far better options for VDI than network booting a Windows installation over iSCSI. Is it an option for the right scenario? Definitely. PXE has its place, and it’s definitely been a fun project to learn about it. The key lesson I’ve learned is that it should be used when you are booting a read-only something, and a whole lot of that something, to justify setting up and configuring a server. If you just want to network boot one media center PC to avoid buying an SSD, this is probably not the solution for you.

Where does network booting have a place?

For installers, where you network boot into a menu to select which installer you want to run off a network store, and install something on the computer, setting up a PXE server and all of the associated ipxe configuration is fantastic. I found a great example file for that here, although you could easily use an embedded script in ipxe to avoid writing rules in the DHCP configuration. You also have the option of identifying clients by MAC or other unique address to serve the right installer.
For an application like a computing lab, you could use Linux and mount the root filesystem read-only using NFS, which lets you deal with common ancestry and deduplication at a file level instead of a block level as in iSCSI. You could also mount the home directory read-write using NFS separately, so every computer boots the golden root image and any changes are lost when it reboots (except the user’s home directory). Since all of the computers are physically identical, drivers and such should be the same.
For a server cluster, it’s not a bad way to distribute the cluster OS and root filesystem either. Again, you’d setup a ‘golden’ image, mount that over iSCSI or NFS, and boot into it. The applications in the cluster must all rely entirely on network storage for this to work, which is a very reasonable expectation of a cluster, and the server would need to configure its networking based on its MAC address. Using DHCP static allocations would be easeiest for this, although if they are all nodes in a cluster they might not need reserved IPs at all and plain DHCP might be fine.
For home use such as media centers or wall mounted tablets, it might be a decent idea to network boot into a Linux OS which again mounts a read only filesystem designed for this purpose, loading the media over the network. For this to be viable you really need a decent number of nodes, and you really need to be running Linux, which is much less of a headache to network boot. The new Raspberry Pi 4 keeps the first stage boot image in EEPROM instead of on the SD card, meaning you can reprogram it to look for other boot devices, including PXE. The older Pi’s would need an SD card to store the first stage boot image, which could then go out and network boot, leaving the SD card otherwise unused. That said, simply booting a read only filesystem from the SD card and loading the media and configuration off the network would be easier to setup and easy to replace if it breaks.
For a home lab which uses GPU partitioning, where you lose the ability to use the physical outputs on the GPU (or use GPUs which never had them), you could network boot a Linux image on all of your computers which in turn launches Parsec as the only graphical environment, to connect remotely to the host on the compute server.
While researching this, I found a number of companies offering software to centrally store the boot disks of Windows corporate workstations, so the company could keep classified data safe overnight without manually removing all of the hard drives from the computers. By removing the drives from the server or keeping it physically secure and disconnected overnight, they could network boot all of the workstations and reduce the chance of physical data theft. There is also some advantage in doing this that you can make a golden corporate image and clone the image for each employees workstation image (or, if you’re really paranoid, wipe them overnight back to the golden image), reducing space on the server since they are all based on a common ancestor, keep central backups of the images, and let any physical workstation boot to anyone’s personal workstation image as required, so no more cloning hard disks when replacing desktops. If your workflow is already heavily based on keeping files on the server and only programs and customizations are stored locally, this could work very well. Linus (of LTT) might be better off network booting all of his editing workstations rather than his living room gaming computer.

What’s a better option for a home lab?

For a media center or home theater that also does gaming, using VFIO (PCIe passthrough or SR-IOV) to a GPU in a compute server and connecting that via long distance HDMI / DisplayPort / USB3 active optical cables is probably cheaper and less power intense way to get 4K/60 video to the client. You’ll be able to use the compute resources you already have in the rack, and unless you’re getting small form factor desktops which can fit a GPU in bulk, probably save money by only adding the GPU to an existing compute node. If you are gaming, you probably don’t want to deal with the latency of streaming over the network, although apparently Parsec isn’t bad (I haven’t used it myself). The downside is that there is no standard for connecting any of these things over networking fiber, so your cheapest option is usually an active fiber cable instead of using standard networking fiber, which means pulling wires which are essentially already dated when installed (i.e. HDMI 2.0 = no 8K without pulling new wires). You also can’t partition the GPU if you use the physical video outputs, since GPU partitioning was never designed to be used with consumer GPUs in the first place (on either the AMD or nVidia side), and the tricks used to partition Geforce GPUs essentially make the drivers believe it’s an equivalent Quadro or Tesla with the same silicon.
Give each user a laptop with a locally installed OS and let them connect to hosts in the compute node (which can use GPU partitioning) for workloads that need more compute or GPU performance.
For a dedicated single user workstation, using dedicated local storage still isn’t an awful choice, it just makes backing up the OS drive more difficult than network booting. The same goes for laptops.
For both laptops and desktops, you have the option of installing the user’s OS inside a hypervisor and using the hypervisor to manage backups. You may use XCP-NG or Proxmox, pass through the physical GPU to the guest, mount the guest from local storage, and have the hypervisor snapshot and push the backup to the server periodically. You’ll have less space available to the guest due to snapshots, but SSDs are cheap enough now. Re-installing the hypervisor on disk failure and then restoring a backup of the guest disk is probably easier than re-installing the guest OS and all of the programs on it, even if you have good data backups. For this to work, you can’t pass through the entire disk to the guest, since you need to manage snapshots through the hypervisor.
Installing Windows programs on a network share is possible, with the right permissions in Samba. You can also locate your Steam library on a network drive. Steam games are very easy to reinstall, but tend to be massive, so keeping them backed up locally just saves internet bandwidth if you have a drive failure.
For macOS, apps are considerably easier to backup since they are usually part of an App Bundle which can be copied freely, so backing up the entire Applications directory isn’t really difficult. There are certainly some apps which don’t respect the App Bundle, but it’s a lot easier than on Windows.
For Linux, writing an Ansible playbook to install your normal desktop apps on top of a fresh distro install is a pretty decent solution.

I hope to explore some of these options in a future adventure.

Home Automation Christmas in September

Mon, 20 Sep 2021 11:00:00 +0000

It’s always a good day to receive new hardware, and today is no different. Over the past few days, I’ve received a bunch of hardware in the mail and I’d like to share my plans with you. Ever since I setup my first automations with my blinds and started automating my lighting and bathroom, I’ve been addicted to automating more and more of the house. So, after spending a lot of money on high end Z-wave motion sensors, dimmers, and switches, I went searching for some more cost-effective products to try out. WiFi, Zigbee, Z-Wave, and 433Mhz (sensors only) were all on the table. At this point, I have no data on how well any of these products work. I will start incorporating them into projects going forward, so follow along for the long-term data on how these end up working out! Here’s the total haul that came in over the last few weeks:

Inovelli Z-Wave Dimmers and Switches

I’ve already used the Inovelli LZW36 dual fan+light switch module, which has an RF controlled canopy module allowing control of the fan and light without dedicated power wires for the fan and the light. I also have an LZW60 4-Way sensor (Motion, Light, Temperature, Humidity), which is used in my Smart Bathroom. After being impressed with this hardware, I decided to buy some of the backordered Inovelli red series dimmers and switches (LZW31-SN Dimmer and LZW30-SN Switch).

I only ordered two dimmers, so I’ll have to plan carefully where I put those, but I bought a 10-pack of switches, so I’ll have plenty of places to put those if I can get away without dimming. I already want to switch the exterior lights based on Frigate motion events, so we don’t have to ’leave the lights on’ when someone is out late. I’m sure I’ll find more uses for the dimmers and switches.

Sonoff (ITEAD) Zigbee Products

I’ve used Sonoff products before, but only their ESP based products, which I flashed with Tasmota. This time, I went specifically for Zigbee. I’d still recommend their Wifi smart plugs (especially the S31 with power monitoring if you’re in the US), but for sensors I’m willing to tolerate Zigbee for better battery life over WiFi. They had a sale on Zigbee products, so I went a bit overboard and bought a lot. I bought directly from ITEAD and shipping via DHL was less than a week.

Overall, I got 3 products:

My plans for the products, and why I chose them:

I’m trying to try a wide variety of low cost sensors to collect as much data on their reliability / functionality as possible before outfitting my future house with a ton of them. The SNZB-02 will go head to head against the much more expensive Z-Wave motion and more sensors (Aeotec 6-way multisensor and Inovelli 4-way multisensor) and the Xiaomi Aqara temperature + humidity sensor.
The IKEA Zigbee motion sensor I have in my bedroom was out of stock, and I want to do more motion controlled lighting. The Sonoff SNZB-03 motion sensor also looks better (it’s physically smaller at least), is physically symmetrical, and has a cone of view optimized for ceiling or wall mounting, whereas the IKEA sensor was designed exclusively for wall mounting. This is important for stairway lighting where the sensor will be above the stairs.
I have more bathrooms with combined fan+light on a single switch that I want to separate into control of the fan and control of the light. The Inovelli LZW-36 fan+light switch I already used is out of stock and not expected to return until 2022, and want to see if I can setup something as functional for less money for the less frequently used bathrooms. I plan on combining the ZBMINI with a smart bulb and a switch on the wall, and may experiment with Zigbee binding so it works during a hub failure.
I didn’t buy the SNZB-04 door contacts from Sonoff, but maybe someday I’ll compare them against the Aqara door contacts. Pricing is about the same for both of them.

Xiaomi Aqara Zigbee Products

Xiaomi is a well-known Chinese brand, and they have a sub-brand called Aqara for some of their home automation products using Zigbee. I purchased their sensors to compare against what I already have from IKEA and Sonoff. Since I went through Aliexpress, I had to wait a bit longer for shipping, but they were significantly cheaper than Amazon and were price competitive to Sonoff, with cheaper shipping, so they’re a great option if you plan ahead enough to be able to wait. I got a bit of a different variety here:

Door And Window Sensor
Water Leak Sensor
Temperature, Pressure, and Humidity Sensor

The box for the Temperature + Humidity sensor contains the sensor (which is really very small), small manual, and an extra foam tape ring (there’s already one attached to the sensor), but the whole white paper box inside is empty, meaning the packaging could have been considerably less wasteful. I setup the first sensor and paired it with my Zigbee network to verify that it works as expected, and it does.

My plans for the products, and why I chose them:

The Aqara sensors seemed reasonably priced for door contacts, and they visually look better than the Visonic ones I have already (leftover from Xfinity, so I didn’t spend money on them). I’m going to try these out with overhead garage doors as well. No temperature data from these, although it’s not really necessary anyway. The Sonoff SNZB-04 door contact doesn’t have temperature either.
The Temp + Pressure + Humid sensor will have a showdown with the similarly priced Sonoff model, which doesn’t include pressure. I don’t expect pressure to add a lot of functionality to the sensor. Additionally, although neither this sensor nor the Sonoff model are listed for outdoor use, this one has a narrower temperature range.
The Water Leak Sensor is one of the cheapest water leak sensors available, so I thought I’d give it a try. As I mentioned in my Septic Control Project, we have previously had water backups in the basement as well as condensate drain failures leading to water on the floor, so I’ll put the 3 sensors I got in the basement and laundry room and see if they ever trigger.

Acurite 433Mhz Outdoor Temperature Module

I already put this one to use in the 433Mhz Sensor project, just a day after receiving it. Usually hardware sits on the bench for awhile before I get around to finishing a project, but the turnaround here was fast enough that the project page is already done before this blog entry. These cost $13 each plus $7 shippping for two and were received in less than a week. Link to Product. Performance is currently under review, data is being logged by Home Assistant and seems to be reasonable so far. rtl_433 has no trouble reading this sensor, and the update rate is pretty good (~30 seconds).

RTL-SDR Dongle (Another One)

After setting up the 433Mhz Sensor project and committing my only SDR radio to be used for data acquisition, I soon after found myself wanting to play with SDR again in the rtlamr failed project. Since I only had one RTL-SDR dongle, I had to take down the 433Mhz sensor project for a day to use the dongle. I have now ordered another RTL-SDR dongle so I can continue to play with SDR based projects without interfering with my ‘production’ hardware. I previously looked for a low cost but not poorly reviewed dongle, and selected the Nooelec model for its low entry price. I could have gotten a nicer dongle this time, but I bought the same one, so maybe in my future third entry into SDR I’ll get serious about higher performance, low noise dongles.

Raspberry Pi 3 (Another One)

With all the Pi’s I use in projects, even temporarily, I’ve found that my aging collection of Raspberry Pi’s is no longer adequate to run a lot of modern software. I’ve been working with Raspberry Pi’s for a long time, and as I re-use Pi’s from older projects they continue to age. A lot of newer software is no longer built for armv6l, or even 32-bit ARM any more. Here’s a bit of my collection:

From left to right, bottom to top:

My first ever Rasperry Pi, the original Model B with the funny mechanical layout and pre-dating the now standard 40 pin GPIO header and 4-hole rectangular mounting pattern, in an Adafruit Case. This Raspberry Pi ran Raspbmc (which later became OSMC), along with an IR receiver chip taped to the case with bacon duct tape, feeding its analog video output to my TV. Circa 2013.
My second ever Rasberry Pi, used for many years as my NAS. I had a 2TB USB HDD attached and served files for XBMC (later Kodi) on the first Raspberry Pi, but I also used this to back up files from my laptop. I eventually setup XBMC MySQL sharing, so all of the TVs in the house could share a common media database and watched information
A ‘New’ improved Model B+, which brought the new standard 40-pin header, mounting hole arrangement, and 4 USB ports, but was otherwise very similar to the OG Model B. This particular one was also used with Raspbmc, over HDMI, for the second TV in the house.
Another Model B+, my second file server for my second 2TB drive. Due to the 100Mbps Ethernet limitation of the Pi, I used two separate Pi’s to reduce bandwidth bottlenecks to the two drives. At this point, I started naming my servers after spacecraft, and named this one ‘Voyager’.
Raspberry Pi 2, which introduced an improved processor over the OG Raspberry Pi, in a 3D printed case. I bought a few of these to try a multicast multi-room audio system, but testing did not go well, so I ended up using one of this batch in a standalone audio setup with a HifiBerry Amp2 and Shairport-Sync along with some nice wired speakers on my deck, which I still use and love. This particular Raspberry Pi was later repurposed to run Octoprint, but I was unhappy with the performance of the older Pi and am rebuilding that for an upcoming project. Another Raspberry Pi 2 from this batch is currently in service as a USB print server, which has been reliable for many years.
The brand new Raspberry Pi 3. At the time of this writing (September 2021), it’s very difficult to find a Raspberry Pi 3, 3+, or lower memory (lower cost) 4 in stock, so I just have one new one, and it’s already allocated to become an Octoprint server.
This Raspberry Pi is in a very special case by Adafruit, designed to hold the OG Model B along with the OG PiTFT (back in the 26-pin days). I used this to build a user interface for a school project, loading the lite version of Raspbian (no desktop environment) and reading the touch drivers / writing the screen framebuffer raw. I published the GUI backend code on github as ‘palscreen’, but did not publish the other project files. Keeping with my space naming theme, I named this one ‘Discovery-One’, after the manned spacecraft from ‘2001: A Space Odyssey’, as the touchscreen makes this a ‘manned’ Raspberry Pi.

Raspberry Pi Follow-Up: Backing Up Zigbee2MQTT

Sat, 18 Sep 2021 19:00:00 +0000

After my scare with the Z-wave controller dying due to SD card failure (See the blog post), I decided that my Zigbee network is important enough to back up, especially because a whole lot more important data is stored on disk rather than in the dongle as with Z-wave. I’m going to follow the same path I took in the Z-wave blog, but for Zigbee2MQTT. Since it’s running ‘bare’ on a raspberry pi, I can’t just backup the whole virtual machine. Here’s the solution I came up with.

First, Install Autofs according to my other blog post. Setup a folder on your NAS to keep backups for Zigbee2MQTT, and mount that directory with autofs to /mnt/backup

Next, we need to find where zigbee2mqtt is storing its local data. In my case, I installed it with npm, and the data directory is at /opt/zigbee2mqtt/data. This contains your configuration.yaml, which is important, along with other internal files used by Zigbee2MQTT (such as database.db).

I’m going to use essentially the same backup script as before, where I take the day of the week and append it to the filename so I automatically keep the last 7 days of backups. I keep the file on the NAS in the backup directory so the backup won’t fill the SD card with backups if the NAS doesn’t mount, since it can’t access the file. I named the shell script /mnt/backup/backup.sh

#!/bin/bash
####################################
#
# Backup the Zigbee2MQTT data directory
#
####################################

# Create archive filename based on the day of the week
day=$(date +%A)

# Backup the files using tar.
tar czf /mnt/backup/zigbee_data_$day.tar.gz /opt/zigbee2mqtt/data

Next, edit the root user’s crontab so it will run all the time once per day:

sudo crontab -e

And the new contents - this will run at the 0 minute of the 0 hour (midnight), all days of all weeks of all months (so every day).

0 0 * * * bash /mnt/backup/backup.sh

The End. After a week, there should be 7x .tar.gz files in the backup directory (Which is mounted from the NAS and can be managed according to your backup and long term storage strategy from there).

Beware when extracting the archive that it restores permissions, so you need to chmod the folder once you restore it to give yourself permissions. I also realized with this that some of my Pi’s never had their timezone set, so they are still on UTC time. If that’s the case, you can sudo raspi-config to change it.

Not Every Project Works, And That's Okay

Thu, 16 Sep 2021 22:00:00 +0000

In my previous tests using Software Defined Radio (SDR), I used rtl_433 to successfully receive data from an outdoor weather station sensor. Always seeking more data, I found rtlamr - a tool which decodes smart meter data. I don’t really need to read smart meter data since the only smart meter I have is my power meter and I already have my own meter for that, but I still was excited to give it a try and see what I could find! But projects don’t always go according to plan, and that’s perfectly normal when trying new things!

After reading the documentation, I ran outside to snap a pic of the meter itself (with all the numbers clearly in view). I found out that I have an Itron AMI-4 meter. According to the RTLAMR Compatible Meters page, it should be supported. So, I figured it would be a simple, fun project to get rtlamr to read my smart meter data, and write some sort of MQTT connector to push the data to Home Assistant since it seems like the only data output path is to InfluxDB at this point (which isn’t a bad option, just not my preference).

Since I currently only own one SDR dongle (the struggle of tying up fun hardware in permanent projects!), I quickly shut down my rtl_433 server and borrowed the SDR dongle, connecting it to my Windows desktop. I played with Windows for a bit to get the software installed, as one does for primarily Linux-focused software. Once I did this, I eventually got rtlamr up and running, and left it for a half hour to try and find packets from the meter (they should come every few minutes).

Packets never came. I tried enabling ‘all’ decoders, a few different frequencies, nothing. No results. At this point, I was slightly disappointed that my easy one-day project wasn’t working out, and I started reading through the rtlamr discussions on Github. I eventually found what I was looking for - The AMI-4 can be configured to support the protocols rtlamr can decode, but just because it is capable does not mean my utility company has configured it this way. It supports 900Mhz and Zigbee, and some of the branding on the meter indicates it’s configured in Zigbee mode. So, I’m out of luck, and my day of experimentation was wasted.

But, despite not getting the project to work, I still enjoyed it, and that’s what matters. I had fun playing with SDR, experimenting with the software, and learning a bit more about how smart meters work. Maybe, in the future, I’ll have a smart meter that does use 900Mhz, but for now, I’m on to the next project.

Low-Cost 433Mhz Sensor Network with rtl_433 on Raspberry Pi

Thu, 16 Sep 2021 08:00:00 +0000

I’m fairly protocol agnostic in my home automation system, and that’s one of the benefits of building something with open source software like Home Assistant - there’s no vendor lock in and you can pretty much connect anything you can pull data from into it. While I’ve set up a Zigbee network for my blinds and ordered a ton of cheap sensors from Aliexpress to test, and set up a reliable Z-Wave network with more expensive sensors and lighting dimmers, I’m always looking to expand the wealth of data I can capture. In this project, I start up a 433Mhz receiving system to capture data from cheap ‘weather stations’ and other consumer electronics which need decently long range wireless and employ poor to no data security.

Reliably Sensing Outdoors

One of the struggles I’ve had with my Onewire network is sensing the outdoor temperature. I’ve replaced the outdoor temperature sensor once already, and both sensors have eventually died outdoors. I could spend more effort potting the sensor so it stops corroding, but I’d rather spend the few dollars to buy an off the shelf solution that’s outdoor resistant. I also really want to measure outdoor humidity, and these cheap ‘weather stations’ are a perfect hardware solution to the problem.

Under the hood, these sensors are very primitive. You mount them outdoors somewhere out of direct sunlight (in my case, under the front porch), and they transmit a signal containing the temperature and relative humidity to an indoor unit, which also measures temperature and relative humidity and displays both of them for you. The whole kit costs only $33 with both units with humidity, and you can buy a replacement outdoor unit for $13. I bought the Acurite 609TXC.

But, how do you pair your replacement outdoor unit with your indoor unit? You don’t. The outdoor unit doesn’t even have a unique ID to pair with. When you put the batteries in, it generates a random number which it calls its ID, and transmits a data packet with no encryption containing that ID, the temperature, the humidity, and a simple checksum periodically until the battery dies. Presumably the indoor unit has some logic to deal with multiple outdoor units in the same area. In my case, I don’t even have an indoor unit, just the replacement outdoor unit, and a Software Defined Radio (SDR) dongle.

When I went to install the sensor, I found an old remote unit that’s been there for at least 15 years at this point, with significant battery corrosion. Hopefully this sensor lasts at least a few years.

Sensor Installed - I'm not the first person to install a sensor here

What is Software Defined Radio (SDR)?

Wikipedia is always a good place to start. In short, SDR implements radio frequency components traditionally implemented in hardware using software combined with a basic analog frontend. In this case, the rtl_433 program uses a dongle which tunes to a specific frequency and returns analog samples, which can be demodulated and converted into a digital signal using software. The hardware does RF downconversion and analog sampling, and that’s it. The analog samples are processed by software.

The rtl_433 project implements decoders for a lot of low cost consumer electronics (including the Acurite 609TXC which I bought), but it requires a SDR backend which implements support for SDR hardware. The normal backend is RTL-SDR (hence the name rtl_433), which is part of the Osmocom (open source mobile communications) family of projects. RTL-SDR implements a SDR receiver using the Realtek RTL2832U chip, which is commonly used inside USB TV tuners. There is now a very healthy ecosystem surrounding the RTL-SDR as it’s a very low cost entry into software defined radio, although being limited to receive only operation. I purchased a Nooelec NESDR Mini 2 which is specifically sold for use with RTL-SDR, and only cost $25 as of this writing. It even includes a little remote antenna.

Installing rtl_433 on a Raspberry Pi

I’m going to install rtl_433 on a Raspberry Pi so I can locate my SDR dongle in a good location in the house, and my Home Assistant install is virtualized making hardware passthrough more work than using a separate Raspberry Pi.

In short:

#Install dependencies - rtl-sdr is currently available for Pi as of this writing
sudo apt-get update
sudo apt-get install libtool libusb-1.0.0-dev librtlsdr-dev rtl-sdr doxygen git cmake

#rtl_433 is not currently available for Pi as a package
#it made it in for Debian Bullseye, but not Buster
git clone https://github.com/merbanan/rtl_433.git

#Generate the makefile with cmake
cd rtl_433
mkdir build
cd build
cmake ../

#Actually build it (this takes awhile)
make

#Install it
sudo make install

Now to set configure it, test it, and set it up to run as a service.

Configuring rtl_433 for 433Mhz sensors and MQTT

There are a few really important config options for rtl_433 itself. The help (rtl_433 -h) is quite… helpful in setting the command line arguments.

What frequency to tune to
Which decoders to enable
How to export the data

In my case, the answer to these questions was:

433Mhz (duh), although 315 and ~900s are also somewhat popular. 433Mhz happens to be the default, so no argument required
The default decoders
MQTT, with the topic containing as much information as possible (-F mqtt), using the default MQTT options. By default, it publishes each measurement to a separate topic, instead of JSON-encoding the resulting data.

The required command line for me was thus:

rtl_433 -C si -F "mqtt://<server>:1883,user=<user>,pass=<pass>"

There are of course more options you can use if you’d like, such as a custom MQTT topic structure, setting the retain flag, and enabling/disabling different demodulators. In my case, 433Mhz is the default frequency (no argument required), SI units are preferred, and I set the MQTT output and broker parameters while using the default topic structure. The default topic structure is:

rtl_433/<hostname>/devices/<type>/<subtype>/<channel>/<id>/

Now it’s time to find out if it’s publishing, using MQTT Explorer

What's that? A tire pressure monitoring sensor? Yes.

Running rtl_433 as a service with systemd

Ideally we can setup a service so rtl_433 starts on boot, and creating a systemd service is a good way to do that.

First we need to create the service file in a location where systemd expects it, then open it for editing:

sudo touch /etc/systemd/system/rtl433.service
sudo chmod 664 etc/systemd/system/rtl433.service
sudo nano /etc/systemd/system/rtl433.service

Then we need to write our service config file for systemd (and save it)

[Unit]
Description=rtl_433 SDR Receiver Daemon
After=network-online.target

[Service]
ExecStart=/usr/local/bin/rtl_433 -C si -F "mqtt://<broker>:1883,user=<user>,pass=<pass>"
Restart=always

[Install]
WantedBy=multi-user.target

This is a pretty basic service that should do what we want. Feel free to read the systemd docs (systemd.service, systemd.unit) for more info on what can go in this file. Note that ExecStart needs to be a full path, so you might need to track down rtl_433 like this:

which rtl_433

After writing the service file, you need to tell systemd to reload it’s daemon files (since we modified them), then start the service (launch it now), then enable the service (set it to launch on boot).

sudo systemctl daemon-reload
sudo systemctl start rtl433
sudo systemctl enable rtl433

You can also view the latest bit of the log to check for any errors

sudo systemctl status rtl433

Hopefully it looks something like this:

  Loaded: loaded (/etc/systemd/system/rtl433.service; enabled; vendor preset: enabled)
  Active: active (running) since Wed 2021-09-15 11:44:23 EDT; 44s ago
 Main PID: 4005 (rtl_433)
    Tasks: 2 (limit: 2059)
 CGroup: /system.slice/rtl433.service
         └─4005 /usr/local/bin/rtl_433 -C si -F mqtt://<broker> 1883

Sep 15 11:44:24 zeus rtl_433[4005]: Found Rafael Micro R820T tuner
Sep 15 11:44:24 zeus rtl_433[4005]: Exact sample rate is: 250000.000414 Hz
Sep 15 11:44:24 zeus rtl_433[4005]: [R82XX] PLL not locked!
Sep 15 11:44:24 zeus rtl_433[4005]: Sample rate set to 250000 S/s.
Sep 15 11:44:24 zeus rtl_433[4005]: Tuner gain set to Auto.
Sep 15 11:44:24 zeus rtl_433[4005]: Tuned to 433.920MHz.
Sep 15 11:44:24 zeus rtl_433[4005]: Allocating 15 zero-copy buffers
Sep 15 11:44:25 zeus rtl_433[4005]: baseband_demod_FM: low pass filter for 250000 Hz at cutoff 25000 Hz, 40.0 us
Sep 15 11:44:26 zeus rtl_433[4005]: MQTT Connected...
Sep 15 11:44:26 zeus rtl_433[4005]: MQTT Connection established.

Adding rtl_433 sensors to Home Assistant

Home Assistant prefers MQTT Discovery, where devices publish to the homeassistant topic with information that points HA on topics of interest and how to decode them. I use this with Zigbee2MQTT, and there are some Python scripts that try to do this with rtl_433, but given the vast number of useless junk that you find on 433Mhz (such as TPMS sensors), I decided to avoid MQTT Discovery and manually enter the sensors in the yaml configuration. In this case, I found that the ID of my sensor is 252 using MQTT Explorer. The ID will change when you replace the batteries and you will have to update Home Assistant’s yaml configuration with the new ID, but it should keep the same entity as before.

The 609TXC publishes 9 variables to MQTT via rtl_433: time, ID, battery_ok (boolean), temperature (number), humidity (number), status, and mic. Of these, Time is not needed (Home Assistant will log the time the message was received), ID is part of the topic, the next three should be read by Home Assistant as sensors or binary_sensors, status always appears to be 2, and mic will always be CHECKSUM. To create sensors in Home Assistant for the three signals we would like to read, we need to edit sensor.yaml and binary_sensor.yaml. If you don’t have either of these files, you need to add to the sensor: and binary_sensor: section in configuration.yaml. Ideally, your configuration.yaml includes pointers to files for sensors and binary sensors, like this:

sensor: !include sensor.yaml
binary_sensor: !include binary_sensor.yaml

Add to binary_sensor.yaml (or your binary_sensor section):

# Acurite outdoor temp/humid sensor
- platform: mqtt
    name: "Outdoor Temperature Humidity Battery OK"
    unique_id: "outdoor_temperature_humidity_battery_ok"
    state_topic: "rtl_433/zeus/devices/Acurite-609TXC/252/battery_ok"
    payload_on: 1
    payload_off: 0

Add to sensor.yaml (or your sensor section):

# Outdoor temperature and humidity sensor (Acurite 609TXC)
- platform: mqtt
    name: "Outdoor Temperature"
    unique_id: "outdoor_temperature"
    state_topic: "rtl_433/zeus/devices/Acurite-609TXC/252/temperature_C"
    unit_of_measurement: "°C"
- platform: mqtt
    name: "Outdoor Humidity"
    unique_id: "outdoor_humidity"
    state_topic: "rtl_433/zeus/devices/Acurite-609TXC/252/humidity"
    unit_of_measurement: "%"

Of course, replace zeus with the hostname of your Pi and 252 with the ID of your sensor, or replace the path entirely if you used a different sensor than I did. You can also copy the path directly from MQTT Explorer.

Make sure to go to Configuration -> Server Control and click ‘Check Configuration’, and that you fix any errors it returns before restarting Home Assistant. It gets angry if you mess up the YAML.

The Project Files and Parts List

Here are all of the files and parts required to replicate this project.

Acurite model 609TXC Outdoor Unit - Select ‘Sensor #2’
Nooelec NESDR Mini 2+ for RTL-SDR
Raspberry Pi - Preferably a Model 2, 3, or 4, but any will work if the Pi has no other tasks
rtl_433 installed on your Pi
MQTT Explorer installed on your host

A Quick Primer on Autofs

Wed, 15 Sep 2021 11:00:00 +0000

As mentioned in my last blog post, I setup an autofs share to mount my NAS for backups. Since I’ve always used fstab in the past to mount this, and it’s quite unreliable for cifs shares, and some internet articles go into way more detail than necessary on setting up autofs, here’s a very quick overview on setting up samba / CIFS shares with autofs on Raspberry Pi OS (or any other Debian / Ubuntu based system). I tested this on Raspberry Pi OS (formerly Rasbian) Buster.

Install Autofs

First you need the package. Often a good idea to reboot after doing things with apt.

sudo apt-get install autofs
sudo shutdown -r now

Add a samba shares file to the master autofs file

Edit the autofs master file:

sudo nano /etc/auto.master

Add a line pointing to a (yet non existant) samba config file

/- /etc/auto.smb.shares --timeout 15 browse

Brekaing down this command:

/- indicates that the following file should be mounted relative to / (i.e. all paths in the file are absolute, if you put /mnt here, then everything in auto.smb.shares would be mounted below /mnt)
/etc/auto.smb.shares is the file we will create to list our SMB shares
timeout is how long to keep the shares mounted
browse seems to work well

Edit the samba shares file we pointed to earlier

sudo nano /etc/auto.smb.shares

Add your new mount point

/mnt/data -fstype=cifs,rw,username=<user>,password=<pass>,credentials=<file>,noperm,dir_mode=0777,file_mode=0777 ://server/share

Not all of these options should be used all the time.

/mnt/data - the path on the local system to mount to
-fstype=cifs - always needed for samba shares
rw - enable read/write access to the share
username=<user> - put the username here or use a credentials file
password=<password> - put the password here or use a dredentials file
credentials=<file> - Alternative to putting the uname/password here, point to a file that has the credentials
noperm - ignore permissions for local users accessing the share
dir_mode / file_mode - default UNIX permissions for directories and files on the Windows share
://server/share - you can put in an IP or hostname, and the share can be multiple folders deep

If you created a credentials file, it needs to look like this:

username=<user> 
password=<pass>

Make sure the mount point exists

sudo mkdir /mnt/data

Restart autofs so it finds the new config entries

sudo systemctl restart autofs

ls /mnt/data

That’s It! Have fun. If you’re having trouble, make sure you rebooted after installing everything, otherwise you might not have the cifs kernel module loaded.

Raspberry Pi SD Failure, and Backing Up ZWaveJS2MQTT

Tue, 14 Sep 2021 11:00:00 +0000

I got my Z-Wave Raspberry Pi setup a few weeks ago, and then spent a ton of time setting up my Bedroom Lights, Bathroom Fan+Light, and ordered even more Z-wave hardware. I also started up an RTL-433 server for a yet unfinished project, and about a week later, Home Assistant suddenly reported all of my Z-wave devices offline. Home Assistant was unable to connect to the WebSocket of ZWaveJS. The Pi was acting really weird. SSH responded but rejected my login, and when I plugged in an HDMI cable and rebooted it got a kernel panic due to mmc driver failure. The SD card had failed. It wasn’t a new SD card, so I guess I should buy a bulk pack and stop re-using them. Since I really liked having a Z-Wave network, I want to make sure that I properly setup the new raspberry pi OS install to prevent excessive logging from taking down the whole thing in the future. This also gives me a huge appreciation for how nice the backup features of virtual machines are, and how much I miss them on the Raspberry Pi. I don’t have a backup of the Z-Wave network, so that’s another feature that would be nice to consider in the second try.

I started with a brand new SD card and imaged it with Raspberry Pi OS in Etcher. I did the usual raspi-config, set the hostname, timezone, apt-get update/upgrade, the usual. Then I proceeded to install ZWaveJS2MQTT according to their instructions (using the Snap package). Thankfully, a lot of the Z-wave network parameters are actually stored in the dongle, so I didn’t have to re-associate all of the devices (whew!). But, the metadata is stored locally, and was lost. Home Assistant didn’t care about the loss of metadata, since it keeps its own metadata for each node ID, but things like node names in zwavejs2mqtt were gone. Not nearly as bad as I thought it would be.

My next plan is to start backing up the zwavejs2mqtt install’s store folder for safe keeping. With the snap package it’s stored in:

/var/snap/zwavejs2mqtt/current

So I can just make an autofs mount to the place I keep backups on my NAS, then tar that directory into the backup folder. This won’t keep revisions, but it will keep the latest copy. What I really want is to keep the last few days of backups (presumably I’d notice something within a few days?), but with as simple a script as possible. I found this script helpful in creating my own. After mounting my desired backup location using autofs to /mnt/backup, I wrote my backup script and added it to root’s crontab.

#!/bin/bash
####################################
#
# Backup the ZWaveJS2MQTT stores directory
#
####################################

# Create archive filename based on the day of the week
day=$(date +%A)

# Backup the files using tar. Need h since 'current' is a symlink from Snap
tar chzf /mnt/backup/zeus_store_$day.tar.gz /var/snap/zwavejs2mqtt/current

After stripping down the script to the bare minimum, it will generate a new tar.gz file for each day of the week, overwriting the previous. Now I need to add a cron job to run the backup script

sudo crontab -e

And the new contents - this will run at the 0 minute of the 0 hour (midnight), all days of all weeks of all months (so every day).

0 0 * * * bash /mnt/backup/backup.sh

After making sure my backups are in order, I need to make the SD card stop dying. After reading some Raspberry Pi specific help, it seems like /var/run used to be on the SD card but is now a symlink to /run, which is a tmpfs in RAM. So that’s good. /var/log is still on the SD card and I could make that a tmpfs as well, but I’m not sure if that’s actually being written to regularly. It doesn’t seem like ZWaveJS2MQTT is creating any trouble on the filesystem, so maybe it was RTL-433 that was on my last setup, and I’ll need to more closely monitor it when I get to fixing that.

At this point, ZWave is working again (very happy about that), so I need to setup RTL-433 again, and write the project page on my 433Mhz sensors. More to come!

Re-Using Old XFINITY Security Hardware in a Real Automation System

Mon, 13 Sep 2021 12:00:00 +0000

A number of years ago, my dad subscribed to Comcast / Xfinity’s security system to get a discount on internet, then un-subscribed when the promotional period ended. Their system relied on a Technicolor touchscreen which acted as a Zigbee hub, connected to a number of Zigbee door switches and a Zigbee wall moounted keypad. They wanted their touchscreen back, but didn’t care about all of the dirt cheap sensors or the keypad, so they’ve been sitting in place in the house for many years now. In this project, I revived them with new batteries and added them to my Zigbee network, trying to find a good use for them.

The Hardware

There were two pieces of hardware which I had available. The door contacts are Visonic MCT-340 SMA magnetic sensors (with temperature sensing, even), and the keypad is a Centralite 3400-D. Both of them are already supported by Zigbee2MQTT, which I use to manage my Zigbee network.

The Visonic sensors were all mounted with double sided foam tape and were kinda ugly, but they’ve been there for years so I don’t want to mess with them. One of them had been installed on top of 3 layers of foam tape to prevent the door from hitting the magnet, so I spent some time with the foam tape getting that sensor and magnet in a less ugly position without moving them too far. The sensors need CR2032 batteries, so I bought a bulk pack since it’s a fairly common size for this low cost automation stuff. In Zigbee2MQTT they report the switch status, the tamper status (if the battery cover is removed), the temperature in integer degrees celsuis, battery %, and link quality. Nothing super special here. They seem to work fine, latency isn’t bad, nothing much to write home about. They were basically free and now they are useful for only the cost of a new battery.

Visonic MCT-340 SMA Magnetic Door/Window Sensor

The Centralite keypad is .. special? It has a wall bracket which is screwed to the wall, but it readily clips off the bracket. It has a tamper switch to detect when it’s removed from the wall bracket, and this switch is also used for pairing, but it doesn’t seem to report the tamper status in Zigbee2MQTT. It does report an ‘occupancy’ status, but I haven’t found out why, it never triggers it, even when waking the keypad up by waving my hand over it. It uses two CR123A batteries, which is also fairly common with low cost automation stuff, so again I bought a bulk pack.

Centralite 3400-D Keypad

Keypad Codes

The keypad reports a topic named ‘Action’ whenever it detects a valid keycode. It does not validate the keycode itself, rather it passes the keys typed to the Zigbee controller which is supposed to process the keycode and return if the key was valid or not. It only accepts 4 digit keycodes, and transmits the entire code as soon as the 4th digit is pressed. I was hoping to be able to enter shorter key combinations to control automations, sorta like a many-button remote, but it absolutely must be 4 digits for the keypad to send it. If you only type 4 digits, it will send a ‘disarm’ command with the keycode. If you first press one of the 3 buttons on the top, it will instead send one of the arm variants (arm all, arm day, arm night). So, you get an event from the keypad with 4 numbers and one of 4 modes associated with it, and you need to respond for the keypad to beep a happy beep or not respond for the keypad to beep a sad beep. In addition, if the code is incorrect (or it doesn’t get a Zigbee response confirming it), it will wait a few seconds before letting you try again.

Because of the complex logic required to check if the keypad code is valid and carry through the transaction ID of the keypad when responding, I decided to implement the log in Node-Red, using the built-in Home Assistant add-on and integration. I wrote a Node-Red flow which demonstrates how to deal with the keypad, have a list of codes to check, and how to change or enable some of these codes from Home Assistant (using input_boolean and input_number, although you could also use a binary_sensor). This only deals with the keypad, you still need to use the associated alarm control panel logic in Home Assistant or your alarm panel integration to make a fully working alarm system. The Node-Red flow for this is available below.

Of course, there’s no reason you need to limit yourself to only treating this keypad as an alarm keypad as I have. It would certainly be possible to set each ‘disarm code’ to turn on or off a different entity, if you wish to use this as something more of a bedside automatino keypad. It’s not great for that, but with a little node-red magic it will definitely function. A Node-Red flow for this is also available.

The Project Files and Parts List

Here are all of the files and parts required to replicate this project.

Visonic MCT-340 SMA as provided by XFINITY - You can do better with another Zigbee vendor, but they were free to me.
Centralite 3400-D keypad as provided by XFINITY - You can probably do better here, although I don’t have a product suggestion to give
Working Zigbee2MQTT install
Node-Red Flow to use Keypad as Alarm Keypad
Node-Red Flow to use Keypad as Automation Keypad

A Quick Infill Project

Sun, 12 Sep 2021 14:17:00 -0400

Everyone who has 3D printed something is at least vaguely aware of the concept of infill. Unless you want your 3D prints to be totally solid, you usually reduce the density of the center of the part, creating a number of solid layers around a central void. The percentage of this central void that is filled with material is known as the infill ratio. A lot of slicers will automatically pick some level of infill and some pattern to distribute the material in the most structurally sound way, but that doesn’t mean you have to stick to these settings. With some clever slicer settings, you can make the infill visible from the outside of the part, giving a unique artistic view into the inards of 3D printing.

Recently, I was asked to help make ‘MAKERSPACE’ letters for a sign for a makerspace classroom. Emphaszing the process of 3D printing, I chose to expose the infill of each letter. I made a CAD model of each letter in FreeCAD, using the clearest font known to man, Helvetica. Being a Mac user, I had access to the TrueType for the true font, not the Arial knockoff. As I wanted the infill to be clearly visible in the final prints, I chose to use the Bold version for the wider stroke width.

Letter A, in PrusaSlicer, using gyroid infill pattern

Letters A as printed using gyroid infill pattern, next to letter M printed with line pattern

How can you tell the real Helvetica? By the capital letter R. The tail should always be vertical, a diagonal tail indicates an inferor clone

Completed sign, using different infill patterns for each letter

Starting my Personal Website

Sat, 11 Sep 2021 12:00:00 +0000

I’ve wanted to document my projects for a few years, after following plenty of other creators with excellent project documentation, but I’ve never found the time to start. This is my start. I’ve built my personal website, created content for back projects worth sharing with the world, and I’m committed to continuing to document and share the things that I make.

All In The Name

I started off with choosing a name. I’ve used the name ‘apalrd’ for many years as a personal account, and I decided to make that the domain name. But, I still had to choose a top-level domain. Do I want the trendy .tech? Personal .me? Slightly less expensive but also trendy .io? I settled for .net since it’s less expensive. It’s also a classic TLD, and fits well with the technology and homelab vibe.

The Wild West of Internet Security

I’ve experimented before with running self-hosted websites, and for awhile in the past I had a personal forum hosted at my house, accessible only to those friends who knew my free dynamic DNS name. I played with administering Linux servers starting all the way back with Ubuntu 6.06 Dapper Drake, on my parents used desktop. I was able to get a basic enough Apache and MySQL setup in the most insecure way, load phpBB and Drupal, and go to town. Back then, AIM was still the messenger of choice for a lot of people, Facebook had just launched to the public a few months prior, and this private forum is how my friends and I communicated with each other from our home computers.

In the years since that forum, I’ve learned a lot about security. In fact, I’ve learned that security is something I really want to leave to the experts as much as possible. I’m also much more conscious of the proliferation of user tracking, and want to minimize how much my site feeds Big Data information on my viewers. To me, the best way to achieve all of this is to make the site completely static, not include any social media ‘share this’ buttons, and not include advertising. Without any server-side code execution, the exploitable surface area is minimal, and with as little client-side code as possible, the site should load quickly on all devices, and should even look reasonably close to correct with Javascript disabled completely. In fact, I audit the view of the site using NoScript to make sure the site looks as I intend for those viewers who have chosen to block Javascript by default (and this is something I’d encourage everyone to use where possible).

A New Hope

I use a tool called Hugo to generate this site, and store the contents of the site in a Git repository, entirely separate from the public server. Pages are written in Markdown, a simple textual language that can be rendered directly to HTML by Hugo, and the site’s theme can be added to all pages when generated. After page generation, the site’s HTML files on my local computer are synchronized to the hosting provider, which is operating as a static content delivery network. This also means I am not responsible for administering a virtual private server, and can let the experts handle hosting security without giving up full control of the exact HTML and Javascript that my viewers get.

I hope you enjoy the collection of projects I work on, and I hope I can continue creating content that’s enjoyable to my viewers. Feel free to contact me by clicking on my email address at the bottom of any page (Javascript must be enabled for it to show up).

Signed, Andrew

3D Printer Mood

Wed, 08 Sep 2021 22:06:32 -0400

As all of you can appreciate, my interest in projects varies with time. At any given time, I have plenty of projects in process, and my moods change which projects I work on at any given time.

Given that information, here’s an update on what I’m working on currently:

Cheap PoE Fisheye Camera for 3D Printing - This project finished months ago, but I wrote a detailed review of the camera and published it.
Simple Polycarbonate Enclosure for Prusa Printers - I just finished the first revision of this, and am starting trials using it.

Since I’m on a 3D printing kick, expect at leat one of these projects to appear in the future, depending on how long I can keep this motivation up:

The Ultimate OctoPi Setup - in which I upgrade my OctoPi setup to a Raspberry Pi 3, print an enclosure to bolt to my polycarbonate box, and setup autofs and all of that good stuff to integrate with my NAS.
Pimping my 3D Printer Lighting - in which I add LED lights to my 3D printer so I can stop using the work light, which is only needed so the camera has a view of the print overnight
Setting up The Spaghetti Detective cloud-free - in which I venture into The Spaghetti Detective to reduce my unattended 3d print messes
How Much Power does my 3D Printer Use - in which I flash a Sonoff S31 power meter smart plug with Tasmota, and then attempt to measure how much power an average 3D print uses.

Stick around and find out what I end up making.

My First Post

Wed, 08 Sep 2021 13:44:05 -0400

This is my first post on my new blog. I’m starting this new endeavour in my life, the creation of my personal website. I hope you enjoy this content.

Simple Polycarbonate Enclosure for Prusa Printers

Wed, 08 Sep 2021 12:00:00 +0000

In 2018 I bought a Prusa i3 MK3s kit, and I’ve been very happy with it. However, I’ve found that it really struggles with early layer curling, even with PLA, in my cold basement. The solution to this problem is to build a box around the printer. The box will retain more of the heat generated by the printer, hopefully resulting in less warping due to cooling of the printed parts. This box should work with any other printer too, as long as you get a sheet of polycarbonate large enough for your printer.

The First Box

To test if the temperature / airflow was causing my print issues, I built a box out of foam floor tiles. The box worked remarkably well, instantly reducing curling of print edges and prints coming off the bed. It wasn’t exactly pretty though.

The Old Box, Closed for printing

The Old Box, Open for viewing

Obviously, this wasn’t a great long term solution. It does have a decent mount for the 3D Printer Camera though.

The Second Box

The second box is designed to use 2’x2’ sheets of polycarbonate. I already had a box of 2x2 sheets of 0.060" polycarbonate leftover from other projects, so that’s the thickness I used. Had I purchased material for this project, I would have gone with a thicker sheet for increased stiffness.

I decided to design and print corner brackets first. These are triangular with 3 sides, each with space for a #8-32 UNF fastener, and a nut pocket that’s a little oversized to avoid the need for a wrench. Since a cube has 8 corners, I need 8 brackets, and made them all with the same design instead of modeling specific variants for bottom or front corners.

After the corner brackets, I designed a similar shaped bracket to hold the camera. I used a #8-32 UNF fastener to mount the camera.

The final model for the initial design is the hinge stiffener bracket. It has 3 holes to mount the hinge. I used VEX Robotics Hinges since I had easy access to them, and they use a 1/2" hole spacing, so the model picks up 3 of the 5 holes, plus spreads the load across a wider area to reduce local bending of the polycarbonate. The width was chosen based on the print bed limits of my Prusa i3 MK3s printer.

While the printer started working, I ordered my fasteners from McMaster-Carr and waited impatiently for them to arrive.

All of the 3D printed parts

Assembling The Box

Newly printed parts and fasteners in hand, I started to assemble the box. I quickly found that there were a few issues with the design that I would need to address:

The nut pockets in the parts were all designed 1mm oversized (flat to flat), which was a bit on the large side, but when I tested the first printed part, I found that certain edges of the hexagon would sag as they were printed unsupported overhangs, and the nuts I had on hand had no problem tightening with that large of an oversize. However, the nuts I had on hand were not nylon locknuts, and didn’t require as much torque to tighten. With the nylon locknuts, one of the three hexagons (the one printed flat without any overhang sag) would spin in the pocket when tightened. The other two were usually fine. I chose to jam a screwdriver into the nut pocket to hold the nut instead of re-printing the parts, but I did go back and fix the original models and export new AMF files for you guys, so the downloads are all corrected with an 0.5" oversize, which I used for the later stiffener parts.
The plastic is way too thin to support the box with only corner brackets. I designed and printed a new side stiffener bracket to mount in the center of each edge, where two side panels meet, to hopefully improve the rigidity of the cube.
The front corners of the box spread easily. This may or may not be a problem, I’m going to see what happens once the stiffeners are installed. I can’t print a single piece to span the gap, but I should be able to fit 3x pieces in the printer and assemble them. Alternatively, I could use rubber feet to try and keep the box in place.
The center sags between the two hinges, and the left and right panels also sag in the front. I need to find a way to stiffen the center aruound the hinges at the top of the box, and interlock the door better to keep the whole box solid. Maybe magnets are the solution here. They are handy sometimes.

Completed Box

After printing more edge stiffeners and linear stiffeners, I assembled the box for the last time, and added the door. I also made 10mm feet for each corner, to lift the box up a little bit to leave room for cables. Due to the hinge design, the door ends about 10-20mm below the bottom of the box, but I didn’t trim it, intead I use this to ‘prop up’ the box since the polycarbonate is so thin. If you copy this, definitely use a thicker plastic, I just used what I had.

The Project Files and Parts List

Here are all of the files and parts required to replicate this project. All design files are licensed Creative Commons CC-BY-SA.

VEX Robotics Hinges
1/16" thick 2’x2’ Polycarbonate Shets - You should buy 1/8" or thicker
#8-32 UNF x 1/2" stainless steel bolt - Increase the length of the bolt if you go above 1/8" thickness
#8-32 UNF stainless steel nylon lock nut
#8 stainless steel washer
3D Models (AMF format)
3D Model originals (FreeCAD format)
Thingiverse Page
PrusaPrinters Page

First (mis)Steps in Bathroom Automation

Mon, 06 Sep 2021 12:00:00 +0000

I’d already had a start in Home Automation with my IKEA Blinds Project and Z-Wave with my Bedroom Lighting, so I was ready for something more advanced. Little did I know that automating a bathroom light and fan switch would require so much logic to avoid being stuck in the dark in the shower at night, and all the other corner cases that come up when you try to implement real automation logic instead of ‘smart home’ party tricks. I also had to deal with an existing bathroom exhaust fan + light that didn’t have separate wiring to control the fan and light, requiring some special hardware.

Existing Fan + Light in future Smart Bathroom

The Magic Fan + Light Dimmer

The existing fan + light had a single power wire (line + neutral) to a single wall switch, which controlled both the fan and the light. This was not ideal, since I’d often leave the light off at night since the fan was so loud, and I can’t use a light dimmer because it would also attempt to dim the fan. I really wanted the light to dim automatically at light, and to turn the fan on and off automatically with humidity due to the shower, so I needed separate control of the fan and light, but I didn’t have the wiring in the wall to accomodate a wall switch and wall dimmer. I was luckliy able to snag a few of the Inovelli Red Series Fan + Light Switch (LZW36), which uses a single-gang space for the wall switch with both a fan and light dimmer, and a RF remote ‘canopy module’ designed to mount on top of a ceiling fan, but controlling the power up at the fan instead of in the switch itself. I took the motor section out of the fan+light, revealing the wiring area and fan duct, wired the canopy module into the wiring area, and tucked it in next to the fan duct. It was an extremely tight fit, and there was basically no wire length for access, so I was very thankful to have Wago push connectors instead of wire nuts. I managed to fit the canopy module in there, and wrapped the black antenna wire in red electrical tape to prevent it from getting damaged when I re-installed the metal fan.

Inovelli Red Fan + Light Switch mounted in the bathroom

Canopy Module tucked very tightly inside fan housing

Sensor Module

I could have looked for separate temperature+humidity and motion sensor modules, but I wanted to try out the Inovelli 4-in-1 Motion Sensor (LZW60), which includes motion, light, temperature, and humidity. I think I could do better on cost in the future by going with Zigbee sensors, but the Inovelli sensor has an absolute ton of configuration ability (parameters like motion reset time, sensor reporting interval, etc.) whereas none of the Zigbee sensors I have can be configured at all.

4-in-1 Motion Sensor installed in the Smart Bathroom

First Pass at Logic

First, I setup an Automation in Home Assistant using the Motion Controlled Lights blueprint that’s built in. I matched up the motion sensor entity with the light entity, and set the time I wanted (4 minutes, minus the 30 seconds built in to the sensor). This functioned, and there was much rejoicing!

Second, I setup an Automation in Home Assistant for the fan humidity control. I used Numeric State trigger, so when the humidity goes above 69%, it will trigger. It turns on the fan, then waits for another Numeric State trigger, for the humidity to go below 66%. I turned the shower on, waited for the fan to come on, turned the shower off, and waited for the fan to turn off automatically, and there was much rejoicing!

After a day of playing with this, I quickly found a bunch of issues with this very basic automation:

The motion sensing light turns off when you’re in the shower, and you’re left in the dark. What a bummer.
It takes a really, really long time for the humidity to come back down after a long shower. Ideally we could keep the fan on until it does go back down, but there’s just too much moisture in the air and the fan is too in effective at venting. I might have to tolerate a higher turn-off humidity, turn the fan on sooner, or something like that, but I need to do something so the fan doesn’t run for hours.
I’d like the light to be dimmer at night, and not so darn bright when I’m barely awake trying to use the bathroom.
Sometimes, people push the up/down buttons on the face of the dimmer and that messes up the level the dimmer returns to when it turns on/off.

Second Pass at Logic

After learning from the basic fan and light automations over a few days, I realized that they still weren’t that great and I could improve.

First, I quicly setup the dimmer to always return to a set level when turned on, instead of the last value. I also added this default brightness parameter to my sunrise/sundown automations (parameter 12 for local control and 13 for Z-wave control), seting the night value to 11% and the day value to 100%, similar to my bedroom lighting. I also set the fan to not be dimmable at all (minimum dim level 3), so it will either be on or off. Being an exhaust fan, it already doesn’t move a ton of air, and going down to lower dim levels makes it basically useless.

Second, I set a timeout of 35 minutes on the fan automation, so it will turn the fan off no more than 35 minutes after it goes on. For a quick shower it should turn off in about 20 minutes, but some people take excessively long showers, and the fan just cna’t remove the moisture in any amount of time without opening the doors and ventilating the room.

Third, I added logic to the fan automation to override the light. It now follows this sequence of events

Trigger when humidity rises above 69%
Turn the fan on
Turn off the bathroom light motion automation, without letting it continue running any subsequent events
Turn on the bathroom light and set the brightness to 100% (even at night, you probably want light in the shower)
Wait for the humidity to fall below 66%, or 35 minutes
Turn the fan off
Turn the bathroom light motion automation back on
Turn the light back off

I then lived with this automation for a few more days, and there was again much rejoicing. However, I did find one corner case where I restarted Home Assistant while the automation was in its 35 minute fan cycle, and the automation ended without finishing, leaving the motion sensing lights disabled and the fan running.

As of the writing of this, I still have a few lingering issues that I might try to resolve with a more complex automation, or possibly by implementing it differently:

Even if the fan and light are turned off manually, the motion sensing lights will remain disabled until the fan would have automatically turned off. They should go back to automatic if the lights are manually turned off, but should not turn on until motion is re-detected (i.e. should not turn on because someone was in the room in the last 30s, as they obviously turned the lights off for a reason)
If the fan is turned off manually, but not the lights, the lights should stay on but revert to motion control.

Manual Backup

After all the work on automation, I still struggled with two unrelated issues:

If you sit on the toilet too long, sometimes, the light turns back off. I played with the motion sensitivity and timeout to reduce the chance of this, but it’s still possible. Waving your hands can get the lights back on, but the position of the motion sensor isn’t perfect.
Since the fan no longer comes on automatically with the light (they used to be physically tied together), and sometimes you wished you turned it on when you’re already on the toilet.

I could have done more work on the automation to fix these issues, but for issue #2 specifically, I just didn’t have a good way to automate the fan. So, I went with the easy way out - I added a pushbutton next to the toilet so you can toggle the fan and light from the toilet. I used one of the two double pushbuttons that came with my IKEA Blinds, and setup an automation to toggle the light in response to the top button and toggle the fan in response to the bottom button.

Sometimes easier user control is better than more automation

The Project Files and Parts List

Here are all of the files and parts required to replicate this project.

Tracking my Cat with Frigate NVR

Sun, 05 Sep 2021 12:00:00 +0000

Previosly, I built a home security camera system using ZoneMinder. Somewhat dissatisfied with the status quo of ZoneMinder, I set out to try a brand new security NVR - Frigate - and see if an NVR written specifically to integrate into Home Assistant could be used for more than just recording and viewing camera footage.

The Old System

I am using the same Dahua cameras I installed in the ZoneMinder System. In fact, I ran both Zoneminder and Frigate for about a month while I validated functionality of the new system, using the same ZFS pool! Read that project page for information on the cameras I chose and why. There may be newer cameras available now, but I’m still satisfied with the image quality from what I already have.

Since writing the article on the old system, I added a few cameras in ZoneMinder just for viewing in the app - the USB security camera connected to my 3D printer’s OctoPrint setup, and the PoE camera I got to monitor my 3D printer (Project Here).

Setting Up the VM

I setup a new Ubuntu VM on my Minilab to run Frigate. Without an AI accelerator, and only giving it 4 vcpu cores so it doesn’t starve Home Assistant, I was able to comfortably run ai inferencing on two cameras and not really on 3. My goal was to run ai on the close-up door cameras at a high enough resolution to detect the cat, which means I can’t run at the low-res D1 (704x480) sub stream on my 1440p cameras and either have to drop the 1440p main stream down and reduce the main stream resolution to the resolution I can do AI at, or do AI at full 1440p main stream. Hence the massive CPU penalty. I was okay running the D1 stream for the far shot cameras where I was more focused on car and person detect, and both cars and people are significantly larger than cats. With this in mind, I ordered two Coral AI accelerators to fit in the two M.2 slots of Minilab.

As you can read in the Minilab project, I really struggled with PCIe passthrough on Xen, and even a bit once I moved to Proxmox, but it ended up working out, and I found myself with a brand new Ubuntu VM with two working Coral AI accelerators for use. I still gave it 4 vcpu cores and 4G of RAM, which is hopefully enough for the full camera system. I setup Docker according to the Docker install guide for Ubuntu, and was off to the races.

Setting up Frigate

At this point, I have the 5 Dahua cameras I setup for the ZoneMinder System, the OctoPrint raspberry pi and it’s USB webcam, and the wide angle PoE camera watching the 3D printer. I setup my Ubuntu VM to load the Frigate directory off a dedicated zfs dataset from the file server, and that includes storage of recordings, clips, and the config.yml file. I relocated the database file off the remote zfs dataset onto the VM’s 30GB OS drive due to file locking issues with Samba shares. I wrote up a config file I was happy with, assuming the Coral AI was all powerful (and I had two of them), using the 5 Dahua cameras and the wide angle PoE camera. I quickly learned a lot:

Frigate does not let you create a camera without a detect behavior, at all. It really wants to try object detection, and I don’t want to do that for my 3D printer cameras.
I expected a bit too much from the little Coral AI chip, even with two of them. Trying to run 4 cameras at 1440p, 15fps analysis and one at 1080p (the 4K camera which supports higher resolution sub-streams), it was not happy.
ffmpeg decoding of so many high resolution streams at high framerate is quite CPU intensive without the ability to offload it to a GPU, and I don’t feel comfortable giving up my only way to access the Proxmox console physically to pass-through the iGPU.

After struggling for a bit, having learned a lot, I decided on a much more reasonable configuration:

The two door cameras will run analysis at 5fps on the 1440p main stream. Only the main stream is used for these cameras. They have person and cat detection enabled.
The two wide field 1440p cameras will run analysis at 5fps on the D1 sub stream, using the D1 stream only for detection and the main stream for everything else. These cameras have car and person detection enabled
The 4K varifocal camera will run analysis at 5fps on a 720p sub stream, using the 4K stream for recording and clips and the 720p stream for real-time viewing and detection. This camera has car, person, and cat detection enabled, although I don’t think the resolution is high enough to pick out cats, I want to give it a try.
The 3D printer cameras were added to Home Assistant directly, and skip Frigate entirely. HA manages the (re)streaming of these cameras to the HA app and web UI.

Cat Detection

As promised, I really wanted to track my cat. Frigate delivered. Here are some gems for you guys to enjoy:

Front Door	Back Door

Staring into your soul	Off on an adventure

A skilled jumper	Supervising the humans

Leaving for fun	Relaxing in the shade

Lovelace View

I added a tab to my HA Lovelace security page specifically for watching the cat

Cat Cam tab

Final Thoughts

Frigate is an excellent NVR and has earned its place in my network. Once I am satisfied with the recording and clip searching, I will deprecate ZoneMinder and switch to using Frigate exclusively for security cameras. I’m already impressed with how reliably Frigate rejects things like bugs, spiders, and swaying tree shadows, and feel like I can trust the filtered events more than ZoneMinder. I already have plans to automate exterior lighting based on person detection, so we don’t have to ’leave the light on’ for people coming home late, and Frigate should reduce false positives in this regard. More to come on this subject for sure.

Getting Started with Automated Lighting with Z-Wave

Thu, 02 Sep 2021 12:00:00 +0000

After playing with my Zigbee-controlled IKEA FYRTUR Blinds, I wanted to experiemnt with automated lighting. Despite already having a functional Zigbee network, I wanted to choose high quality, reliable lighting components. After spending a ton of time researching on the internet, I decided to start a new Z-Wave network, using Inovelli dimmer switches. This project is my first attempt to get the network functioning.

The Choice of a Network

After my (somewhat poor) experience with Zigbee, I wasn’t eager to use Zigbee hardware again for something critical like lighting. I know hardwired devices behave better overall than battery powered ones as they don’t have to be woken up to change settings, but I’d been spending some research time comparing Zigbee with Z-Wave and Lutron Caseta, and I decided to try setting up a Z-Wave network for this project. Lutron Caseta has a fantastic reputation, but it’s pretty expensive for what you get compared to the newest generation of Z-Wave products, and has a 75 device limit. I don’t need 75 devices currently, but at this point in my life I’m already planning on my first new house build and I could easily run up on that 75 device limit in that application, so I’d like to experiment with technologies I’d actually put in my house build.

I started with my bedroom combined fan and light. I am planning on adding additional lighting in the future, but that is another project. For now, I selected the Inovelli Red Series Fan + Light Switch (LZW36). As of this writing it’s out of stock, but I was able to snag two by closely following the forums. It uses Z-Wave Plus, sports a double button plus double up/down toggle control, and two of the RGB indicator bargraphs that Inovelli is famous for. The Fan + Light switch has a remote module which actually controls the fan and light, and mounts like an RF fan remote module would.

Setting Up the Network

For the Z-Wave controller, I got an Aeotec Z-Stick Gen5+ because the Gen7 was out of stock. My existing raspberry pi in the laundry room (located in the center of all of the bedrooms) which is running Zigbee2MQTT is an old Raspberry Pi 2, which is too old to run the latest versions of Node.JS required for ZWaveJS. So, time to track down a newer Raspberry Pi. I’ve been involved with Raspberry Pi’s for a long time and I’ve gotten most of my older ones embedded into projects by this point, but I bought 10+ of them in the ARMv6L days, so most of them are still older. Given the current Raspberry Pi shortages and the lack of software to run on ARMv6L, I’ll probably have to start slowly buying Model 3 and 4’s as I can. I was able to find a Raspberry Pi 3 not involved in a project, and set it up as my Z-Wave master using ZWaveJS2MQTT. I chose to use ZWaveJS2MQTT to use it’s good web UI, and ran it remotely from Home Assistant to keep it running when I restart the HA server and locate it in a better place for coverage than the depths of the basement lab.

Automating my Lights

After setting up the Z-Wave controller, adding the ZWaveJS integration to Home Assistant and pointing it to my ZWaveJS2MQTT installation on the Raspberry Pi, I paired the LZW36 in my bedroom to the network. I was instantly impressed by the breath of configurationability of the LZW36, compared to my Zigbee network where all of my devices have essentially no configuration ability. I played with the dimming ramp rates, default levels, and even set the LEDs to different colors for fun. Super easy with ZWaveJS2MQTT.

I already bought an IKEA TRÅDFRI Motion Sensor when I bought my blinds, since it was really cheap and didn’t cost extra to ship, and I decided to try that in my bedroom instead of buying a new Z-Wave motion sensor. This means I won’t get all of the features of something like the Inovelli LZW60, but it’s functional and 1/3 the price. There are no settings to adjust on the motion sensor, it returns a boolean sensor in Home Assistant with a 90 second turn off delay. I configured the lights with the Home Assistant built-in ‘motion controlled lighting’ blueprint, matching the motion sensor with the light device. It works pretty well, and I’m generally happy with it.

After living with it for a few days, I found that the automation is perfect during the day, but can be irritating at night. The motion sensor also doesn’t have a great view of my bed, so sometimes it turns itself off if I’m using my laptop in bed. My solution to this is to expand the sunset automation I started in my Smart Blinds Project, making a pair of automations for sunrise and sunset tasks. I change the default brightness of the switch (parameter 12 for local and 13 for Z-wave ‘on’ command) to 66% after sundown and back to 100% after sunrise, and disable the motion sensing automation between sunset and sunrise so the lights don’t accidentaly turn on at night. I’ll have to suffer with only manual or app control in those situations, which is still better than what I had before this, so I’m pretty happy overall.

I haven’t experimented with Inovelli’s RGB indicator for notifications yet other than manually changing the color for fun, but that’s certainly a project for the future. I already love the look and function of my automated light switch, and I’ve ordered a 10-pack of switches and a few dimmers to automate a whole bunch of other stuff in my house.

The Project Files and Parts List

Here are all of the files and parts required to replicate this project.

Aeotec Z-Stick Gen5+ - NOTE - They have a Gen7 but it was out of stock so I got the Gen5+
ZWaveJS2MQTT
Inovelli Red Series Fan + Light Switch (LZW36)
IKEA TRÅDFRI Motion Sensor

How I Solved My Afternoon Sun Glare with IKEA and Zigbee

Sat, 24 Jul 2021 12:00:00 +0000

My bedroom faces to the West. As with most McMansions in the United States, the architect had absolutely no consideration for the angles of the sun in each room throughout the day. In fact, the architect wasn’t even involved in building this house, the plans were purchased as a set. As a result, I get blasted in the mid afternoon summer sun, greatly raising the temperature in my room and causing far too much screen glare for my taste. My solution to this is to cover my windows with motorized roller blinds.

IKEA To The Rescue

As with all things in interior decorating and the finest Swedish meatballs, my very first place to look is IKEA. IKEA has actually come out with some low-cost home automation products based on Zigbee, their TRÅDFRI line, which indludes a Zigbee gateway, app, low cost Zigbee smart buttons, motion sensors, light bulbs, etc… and of course their FYRTUR motorized roller blinds!

Blinds as they move

Without much information at all on these, but knowing they came in a complete kit (including a repeater/controller and pre-paired button), and they look pretty decent as far as blinds go, I bought a pair of them for my two windows. They only came in 2" width increments, and all come with a ton of height (which you can adjust to the exact window height during setup). After mounting it in my windows, they come pretty close to completely covering the glass, but it’s good enough.

Clips that attach the blinds to the window frame

The FYRTUR motorized roller blinds come with a right angle USB power supply, a Zigbee controller/repeater with a USB power passthrough, a battery pack with micro-USB input, a micro USB cord to connect to the passthrough on the repeater and charge the battery pack, and a two button battery powered Zigbee remote.

Included Zigbee controller/repeater with USB passthrough and power supply

Living with Manual Control

Since the blinds come pre-paired, I just needed to power the included Zigbee controller and the included button would work with the blinds. Since the controller has a USB in and out, I could stack them all up on the same USB power supply. It’s possible to pair multiple blinds with a single controller and button without their hub, but I left them as-is and had a separate button for my left and right blind. This is honestly a pretty adequate way to use these roller blinds, with the buttons on my nightstand next to my bed.

Included button (integrated magnet), plus included metal button base

How Long does the Battery Last?

The battery is an 18 Ah Li-Ion pack with integrated charger with a micro-USB port for charging. The battery is charged by removing it from the blinds and plugging it in to any micro USB charger. I’ve been using these for about 6 months and the batteries haven’t died yet. I will update this page when I get better battery voltage tracking.

Battery Door with battery installed

Battery

Starting my first Zigbee network

To better automate these blinds based on weather and time of day, I sought to integrate them with the Home Assistant system I’ve been playing with for a bit. I decided to run a Zigbee2MQTT system, and bought a Texas Instruments CC2531 based Zigbee controller pre-flashed to work with Zigbee2MQTT. Read more on the Z2M page about supported adapters. When I selected the CC2531, it was noted as having a poor antenna and not recommended for large networks, but there weren’t many options listed on the Z2M site. Over the past year, Z2M have added a lot more options for Zigbee USB sticks, and I’d recommend you read their updated page before selecting one. I’ve certainly noticed that the CC2531 has terrible link quality.

Despite the poor link quality, I paired both of the included IKEA controllers as repeaters to my Z2M network, placing one on the first floor and one in the laundry room (the same room where the Z2M raspberry pi is located, but on the other side of the room). By pairing as many devices as possible through these two repeaters, I’ve gotten much better signal strength than pairing directly to the controller.

I paired all of the IKEA hardware (both buttons, both blinds, both repeaters) individually, and removed all associations in the Zigbee network. I also connected Z2M to my MQTT broker. This paved the way for Home Assistant to manage the show.

I setup Home Assistant on my Minilab, connecting it to my MQTT broker on Telstar. Enabling MQTT Discovery in Z2M was very easy, and the devices showed up quickly in Home Assistant. I renamed them all to make sense, and set the location so I can use Area based automations.

My first two automations run when the up and down button are pressed on one of the Zigbee remotes. This directly commands the blinds to open and close. This restores the functionality that I had previously, but using only one remote for both blinds.

My first real automation opens the blinds automatically at sunset. This ensures that I can wake up naturally with the sun, as previously I would accidentally leave the blinds closed at night and not wake up with the sun as I prefer. After that, I started experimenting with automation rules that close the blinds in the afternoon depending on the weather. As I improve the sensor suite the automation system has access to, I hope to base the blind automation based on UV Index instead of just ‘sunny’ or ‘partly cloudy’.

I always hate how most people consider ‘smart’ devices as having app control, but no real automation ability. I didn’t even consider the Home Assistant app until many weeks of tweaking the automation rules so the system works for me. I already had a wireless remote I could have used, so why should I use an app instead? My goal is to open the app as infrequently as possible, and take the time upfront to integrate everything properly so it just works as I expect. This project was pretty simple in that regard, but I’m sure I’ll expand my usage of Home Assistant in the future with much more complex automations.

The Project Files and Parts List

Here are all of the files and parts required to replicate this project.

IKEA FYRTUR Zigbee Roller Blinds
Zigbee2MQTT Supported Adapter
Raspberry Pi (I used a Raspberrry Pi 2 Model B since it was already in the laundry room, but you really should use an ARMv8 64-bit version 3/3+/4).

Cheap PoE Fisheye Camera for 3D Printing

Mon, 10 May 2021 12:00:00 +0000

The quest for the best camera/angle for my 3D Printer

My original plan was to use Octoprint with a USB camera (since that’s the cool thing to do, right?). I got a Logitech C270 USB webcam and was very underwhelmed by the image quality. I found that the field of view was just too narrow to get a good shot of the entire 3d print, especially if I was printing something big. I ended up mounting it to the back of a chair and moving it around.

I connected it to my OctoPi (OctoPrint on a Raspberry Pi), and streamed it, and it functioned. I could watch it remotely. At this point, I was not using OctoPrint for the printer, since I was having printing troubles and was trying to minimize the number of variables. I would later find that the 3D Printer Box solved the printing issues, and am planning on building a new, evolved OctoPi setup in the future.

The New Camera

I set the following goals for my next camera:

Camera must have 120° or wider field of view for close mounting
Camera must be as small as posible to fit in the 3d printer enclosure
Camera must be 1080p (or higher)
Camera must be Ethernet based and support h.264 natively - I don’t want to rely on my Raspberry Pi for streaming, and Octoprint plugins always take a stream or image URL instead of a local camera anyway

From this set of requirements, since I prefer not to purchase from Amazon, I went to Aliexpress to search for a low cost camera. I ended up buying this (affiliate link):

JIENUO 5MP Mini Panoramic Camera POE

I obviously chose the PoE version, to reduce cable clutter around the printer, since I already use PoE extensively for IoT devices and cameras.

The New Camera

Another View

PoE power electronics in-line in the cable

Testing the new camera

I mounted the new camera on the foam tile printer box (see the 3D Printer Box backstory). I connected it to the network, configured it with the web UI, but couldn’t figure out what the stream and image URLs were from the minimal documentation. Thankfully, it is truly ONVIF compliant, so I used ONVIF Device Manager to figure out the stream URL.

tl;dr, here are the stream URLs you need:

For the main stream:

  rtsp://user:password@<ip_address>:554/11

For the sub stream:

  rtsp://user:password@<ip_address>:554/12

For the still image:
```
  http://<ip_address>/tmpfs/auto.jpg
```

The Web UI uses HTML frames, but this is what one of the configuration frames looks like. Very basic HTML, but functional.

A view of the web interface

The main stream supports up to 2560x1920 and the sub stream supports up to 800x600. Both support up to 30fps, and either h.264 or h.265 (although both streams must use the same codec!). I chose h.264 over h.265 since most web browsers (except Safari) do not currently support h.264 natively, and compatibility without transcoding was important to me.

A snapshot from the camera (click for an unmodified image)

Home Assistant Config

If you want to use this camera with Home Assistant, here’s a sample YAML section (goes under ‘camera’ section, or camera: !include camera.yaml)

# Fisheye 3D Printer Camera
- platform: generic
  name: "Printer Camera"
  still_image_url: 'http://10.0.0.1/tmpfs/auto.jpg?'
  stream_source: 'rtsp://admin:admin@10.0.0.1:554/11'
  #/12 is the low res stream
  username: 'admin'
  password: 'admin'

Accurate Water Measurement with ESP32

Tue, 04 May 2021 12:00:00 +0000

I am currently living in my parents house, which is fed water by a private well, and we have no way of knowing exactly how much water we use. Since I’m planning on building my own house, I wanted to know how much water I (and everyone else in the house) realistically used. In addition, I wanted to know how much hot water we use and how frequently we use hot water, so I could model hot water heating systems.

The Existing Plumbing Setup

As I mentioned, we have a private well. The well has a submersible pump near the bottom of the well, with a wire and pipe which runs into the house, where the pressure switch and fusible disconnect are located. After the pressure switch and pressure tank T off, there is a PVC manifold which directs the water to 3 places - the automated irrigation system, the hose bibs around the house, and the water softener. For those of you with city water, a water softener is a type of ion exchange filter which uses salt (sodium chloride) to replace iron and other metals in the water with sodium. Iron is not hazardous, but it does stain everything, so reducing it is desirable, but the water still isn’t perfect. Anyway, going out of the water softener the manifold splits again into two 3/4" PEX lines, one of which goes directly to the cold water inlet of the water heater and the other of which feeds all of the cold water outlets within the house.

tl;dr I have four separate pipes coming out of the well manifold and by measuring two of them I can get hot and cold water usage directly

The Design

Given my desire to measure the potable water usage in the house and also to measure specifically hot water flow, I decided to buy two NSF certified, revenue grade 3/4" cold water meters and install one on each of the cold water lines leaving the water softener. This means one meter measures hot water, but before it’s heated, so I can use a cheaper meter that isn’t rated for hot water. I ended up buying these meters: Assured Automation Economy Plastic Water Meter - WM-PC Series. I selected the 3/4" model, measuring gallons, with 10 pulses per gallon output. Since it has 10 pulses per output, I can measure both rising and falling edges resulting in 20 counts per gallon, but the pulse width is not close to 50% so there is a bit of noise in the signal.

The Electronics

Since I want this to integrate with my home automation system, the data from this water meter needs to end up on Ethernet, and pushed to an MQTT server. In 2021, the microcontroller of choice for WiFi or Ethernet is the Espressif ESP32. I specifically chose the Olimex ESP32-PoE board, which features wired Ethernet with Power over Ethernet, although the power supply on the board is not capable of handling anywhere near the 12.5W the PoE spec should allow. For this particular design, the low power is not important, we are just powering the ESP32 itself and the two dry contact switches with pull up resistors. It also helped that I’ve used them before in more than a few projects.

The circuit is very simple - I used RCA connectors to connect each meter, wired both grounds to the ESP32 ground, and wired each signal to a digital pin on the ESP32, with a 4.7K pull up resistor to 3.3V. All of this was done using solid electronics wire and heat shrink without a PCB.

The Enclosure

Astute observers will notice the similarities between this enclosure and the Salt Level Sensor. I built both of these projects around the same time, and chose to use the same hexagonal case design as that sensor for this project (you’ll also notice later that I copied and pasted the code, too). While there’s no reason for the case to be hexagonal, it is, and I printed it in a nice blue. Here’s a 3D model of the case design, showing the ESP32-PoE with Ethernet jack visible, the lower case with holes to mount both RCA jacks, and the upper case, which is an identical part to the Salt Level Sesor.

Transparent housing showing electronics mounting

Interior view of main housing showing PCB mount (will be hot glued)

Ethernet-end view of housing, cap transparent

The Software

As mentioned earlier, the software for this project was largely copied from the Salt Level Sensor. It’s built in Visual Studio Code using PlatformIO, using the ESP-IDF backend for ESP32 (instead of Arduino). All of the parameters are essentially hardcoded, and I compile/flash the board when I make changes. It isn’t as elegant as a nice web UI to set configuration parameters, but it works for me. Maybe in the future I’ll spend time developing a clean submodule for ESP-IDF projects to configure the Ethernet/MQTT backend components. Until then, you can feel free to browse the code on github.. After developing both this and the Salt Level Sensor, I realized that I didn’t ever set the hostname to match the MQTT client identifier - so both of these sensors show up as ’espressif’ on the network and fight over the DNS name assigned to DHCP clients. Not a big deal, but it’s a bug I would fix if I revisit this.

The software reads the two pulse counter pins in a 10ms timed tasks. Once the pin states are read, they are logged in a bit array, such that each bit in an integer contains a previous value of the pin, and by bit shifting the integer the previous values are copied one entry further back in the array. This method stores the last 32 pin states (320ms), but only the last 6 are used. If all of the last 6 pin states are the same (all 1 or all 0), then the pin state is determined, otherwise, the pin state is not changed from the last known state. If the pin state is determined and is different from the last known state, the counter is incremented. This provides sufficient debouncing for the water meter at maximum flow rate and provides some resilience to electrical noise (combined with the 4.7K pull up resistor). In a separate task, the data is periodically published to MQTT. At a minimum, it will publish a payload containing the two counters every 30 seconds, but it will publish instantly if either counter changes, with a maximum publishing frequency of every 1 seconds. The payload is JSON encoded, with each field named SenseX and each value being a number

The MQTT data is picked up by Node-Red (Telstar) and converted into gallons (using a scaling factor of 20 pulses per gallon), and re-published as cumulative gallon data to be logged in InfluxDB. In Grafana, I take the non-negative-difference followed by the sum to get the daily water consumption. Here are some helpful equations for dealing with the cumulative water consumption numbers when the numbers reset to zero whenever the sensor node reboots:

Graph the consumption of water over the time window in Grafana

  SELECT cumulative_sum(non_negative_difference(mean("Cold"))) FROM "mqtt_consumer" WHERE ("topic" = 'energy/water/meters') AND $timeFilter GROUP BY time($__interval) fill(null)

Return an array of differences, which can be summed by the Gauge visualization in Grafana

  SELECT non_negative_difference(max("Cold")) FROM "mqtt_consumer" WHERE ("topic" = 'energy/water/meters') AND $timeFilter GROUP BY time($__interval) fill(null)

Return the derivative (flow rate)

  SELECT non_negative_derivative(max("Cold"),1m) FROM "mqtt_consumer" WHERE ("topic" = 'energy/water/meters') AND $timeFilter GROUP BY time($__interval) fill(null)

Putting It All Together

Parts in hand, this was a fairly simple project to install. I used 3/4" Sharkbite to 3/4" FNPT fittings along with plenty of Teflon tape to attach the water meter fittings to the existing PEX lines, one feeding the house and the other feeding the water heater. Water had to be shut off a few times during the install, but it was a fairly painless process overall. I’m not a fan of using either Sharkbite or NPT fittings, especially because NPT fittings rely on interference of the threads to seal, which isn’t a well engineered solution, but it’s just impossible to avoid them in home plumbing.

Water meter installed in water line feeding hot water heater

Once the meters were installed and the water was back on, I got to setting up the electronics. I bench tested the electronics module, sealed the cover with hot glue, and installed the electronics. The cables on the water meters are long enough that I could mount the electronics module a bit out of the way, in case there is a water leak in the future. I mounted the electronics using special zip-ties with a hole in them to mount using a screw, they are very handy for stuff like this. I neatened the cables, pulled a new cat5e from the sensor back to my patch panel, and it was good to go.

Testing the electronics before final mounting

Final installed electronics

The Project Files and Parts List

Here are links to the files and parts required to make this project. As usual, all design files are licensed Creative Commons CC-BY-SA unless otherwise noted.

Measuring Water Softener Salt Remaining with ESP32

Fri, 30 Apr 2021 12:00:00 +0000

I am currently living in my parents house, which is fed water by a private well. Due to the mineral content in most well water, we use a water softener. For those of you with city water, a water softener is a type of ion exchange filter which uses salt (sodium chloride) to replace calcium, magnesium, and other metals in the water with sodium. None of these minerals are hazardous, but they do stain everything, so reducing them is desirable. This process consumes salt, and the brine tank containing salt and a bit of brine water must be periodically refilled with bags of salt from the hardware store. These bags are cheap but heavy, and since it takes many months to go through a full tank of salt, it’s an easy task to forget until the water quality drops off. Thus, it’s a prime candidate for automated monitoring and alerting.

Since this is a long project, I’ve left some links below for you to skip around. Enjoy!

The salt tank to be measured

The Video

There is a video corresponding to this project! Click the thumbnail below to see it.

Design Concept

The design concept for this sensor was to take an off-the-shelf, sealed, ultrasonic distance measuring sensor and thread it through a hole in the lid of the salt brine tank. The sensor I chose was the Maxbotix MB7052 with a 1" NPS (National Pipe Straight) housing. An NPS thread uses the same thread profile and pitch as the more common NPT (National Pipe Taper), but without a taper, meaning it isn’t designed to seal. As the sensor has a 1" NPS housing, I need a 1" NPS nut on each side of the tank lid to secure the sensor. I found a suitable nut at McMaster-Carr, and downloaded their CAD file. I printed the CAD file as-is initially, to prove that the printer could resolve the thread detail well enough, and was satisfied with the fit between the printed part and the sensor. I then imported the CAD model with threads and used the hexagonal shape of the nut as the basis for the hexagonal design of the whole module. I thought it looked cool, the ESP32 board packaged well, and the Ethernet cable stuck out the top. The whole electronics module threads onto the back of the sensor, the sensor is hot-glued into the electroincs module, the wiring is completed, and finally the whole assembly is threaded through the tank and the bottom nut tightened. I eventually used this same hexagonal case on the Water Meter project, although there was no reason to stick with the hexagon other than aesthetics on that project. In fact, that project also copied the software from this project as a starting point.

The Hexagonal Housing, minus lid

Electronics

Given the desire to push all of my automation data to Ethernet as soon as possible, a logical choice for this project would be the ESP8266/ESP32 family of microcontrollers from Espressif. These low-cost MCUs contain built-in WiFi, and the newer ESP32 features built-in WiFi and Bluetooth (but not at the same time!), and support for many more peripherals, including optional wired 100Mbit Ethernet. There aren’t a lot of development boards with Ethernet PHYs on board, but the Olimex ESP32-PoE board is what I ended up choosing, having previously used it for the Air Quality Project. It’s reasonably priced, supports PoE (even if the power supply can only handle a few watts, something I’d struggle with later), and using wired Ethernet with PoE for a sensor in my basement makes power and data that much easier to deal with.

Aside from the ESP32 board, the Maxbotix sensor is wired directly to pins on the board and no other circuitry is required. I did end up adding a large capacitor across the power leads on the Maxbotix sensor to smooth power pulses during ultrasonic pings, and this greatly improved stability of the whole device when operating on PoE power. The Maxbotix sensor will run continuously by default, so I only need to wire power, ground, and UART TX from the sensor to UART RX on the ESP32.

Software

As mentioned earlier, the software for this project is the basis for the Water Meter. It’s built in Visual Studio Code using PlatformIO, using the ESP-IDF backend for ESP32 (instead of Arduino). After using Arduino for ESP32 in the Air Quality Project, using the well-designed and well-documented ESP-IDF and PlatformIO is a very nice improvement to the code quality. All of the parameters are essentially hardcoded, and I compile/flash the board when I make changes. It isn’t as elegant as a nice web UI to set configuration parameters, but it works for me. Maybe in the future I’ll spend time developing a clean submodule for ESP-IDF projects to configure the Ethernet/MQTT backend components. Until then, you can feel free to browse the code on github.

The Maxbotix sensor is connected to a UART input on the ESP32, which reads and decodes the serial data stream. As the data is passed as ASCII text, the UART is configured to detect a pattern, which corresponds to a new line. When a new line is detected, UART driver adds the message to a queue, which is read by the Maxbotix task. Resulting messages are decoded into samples, which are stored in a circular buffer to be processed. If the time between a sample and the previous sample is excessively long, the circular buffer is emptied and the collection process starts over.

Every 60 seconds, the data buffer is processed and the results are published to MQTT. A median filter is implemented. The median filter copies the last 64 samples into a local buffer, sorts the local buffer, removes the top 20% and bottom 20% of samples, and takes the mean of the resulting samples. As the Maxbotix sensor reports in integer centimeters but contains some sample noise, this is intended to improve accuracy of the number. The resulting sampled value, as well as the number of samples used to compute the median, are published to MQTT in JSON format. A low number of samples indicates a problem with the Maxbotix sensor. Due to the high power draw of the sensor during ‘ping’ events, I had trouble getting the sensor to work reliably using PoE power, but eventually resolved this with large enough capacitors on the sensor power input.

Putting It All Together

Assembly was pretty simple. I found the nearest hole saw size to the outer diameter of a 1" NPS thread. As with many other American units, 1" does not mean 1", in this case it means the size of an iron pipe with a 1" inner diameter, so actual outer diamter is 1.315". Imperial fist shake! The sensor had already been bench tested during software development, so I threaded the sensor and ESP32 through the housing for the last time and secured the sensor from the inside with hot glue. I hot glued the cover on, slipped the sensor into the hole in the tank lid, and secured it with the nut I’d printed earlier.

Sensor attached to tank and functioning

View from the inside of the tank

The measurement from the sensor is published in floating point centimeters. The tank is about a meter high, so I need to subtract the sensor reading from the height of the tank to get the centimeters of salt remaining in the tank. However, the bottom of the tank is always full of ~20cm of water, as the softener fills the bottom of the tank with water to dissolve the salt into brine, then pumps some of this water through the ion exchange resin during the regeneration process. The data is processed by Node-Red, where it is pushed to InfluxDB so I can view it in Grafana. The salt level periodically drops as the water softener regenerates, then it starts cycling between the high and low water levels frequently as the level of salt is lower than the level of the water. This is when I know it’s time to refill the salt.

Graph of 3 months of data in Grafana, including one vacation and one salt refill

Adding Home Assistant Support

After almost a year with this sensor, I’ve migrated the home automation from Node-Red to Home Assistant, so I updated this page with the new HA configuration bits. I’m using an MQTT sensor to bring in the raw distance reading from the sensor, and a template sensor to convert from raw distance to a tank full percentage. Since the sensor measures down from the top, but we would like a percentage up from the bottom, we have to do a tiny bit of math. Both of the yaml snippets are below.

Salt Level Sensor

This should go in sensors.yaml. You’ll need to figure out the ESP’s MAC address, using a tool such as MQTT Explorer. In this case, it’s esp-6aba77, so the last 3 bytes of the MAC are 0x6ABA77.

#Salt Level Sensor Reading
- platform: mqtt
  name: "Salt Level"
  unique_id: "salt_level"
  state_topic: "raw/esp-6aba77/ultra"
  value_template: "{{ value_json.Dist }}"
  unit_of_measurement: "cm"

Salt Level Percent Template Sensor

Obviously if you already have a template and sensor section in your configuration.yaml, put this inside instead of creating a new one.

template: 
 - sensor:
      #Salt Level Percent
      - name: "Salt Level Percent"
        unique_id: "salt_level_percent"
        unit_of_measurement: "%"
        #In this example, the tank is 140cm tall.
        #Substitute 140 with the total height of your tank in both places
        state: "{{ (140 - (states('sensor.salt_level') | float))/140*100 }}"

Project Files and Parts List

Here are all of the files and parts required to replicate this project. As usual, all design files are licensed Creative Commons CC-BY-SA unless otherwise noted.

My Very Tiny Virtualization Lab

Tue, 16 Feb 2021 12:00:00 +0000

After my experience with FreeBSD Jails and LXC containers, I wanted to get into ‘real’ virtualization - and all of the advantages that come with it, like VM snapshot and restore features, moving VMs around between my workstation and production environment, and separating my storage from my compute. To this end, I built the Minilab, a small scale virtualization lab that will be at home in any house or appartment.

The Choice of Hardware

I already had ‘production’ work running on my main server (including my very overworked automation server Telstar and my security camera recorder ZoneMinder), so I needed some new hardware to avoid downtime in the transition process. I built a my new virtualization server around an Asrock Deskmini A300, which is a small form factor PC barebones which supports fairly modern AMD Ryzen APUs. It sported a few things that made this attractive to me as a minilab:

AMD Ryzen APUs are quite power efficient, and include a decently powerful Vega GPU (if I can ever pass that through to a VM without breaking the host)
One M.2 slot for WiFi cards and one M.2 slot for NVMe storage (but not SATA, as I would learn)
Space and cabling for a 2.5" SATA SSD
A full suite of video outputs (HDMI, DisplayPort, and VGA), which is important since I have an old VGA monitor next to my server rack
On-board Gigabit Ethernet, USB-2, USB-3, and USB-C
Single 19V power brick with all power supply circuitry on the motherboard
No extra IO that I don’t need

In all truth, I originally bought this SFF PC to be the basis of a 3D-prined PC project (since it’s small enough to 3D print a case on my Prusa), but I never actually built that project, and it was available, so it became the virtualization lab. I fitted it out with 16Gb of DDR4-3000, a 240Gb 2.5" SATA SSD, and most importantly, an AMD Ryzen 2400G APU. I know the 2400G is older than the 2400G in my production server, but I bought the parts for this back when the 3400G was brand new and the 2400G was on sale, so it’s what I’m going with.

Setting up the Minilab

The first choice for software on the Minilab was XCP-NG, a hypervisor operating system based around the Xen hypervisor. Xen is the open-source basis of a number of hypervisors including Cirtix XenServer, the open-source XCP-NG, and the privacy focused Quebes OS project, among others. XCP-NG combined with Xen Orchestra for management seemed like a good choice to get started with, given the desire to stay with open-source software (so Hyper-V and VMWare are out), and the great snapshot and backup features that Xen Orchestra provides on top.

The Minilab has a 240Gb SSD which is available for VMs, and the Xen Orchesta VM is stored there (so I can manage the minilab if the remote storage has an issue). I also attached a zfs dataset on my primary server over the network for VM storage, and VMs can of course map their own network shares on that server as well. All seems well at this point

Setting up Home Assistant

The first ‘real’ VM I ran on the new minilab was Home Assistant. I imported the image of Home Assistant OS, expanded the filesystem to 32Gb, and started playing around. I don’t have a project article on my initial dealing with Home Assistant (I’m certainly not an expert on the topic yet), but it was the first real test of the Minilab.

Setting up Frigate

After many months of using the Minilab with Home Assistant, I wanted to add Frigate NVR as a new VM. Home Assistant OS supports running it as a supervised add-on, but I wanted to store the video data on my primary storage pool over the network, and Home Assistant OS doesn’t allow any package management or file mapping within the OS (it’s purely a Docker host managed through Home Assistant Supervisor). So, I created a new Ubuntu VM to run Frigate, and bought a pair of Coral AI devices to offload AI inferencing for Frigate. I was able to install Frigate and run it, but with just CPU it was unable to handle my full suite of 5 security cameras. I kept it going for just one camera, and let it run for a few weeks to get a feel for how well Frigate works with Home Assistant. This is the first time I’ve gone straight to the minilab to test new software instead of running a VM on my workstation in VirtualBox, and I’m really enjoying having a compute server to use, even if it’s not very powerful.

Unfortunately, when the two coral.ai devices came (one M.2 B+M key and one M.2 E key, finally a use for the WiFi slot on the A300 Deskmini!), I struggled to get PCIe passthrough working in XCP-NG. I was able to drop the devices from dom0 (the Linux host which manages the Xen hypervisor), but couldn’t get them to pass through. Xen was not showing IOMMU as enabled, even though the CPU supported it, and IOMMU support is required for PCIe passthrough.

Switching to Proxmox

Still overall happy with XCP-NG, I decided to switch to Proxmox to get better hardware passthrough support. It’s based on Debian Linux and uses the KVM hypervisor (Linux Kernel-based Virtual Machine), so it should have the advantage of Linux’s excellent hardware support. As expected, I just had to enable AMD IOMMU in the Linux boot options and it seemed to work correctly. The host Debian system kept trying to use the Coral’s itself, so I had to blacklist the drivers for those, but they still didn’t show up in the PCIe passthrough menu. I was able to get through it using the command line, but it wasn’t a smooth process. But, now, I have an Ubuntu 20.04 VM which has the two Coral AI accelerators available for use, and I can install Frigate.

Because I blew away my XCP-NG install, I had to backup and reload my Home Assistant installation across hypervisors. I decided to do a backup in Home Assistant OS and also a backup in Xen Orchestra, so I’d have options in the restore process. When restoring, I created a new VM in Proxmox with Home Assistant OS, using the latest install image, and reloaded the backup file there. After a few minutes it had figured itself out and was back exactly as I’d left it, no need for the disk image from Xen Orchestra.

The Final Verdict

I liked XCP-NG and Xen Orchestra, but I needed better hardware support that Proxmox provides. I’m not sure which I’ll end up going with in my next homelab, but I’m certainly happy to get away from the console LXC commands of my current server. I’m not using the ZFS features on Proxmox since I already have a ZFS pool on the storage server, and I like the separation of compute from storage, since a lot of my storage is accessed directly, not via the VMs. In my mind, this will become an advantage as I scale up and the load on the storage part of the system increases, but in reality it’s still mostly just a fictional future load for a future homelab.

My Experiences with a ZoneMinder Home Security Camera System

Sat, 04 Jul 2020 12:00:00 +0000

This project describes my process of building a home security camera system using Dahua PoE cameras and ZoneMinder. Overall the system is functional, but Zoneminder leaves a bit to be desired. I will revisit this project in the future.

The Beginnings of a Camera System

The real OG camera system was installed by my dad a decade ago using analog ‘960H’ cameras (which are grossly misleading in their advertising since they market the horizontal resolution instead of the usual vertical resolution, they are actually 480x960 at best). The system used a Chinese DVR which was limited to 8 cameras, and one after another they failed due mostly to poor quality cabling. The image quality was also awful. So, a new setup was desirable. We don’t live in a very dangerous area, but we like to look at the cameras to check on things when we go on vacation, and they can also be useful to keep track of deliveries and who’s home or away.

For the ‘New’ first camera system, I wanted to go all IP based, with plenty of room to upgrade in the future, and complete local control. I wanted to be free of any cloud services. I didn’t mind paying for licensed software when it’s the best option, but I’d prefer an open source solution and absolutely require Linux or BSD for servers. I already had a PoE managed network switch (Mikrotik CRS328-24P-4S+RM), so the next choice was the cameras.

Based on the camera review of intermit.tech, I looked at Dahua, one of the two major Chinese camera OEMs (the other is Hikvision). I decided to try the DH-IPC-HDW2431TP (4MP/1440P ‘Lite’ series) and DH-IPC-HDW5831RP-ZE (8MP/4K ‘Pro’ series, the one he calls the ‘old champ’ in a few videos). The Pro series cameras have a better build quality, higher resolution, and a varifocal lens that can be considerably sharper in certain scenarios. The Lite series cost 1/3 the price with a fixed focus lens, so you have to design around the focal range of the lens. I ran an Ethernet cable to my first location and setup the Pro for a week, followed by the Lite, and compared the two. Based on this test, I decided that the Lite series was perfectly adequate for most of the views I wanted to capture. Most of the camera locations I wanted were near entry doors, so the fixed focus lens was a good match for close range imaging above each door. I then bought a 5-pack of the Lite cameras to build out the whole system (7 cameras in hand now).

Short Review of the Dahua Cameras

I highly recommend you review intermit.tech’s IP Camera series (Blog Link)(YouTube Link). He’s done a good job at showing the features and visual comparisons of each camera. The right camera for you is very subjective. In my case, I was on a fairly low budget, and not trying to spot a license plate or face at the end of my driveway, especially not at night.

In my case, here are my observations which may be useful to you:

The software security of the cameras impressed me, given the my very low expectations based on the reputation of cheap IoT devices. When first connecting to the camera, it made me choose a new password, offered me the choice to enable or disable cloud services, and hasn’t made any attempt to call home as verified by my router. This is a professional product, albeit from China, and it’s software design shows some care has been given to security.
The Web UI is a bit basic, but it’s functional to navitage and configure the camera. The cameras support both h.264 and h.265, although I operate them in h.264 mode to remove the need to transcode the recordings to view them (since no web browser other than Safari natively supports h.265 as of the writing of this).
The Lite cameras support a main stream of up to 2560x1440x30fps (1440p) and a single sub stream of up to 704x480x30fps (D1). You can independently select the resolution, framerate, encoding mode (constant/variable bit rate), maximum bit rate, i-frame interval, and codec (h.264 or h.265) for each stream.
The Pro cameras support three streams. The main stream supports up to 3840x2160x15fps (4K), and the two sub streams support up to 704x480x15fps (D1) for sub stream 1 and 1920x1080x15fps (1080p) for sub stream 2. All 3 streams can be enabled at once, and again have independent configuration of all parameters
The cameras show up in ONVIF Device Manager, as they should, being advertised as ONVIF compliant. I don’t use an ONVIF compliant NVR, so I can’t test intgration with other hardware, but ONVIF Device Manager can help you identify the streams if you need help with that.

When wiring the cameras, I bought Ubiquiti Toughcable, which has a black polyethylene (PE) jacket instead of the usual PVC, so it does not degrade in sunlight. It seems like they have discontinued this cable, but any outdoor rated cable, preferrably non gel filled, will work for outdoor wiring. I had to run some cameras 50’ or more outdoors along a brick ledge between the first and second floor of the house, but for short runs of a few feet an indoor rated cable should be fine.

The Great NVR Test Campaign

I initially tried a number of open source NVRs in a virtual machine on my personal workstation using Virtualbox. The goal of these tests was to estimate the CPU and RAM requirements for a single camera (by limiting the VM to one core or less on my Ryzen Threadripper 1950X), test scalability with the 2 cameras I had installed at the time, and evaluate the installation process, maintenance, UI, and apps.

The tests included the long-running open source project ZoneMinder, Motion, and Shinobi. ZoneMinder was functional but required a lot of menus to setup, but offered a ton of configuration of exactly how to extract the video (using ffmpeg internally) and manage the streams. Motion seemed to lack the comprehensive system feature set I wanted, and more focused on being lightweight. At the time of this review (2020), Shinobi’s setup did not function correctly in my Ubuntu virtual machine and overall it was a very painful process. Considering the recent age of the project it’s possible the developer has improved it.

I also tried trial versions of a few commercial products including Xenoma and Blue Iris, but was dissatisfied with both. Xenoma licensing does not allow running personal licenses in a virtual machine, which would drastically raise the price for me running virtual machines at home. Blue Iris only supports Windows, and I’d like to avoid a Windows virtual machine just for this. My goal was to run this software in a LXC container on Ubuntu (18.04 Bionic Beaver), with the host running a ZFS pool for storage. At this point in the project, the NVR was expected to run on the existing storage server, which was not very powerful and the low overhead LXC container was preferrable over a paravirtualized Linux or hardware virtualized Windows virtual machine. My knowledge of virtual machines was also poor going in to this project.

The Choice of Zoneminder

Zoneminder is very functional and reliable, as you would expect from a nearly 20 year project still in active development. However, it’s design age is showing. When the project was started, it was expected that analog cameras would be connected to a v4l capture card, and presented as a raw framebuffer for use by the processing tasks. It still somewhat operates this way internally, with the incoming camera stream decoded into a framebuffer for analysis. All detection is done purely by the CPU on raw decoded frames, and there is no native support for using lower resolution streams for motion detection and higher resolution streams for recording. The solution around this limitation is to setup two monitors, one with the low-res stream and one with the high-res stream, and have the low-res stream run detection and trigger the high-res camera so both record on motion. This results in twice as many recordings (one for each stream), although the low-res recordings are significantly smaller. It also means that analyzing and tweaking the motion analysis requires watching the low resolution 704x480 D1 resolution video to see the analysis results. After all of these negatives, I still liked the software and went forward with setting it up on the production server

Upgrading to New Hardware

The production server at this point in time was an AMD Bulldozer family APU designed for mobile applications, but I bought it as an embedded mini-ITX board with a soldered CPU. This APU performed well running Ubuntu 18.04 Bionic Beaver, as well as ZFS on Linux for the data pool. Prior to the installation of Zoneminder, it was already hosting an LXC container for the first generation home automation system, which comprised of an MQTT broker, InfluxDB server, Grafana server, and Node-Red server, with all logic being done by Node-Red. The Bulldozer CPU was a quad-core, but the AMD construction equipment family (Bulldozer/Piledriver/Excavator) was famous for having tightly coupled core pairs sharing execution units more similar to hyperthreading than truly distinct cores. To attempt to simulate the performance of this low end CPU, I ran my Zoneminder virtual machine with 2 cores of the Threadripper and an execution cap of 1/2 each CPU. It ran adequately with the 2 cameras in my test setup, although it was certainly reaching the limit of that hardware. Installed on the Bulldozer, it was unable to run more than two cameras with low resolution motion detection.

As it was unable to run the 2 cameras I already had installed and I still had 5 more cameras to install, a hardware upgrade was in order. I ended up replacing as few components as possible, resulting in the following hardware setup:

AMD Ryzen 5 3400G APU (4c/8t)
ASRock B450M Micro-ATX motherboard
Intel X520-DA2 10G SFP+ network card (half-height)
16Gb DDR4-3200 RAM in two sticks of 8gb each
Existing 240Gb SSD boot drive and data storage pool were retained
Existing 120W PicoPSU was retained, along with the AC to DC converter, which was secured into the case with double sided foam tape.
Existing 80mm case fans were upgraded to Noctua equivalents since the cheap Rosewill case was very loud

With the new hardware setup, I was able to install the Ubuntu HWE (HardWare Enablement) package to get the latest kernel graphics drivers and permit the LXC container to access the GPU for hardware accelerated h.264 decoding. I was also able to connect the server to the switch with a 10G SFP+ DAC cable, although at this point in time there are no clients which can use 10G anyway, so at best it currently just allows higher aggregate throughput for multiple clients, and in reality it’s just the start of a future expansion to a 10G backbone.

Final Verdict

ZoneMinder is currently functioning and has been for several months. I continuously tweak the motion zones and thresholds to reduce the false triggers due to swaying tree shadows, flies, and spiders making webs in front of the camera. The visible area from two cameras could be improved by tree trimming. The LXC container has a 1Tb ZFS quota, ensuring it doesn’t fill my entire storage pool with months of recordings. The mobile app (zmNinja) works well, when combined with my personal home VPN (direct from my phone to my home router).

At a rate of approximately one per month, I installed 3 of the remaining 5 cameras around the house, leaving the last two as winter approached and I didn’t want to deal with running another conduit through brick to access the rear walkout of the house. But, there aren’t packages or cats waiting at that door, so it’s not high priority at this point in time.

Fall 2021 Update

The system described in this project was operational for 2 years without major issues. In the future, this setup was replaced with Frigate NVR, but the cameras, wiring advice, and software testing listed here is still valid.

Installing a Whole-House Per Circuit Energy Meter

Thu, 04 Jun 2020 12:00:00 +0000

After setting up my 1-Wire Temperature Network, my next sensory target was my AC breaker panel. I watched reviews for products such as the Sense energy monitor, The Energy Detective, and a few others, and felt like none of them did what I wanted. I know that big loads in my house use a lot of power (duh), I want to know how much power specific people and rooms in the house use, so I can improve my home energy modeling skills. This, to me, required a sensor on each device or circuit, to directly measure as much as possible and not rely on ‘AI’ buzz to pick out a few major appliances.

The Brultech GreenEye Monitor

The energy monitor I chose is the Brultech GreenEye Monitor (GEM). I chose this monitor because of the following attributes:

Offers WiFi and hardwired Ethernet, with a well-documented and purely local API (no cloud required)
Offers 32 channels of true power monitoring (although it only measures phase voltage on one phase of a split-phase North American supply, so there will be a very slight error on half of the channels). This is a lot of channels for one device
Packages include donut CTs for regular circuits (40A or less) at an excellent price per circuit compared to anything else on the market
It also has pulse and one-wire monitoring, but I already have a 1-wire network so I won’t be using either of those features

I ended up buying the GEM+CT Package and getting 32x 40A donut CTs, plus buying an extra 3x 60A split CTs for really big circuits. Since the total number of circuits in my panel is 33 (including all double-pole circuits once), I left out the master bath jet pump circuit since it’s infrequently used and did not monitor the mains. I had a 20A and a 15A double pole circuit which I used with one 40A donut CT each (same as all the other 15A and 20A circuits), since they are for motors with no neutral. I had a 60A air conditioner which got a single Split-60, and a 50A stove and cooktop which got a pair of Split-60s wired electrically to the same channel on the GEM.

Installation Process

Installing the CTs took awhile, and with the power off to the whole building, I didn’t get any pictures of the process. Before I turned off the power, I made heat-shrink labels for each CT with a number on them so I could keep track of the CTs. I turned off the main breaker, removed the cover, and planned the wiring route for the 33x CT wires. I mounted the GEM next to the breaker panel but close enough that I could route the wires around the side without crossing over a stud, and ran the wires through a plastic bushing in the bottom of the panel, through the wall, and into the GEM. Each 40A donut required me to unscrew the wire from the breaker, remove it, slip it through the CT, and screw it back in to the breaker. Every few circuits I zip-tied the CT wires together and neatly dressed them down both sides of the panel, across the bottom, and out. I went down one side with the CTs numbered 1-15, and then down the other side with the rest of the 40A donuts, and then did the 60A ones last. I knew I’d be able to count breakers later to match CT numbers to circuits. Once the GEM was installed, I planned on running a tap off an exsting 15A circuit to a new outlet box right at the panel (the nearest outlet is ~8’ away), but I didn’t end up doing that. I closed the panel and power was restored, and there was much rejoicing.

After physically installing the GEM, I pulled a new Cat5e wire from the patch panel into the GEM and crimped it with an RJ45 male. I ran a 6’ 2-wire extension cord to the GEM to power both of the power supplies - the GEM needs an AC reference voltage which it gets from an AC wall-wart, plus a 5V DC supply to power itself.

GEM installed, with PoE splitter attached to lid

Setting up the Software

I had just finished setting up Telstar, my automation server, and was eager to capture the data from the GEM in Node-Red. I ended up setting the GEM to do an HTTP request to a webserver at a fixed interval (5sec) with each value as a parameter. I setup Node-Red with an HTTP In node to capture this, then used some node-red commands to filter out and parse the strigns into floats (they all come in as strings, unfortunately), then I wrote some Function nodes to take the derivative of the energy totals and re-bundle everything into JSON to pass to MQTT. The Node-Red flow I used to do this is available below.

Shortly after getting all of this working, I realized that I can’t tell the difference between power being out (actually out), something wrong with the GEM, or someone accidentally unplugging my 6’ extension cord. To remedy this, I got a PoE splitter and taped it to the case of the GEM with double sided foam tape. It outputs the 5v that the GEM needs to power itself, and is powered by my PoE switch which is backed up by a UPS. Now, I can look at the voltage reported by the GEM (which should be around 120V) and tell if either the power is out or someone unplugged the potential transformer, and fix the situation, without losing connection to the GEM in the process. The PoE splitter I used is a TP-Link TL-POE10R.

Using my brand new Grafana server, I started experimenting with the best way to view the data. I’m not ready to share the entire graph code, but I have a dashboard showing total power consumption today, current line voltage, and pie charts showing total energy usage for each floor of the house. I also have a few charts of ’everything’, which are really difficult to view but are designed to be exported to a csv for analysis.

Top of the Grafana page showing the daily totals and data from the first floor

The Project Files and Parts List

Here are links to the files and parts required to make this project.

Update Spring 2021

Someone tripped over the AC reference supply and broke it, since I never finished wiring the outlet right at the breaker panel. I emailed Brultech, since I was unable to find a power supply with a 3.5mm jack. I provided my serial number, and they sent me a new one for just the cost of shipping (from Canada). The new transformer arrived and power measurement is working again. Happy with the customer support from this company.