mtbales.com

I found myself needing to do an import of several /24 subnets into our new ISC-Kea servers and was unable to find an existing solution. So i wrote a quick and dirty importer that accepts isc-dhcp conf files, pulls all the required reservation info (does not include options and other non-essential data) and puts it directly into the kea postgres host database. There is an existing project that uses the isc-kea web / stork web api to convert memfile to db and one to convert the dhcpd.conf file into the kean.conf file json format, but nothing for importing from dhcpd.conf to a database. You can check it out on my GitHub page here: https://github.com/slapplebags/ipv4-dhcpd-to-kea

Additionally I wanted to get AD / LDAP auth working with isc-stork, the web front end for isc-kea and had some difficulty with the very spartan documentation. Rather than LDAP I am using Samba4 as an Active Directory domain so some settings are specific to that. Below is a working config, note that the lines starting with # are commented out as i am not using them or they don’t work as expected.First you’ll need to install the isc-stork-server-hook-ldap package or from sources, then add the following (modified for your environment) to /etc/stork/server.env

### LDAP / AD auth settings
STORK_SERVER_HOOK_LDAP_URL=ldap://127.0.0.1:389
STORK_SERVER_HOOK_LDAP_ROOT=dc=foo,dc=bar,dc=com
STORK_SERVER_HOOK_LDAP_BIND_USERNAME=stork(,ou=services (this is only needed if your bind account is in a different ou from your users)
STORK_SERVER_HOOK_LDAP_BIND_PASSWORD=password
STORK_SERVER_HOOK_LDAP_SKIP_SERVER_TLS_VERIFICATION=true
STORK_SERVER_HOOK_LDAP_GROUP_ALLOW=StorkAdmins
STORK_SERVER_HOOK_LDAP_MAP_GROUPS=true
#STORK_SERVER_HOOK_LDAP_DEBUG=true
#STORK_SERVER_HOOK_LDAP_GROUP_ADMIN=cn=StorkAdmins,ou=Users,dc=foo,dc=bar,dc=com (as of 1.16.0 this does not work, you have to create a group with the default name: stork-admin)
#STORK_SERVER_HOOK_LDAP_GROUP_SUPER_ADMIN=cn=StorkAdmins,ou=GRIT Users,dc=grit,dc=ucsb,dc=edu(as of 1.16.0 this does not work, you have to create a group with the default name: stork-super-admin)
STORK_SERVER_HOOK_LDAP_OBJECT_CLASS_GROUP=group #AD specific
STORK_SERVER_HOOK_LDAP_OBJECT_CLASS_GROUP_MEMBER=member #AD specific
#STORK_SERVER_HOOK_LDAP_OBJECT_CLASS_GROUP_COMMON_NAME=
#STORK_SERVER_HOOK_LDAP_OBJECT_CLASS_USER=
STORK_SERVER_HOOK_LDAP_OBJECT_CLASS_USER_ID=sAMAccountName #AD specific
#STORK_SERVER_HOOK_LDAP_OBJECT_CLASS_USER_FIRST_NAME=
#STORK_SERVER_HOOK_LDAP_OBJECT_CLASS_USER_LAST_NAME=
#STORK_SERVER_HOOK_LDAP_OBJECT_CLASS_USER_EMAIL=

With this config i am able to have members of the StorkAdmins group (and the stork-admin group) login and manage isc-kea dhcp via the stork web ui. There is a ticket in for the group mapping issues: https://gitlab.isc.org/isc-projects/stork/-/issues/1369.

I recently started a little side project using wiki.js and wanted to be able to automatically create, search for, and edit wiki pages. wiki.js provides a graphql API to do this but documentation is a bit sparse. Here is a working example for creating a new page:

mutation = '''
mutation Page (
  $content: String!, 
  $description: String!, 
  $editor:String!, 
  $isPublished:Boolean!, 
  $isPrivate:Boolean!, 
  $locale:String!, 
  $path:String!,
  $tags:[String]!, 
  $title:String!
) {
  pages {
    create (
      content:$content, 
      description:$description, 
      editor: $editor, 
      isPublished: $isPublished, 
      isPrivate: $isPrivate, 
      locale: $locale, 
      path: $path, 
      tags: $tags, 
      title:$title
    ) {
      responseResult {
        succeeded,
        errorCode,
        slug,
        message
      },
      page {
        id,
        path,
        title
      }
    }
  }
}
'''

variables = {
  'content': output_str, #string containing the page contents
  'description': description, #string containing the page description
  'editor': 'markdown', #I like to use the markdown editor
  'isPublished': True, #should the page be published? i haven't found a way to publish it after the fact
  'isPrivate': False, #should the page be visible to all users? i haven't found a way to publish it after the fact
  'locale': 'en', #other languages also available
  'path': '/show-notes/'+formatted_date, #URL path ie wiki.com/en/<your/path>
  'tags': [entry.published.strip('+0000'), description], #optional strings to tag pages to speed up searching 
  'title': formatted_date #page title
}

response = requests.post(url, headers=headers, json={'query': mutation, 'variables': variables})
print(response.json())

Searching for a page (by title in this example):

#SEARCH
search_query = '''
query SearchPage($titles: String!) {
  pages {
    search(query: $titles) {
      totalHits
      results {
        path
        id
        title
      }
    }
  }
}
'''

search_variables = {
  'titles': formatted_date #page title you are searching for
}
SEARCH
search_query = '''
query SearchPage($titles: String!) {
  pages {
    search(query: $titles) {
      totalHits
      results {
        path
        id
        title
      }
    }
  }
}
'''

search_variables = {
  'titles': formatted_date
}

And lastly the one i had the most trouble with, updating a page:

mutation = '''
mutation UpdatePage($id: Int!, $tags: [String]!, $content: String!, $isPublished: Boolean!) {
  pages {
    update(id: $id, tags: $tags, content: $content, isPublished: $isPublished) {
      responseResult {
        message
      }
      page {
        tags {
          id
          tag
          title
        }
      }
    }
  }
}
'''

variables = {
    'id': 66, #page ID which can be found via the search function noted above
    'tags': ['test', 'test2'], #tags to overwrite existing tages
    'content': 'Your updated page content here', #string containing data to overwrite the current page data
    'isPublished': True
}

response = requests.post(url, headers=headers, json={'query': mutation, 'variables': variables})
print(response.json())

Note that the tags and page content must be updated or you will receive an error. The wiki.js site indicates that this will be changed in version 3 of wiki.js.

I recently updated our OpenNebula stack to 6.4 CE from 6.2 CE and all was well until i tried to add a new GPU server to OpenNebula and do GPU passthrough to a VM. The Ubuntu 20.04 host was setup for GPU passthrough just like I’d done in the past but when added to OpenNebula the GPU was not show in the list of PCI devices. I tried adjusting filters, double checking the GPU passthrough setup on the host, all to no avail. After some searching I found the following issue in GitHub which is exactly the issue I was facing. Since we run the CE version no updates are available and the patch has to be applied by hand. To resolve the issue with GPU passthrough follow the instructions below:

• mv the file here: /var/lib/one/remotes/im/kvm-probes.d/host/system/pci.rb somewhere safe just in case
• Run the following to grab the patched pci.rb file from github: wget -P /var/lib/one/remotes/im/kvm-probes.d/host/system/pci.rb https://raw.githubusercontent.com/OpenNebula/one/3f300f3bf9ccd90fd082cb8b1ad85c0037911304/src/im_mad/remotes/node-probes.d/pci.rb
• after downloading the patched file make it executable: chmod +x /var/lib/one/remotes/im/kvm-probes.d/host/system/pci.rb
• restart the opennebula services
  • If you’ve already added a host and run into this issue you’ll need to remove the host
  • modify /var/lib/one/remotes/etc/im/kvm-probes.d/pci.conf and change the filter to '0:0' to filter all PCI devices.
  • Once that change has been made re-add the host and remove it again (this is required to clear the PCI data already associated with the host in ONE). Last step is remove the ‘0:0’ filter with your previous filters, restart your OpenNebula services and re-add your host and you should now see your PCI devices.

Been working on setting up a Linstor deployment to support several terabytes of data and ran into an issue that took some fiddling to sort out. I installed Linstor on a fresh Ubuntu 20.04 install per Linbit’s instructions and was able to enroll the node but it showed up as offline. The Linstor service was running on the satellite and firewall was open between the two but I just couldn’t get it online.

After a fair bit of fiddling i ran cat /proc/drbd and got the following:

version: 9.2.1 (api:2/proto:86-121)
GIT-hash: 86ec2326fef3aede9f4d46f52bfd35aac4d5eb7e build by <server1>, 2023-03-28 06:03:07
Transports (api:18):

While on a normal healthy node i got this:

version: 9.2.1 (api:2/proto:86-121)
GIT-hash: 86ec2326fef3aede9f4d46f52bfd35aac4d5eb7e build by <server2>, 2023-04-28 06:35:08
Transports (api:18): tcp (9.2.1)

The fix was easy, just run the following:

sudo modprobe drbd_transport_tcp

Also ran into an issue when setting up high availability for linstor converting existing nodes from satellite to combined. I couldnt find it in Linbit’s documentation but you can switch between modes:

linstor node modify <node name> --node-type <combined|satellite|controller>

I recently ran into some issues limiting DNS AXFR using the Turnkey Linux Domain Controller appliance. The Turnkey Linux image (version 16.9 at the time of writing) ships with Debian 10 Buster as the base image and includes Samba 4.9. There are two DNS options with versions 4.4 – 4.15, internal DNS which does not allow AXFR at all, and Bind9-DLZ which has no way to disable or limit AXFR.

Our use case requires DNS AXFR to be limited to just 2 up stream DNS servers. Given this limitation I had to use Bind9-DLZ but had no way to limit DNS AXFR requests. Shortly after giving up and replacing our TKL Samba4 BDC with an Ubuntu BDC which has packages for samba 4.15 I found a work around. Enter DNSDIST and a blog post describing using it to limit Samba4 Bind-DLZ AXFR requests.

DNSDIST is a package which provides DNS proxying and has options to limit AXFR and other requests. The blog post above didn’t work for me and needed a little tweaking to run. Below is the config I found works as of 3/2022 with Turnkey Linux DC v16.9:

first install and configure DNSDIST:

sudo apt install dnsdist
sudo vi /etc/dnsdist/dnsdist.conf

-- dnsdist configuration file

-- disable security status polling via DNS
setSecurityPollSuffix("")

-- listen address
addLocal('10.10.1.2:53')

-- allow queries from
setACL({'10.10.0.0/16'})

-- backend server
newServer("127.0.0.1:54")

-- drop every query type SOA, AXFR, IXFR with exception of trusted servers
trusted_servers = newNMG()
trusted_servers:addMask("10.10.1.1")
trusted_servers:addMask("10.11.1.0/30")
addAction(
  AndRule({
    NotRule(NetmaskGroupRule(trusted_servers)),
    OrRule({QTypeRule(dnsdist.AXFR), QTypeRule(dnsdist.IXFR)})
  }),
  DropAction()
)

next verify the DNSDIST config:

dnsdist --check-config

If it comes back OK restart bind and DNSDIST and verify that your DNS AXFR requests are limited as expect.

I was recently tasked with setting up a 24/7 livestream of the surf on a local beach from a PoE network camera to a public accessible webpage. Previously this was done with a 640×480 webcam sending an MJPEG stream to an Ubuntu server running FFMPEG which would then update the image on a webpage on ever 2 seconds. In addition to getting a better image at a double digit frame rate I wanted to include data from the local NOAA buoy in the stream.

First I went looking on a suitable replacement camera and settled on the Reolink 811a. Its PoE, affordable, high def, and given that the subject of the video stream is 200+ feet away the 5x optical zoom allows for less unwanted scenery in the shot. Next was to replace the static Apache webpage showing the 2hz image with a CDN to reduce our bandwidth and server requirements. YouTube was the obvious answer though after having completed the project I think I’ll investigate alternatives for future projects. Last step was to get FFMPEG to cooperate and this took by far the longest.

I had never used FFMPEG directly before so this was a learning experience. FFMPEG has lots of flags and while the documentation is exhaustive its difficult to parse and decipher. Looking around at forums for answers shows that there is a lot of varying opinions on what some FFMPEG flags should be used and what they do which doesnt help matters. I’m still honing my script but here’s what I have so far that seems to work to pull the h.264 stream from the reolink camera and send it to YouTube at 720p (I’m limited by the uplink from the camera being 10/100 ethernet):

ffmpeg -f lavfi -i anullsrc -rtsp_transport udp -i rtsp://surfcam:Rfs387Ax@128.111.28.194:554//h264Preview_01_main -vf "drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf:textfile=/surf/data.txt:reload=1:fontcolor=white:fontsize=44:box=1:boxcolor=black@0.5:boxborderw=5:x=(w-text_w)/2:y=1300" -preset ultrafast -tune fastdecode -c:v libx264 -pix_fmt yuv420p -b:v 9500k -maxrate 9500k -bufsize 9500k -f flv -g 4 rtmp://a.rtmp.youtube.com/live2/9qwd-1s7c-2dpk-dc73-a5ys

I came up with this after a lot of trial and error and while i’m not 100% on exactly what all this does here is what I do know:

• -f lavfi -i anullsrc: sends a null audio stream to YouTube as they require audio
• -rtsp_transport udp: use RTSP as the transport method via UDP, I tried TCP for a while and that was not reliable
• rtsp://<username>:<password>@<IP>:554//h264Preview_01_main: how you pull the HD RTSP video stream from the reolink 811a
• -vf “drawtext=fontfile=/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf: draw text using a fixed width font that ships with Ubuntu 22.04
• :textfile=/surf/data.txt:reload=1: use the contents of a text file as the subtitles and update the subtitles if the contents of the file changes
• :fontcolor=white:fontsize=44:box=1:boxcolor=black@0.5:boxborderw=5:x=(w-text_w)/2:y=1300″: white font, 44pt, within a translucent black bounding box the width of the text + 1/2 located 1300 pixels down the page
• -preset ultrafast -tune fastdecode: no idea but it works
• -pix_fmt yuv420p: set color space to what youtube accepts
• rtmp://a.rtmp.youtube.com/live2/<streamkey>: YouTube endpoint and the streamkey
• -b:v 9500k -maxrate 9500k -bufsize 9500k: set stream rate at 9500k which is what youtube wants for a 2k stream and set a buffer

Definitely a lot to unpack and it took a lot of iterating to get to this point. With this setup I still have to restart the FFMPEG script about ever 3 hours to prevent the YouTube stream from dropping about once every 24 hours. I set this up as a systemd service that auto starts on boot and setup a cronjob to restart it every 3 hours. I tried starting with 12 hours and worked my way down til I found something that was stable. Next was getting the buoy data from NOAA which was simple as they publish a text file once every 10 minutes with the latest data. I wrote a quick and dirty Python script to scrape it and a cron job to replace the data.txt file with the latest data.

Now I had a working stream to YouTube that checked all the project requirements last step was to add some redundancy. YouTube provides a backup stream that it will automatically fail over to if the primary stream goes down and during testing this happened a lot. I setup a second lightweight VM to stream a static technical difficulties image with this FFMPEG script:

ffmpeg -loop 1 -f image2 -i '/surf/psb.jpeg' -f lavfi -i anullsrc -vf realtime,scale=1280:720,format=yuv420p -r 30 -g 60 -c:v libx264 -x264-params keyint=60 -bufsize 500k -c:a aac -ar 44100 -b:a 128k fps=20 -f flv rtmp://b.rtmp.youtube.com/live2?backup=1/<streamkey>

Breaking down this FFMPEG command we have:

• -loop 1 -f image2: loop an input image indefinitely
• -i ‘/surf/psb.jpeg’: input image
• -f lavfi -i anullsrc: more null audio
• -vf realtime,scale=1280:720,format=yuv420p: no idea on the first part, image is 720p to match the primary stream and again no idea on the last part
• -r 30 -g 60: ???
• -c:v libx264 -x264-params keyint=60 -bufsize 500k: ouput to h.264 with a key frame every 60 frames, and set the buffer size I think
• -c:a aac -ar 44100 -b:a 128k: i think this sets the audio format and bitrate
• fps=20 -f flv rtmp://b.rtmp.youtube.com/live2?backup=1/<streamkey>: stream at 20fps to match the primary or YouTube complains, point the stream at the backup endpoint

This will reliably send out a static image and YouTube will happily stream it until the primary stream is back. Annoyingly YouTube will not automatically switch back to the primary stream, you have to stop the backup for 30 or so seconds before YouTube will try the primary stream again. There is good reason to have a reliable backup stream, YouTube doesn’t give you a permanent stream address immediately so every time your stream ends a new URL is generated. So if you want to embed the video in a webpage you’ll have to manually update the embed code every time the stream ends.

I plan on continuing to refine things to improve performance and reliability, the full setup is available via github.

6 month update:

After 6 months of running the stream here is where we are:

We had lots of crashes irregular intervals, from days to weeks so the text overlay has been removed. This seems to have improved things but I haven’t tested to verify if this is what fixed it.

I can now reliably run the stream at 1440P instead of 720P

We still had minor networking issues that were out of our hands that would kill the stream, so I wrote a python script to handle restarting the stream. This was surprisingly difficult to put together so to save others some time, since I couldn’t find a good example elsewhere, here it is: (note this still wont auto start the stream but it can at least be used for alerts if the stream dies)

#!/usr/bin/python3
import subprocess
import httplib2
import os
import sys
import time
from datetime import datetime, timedelta

from googleapiclient.discovery import build
from oauth2client.client import flow_from_clientsecrets
from oauth2client.file import Storage
from oauth2client.tools import argparser, run_flow

now = datetime.utcnow()
start_time = now + timedelta(minutes=2)
start_time_iso = start_time.strftime("%Y-%m-%dT%H:%M:%S.%fZ")
command_stop = "sudo systemctl stop surfcam.service"
command_start = "sudo systemctl start surfcam.service"
CLIENT_SECRETS_FILE = "/surf/client_secrets.json"

MISSING_CLIENT_SECRETS_MESSAGE = """
WARNING: Please configure OAuth 2.0
To make this sample run you will need to populate the client_secrets.json file
found at:
   %s
with information from the Developers Console
https://console.developers.google.com/
For more information about the client_secrets.json file format, please visit:
https://developers.google.com/api-client-library/python/guide/aaa_client_secrets
""" % os.path.abspath(os.path.join(os.path.dirname(__file__),
                                   CLIENT_SECRETS_FILE))

YOUTUBE_SCOPE = "https://www.googleapis.com/auth/youtube"
YOUTUBE_API_SERVICE_NAME = "youtube"
YOUTUBE_API_VERSION = "v3"

flow = flow_from_clientsecrets(CLIENT_SECRETS_FILE,
  message=MISSING_CLIENT_SECRETS_MESSAGE,
  scope=YOUTUBE_SCOPE)

storage = Storage("/surf/main.py-oauth2.json")
credentials = storage.get()

if credentials is None or credentials.invalid:
  flags = argparser.parse_args()
  credentials = run_flow(flow, storage, flags)
youtube = build(YOUTUBE_API_SERVICE_NAME, YOUTUBE_API_VERSION,
  http=credentials.authorize(httplib2.Http()))

request = youtube.liveBroadcasts().list(
        part="status",
        broadcastStatus="active",
        broadcastType="all"
    )
response = request.execute()
print(response)
if response['items']:
    if response['items'][0]['status']['lifeCycleStatus'] == "live":
        print("running")
        pass
else:
    print("dead")
    stop = subprocess.run(command_stop, shell=True, text=True, capture_output=True)
    print(stop.stdout)
    time.sleep(15)
    print("starting stream")

    stream = youtube.liveStreams().insert(
        part="snippet,cdn",
        body={
            "snippet": {
                "title": "Surf Cam",
                "description": "Surf Cam"
            },
            "cdn": {
                "resolution": "1440p",
                "frameRate": "30fps",
                "ingestionType": "rtmp"
            }
        }
    ).execute()

    time.sleep(30)
    start = subprocess.run(command_start, shell=True, text=True, capture_output=True)
    print(start.stdout)
    
    # Check if the stream is active
    stream_status = youtube.liveStreams().list(
        part="status",
        id=stream["id"]
    ).execute()['items'][0]['status']['streamStatus']

    if stream_status == "active":
        # Bind the broadcast to the stream
        bind_broadcast = youtube.liveBroadcasts().bind(
            part="id,contentDetails",
            id=broadcast["id"],
            streamId=stream["id"]
        ).execute()

        # Transition the broadcast to live
        transition_broadcast = youtube.liveBroadcasts().transition(
            broadcastStatus="live",
            id=broadcast["id"],
            part="id,contentDetails"
        ).execute()

print("done")

I recently started work on deploying Microstack to replace a lot of manual management of VMs / networks. Why Microstack? because i’m very lazy and doing a full Openstack install on more than one server does not sound like my idea of a good time.

Getting started with Microstack is very easy, just a SNAP install on each host and a couple of quick commands and you’re up and running with a multi-node Openstack environment. However doing much more than spinning up the CirrOS VM covered in the tutorial and you’ll start to run into issues. Note that this guide currently covers Ubuntu 20.04.02 LTS and Microstack version Ussuri 2020-11-23 (222) installed via the guide available here

(Edit: This has been fixed as of snap 233) The first issue I discovered was that image files over one gig could not be uploaded via the web ui, and thanks to the way SNAP works works it does a great job of hiding where the image files live (/var/snap/microstack/common/images/). The issue is caused by the default nginx config not allowing large file uploads and can be fixed by editing the following file:

sudo vi /var/snap/microstack/common/etc/nginx/snap/nginx.conf

to include this line:

client_max_body_size 4096M;

this will allow up to 4GB uploads via the web ui. Once able to upload my own images I started spinning up VMs and seeing what Microstack can do. But it wasn’t long before I ran into another issue that was a little more work to fix.

The issue I ran into was out of the box I was unable to migrate instances between hosts. Microstack’s setup sifting through logs a bit difficult, just dumping them all into syslog, but eventually I found a few issues. Firstly that the two old mac minis i was using as a lab environment had different generations of processor causing incompatibility between them. This will almost certainly be an issue in production as well unless you buy and replace all your servers at once. To resolve the issue edit the following file on all nodes:

sudo vi /var/snap/microstack/common/etc/nova/nova.conf.d/hypervisor.conf

with the following:

[DEFAULT]
compute_driver = libvirt.LibvirtDriver

[workarounds]
disable_rootwrap = True

[libvirt]
virt_type = kvm
#cpu_mode = host-passthrough
cpu_mode = custom
cpu_model = kvm64

Note: this may have changed to /var/snap/microstack/common/etc/nova/nova.conf.d/nova-snap.conf as of snap 233, doing some digging on this. Additionally is looks like libvirtd is now listening on the correct ports out of the box.

edit 6/16/2021:

for snap 223 to get HVM working instead of the default PV edit /var/snap/microstack/common/etc/nova/nova.conf.d/nova-snap.conf libvirt section to look like the following on all hosts. Also make sure that hosts are able to resolve each other by hostname, you may need to edit /ets/hosts. This may require running as root to edit so try sudo su.

[libvirt]
virt_type = kvm
cpu_mode = custom
cpu_models = kvm64

This will make libvirt ignore the hypervisor processor and allow that check to pass. With that solved it was on to the next issue. Now when attempting to migrate instances the CLI said it completed, and the GUI showed no errors, however the VM would not migrate.

Sifting through logs showed that libvirtd was unable to connect to the other node (via TCP port 16509 by default) to initiate the migration. I was unable to find much in the way of a resolution for this, or even many others with the same issue. Some digging into libvirtd showed that there was a recent change to how to place it into listen mode and select a port. Relevant bug from Redhat here and man page. It seems this change was missed in Microstack’s default config. The required socket files for libvirt to open and listen on a port were not present in /etc/systemd/system. To resolve this do the following:

copy these files:

/snap/microstack/222/usr/lib/systemd/system/libvirtd-tcp.socket
/snap/microstack/222/usr/lib/systemd/system/libvirtd.socket

to /etc/systemd/system/. Next you’ll need to make a few minor edits. In both libvirtd.socket and libvirtd-tcp.socket you’ll change two lines:

#Before=libvirtd.service
Before=snap.microstack.libvirtd.service

#Service=libvirtd.service
Service=snap.microstack.libvirtd.service

remove or comment the line SocketGroup=libvirt from libvirtd.socket. next you’ll run systemctl daemon-reload, systemctl enable libvirtd-tcp.socket and lastly systemctl enable libvirtd.socket. Once that is done on any and all microstack compute / controller nodes reboot them and you should be able to live migrate instances between hosts.

Update 4/23/2021

If you get the error:

Error: Could not find default role "_member_" in Keystone

when attempting to create a new project in Microstack there is a simple fix. Just go to the Identity > Roles page and add a role named: _member_ and then try again and the error should be resolved.

If your VMs are unable to access the external network run this command to see if ip forwarding is enabled on the hypervisor:

sudo sysctl net.ipv4.ip_forward

if you get a result of 0 then run the following command to enable forwarding:

sudo sysctl -w net.ipv4.ip_forward=1

To make this run at boot edit:

/etc/sysctl.conf

and uncomment the line:

net.ipv4.ip_forward=1

now networking will work for VMs after a reboot.

edit 4/24/2021:

(note, this needs to be reviewed and possibly updated)

Cinder doesnt work out of the box and after much fiddling it was as simple as starting the cinder scripts:

snap.microstack.cinder-backup.service
snap.microstack.cinder-scheduler.service
snap.microstack.cinder-uwsgi.service
snap.microstack.cinder-volume.service

and to run them at boot just run the following for each cinder service:

sudo systemctl enable snap.microstack.cinder-<replace>.service

and then reload the systemctl daemon:

sudo systemctl daemon-reload

Update: fixed this chunk on 6/10/2021

With cinder now running we can create a disk to store our volumes. You’ll need a blank disk mounted to /dev/<drive name> and then we’ll run a couple of commands to complete the setup:

sudo pvcreate /dev/<drive name>
sudo vgcreate cinder-volumes /dev/<drive name>

to get volumes to attach to CentOS / RedHat instances. I’ll add this when i run into the issue.

Update 5/20/2021:

Verified that getting instances directly connected to the external network is a bit of an issue, but the excellent workaround available here: https://connection.rnascimento.com/2021/03/08/openstack-single-node-microstack/ mostly worked to resolve the issue. The one issue i ran into is that the work around script doesnt add a default gateway which caused the server to be unreachable. After following the directions in the link above I modified the workaround script thusly:

#!/bin/bash
#
# Workaround to enable physical network access to MicroStack
#
# Adds the server physical ip address to br-ex. Replace the physicalcidr and gateway values to match your NIC's IP address and gateway

physicalcidr=<your ip>
gateway=<your gateway ip>
# Add IP address to br-ex
ip address add $physicalcidr dev br-ex || :
ip route del default via $gateway || :
ip route add default via $gateway dev br-ex ||:

Edit 6/16/2021:

Next you’ll need to update netplan (assuming you’re using ubuntu) to the following:

network:
  ethernets:
    <physical interface>:
      dhcp4: false
    br-ex:
        addresses:
             - <server IP/subnet>
        gateway4: <gateway ip>
        nameservers:
             addresses: [<dns server IP(s)>]
  version: 2

then run netplan apply and now everything is working as expected after rebooting the server.

Update 5/26/2021:

been doing a fair amount of work on this, will be pushing some more updates soon.

Update 6/2/2021:

collecting all the useful command aliases i’ve found and used:

sudo snap alias microstack.openstack openstack
sudo snap alias microstack.ovs-vsctl ovs-vsctl

Update 6/11/2021:

Tested the steps for direct network connectivity of VMs listed above and verified they also work in a multi-node setup.

To Do:

get HTTPS working on horizon

I recently acquired a new Sunpowere home solar system and wanted to build a dashboard to easily see my current generating and usage at a glance. I purchased a M5Paper development board which has a 960×540 eink display powered by an ESP32 to act as the dashboard.

Unfortunately Sunpower doesn’t have an accessible API so I had to scrape their monitor webpage. They dont make it easy so I ended up writing a Python / Selenium script that nagivates the monitor page and downloads a CSV Sunpower provides of the daily generation hour by hour. The Python script is run by a raspberry pi zero thatt does a few other home automation tasks. The Python script then pulls the current data and compares it against the previous hour, as well as building a graph of all hours so far that day. It then pushes images of the generated graphs to a github repo.

The M5paper dashboard is running a simple script to pull the jpeg images from the github repo and display them on the eink display.

I’ll be getting a dog soon and started looking around for a way to track it down should it ever escape. There are lots of neat commercial options but most require a monthly subscription, and those that don’t seem to be of dubious quality, so i decided to build my own.

I’ve based the build off the LilyGo-T-SIM7000G which includes an ESP32, SimCom7000G LTE cell modem / GPS receiver, and battery hold / charger for an 18650 lithium battery. The boards can be had from the usual suppliers for around $40, quite a deal for so much functionality. I used the Botletics fork of the Adafruit Fona library as a jumping off point to get the basics working. For cellular connectivity I use Google Fi as my cell provider, so I ordered a Fi data only SIM card. Getting the SIM card was the easy part, connecting the SimCom7000G module to connect to the Fi network was a bit more work. I couldn’t find any reliable documentation on getting the two to work together so it was a bit of trial and error to find that this works reliably:

fona.setNetworkSettings(F("h2g2"), F("h2g2"));

With the board on the network and getting a GPS lock remarkably quickly the next step was to push data to someplace accessible remotely. The Google Fi data only SIM card blocks SMS messages so that was out. The example library I am using references dweet.io so I thought I’d give it a try. Within just a few minutes i was able to post data to dweet and check it via the web. With the basics done and tested its time to dream add some features and optimizations.

More to come as the build continues. You can follow along at the GitHub project here.

I was recently tasked with creating a digital menu for a local company. We went through several options and settled on running dual 4k monitors with a Raspberry Pi 4 displaying a menu made in Google Sheets. Sounds pretty straight forward, and it was, until it came to automatically opening two chromium instances to separate web pages and on different displays.

• 

Auto starting one or more instances of chrome is a snap, just add the following to /etc/xdg/lxsession/LXDE-pi/autostart:

@/usr/bin/chromium-browser --new-window --kiosk --disable-restore-session-state http://exampleemailaddress.com

But to make two separate windows and have one auto open on each monitor you’ll need to add the following to the end of the command listed above in /etc/xdg/lxsession/LXDE-pi/autostart:

-user-data-dir=default0 --window-position=0,0

and for the second monitor / session you’ll add another @/usr/bin/chromium-browser --new-window --kiosk --disable-restore-session-state http://exampleemailaddress.com -user-data-dir=default0 --window-position=0,0 to /etc/xdg/lxsession/LXDE-pi/autostart changing the last two flags to: -user-data-dir=default1 --window-position=1080,0 (note that this is for a pair of vertically mounted 1080P displays, your value may vary).

Your final /etc/xdg/lxsession/LXDE-pi/autostart will have the following two lines at the bottom:

@/usr/bin/chromium-browser –new-window –kiosk –disable-restore-session-state http://exampleemailaddress.com -user-data-dir=default0 –window-position=0,0

@/usr/bin/chromium-browser –new-window –kiosk –disable-restore-session-state http://exampleemailaddress.com -user-data-dir=default0 –window-position=1080,0

Lastly you can add the unclutter application to hide the mouse cursor. To add unclutter just open an command prompt and run: sudo apt-get install unclutter then add @unlutter to your /etc/xdg/lxsession/LXDE-pi/autostart file.

Update 2/9/2020:

The menu system has been working fairly well with two exceptions. Updating the menus required using VNC to connect to the raspberry pi, exit presentation mode on both displays, and then re-enter presentation mode. Additionally after reboot the menu’s required manual intervention to be placed into presentation mode. I solved this issue using xdotool, installed by running sudo apt-get install xdotool. This allows me to script keyboard presses so after boot a script presses ctrl+shift+F5 and alt+Escape to enter presentation mode on the selected chrome / google slides instance, then switch to the other instance to repeat.

I solved the refresh issue by slapping together a quick and dirty Python / Flask app that allows users to call the script to exit presentation mode one each chrome instance which will allow slides to grab any changes made to the slides, then after a short delay reenter presentation mode. I also added an option to reboot the Pi’s via the web ui. Here is the Python:

from flask import Flask, render_template, request
import subprocess

app = Flask(__name__)


@app.route('/', methods=['GET', 'POST'])
def index():
    if request.form.get('refresh_button') == 'refresh menu':
        print('resfresh')  # do something
        cmd = ["/somepath/refresh_script.sh"]
        p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE,
                             stdin=subprocess.PIPE)
        out = p.communicate()
        print(out)
    else:
        print('reboot')  # do something else
        cmd = ["/somepath/reboot_script.sh"]
        p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                             stderr=subprocess.PIPE,
                             stdin=subprocess.PIPE)
        out1 = p.communicate()
        print(out1)
    return render_template('index.html')


if __name__ == '__main__':
    app.run()

and index.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Menu Control</title>
</head>
<body>
<form method="post" action="/">
    <p style="text-align: center;font-size:30px">Menu Controller</p>
<div style="text-align: center; border: 1px solid">
    <p>Press Refresh Menu to update the displays. Use this option after updating menus in google slides.</p>
    <p><input type = "submit" name = "refresh_button" class="btn btn-outline-primary" value = "refresh menu" style="height:75px;width:200px;background-color:green;color:black;font-size:30px"/></p>
</div>
    <div style="text-align: center; border: 1px solid">
        <p>Press Reboot to reboot the menus, this will take several minutes to complete. Only use this if the menus are not functioning correctly</p>
<p><input type = "submit" name = "restart_button" class="btn btn-outline-primary" style="background-color:red;color:black" value = "restart menu"  /></p>
    </div>
</form>
</body>
</html>

Its not pretty but its functional and much easier than doing it manually. and finally the bit of bash called to refresh the menu / enter presentation mode on boot:

#!/bin/bash
sleep 30
xdotool key ctrl+shift+F5
sleep 30
xdotool key alt+Escape
sleep 30
xdotool key ctrl+shift+F5