Adding Yoctopuce Sensors to Observium

I came across the Yoctopuce USB sensors recently and thought it might be fun to use one to monitor the closet I keep my network rack & servers in. I picked up one of the Yocto-Meto sensors, which combines humidity, pressure, and temperature sensors, and hooked it up to the server with a USB cable.

Observium primarily deals with SNMP, but also includes a Unix Agent which allows it to collect other system and application metrics. The agent essentially executes a folder full of shell scripts, each of which is responsible for writing some application or service metrics to stdout.

I wrote a short python script to poll the sensor and output the current values using the unix agent format, and saved it to /usr/local/lib/observium-agent/yoctopuce:

The agent is normally configured to execute using xinetd. On macOS we can use launchctl to listen on a port and execute our script by adding the following to ~/Library/LaunchAgents/org.observium.agent.plist and enabling it with launchctl load ~/Library/LaunchAgents/org.observium.agent.plist.

After loading the agent it can be tested by running telnet 127.0.0.1 36602, which will spit out the output of the script above and then disconnect.

<<<yoctopuce>>>
temperature:rack:20
humidity:rack:55
pressure:rack:1.016

It took me a bit of digging around to work out what I needed to change to get this data into Observium. I’m running the CE edition, which is a bit out of date now so things could have changed since this release. Since temperature, pressure and humidity are already built in sensor types, this seems to be all that’s needed to get the sensors discovered. I saved it into /opt/observium/includes/polling/unix-agent/yoctopuce.inc.php.

After enabling unix agent polling for the server, Observium picks it up based on the <<<yoctopuce>>> header in the output and creates an RRD for each sensor.

Restoring a Failed Hard Drive from Backblaze

I was recently unlucky enough to discover one of the external drives attached to my Plex server had failed. The drive actually appeared ok in Finder - the way I found out was actually a warning email from Backblaze explaining that the drive hadn’t been connected for 14 days. If I’d been watching more TV I might have noticed the issue earlier. I have the server backed up to Backblaze, and since we have a pretty decent fiber connection I assumed it wouldn’t be too much of an issue to just download the backup.

The first issue I ran into was the interface for restoring files from external drives that are no longer attached. When a drive is disconnected, it seems Backblaze treats the files as deleted, and since the notification of the disconnected drive took 2 weeks, the granularity available for restore was reduced considerably. This wasn’t a concern for me in this case, but it’s worth being aware of. Of greater concern for me was that I’d now wasted half of Backblaze’s 4 week retention period. That still left plenty of time to download the files, but I didn’t want to cut it too fine.

I decided to split my restore into two downloads. Once Backblaze was done zipping up my restores, the downloads were listed at 250GB and 500GB.

I installed the Backblaze Downloader, and set it to restore the smaller file using 10 threads. The result was pretty impressive, maintaining over 70Mbps on my 100Mbps connection.

Once the first file was downloaded and the zip extracted (to a new drive) I felt confident that this was going to be a piece of cake, and kicked off the second download. The download ran at 65 - 70Mbps for most of Sunday and Monday (except when it was paused so I could watch Netflix). By Monday evening the download was finished. It was only when I tried to extract the archive that things started going wrong.

$ unzip Server_4-27-21-44-11.zip -d /Volumes/Mercury/
Archive: Server_4-27-21-44-11.zip
warning [Server_4-27-21-44-11.zip]: 122632979645 extra bytes at beginning or within zipfile
(attempting to process anyway)
error [Server_4-27-21-44-11.zip]: start of central directory not found;
zipfile corrupt.
(please check that you have transferred or created the zipfile in the
appropriate BINARY mode and that you have compiled UnZip properly)

I opened a Backblaze support ticket to find out the archives had a checksum I could use to verify the download (none is displayed on the download page). After a bit of back and forth I discovered that there is no checksum generated on their end that I could use, and their suggestion was to create a new restore and try the download again. Unfortunately the second download attempt was much slower.

Clearly something was throttling my download speed. I restarted everything I could thing of, and reinstalled the downloader, but nothing seemed to make any difference. Backblaze support assured me they don’t throttle the downloads, and suggested increasing the number of threads used by the downloader.

They also revealed an interesting tip: backup retention is calculated from the last backup date, so it’s possible to keep them longer by uninstalling Backblaze from the computer.

After almost a week the download finished, and (somewhat unsurprisingly) I got the same error extracting the archive.

I tried Backblaze support again, and this time they suggested trying either The Unarchiver or BetterZip to unzip the archive. The Unarchiver was partially successful, managing to extract a handful of files from the zip after complaining that it was corrupt. BetterZip on the other hand was actually able to open the file and extract everything! It took a while, but as far as I can tell everything seems to be restored.

Overall I’m pretty happy with this. I’m paying a very low price to back up over 3TB of data, and I feel much more confident recommending Backblaze to others now that I’ve actually been through the process of successfully restoring a large amount data.

Some things I’ll keep in mind for the future:

• Backup retention with Backblaze is calculated from the last backup date. You can stop backups from being aged off by uninstalling Backblaze (or disconnecting it from the internet) until the data has been restored.
• Using the Backblaze Downloader with multiple threads can make the best use of your available bandwidth, but a fast internet connection is no use if something upstream is throttling you.
• BetterZip was the only tool I could find that would open and extract the 500GB zip file.
• Test your backups!

Setting up SNMPD on ReadyNAS Duo v1

The ReadyNas Duo v1 is fairly old now, but is still perfectly capable of providing up to 2TB of shared RAID storage over gigabit ethernet. While it does offer email alerting, I’d prefer to add it to Observium with the rest of my lab, as that’s where the rest of my monitoring is done. Turns out this is actually fairly easy to set up.

1. Power on with new drives and follow reset process.
2. Set up networking and connect to network.
3. Upgrade to latest firmware.
4. Install Toggle SSH patch
5. Install Enable Root SSH patch
6. Install APT patch
7. SSH to NAS ssh root@nas-ip using admin password
8. Install snmpd apt-get install snmpd
9. Configure snmpd.conf

rocommunity <secret>
syslocation Auckland, New Zealand
syscontact Tom Henderson

10. Restart snmpd service /etc/init.d/snmpd restart
11. You should now be able to connect to the NAS over SNMP
12. Finally we need to make sure snmpd starts if the system reboots, by running ln -s ../init.d/snmpd /etc/rc3.d/S99smnp

EdgeRouter L2TP Remote Access VPN

With the release of iOS10 and macOS Sierra, Apple has removed PPTP as a supported VPN connection. Previously I had set up a PPTP VPN for remote access to my home network, so to keep this working I needed to switch to another type of VPN. iOS supports L2TP, IKEv2 and IPSec, and of these the EdgeRouter only supports L2TP as a remote access VPN.

The setup is fairly straightforward, and very similar to the PPTP configuration. The only real difference is the need for a pre-shared secret in the IPSec settings, and firewall rules on WAN_LOCAL to allow IKE, L2TP, ESP and optionally NAT-T.

The EdgeRouter also requires that we define either an outside IP address to listen on, or a DHCP interface to listen on. My pppoe interface has a dynamic IP, but doesn’t use DHCP, so the simplest option seems to be to use 0.0.0.0 as the outside interface. I assume this means that all interfaces are listening, but in practice only the pppoe interface will have the required firewall rules to allow a connection.

I ended up with this config with seems to work perfectly.

Note that my previous configuration had already set set service dns forwarding options "listen-address=10.0.0.1", which is probably also required here.

Azure Automation with Runbooks

Running PowerShell scripts against Azure requires a fair bit of setup - we need a windows computer, Azure PowerShell components, a suitable network to run from etc - and if we need to run them regularly that computer needs to stay online. Fortunately there’s a better way.

By adding an Automation Account to a subscription we can upload scripts (‘runbooks’), and have them run on a schedule, or connect them to a webhook to allow other systems to trigger them by making an http request. There are also tools to integrate with GitHub so your runbooks are stored safely in version control.

Updating a dynamic local gateway IP

I keep a VPN up to extend my home network into Azure. Unfortunately since I don’t have a static IP on my internet connection, this goes down if the IP changes (unfortunately Azure Local Network Gateways require an IP address, not a DNS name). I have a dynamic DNS name pointing at my home IP, so updating this seems like an ideal task for a runbook. All we need to do is resolve my dynamic DNS name, and if it doesn’t match the IP address of the local gateway, update it. Here’s the script:

To get it running in Azure only a few modifications are required.

The Automation Account comes preconfigured with a reasonable set of PowerShell modules, but this script needs AzureRM.Network, which isn’t available by default. It can be added under Assets > Modules > Browse Gallery. We also need to add AzureRM.profile, which is a dependency of AzureRM.Network.

To run this manually we would need to use Login-AzureRMAccount to authenticate with Azure, but since this needs to run non-interactively we need another way. When you create the Automation Account a service principal is added to the directory. You can find it at https://manage.windowsazure.com on the Active Directory tab, in your directory under Applications > Applications my Company Owns.

Back in the Automation Account in the ARM portal, under Assets, we also have two certificates, and two connections for connecting to either the ARM portal or the classic portal using this service principal. To use the connection we just need to add this code to the top of the script:

To add the script we go to the runbooks tab in the Automation Account and create a new PowerShell runbook. Paste the script into the editor, click save, then open the test pane from the toolbar to try out the script (which takes longer to run than it would locally because in the background Azure is creating a fresh VM for it to run from each time). Assuming it runs ok we can publish it, and then create a schedule to run the script.

I now have this running hourly so if my home IP changes the VPN will come back up automatically the next time it runs.

$connectionName = "AzureRunAsConnection"
try {
    $servicePrincipalConnection = Get-AutomationConnection -Name $connectionName         

    Add-AzureRmAccount `
        -ServicePrincipal `
        -TenantId $servicePrincipalConnection.TenantId `
        -ApplicationId $servicePrincipalConnection.ApplicationId `
        -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint 
} catch {
    if (!$servicePrincipalConnection) {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    } else {
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
}

$resourceGroup    = 'RG-Network'
$localGatewayName = 'GW-Local'
$hostName = 'host.example.com'
$localGatewayIP   = [system.net.dns]::GetHostByName($hostName).AddressList.IPAddressToString

Write-Output "Checking IP address of $localGatewayName"

$localGateway = Get-AzureRmLocalNetworkGateway -Name $localGatewayName -ResourceGroupName $resourceGroup
$localAddressSpace = $localGateway.AddressSpaceText | ConvertFrom-Json

if ($localGateway.GatewayIpAddress -ne $localGatewayIP) {
    Write-Output "Gateway IP is $($localGateway.GatewayIpAddress), should be $localGatewayIP"
    $localGateway.GatewayIpAddress = $localGatewayIP
    try {
        Set-AzureRmLocalNetworkGateway -LocalNetworkGateway $localGateway -AddressPrefix @($localAddressSpace.AddressPrefixes)
    } catch {
        Write-Error "Failed to change $localGatewayName gateway IP address to $localGatewayIP"
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
} else {
    Write-Output "Gateway IP is correct: $localGatewayIP"
}