Package management

You have just created your first package, be it on Python, Node.js or Debian. Each one have several ways they can be distributed to users and each community has its preferred method for installing software. It can be a bit daunting to find the right place to upload your package, so we will talk about one the options here, which is the one we are following with the WLAN Pi project.

For the past few weeks we have been playing around with PackageCloud.io and its various features. It’s a complete solution for cloud package repository, while still being simple to use. It can handle several types of packages on a common interface and API. In this blog I’ll explain our journey until this point.

Debian repository

Our main requirement on WLAN Pi is an online repository for Debian packages. There are only a few offerings available as hosted solutions on the cloud, and this was an important requirement for us because we wanted to avoid the hassle of managing our on self-hosted repository, handling security updates, configuration management, load balancing and that comes with it. After some research, we were between two options, GemFury and PackageCloud.

Initial usage

We used GemFury for quite some time, as it had a free tier so we setup the account and started using it. WLAN Pi 2.0 was shipped with this configuration and our custom packages are installed from there. While it does work, there were some limitations that made us consider moving to another provider. Specifically, those were the main limitations:

  • GemFury didn’t have the option to setup SSL, so everything was plaintext HTTP;
  • The web interface was confusing and hard to manage the packages online;
  • No option to promote a package from a development repository to a final deploy.

The move to a better solution

That made us consider moving to another provider and then we applied for the OSS (Open Source Software) plan on PackageCloud. This plan provides a good amount of resources free of cost for Open Source projects. While this is great, it’s more important to fulfill our needs and requirements – not everything that’s free is good.

PackageCloud offers an out-of-the-box solution with SSL enabled by default for HTTPS, packages can be signed with GPG for extra security, the interface is straightforward and promoting a package from development to deployment repository takes 3 clicks. This is a lot of added value hosted solution and it solved all the problems we had before.

Package uploaded in one of the repositories

CI integration

We use GitHub Actions for our CI pipeline, which is not supported by default by PackageCloud. To facilitate integration in our pipeline, I created a new Action and published it on GitHub Action marketplace, you can find it here. This encapsulates the upload commands and can handle any type of package supported by PackageCloud, only needs an account already set up.

GitHub Action for uploading packages to PackageCloud

With accounts set up, configuring our CI environment to upload our packages took about 1 hour, including creating the GitHub Action, publishing it and testing the setup with a demo repository.

Final thoughts

PackageCloud has checked all the boxes so far. It has improved the security for all our users installing and upgrading our packages, even if they are not aware of this. Now we can also upload packages to a development environment, where we can test all the new features without fear that someone might upgrade for a broken version of the software. When we consider it stable, with 3 clicks the package is promoted to our deployment repository, where all users can then download it and enjoy all the new features and bug fixes.

Let’s work to get those updates out there now.

Remote access to a device behind NAT/Firewall

When using a debug device – like the WLAN Pi, it can be helpful to be able to access it remotely behind a firewall or NAT service. Of course we could create routing rules to allow direct access to our device, but this is not always possible.

We can facilitate this connection by providing an easy-to-setup solution. It this article, I’ll show how to setup the device to “call home” – creating a reverse tunnel to a server we own, allowing us to connect to it without any headaches.

A simple setup example conecting the WLAN Pi to a home router

Setup

1. A publicly accessible server
This can be a VPS (Virtual Private Server) or dedicated server on the cloud or on premise. It can also be a service like https://ngrok.com/, which has free and paid plans. For this article, let’s consider the server setup – we can cover ngrok later.

The server will be the meeting point, it will be running an SSH server, listening on a selected port, let’s say 2222. If we want to connect to this server we would execute:

$ ssh -p 2222 user@server-address.com

Now we must configure public key authentication to the server. This is needed for the next steps.

1.1 Public key authentication
First step it to create a public/private key pair on the WLAN Pi. This is done with ssh-keygen.

$ ssh-keygen

Then we can just press enter on all the options or configure according to your needs. After the keys are created, just copy the public key to the server with:

$ ssh-copy-id -i ~/.ssh/id_rsa.pub -p 2222 user@server-address.com

Now you should be able to connect from the WLAN Pi to the server without typing your password. Read here if you want to know more about public key authentication.

2. Device configured to “call home”
Let’s consider the device is a WLAN Pi, we need to make sure it will create a reverse SSH tunnel to our server when it gets connected to the internet.

The command to create a reverse SSH connection with port forwarding is:

$ ssh -NT -R *:12345:localhost:22 -p 2222 user@server-address.com

This will SSH into the server and configure a reverse port forwarding, listening on port 12345 on the server and forwarding any connection attempts to port 22 on the WLAN Pi. The options -NT prevent having a shell after the SSH command, since we are only interested in the port forwarding at this step – which is accomplished by the -R argument.

Now we can access the WLAN Pi by ssh’ing to our server, than ssh’ing to localhost:12345.

$ ssh -p 2222 user@server-address.com
$$ ssh -p 12345 localhost

2.1 Listen on external interface
In order to be able to ssh directly to the WLAN Pi, we need to configure sshd to allow binding of forwarded ports to interfaces other than the loopback interface. This is done with GatewayPorts configuration on sshd_config.

To change this configuration, edit /etc/ssh/sshd_config on the server and change GatewayPorts to:

GatewayPorts clientspecified

Then restart the sshd service with:

# systemctl restart ssh

This will allow us to specify which interfaces we want to bind our port forwarding to.

This is done with the * on -R *:12345:localhost:22, which makes ssh listen to port 12345 on all interfaces – allowing us to directly SSH into the WLAN Pi, without SSH’ing to our server first.

That’s good, but it still depends on someone typing this command on the WLAN Pi. In order to automate it, the command needs to execute on boot and restart in the case of any disconnections.

2.2 Connection retry
In case SSH connection drops for any reason, it has to have a mechanism to retry the connection. This is where autossh is needed. It is just a wrapper that calls SSH again in case it disconnects. The wrapped command looks like this:

$ autossh -f -M0 -o "ServerAliveInterval 20" -o "ServerAliveCountMax 3" -NT -R *:12345:localhost:22 -p 2222 user@server-address.com

There are four extra arguments there (from the normal SSH command). The first is -f it makes autossh go to background before executing ssh. Then we have –M0, it just tells autossh to retry when connection is lost. If you give a higher number to -M, it will create another connection on this port to check for connectivity. To make things simple, we won’t use that and instead just use an SSH keep alive on the tunnel we already have. This is done with the two -o arguments.

ServerAliveInterval tells how long the interval will be between the null packets sent for the keep alive. ServerAliveCountMax is how many keep alive packets can be lost before closing the connection. Therefore, in this case, if the WLAN Pi loses internet connection for more than 1 minute, the tunnel will be closed and autossh will start the connection retries.

2.2 Start SSH at boot
SSH should be started at boot or, more specifically, when we connect to the network. There are several ways to make this happen. On Debian (used on the WLAN Pi), the boot is controlled by systemd, other systems might manage boot with init.

systemd is a robust and easy way to start services during boot and we can even choose the order which will be executed. Although it’s easy to configure, it’s implications are not so easy to understand – adding a service to boot at the wrong time might delay boot by several seconds (or even minutes). I will cover systemd at a later post, for now let’s just use a simple solution that will get the job done.

An easy way to execute something at boot is to add it to /etc/rc.local. This is a legacy script from init that was ported to work on systemd as well. It will execute whatever is placed inside once the boot finishes, that can be considered when the login screen would appear.

For our purpose let’s just add the autossh command we had to this file:

autossh -f -M0 -o "ServerAliveInterval 20" -o "ServerAliveCountMax 3" -NT -R *:12345:localhost:22 -p 2222 user@server-address.com

You will notice that rc.local finishes with exit 0, this should remain there and we need to add our command before it.

Done

Now your are done. As soon as you boot your WLAN Pi it will start trying to create the reverse tunnel to your server. If it doesn’t have internet access, it will keep retrying. When it gets connected to the internet, the tunnel will open and you will be able to connect to it using:

$ ssh -p 12345 wlanpiuser@server-address.com

I recommend using public key authentication on all ssh connections, as you save the trouble of always typing the password, as well as being able to disable password access to ssh, which makes the system more secure.

Conclusion

This is my first blog post, so please let me know what I could improve and if it helped you. I’ll work on writing about ngrok and systemd on the next posts.

Also putting together a script to help configuring this on the WLAN Pi, will update here once it’s done.