SSH tunnels on steroids

I have been using SSH tunnels for year, setting up a dynamic tunnel and configuring socks proxy in the browser has been my “way-to-go” tool for getting access to services at my home network, and to bypass geo blocking for varouis services in my home country.

This week i stumbled across sshuttle a tool that feels like traditional SSH tunnels on steriods. I have tried several solutions during the years, including tailscale and cloudflare, but i always ended up in going back to plain SSH because of simplicity and ease of use.

sshuttle is almost as easy as plain ssh, and it does not require anything (other than SSH and python) on the server side. The connection is initiated with pure SSH, and the serverside configuration is automatically done by the client copying the python script to the host, and setting everything up.

The application takes some parameters, and it can be seen with “sshuttle -h”
At first it seems a bit overwhelming, but have no fear, examples are available on github.

You need to let sshuttle know what subnets to route, you can use 0.0.0.0/0 for everytning, or limit it to do something similar to a “split tunnel VPN”. Just use the CIDR syntax to set the right network. I have created an alias in my ~/.profile file, and can now connect using “connect-home”

My alias looks like this:
alias connect-home='sshuttle -r username@home --dns 192.168.5.0/24 --to-ns=192.168.5.254 -no-latency-control -D'

  • -r tells sshuttle about the remote host, you can use entries from your ssh config file.
  • –dns tells to use DNS on the remote end of the tunnel
  • 192.168.5.0./24 specifies the network to route
  • –to-dns tells what IP to use for remote DNS lookups (default is the SSH server)
  • –no-latency-control speeds up the bandwidth but sacrifices latency
  • -D is for daemon mode = run in background

You can add or exclude DNS servers, subnets and specific hosts, and a lot more, check it out at https://github.com/sshuttle/sshuttle

BR Kasper

Puppetlabs apt GPG key expired

Seems puppetlabs is trying to force people to the puppet core product, making live difficult for existing puppetlabs users.
Lately the “official” apt GPG key expired at apt.puppetlabs.com, and trying to find help in the puppetlabs forum will give you the idea, that you need to use their repoe, which requires a license.

Until now, the fix for at least Ubuntu, and probably other debian based distroes is to add the DEB-GPG-KEY-FUTURE found here: https://apt.puppetlabs.com/DEB-GPG-KEY-future
Add it with the following command:

wget -O- https://apt.puppetlabs.com/DEB-GPG-KEY-future | sudo apt-key --keyring /etc/apt/trusted.gpg.d/puppet8-keyring.gpg add

You might need to clean up the old key, I overwrote it (same file name), and it worked as expectd.

Making the windows terminal great again

Once again my employeer forces me to work from an inferior platform, so what are my options to get the best possible working environment? Heres some tips to what I’ve done so far.

The windows terminal is one of the better things produced by Microsoft the last couple of years. I have already written a bit about it before, so search for windows terminal, and you will get the other articles. Even though the terminal is nice, there are some minor annoyances, when you are used to work in a bash environment.

  1. No nice “inline” editor
  2. Does not obey ctrl-d to exit the session
  3. Does not have tab completion on entries in the ssh config file

To fix these three annoyances, we need to download vim from www.vim.org, install it and the rest is done creating and configuring a powershell profile, lets dive into it.

Not only do I have to work from a windows machine (my department does primarily FOSS solutions) I also don’t have any elevated permissions on my machine. IT proffesionlas are treated just the same as office workers šŸ™

This means I cannot install any software, besides whats picked by the Windows admins. Luckily vim is not picky, and you can install it in your own home folder. So lauch the installer and change the install path accordingly.

When vim is installed, all we need to do is to create a powershell profile file. The file has to be named Microsoft.PowerShell_Profile.ps1 and has to be placed under $home\Documents\PowerShell\
When placed here it will be a user specific profile, you can also do system-wide profiles, take a look at $pshome.

The profile file I have created looks like this:

using namespace System.Management.Automation

# Set aliases

Set-Alias -name vim $home\Vim\vim90\vim.exe
Set-Alias -name vi $home\Vim\vim90\vim.exe

# Obey ctrl-d

Set-PSReadlineKeyHandler -Key ctrl+d -Function ViExit

# Enable ssh tab completion
# Credits to backerman and contributers at github for creating the code

Register-ArgumentCompleter -CommandName ssh,scp,sftp -Native -ScriptBlock {
param($wordToComplete, $commandAst, $cursorPosition)

function Get-SSHHostList($sshConfigPath) {
    Get-Content -Path $sshConfigPath `
    | Select-String -Pattern '^Host ' `
    | ForEach-Object { $_ -replace 'Host ', '' } `
    | ForEach-Object { $_ -split ' ' } `
    | Sort-Object -Unique `
    | Select-String -Pattern '^.*[*!?].*$' -NotMatch
}

function Get-SSHConfigFileList ($sshConfigFilePath) {
    $sshConfigDir = Split-Path -Path $sshConfigFilePath -Resolve -Parent

    $sshConfigFilePaths = @()
    $sshConfigFilePaths += $sshConfigFilePath

    $pathsPatterns = @()
    Get-Content -Path $sshConfigFilePath `
    | Select-String -Pattern '^Include ' `
    | ForEach-Object { $_ -replace 'Include ', '' }  `
    | ForEach-Object { $_ -replace '~', $Env:USERPROFILE } `
    | ForEach-Object { $_ -replace '\$Env:USERPROFILE', $Env:USERPROFILE } `
    | ForEach-Object { $_ -replace '\$Env:HOMEPATH', $Env:USERPROFILE } `
    | ForEach-Object { 
    $sshConfigFilePaths += $(Get-ChildItem -Path $sshConfigDir\$_ -File -ErrorAction SilentlyContinue -Force).FullName `
    | ForEach-Object { Get-SSHConfigFileList $_ } 
    }

    if (($sshConfigFilePaths.Length -eq 1) -and ($sshConfigFilePaths.item(0) -eq $sshConfigFilePath) ) {
        return $sshConfigFilePath
    }

    return $sshConfigFilePaths | Sort-Object -Unique
}

$sshPath = "$Env:USERPROFILE\.ssh"
$hosts = Get-SSHConfigFileList "$sshPath\config" `
| ForEach-Object { Get-SSHHostList $_ } `

# For now just assume it's a hostname.
$textToComplete = $wordToComplete
$generateCompletionText = {
    param($x)
    $x
}
if ($wordToComplete -match "^(?<user>[-\w/\\]+)@(?<host>[-.\w]+)$") {
    $textToComplete = $Matches["host"]
    $generateCompletionText = {
        param($hostname)
        $Matches["user"] + "@" + $hostname
    }
}

$hosts `
| Where-Object { $_ -like "${textToComplete}*" } `
| ForEach-Object { [CompletionResult]::new((&$generateCompletionText($_)), $_, [CompletionResultType]::ParameterValue, $_) }}

Exit your terminal and lauch it again, hopefully you can be a bit more productive with the new “features”

Re-Volt, even easier with rvgl launcher

The RVGL launcher project has made it even easier to install and play the good old racing game ReVolt.

To play on ubuntu, there are some prereqs you need to have in place, and not all of them are mentioned on the official RVGL launcher. To help you, I have gathered everything in a one liner, just here:

sudo apt install python3-wxgtk4.0 python3-requests python3-packaging p7zip-full xdg-utils libsdl2-2.0-0 libsdl2-image-2.0-0 libopenal1 libenet7 libunistring2 libjpeg8 libpng16-16 libtiff5 libwebp7 libvorbisfile3 libflac8 libmpg123-0

If you get an error on libwebp7, please try libwebp6

When the above is installed, download, extract and launch RVGL from here: https://re-volt.gitlab.io/rvgl-launcher/

When lauched, you may get an error about a lock file, dont worry, it’s only the first time you run it. Press “Download and Install” in th bottom, and you are ready to play. Remember to explore additional options in the rvgl launcher, it’s packed with cool features šŸ™‚

Get ready, set, race!

Gnome NetworkManager and vpnc – use nm-connection-editor

Gnome NetworkManager is awsome, supporting almost all kinds of VPN solutions, and all you need to do is install a plugin. My main issue is, that my customers often have a pre-build package (for Windows) containing a VPN client including configuration, so they actually dont know how it’s configured.

I usaually figure it out, and then its a breeze to click the network icon, select VPN and activate the connection, this is how it should be on all OS’es.

Unfortunatly there are some minor issues, one of them beeing that if you install the network-manager-vpnc module, you will have the option to create a VPNC based connection, and if you type everything right the first time, it will work. But….

You will not be able to edit the connection, it’s simply not listed when you select VPN settings (Ubuntu 22.04). using nmcli will list the connection, and you will be able to modify it, but I found another soluition, that is much easier.

nm-connection-editor

It will open a small GUI, letting you create, modify or delete all connections.

I have created a bug report here:
https://bugs.launchpad.net/ubuntu/+source/network-manager-vpnc/+bug/1951313
It has been confirmed, bot not fixed. Hopefylle it will be fixed in later releases.

Another issue, that I discovered today, is that you are not allowed to add the connection when creating it šŸ™ See some discussion here:
https://askubuntu.com/questions/1403896/cant-add-cisco-compatible-vpn-vpnc-on-network-manager-ubuntu-22-04
This issue is also not present in nm-connecttion-editor.

The connections will be working perfectly from the build in NetworkManager application in gnome.

systemd.mount replacing autofs

I have been a very satisfied user of autofs for several years. Using a laptop in multiple location without reboot, could often cause annoying timeouts with hard mounted NFS shares.

So I found autofs, which did a great job, especially the ghost option is nice, making you able to browse the filesystem when it’s not mounted, and only seeing a small delay when the autofs mounts it for you at your wish.

I have now discovered that systemd has a mount option: https://www.freedesktop.org/software/systemd/man/systemd.mount.html
It can be used as systemd units, but also directly from /etc/fstab where I prefer to have my mount options.

To replicate the autofs functionality, add something like this to your mount options:

noauto,x-systemd.automount,x-systemd.idle-timeout=60,x-systemd.mount-timeout=30,_netdev

The above options will mount when you try to access the share, and it will unmount after 1 minute of idling, and the _netdev tells the system not to mount it before the network is ready.
More or less the same functionality as autofs, but no need to install additional software.


Working with yaml files in vim

when working with yaml files, indentation is crucial. You can setup vim to make it a little easier, like this:

:set ai ts=2 sw=2 expandtab

The above will set tabstop and shiftwidth to 2 spaces, and make the tab key create spaces instead of tabs. So with this config, you will type 2 spaces, whenever you press tab, just like yaml likes it.

This can of course be put in your vimrc file.

zfs filling up the /boot device with snapshots

So, you are trying to update your system and get this:

Requesting to save current system state
ERROR couldn’t save system state: Minimum free space to take a snapshot and preserve ZFS performance is 20%.
Free space on pool “bpool” is xx%

The zfs filesystem is creating a snapshot every time you install a new kernel, this will add up, and suddenly you are not able to update anymore, and you are seeing a lot of error caused by this. If you are unlucky and reboot, it might even brake your system!

Running Ubuntu with zfs will create a pool called bpool as your /boot device. Continuing I will refer to this as bpool.


To get an overview of your zfs pools, use this command: zfspool list

kasper@AsusPro:~$ zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bpool 1.88G 1.64G 238M – – 32% 87% 1.00x ONLINE –
rpool 114G 26.3G 87.7G – – 7% 23% 1.00x ONLINE

As you see, my bpool pool is 87% filled up, I have a problem.

To get an overview of snapshots in the bpool pool, use this command: zfs list -t snapshot | grep bpool

kasper@AsusPro:~$ zfs list -t snapshot | grep bpool
bpool/BOOT/ubuntu_nijvpt@autozsys_646r7v 8K – 92.8M –
bpool/BOOT/ubuntu_nijvpt@autozsys_5z8j40 8K – 92.8M –
bpool/BOOT/ubuntu_nijvpt@autozsys_rknbhc 0B – 92.8M –
bpool/BOOT/ubuntu_nijvpt@autozsys_cd3k48 0B – 92.8M –
bpool/BOOT/ubuntu_nijvpt@autozsys_e3s23t 0B – 185M –
bpool/BOOT/ubuntu_nijvpt@autozsys_teumkd 0B – 185M –
bpool/BOOT/ubuntu_nijvpt@autozsys_iaxh6k 88K – 279M –
bpool/BOOT/ubuntu_nijvpt@autozsys_8uh3k1 88K – 372M –
bpool/BOOT/ubuntu_nijvpt@autozsys_kpxpbq 88K – 187M –
bpool/BOOT/ubuntu_nijvpt@autozsys_3lafml 0B – 280M –
bpool/BOOT/ubuntu_nijvpt@autozsys_0gvki1 0B – 280M –
bpool/BOOT/ubuntu_nijvpt@autozsys_8j52ch 82.3M – 187M –
bpool/BOOT/ubuntu_nijvpt@autozsys_2sibtv 72K – 284M –
bpool/BOOT/ubuntu_nijvpt@autozsys_3t44c8 56K – 284M –
bpool/BOOT/ubuntu_nijvpt@autozsys_u58usc 0B – 284M –
bpool/BOOT/ubuntu_nijvpt@autozsys_3okd08 0B – 284M –
bpool/BOOT/ubuntu_nijvpt@autozsys_v18vua 0B – 284M –
bpool/BOOT/ubuntu_nijvpt@autozsys_f5185b 0B – 284M –
bpool/BOOT/ubuntu_nijvpt@autozsys_12i3wc 0B – 288M –
bpool/BOOT/ubuntu_nijvpt@autozsys_o6sgfl 0B – 288M –

Seems like zfs is not doing any housecleaning by itself, we need to help!
To remove a snapshot, we need to destroy it. This is done with the destroy option, like this:

sudo zfs destroy /path/to/snapshot

To remove all snashots in the bpool, you can use the following:

for i in $(zfs list -t snapshot | grep bpool | cut -d ” ” -f 1);do sudo zfs destroy $i; done

This will destroy all snapshots and free up your bpool pool.

If you don’t want to remove all snapshots, pipe your snapshots listing into tail or head. You can add a creation column to the list like this:

zfs list -t snapshot -o name,creation -s creation

This will list your snapshots in order of creation date. Use wc -l to count them and use head to get x amount fewer.

zfs list -t snapshot -o name,creation -s creation | wc -l
20

for i in $(zfs list -t snapshot | grep bpool | cut -d ” ” -f 1 | head -15);do sudo zfs destroy $i; done

I did try to format the commands, but as usual Word Press make things so complicated to use, that i had to revert all the special formatting, instead of spending the entire day figuring out why html tags are not respected in “enlighter inline code”

Scan for new drives on Linux – the easy way

Adding storage to servers is part of every day work. this can be anything from a virtual disk on a virtual machine, to LUN’s presented from some external storage system.

When the new disk has been added, ususally nothing happens. You need to scan for the new disk/disks in order to operate them. Theres a tool called rescan-scsi-bus.sh for doing this specific task, but sometimes you don’t have that, then you need to do it manually, and it’s actually not that difficult.

You then need to echo “- – -” to the scsi system. I have found a lot of different methods to do the same thing, a lot of them telling you to do something like: echo “- – -” >> /sys/class/scsi_host/hostxx/scan where xx can be anything, and sometimes there’s a lot, so typing all the different host numbers is not optimal. Then someone suggests a for loop, like: for i in $(/sys/class/scsi_host/hosts); do echo “- – -” /sys/class/scsi_host/host$i/scan; done

I have tried them all (I think) but in my opinion, the best and easiest solution is the one below, as it is pretty easy to remember:

echo "- - -" | sudo tee /sys/class/scsi_host/host*/scan

List your newly installed disk with lsblk and partition them with fdisk