UP | HOME

Small snippets worth sharing: A collection of tricks to solve common problems

These are tools and tricks I use regularly which are too small to give them full articles but too useful not to describe them.


PDF (drucken)

Calculate the CSP script-src hash for an inline script-tag

echo -n 'var inline = 1;' > /tmp/foobar
sha256sum /tmp/foobar | cut -d " " -f 1 | xxd -r -p | base64 

For background, see mdn: Content-Security-Policy/script-src.

Build Freenet with Java 8 on Guix

guix environment --ad-hoc icedtea:jdk -- \
  bash -c 'ssh kav freenet/run.sh stop; \
    ./gradlew --no-daemon clean build -x test && \
    for i in freenet.jar freenet.jar.new; do \
      scp build/libs/freenet.jar kav:freenet/$i; \
    done; \
    ssh kav freenet/run.sh start'

jump between versions within a filesystem tree

# start ~/path/to/source/branch-7.9/src/path/to/folder
cd $(echo $PWD | sed s/7.9/master/)
# now at ~/path/to/source/branch-master/src/path/to/folder

I use this regularly at work to avoid deep navigation. Typically via C-r 7.9/master — and for the reverse C-r master/7.9.

Optimize bash defaults: increase history size, update history instantly, share history

This is essential to re-use commands without typing them.

Add the following to ~/.bashrc:

# better bash history handling
# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
export HISTSIZE=100000
export HISTFILESIZE=1000000
# append to the history file, don't overwrite it
shopt -s histappend
# don't put duplicate lines or lines starting with space in the history.
# See bash(1) for more options
export HISTCONTROL=ignoredups:erasedups
# update the history with every command, not only at exit
export PROMPT_COMMAND="history -a;$PROMPT_COMMAND"

# check the window size after each command and, if necessary,
# update the values of LINES and COLUMNS.
shopt -s checkwinsize

# If set, the pattern "**" used in a pathname expansion context will
# match all files and zero or more directories and subdirectories.
shopt -s globstar

Activate readline and colors in the Guile REPL

To be enjoyable to use interactively, Guile requires readline and colors.

Just put the following in ~/.guile:

(cond ((false-if-exception (resolve-interface '(ice-9 readline)))
       =>
       (lambda (module)
         ;; Enable completion and input history at the REPL.
         ((module-ref module 'activate-readline))))
      (else
       (display "Consider installing the 'guile-readline' package for
convenient interactive line editing and input history.\n\n")))

      (unless (getenv "INSIDE_EMACS")
        (cond ((false-if-exception (resolve-interface '(ice-9 colorized)))
               =>
               (lambda (module)
                 ;; Enable completion and input history at the REPL.
                 ((module-ref module 'activate-colorized))))
              (else
               (display "Consider installing the 'guile-colorized' package
for a colorful Guile experience.\n\n"))))

optimize scanned image for homework in Gnome

With this trick you get a right-click menu in Nautilus (Gnome file manager) that optimizes a scanned file for sending as homework assignment result.

Save the following as ~/.local/share/nautilus/scripts/optimize-scan-for-homework and run chmod + ~/.local/share/nautilus/scripts/optimize-scan-for-homework.

#!/run/current-system/profile/bin/bash

# This script makes a scanned image suitable (=small enough) for
# sending as homework assignment by replacing almost-white pixes from
# scans by white pixels and then running pngquant

# thanks for the urldecoder goes to to https://stackoverflow.com/a/37840948
# license: cc by-sa (as this is stackoverflow)
urldecode() { : "${*//+/ }"; echo -e "${_//%/\\x}"; }
base="$(echo $(urldecode "$NAUTILUS_SCRIPT_CURRENT_URI") | cut -d / -f3-)"
while [ ! -z "$1" -a ! -e "$base/$1" ]; do shift; done
filename="$base/$1"

if [ -f "$filename" ] && [[ x"$(file -b --mime-type "$filename" | sed s,/.*,,)" = x'image' ]]; then
    COLORFILE="${filename%.*}-colors.${filename#*.}"
    if [ -f "${COLORFILE}" ]; then
       zenity --error --width 400 \
              --text "Temporary file ${COLORFILE} already exists, not overwriting" \
              --title "temp file exists: ${COLORFILE}";
    else
        convert "${filename}" -contrast-stretch 1%x80% "${COLORFILE}"
        pngquant --skip-if-larger --strip --speed 1 "${COLORFILE}" || zenity --error \
                 --width 400 \
                 --text "running pngquant on ${COLORFILE} failed";
    fi
else
    zenity --error --width 400 \
                   --text "image optimization needs an image file, 
but \n${filename}\n is not an image file.\n
Its mime type is $(file -b --mime-type "$filename")" \
                   --title "not an image: ${filename}"
fi

Evaluate website logs with goaccess

uses goaccess.

cp logs/access_log_2021*.gz /tmp/
cd /tmp/
gunzip access_log_2021-0*
cat access_log* > aggregated_log.log
goaccess --all-static-files --ignore-crawlers -f aggregated_log.log

Now hit the number of the part you’re interested in, jump to next with tab.

Sort with s, expand with enter or space or o, scroll down with page down or CTRL-f and up with page up or CTRL-b.

Hit ? for more info.

Example, unique visitors per day, ordered by number of visitors:

goaccess-draketo-ua-example.png

To create a csv file:

goaccess --date-format='%d/%b/%Y' \
         --time-format='%H:%M:%S' \
         --log-format='%h %^[%d:%t %^] "%r" %s %b "%R" "%u"' \
         --max-items=99999999 --all-static-files --ignore-crawlers \
         -f /tmp/aggregated_log.log \
         -o /tmp/agg.csv

url-encode / url-decode

To encode unicode chars for a URI, just just save this as ~/.local/bin/url-encode and make it executable

#!/usr/bin/env bash
exec -a "$0" emacs --batch --eval "(progn (require 'package) (package-initialize) 
  (add-to-list 'package-archives '(\"melpa\" . \"https://melpa.org/packages/\")) 
  (package-refresh-contents)(package-install 'urlenc) (require 'urlenc))" \
  --eval "(princ (urlenc:encode-string \""$@"\" urlenc:default-coding-system))" \
  --eval '(princ "\n")'

and this as ~/.local/bin/url-decode and make it executable

#!/usr/bin/env bash
exec -a "$0" emacs --batch --eval "(progn (require 'package) (package-initialize) 
  (add-to-list 'package-archives '(\"melpa\" . \"https://melpa.org/packages/\")) 
  (package-refresh-contents)(package-install 'urlenc) (require 'urlenc))" \
  --eval "(princ (urlenc:encode-string \""$@"\" urlenc:default-coding-system))" \
  --eval '(princ "\n")'

Usage:

url-decode $(url-encode '1,2+3!4')
# the single quotes prevent the '!' from mangling your command

Prepare this in one command:

echo '#!/usr/bin/env bash'"
exec -a \"\$0\" emacs --batch --eval \"(progn (require 'package) (package-initialize)
  (add-to-list 'package-archives '(\\\"melpa\\\" . \\\"https://melpa.org/packages/\\\"))
  (package-refresh-contents)(package-install 'urlenc) (require 'urlenc))\" \
  --eval \"(princ (urlenc:encode-string \\\"\"\$@\"\\\" urlenc:default-coding-system))\" \
  --eval '(princ \"\\n\")'
" > ~/.local/bin/url-encode
echo '#!/usr/bin/env bash'"
exec -a \"\$0\" emacs --batch --eval \"(progn (require 'package) (package-initialize)
  (add-to-list 'package-archives '(\\\"melpa\\\" . \\\"https://melpa.org/packages/\\\"))
  (package-refresh-contents)(package-install 'urlenc) (require 'urlenc))\" \
  --eval \"(princ (urlenc:decode-string \\\"\"\$@\"\\\" urlenc:default-coding-system))\" \
  --eval '(princ \"\\n\")'
" > ~/.local/bin/url-decode
chmod +x ~/.local/bin/url-{de,en}code

This is not the fastest command, because it spins up a full Emacs, but definitely faster than opening a website — and privacy preserving.

dump all quassel irc channel logs

This uses the quassel dumplog script to extract all channel logs from a quassel db into plaintext logs.

DB="path/to/quassel-storage.sqlite"
OUT="path/to/target-directory"
USER="user"
for i in $(python2 dumplog.py -u "${USER}" -d "${DB}"); do
    for j in $(python2 dumplog.py -u "${USER}" -d "${DB}" -n "$i" | head -n -2 | tail -n +3); do
        python2 dumplog.py -u "${USER}" -d "${DB}" -n "$i" -c "$j" -o "${OUT}"--"$i"--"$j".log
    done
done

To work with logs from FLIP, I had to patch in some rudimentary error recovery:

diff -u dumplog-0.0.1/quasseltool.py dumplog-0.0.1/quasseltool.py
--- dumplog-0.0.1/quasseltool.py
+++ dumplog-0.0.1/quasseltool.py
@@ -252,7 +252,10 @@
             self.mynick = sender[0]
             return "\nSession Start: %s\nSession Ident: %s\n"%(self._now2(row[2]), self.buffer)
         else:
-            return "%s *** %s (%s) has joined %s\n"%(self._now(row[2]), sender[0], sender[1], row[4])
+            try:
+                return "%s *** %s (%s) has joined %s\n"%(self._now(row[2]), sender[0], sender[1], row[4])
+            except IndexError:
+                return (str(sender) + str(row)).replace("\n", "----")

     def part(self, row):
         sender = row[3].split("!")
@@ -260,7 +263,10 @@
             self.mynick = ""
             return "Session Close: %s\n"%self._now2(row[2])
         else:
-            return "%s *** %s (%s) has left %s\n"%(self._now(row[2]), sender[0], sender[1], self.buffer)
+            try:
+                return "%s *** %s (%s) has left %s\n"%(self._now(row[2]), sender[0], sender[1], self.buffer)
+            except IndexError:
+                return (str(sender) + str(row)).replace("\n", "----")

     def quit(self, row):
         sender = row[3].split("!")

Diff finished.  Thu Aug 12 21:51:53 2021

Also see dropping old logs from quassel, but with adjustment: You get the time with

sqlite3 quassel-storage.sqlite
select strftime('%s','now','-90 day');

And get something like:

1621148264

Now you add three zeros and check what you’d get:

select * from backlog where time < 1621148264000;

As a sanity test, check whether there are messages from the future. Since you stopped the quasselcore before these tests (you did, right?), there should be none:

select strftime('%s','now','-1 second'); -- right now it is 1628924157
select * from backlog where time > 1628924157000;

Now you can create a backup and then drop everything older than 90 days:

cp quassel-storage.sqlite quassel-storage.sqlite.bak
-- First doublecheck again
select COUNT(*) from backlog where time > 1621148264000 ; -- newer messages: 748458
select COUNT(*) from backlog where time < 1621148264000 ; -- old messages: 25471749
-- now drop the old messages
delete from backlog where time < 1621148264000 ; -- careful: there is no going back.
-- check whether it worked
select COUNT(*) from backlog; -- 748458
-- actually free the diskspace, this saved 2.7GiB of diskspace for me
VACUUM;

this trick wasn’t really a shell-trick, but rather a commandline trick. I’d rather have a not so narrow view here on the tricks here where it makes them more useful :-) .

loop over files with spaces by time, oldest first

SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
for i in $(ls --sort=time -r)
do
  echo "$i"
done
IFS=$SAVEIFS

This adjusts IFS to avoid splitting by spaces. For more IFS tricks, see BASH Shell: For Loop File Names With Spaces.

[2021-09-17 Fr]

Update value in sqlite

This is here because I had to look it up to update my config for FLIP and FMS.

# accesss the database
sqlite3 flip.db3
-- get help
.help
-- see the tables in the database
.tables
-- activate table headers to see the names of the options
.headers on
-- see the values in tblOption (the columns are given in the headers)
select * from tblOption;
-- set the OptionValue for Option FCPPort
update tblOption set OptionValue = 9481 where Option = "FCPPort";
-- exit
.quit

[2021-10-13 Mi]

use xargs with ls: linebreak as separator

I often want to call some command for every file in a folder. Normal xargs fails for files with spaces in them.

xargs -d "\n" to the rescue: It uses the newline as argument separator instead of the space.

So executing a command on every file in a directory therefore only requires

ls | xargs -d "\n" command

Typical usage: Shuffle-play the 30 newest videos in a folder with mpv:

ls --sort=time -c1 ~/path/to/videos/* | head -n 30 | xargs -d "\n" mpv --shuffle

[2021-11-11 Do]

arrange multiple images on a page, keeping space around the images

pdfjam --trim="-1cm -1cm -1cm -1cm" true --nup 2x3 …images

(optionally use --landscape for landscape view)

[2021-11-24 Mi]

multiprocessing made easy with xargs

A very common usecase: I want to run a command with many different arguments, but I want to run at most 32 processes at a time to avoid insta-OOMing my machine:

for i in {1..64}; do echo $i; done | xargs -P 32 -I % echo %

-P 32 allows for up to 32 simultaneous processes.

-I % sets the input placeholder to %, so I can use echo % similar to echo $i in a direct loop.

[2021-12-03 Fr]

Stop and continue a process by PID (i.e. in another terminal)

So you started Emacs from the terminal, then activated exwm as window manager, switched back to the terminal and stopped Emacs with CTRL-z so your X-session is stopped and you cannot enter fg or bg? How to get it active again?

You can continue an arbitrary PID with kill — or by name with pkill:

pkill -CONT emacs

as mirror operation you can use the signal stop to pause any process by PID:

# hard stop
kill -STOP PID
# polite stop (keyboard stop, may be ignored)
kill -TSTP PID

sidenote: it did not actually restart the exwm for me, but stop and cont by PID should still come in handy.

[2022-01-05 Mi]

Transparently run script as root via sudo — if needed

If you have a script that needs to run as root and you want to ensure that users know that sudo will be run, you can add a header to conditionally run it with sudo:

if [ "$EUID" -ne 0 ]; then
  echo This script needs root priviledges. 1>&2
  echo Executing sudo --login $(realpath "$0") in 3 seconds 1>&2
  for i in {1..3}; do
      echo -n .
      sleep 1
  done
  echo " " now executing sudo and sudo --login $(realpath "$0")
  sudo echo || exit # ensure nicer exit if the user aborts
  exec sudo --login $(realpath "$0")
fi
# run your privileged code here.

[2022-03-27 So]

select specific characters from a string (splicing in bash)

You can use parameter expansion in bash to select a string by index and length (zero-indexed):

V=abcxyz
echo ${V:3:2}
xy

Use ${#var}-N to index from the end.

V=abcxyz
echo ${V:${#V}-5:3}

Parameter expansion can do a LOT more. Do read up on it in the manual.

[2022-04-23 Sa]

Where does this command come from? (in Guix)

When I want to know which package brought a given command, I use ls -l $(which prog), because Guix just symlinks the binaries from packages in /gnu/store/<unreadable-hash>-package-name-and-version/.../bin/program-name into ~/guix-profile/bin/

That has proven to be pretty useful more often than I expected.

[2022-06-10 Fr]

cron-job at randomly selected time: guard with random

If you want to run repeated jobs in a way that does not expose when your computer is active, you need some waiting. The usual tool for repeated jobs is cron, but waiting (sleep seconds) isn’t a good idea there, because it can block other jobs.

A simpler method is to select multiple possible times of the day and then guard your cron-job with a test for random. Example:

# run at a random afternoon hour every day
0 14,15,16,17,18 * * * test 0 -eq $(($RANDOM % 5)) && date >> /tmp/randomly-selected-execution-times

This runs on average once per day, but it is not guaranteed to run exactly once per day to prevent providing information about the days your computer isn’t running. It could skip one day and run three times during the next day; it could even run 5 times on one day, but on average it should run once per day.

If instead you want to run program exactly once per day but at a random time, you can use a cron-job that delegates the exact timing to at.

[2022-06-19 So]

sanitize filenames

Filenames with spaces and non-alphanumeric letters create problems in many contexts. This replaces all the dangerous letters by underscores:

for i in *" "*; do mv "$i" /tmp/"$(echo $i | sed 's/[^\.A-Za-z0-9/-]/_/g')"; done

[2022-07-01 Fr]

ArneBab 2020-05-18 Mo 00:00 - Impressum - GPLv3 or later (code), cc by-sa (rest)