Entries tagged 'cat:Software' (Page 3)

backup.sh

Entry created on 2021-12-05 Authors: steeph (376) Categories: Backups (4) Bash (31) Code (31) Computer (78) Linux (36) Projects (41) Scripts (28) Software (53) Languages used: en (257) Topics: Projects → Code (26) edit
This entry is an update of the entry 'Backup Shell Script'.

I've updated/improved my backup script again.

Download


#!/bin/bash
#2021-11-03

budir="/mnt/backup"
num=8
list="/root/backup-list"
hostname=$(</etc/hostname)
srcdir="$1"
#NAME=${SRCDIR//\//--}
name="$2"

function usage () {
  echo "This script requires either one or two arguments."
  echo "Usage:"
  echo "$0 JOBLIST"
  echo "$0 SOURCE_DIRECTORY NAME"
  echo
  echo "JOBLIST File that contains one backup job per line in the format SOURCE_DIRECTORY NAME"
  echo "SOURCE_DIRECTORY Directory that should be backed up. Please, no trailing slash."
  echo "NAME A string that is used to name the backup in the destination."
  exit 1
}

function single() {
  if grep -qs "$budir " /proc/mounts
  then
    fullname="$budir/$hostname/$name"
    printf "\n"
    printf "Attempting backup of source directory '%s' to '%s'.\n" "$srcdir" "$fullname"
    printf "Number of differential backups to keep: %s\n" "$num"
    printf "Removing oldest backup... "
#    rm -rf "$fullname.$num"
    rsync --archive --delete "$budir/empty/" "$fullname.$num/"		# This is quicker than rm.
    printf "Done.\n"
    for ((i=$num; i>=2; i--)); do
      printf "Renaming '%s' to '%s' ... " "$fullname.$((i-1))" "$fullname.$i"
      mv "$fullname.$((i-1))" "$fullname.$i"
      printf "Done.\n"
    done
    printf "Duplicating last backup ('%s' to '%s')... " "$fullname.0" "$fullname.1"
    [[ -d $fullname.1 ]] && exit 1							# This directory should not exist at that point.
    cp -al "$fullname.0" "$fullname.1"
#    rsync --archive --acls --xattrs --hard-links "$fullname.0/" "$fullname.1/"
    printf "Done.\n"
    now=$(date)
    printf "\n\n\n" >> "$fullname.log"
    printf "STARTING INCREMENTAL BACKUP AT %s\n" "$now" >> "$fullname.log"
    printf "Starting new incremental backy uppy at '%s.0'..." "$fullname"
    if rsync --exclude ".cache" --archive --no-links --delete "$srcdir/" "$fullname.0/" 2>&1 | tee -a "$fullname.log"
    then
      printf " Done.\n"
      printf "Done.\n" >> "$fullname.log"
    else
      printf " Failed.\n"
      printf "Failed.\n" >> "$fullname.log"
    fi
    now=$(date)
    printf "\nIf this line is here the script finished (with or without errors) at %s\n" "$now" >> "$fullname.log"
  else
    printf "'%s' is not mounted. Aborting." "$budir"
    exit 1
  fi
}

function fromlist() {
  while read job; do
    if [ -n "$job" ]; then
      name="${job#* }"
      srcdir="${job% *}"
      single
    fi
  done < "$list"
}

mkdir "$budir/empty"									# Empty directory for a quicker method to delete a large directory.

case $# in
0)
  mount "$budir"
  printf "Reading backup jobs from list. Defaulting to %s.\n" "$list"
  fromlist
  ;;
1)
  mount "$budir"
  list="$1"
  printf "Reading backup jobs from %s." "$list"
  fromlist
  ;;
2)
  mount "$budir"
  single
  ;;
3)
#  I can't decide whether to make the third argument num or budir. I don't need it anyway.
#  num=$3
#  budir=$3
#  mount $budir
#  single
  usage
  ;;
*)
  usage
  ;;
esac

exit 0
Comment via email

SBWG - The Pathshortener And Other Recent Changes

Entry created on 2021-12-05 Authors: steeph (376) Categories: Bash (31) Code (31) Computer (78) Linux (36) Projects (41) SBWG (18) Scripts (28) Software (53) Languages used: en (257) Topics: Projects → Code → Bash Scripts → SBWG (16) edit

I made it my goal to harden SBWG before I start to implement new features. Before I call the next version of this project 1.0 I want to make sure that unexpected input from the command line or from source files, absurd numbers of absurdly long tags and content items, stupidly weird filenames or random binary data as tag values as well as purposfully created traps in the various places where input is processed are handled well, meaning that nothing fails unless there is no sensible way around it, and if something fails, that nothing breaks. Data should be filtered carefully, errors should be handled well and whereever possible data should be made processible if it was supplied in an unprocessible form to reduce the chances of errors. On top of that I wanted to make sure that the script did its job in a reasonable amount of time considering the circumstances. I mean, it will never be very fast. Bash is just not the right language for that. But there certainly were some repetitive tasks that could be improved. Fot the latter I created a simple caching functionality that will probably be extended in the future. I managed to reduce the (calculated/estimated) generation time of my biggest test web site from almost 300 years to a few days. Actual web sites will of course not take that long to generate, even on a slow machine. A huge web site will maybe take up to one day to generate completely, even without the new options that keep the script from re-generating existing unchanged parts of the web site. But before sombody will try to create such a big web site with SBWG I will probably have improved speed further. And even then it's a worst-case time.

As part of the aforementioned goals I have started working on last new feature before version 1.0. I call it the pathreducer. Since many of the files created by SBWG are named after the tags they represent, they can become quite long and contain almost any printible character, including multi-byte unicode characters or characters of character sets I haven't even heard of. I definitely don't want to restrict more than I already have what characters and how many of them tag values can contain. Especially filesystems used by operating systems from Microsoft are relatively restrictive in maximum allowed directory, path and filename length and allowed characters. By default the pathreducer is not used. But if enabled via command line option or in a web site's settings file, it will filter directory and filenames and shorten them to a user-defined maximum length. If the pathreducer decides to change a path elements it also adds a 6-character hash value to make shortened or otherwise reduced path elements as good as unique.

That works well for now and even can create 8.3 or 6.3 filenames for old DOS filesystems. But the result is not very nice because it isn't aware of what filesystem it is going to write a file to. To be save it removes more characters than it would have to for ext and NTFS filesystems. In the future I may extend the pathreducer to detect the filesystem at least of the root of the output directory automatically and decide how exactly path elements should be reduced according to the actual limitations of the present filesystem. Than it may even be enabled by default, even though it can increase the generation time quite a bit.

There are still some tests that I want to do and I will probably find some more things that I want to fix before version 1.0. But I see light.

Comment via email

Generating Bitmap Files With Bash

Entry created on 2021-10-17 Authors: steeph (376) Categories: Bash (31) Bitmaps (1) Code (31) Computer (78) Linux (36) Scripts (28) Software (53) Languages used: en (257) Topics: Projects → Code → Bash Scripts (23) edit

I needed a large amount of image files to try something. I wanted them to be different images. But what's in them didn't matter. So I looked at a BMP file to see how I could create one byte for byte automatically. Bitmap is just the first uncompressed format that I thought of. This is what I came up with:


rbmp() {
  echo -n -e '\x42\x4D\x2A\x02\x00\x00\x00\x00\x00\x00\x7A\x00\x00\x00\x6C\x00\x00\x00\x10\x00\x00\x00\x09\x00\x00\x00\x01\x00\x18\x00\x00\x00\x00\x00\xB0\x01\x00\x00\x23\x2E\x00\x00\x23\x2E\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x42\x47\x52\x73\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x'
  c=$(</dev/urandom tr -dc '0123456789ABCDEF' | head -c864 | sed 's/.\{2\}/&\\x/g')
  echo -n -e "$c"'00'
}

Or, if you would like to do run a command for each pixel/color before it is generated, you can do it like this:


# Create a 24 bit bitmap file of 16x9 randomly colored pixels
randombmp() {
  echo -n -e '\x42\x4D\x2A\x02\x00\x00\x00\x00\x00\x00\x7A\x00\x00\x00\x6C\x00\x00\x00\x10\x00\x00\x00\x09\x00\x00\x00\x01\x00\x18\x00\x00\x00\x00\x00\xB0\x01\x00\x00\x23\x2E\x00\x00\x23\x2E\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x42\x47\x52\x73\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
  i=1
  while (( i < 433 )); do
    c=$(</dev/urandom tr -dc '0123456789ABCDEF' | head -c2)
    echo -n -e '\x'"$c"
    i=$((i+1))
  done
}

I'm sure I did something wrong. I just copied the header of some BMP file with the right format that I don't even know what it was created with. I didn't actually look up how the header of a BMP file is composed. But it worked.

Comment via email

The Three Bad Reasons Why I Don't Use Git

Entry created on 2021-09-01 Authors: steeph (376) Categories: Computer (78) Git (1) Personal (13) Software (53) Version Control (1) Languages used: en (257) edit

1. Github or other public Git repositories: Wouldn't be complete

Haven't published everything always, don't want to publish everything, would want to include everything but couldn't. So using public Git repositories would always be accompanied by a feeling of imperfection.

2. I've never used Git for anything really.

Apart from cloning and occasionally updating others' repositories I've never used them. I'm not used to using Git and I don't struggle with not using it. So starting to do so now would require me to hurdle quite some hurdle. I never used it. Why would I start now?

3. It's too late to start now.

I've noticed that the point where it would have been a good idea and would have made a lot of sense to get accustomed to using Git has long passed. So by starting now or in the future I would admit that I didn't take the hurdle when it would have been the right thing to do - when the best time to do so was near the present. I would admit to doing some things not the right way in the past if I would start to do them right from now on. It's easier to pretend that the way it always has been is the right way - the way I'm used to doing things.

That all makes no logical sense. It would be an improvement to start using Git for some things, be it coding projects, any category of texts that I have on my computers, any collection of files, ... The costs of these improvements would be hard disk space, which isn't all that rare for me nowadays and getting used to using Git, which isn't complicated.

So why don't I even try to use it in some cases? Well, I've just honestly told you my three reasons.

Comment via email

SBWG 0.9.6

Entry created on 2021-08-29 Authors: steeph (376) Categories: Bash (31) Code (31) Computer (78) Linux (36) Projects (41) SBWG (18) Scripts (28) Software (53) Languages used: en (257) Topics: Projects → Code → Bash Scripts → SBWG (16) edit

As foreseen I've made slow progress in development of SBWG, the script that generates this web site, because I want to test and improve it with the current feature set before I start to implement new features. The former is necessary. The latter is more fun. But today I've reached a point where I can say that the only thing left to do before I call it v1.0.0 is testing edge cases and things that I didn't think of testing before, as well as fixing potential bugs discovered from this testing.

So, version 0.9.6 is out, everything is working, The README file, other documentation included in the package, the example website, the included style sets and partly even the code quality has been deemed satisfactory, and I hope I'll find the time to test all sorts of weird stuff and discover and fix some bugs next month, at which point version 1.0.0 will be published and I'll finally be able to allow myself to start working on new features, some of which are awaited by both regular users of the script.

Comment via email

Backmatch - A Simple Dual-N-Back-Inspired Performance Task Trainer For Bash

Entry created on 2021-07-31 Authors: steeph (376) Categories: Bash (31) Brain Training (1) Code (31) Computer (78) Scripts (28) Software (53) Languages used: en (257) Topics: Projects → Code → Bash Scripts (23) edit

Here is a script that I wrote as a short side project because I wanted my own n-back trainer. I'm aware that the practical memory improments of n-back training, even when using a proper audiovisual dual n-back trainer, is not as great as it's often said to be. I just wanted to try it and see whether I like the training.

Usage

Execute the script and pass it a number that resembles the difficulty level. For example bash backmatch 3 starts the script with a 3-back task. On your keyboard press the key of the letter that was displayed n letters ago (in this example 3 letters ago). When you press a key the next letter is presented immedietly. If you don't press a key for 3 seconds (the time value can be changed by changing the variable sec) the next letter is presented and you don't get a point for this letter. When you exit the script by pressing ctrl+c your keypresses get compared to what was presented and you get your score.


#!/bin/bash

if [[ ${#} -ge 1 ]] && [[ ${1} =~ ^[0-9]*$ ]]
then
  n=${1}
else
  echo "'${@}' is not a number, is it?"
  exit 1
fi

echo "Ctrl+C pressing is for quitters."
npo=$((n+1))
sec=3

end() {
  echo -en "           \n"
  if [[ ${#str} -ge ${npo} ]]							# If enough characters had been generated
  then
    for i in $(seq $n); do echo -n "-"; done					# offset by $n dashes
    echo ${str}
    echo ${you}
    got=0
    for i in $(seq ${#str})
    do
      [[ ${str:$i:1} == ${you:$((i+n)):1} ]] && got=$((got+1))
    done
    echo -e "\n${got} out of $((${#str}-npo)) correct"
  else
    echo "Not enough data to judge you."
  fi
}

trap end EXIT

while true
do
  str+=$(cat /dev/urandom | tr -dc 'A-Z' | head -c 1)				# Get a random new letter.
  echo -en "\r       \r${str: -1}"						# Print the last character in the string (the new letter).
    read -n 1 -t ${sec} key							# Get a single character input, timeout $sec seconds.
    [[ -n ${key} ]] && you+=${key^^} || you+="-"
    [[ ${key^^} == ${str: -$npo:1} ]] \
      && echo -en "\r       good" \
      || echo -en "\r        bad"						# Check if uppercase input char is the same as the nth char from the back.
done

end

exit 1
Comment via email

SBWG 0.8.10

Entry created on 2021-05-24 Authors: steeph (376) Categories: Bash (31) Code (31) Computer (78) Linux (36) Projects (41) SBWG (18) Scripts (28) Software (53) Languages used: en (257) Topics: Projects → Code → Bash Scripts → SBWG (16) edit

So, I'm still making slow process with SBWG. I had more fun with it when I was out and implementing new features. But I find it important to finish version 1.0.0 with the currently defined set of festures and goals, which include finishing documentation, testing and code hardening, which are less fun for me.

I've always treated the third level of the version number (x.x.thisone) as a means to declare a new version done when I feel like having achived something. So today I declare version 0.8.10 as done. There really isn't much left to do to meet my milestones for v0.9.0. And from there on it will only be testing and possibly a little bit of code improvements to get to my set goals for v1.0.0.

I'm looking forward to this not only because I'll like the feeling of having achived a goal, but also it will mean that I'll be free again to introduce new features. I still have more ideas than necessary about what to do with SBWG.

But right now I'm enjoing the fact that I'm able to make myself believe that it's okay to move on as slowly as I want and let my colloquial executive dysfunction do its think without impacting my feeling of self-worth oo much.

Comment via email

Bash script to burn your CPU easily

Entry created on 2021-05-16 Authors: steeph (376) Categories: Bash (31) CPUs (5) Code (31) Computer (78) Scripts (28) Software (53) Languages used: en (257) Topics: Projects → Code → Bash Scripts (23) edit

When making my own CPU cooler or testing some other stuff I sometimes need to create as high of a CPU load as possible. After typing while true; do true; done for the one-too-manyth time I finally decided to create an alias for that. But that didn't feel neat enough. So I made this simple Bash script.


#!/bin/bash

if [ "$#" -ne "1" ]
then
  echo "I need one argument and one argument only: Number of threads to start."
  echo "Hint: You have $(nproc) CPU cores."
  exit 1
fi

if [[ "$1" =~ ^[0-9]+$ ]]
then
  n="$1"
else
  echo "Argument needs to be a number: Number of threads to start."
  echo "Hint: You have $(nproc) CPU cores."
  exit 1
fi

trap "pkill -P $$; exit 0" SIGHUP SIGINT SIGTERM

while [ "$n" -gt 1 ]
do
  while true; do true; done &
  n=$((n-1))
done
echo "Started $1 threads to burn your CPU."
pidof -x "$(basename -- $0)"
while true; do true; done

Download

For a simpler script, you can also just use this three-liner without any arguments to stress all CPU cores:


#!/bin/bash
yeah() { while true; do true; done& }
threads=$(nproc)
for i in $(seq 1 $threads); do yeah; done
Comment via email
Go To Navigation Page
Show/Hide Navigation
Mastodon