davidg | David's Blog

Raspberry Pi and i3 window manager

I am really liking the combination of Raspbian and i3. I’ve always liked tiling windows; let the computer do the work for me to maximize the screen size of what I can get done. Me messing with window sizes is a waste of my time. Windows got particularly bad when the “tile” function added an extra blank column for people with touch-screens. I don’t have a touch-screen; so why‽‽‽ But I digress.

Novaspirit Tech has a nice video about how to add i3 to Raspbian.

The short form list of steps is:

cd Downloads
git clone https://github.com/Airblader/i3
1. So this one turned out to be a little more difficult than it is portrayed in the video, because since then, Git Hub has decided that passwords are bad, and I should have to log in, create a token, and use the token in place of the password. Okay, I jumped through those hoops.
sudo apt install meson
sudo apt install dh-autoreconf libxcb-keysyms1-dev libpango1.0-dev libxcb-util0-dev xcb libxcb1-dev libxcb-icccm4-dev libyajl-dev libev-dev libxcb-xkb-dev libxcb-cursor-dev libxkbcommon-dev libxcb-xinerama0-dev libxkbcommon-x11-dev libstartup-notification0-dev libxcb-randr0-dev libxcb-xrm0 libxcb-xrm-dev libxcb-shape0 libxcb-shape0-dev
cd i3
mkdir build; cd build
meson ..
ninja
sudo ninja install
cd /etc/xdg
cd lxsession
cd LXDE-pi
1. One quibble I have with this video is that using tab completion is a good thing, and this step was an excellent opportunity to demonstrate it. Essentially, we have done cd /etc/xdg/lxsession and now would be the right time to do ll
  1. Well, I do ll because I have set up an alias for ls -l
  2. There are two directories listed, and both begin with “LXDE”
  3. cd (space) and just hit the tab key is the answer here. The result is the same as typing cd LXDE (without hitting the Enter key). If the plain LXDE directory was the right one (it is not) I could just hit the Enter key. Since it is the other one I want, I type the next character (the dash) and hit the Tab key again. Then I hit Enter. I never had to hit the Shift key to get to that uppercase L for LXDE. The computer did the work for me.
Edit both desktop.conf and autostart
1. Near the top of desktop.conf, change window_manager=i3 then save and exit.
2. In autostart, comment out @lxpanel and @pcmanfm, then save and exit.
sudo reboot
On start, you will be prompted to configure i3. Yes, generate a config
Pick your modifier key. I do like the key on the left between Ctrl and Alt. I also like the little stickers that put a proper penguin over the Winders logo. I’m going to write this as $mod although the $ is not typed; it is just an indicator that whichever key is the the modifier key is variable and can change depending on your choice.
sudo apt install i3status
$mod+shift+r for restart i3
1. This gets us the i3 status line at the bottom.
sudo apt install dmenu
sudo apt install rofi
cd .config/i3
1. because of the reboot above, we started in our home directory. .config is a hidden directory (because of the leading dot in the file name). i3 is the sub-directory underneath, where the file config exists.
edit the file config
1. There is some good instruction in the video here about checking out which keys do what. For example, seeing that $mod+Enter launches a terminal is important; ditto $mod+d launches drun (which is apparently desktop run ?)
  1. $mod+arrow keys switch the focus between active windows.
2. Find your way to bindsym $mod+d exec --no-startup-id and replace the end dmenu_run with rofi -show drun -width 400 -lines 5
3. Save and exit config
4. $mod+shift+r to restart i3
  1. Now $mod+d brings up a searchable list of things to run (like Firefox).

And now I have a beautiful set up to install WordPress on my Raspberry Pi. I’ve got a terminal side-by-side with the web browser instructions, and I didn’t have to mess with screen sizing at all. The windows are as useful as possible, and the computer did that work for me.

I may have enjoyed this somewhat more than normal because of also listening to Noisestorm on YouTube. 😀

Raspberry Pi – sed for customization

I like automation, and have a project to migrate a friend’s web site to WordPress. There are a lot of MP3 files to move, so I’d like to script it.

I prefer to not test in production (in other words I’m not insane when it comes to computer operations) but that means having a consistent development environment. The Raspberry Pi is a very capable piece of equipment for a very reasonable price. So I think I want to automate building a development environment there.

I’m going to go with the Raspberry Pi OS with desktop

The short form of installing this is:

Download the correct image
1. I went with Raspbian instead of OpenSUSE because I want sound to work. Apparently the sound driver for Raspberry Pi has not been merged into Linux officially. As soon as it is, then OpenSUSE will have the sound driver baked in. But until then, I want to go with a Linux distribution that adds in the drivers for the Raspberry Pi sound hardware.
Copy the image to the Micro SD card
1. Open a terminal prompt
2. unzip -p 2021-05-07-raspios-buster-armhf.zip | sudo dd bs=4M of=/dev/sdc iflag=fullblock oflag=direct status=progress; sync
  1. This assumes that current working directory is the directory where the 2021-05-07-raspios-buster-armhf.zip file is.
  2. This comes from https://scribles.net/writing-raspbian-os-image-to-sd-card-on-linux/ – Thank you Max
  3. Note that the “out file” part of the DD command has a device specifier, and if you put in the wrong one, you can completely destroy your machine. Before you plug the Micro SD card in, do df -h and note the list of devices. None of the listed devices must be in your of=/dev/sdx option of the dd command.
  4. For example: I do df -h and I see a bunch of /dev/sda and a few /dev/sdb listed. This tells me that I must not use either of those for my outfile device. In my case, it happens to be /dev/sdc for my outfile device.
  5. My machine does not have a built in Micro SD slot. I bought an external USB dongle that lets me plug in memory cards. When I plug the dongle in, then /dev/sdc becomes an available device. This is correct – I don’t want dd overwriting any of my permanent devices.

After the image is copied to the Micro SD card, I can (software) eject the removable device, plug it into the Raspberry Pi and power on the Pi. After first boot, there are security updates to install; but after that I’ve got a virgin machine with bog standard software on it. Development is ready to begin!

First, create a couple of aliases

echo "..='cd ..'" > ~/.bash_aliases

echo "ll='ls -l'" >> ~/.bash_aliases

Finally, sed to edit /etc/inputrc

sudo sed --in-place=.original -e '/^# "\\e\[5/ s/# //' -e '/^# "\\e[\6/ s/# //' /etc/inputrc

Okay, so what does this do?

sed is the stream editor that let’s one search and replace inside files.

sudo is the command that grants permissions in Raspbian to edit files in /etc

--in-place=.original is the sed option to copy the file to a new name, first, before editing the file. The file will named the same as it was before, but with .original appended to the file name. I could just have easily used .backup or .bak

-e is the sed short form of --expression

It tells sed that what is between the single quotes is an expression to process. Note that the above command line has two expressions.

The two expressions are almost identical. Let’s break the first one down:

/^# "\\e\[5/ s/# //

First is a search, designated by the first /…/ pair. Second is a substitution, designated by the /…/…/ triplet (but the reason it is a substitution is because of the s in front of the triplet).

In this case, what we are searching for is:

^# "\\e\[5

The line in /etc/inputrc literally has this:

# "\e[5~": history-search-backward

You can see that the sed search expression ^# "\\e\[5 differs from the actual line by the leading carat character ^ and a couple extra backslashes.

That leading ^ (carat) is the instruction to sed that the match only works at the beginning of a line. First find the beginning of a line, then proceed.

The next few characters are just an exact match: # "

The line in inputrc has \e following the double quote mark; but, because this is going to processed on a command line (or by a shell script), we have to warn the command line interpreter that there is an actual backslash in the expression. Backslashes are normally an “escape” character on the command line, warning that “whatever you would normally do with the next character that you see, don’t do it. Even though the character is in the list of special characters that get special handling, just ignore that and deal with it as if it were not a special character”. Which is a long way of saying that to deal with \ in the data, we need to specify \\ in the script or command line.

Same thing applies with the [ character; it is normally a special character. The inputrc file has e[5 in the data, so we need e\[5 in the expression.

So the search part within the first two slashes matches when there is a beginning of a new line, a pound sign character, a space character, a (double) quote character, a backslash character, the letter e, an (opening) square bracket character, and the numeral 5.

The substitution part of the expression matches on the pound sign character and the space character, and replaces them with nothing. That’s what s/# // does: “s” in front of the slash triplet says we are going to do substitution, what to substitute are the quote and space, and what to substitute them with is the empty set: between the second and third slashes is nothing.

The second expression differs from the first by 6 instead of 5. This is from Changing the history search keystrokes

Easy button:

update_system_defaults.sh

(One should always look at the contents of the file instead of just blindly running it)

Firefox / Mozilla – on Windows – decided I did not need a Bookmarks button. WTH?

Two environments: work and home. Two platforms: Linux and Windows. Symptom is only on Windows, but spanned both work and home.

I do use the Firefox Sync feature, but of course two different email accounts for work versus home. I realize I am kind of stretching things; but, I use the Firefox sync thing across both Windows and Linux for both work and home.

Once I finally figured out how to get the Bookmarks button back, I tried to synchronize now to get the good/repaired installation of Firefox on Windows to propagate over to the broken versions. That did not work.

So: Firefox on Linux (both work and home) did not lose the Bookmarks button. Firefox on Windows (all machines) lost the bookmarks button. If it were a Firefox Sync thing, I would have thought that the repaired Windows installation would have propagated over to the other Windows machines.

Also, if it were a Firefox Sync thing, I don’t think it would have happened at the same time on two different accounts (work and home). Sure, I could have hit a bad keystroke and removed this button I wanted in one browser, and then Sync propagated the bad config. But that it happened at both work and home at the same time makes me think that it was a new build of Firefox that just clobbered the thing.

I am disappointed that 1) Mozilla didn’t have quality checks in place to catch this first, and 2) that Firefox Sync does not propagate the repair to the synchronized machines.

So, on every Windows machine, I now need to do:

(hamburger menu) –> More Tools –> Customize Toolbar

Then on the new screen, find the Bookmarks Menu button, and drag it to the toolbar. Repeat for all the Windows machines you have.

Microsoft as bully, yet again

Personally, I think that people have the capacity to be both humble and bullies. But it is a conflict, and, some people think they are being helpful when actually they are bullying. “If only you did things my way, and then everything will be better” says the bully under the delusion of being helpful.

Recently, Microsoft pushed out an update to Windows 10 which adds a “News and Interests” widget to the Windows task bar. You don’t get a choice; it will be installed whether you want it or not. You can turn it off after the fact, of course. But what the person at Microsoft behind this change does not care to hear is that I didn’t want to be violated in the first place.

All it really does is remind me that I don’t have control of this machine; Microsoft does.

Thanks, Microsoft. I already dislike you, but, I hadn’t gotten a recent reminder of why.

“What’s the big deal‽‽‽ It’s just a little thing. I was being helpful and making your life better” says the bully. Yeah, no. I hear what you are saying, and I see through the deceit (conceit) that this is somehow for my good. It is not. It is an ego stroke for yourself and nothing more.

If it were really for my good, it would be turned off by default, and not installed by default. Microsoft could say “we added a new feature, if you want”, and I’d be fine with that. But pushing it without permission tells the truth of the act.

Abandoned LastPass

LastPass was, for seven years, my password manager of choice. I liked that Steve Gibson of Gibson Research Corporation liked it. I liked that it had Yubikey support. I liked that it had an app for my iPhone. I liked that because it was a cloud service, my passwords followed me around.

The idea is a good one, too: memorize a single complex password, use it plus 2FA (“second factor authentication” in this case my Yubikey) and then make the passwords on every other web site extremely complex. Like Hn6k344SdRt#CT_Epste1nd1dn’tk1llh1mself_PFnPr2XP#J$4P*@Lyxi!r complex.

I did not mind paying for that service, since I know that it costs money to run servers and pay employees and such. The price really wasn’t too bad, either.

But somewhere along the line, the creators of LastPass decided to cash out. They sold the company to LogMeIn. But now, the LogMeIn folks are out cash money, and they need to make that money back; the quicker, the better. Suddenly I and all their other customers began to look like marks to be played.

Sigh.

For several months, I wanted to take a screen shot of my LastPass initial login screen, and then post the screenshot to the Reddit Asshole Design community. What it was, was that all of a sudden, LastPass would post a fake “Warning – We Care About Your Security” alert every time I logged in. But what it really was, was a request to put my credit card number on file with them, so they could auto-renew. It wasn’t about my security. It was about theirs. I’m going to need to buy a pair of cowboy boots, the bullshit is getting so deep around here.

This was a constant reminder that the company had changed, and wasn’t the same company I signed up with. What finally pushed me over the edge was their announcement that as of March 16, 2021 you have to pay, or else “no passwords for you!” if you want to use LastPass on mobile. So now we see their true character: “I Am Altering the Deal, Pray I Don’t Alter It Any Further.”

And I’m out.

Need to print + OpenSUSE 15.3 upgrade – What could go wrong?

I needed to go to a new doctor yesterday. The day before, they had called and left a message that I would also need to bring along a list of my current pharmaceutical prescriptions. I got the bright idea to log in to the online pharmacy web page and print my current list. This is about 40 minutes before I need to step in to the home office to report to work.

It went poorly.

Still, if this is the worst thing to happen to me this month, I am a fortunate man. I’m a fortunate man who cannot print from Linux, but I’m still a fortunate man.

Certainly part of the problem is my fault; I had upgraded from OpenSUSE 15.2 to 15.3. 15.3 was released two weeks ago; I upgraded about ten days ago. This was not enough bake time. I should have listened to my own advice: do less yeet and more tootle. But yeeted I had, so the story unfolds ….

Okay, so I logged in to the pharmacy web page, and used the browser to print. Got no printer noises, but no error about anything, either.

The print driver I’m using is from OpenPrinting.org and it did previously work. I did print something three weeks ago. But today, nada.

Go into the printer manager in OpenSUSE 15.3 and do a test print. No printer noise, but no error alert either. It asks if the print worked; (no) so says do journalctl to see what went wrong.

I don’t like journalctl. It spits at me about permissions, and I used to just be able to just grep a log file – any log file – and search for terms like “error” or “warn” or “cups”. I just want to print, man.

Okay, dig in and find that there is an error with the driver. Reinstall the driver. The driver will not install.

The driver is dependent on LSB. LSB = Linux Standard Base, which was the idea that the various packagers of Linux would all agree on what should be in a base install of Linux (that supports LSB). Software vendors could count on the base packages being there, or worst case say, “this software needs LSB, please install it”.

I had previously installed LSB (to get the printer to work), but now it’s missing. That must have happened during the 15.2 to 15.3 upgrade.

Okay, no big deal: zypper in lsb

Problem: nothing provides ‘/usr/bin/pidof’ needed by the to be installed lsb-foo

Well that’s darling. It’s a bug, and it is fixed in OpenSUSE Factory. I just want to print, man, and it’s now 20 minutes before work.

Okay, go to the fallback position: print to PDF, copy the file to a Winders box, and print from there.

I have Nextcloud client installed and running on most of my machines. Copy the file to my Nextcloud folder. Go to a Windows machine – there are no new files in the Nextcloud folder. Machine is acting wonky anyway, so I reboot^(yeet!)

Microsoft decided I needed a Weather widget in my taskbar, so they inserted one without asking. I need to lose some time praying to remove the murderous rage I have toward Microsoft for being so un-invitingly forcefully helpful.
Nextcloud client has an update, would I like to install? Yes, please. What was that about less yeet and more tootle?
Nextcloud client version 3.2.2 is no longer compatible with your older Nextcloud server. Have a nice day!
It is now 10 minutes before work. I just want to print, man; my travel time to the doctor did not pad with lead time for print fixing, and as the famous mage once said: “Outlook not so good”.

Okay, what about the web page version of the Nextcloud server? Right, dang, I forgot I was going to need my physical 2nd Factor authentication key. Back to the living room to get it.

Logged in on the Windows box. The file is not there.

Dang it! The Nextcloud client on my Linux box has the same version problem. Back to the living room, open up the Nextcloud files web portal, do the physical physical 2nd Factor authentication thing and copy the file up. Back to the home office, open the PDF in the Nextcloud files web portal in Firefox and hit the print button. Finally, noise from the printer in the living room.

Put on a shirt and my shorts, get an energy drink out of the refrigerator, and I’ve got 30 seconds to spare.

“One does not simply press print”

Next week, I’m going to install a firewall router!

New toy: Raspberry Pi 4

So of course, the first thing I want to do (after installing Raspbian and applying updates) was to add some aliases and switch the editor and history search commands.

Changing the editor to vim

Changing the editor to vim was easier, once I knew how.

Install vim
update-alternatives

Vim does not come installed by default on a Raspbian. But adding it is easy enough:

sudo apt-get install vim

I had found someone that said (when thing weren’t working the way I wanted) to sudo apt-get install vim-full ← this does not work! There is no package “vim-full” from which to install. At least not on my fresh-out-of-the-box Raspbian machine.

sudo update-alternatives --config editor

This opened up a list of available editors, numbered for selection, with an asterisk next to the default. It was set to nano, but I wanted vim.

Except that there were two vim choices: vim.basic and vim.tiny

Which one to choose? Neither looks like vim.full to me. 😉

Turns out I wanted vim.basic

Now, I can be in the less utility, and if I want to edit the file (and I already have permission to do so) I can hit the v key and be editing in vim.

Adding some aliases

This was super easy. I created a file, ~/.bash_aliases and added the alias commands to it.

alias ..='cd ..'
alias ll='ls -l'

Changing the history search keystrokes

This one was the closest to what is described in my How to make Ubuntu have a nice bash shell like OpenSuSE post.

sudo vim /etc/inputrc

Find the commented out commands, and uncomment them:

# alternate mappings for "page up" and "page down" to search the history
# "\e[5~": history-search-backward
# "\e[6~": history-search-forward

All I have to do is to delete the “#” character that declares \e[5~ and \e[6~ to be a comment. With vim, this is the x key

Once you’ve been bamboozled ….

Once you’ve been bamboozled, it is almost impossible to become un-bamboozled.

This was from an AskReddit question about “What was the best quote or life changing saying or most profound advice people had heard?” Something like that; but my search did not find the exact entry to cite. One of the answers was this one. It’s great. I mentioned this to a friend of mine; he thought Carl Sagan had said it. Well, essentially yes, but not exactly. Carl Sagan’s quote goes like this:

“One of the saddest lessons of history is this: If we’ve been bamboozled long enough, we tend to reject any evidence of the bamboozle. We’re no longer interested in finding out the truth. The bamboozle has captured us. It’s simply too painful to acknowledge, even to ourselves, that we’ve been taken. Once you give a charlatan power over you, you almost never get it back.”
https://www.goodreads.com/quotes/85171-one-of-the-saddest-lessons-of-history-is-this-if

This is well said, but also really wordy, plus throws in a ten dollar word: charlatan. I like the short and sweet version.

It seems to me that the almost the entire USA has been bamboozled about politics.

The Left has been bamboozled that Donald Trump Is A Bad Man.

The Right has been bamboozled that Donald Trump Is A Good Man.

I remember seeing a cartoon not that long ago (within a year or two) that had a King on a balcony with an advisor, overlooking an angry mob. Here it is (I linked to the original source, so you can get to that web page – credit where credit is due):

The advisor was saying to the King: “Oh, You don’t need to fight them – you just need to convince the pitchfork people that the torch people want to take away their pitchforks.”

When I went looking, Google search failed to find this cartoon. I mentioned it to a friend, and he saw it on Facebook. I asked him to forward it to me. From there, I was able upload it to Google Image Search, and then finally find the original publisher. The ~~conspiracy theorist~~ spoiler alerter in me thinks the search engines of the day have de-ranked or removed this image in search results because it spoils the narrative.

The idea here is an intersection between the two old sayings A house divided against itself cannot stand and The People restrain themselves and anxiously hope for just two things: bread and circuses.

The Left is thoroughly convinced that The Right has been bamboozled. The Right is thoroughly convinced that The Left has been bamboozled.

I am convinced both have been bamboozled by the deep state and it’s unholy alliance with mass media. When I say mass media, I’m also looking at you: Facebook and Google and Twitter.

Here’s the thing about Donald Trump: he was never supposed to be President.

The deep state mass media planned to get Hillary Clinton. They thought they earned Hillary Clinton. By knocking out every good opposing candidate, there was no way that Hillary could lose. There was no way that Hillary Clinton could lose against Donald Fucking Trump. Knock out every other candidate, and the election was a done deal.

This was perfect for the deep state, because Bill and Hillary Clinton were already players. They’d played ball before, and were happy to play again. As insiders, their keepers had leverage on them, and as players, they knew their keepers would be comforted with them as lackeys. It was a win-win situation.

But (“oh by the way”) Hillary Clinton was the worst possible candidate for President.

Which is proven out, because she lost to Donald Fucking Trump, dontcha know. Fair and square, she was simply that BAD of a candidate. And to be fair, Donald was actually a very good campaigner, and a master of Twitter trolling. His campaign speeches were super entertaining. The deep state completely underestimated how well Donald would perform.

Donald was an outsider. This was a disaster for the deep state.

Chuck Schumer delivering the deep state warning to Donald Trump to play ball or else (after election but before inauguration).

Donald bristled at being told to take his role of lackey. Now the deep state is on his shit list.
If Donald made it to a second term, there was no remaining leverage to keep him from ravaging the deep state.

Does it appear to you that Donald rolled over and became a lackey?

The only choice the deep state had was to backstab the sitting President every chance they could get.

Wow did they ever.

The Commander In Chief: that is who the deep state is supposed to obey. Instead, they did everything they could to subvert CIC/POTUS. They became traitors to the rule of law.

And you, dear reader, got taken in by the charlatans that Donald Trump Is A <‽> Man.

rsync is wonderful, but ….

rsync /datastore/61/E4 /newserver/61/E4

is wrong and will mess you up!

Imagine if you will, that you have a whole bunch of data stored on an old server, and you need to copy it to a new server. The rsync utility would be an obvious way to go. There are things about the job and rsync that you might want to tweak, though, and that’s where things get ugly. Part of this is bash’ fault.

Imagine if you will, that your data store is 120 million small files (emails) stored in 256**3 directories. 256 cubed is 16,777,216 sub directories.

The programmer that created the data store to hold all these files needed subdirectories to put the files in. Linux doesn’t really like 20,000+ files in one directory. It would be better to have more subdirectories, with less files per subdirectory. So the programmer started with a loop:

for 00 .. FF mkdir

Then the programmer did a change directory into each of those directories he just made, and did the exact same thing.

cd 00;for 00 .. FF mkdir;cd ..
cd 01;for 00 .. FF mkdir;cd ..
cd 02;for 00 .. FF mkdir;cd ..
...
cd FF;for 00 .. FF mkdir;cd ..

That gets you to 256 squared, which is 65,536

And then the programmer did a change directory into each of those directories he just made, and did the exact same thing. All 65,536 second level subdirectories got a third level of another 256 subdirectories. That gets you to 16,777,216 which is 256 cubed.

So your file server directory structure might contain this:

/datastore/61/E4/7D

Inside good old 61/E4/7D there might be twenty to thirty files, each one holding the content of an email, or a metadata file about the email. The programmer was pretty good about filling all of the datastore subdirectories to nineteen files each, then twenty files each, then twenty one files each. No Linux system is going to have a problem with twenty one files in a subdirectory.

The only real problem here is if you need to traverse everything in /datastore – this takes forever

Back to the problem of copying everything from /datastore to /newserver. Let’s assume that /newserver in on a different machine, and we are using remote file system mount command to make the remote machine appear to be a local disk (mount point).

You might think the rsync command ought to look like this:

rsync --archive /datastore /newserver

There are two things that make this sub-optimal. First, it is single-threaded. Second, there is no progress feedback.

The single threaded part isn’t so bad; it just means that we are losing speed due to rsync overhead. The server has twelve cores, the network is 10 Gbps Fibre Channel, the /datastore disk has multiple spindles, but rsync was designed for slow networks way back when in the bad old days.

At this point, you might ask “why not do a straight cp -r” (copy command, recursive)? It’s not a terrible idea; but, what if there were a network glitch? The entire cp -r would have to be started over, and every bit already copied would be copied again. This is where rsync shines: if the files in the destination are the same as the source, the copy is skipped. cp -r also suffers from the same lack of progress feedback.

Did I mention that the 120 million files are also 9.3 terabytes of files? I really don’t want to get to 98% done and then have a network glitch cause me to copy another 9.3 TB over, which would be the case with cp -r

The tests I’ve done indicate that four rsync commands, running simultaneously, copied the most data in the shortest period of time in my environment^*. More than four rsync commands at once, and I started to saturate the disk channel. Less than four rsync commands, and something is waiting around, twiddling it’s thumbs, waiting for rsync to get busy with the copying again, which it will do, as soon as it finishes up with the overhead it’s working on.

The other problem is a lack of progress feedback. The copy is going to take multiple days. It would be nice to know if we are at 8% complete or 41% complete or 93% complete. It would be nice to be able to compute what the percentage complete is.

Well, how about 64K rsync commands, each with a print statement of the directory it is processing? And if we could run four of them in parallel, we could get the multiple jobs speedup too.

You might think the rsync commands ought to look like this:

rsync --archive /datastore/00/00 /newserver/00/00
rsync --archive /datastore/00/01 /newserver/00/01
rsync --archive /datastore/00/02 /newserver/00/02
rsync --archive /datastore/00/03 /newserver/00/03
rsync --archive /datastore/00/04 /newserver/00/04
...
rsync --archive /datastore/FF/FF /newserver/FF/FF

but WOW would you ever be wrong!

Remember old /datastore/61/E4/7D up there? This format for rsync would put E4 in the source under E4 in the destination! In other words, although the source looks like this: /datastore/61/E4/7D the destination would look like this: /newserver/61/E4/E4/7D

To be done right, the command needs to look like this:

rsync --archive /datastore/00/00/* /newserver/00/00/
rsync --archive /datastore/00/01/* /newserver/00/01/
rsync --archive /datastore/00/02/* /newserver/00/02/
rsync --archive /datastore/00/03/* /newserver/00/03/
rsync --archive /datastore/00/04/* /newserver/00/04/
...
rsync --archive /datastore/FF/FF/* /newserver/FF/FF/

The source needs a trailing slash and asterisk to tell rsync to copy the stuff underneath the source (not the source itself) to the destination (which is finished with a slash).

Enter the problem where bash is a pain in the ass.

Well, before I go there, let me mention that it wasn’t too bad to write a Perl script to write this bash script, and do three things per source and destination pair:

echo "rsync --archive /datastore/00/00/* /newserver/00/00/"
rsync --archive /datastore/00/00/* /newserver/00/00/
echo "/newserver/00/00/" > /tmp/tracking_report_file

The first line prints the current status to the screen. The second line launches the rsync. The third line overwrites a file, tracking_report_file, with the last rsync finished.

So, crank up screen first, launch the bash script, and some number of days from now, the copying will be done.

That /tmp/tracking_report_file gives me a pair of hexadecimal pairs, which I can then use to compute percentage complete. For example, when /newserver/7F/FF updates to /newserver/80/00, then we are going to be just over 50% done.

Heck, I can detach from screen, and I don’t even have to watch the rsyncs happen. I mean that I do need to, but I don’t have to. Better yet, I can take the same routine that converts the pair of hexadecimal pairs into percentage complete and wrap that inside a cron job that sends an email. Progress status tracking accomplished!

But this does not solve the single-threaded rsync problem.

And ultimately, I could not get it done.

What looked to be an okay solution was using the find command, to feed into xargs which could do shell stuff in parallel. I even got as far as getting bash shell variables to create the rsync --archive /datastore/00/00/00 /newserver/00/00/00 part.

Okay, that would be 16 million smaller rsyncs instead of 64 thousand larger ones, but I might even be able to bump up the parallelism to six or eight or nine.

But the serious problem the rsync –archive /datastore/00/00/00 /newserver/00/00/00 command has, is the naive problem: the missing trailing slash and asterisk are going to put the source underneath a destination. I need to put the trailing slash and asterisk on there.

And bash says “that’s a nope”

Trailing slashes and asterisks are automatically culled from output, because (reasons).

Oh well. The find command also spits out the directories it finds in rather random order. My bash script with sequential rsyncs by sorted order means that the last one complete really is some-percentage-of-the-total done. But if find chooses to spit out /datastore/b3/8e/76 instead of /datastore/00/00/00 then my status tracking doesn’t actually work. I would be forced to traverse all of /newserver/ and count which of the 17 million are complete; which would take freaking forever.

Yes, I said 17 million. Did you notice that the programmer that created subdirectories did some of them in lowercase hexadecimal? That happened when we brought in another email system (Exchange). Lovely.

^*the last time I did this migration, although it was on a four core box, then.

Microsoft fouled up when they got rid of gallery.technet.microsoft.com

In the real world, people like to find solutions and then link to the solution as a form of documentation. It is a way of being helpful. I can feel good about myself if I help you out (or at least I’m trying to help you out). The result is that there tend to be a lot of forum discussions and blog posts that have a link to a gallery.technet.microsoft.com script that solves the problem.

As solutions go, gallery.technet.microsoft.com was a great idea. People write a script, that script works, so the author donates it to world at large by publishing it (free of charge and disclaiming all complaints about damage). Microsoft benefited because if you knew nothing else, you knew to go search there for possible help. If I’m trying to solve a problem, and I find the solution on gallery.technet.microsoft.com, then I’m helping if I tell people “here was my problem, and I found the solution: foobarbaz at gallery.technet.microsoft.com”

Unfortunately for us, someone at Microsoft felt the need to push the world into complying with their grandiose idea: “Let’s get rid of Technet and replace it with docs.microsoft.com !”

This was a terrible idea.

And no-one at Microsoft was grown-up enough to stop it.

So now, the world wide web is littered with broken links to solutions that used to be helpful, but now go to https://docs.microsoft.com/en-us/samples/browse/?redirectedfrom=SomethingThatUsedToBeGreatButWeKilledIt

I don’t know what they were thinking, but it was probably someone wanting to pridefully change the world to comply with the way they thought the world ought to be. All they really did is break a previously good thing.