Category: Nextcloud

Let’s Encrypt for my internal domain

It is time to renew my wildcard SSL certificate for an internal domain I have, and here are the steps I went through to solve it. When I say internal domain I’m referring to a DNS domain that exists on the public Internet, but which wholly and only points to the IP address of my home broadband router. That router has pass-through enabled, so that essentially, my pfSense box is my presence on the Internet for everything inside my home.

I turned off HAProxy so that pfSense wouldn’t be sending the challenge traffic to the only internal server I put out there. The internal server, Nextcloud, doesn’t play nice with others; in order to keep things consistent, they want it to be an appliance where the only stuff running on the box is their code. Okay, I get that. This wouldn’t be so annoying if it wasn’t bug-riddled junk that is in a huge rush to implement new features. Can you say “AI”? But I digress.

I created a new Linode API key in case the problem was that the old API key didn’t have access. Well, the first new key had the wrong selector, and resulted in “Your OAuth token is not authorized to use this endpoint”.

The problem is that the pfSense script is trying to generate a challenge key and insert it into a web server that doesn’t exist. The pfSense web admin portal is that web server. When I turned off HAProxy, that should have opened it up. It did, but I couldn’t tell because the Linode API key was wrong.

Okay, maybe I need to log in to the pfSense box and manually use a generated challenge key? How to log in to the pfSense box? When was the last time I did that?

Here’s a convenient command:

 history | awk '{$1="";print substr($0,2)}' | grep "ssh " | grep -v history | sort | uniq

We run the output of the history command through awk to remove line numbers, then search for "ssh " (the trailing space omits ssh-copy-id and such), run that through sort, and run that through uniq. Et voilà, and I have a list of all twelve boxes I’ve logged in to since history.

Sigh: pfSense isn’t one of them.

But this was a good exercise: I did get logged into pfSense, and did find the “Your OAuth token is not authorized to use this endpoint” problem.

I deleted the previous Linode v4 API certificate specifications, and it worked.

Time to turn HAProxy back on.

Okay, the short form is:

  1. Generated a new Linode API access token with Domain read/write access
    • This probably won’t be required if the access token hasn’t expired.
  2. pfSense > Services > HAProxy > Settings > disable and apply
  3. pfSense > Services > Acme > Certificates > pick certificate and Edit > delete the Domain SAN list entry > Add a new Domain SAN list entry with the new Linode API access token > Save
  4. pfSense > Services > Acme > Certificates > pick certificate and hit Renew
  5. Do the other certificate in the list
  6. pfSense > Services > HAProxy > Settings > Enable and apply

Nextcloud has far too many bugs

I like Nextcloud: it is probably my favorite piece of software that I run. But man it has a ton of bugs. Their support forums are full of people reporting problems, and there is no solution.

Sometimes, the support forums do report that “Yes, this bug is listed in the Github bug tracker.”

There are more than 2,300 open bugs.

It is ridiculous. I saw a changelog that said the update would fix a bug I was seeing. I installed the update. The bug was still there. Quality control in this project is deplorable. I have another bug that I experience daily that has been open for almost two years.

What is dismaying to me is that the main developers have a Microsoft mentality: let’s add new features! No, we’re not going to work on bug fixes: debugging is boring.

In a couple of years, after I retire, I may decide to learn PHP programming. I haven’t really been fond of PHP.

Way back when, I read (well, got through the first few chapters) of a programming book that pointed out that software can be written to be mathematically provably correct. For every memory allocation, the math can add to the sum of debt. Memory should be specifically de-allocated, which subtracts from the sum of debt. When at the end of your source code, the sum is zero, you’ve handled all cases of allocating and de-allocating.

Nothing about PHP makes me think this is true for that language. Maybe I just don’t know the language well enough.

But, PHP does run a ton of super successful projects. So there must be something there which is valuable.

But yeah, I’m not going to be competent at writing PHP for years. Hopefully, someone at Nextcloud will get tasked with fixing bugs before then.

Temporary fix for Nextcloud calendar broken sync

Nextcloud has a nice home page called the Dashboard, which has calendar items and to-do list on it. But ever since Calendar App version 4.5, it has been broken for items sourced outside of Nextcloud. In other words, if you create a calendar item on your smartphone and sync it in to Nextcloud, on the Calendar web page you can see the item, but on the Dashboard home page it will be missing. The solution is to downgrade the Calendar app to version 4.4.5

Steps to perform:

  1. In the Nextcloud admin interface, find the Calendar app and disable it
  2. ssh into your Nextcloud instance
  3. cd /var/www/html/nextcloud/apps/
  4. mv calendar calendar-old
  5. wget -q https://github.com/nextcloud-releases/calendar/releases/download/v4.4.5/calendar-v4.4.5.tar.gz
  6. tar xvf calendar-v4.4.5.tar.gz
  7. chown -R user:group calendar
  8. In the Nextcloud admin interface, select the Disabled apps section. Then Enable (but not update) the Calendar 4.4.5 app.

And now, when you go back to your Dashboard home page, your calendar will have all the items on it. 🙂

You do get to apply this fix after every update. 🙁

Technically, this post title is somewhat misleading: sync is not broken. What is broken is that items that sync in from CalDAV sources apparently have something that, when it is present, causes the Dashboard page to skip those calendar items. It just looks like sync is broken because you knew the items were on your calendar: but when you look at the Dashboard for today, they are missing. I suppose a better title would be Temporary fix for Nextcloud calendar (some) items missing from Dashboard

Ogg > MP3 (thanks, Apple) (not)

I have several CDs (Compact Discs, not Certificates of Deposit) of music that I like. When I popped them into my PC, I got several folders of files I could copy from. I chose to copy the .ogg files because I liked the idea of using an encoding format without weird licensing issues.

Apple has foiled that plan. If I try to play a playlist on an Apple device, the .ogg files get skipped because (apparently) Apple doesn’t feel like playing nice with the Open Source community. They may have more money than God, but adding another codec – that doesn’t have license issues – to their devices isn’t something they are going to spend money on.

When I work on-premises in the office, my co-workers are often noisy and annoying. I want to pop in my Airpods and play background music to drown out their inane chatter. I don’t want to carry the music files on my device; but I do have a Nextcloud server at home that can stream the audio from the Music app web page. I can log in on my iPhone and play the playlist.

But because it’s an iPhone, it auto-skips the Ogg Vorbis files. This doesn’t happen when I’m at home playing the same playlists on Linux or Windows.

So now I get to re-copy the files from the physical media to my NAS (network attached storage) which in this case is a Synology.

First, I get to delete the files with the .ogg file extension. Two steps (for example):

exiftool -p '$filename' -if '$album =~ /WOW Worship: Yellow \(disc 1\)/' *.ogg > wow_worship_ogg_file_list
This generates a file, wow_worship_ogg_file_list, which has the file names in a list.

then to delete them:

xargs -I{} rm -r "{}" < /path/wow_worship_ogg_file_list

Second, after having cleared out the disk space, I can copy from my physical CD to my NAS. That takes a while; and, after it is done, the file names aren’t wonderful. Rename music files to their title to the rescue.

Except, of course, for a duplicate file name. I have an MP3 file I bought from Amazon (published by Monstercat) with the same title as one of the files from the WOW Worship CD. I would prefer to rename the Monstercat file, but really if I’m going to be running the rename music files to their title command often, I need to change the Title inside the .mp3 file. If I don’t, the next time I run it, it will attempt to rename the file to a duplicate name that is already in use.

Exiftool doesn’t really write new Titles, apparently. I think it can, depending on the file type. I wonder if the weird license problems of MP3 are at the root of the problem. Whatever: the answer was to add the id3v2 program and use it instead.

id3v2 -t 'Title by Artist' file.mp3

Now the rename music files to their title script moves the one file to the new file name, and the other file to its simpler file name than what came off the CD.

New Nextcloud setup with cron and transactional file locking problems (solved)

I set up Nextcloud on a new instance of Debian, and thought I had added all the pieces for memory cache and file cache, and had set up cron to run php -f /var/www/html/nextcloud/cron.php correctly. But in the Administration Overview screen I was still seeing this:

  • Last background job execution ran 2 hours ago. Something seems wrong.
  • The database is used for transactional file locking. To enhance performance, please configure memcache, if available.

But had installed Redis and APCu and configured them … so what is wrong?

I should mention that I’m using php 8.2. Apparently, with that new version of php, the APCu code now needs an additional setting that wasn’t needed before.

Find your way to /etc/php/8.2/mods-available and edit the apcu.ini file. Add this:

apc.enable_cli=1

Finally! I have the green check mark: All checks passed.

How to test if you cron job is going to run correctly:

sudo -u www-data php -f /var/www/html/nextcloud/cron.php

I had to add the sudo package to Debian, because the basic server build did not come with that. But what it does do, is let me switch user and do the command. First, I specify the same user that Apache is going to use: www-data and then I run the PHP interpreter, using the file /var/www/html/nextcloud/cron.php

Prior to the change, it erred out with a rather ugly OCP\HintException: [0]: Memcache \OC\Memcache\APCu not available for local cache (Is the matching PHP module installed and enabled?)

Now after the change it simply runs without reporting anything (everything ran sucessful)

Nextcloud web on mobile: remove iOS App prompt

New update: A little while ago, I updated my Nextcloud client on my iPhone, and a feature has been added to the “Use The App!” pop-up to turn it off. Thank you! I no longer need to mess with layout.user.php file on the server.

Log in to MySQL, and run this:

insert into oc_appconfig(appid,configkey,configvalue) values ('theming','iTunesAppId','');

There is a file in Nextcloud, layout.user.php which pushes a link to the iOS app. I wish to remove that. It will come back, every time Nextcloud pushes an update, so here is a note on how to remove it.

The file is (web root)/core/templates/layout.user.php

The three lines to delete are:

<?php if ($theme->getiTunesAppId() !== '') { ?>
<meta name="apple-itunes-app" content="app-id=<?php p($theme->getiTunesAppId()); ?>">
<?php } ?>

I don’t know if there is a setting anywhere in Nextcloud which would let me simply disable the prompt to use the app. I don’t see such; so the fallback is to edit the source code. It would be nice if there were a setting stored in the database instead. When the Nextcloud people publish an update, the recorded setting could keep the “Use The App!” prompt disabled. But I don’t think there is a setting I can get at which lets me control this.

That php code implies that the theme could have a setting; but I don’t see anything in the theme setting page for a flag of getiTunesAppId.

Mostly, I use the other pieces of Nextcloud: Calendar, Tasks, Music, and Dashboard. For those, that “Use The App!” prompt is a waste of screen real estate on a platform that doesn’t have a lot of screen real estate to waste.

I wouldn’t terribly mind if the “Use The App!” prompt showed up if I were to visit the Files part of the Nextcloud web UI: that’s the only thing the iOS app is good for. But I almost never use the Files part of Nextcloud when on mobile.

I’ve asked the people in the Nextcloud support forum if I’m missing something; this could be a solved problem already. If I get an answer back, I’ll update this post.

User dakkar-tn did answer my post with the database query above. I am very appreciative of his help.

exiftool to playlist file

Assuming that you are currently in the subdirectory with music files, and those files are of type .ogg and you want to create a playlist file named _great.m3u which contains the names of the songs with “World of Warcraft” in the album name, this one liner creates such a file:

exiftool -p '$filename' -if '$album =~ /World of Warcraft/' *.ogg 2> /dev/null > _great.m3u

Assuming that you then wanted to add the files of type mp3 from the artist E.S. Posthumus, this one liner adds to that file:

exiftool -p '$filename' -if '$artist =~ /E.S. Posthumus/' *.mp3 2> /dev/null >> _great.m3u

Assuming that you then wanted to add files of type mp3 with the Genre of “Classical”, this one liner adds these to that file:

exiftool -p '$filename' -if '$genre =~ /Classical/' *.mp3 2> /dev/null >> _great.m3u

Assuming that you then wanted to add files of type mp3 with the Comment of “VIVA EL PRESIDENTE!”, this one liner adds these to that file:

exiftool -p '$filename' -if '$comment =~ /VIVA EL PRESIDENTE!/' *.mp3 2> /dev/null >> _great.m3u

exiftool is the wonderful utility written and maintained by Phil Harvey

-p '$filename' prints the file name. We later strip off the other stuff by redirecting stderr to null. That’s the 2> /dev/null part.

-if '$album =~ /World of Warcraft/' and -if '$artist =~ /E.S. Posthumus/' are matches against a regular expression. =~ says we are doing a match and the text between the slashes are what need to be present for the match to report true.

> _great.m3u overwrites the existing file, but then >> _great.m3u appends to it.

Care to guess who purchased the collector’s editions of some of the games so I could get a CD of the game music (or files from Steam)?

One thing (I don’t know that it’s a problem, really) is that Artist =~ /E.S. Posthumus/ will find the same file as Genre =~ /Classical/ so the same songs end up in the playlist twice. Maybe I just like E.S. Posthumus so much that I want their chance of being picked by the shuffler better than average. 😉

But if that’s not your bag, this will make a new file (with a new name) which contains only unique song file names:

sort _great.m3u --unique --output=_great-unique.m3u

If you happen to have your own Nextcloud with the Music Player app, you can import this _great-unique.m3u file directly into a new playlist.

The Helm migration is complete

As I mentioned before, The Helm email appliance company is calling it quits, which I understand. If the business isn’t going to make it, it is better to pull the plug than just keep letting things linger. Best of luck to them on their next adventure.

So, what did I do?

  • (there was a detour while Amazon pissed on their customers wanting to run Mail-In-A-Box) (me)
  • I provisioned the smallest Ubuntu 22.04 LTS machine that Linode has.
    • Mildly annoyed that it doesn’t really support LVM (Logical Volume Manager); they have a backup service that runs an agent inside their machines, and that agent doesn’t do LVM. Still, I know that I’m going to need to grow disks, so I had to learn how to re-partition the Linode so I could do LVM. LVM done.
  • I made a mail server on the Linode machine at a domain name I have that I don’t really use. I followed the excellent guide from Christoph Haas at workaround.org: ISPmail guide for Debian 11 “Bullseye”
  • I got RoundCube webmail working for the domain name; complete with SPF and DKIM.
  • I got Thunderbird to send and receive from the domain name.
  • Then I added Nextcloud to the same box. I wanted CalDav for contacts and calendar, when I eventually hook my iPhone to it.
    • The Nextcloud documentation really needs a lot of work here. If I were retired, I would like to help them with their documentation.
    • Finally, I have the files.example.tld function of The Helm replaced, although at a different domain name.
    • Rspamd uses Redis, but so does Nextcloud. But one uses the network stack, and the other, Unix sockets. Get them both set same.
  • Then I added Duplicati backup. This wasn’t great, as it added a ton of overhead in the form of Mono, just for a graphical user interface.
  • I realize that I’m going to want to host my WordPress here too. I don’t want to have to wrangle four Let’s Encrypt SSL certificates, one for each domain. What about a single wildcard SSL certificate?
    • Yes, that can be done, but: my domain names registrar doesn’t support it. Linode does, though. I install the Linode DNS agent on my machine, and spin up Linode DNS servers to do the DNS work. I have to configure my domain names registrar to tell the rest of the world that Linode is where my name servers are.
    • Somewhere in there I installed the Unbounded DNS resolver. Looks like I need this on my home machine, too, for Home Assistant.io1
  • I got to the point where I could request the domain name transfer. Turns out the people at The Helm were going through Ghandi.net. Ghandi.net tooks as long as they legally could, before actually doing the DNS transfer.
    • Ghandi –> registrar, then registrar to point to Linode. Linode DNS needs to be reconfigured for SPF and DKIM. I had gotten some DNS records wrong, too.
  • Thunderbird to connect to the mail.domain.tld, and though the name hasn’t changed, everything underneath has. Thunderbird is not happy; I lose all my old mail.
    • Well, I didn’t, but it is in a new folder now, so that I’ve got an old version of my mailbox and a new version of my mailbox, and they are separate. Not ideal. Perhaps I could have done an IMAP to IMAP transfer, if I hadn’t already moved the domain name.
  • Hey, looky there: one of the volumes filled up (but everything else was unaffected). Time to grow a disk using LVM.
  • iPhone to connect to CalDAV; phew that was not well documented and had tons of conflicting information.
  • Not really happy with Duplicati, so I remove it and Mono, and install Restic backup instead.
  • Okay, so the last thing left to do is to migrate this blog from Amazon to this new Linode machine. The transfer using NS Cloner goes well, as it usually does. But domain names need to be updated via Let’s Encrypt certbot.
    • Crud. I’m on holiday out of town with family, and have only a Windows laptop with me. Per best practice security protocols, I can only ssh in from home. Logging in via root@ is blocked, and I don’t think I can even do a ssh-copy-id without getting in first and lowering the root login barrier. The certbot to add gerisch.org to the domains list is going to have to wait.
  • Here I am, at home, and I’m done. Dovecot, Postfix, RoundCube, Nextcloud, and WordPress all on one box.
  • While I was on holiday, I took the .mp3 files on the Nextcloud, and made Nextcloud Music Player playlists for the different types of files. Then on the 16 hour drive home, my iPhone logged in to the Nextcloud web interface and played playlists.
    • It’s a bit of nirvana to me, to have a large list of songs (randomized of course) playing absolutely advertising-free because I paid for the songs in the first place.
  1. I ended up not connecting Home Assistant to their cloud ↩︎

Need to print + OpenSUSE 15.3 upgrade – What could go wrong?

I needed to go to a new doctor yesterday. The day before, they had called and left a message that I would also need to bring along a list of my current pharmaceutical prescriptions. I got the bright idea to log in to the online pharmacy web page and print my current list. This is about 40 minutes before I need to step in to the home office to report to work.

It went poorly.

Still, if this is the worst thing to happen to me this month, I am a fortunate man. I’m a fortunate man who cannot print from Linux, but I’m still a fortunate man.

Certainly part of the problem is my fault; I had upgraded from OpenSUSE 15.2 to 15.3. 15.3 was released two weeks ago; I upgraded about ten days ago. This was not enough bake time. I should have listened to my own advice: do less yeet and more tootle. But yeeted I had, so the story unfolds ….

Okay, so I logged in to the pharmacy web page, and used the browser to print. Got no printer noises, but no error about anything, either.

The print driver I’m using is from OpenPrinting.org and it did previously work. I did print something three weeks ago. But today, nada.

Go into the printer manager in OpenSUSE 15.3 and do a test print. No printer noise, but no error alert either. It asks if the print worked; (no) so says do journalctl to see what went wrong.

I don’t like journalctl. It spits at me about permissions, and I used to just be able to just grep a log file – any log file – and search for terms like “error” or “warn” or “cups”. I just want to print, man.

Okay, dig in and find that there is an error with the driver. Reinstall the driver. The driver will not install.

The driver is dependent on LSB. LSB = Linux Standard Base, which was the idea that the various packagers of Linux would all agree on what should be in a base install of Linux (that supports LSB). Software vendors could count on the base packages being there, or worst case say, “this software needs LSB, please install it”.

I had previously installed LSB (to get the printer to work), but now it’s missing. That must have happened during the 15.2 to 15.3 upgrade.

Okay, no big deal: zypper in lsb

Problem: nothing provides ‘/usr/bin/pidof’ needed by the to be installed lsb-foo

Well that’s darling. It’s a bug, and it is fixed in OpenSUSE Factory. I just want to print, man, and it’s now 20 minutes before work.

Okay, go to the fallback position: print to PDF, copy the file to a Winders box, and print from there.

I have Nextcloud client installed and running on most of my machines. Copy the file to my Nextcloud folder. Go to a Windows machine – there are no new files in the Nextcloud folder. Machine is acting wonky anyway, so I reboot (yeet!)

  1. Microsoft decided I needed a Weather widget in my taskbar, so they inserted one without asking. I need to lose some time praying to remove the murderous rage I have toward Microsoft for being so un-invitingly forcefully helpful.
  2. Nextcloud client has an update, would I like to install? Yes, please. What was that about less yeet and more tootle?
  3. Nextcloud client version 3.2.2 is no longer compatible with your older Nextcloud server. Have a nice day!
  4. It is now 10 minutes before work. I just want to print, man; my travel time to the doctor did not pad with lead time for print fixing, and as the famous mage once said: “Outlook not so good”.

Okay, what about the web page version of the Nextcloud server? Right, dang, I forgot I was going to need my physical 2nd Factor authentication key. Back to the living room to get it.

Logged in on the Windows box. The file is not there.

Dang it! The Nextcloud client on my Linux box has the same version problem. Back to the living room, open up the Nextcloud files web portal, do the physical physical 2nd Factor authentication thing and copy the file up. Back to the home office, open the PDF in the Nextcloud files web portal in Firefox and hit the print button. Finally, noise from the printer in the living room.

Put on a shirt and my shorts, get an energy drink out of the refrigerator, and I’ve got 30 seconds to spare.

“One does not simply press print”

Next week, I’m going to install a firewall router!