Richard Vrijhof's blog and stuff

I'm pleased to announce my server is running without freezes again!

I finally took the decision of buying a SATA controller and swapping out the SAS controller. I've also started the Samba daemon again. That was one and a half weeks ago. Since then no freezes. I'm officially declaring my server to be stable again!

My guess the SAS controller (or one or more of the cables, but I think the chance of that being the culprit is lower) was causing the problems was apparently correct. In hindsight, buying the thing was a costly mistake in the first place: close to five times more expensive than the SATA controller and vague errors because the SATA disks are communicating with a SAS controller (so a different protocol). Yes, the SAS controller can accomodate 8 disks and the SATA controller only 4, but I still have two ports free, partly using the SATA ports on the motherboard, so no problem there. I've learned an important lesson and will never buy a SAS controller again when using SATA disks!

I've checked for bad blocks on all disks that are part of the RAID array connected to my SAS controller: none found, which in itself is a good thing, but still doesn't provide an explanation about the server freezes. As it's a RAID 5 array, I could only take out one disk, check it for bad blocks, re-add it to the array, wait for the disk to have re-synced with the array, before I could go ahead with the next disk. That all takes quite some time.

There's also the possibility to check for bad blocks with a read/write test, instead of just read-only. This test takes a whole lot longer to complete, but should I have done it anyway? What's the advantage of this test over the read-only test? Does it find bad blocks that would otherwise not be found?

I'm running with the Samba daemon stopped for a few weeks now. No freezes anymore, but making do without Samba is a real pain, as this means there's no NAS on the LAN anymore. That's why checking for bad blocks took over two weeks to complete, because I was copying a folder of more than 2 TiBs over SFTP to my desktop pc, which makes the re-sync on the RAID array very, very slow. As it does seem to be the culprit, but I still refuse to believe the freezes are caused by software but instead by hardware, there are only two possibilities left in my opinion: the SAS controller or the CPU, with the SAS controller being the most likely candidate. Sigh, decisions, decisions…

I've switched to using Let's Encrypt for my certificates and am now also using SNI, so I don't have to use several ports for the different websites over SSL (sorry, Internet Exploder on Windows XP users...)! It looks like it's working just fine.

Thanks to StartSSL for their free certificates over the years, but Let's Encrypt issues free certificates too and takes the pain out of renewing, so that's why I've switched.

I've run some CPU stress tests for a few hours, but no errors or freezes...

Stopping Samba didn't work, as yesterday evening my server again froze, but at least I didn't see any kernel errors about stuck processes. I've been talking about the COM port stopping, but I have to rephrase that: the UPS software reported it lost communications with the UPS. Maybe just stopping and starting the UPS software could solve that; I'll have to find out when that issue pops up again. Anyway, of course it's still weird, because the UPS software never reported having lost communications before in the last year or so this server is running.

Maybe the SAS controller makes my server hang, by slurping up all I/O or something?

I really have to think about how to proceed further. Until the next update!

Unfortunately exchanging the motherboard hasn't changed anything. The server is still unstable, though I must say, it hasn't hung, the only problems were the COM port that stopped working and kernel messages about a stuck process on one of the CPU cores.

I've run a memtest too. All RAM working without errors.

In syslog I see the stuck process is always smbd (the Samba daemon). For now, I've stopped Samba, but that still doesn't answer the question why the COM port stops working and what is the cause of the stuck process (hardware, like the SAS controller or one of the SATA hard disks, or software, which had been updated several days before the problems began, so it is strange the server was stable for several days before the problems started). For now, I'm a bit at a loss and don't really know how to proceed further.

Maybe the CPU is the culprit? I will probably have to run some stress tests to try to find that out.

To be continued, that's for sure.

As you may have noticed (but probably not), my server is really very, very unstable and seemingly randomly hangs. The problems started about last Sunday evening.

At first I thought an update was causing the problems (although I've never seen a Linux server completely hanging because of user space software), but that doesn't seem to be the case. Looking at statistical data about my system like CPU, memory, disk, network usage and temperature sensor information neither revealed any clues. Finally, when yesterday the serial port (that is being used to communicate with an old, but just fine APC UPS) just stopped working (but after restarting the server started working again) and also seeing strange kernel errors in syslog, it hit me it must be hardware issues and in this case most probably the motherboard.

Sending the motherboard back for repairs and then having to wait for probably weeks isn't an option, so today I ordered a new motherboard (the same model, so hopefully this instability issue after less than a year isn't a symptom of structurally using sub-par components) and this motherboard is expected to arrive in a few days' time. It will probably be next week the server is running stably again, I hope, but for now, it's sadly a question of keeping an eye on the server and switching it off and on again (just like Roy from The IT Crowd) when it hangs.

When you want to download mutations from your banking website, sometimes the only (usable) format of the downloaded file is MT940, as in my case (I have a bank account with the ABN-AMRO bank). I use the term "usable" when talking about wanting to import that file into GnuCash, because the other choice is downloading it in TSV format (Tab Separated Values), which means every freaking time when importing, you have to tell GnuCash what every column is about (or I'm too stupid to work with GnuCash, which I don't rule out entirely).

GnuCash doesn't support MT940 by itself, but you can convert it to OFX format by means of a nice little Java program made by Pieter van Harmelen (on SourceForge at http://sf.net/projects/mtconverter/), but it's a console application and I'm lazy... Thus I wrote a small wrapper in C#, so you can click your way to an OFX file.

It makes the following assumptions:

• Java is installed and java.exe is in your path
• mtconverter.jar is in C:\Windows (yes, hard-coded and with the name "mtconverter.jar")

Furthermore, it needs .NET 4.5 to be installed.

You can find the executable and Visual Studio 2012 C# project on my download page, but here are the direct links too:

• ConvertMTtoOFX.zip for the executable
• ConvertMTtoOFX - Visual Studio 2012 C# project.zip for the Visual Studio 2012 C# project

Enjoy!

The challenge

My keyboard of choice at home is the Microsoft Natural Ergonomic Keyboard 4000 and it's a great keyboard for touch typists. However, I did have some problems using the special keys to control MediaMonkey, which is my media player of choice for music. Here I'll describe how I tackled that by making tiny executables for this purpose by means of Visual C# (Visual Studio Express 2012 for Desktop). I realize there are many ways to tackle this challenge and this is just one of them, but it's a way I found performs so well that I decided to dedicate a blog entry to it.

Of course I realize you don't need a Microsoft Natural Ergonomic Keyboard 4000 for this to work or even a Microsoft keyboard, as long as it is a keyboard with some extra keys (preferably media keys) and you can use some software to configure the keys to run an executable when pressed. In that case, the steps necessary to accomplish the same goal will of course differ somewhat.

First try: PowerShell

MediaMonkey makes it easy to be controlled from the outside by exposing a COM object called SongsDB. Initially I used PowerShell for MediaMonkey control, but it had the following disadvantages:

• Slow to start up: it took up to several seconds until MediaMonkey reacted to the press of a key
• It got in the way: because running PowerShell involved opening a command prompt, a command prompt window would pop up briefly. Using the start command was no option, because it is integrated into cmd.exe (there is no start.exe), and because PowerShell insists on running an executable instead of only a batch command like start (which of course makes sense, because start needs cmd.exe to even work).
• Crash and burn: if MediaMonkey wasn't already running when pressing one of the keys to control MediaMonkey, because its COM object was referenced, MediaMonkey would start automatically and then crash immediately after starting.

In the settings of the Microsoft Mouse and Keyboard Center, I had configured the following special keys to run the following commands:

• the Play / Pause key (open a program, webpage, or file):
  Select code

  c:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -WindowStyle Hidden -Command "&{$MM = New-Object -com SongsDB.SDBApplication; if (! $MM.Player.isPlaying) { $MM.Player.Play() } else { $MM.Player.Pause() }}"
• the Browser Forward key (open a program, webpage, or file):
  Select code

  c:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -WindowStyle Hidden -Command "&{$MM = New-Object -com SongsDB.SDBApplication; if ($MM.Player.isPlaying) { $MM.Player.Next() }}"
• the Browser Back key (open a program, webpage, or file):
  Select code

  c:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -WindowStyle Hidden -Command "&{$MM = New-Object -com SongsDB.SDBApplication; if ($MM.Player.isPlaying) { $MM.Player.Previous() }}"

The better way: running a tiny executable

What I finally settled on was making an executable program by means of Visual C# (in Visual Studio Express 2012 for Desktop). The resulting executables are tiny (about 6 KiB), so they start up fast and the steps in Visual Studio and the C# code necessary to accomplish this, are not too bad either. Because I started a new project configured as a console application, which I later changed to Windows application (but without any code to generate a window), the executable conveniently runs on the background without any visible window at all.

I'll show here the steps necessary to generate the executables by starting a new project for every executable, each for a key used to control MediaMonkey.

1. Start a new project in Visual Studio:
   
2. Make sure it's a Console Application, give it a name something like MediaMonkeyKeyAction (and give the solution the same name; I've named mine MediaMonkeyPlayPause) while also making sure Create directory for solution is checked:
   
3. Now, in the Solution Explorer pane on the right, right-click on the project and choose Properties:
   
4. Change the output type to Windows Application (so no window will pop up):
   
5. In the Solution Explorer pane on the right, right-click on the solution and choose Properties:
   
6. Choose Configuration Properties on the left and change the Configuration column to Release:
   
7. Again, in the Solution Explorer on the right, right-click on References (expanding the project if necessary) and choose Add Reference...:
   
8. Expand COM on the left, choose Type Libraries, tick the checkbox to the left of MediaMonkey Library and click on the OK button (if for whatever reason MediaMonkey Library is not in the COM type libraries list, just choose Browse on the left, click the Browse... button, browse to the directory where you've installed MediaMonkey, select MediaMonkey.exe, click the Add button, make sure the checkbox to the left of MediaMonkey.exe is ticked and click the OK button):
   
9. Type or paste the following code in the Program.cs editor window:
   Select code

   using Microsoft.Win32; using SongsDB; using System; using System.Diagnostics; namespace MediaMonkeyPlayPause { class Program { static void Main(string[] args) { string mmPath = (string) Registry.GetValue( @"HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Media\MediaMonkey\" + @"shell\open\command", null, false ); Process[] mmArr = Process.GetProcessesByName("MediaMonkey"); if (mmArr.Length < 1) { Process mm = Process.Start(@mmPath); mm.WaitForInputIdle(); } SDBPlayer player = new SDBPlayer(); if (! player.isPlaying) { player.Play(); } else { if (! player.isPaused) { player.Pause(); } else { player.Play(); } } } } }
10. Now, choose Build Solution in the Build menu (or press F7):
    
11. The Build Output pane on the bottom should show something like this (if not, you've probably done something wrong):
    
12. Repeat the above steps for MediaMonkeyPlayNext, only this time type/paste the following code in the Program.cs editor window:
    Select code

    using Microsoft.Win32; using SongsDB; using System; using System.Diagnostics; namespace MediaMonkeyPlayNext { class Program { static void Main(string[] args) { string mmPath = (string) Registry.GetValue( @"HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Media\MediaMonkey\" + @"shell\open\command", null, false ); Process[] mmArr = Process.GetProcessesByName("MediaMonkey"); if (mmArr.Length < 1) { Process mm = Process.Start(@mmPath); mm.WaitForInputIdle(); } SDBPlayer player = new SDBPlayer(); if (player.isPlaying && ! player.isPaused) { player.Next(); } else { player.Next(); player.Play(); } } } }
13. Then repeat the above steps for MediaMonkeyPlayPrevious, only this time type/paste the following code in the Program.cs editor window:
    Select code

    using Microsoft.Win32; using SongsDB; using System; using System.Diagnostics; namespace MediaMonkeyPlayPrevious { class Program { static void Main(string[] args) { string mmPath = (string) Registry.GetValue( @"HKEY_LOCAL_MACHINE\SOFTWARE\Clients\Media\MediaMonkey\" + @"shell\open\command", null, false ); Process[] mmArr = Process.GetProcessesByName("MediaMonkey"); if (mmArr.Length < 1) { Process mm = Process.Start(@mmPath); mm.WaitForInputIdle(); } SDBPlayer player = new SDBPlayer(); if (player.isPlaying && ! player.isPaused) { player.Previous(); } else { player.Previous(); player.Play(); } } } }
14. You may have noticed that the resulting executables even start MediaMonkey for you if it isn't running when you press one of the MediaMonkey control keys!
15. Now, copy the resulting executables to a directory of your choice (I've just made a new directory called C:\Program Files (x86)\MediaMonkeyControl), make sure the Play / Pause, Browser Forward and Browser Back keys are reconfigured to run these executables and presto, you're done!
16. Finally, don't forget to have fun and think of me from time to time while feeling a warm, fuzzy feeling in your heart!

Check it out! I have made an e-books page for the e-books I've written (one until now), where you can view information about them and download them. They are (and will be) in the EPUB format and don't have any DRM whatsoever (because I hate DRM)! I hope you'll like them.

Also have a computer with both Linux and Windows 7 (or XP or Vista) in dual-boot configuration and having the problem that every time you start Windows, your clock is on UTC time? I did, at least, and the problem was, that you could set an internet time server to sync with, but Windows would not sync at startup, only once a week. Here is how I solved it.

Make a new task in the Task Scheduler, running under the SYSTEM account, triggered to run at system startup, starting the program "w32tm /resync" and with the further condition that the network connection should be up-and-running (I think this last possibility is missing from XP, so here the task will probably fail because it is too early). Voilà, finally solved!

Apparently something was wrong with downloading some files from the download page, as they were compressed (again) by the web server, which is a waste of time and effort (as I'm talking here about the compressed files) and looked like it was responsible for corrupt downloads in some cases. That should have been fixed now. Sorry for any inconvenience.

I've received a paper that says there will be a power outage on October 30, between 08:00h and 16:00h (CEST, so +0100), for at least five hours, because of maintenance to the grid. That means my web and mail server will be offline during that period. When power comes online again, the server should power up again automatically and be online some minutes later.

I don't have the illusion that a lot of all my loyal readers (all hundred thousand of them) are now thinking: "Oh, no! My god! I have to miss Richy's website for a whole day! I just cannot bear the thought! Better go kill myself!", though...

But, at least, now you know.

 

It looks like the connection problems are over (at least mine are). Finally! I've never experienced an outage that lasted this long. Hopefully now we've seen the end of it.

The whole country is suffering from a huge outage of internet connections, and I'm one of them. It started yesterday morning and it looked like it was solved this morning. During the day I could connect to my server without any problems, but coming home, it took a turn for the worse. I don't know how long this is going to last, but it surely is nasty!

My server is now running on a Pentium 4 instead of a Pentium 3, with 80-pin ATA instead of 40-pin ATA, and I can really notice the difference! I was first worried about how much extra energy this server would use, but after connecting it to my UPS (with the three hard disks and the DVD writer of course), it looks like it is only a few percent more when not very busy and maybe ten percent more when really busy (like backing up, or remirroring).

Pity for my girlfriend; she'll have to make do with her Pentium 3 for now...