Coretutils 8.25+ adds single quotes to filenames containing spaces  when listed with ls

ls -al
-rw-r–r– 1 root root 0 May 14 11:41 ‘file with spaces’

When you first see this you may think “WTF has happened to my files” – stay frosty :

• It was introduced in coreutils v8.25, and alignment improved in v8.26
• It only happens when outputting to terminals so doesn’t break scripts
• It disambiguates the output for users for files containing whitespace
• It sanitizes output so it is safe to copy and paste
• Output is now always valid to copy and paste back to shell
• you can get back to the old format by adding -N to the ls alias or set export QUOTING_STYLE=literal in your .bashrc

And relax – all is well…

Taken from https://unix.stackexchange.com/questions/258679/why-is-ls-suddenly-wrapping-items-with-spaces-in-single-quotes

php mail() returns FALSE

My Ubuntu mail setup uses sendmail with exim4. After the upgrade I was occasionally seeing the following exim4 error

2014-10-29 08:00:50 unable to set gid=33 or uid=0 (euid=0): forcing real = effective

Normal email was working fine and php mail from the command line, or cron jobs was also working fine. It was only email sent via web sites i.e. with online forms using the php mail command i.e.

$mail=\mail($email, $subject, $message);

$mail would always return FALSE, and an exim4 error would be logged.

The exim error points to a permissions problem with the Apache user group – www-data in Ubuntu. Group ID 33 is www-data.

After googling around for a while and looking at various configuration options in php.ini the resolution lay with the Apache ITK MPM module.

It seems that a default configuration in this module with Apache 2.7 has some kind of influence on the permissions associated with the www-data user/group when communicating with exim4.

The fix is to explicitly define the ITK MPM module LimitUIDRange parameter in the /etc/apache2/mods-available/mpm_prefork.conf module configuration to something like

LimitUIDRange 0 2000

Restart Apache and Exim 4 and PHP mail via Apache works again.

My main production server was running Ubuntu 12.04 LTS with PHP 5.3, and although the 14.04 LTS had been released it wasn’t available as a release upgrade until the 14.04.01 release was out, so I was waiting for that before I would even think about upgrading.

So the release upgrade to Ubuntu 14.04.01 LTS has been available for a while and I was putting off the upgrade until I had some time to test it on a development server which I did and everything seemed to go ok. I did however make a major mistake in that I upgraded my dev vmware server and assumed that just because the upgrade worked, everything else worked and went ahead with my production server upgrade.

do-release-upgrade

I love Ubuntu and have been using it since Hardy Heron (8.04) so far my experiences with upgrading Ubuntu Server have been pretty good, as long as I stick with the LTS versions, minor and major upgrades haven’t caused me any major heartaches.

Simply run do-release-upgrade from the console, make a coffee and put your feet up.

UBUNTU 14.04.01 LTS Trusty THAR

As I mentioned my main goal of upgrading was to upgrade PHP, Ubuntu 14.04.01 upgrades my most crucial apps to:

APACHE 2.4.7 Errors

NameVirtualHost

As the upgrade finishes one of the last packages to install in Apache and you will quite possible see that apache fails to start after the upgrade.

This is quite likely to be due to older configurations lurking in your Apache config files, for me there was an error in ports.conf where I had an old virtual hosts statement : NameVirtualHost *:443 which is no longer supported.

I removed the NameVirtualHost statements (ports.conf just requires a Listen statement).

After editing ports.conf apache started ok.

Forbidden You don’t have permission to access / on this server. Apache/2.4.7 (Ubuntu) Server at localhost Port 80

I eagerly tried to load a website on the server and got the above error, in fact all my websites were down.

It took a while to track down the causes of this error but the main cause of the problem I think is that my apache2.conf file was overwritten by the upgrade even though I am sure I was asked to keep my original files by the installer.

My default WWW folder is not in the default /var/www and this was the cause of the first problem. Edit apache.conf and add the directory configuration for your default WWW parent folder. Reload apache and…

No websites work.

APACHE 2.4.7 UPGRADE RENAME SITES-AVAILABLE CONFIG FILES to .CONF

Apache 2.4.7 requires all apache site config files to have a .conf extension. As the release notes nicely point out all files without a .conf extension are silently ignored.

I am not sure why my site config files never had a .conf extension, I guess either I picked up a bad habit or thats just the way most people create them. Basically all my site config files were being ignored so I still had no websites.

To resolve this obviously rename the files so they have a .conf extension and then add the sites again with a2ensite config.conf and reload apache.

No websites work.

The final problem I had was that access control commands have changed in apache 2.4.7 instead of

Order allow,deny
Allow from all

You should use

Require all granted

Another thing that might also catch you out is that the options statement cannot be mixed with + or – indicators

Options -Indexes Multiviews FollowSymLinks

Will NOT work. Either all options must have a plus minus enabler or none should have one, so use Options -Indexes +Multiviews +FollowSymLinks  instead.

If you have a lot of sites enabled, be prepared for this to take a while as you rename and edit each site file, save it, re-add the site to apache and reload the apache config.

Eventually after going through all my site configs, my websites slowly came back online.

PHP 5.5 ERRORS

Unfortunately that wasn’t the end of my troubles, two main issues came out of the PHP upgrade.

PHP.INI overwritten

Like the Apache conf my php.ini files were also (I think) overwritten and I am not entirely sure why. The main issue from this for me was that my error settings had changed so I was now seeing STRICT and DEPRECATED errors that I was safely ignoring before.

My PHP cron jobs were also failing due to a similar configuration error in the PHP-CLI ini file.

MCRYPT DISABLED

Another problem I noted was that php-mcrypt was not enabled and I was seeing fatal error Call to undefined function mcrypt_decrypt() – even though  php5-mcrypt was installed.

To resolve this problem (which is probably also associated with the config problems) re-enable the mcrypt module with php5enmod mcrypt and restart Apache.

HTTP custom headers

Finally another error related to Apache is that custom HTTP headers cannot contain the underscore character any more. Apache will silently ignore them and they will not appear in the PHP SERVER array.

This was affecting all my XHR javascript file uploads due to a filename being set with a custom header using underscores:

xhr.setRequestHeader(“X_FILENAME”, file.name);

Fortunately the work around is to simply replace the underscore with a minus

xhr.setRequestHeader(“X-FILENAME”, file.name);

Interestingly on the server side you still see the header as HTTP_X_FILENAME so your server side PHP code does not need to change, but any client side code creating custom headers cannot use underscores in the header name.

LOG ROTATE ERRORS

The next day, as system tasks started running I saw this error

error: skipping “/var/log/mysql.log” because parent directory has insecure permissions

This is resolved by editing /etc/logrotate.conf and adding the following to the configuration

# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root syslog

Lessons Learned

What can we learn from this, firstly always, always, always have backups. My whole system was backed up so simply restoring back to 12.04 LTS was a last result option.

Don’t try to fix it if it isn’t broken. Did my server really need to be upgraded? Yes I think so, although 12.04 is supported until 2017 I needed PHP 5.4+ and I Would have been faced with these upgrade issues at some point anyway.

Read the release notes. Yeah right. I guess I could have read the Apache release notes but no one really does that in the real world.

Better testing. I thought I had tested the upgrade but my dev server did not properly mirror my production server so I was unable to test everything.

Fresh install instead of upgrade. I would never consider doing this on an older LTS. Build a new server with a fresh install of LTS and migrate your production sites and software over to it.

OTHER TESTS

Magento 1.9.x is running without issues after the upgrade

WordPress is running without issues

SMF is running without issues

MYSQL (apart from the fact that it is deprecated) is running without issues

memcached is running without issues

dropbox is running without issues

exim4 and spam assasin (mail) is running without issues

shoutcast and darwin media server are running without issues