Magento Product Attribute Search and Filter

Magento Product Attribute Advanced Search and FilterThe Magento product database is pretty flexible when it comes to extracting product information. Sometimes however, extracting all the information you want for a product is not so easy.

For example you want to extract the main product info, price, title, SKU etc and all the product attributes names and the product category. This might require multiple product / collection calls and when you multiply this by a search across 1000’s of products can become slow.

Magento Product Attribute Search and Filter

I wanted to provide a quick and easy way to search and filter magento products by attributes, in this case by colour, so customers could quickly narrow product selection down by selecting a colour and a filtering by category.

To extract the data I developed my Magento Product Data and Attribute exporter. This let me initially see how many attributes I had to work with, in this case how many different product colours were already configured.

To make colour selection simpler I created a new Magento attribute product_primary_colour and added a primary colour to each product so that I ended up with about 16 main colours that would provide a match to all the existing product attribute colours.

Once I had all my exported product data in a tab delimited text file I quickly realised that it would be relatively simple to search and filter this data via ajax requests.

I had initially thought it would be more effective to do the search within Magento using product collections but the speed of the search against the delimited txt data file made me think again. When I then added cached search results using Memcache I was happy that this solution would work.

The benefits of using the flat txt file are that it is quick and can run completely external from Magento in another webapp or in WordPress or Facebook.

The Magento attribute search and filter is easily customiseable and can search the data created by the export script filtering results from any of the product data columns in the export file. This could be colour, size, type etc, there are no limits.

If your product data is changing regularly simply export the data regularly via a cron job to keep it up to date.

if you need to filter your data by a completely new attribute simply pull the export data into Excel, create your new attribute data and corresponding Magento attribute and import the data back into Magento. Now you can export this data and easily search and filter it.

Here is an example of the attribute search running in an iframe using the Magento test product data. Here I have exported the color attribute, and am filtering against existing product categories like Womens, Mens and Shirts (note Mens matches Womens, some refinement needed there!). This will also easily run in Magento when setup in a static block containing the iframe data.

You can see how quickly the data is filtered and returned, try a search for Red to see some data. Click around the categories and check the debug footer which shows if the results were cached or not and how long the ajax query took to run.

Data from Magento DEV Store running Magento 1.9.x

Magento Product Data and Attribute Exporter

Export Magento Product and Attribute DataOver the last few years I have developed a few different PHP scripts to export Magento product data in order to manipulate product pricing, generate Magento  EAN barcodes or create product import data.

Recently I had to export Magento product data to be used as import data for Amazon product listings. Specifically the products had a lot of different attributes associated with them and I wanted to export each product with all of its associated attribute data to map to the relevant Amazon product data.

To accomplish this I returned to my previous code and enhanced it to be able to export user defined Magento product and attribute data to a delimited text file (as well as creating product EAN barcodes which was a function of the original script.)

The script can extract the usual global product data e.g. price, description etc. as well as any custom product attributes.

The export data is configured in an array;

‘name’ => true,
‘description’ => false,
‘imageurl’ => true,
‘producturl’ => true,
‘type’ => false,
‘ean’ => false,
‘category’ => true

We set the array key value to true or false for the product data we want to export. The same applies to attribute data:

‘colour1′ => ‘color’,
‘custom’ => ‘my_custom_attribute’

So here we are exporting the color product attribute as well as a custom attribute.

The tab delimited data is easily imported into Excel for manipulation and further import/export, for example adaption to an Excel format compatible with Amazon product imports.

This data can also become really useful if you want to provide fast external access to your product info for example, in an external app, on Facebook or even within Magento. See my product attribute search app for more information.

You can download the source code for this export script from my Github.

To run the script copy the files to your server and edit the configuration php file config/applicationConfig.php. Configure the paths to your Magento installation and the folder to export to.

Configure the product arrays by editing php/Application.php, edit $_getProductData and $_getProductAttributes.

To run the script execute it with php export.php debug from the command line. To run silently from a cron job leave the debug switch out.

You can also run the script from your web browser if you install it in a web accessible folder.

Here is an example of the output from my Magento dev store ::

id	sku	name	imageurl	producturl	category	colour1
51	1111	Ottoman	Living Room	Red
52	1112	Chair	Living Room	Red
53	1113	Couch	Living Room	Red
83	cn	CN Clogs Beach/Garden Clog	Mens	xxx
29	cn_3	CN Clogs Beach/Garden Clog	Mens	Blue
84	cn_4	CN Clogs Beach/Garden Clog	Mens	Blue
85	cn_5	CN Clogs Beach/Garden Clog	Mens	Blue


Facebook API v2.0 Ubuntu PHP 5.4 #fail

Facebook #fail, I do not like thisSo I finally caught on to the fact that Facebook released version 2.0 of their API at the end of April (to coincide with the F8 developer conference.) and will discontinue pre v2.0 API calls in April 2015.

I thought I had better take a look as I support a number of PHP apps that run on Facebook or integrate with Facebook for login/authentication etc.

v2.0 of the Facebook API has been completely rewritten from the fairly basic previous API code that I had been using, there are a lot of changes including new login features and permissions.

‘This is great’, I thought…

Facebook PHP API v2.0 requires PHP v5.4

However there is a problem that will cause a whole lot of people a whole lot of problems. The new PHP SDK for v2.0 of the API requires PHP v5.4.

if (version_compare(PHP_VERSION, '5.4.0', '<')) {
 throw new Exception('The Facebook SDK v4 requires PHP version 5.4 or higher.');

My main production server runs the latest (as of the time of writing) LTS of Ubuntu v12.04, it’s up to date but a quick php -v from the command line reveals

PHP 5.3.10-1ubuntu3.13

The 14.04 LTS is not available to apt until the first point release which I think is any day now. 14.04 LTS will include PHP 5.5, but I’m not really planning on upgrading my live server straight away, 12.04 LTS is supported up to 2018 – that’s what Long Term Support means.

I currently work with live production Ubuntu servers ranging from v8 to 12 that I would not even dream of trying to upgrade.

Bearing in mind that a whole lot of people won’t even be running an Ubuntu version above 12 on their production servers getting to PHP 5.4 is going to present them with a considerable upgrade challenge.

You are going to have to be pretty brave to manually upgrade PHP, and if you do it without proper testing then you are also going to have to be particularly stupid.

Upgrading PHP without testing is going to break applications that are running PHP code that has been deprecated, older versions of Magento spring to mind, not to mention the problem people who are on a virtual or shared hosted systems where they have absolutely no control of the version of PHP running on their server.

Facebook #fail

So it looks like it’s a real #fail from Facebook to implement a PHP SDK totally dependent on a PHP version that a lot of people are not running and possibly cannot run.

What does this mean for your existing Facebook apps?

  • For apps that existed before April 30th 2014, making an API call without specifying a version number (‘unversioned’) is equivalent to making a call to the v1.0 of the API.
  • For apps created on or after April 30th 2014, making an API call without a specifing a version number is equivalent to making a call to v2.0 of the API.
  • Apps that were inactive or have a creation date on or after April 30th, 2014 will not be able to make calls to v1.0 of the API. They must use v2.0.

This means your existing pre April 30th 2014 apps will be allowed to still make pre v2.0 api calls until April 30th 2015 after that they will stop working.

  • If you want to create a new Facebook App you have to use the v2.0 API.
  • Inactive apps will no longer work. (define inactive please Facebook.)
  • if you want to create a new Facebook App you need to upgrade to PHP 5.4 or for Ubuntu admins be running the latest LTS (14.04).
  • if you have code on Envato using the old API it will be removed in August 2014 – goodbye to my Magento Facebook storefront!

Man this sucks

I think this really sucks, especially if you are authenticating users via Facebook login if you don’t do anything your users will not be able to login to your website after April 30th 2015. I am kinda glad I didn’t implement that Facebook login feature on our eCommerce site now…

if your eCommerce site is running on shared hosting, or does not support PHP5.4 (i.e. Magento 1.3x) then you are pretty well stuffed I think.

The new API has a learning curve for devs too and I am already coming to this pretty late so I guess I better look at what options I have and what apps are going to stop working next year if I do nothing, or am unable to get my code and server to PHP 5.4 by then.

Having said all this I would bet my bottom dollar that Facebook change these deadlines and extends the v1.0 API support beyond 2015 – watch this space…

Because I am now depressed I am going to include a funny picture to cheer me up.








How to move the Dropbox cache folder

DropboxHow to move the Dropbox cache folder

Dropbox is extremely useful, it can be used in many different ways to allow distributed sharing of files between various systems, Windows, Mac, Unix, IOS devices. The list of ways to use Dropbox are endless.

I use one dropbox account to backup SQL backups from various systems on a Windows 2003 server however I noticed today the disk dropbox was installed on had ran out of disk space and the problem was the Dropbox cache folder.

Dropbox Cache Folder too large?

The Dropbox cache folder is a hidden folder within the Dropbox folder which caches older versions of Dropbox files. The cache contents are cleaned out every few days by Dropbox but the cache files still seem to have the ability to grow very large, very quickly – my cache folder was 20GB, which on a relatively older 250GB disk was a large percentage of the free disk space.

Whilst this may not be a problem on some desktop PC’s or Laptops if you have Dropbox installed on a server with limited disk capacity you might find yourself running low on disk space. Even worse, if you installed Dropbox on a relatively small system partition, your system will grind to a halt.

You can manually delete the cache folders, but you will see that they very quickly reappear and start to grow in size.

Move Dropbox cache folder to an external drive

The solution is to move the Dropbox cache folder (NOT the data folder) to an external drive.

Unix users will be familiar with symbolic links which allow you to link a virtual file or folder to a target file or folder. This is possible on newer versions of Windows with the Mklink command, and on older versions of Windows Server with the Junction command.

To move my Dropbox cache folder on Windows 2003 to an external usb drive, I used junction with the following command:

junction “d:\DATA\Dropbox\.dropbox.cache” “g:\EXTERNAL\dropbox-cache”

with Mklink on Windows 7/8 the command would be

mklink /d /h  “d:\DATA\Dropbox\.dropbox.cache” “g:\EXTERNAL\dropbox-cache”

The external drive is large so the Dropbox cache folder can grow there without any problems.



Upgrade Magento 1.8 to 1.9

Magento Upgrade

Magento CE 1.9 has been released. It appears to be a relatively small update introducing a new responsive GUI. See the release notes here.

Here are my notes after upgrading my dev store from 1.8.x to 1.9 using the command line.

Backup existing install folder and database

1. Magento source
tar -cvvf /home/backup/magento.tar /home/www/magentoinstallfolder/

2. Magento DB
/usr/bin/mysqldump -h localhost -u USER -pPASS magento | gzip > ~/backup/folder/db-magento.sql.gz

1. Remove cached files and temp files

rm -rf downloader/pearlib/cache/*
rm -rf downloader/pearlib/download/*
rm -rf downloader/var/cache/*
rm -rf downloader/var/report/*
rm -rf downloader/var/tmp/*
rm -rf media/catalog/product/cache/*
rm -rf media/tmp/*
rm -rf var/cache/*
rm -rf var/session/*

2. Restart memcached (if being used)

3. Sync mage

./mage mage-setup

./mage sync

4, Run upgrade

./mage upgrade-all

5. Reset ownership and permissions on all files and folders.

Check Magento is operational, login and refresh caches.

If all is well take a well deserved break and maybe eat a cream cake. If all is not well restore from backup…



Google Universal Analytics Update for Magento Version v.1.3.X Old Versions


Google Universal Analytics went out of BETA recently (May 2014) and is now available for all users. Universal Analytics has “more features, better insights”.

All versions of Magento include the client side analytics code which can be enabled in Magento Admin. When enabled Magento creates the client side html and javascript and appends it to the footer section of each page.

On the checkout completion page additional code is created to track sales.

Google Univeral Analytics Magento

Google Universal Analytics uses new client side javascript code, this means Magento does not yet support Google Universal Analytics by default, however it is relatively easy (if you are using new versions of Magento) to override the base/default Magento code that creates the client side analytics html and javascript.

If you simply turn off the built in Magento analytics feature and add the new Google Universal Analytics code to your page templates (as Google suggests you do) then you will lose all eCommerce tracking.

You can update your Magento core code to support Google Universal Analytics using the information below.

New Versions > 1.4.x

If you are using a newer version of Magento i.e. > 1.4.1.x you can update the core analytics functionality following the guidelines here.

Old Versions < 1.4.x

if you are using an older version of Magento (before 1.4.1) i.e. Magento 1.3.x, I have adapted the Magento v1.3.x google analytics core module code to support Google Universal Analytics. The code can be installed as a local module and will override the core module.

Download the code from my GitHub

To install the module copy the files to your Magento app folder, refresh your Magento configuration and configure the settings under Configuration -> Google API -> Google Analytics.

To view the analytics code look at the source code of any page, scroll to the bottom and look for the code between the Google Universal Analytics Code comment tags. For order tracking information, check the code on the Checkout/Success page.

Use GEOIP2 for PHP Geo IP data in WordPress, Magento, HTML

The above iFrame makes an ajax request to a PHP class querying GeoLite2 data created by MaxMind, available from
This example displays the location of a single visitor i.e. YOU. Click here to see the logging data for this WordPress blog which demonstrates a PHP memcache data logging solution with geo ip visitor data displayed dynamically on a google map.

Magento SEPA Migration Check List

The Single Euro Payments Area (SEPA) is a payment-integration initiative of the European Union for simplification of bank transfers denominated in euro.

If you make or accept payments via direct bank transfers then you will know all about SEPA. As a customer online payment method direct bank transfers are popular in some countries (i.e. Germany) and not so popular in others (i.e the UK).

The deadline for SEPA payments was 1st February 2014 but (as of the time of writing) has been extended 6 months to 1st August 2014.

SEPA standardises bank transfers in Europe, replacing bank specific account and sort code numbers with an IBAN (International Bank Account Number) and BIC (Business Identifier Code).

For most businesses the migration to SEPA involves communicating the new changes to customers and migrating bank account numbers and sort codes to IBAN and BIC.

For Magento stores if you have a payment method that allows customers to pay via a direct bank transfer then you need to ensure that you can capture IBAN and BIC numbers at checkout. Until 1st August 2014 you can decide to give the customer the option of entering either bank account number and sort code or IBAN and BIC. From August 1st you *should* only be accepting IBAN and BIC.

Here is what you need to do to get your Magento Store ready for SEPA payments

  • Make sure you are capturing bank account information in the SEPA format – IBAN and BIC at checkout.
  • Check that your validation rules work for the IBAN and BIC formats.
  • Make a note to remove input fields and validation rules for old account formats, bank account number and sort code.
  • Upgrade your existing debit payment methods to make sure they support SEPA.
  • Implement a validation system for IBAN and BIC

This debit module is perfect for German Magento shops implengint Direc Debit (Lastschrift) payment methods and supports SEPA :

To validate IBAN account numbers with PHP take a look at



Javascript only EU Law Cookie Notification Message for WordPress and Magento

You know sometimes you just want a quick solution to a problem and wherever you look there are just complex, overpriced scripts that make your simple problem sound like something that needs a lot of code and money to get to work.

I needed to display a cookie notification message on a wordpress and magento site, to avoid compatibility problems I wanted to avoid using jQuery and just do it as simply as possible in javascript.

After looking around at a lot of $10 jQuery solutions I found a free javascript only solution, modified it for Magento and WordPress and implemented it.

Download the code from Github here

Whatever your opinion of the EU Cookie Law which came into effect in 2012, sooner or later if you are in Europe you may need to show that you are doing something about it. Err actually it’s 2014, so I guess we are already a teeny weeny bit late.

From what I understand the law may differ in various countries (certainly outside Europe), so depending on where your website is, and who visits your site the law may or may not apply to you.

This script will let  you quickly implement a cookie notification banner with a link to your privacy policy. The notification itself uses cookies and will not be show again when closed.

You can implement this solution on any website by copying the install files to your web server and adding the following code to the footer of your site.

<script type=”text/javascript”>

var cookieNotificationURL='/cookies/';
var cookieNotificationLifetime=180;
var cookieNotificationMessage='This website uses cookies. For more information please ';
var cookieNotificationClickMessage='click here';
var cookieNotificationMessageFade=true;
var cookieNotificationPath='/path-to-install-folder/

<script type="text/javascript" src="/js/cookienotification/cookie.notification.min.js"></script>

This sets up some variables used by the script, the URL of the privacy info page, the lifetime of the cookie used by the script in days, messages displayed, the option to fade the message automatically and the path to the install folder – used to display the close image.

For wordpress simply paste this into the footer php file of your theme.

For Magento paste this into the footer.phtml template file of your Magento theme. You can translate the messages, and perhaps a good idea, choose to display the message only on non SSL pages – we don’t want this javascript to somehow screw up a sale!

<?php if (empty($_SERVER['HTTPS']) && $_SERVER['HTTPS'] = ‘off’): ?>

<!-- Cookie Notification -->
<script type="text/javascript">
var cookieNotificationURL='/datenschutzerklaerung/';
var cookieNotificationLifetime=180;

<?php if (Mage::app()->getStore()->getId()==2): ?>
var cookieNotificationMessage='This website uses cookies. For more information please ';
var cookieNotificationClickMessage='click here';
<?php else: ?>
var cookieNotificationMessage='Diese Website verwendet Cookies. ';
var cookieNotificationClickMessage='Klicken Sie hier für weitere Informationen.';
<?php endif ?>

<script type="text/javascript" src="/js/cookienotification/cookie.notification.js"></script>
<?php endif ?>

Here I am manually checking the store id and translating the text to German. Remember to refresh your Magento cache after editing and saving the footer template file.

You can see the code working on this WordPress blog below, and also with Magento on my Magento development site.

Tested with Magento 1.8.x, WordPress 3.blah, Chrome, Firefox, IE11. Detects mobile devices and modifies banner size accordingly.

IOS 7.x Jailbreak SSH Access / SSH Tunnel

There are 3 reasons why I always “Jailbreak” my iPad.

  1. The Apple iPad is a (bloody expensive) computer. When I buy a (bloody expensive) computer I expect to have 100% usability from it. The restrictions imposed by Apple on their i device operating system (IOS) inhibit it’s practical usability considerably, and I really don’t like that.
  2. When I am travelling I use the excellent MyWi app to turn my iPad into a wireless hotspot.
  3. For privacy and security I like to be able to tunnel my iPad traffic through a secure SSH connection to my server.

“Jailbreaking” exploits vulnerabilities in IOS to achieve root access to the operating system thus “liberating” IOS and allowing ad hoc software to be installed and executed. If you come from a Unix background this is great, because your (bloody expensive) computer now becomes really useful in accessing other Unix based systems and doing geeky Unix type stuff.

I have been Jailbreaking my iPad since IOS v.5 – I find little point in Jailbreaking my iPhone as the screen size limits use.

The Jailbreak process itself is always seamless:

  • Backup with iTunes
  • Install update from iTunes (not via OTA update)
  • Restore from iTunes
  • Jailbreak
  • Restore Cydia software and purchases, i.e. MyWi.

The IOS 7.x Jailbreak was released recently and I finally decided to upgrade my iPad from IOS 6.x to IOS 7.x and apply the Jailbreak, as usual this worked pretty well, but some changes in IOS 7 caused problems:

  • IOS 7 stops applications from connecting to localhost SSH on port 22
  • IOS 7 multi tasking affects SSH background connections

IOS 7 Stops Applications from Connecting to localhost SSH on Port 22

This was a devious software change by Apple. Normally after Jailbreaking the first task is to install Open SSH via Cydia – this gives you normal SSH terminal access to the device, and then use a terminal application such as iSSH to login to the device (localhost) as root.

With IOS 7 Apple have hardcoded a restriction into the operating system that stops (App Store) Apps from making an SSH connection to localhost on the default SSH port 22. When you try and connect with iSSH you will get a connection cancelled error. You can still SSH from an external device, but not locally.

The workaround for this is to change the listening TCP/UDP ports used by the SSH daemon to something other than 22.

To do this you need to edit a couple of system files. An easy way to edit the files is with the Cydia app iFile.

Take a look at /etc/services this file defines network services including SSH. Find the entries for SSH:

ssh    22/udp    # SSH Remote Login Protocol
ssh    22/tcp      # SSH Remote Login Protocol

and duplicate them creating a new service called ssh2

ssh2    52222/udp    # SSH Server
ssh2   52222/tcp      # SSH Server

Save the file.

Here I am using 52222 for the UDP/TCP ports, you can use other port numbers but stay clear of well known ports from 0 – 1023 (dynamic/private ports 49152 to 655535 are preferable).

Now edit /Library/LaunchDaemons/com.openssh.sshd.plist  and change the SockServiceName string to ssh2.


Save the file and reboot.

We are basically telling the operating system to continue using port 22 for SSH connections but to listen for SSH connections on a different port.

You can now connect using SSH on the port you specified i.e.

ssh root@my.ipad.address:52222

Remember to change the root and mobile default passwords of your i device when you login.

IOS 7 multi tasking affects SSH background CONNECTIONS

So now I have root access to my IOS 7.0 device I can run SSH to create a secure tunnel to my Ubuntu server:

ssh -N -g -D XXXX

This creates a SOCKS proxy tunnel on port XXXX over SSH to my server, the i device can be configured to send all traffic via this proxy with a proxy auto config (PAC) file.

On IOS 7.0 this worked as expected, hurrah! I ran the shh tunnel changed my Wifi proxy settings to auto using my PAC file URL, switched apps to Chrome, checked my IP address to confirm I was proxying via the SOCKS tunnel and was happy that my iPad data was going through the “secure” tunnel –  until a few minutes later when it stopped working, doh!

The tunnel stops working because shortly after switching apps the SSH connection to the iPad is terminated, also terminating the SSH tunnel. This is because Apple has changed the way App multitasking works in IOS 7.x

When you switch apps in IOS 7 some apps continue to run for a short while and are then set to a suspended state to reduce system resources. They will “instantly” launch when you return to them. Of course when an app like iSSH switches to the background and is suspended by the operating system any active SSH connections will quickly timeout and terminate. This means our session running the SSH tunnel will be terminated, closing the tunnel.

Some apps are allowed to update in the background, and this is controlled via the background refresh options in settings, but as of the time of writing iSSH (and Prompt) do not appear in this list. (in IOS 6 apps were allowed to run for 10 minutes in the background and iSSH used to prompt you to return to the app to keep connections alive).

Fortunately there is a simple work around to this problem, install the mobile terminal app via Cydia and reboot. The mobile terminal app has been around for a while and gives you direct command line access as the mobile user. Although Cydia says it only supports IOS v4 to v6 it installs and runs perfectly on IOS v7.x too.

The great thing about mobile terminal is that it creates a direct local login session. When you switch the app to the background this session will keep running even when the mobile terminal app is suspended and reset. In fact if you install adv-cmds via Cydia you can login via SSH and see this login session running as a process with the ps command.

So we execute our SOCKS proxy ssh command in mobile terminal and setup the tunnel, when the mobile terminal app switches to the background the tunnel will stay open in the login session indefinitely, or until you kill the session manually from another command line using the kill process command.

If you don’t want to use mobile terminal have a look at the cydia implementations of screen and autossh.

Now I have full SSH functionality from my (bloody expensive) IOS 7.x computer again!

Here is the pac file I use for my proxy auto config:

function FindProxyForURL(url, host) {
return “SOCKS localhost:XXXX”;