It’s a question I’ve been asking myself.

I started this blog way back in 2010.
For years, it was a place where I shared thoughts, ideas, and whatever else was on my mind.
But then it kinda got left behind, life happened, things changed.
And like many others, I just stopped posting here.

At first, I blamed it on being too busy.
Then on not having anything to say.
But if I’m honest, the real reason has become clearer lately:

The internet has changed.

In 2025, it feels like blogs — once vibrant, personal spaces — have been pushed to the sidelines.
Google has changed too.
It’s not the same search engine we use to know, now, it feels like AI is answering everything before you even have a chance to click a link.

People don’t “browse” the web like they used to.
They ask AI a question and get an answer immediately.
Quick.
Effortless.
Efficient.

Who needs to scroll through endless blog posts anymore?

Right…?

But here’s the thing.
Last night, I realised something.
Maybe blogs aren’t dead.
Maybe they’ve just evolved — like everything else online.

A blog is still a place where you own your words.
Where you choose the conversation.
Where you’re not boxed in by an algorithm, a feed, or a chatbot’s summarised answer.

AI can give you an answer.
But a blog gives you a person.
A real voice, not just a response.

So maybe it’s not that blogs are dead.
Maybe we just forgot why they mattered in the first place.

Maybe it’s time to start posting again…

If you have already used any dapps you will be familiar with crypto wallet apps such as MetaMask that allow you to access your accounts on various blockchains. I was interested in how these apps connect to MetaMask and came across a github project using MetaMask as a passwordless user authentication system for web apps and thought this would be a great way to do user logins for some of my applications.

The demo below shows how to connect with a web3 wallet and then authenticate the user to a PHP application by signing a message within the wallet to validate the account.

How it works

Let’s look at the workflow used by the client javascript and server php code.

First on the client we try and initialise a web3 provider and connect to a wallet with web3Connect(). This will fetch and initialise account information variables and update the gui. If no provider is found we launch a new Web3Modal window which is a single Web3 / Ethereum provider solution for all Wallets.

With the wallet connected and the account public address identified we can offer a user login using the public address as the unique identifier. First we call web3Login() which initiates the backend login process. We are using the axios plugin to post the login data to the backend php script which queries an sql database to check if the user account exists / creates a new account. The backend generates an Cryptographic nonce which is passed back as a sign request to the client. The client requests that the message be signed in the wallet and the signed response is sent back to the server to be authenticated.

We now have the server generated message, the same message signed by the user and the users public address. The backend performs some cryptographic magic in order to determine if the original message was signed with the same private key to which the public address belongs. The public address also works as a username to identify the account. If the signed message and public address belong to the same private key, it means that the user who is trying to log in is also the owner of the account.

After authentication the backend creates a JSON Web Token (JWT) to authenticate further user requests. PHP Session data is created by the backend which allows the authentication to persist between visits, with the backend using the JWT token to authenticate the user with each page request. The user is now logged in and the client updates the frontend gui accordingly.

To complete the demo a logout button is included to log the user out. In this demo anyone can create a new user account and “login”. In practice to restrict user access the backend would have a user approval process to enable new accounts, additionally user groups can be created to apply permissions to the account which are then used by the backend pages to determine which content is available to the user.

–

Demo page design based on a template from HTML5UP

Recently I got really confused about the correct way to configure Varnish for Magento 2, so for me and anyone else confused about the configuration here is the definitive guide to configuring Varnish for Magento 2.

The Definitive Guide to Configuring Varnish for Magento 2

To configure Varnish you need to know:

• Varnish server name
• Varnish listener TCP port – defaults to 6081
• Magento content server name
• Magento content server TCP port

If you are working with a single host the server name for Varnish and Magento will be localhost (127.0.0.1), if you are working in a Docker environment the server name for Varnish and Magento will be the container names of the Varnish and Magento web/content server services.

There are two areas in Magento where you must configure Varnish settings:

1. Magento Core Config : app/etc/env.php
2. Magento admin Stores -> Configuration -> System -> Full Page Cache

Core Config

The core config for Varnish in app/etc/env.php looks something like this:

    'http_cache_hosts' => [
        [
            'host' => 'varnish_server_hostname',
            'port' => '6081'
        ]
    ],

• HOST=hostname / ip address of Varnish server
• PORT=TCP listener port of Varnish server

You can configure these settings from the command line using:

php bin/magento setup:config:set --http-cache-hosts=varnish_server_hostname:6081

Admin Config

The admin config for Varnish in Stores -> Configuration -> System -> Full Page Cache looks something like this:

• ACCESS LIST=hostname / ip address of Magento content server/s
• BACKEND HOST=hostname / ip address of Magento content server/s
• BACKEND PORT=TCP listener port of Magento content server
• EXPORT CONFIGURATION=Click here to export a the Varnish VCL file

Note that the Varnish Configuration section settings here are only used to generate the Varnish VCL file.

If you are using Docker set the acl purge list in the Varnish VCL to all Docker private networks.

# purge set to docker nets
# 172.0.0.0/12 192.168.0.0/16
acl purge {
    "172.16.0.0"/12;
    "192.168.0.0"/16;
}

Troubleshooting

You can verify your Varnish configuration with n98-magerun2. Use the command:

n98-magerun2.phar config:show | grep full_page_cache

You should see something like the following where magento2_php-apache_1is the hostname of your Magento 2 content server and backend_port is the tcp port of the content (Magento) server

system/full_page_cache/caching_application - 2
system/full_page_cache/varnish/access_list - magento2_php-apache_1
system/full_page_cache/varnish/backend_host - magento2_php-apache_1
system/full_page_cache/varnish/backend_port - 8080
system/full_page_cache/varnish/grace_period - 300

Don’t get the ports mixed up :

• By default Varnish is configured to listen for incoming external client http requests on TCP 6081.
• The backend_port configured in admin is only used for the vcl config generation.
• The env.php http_cache_hosts port is the port used to communicate with varnish.

To confirm your Varnish cache is working examine the headers returned by your Varnish server when browsing Magento frontend pages. You can also inspect the headers using curl

curl -I -H "host: magento2.gaiterjones.com" 127.0.0.1:80

X-Varnish: 762904605
Age: 0
X-Magento-Cache-Debug: HIT
Pragma: no-cache
Expires: -1
Cache-Control: no-store

Here we can see the X-Magento-Cache-Debug header showing a cache hit. Note – this header will be disabled in production mode.

Remember that Varnish has no support for TLS connections over HTTPS. To use an encrypted TLS connection to Magento 2 with Varnish FPC you need to use a frontend proxy such as NGINX.

What are Asynchronous Message Queues?

The normal way for a module to process data caused by some kind of front or backend action is to create an observer that listens for the specific Magento event and then processes the event data in some way. For example if you want to process order data whenever an order is placed, you would create a custom module with an order event observer. When a customer places an order the event will be fired and your module can process the order data.

This event type processing occurs synchronously with the event. This can be a problem if your process is intensive and takes some time to complete, or if for some reason it causes an error. Synchronous processing of events may cause delays for the customer in frontend processes.

Asynchronous Message Queues allow your event to be placed in a queuing system and processed by your modules consumer as soon as possible.

The Magento Message Queue Framework consists of the following components:

• Publisher
  A publisher is a component that sends messages to an exchange.
• Exchange
  An exchange receives messages from publishers and sends them to queues.
• Queue
  A queue is a buffer that stores messages.
• Consumer
  A consumer receives messages. It knows which queue to consume.

By default Magento 2 uses the MySQL database as the Exchange and Queue system for messages. It does this with a MySQL adapter, Message data is stored in the queue, queue_message, queue_message_status tables. Magento uses the cron job consumers_runner to manage queued messages by starting (or restarting) message consumers.

The fastest this can happen with the Magento cron system is once every 60 seconds. The job is configured in the MessageQueue module.

job name="consumers_runner" instance="Magento\MessageQueue\Model\Cron\ConsumersRunner" method="run">
    <schedule>* * * * *</schedule>
</job>

I recommend looking at Magento\MessageQueue\Model\Cron\ConsumersRunner to understand how the default Magento consumers are managed.

If you want to list all the default consumers available in Magento 2 use the command bin/magento queue:consumers:list

Using the Magento database for messaging is not very scalable

RabbitMQ should be used whenever possible.

Converting Magento 2 Message Queues to Rabbit MQ AMQP

RabbitMQ is an open source message broker system and the Magento Message Queue Framework has built in support for RabbitMQ as a scalable platform for sending and receiving messages. RabbitMQ is based on the Advanced Message Queuing Protocol (AMQP) 0.9.1 specification.

Configuring Magento 2 to use RabbitMQ requires the addition of a new queue node in app/etc/env.php

    'queue' => [
        'amqp' => [
            'host' => '<rabbitmq_host>',
            'port' => '<rabbitmq_port>',
            'user' => '<rabbitmq_user>',
            'password' => '<rabbitmq_pass>',
            'virtualhost' => '/',
            'ssl' => false
        ],
     ]

Enabling message queues for RabbitMQ AMQP is simply a case of changing the configuration of the queue from DB to AMQP.

connection="amqp"

One built in message queue is by default configured to use AMQP this is the async.operations.all queue. If you have successfully configured RabbitMQ you should see this queue in the admin interface

When creating message queues for new modules you should configure them to use AMQP, check out this module for an example module using a product save event message.

If you want to convert existing MySQL DB message queues to use AMQP you can accomplish this using extra nodes in the env.php queue configuration. This example is taken from the official documentation for product_action_attribute.update queue:

'queue' => [
    'topics' => [
        'product_action_attribute.update' => [
            'publisher' => 'amqp-magento'
        ]
    ],
    'config' => [
        'publishers' => [
            'product_action_attribute.update' => [
                'connections' => [
                    'amqp' => [
                        'name' => 'amqp',
                        'exchange' => 'magento',
                        'disabled' => false
                    ],
                    'db' => [
                        'name' => 'db',
                        'disabled' => true
                    ]
                ]
            ]
        ]
    ],
    'consumers' => [
        'product_action_attribute.update' => [
            'connection' => 'amqp',
        ],
    ],
],

In theory you can try and convert all Magento 2 MySQL message queues to AMQP RabbitMQ message queues. If you do this you will also probably want to create your own external consumers to process RabbitMQ message queues more efficiently.

Keep in mind that if you create external message queue consumers you should ensure that the consumer processes are always running. There are a few ways to accomplish this for example by using a supervisord process control system. If you are working with a Docker environment I recommend creating one or more consumer containers to manage Magento 2 AMQP messages.

You can configure the default Magento 2 cron_runners_runner cron job via env.php settings

    'cron_consumers_runner' => [
        'cron_run' => true,
        'max_messages' => 20000,
        'consumers' => [
            'consumer1',
            'consumer2',
        ]
    ],

Note that if you set cron_run here to false you are completely disabling the default consumer cron job in Magento. If you are using external consumers for some or all message queues think carefully before completely disabling this cron job.

Note – during testing I was unable to convert some existing Magento 2.4.2 MySQL queues to AMQP

Docker Consumers and Message Management

I like to use dedicated Docker containers to manage message queue consumer processes in my Magento 2 Docker environment. In theory a container should manage a single consumer, in practice it can be easier to run multiple consumers in one container. The container needs to know which consumers to start, and also needs to monitor consumer processes to ensure that they remain running.

To manage this process and make it easier to convert multiple message queues from MySQL db to AMQP I created a Message Manager module.

This module allows you to convert some or all queues to AMQP, automatically updating the env.php settings to configure the changes required for each module. The module also feeds data to my Docker container telling the container which consumer processes it should manage.

After installing the module you can display all existing message consumers with:

bin/magento messagemanager:getconsumers

This will return an array of consumers showing the queue name and the current connection method – db for MySQL, – amqp for RabbitMQ AMQP.

To convert specific queues to AMQP you can define a whitelist in Gaiterjones\MessageManager\Helper\Data

    public function whitelist(): array
    {
        return array(
            'gaiterjones_message_manager',
            'product_action_attribute.update'
        );
    }

Here I am specifying two queues that I want to use with RabbitMQ AMQP. I know gaiterjones_message_manager is already configured for AMQP and I want to convert product_action_attribute.update from MySQL to AMQP.

To do this I can use the getconfig command. First check your current AMQP config with bin/magento messagemanager:getconfig ensure that your RabbitMQ server is configured.

    [amqp] => Array
        (
            [host] => magento2_rabbitmq_1
            [port] => 5672
            [user] => blah
            [password] => blah
            [virtualhost] => /
        )

Next display the configuration required to convert your whitelisted queues to AMQP

bin/magento messagemanager:getconfig --buildconfig --whitelist

(To display the configuration for all MySQL configured queues use bin/magento messagemanager:getconfig --buildconfig)

After reviewing the configuration use the saveconfig switch to save this configuration to env.php :

bin/magento messagemanager:getconfig --buildconfig --whitelist --saveconfig

Make sure you backup env.php before running this command!

After changing the config run setup:upgrade to reconfigure the module queues. If the queue config creates an error restore your env.php settings and run setup:upgrade again to restore the configuration.

Now you should see the converted queues in RabbitMQ admin.

The gaiterjones_message_manager queue is installed with the module, to test RabbitMQ is working use the test queue command bin/magento messagemanager:testqueue

You should see a new message created in RabbitMQ.

Summary

It is very likely that future versions of Magento 2 will stop using MySQL as a messaging broker. In the same way that Search was moved to Elasticsearch we may have to use RabbitMQ in the future for all Magento async messaging.

Message queues are cool! If you are building production Magento 2 sites in a Docker environment it makes sense to start using RabbitMQ and external consumers now.

More Reading

In conclusion I recommend reading through the official documentation for more information on this subject.

• Manage Message Queues
• Message Queues Overview
• consumers_wait_for_messages
• Message Queues

Credits

Thanks to Diana Botean for her helpful comments.

The attribute appeared to be created correctly in the eav_attribute database table, but the frontend value was not being saved to the database when the customer registered.

The attribute was a checkbox, which has a boolean true/false value. As I set the checkbox to be checked by default, my mistake was simply not to set a default value which meant that no value for the custom attribute was being passed via the registration form to the backend customer registration / creation process.

    <div class="field gj_custom_boolean_attribute">
        <label for="gj_custom_boolean_attribute" class="label"><span><?php /* @escapeNotVerified */
                echo __('Customer Custom Boolean Attribute') ?></span>
        </label>
        <div class="control">
            <input type="checkbox" checked name="gj_custom_boolean_attribute" id="gj_custom_boolean_attribute" title="<?php /* @escapeNotVerified */
            echo __('Customer Custom Boolean Attribute') ?>" class="input-text" autocomplete="off" value="1">
        </div>
    </div>

In case anyone else is trying to create a custom checkbox (boolean) attribute and experiencing the same mind numbingly annoying problem here is a module with a demonstration of two custom Magento 2 attributes – checkbox and text input that demonstrates the correct working code.

https://github.com/gaiterjones/Magento2_Gaiterjones_CustomerRegistration

The module creates two attributes:

• gj_custom_boolean_attribute – a sample boolean checkbox attribute
• gj_custom_text_attribute – a sample text attribute

The additional registration html is in the additionalinfocustomer.phtml template file. Note here that the checkbox requires a default value

value="1"

Migrating the Magento 2 catalog search engine to Smile ElasticSuite will resolve pretty much all the issues you might be experiencing with Magento 2 native ElasticSearch catalog search so go ahead and Jump to the ElasticSuite installation.

If you are new to ElasticSearch or want to find out how to customise ElasticSearch in Magento 2 read on!

Magento Catalog Search

Up until version 2.3 the default catalog search engine for Magento used the MySql Magento database. Using MySql for search was adequate but it lacked the features and scalability of enterprise search solutions. In version 2.3 Magento built in support for ElasticSearch as the catalog search engine and announced in 2019 that MySql search would be deprecated. As of version 2.4 in July 2020 the MySql catalog search engine was removed completely from Magento 2.

Native support for ElasticSearch was good news for Merchants, Elasticsearch is a java based open-source, RESTful, distributed search and analytics engine built on Apache Lucene. Since its release in 2010, Elasticsearch has quickly become the most popular search engine.

It’s worth mentioning that ElasticSearch in Magento 2 is not just used for user full text search queries, the catalog search engine is responsible for returning all catalog queries including category products and filtered navigation product lists.

For customers ElasticSearch should provide faster and more relevant search experiences – the problem for merchants is that out of the box Magento 2 ElasticSearch just doesn’t do this – catalog search results and relevance have a tendency to be extremely poor. The built in search struggles to provide accurate and relevant results for simple queries such as SKUs.

Whilst MySql catalog search had some admin options to refine search results, there are no options available to customise catalog search with ElasticSearch. ElasticSearch is a great search engine but the native Magento 2 catalog full text search implementation is very disappointing.

Let’s look at ways to customise ElasticSearch catalog search in Magento using your own module to improve some areas of search relevance.

Simple SKU Search

Poor search results or search relevance with native Magento ElasticSearch is very apparent when searching for SKUs. Let’s look at a simple SKU search for one of the sample products provided in the Magento 2 sample data.

Article MH03 is a range of Hoodies. Searching for ‘MH03’ correctly returns all 16 products. But what if you want to search for MH03-XL?

Here we see that 112 items are returned when in fact only the first 3 were 100% matches for the search term. Native search really struggles with search terms containing special characters such as the hyphen commonly used in SKUs resulting in extremely poor search results. To look at why we are seeing so many results returned we need to look at the relevance score of the search results.

Customise Elastic Search

To capture the data returned by an ElasticSearch frontend full text search query we need to create a debug plugin for Magento\Elasticsearch\SearchAdapter\ResponseFactory that will let us analyse the search data and log it.

<type name="Magento\Elasticsearch\SearchAdapter\ResponseFactory">
<plugin disabled="false" sortOrder="2" name="gaiterjones_elasticsearch_queryresultsdebug" type="Gaiterjones\Elasticsearch\Plugin\QueryResultsDebug"/>
</type>

The plugin dumps the search data to the debug log in var/log, this allows us to look more closely at the ElasticSearch results for the MH03-XL SKU full text search :

Array
(
    [results] => 112
    [scores] => Array
        (
            [0] => 40.57959
            [1] => 40.57959
            [2] => 40.57959
            [3] => 29.099976
            [4] => 12.694372
            SNIP...
            [108] => 8.300152
            [109] => 8.300152
            [110] => 7.9659967
            [111] => 7.9659967
        )

    [min_relevance_score] => 1
    [min_score] => 7.9659967
    [max_score] => 40.57959
)

The debug shows the ElasticSearch rawdocument score  for the 112 search results, you can see that the score value for the search ranges from 7.9 to 40.5 with the most relevant results having a higher score. If we were to define a minimum relevance score of 40 the search results would be much more accurate.

We can do this with another plugin :

<type name="Magento\Elasticsearch\Elasticsearch5\SearchAdapter\Mapper">
<plugin disabled="false" sortOrder="1" name="gaiterjones_elasticsearch_searchadapter_mapperplugin" type="Gaiterjones\Elasticsearch\Plugin\Elasticsearch5\SearchAdapter\MapperPlugin"/>
</type>

Here we set a min_score value for the search query. Setting this to 40 would return just three results for the MH03-XL SKU search.

This looks much better, we can improve the relevance of the search results by filtering out results that have a low ElasticSearch score. The tricky part here is deciding what the minimum score value should be – it can be difficult to find a value that works well for different search queries.

Another useful ElasticSearch customisation is changing the ngram values when indexing the catalog. The ngram tokenizer helps break search terms up into smaller words. We can change the ngram value with another plugin

<type name="Magento\Elasticsearch\Model\Adapter\Index\Builder">
<plugin disabled="false" sortOrder="3" name="gaiterjones_elasticsearch_indexbuilder" type="Gaiterjones\Elasticsearch\Plugin\IndexBuilder"></plugin>
</type>

In this case we are increasing the ngram value to 3. This will take effect at the next catalog search full text reindex.

To test these customisation values for yourself you can download the full module here https://github.com/gaiterjones/Magento2_Gaiterjones_Elasticsearch

The module adds a configuration value to Stores -> Configuration -> Catalog -> Search where you can set the minimum score value for Elastic Search results.

It’s a real shame that ElasticSearch customisation options such as these are not built into Magento 2 by default to help Merchants improve the search experience. ElasticSearch is new to me, and will be to a lot of merchants and Magento devs upgrading to Magento 2.4. It’s a very complex system to understand and although we can tweak some values as shown to improve results this is not a great solution to the problem.

If like me you are still not happy with the native Magento 2 ElasticSearch catalog search results the absolute best solution I have found is to migrate to Smile ElasticSuite

Simply put, installing the Smile ElasticSuite modules and changing catalog search to ElasticSuite will immediately give you almost perfect search results. ElasticSuite is very simple to install and works out of the box improving search results and search relevance.

Install Smile ElasticSuite

Here are the steps required to install ElasticSuite.

Note that ElasticSuite includes it’s own custom layered navigation, if you are already using third party layered navigation modules you will need to disable these first before installing elasticsuite.

You will need to choose the correct ElasticSuite version for the version of Magento 2 you are using. Here are the options for Magento 2.3.x and 2.4

• Magento 2.3.3 -> 2.3.5
• • ElasticSuite 2.8.x : composer require smile/elasticsuite ~2.8.0
• Magento 2.3.5 +
• • ElasticSuite 2.9.x : composer require smile/elasticsuite ~2.9.0
• Magento 2.4.0
• • ElasticSuite 2.10.x : composer require smile/elasticsuite ~2.10.0

After installing the modules run setup:upgrade and then configure your ElasticSearch server hostname.

• bin/magento s:up
• bin/magento config:set -l smile_elasticsuite_core_base_settings/es_client/servers elasticsearch_server_hostname:9200
• bin/magento s:up
• bin/magento index:reindex

To change your catalog search engine to ElasticSuite navigate to Stores -> Configuration -> Catalog -> Search and select ElasticSuite as the new catalog search engine.

Refresh caches and the ElasticSuite catalog search engine should now be setup and working – congratulations – Magento 2 full text catalog search just got a whole lot better!

if you see the following error in the frontend, simply run the indexer again.

Exception #0 (LogicException): catalog_product index does not exist yet. Make sure everything is reindexed.

ElasticSuite has some great features :

• A new search input form with automatic and relevant search suggestions in a dropdown list
• Layered navigation with price sliders and advanced filters
• Automatic redirect to product when search returns a single matching product
• Automatic spell checked search terms
• Smart virtual product categories
• Search analysis in admin

You will notice straight away when searching for SKUs that ElasticSuite returns more relevant results than native search.

Using the SKU search example you can search for all or part of the SKU with or without the hyphen and accurate search results will be returned. Notice below the search for MH03 XL without the hyphen returns the correct results

The redirect to product feature when a single matching product is found in search is really useful taking the customer directly to the relevant product.

The search analysis in admin is a great feature allowing you to see how search is being utilised by your customers and which search terms lead to conversions.

For more information on ElasticSuite features and configuration consult the ElasticSuite Wiki or visit the website.

Many thanks to the Smile team for making this module freely available to the Magento 2 community.

Acknowledgements

• Smile ElasticSuite
• Moore Digital Magento 2 Elasticsearch Relevance
• Sandip – ngram plugin

Compared to Magento 1, Magento 2 configurable products are just containers for simple products (variations). You can no longer configure pricing data directly in the configurable product as the parent configurable inherits all it’s unit, tier price and inventory data from the child simple products. It’s also much easier to create products with variations such as size, colour etc. and display them in the frontend in various different ways i.e. visual swatches.

Large Configurable Products are slow to Load!

One of the downsides to Magento 2 configurable products is that large configurable products can be slow to load in the frontend if a lot of variations are configured. I worked on a store with a product that is available in over 250 colours and four sizes. This resulted in a configurable product with over 1000 child products and whilst theoretically there is no limit to the amount of simple products in a configurable container product in practice, the way Magento 2 builds the frontend product can lead to very slow load times.

In the frontend, Magento 2 loads all variations in a giant JSON object and renders that into the DOM. This JSON object is 20 megabytes for 10,000 variations. In the backend, this JSON is also built and passed to a UI component wrapped in XML. PHP’s xmllib is not able to append extremely large XML structures to an existing XML structure.

Even with 1000 variations page load time for an uncached configurable product was in excess of 30 seconds.

Elgentos LCP

Fortunately the nice people at Elgentos open sourced a module developed for a customer experiencing exactly this slow loading Magento 2 configurable product problem. elgentos/LargeConfigProducts greatly improves the loading time of large configurable products by pre-caching the product variation data in the backend and loading the frontend variation JSON as an asynchronous ajax request. This results in a much faster load time of the parent product and the cached variation data.

When I tested the module there were some issues with Magento 2.3.x compatibility which the developer had not had time to correct. I made some changes to make it compatible and also added AQMP/RabbitMQ integration. I am using the module in production without any issues and it has made a big difference to page loading times.

Here are a few notes on installing and using the module with Magento 2.3.x.

After installing elgentos/LargeConfigProducts you should configure the prewarm settings

“Prewarming” is the process of creating and caching the variation data for configurable products. The module uses Redis to cache the data and you should specify your Redis hostname/IP, TCP port and choose a new database for the data.

The module includes a new indexer that will prewarm all configurable products when you manually reindex with bin/magento index:reindex

With the module configured and enabled all configurable products will now load variation data via an Ajax request. If variation product data has not been prewarmed or cached the cache will be updated when the product loads. You can also manually create the product variation data cache using a console command

bin/magento lcp:prewarm --products 1234,12345 -force

This will force a prewarm of variation data for a configurable product with the ids 1234 and 12345.

When you make a change to a configurable product, or a child of a configurable product the module uses a message queue to update the configurable product cached data. Magento 2.3 has built in AQMP/RabbitMQ integration and you can add a Rabbit MQ server to your Magento 2 system by using the following env.php configuration :


'queue' => [
'amqp' => [
'host' => 'RABBITMQ_HOST_NAME',
'port' => 5672,
'user' => 'guest',
'password' => 'guest',
'virtualhost' => '/'
] ],


Messages are created by a publisher and actioned by a consumer process in the module. To list all the configured Magento 2 consumer queues use:

bin/magento queue:consumers:list

You will see that elgentos_magento_lcp_product_prewarm is listed. To run the prewarm consumer use bin/magento queue:consumers:start elgentos_magento_lcp_product_prewarm this will start processing all messages generated by the module and updating the product variation data cache for any products that have been changed.

You should ensure that your consumer process is always running. If you use Docker you can create a small consumer container for this purpose.

    consumer1:
        hostname: shop01_consumer
        build:
            context: ./consumer/
        volumes:
            - "${CONTAINERDATA}/${PROJECT_NAME}/www1/dev/magento2:/var/www/dev/magento2"
        depends_on:
            - mysql
            - rabbitmq
        restart: always
        entrypoint: ["php", "./bin/magento", "queue:consumers:start", "elgentos_magento_lcp_product_prewarm"]

I can also recommend using the RabbitMQ Docker container image: rabbitmq:management the built in management gui is useful for monitoring message data here you can see the lcp message generation for the prewarm consumer after performing a reindex

In my opinion this functionality should be built into Magento by default to improve the loading time of large configurable products. Changes might be coming to configurable products in Magento 2.4 so perhaps there will be improvements made in this area.

Many thanks to Elgentos and Peter Jaap Blaakmeer for making this module freely available to the community and allowing me to contribute to it.

I was working on my procedures for applying updates to a production Magento 2 site recently and decided it was a pretty good idea to put Magento into maintenance mode first before making any changes or updates that might temporarily break the site and return a nasty error message. The default production maintenance page for Magento 2 looks like this

It’s not exactly what I would call a thing of beauty. A Google search reveals a plethora of solutions – but I really wanted something simple. In my mind a custom module with a thousand customisation options for a maintenance page is somewhat overkill. You can also create your own custom response by editing or extending the 503.phtml file in pub/errors/default.

503 Service Unavailable

Notice that in production mode the maintenance page returns a 503 error which is correct as we want any visitors (and crawlers) to know that the site is temporarily unavailable. (In development mode this is a much more unfriendlier http 500 error!)

There is however a problem associated with your Magento site returning a 503 error in maintenance mode. If you are using Varnish, and especially if you are using the health probe in varnish the 503 error will cause varnish to eventually announce the server as sick and throw it’s own extremely unfriendly error – something about meditating gurus.

If you look at the Magento docs they actually suggest creating a custom maintenance page via the web server – Apache or NginX. The examples show a configuration whereby the web server redirects to a custom url when a maintenance file is present on the system.

server {
     listen 80;
     set $MAGE_ROOT /var/www/html/magento2;

     set $maintenance off;

     if (-f $MAGE_ROOT/maintenance.enable) {
         set $maintenance on;
     }

     if ($remote_addr ~ (192.0.2.110|192.0.2.115)) {
         set $maintenance off;
     }

     if ($maintenance = on) {
         return 503;
     }

     location /maintenance {
     }

     error_page 503 @maintenance;

     location @maintenance {
     root $MAGE_ROOT;
     rewrite ^(.*)$ /maintenance.html break;
 }

     include /var/www/html/magento2/nginx.conf;
}

Here they are suggesting that if the file maintenance.enable is present NginX will 503 redirect to a maintenance page. A similar config example is available for Apache.

This also works quite well and if you change the file detection to the Magento 2 system generated maintenance file /var/.maintenance.flag As soon as you place Magento into maintenance mode the custom page would be shown – cool!

But there are still a couple of drawbacks, first with your site returning 503 for all pages your maintenance page can’t load any external js or css hosted on your Magento server so your maintenance page needs to be pretty basic. Second you are still returning a 503 to Varnish which will eventually cause a health error.

Chances are if you are using Varnish you also have an NginX reverse proxy in front of Varnish providing TLS encryption. Or if you are using Docker, NginX is reverse proxying http/s to your containers. If so then this is best place to configure your custom maintenance page and you can create a really nice looking dynamic Magento custom maintenance page that will appear as soon as you place Magento into maintenance mode – or whenever Magento or Varnish return 503 errors.

For Docker you will need to mount a volume on NginX giving it access to the var/ folder in Magento so that it can detect the .maintenance.flag file.

The NginX config looks like this

    # MAGENTO 2 Maintenance Mode
    set $MAGE2_ROOT /var/www/gaiterjones/magento2/;
    set $maintenance off;
    if (-f $MAGE2_ROOT/.maintenance.flag) {
        set $maintenance on;
    }
    if ($maintenance = on) {
        return 503;
    }
    error_page 503 @maintenance;
    location @maintenance {
        root /var/www/html;
        rewrite ^(.*)$ /magento2-maintenance.html break;
    }

Here you can see the Magento var folder is mounted to var/www/gaiterjones/magento2 in NginX and if the maintenance file exists we redirect to a local maintenance.html page in var/www/html

The custom maintenance.html page can be any kind of page you want, I’m using a nice responsive page that you can see below.

As soon as you do a bin/magento maintenance:enable NginX will show the maintenance page returning a 503 code to any visiting customers (or search engines). My page refreshes every 30 seconds so as soon as you do bin/magento maintenance:disable customers will automatically see your shop again (hopefully).

Responsive HTML template by HTML5UP download this template here

Magento 2 uses the RequireJs javascript module system to load Javascript source code required for each Magento 2 page. If you have a lot of custom features with modules implementing additional Magento 2 Javascript mixins the number of Javascript resources in addition to the core javascript code required by Magento will increase and adversely affect page loading performance. As an example, here is the network console log from a really simple product page from my development site, you can see that there are 194 requests for Javascript resources!

There are various ways to try and reduce the performance impact of loading lots of Javascript including using http2 which is great at handling small file requests quickly or minifying the Javascript source to reduce it’s size but the most effective way of optimising Javascript loading is to use bundling.

Javascript bundling is a technique that combines or bundles multiple files in order to reduce the number of HTTP requests that are required to load a page.

Magento 2 has a built in javascript bundler that is extremely ineffective! Users report it creating a huge multi megabyte javascript file that decreases performance instead of improving it. You will actually see the recommendation not to use the built in Magento 2 bundling referenced in Lighthouse reports – “Disable Magento’s built-in JavaScript bundling and minification, and consider using baler instead.”

Baler mentioned here is an AMD (Asynchronous Module Definition) module bundler / preloader for Magento 2 stores. You will find a lot of Magento 2 js bundling guides that recommend using Baler but for the average developer (like me) or Magento 2 merchant the bundling process with Baler can be quite complex and daunting. There is however a new Magento 2 js bundler available that is much easier to use.

MageSuite Magepack

The Magepack from MageSuite is a “Next generation Magento 2 advanced JavaScript bundler” it’s pretty easy to implement and as of version 2.0 the results it achieves are very impressive.

• Up to 91 points mobile score in Google Lighthouse.
• Up to 98% reduction in JavaScript file requests.
• Up to 44% reduction in transferred JavaScript size.
• Up to 75% reduction in total load time.
• Works with Magento’s JavaScript minification and merging enabled.
• Uses custom solution (inspired by Baler)

I installed Magepack on my Magento 2 development site in May 2020 and achieved a 100 desktop performance score with PageSpeed  –

This is a simple product page, using the default Luma theme and I am also using Nginx as a container proxy running the PageSpeed module, so you probably won’t achieve this kind of result on a real world product page but you will see a huge improvement. Check the results yourself here.

Let’s look at how to setup and install MagePack for Magento 2.3.x / 2.4.x.

Setup and install Magepack for Magento 2.3.x and 2.4.x

MagePack consists of a NodeJS bundler app and a Magento 2 module. The bundler app runs on Node JS v10 or higher. I’m running MagePack in my Docker Magento 2 php container, it’s running Ubuntu server 20.04LTS and I’ve tested Magepack with Magento 2.3.3, 2.3.5 and 2.4.1. To install Node JS simply run

curl -sL https://deb.nodesource.com/setup_10.x | bash -
apt-get install -y nodejs

Ubuntu will probably need some more dependencies before MagePack will install

apt-get install gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget

Finally to install the NodeJS MagePack app itself run
npm install -g magepack --unsafe-perm=true --allow-root

You will see that Magepack pulls down Chromium – it needs a web browser to analyse your Magento 2 site, most of the dependencies installed earlier are required for Chromium.

With Magepack installed, we now need to install the Magepack Magento 2 module

composer require creativestyle/magesuite-magepack

Next, depending on the version of Magento 2 you are running you might need to install some patches.

• For Magento 2.3.3 and earlier 7 patches are required
• For Magento 2.3.4 and 2.3.5 1 patch is required
• For Magento 2.4.1 no patches are required (tested November 2020)

The most painless way of patching Magento 2 is to use Cweagans/Composer-Patches

composer require cweagans/composer-patches

You will find all the patches you need here : https://github.com/integer-net/magento2-requirejs-bundling

In your Magento 2 installation folder create a patches folder copy the patches into it and edit your Magento 2 composer.json file to include the following composer extra patches config.

If you see the error Evaluation failed: ReferenceError: BASE_URL is not defined running magepack generate with 2.3.5+ or 2.4.x check your CSP configuration, or try disabling the CSP module temporarily – errors reported by CSP may create a magepack error.

Now run composer update Magento 2 will be patched and we are good to go.

Let’s get ready to bundle

Magepack needs to analyse pages from your Magento 2 store to determine the Javascript files your store is using and how they can be bundled. It saves this information in a configuration file called magepack.config.js. The magepack config file is generated by analysing three different type of pages from your Magento 2 store, a cms page i.e. the home page, a category page and a product page. This is done using the magepack generate command and supplying three store urls.

magepack generate --cms-url="http://magento2.gaiterjones.com/" --category-url="http://magento2.gaiterjones.com/en/buy-x-get-y.html" --product-url="http://magento2.gaiterjones.com/en/affirm-water-bottle.html"

Run this command in the root folder of your Magento 2 installation to create the magepack.config.js file. It’s worth noting that you could run this generate command from any system, and just copy the generated config file to your Magento 2 server.

If you take a look at magepack.config.js you will see it contains references to all the javascript required to load Magento pages. Below is an example from a product page.

example product section from magepack.config.js

[code language=”javascript”] name: ‘product’,
modules: {
‘Magento_Catalog/js/price-utils’: ‘Magento_Catalog/js/price-utils’,
‘Magento_Catalog/js/price-box’: ‘Magento_Catalog/js/price-box’,
‘Magento_Wishlist/js/add-to-wishlist’: ‘Magento_Wishlist/js/add-to-wishlist’,
‘Magento_Cookie/js/require-cookie’: ‘Magento_Cookie/js/require-cookie’,
‘Magento_Swatches/js/configurable-customer-data’: ‘Magento_Swatches/js/configurable-customer-data’,
‘Magento_Review/js/error-placement’: ‘Magento_Review/js/error-placement’,
‘Magento_Review/js/process-reviews’: ‘Magento_Review/js/process-reviews’,
‘Elgentos_LargeConfigProducts/js/swatch-renderer-mixin’: ‘Elgentos_LargeConfigProducts/js/swatch-renderer-mixin’,
‘text!Magento_Theme/templates/breadcrumbs.html’: ‘Magento_Theme/templates/breadcrumbs.html’,
‘magnifier/magnifier’: ‘magnifier/magnifier’,
‘magnifier/magnify’: ‘magnifier/magnify’,
‘Magento_Catalog/js/gallery’: ‘Magento_Catalog/js/gallery’,
‘Magento_ProductVideo/js/load-player’: ‘Magento_ProductVideo/js/load-player’,
‘Magento_ProductVideo/js/fotorama-add-video-events’: ‘Magento_ProductVideo/js/fotorama-add-video-events’,
‘Magento_Theme/js/model/breadcrumb-list’: ‘Magento_Theme/js/model/breadcrumb-list’,
‘Magento_Theme/js/view/breadcrumbs’: ‘Magento_Theme/js/view/breadcrumbs’,
‘Magento_Theme/js/view/add-home-breadcrumb’: ‘Magento_Theme/js/view/add-home-breadcrumb’,
‘Magento_Catalog/js/product/breadcrumbs’: ‘Magento_Catalog/js/product/breadcrumbs’,
‘jquery/jquery.parsequery’: ‘jquery/jquery.parsequery’,
‘Magento_ConfigurableProduct/js/options-updater’: ‘Magento_ConfigurableProduct/js/options-updater’,
‘Magento_Review/js/validate-review’: ‘Magento_Review/js/validate-review’,
‘Magento_Swatches/js/swatch-renderer’: ‘Magento_Swatches/js/swatch-renderer’,
‘Magento_Catalog/product/view/validation’: ‘Magento_Catalog/product/view/validation’,
‘Magento_Catalog/js/product/view/product-ids’: ‘Magento_Catalog/js/product/view/product-ids’,
‘Magento_Catalog/js/product/view/product-ids-resolver’: ‘Magento_Catalog/js/product/view/product-ids-resolver’,
‘Magento_Catalog/js/catalog-add-to-cart’: ‘Magento_Catalog/js/catalog-add-to-cart’,
‘Magento_Catalog/js/validate-product’: ‘Magento_Catalog/js/validate-product’,
‘Magento_Catalog/js/product/view/provider’: ‘Magento_Catalog/js/product/view/provider’,
‘text!mage/gallery/gallery.html’: ‘mage/gallery/gallery.html’,
‘text!Magento_InstantPurchase/template/confirmation.html’: ‘Magento_InstantPurchase/template/confirmation.html’,
‘Magento_InstantPurchase/js/view/instant-purchase’: ‘Magento_InstantPurchase/js/view/instant-purchase’,
‘Magento_Review/js/view/review’: ‘Magento_Review/js/view/review’,
‘fotorama/fotorama’: ‘fotorama/fotorama’,
‘mage/gallery/gallery’: ‘mage/gallery/gallery’,
‘text!Magento_InstantPurchase/template/instant-purchase.html’: ‘Magento_InstantPurchase/template/instant-purchase.html’
}
},
[/code]

All that remains now is for us to create the bundle files and deploy them for all our store views and themes. This is simply done with the magepack bundle command which you can execute from the Magento installation root folder. If you are running in development mode, deploy frontend static files first.

magepack bundle

Finally enable Magepack Javascript bundling in admin :

Stores – Configuration – Advanced – Developer – Javascript Settings

If you are in production mode these options will be hidden in admin. To enable Magepack Javascript bundling from the command line use

bin/magento config:set dev/js/enable_magepack_js_bundling 1

Note that you should also enable the other Javascript optimisation options here including minfy javascript files and move js code to the bottom of the page – but don’t enable the default bundling!

MagePack Javascript bundling should now be enabled. To check it’s working go to a Magento 2 product page and look at the source code, do a search for “bundle” and you should see the magepack javascript bundles

Now refresh the page and have a look at your network log

Instead of loading 194 Javascript files, the product page now loads 7, Magepack has bundled all the Javascript into two main bundle files.

I guess it’s now time to look at the PageSpeed Lighthouse performance reports for your optimised Magento 2 pages. If you are using the Chrome browser simply run a Lighthouse report from the DevTools page. You can also use Googles PageSpeed insights tool at https://developers.google.com/speed/pagespeed/insights/

This is the improvement I saw in a live production Magento 2 site

If you don’t see a big improvement remember there are a lot of other factors taken into Lighthouse performance reports. Work through the report and try to find out where you can make further improvements.

Deployment in production

Whenever you flush your sites static files you will need to remember to run magepack bundle again. In production mode you should add this to your deployment process

bin/magento setup:upgrade
bin/magento setup:di:compile
bin/magento setup:static-content:deploy en_GB --area adminhtml
bin/magento setup:static-content:deploy en_GB --area frontend --theme MY/Theme -f
magepack bundle
bin/magento cache:clean

Top Tip – if for any reason you want to generate the magepack.config.js bundle config again remember to disable the Magepack module first!

Testing and Troubleshooting

You should test your store thoroughly to make sure there are no Javascript problems caused by the bundling process. Magepack cannot always 100% bundle all the Javascript required by some pages. Check your web browser console for errors. If you find some features of your store are not working, try and identify if the code was included in the magepack.config.js file. Try removing the code from the bundle and test again.

Selective Bundling

If you do not want to bundle at checkout, or any other specific pages take a look at the isEnabled method in the Block\BundlesLoader class. This method determines whether the magepack module is enabled and if bundling should be activated. Simply detect any page here i.e. checkout pages and return false to disable bundling at checkout.

        \Magento\Framework\App\Request\Http $request,
        ....
        $this->request = $request;
        ....
public function isEnabled()
{
        if ($this->request->getFullActionName() == 'checkout_index_index') {

            // disable for checkout
            //
            return false;
        }

...

Magepack for Magento Cloud

You need to have magepack.config in repo, add this to magento.app.yaml:

echo "\n================================== Install and configure Magepack - Start ==============================="
unset NPM_CONFIG_PREFIX
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.35.1/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh"  ] && \. "$NVM_DIR/nvm.sh"
nvm install --lts=dubnium
npm install -g magepack
echo "\n================================== create bundle ==============================="
magepack bundle
echo "\n================================== Install and configure Magepack - END ================================="

Thanks to Solteq for this information.

Magepack is pretty new with updates being made regularly, be sure to check out the projects GitHub page for new issues.

References

https://github.com/magesuite/magepack
https://github.com/magesuite/magepack-magento
https://www.integer-net.com/magento-2-javascript-bundling-integer-net-2/
https://gist.github.com/tdgroot/f95c398c565d9bbb83e0a650cdf67617
https://github.com/integer-net/magento2-requirejs-bundling
https://developers.google.com/speed/pagespeed/module

PageSpeed optimisation in Magento 1

To do this you configure an alias account in Gmail settings configured with the credentials of your mail service. This is useful if you want to consolidate email accounts or if you have a private email server and want to use Gmail to send email via this server using your various private email address domains.

At the start of April 2020 Google rolled out a security update that affected mail delivery using third party accounts. All emails sent via the alias account would not deliver, bouncing back with the error TLS Negotiation failed, the certificate doesn’t match the host.

TLS Negotiation failed, the certificate doesn’t match the host.

Google appears to be enforcing a new email encryption policy for secureTLS connections including validating that the host name on the mail server TLS certificate matches the canonical hostname (MX record) of the third party mail account.

If the host name does not match you can no longer use an encrypted TLS connection in Gmail to send email via your (or your ISP’s) mail servers.

For example, if your MX record resolves to mail.domain.com but the TLS certificate presented is for smtp.domain.com then Gmail will not connect to your mail server. For some users the only option to get mail working again is to revert to an unencrypted connection – strange that Google even allow that!

Google have updated the help article for Send emails from a different address or alias to include a section about this problem.

Google also no longer accept self signed certificates for TLS mail connections.

I use an EXIM4 docker container for my private mail relay, and use Gmail as the hub for my email send/receive. To workaround this problem I created a docker Certbot container and issued new LetsEncrypt TLS certificates for all my private mail domains used with Gmail as well as the primary TLS certificate for my Mail server.

I can confirm this resolves the problem and third party provider email sending via Gmail is now working again.

For anyone using Exim4 the way to configure Exim to use multiple TLS certificates is to dynamically match them to your mail domain, I did this using

tls_privatekey=${if exists{/etc/exim4/tls/exim.key.${tls_sni}}{/etc/exim4/tls/exim.key.${tls_sni}}{/etc/exim4/tls/exim.key.mail.defaultdomain.com}}
tls_certificate=${if exists{/etc/exim4/tls/exim.crt.${tls_sni}}{/etc/exim4/tls/exim.crt.${tls_sni}}{/etc/exim4/tls/exim.crt.mail.defaultdomain.com}}

Thousands of people have been affected by this. Considering the amount of people working from home or struggling to work at all during the Corona Virus pandemic its really bad timing by Google to implement new email security policies that are service affecting for a lot of users.