magento2 Archives - gj https://blog.gaiterjones.com/tag/magento2/ gaiterjones Wed, 10 Feb 2021 10:03:59 +0000 en-US hourly 1 https://wordpress.org/?v=6.8 How to Improve Magento 2 ElasticSearch Catalog Search Results and Relevance https://blog.gaiterjones.com/how-to-improve-magento-2-elasticsearch-catalog-search-results-and-relevance/ https://blog.gaiterjones.com/how-to-improve-magento-2-elasticsearch-catalog-search-results-and-relevance/#respond Thu, 20 Aug 2020 17:15:25 +0000 https://blog.gaiterjones.com/?p=2275 Continue reading]]> TL;DR

Migrating the Magento 2 catalog search engine to Smile ElasticSuite will resolve pretty much all the issues you might be experiencing with Magento 2 native ElasticSearch catalog search so go ahead and Jump to the ElasticSuite installation.

If you are new to ElasticSearch or want to find out how to customise ElasticSearch in Magento 2 read on!

Magento Catalog Search

Up until version 2.3 the default catalog search engine for Magento used the MySql Magento database. Using MySql for search was adequate but it lacked the features and scalability of enterprise search solutions. In version 2.3 Magento built in support for ElasticSearch as the catalog search engine and announced in 2019 that MySql search would be deprecated. As of version 2.4 in July 2020 the MySql catalog search engine was removed completely from Magento 2.


MySql catalog search deprecation notice

Native support for ElasticSearch was good news for Merchants, Elasticsearch is a java based open-source, RESTful, distributed search and analytics engine built on Apache Lucene. Since its release in 2010, Elasticsearch has quickly become the most popular search engine.

It’s worth mentioning that ElasticSearch in Magento 2 is not just used for user full text search queries, the catalog search engine is responsible for returning all catalog queries including category products and filtered navigation product lists.

For customers ElasticSearch should provide faster and more relevant search experiences – the problem for merchants is that out of the box Magento 2 ElasticSearch just doesn’t do this – catalog search results and relevance have a tendency to be extremely poor. The built in search struggles to provide accurate and relevant results for simple queries such as SKUs.

Whilst MySql catalog search had some admin options to refine search results, there are no options available to customise catalog search with ElasticSearch. ElasticSearch is a great search engine but the native Magento 2 catalog full text search implementation is very disappointing.

Let’s look at ways to customise ElasticSearch catalog search in Magento using your own module to improve some areas of search relevance.

Simple SKU Search

Poor search results or search relevance with native Magento ElasticSearch is very apparent when searching for SKUs. Let’s look at a simple SKU search for one of the sample products provided in the Magento 2 sample data.


SKU Search for Magento 2 sample product

Article MH03 is a range of Hoodies. Searching for ‘MH03’ correctly returns all 16 products. But what if you want to search for MH03-XL?


Refined SKU Search for Magento 2 sample data

Here we see that 112 items are returned when in fact only the first 3 were 100% matches for the search term. Native search really struggles with search terms containing special characters such as the hyphen commonly used in SKUs resulting in extremely poor search results. To look at why we are seeing so many results returned we need to look at the relevance score of the search results.

Customise Elastic Search

To capture the data returned by an ElasticSearch frontend full text search query we need to create a debug plugin for Magento\Elasticsearch\SearchAdapter\ResponseFactory that will let us analyse the search data and log it.

<type name="Magento\Elasticsearch\SearchAdapter\ResponseFactory">
<plugin disabled="false" sortOrder="2" name="gaiterjones_elasticsearch_queryresultsdebug" type="Gaiterjones\Elasticsearch\Plugin\QueryResultsDebug"/>
</type>

beforeCreate plugin for SearchAdapterResponseFactory
[code language=”php”] public function beforeCreate(\Magento\Elasticsearch\SearchAdapter\ResponseFactory $subject, $result)
{
if($this->debug)
{
if(!is_array($result) || empty($result)) {return false;}

$scores=array();

foreach ($result[‘documents’] as $rawDocument) {
$this->logger->debug(‘ELASTIC_SEARCH_QUERY_RESULTS_DEBUG’,$rawDocument);
array_push($scores,$rawDocument[‘_score’]);
}

if (count($result[‘documents’]) > 0)
{
$_debug=array(
‘results’ => count($result[‘documents’]),
‘scores’ => $scores,
‘min_relevance_score’ => $this->configuration->getMinScore(),
‘min_score’ => min($scores),
‘max_score’ => max($scores)
);

$this->logger->debug(‘ELASTIC_SEARCH_QUERY_RESULTS_DEBUG’,$_debug);
}
}
}
[/code]

The plugin dumps the search data to the debug log in var/log, this allows us to look more closely at the ElasticSearch results for the MH03-XL SKU full text search :

Array
(
    [results] => 112
    [scores] => Array
        (
            [0] => 40.57959
            [1] => 40.57959
            [2] => 40.57959
            [3] => 29.099976
            [4] => 12.694372
            SNIP...
            [108] => 8.300152
            [109] => 8.300152
            [110] => 7.9659967
            [111] => 7.9659967
        )

    [min_relevance_score] => 1
    [min_score] => 7.9659967
    [max_score] => 40.57959
)

The debug shows the ElasticSearch rawdocument score  for the 112 search results, you can see that the score value for the search ranges from 7.9 to 40.5 with the most relevant results having a higher score. If we were to define a minimum relevance score of 40 the search results would be much more accurate.

We can do this with another plugin :

<type name="Magento\Elasticsearch\Elasticsearch5\SearchAdapter\Mapper">
<plugin disabled="false" sortOrder="1" name="gaiterjones_elasticsearch_searchadapter_mapperplugin" type="Gaiterjones\Elasticsearch\Plugin\Elasticsearch5\SearchAdapter\MapperPlugin"/>
</type>

aroundBuildQuery plugin for SearchAdapterMapperPlugin
[code language=”php”] public function aroundBuildQuery(
Mapper $subject,
callable $proceed,
RequestInterface $request
) {
$searchQuery = $proceed($request);

if ($request->getName() === ‘quick_search_container’) {
$searchQuery[‘body’][‘min_score’] = $this->configuration->getMinScore();
}

return $searchQuery;
}
[/code]

Here we set a min_score value for the search query. Setting this to 40 would return just three results for the MH03-XL SKU search.


SKU Search for Magento 2 sample products with min_score value

This looks much better, we can improve the relevance of the search results by filtering out results that have a low ElasticSearch score. The tricky part here is deciding what the minimum score value should be – it can be difficult to find a value that works well for different search queries.

Another useful ElasticSearch customisation is changing the ngram values when indexing the catalog. The ngram tokenizer helps break search terms up into smaller words. We can change the ngram value with another plugin

<type name="Magento\Elasticsearch\Model\Adapter\Index\Builder">
<plugin disabled="false" sortOrder="3" name="gaiterjones_elasticsearch_indexbuilder" type="Gaiterjones\Elasticsearch\Plugin\IndexBuilder"></plugin>
</type>

plugin afterBuild method for IndexBuilder
[code language=”php”] public function afterBuild(\Magento\Elasticsearch\Model\Adapter\Index\Builder $subject, $result)
{
$likeToken = $this->getLikeTokenizer();
$result[‘analysis’][‘tokenizer’] = $likeToken;
$result[‘analysis’][‘filter’][‘trigrams_filter’] = [
‘type’ => ‘ngram’,
‘min_gram’ => 3,
‘max_gram’ => 3
];
$result[‘analysis’][‘analyzer’][‘my_analyzer’] = [
‘type’ => ‘custom’,
‘tokenizer’ => ‘standard’,
‘filter’ => [
‘lowercase’, ‘trigrams_filter’
] ];
return $result;
}

protected function getLikeTokenizer()
{
$tokenizer = [
‘default_tokenizer’ => [
‘type’ => ‘ngram’
],
];
return $tokenizer;
}
[/code]

In this case we are increasing the ngram value to 3. This will take effect at the next catalog search full text reindex.

To test these customisation values for yourself you can download the full module here https://github.com/gaiterjones/Magento2_Gaiterjones_Elasticsearch

The module adds a configuration value to Stores -> Configuration -> Catalog -> Search where you can set the minimum score value for Elastic Search results.


Configure search minimum score value in admin

It’s a real shame that ElasticSearch customisation options such as these are not built into Magento 2 by default to help Merchants improve the search experience. ElasticSearch is new to me, and will be to a lot of merchants and Magento devs upgrading to Magento 2.4. It’s a very complex system to understand and although we can tweak some values as shown to improve results this is not a great solution to the problem.

If like me you are still not happy with the native Magento 2 ElasticSearch catalog search results the absolute best solution I have found is to migrate to Smile ElasticSuite

Simply put, installing the Smile ElasticSuite modules and changing catalog search to ElasticSuite will immediately give you almost perfect search results. ElasticSuite is very simple to install and works out of the box improving search results and search relevance.

Install Smile ElasticSuite

Here are the steps required to install ElasticSuite.

Note that ElasticSuite includes it’s own custom layered navigation, if you are already using third party layered navigation modules you will need to disable these first before installing elasticsuite.

You will need to choose the correct ElasticSuite version for the version of Magento 2 you are using. Here are the options for Magento 2.3.x and 2.4

  • Magento 2.3.3 -> 2.3.5
    • ElasticSuite 2.8.x : composer require smile/elasticsuite ~2.8.0
  • Magento 2.3.5 +
    • ElasticSuite 2.9.x : composer require smile/elasticsuite ~2.9.0
  • Magento 2.4.0
    • ElasticSuite 2.10.x : composer require smile/elasticsuite ~2.10.0

    After installing the modules run setup:upgrade and then configure your ElasticSearch server hostname.

  • bin/magento s:up
  • bin/magento config:set -l smile_elasticsuite_core_base_settings/es_client/servers elasticsearch_server_hostname:9200
  • bin/magento s:up
  • bin/magento index:reindex

To change your catalog search engine to ElasticSuite navigate to Stores -> Configuration -> Catalog -> Search and select ElasticSuite as the new catalog search engine.

Configure ElasticSuite as Catalog Search Engine
Configure ElasticSuite as Catalog Search Engine

Refresh caches and the ElasticSuite catalog search engine should now be setup and working – congratulations – Magento 2 full text catalog search just got a whole lot better!

if you see the following error in the frontend, simply run the indexer again.

Exception #0 (LogicException): catalog_product index does not exist yet. Make sure everything is reindexed.

ElasticSuite has some great features :

  • A new search input form with automatic and relevant search suggestions in a dropdown list
  • Layered navigation with price sliders and advanced filters
  • Automatic redirect to product when search returns a single matching product
  • Automatic spell checked search terms
  • Smart virtual product categories
  • Search analysis in admin

You will notice straight away when searching for SKUs that ElasticSuite returns more relevant results than native search.

ElasticSuite sample product SKU search
ElasticSuite sample product SKU search

Using the SKU search example you can search for all or part of the SKU with or without the hyphen and accurate search results will be returned. Notice below the search for MH03 XL without the hyphen returns the correct results

ElasticSuite sample product SKU search
ElasticSuite sample product SKU search

The redirect to product feature when a single matching product is found in search is really useful taking the customer directly to the relevant product.

The search analysis in admin is a great feature allowing you to see how search is being utilised by your customers and which search terms lead to conversions.

ElasticSuite search analysis
ElasticSuite search analysis

For more information on ElasticSuite features and configuration consult the ElasticSuite Wiki or visit the website.

Many thanks to the Smile team for making this module freely available to the Magento 2 community.

Acknowledgements

]]> https://blog.gaiterjones.com/how-to-improve-magento-2-elasticsearch-catalog-search-results-and-relevance/feed/ 0 Magento 2 Javascript Bundling with Magepack https://blog.gaiterjones.com/magento-2-javascript-bundling-with-magepack/ https://blog.gaiterjones.com/magento-2-javascript-bundling-with-magepack/#comments Sun, 03 May 2020 19:51:28 +0000 https://blog.gaiterjones.com/?p=2171 Continue reading]]> Magento 2 PageSpeed (Lighthouse) performance audit results for mobile and desktop are notoriously bad. Imagine you have worked for months on a new Magento 2 eCommerce store, followed best practices for setup and optimisation, the store seems to be running fine but the first time you run a Lighthouse report you see a performance score like this:There are a lot of factors that can affect the Lighthouse performance results for any website but for Magento 2 a big performance killer is the sheer amount of external resources required to render a page whether it be a product page, cms page or category page. Some of these render blocking resources such as Javascript or CSS can cause significant delays in page loading and affect performance. You will see this type of performance problem identified in Lighthouse as “Eliminate render-blocking resources”.

Magento 2 uses the RequireJs javascript module system to load Javascript source code required for each Magento 2 page. If you have a lot of custom features with modules implementing additional Magento 2 Javascript mixins the number of Javascript resources in addition to the core javascript code required by Magento will increase and adversely affect page loading performance. As an example, here is the network console log from a really simple product page from my development site, you can see that there are 194 requests for Javascript resources!

Simple Magento 2 product page loads 194 Javascript files
Simple Magento 2 product page loads 194 Javascript files

There are various ways to try and reduce the performance impact of loading lots of Javascript including using http2 which is great at handling small file requests quickly or minifying the Javascript source to reduce it’s size but the most effective way of optimising Javascript loading is to use bundling.

Javascript bundling is a technique that combines or bundles multiple files in order to reduce the number of HTTP requests that are required to load a page.

Magento 2 has a built in javascript bundler that is extremely ineffective! Users report it creating a huge multi megabyte javascript file that decreases performance instead of improving it. You will actually see the recommendation not to use the built in Magento 2 bundling referenced in Lighthouse reports – “Disable Magento’s built-in JavaScript bundling and minification, and consider using baler instead.”

Baler mentioned here is an AMD (Asynchronous Module Definition) module bundler / preloader for Magento 2 stores. You will find a lot of Magento 2 js bundling guides that recommend using Baler but for the average developer (like me) or Magento 2 merchant the bundling process with Baler can be quite complex and daunting. There is however a new Magento 2 js bundler available that is much easier to use.

MageSuite Magepack

The Magepack from MageSuite is a “Next generation Magento 2 advanced JavaScript bundler” it’s pretty easy to implement and as of version 2.0 the results it achieves are very impressive.

  • Up to 91 points mobile score in Google Lighthouse.
  • Up to 98% reduction in JavaScript file requests.
  • Up to 44% reduction in transferred JavaScript size.
  • Up to 75% reduction in total load time.
  • Works with Magento’s JavaScript minification and merging enabled.
  • Uses custom solution (inspired by Baler)

I installed Magepack on my Magento 2 development site in May 2020 and achieved a 100 desktop performance score with PageSpeed  –

This is a simple product page, using the default Luma theme and I am also using Nginx as a container proxy running the PageSpeed module, so you probably won’t achieve this kind of result on a real world product page but you will see a huge improvement. Check the results yourself here.

Let’s look at how to setup and install MagePack for Magento 2.3.x / 2.4.x.

Setup and install Magepack for Magento 2.3.x and 2.4.x

MagePack consists of a NodeJS bundler app and a Magento 2 module. The bundler app runs on Node JS v10 or higher. I’m running MagePack in my Docker Magento 2 php container, it’s running Ubuntu server 20.04LTS and I’ve tested Magepack with Magento 2.3.3, 2.3.5 and 2.4.1. To install Node JS simply run

curl -sL https://deb.nodesource.com/setup_10.x | bash -
apt-get install -y nodejs

Ubuntu will probably need some more dependencies before MagePack will install

apt-get install gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget

Finally to install the NodeJS MagePack app itself run
npm install -g magepack --unsafe-perm=true --allow-root

Installing Magepack NodeJS app and dependencies
Installing Magepack NodeJS app and dependencies

You will see that Magepack pulls down Chromium – it needs a web browser to analyse your Magento 2 site, most of the dependencies installed earlier are required for Chromium.

With Magepack installed, we now need to install the Magepack Magento 2 module

composer require creativestyle/magesuite-magepack

Next, depending on the version of Magento 2 you are running you might need to install some patches.

  • For Magento 2.3.3 and earlier 7 patches are required
  • For Magento 2.3.4 and 2.3.5 1 patch is required
  • For Magento 2.4.1 no patches are required (tested November 2020)

The most painless way of patching Magento 2 is to use Cweagans/Composer-Patches

composer require cweagans/composer-patches

You will find all the patches you need here : https://github.com/integer-net/magento2-requirejs-bundling

In your Magento 2 installation folder create a patches folder copy the patches into it and edit your Magento 2 composer.json file to include the following composer extra patches config.

If you see the error Evaluation failed: ReferenceError: BASE_URL is not defined running magepack generate with 2.3.5+ or 2.4.x check your CSP configuration, or try disabling the CSP module temporarily – errors reported by CSP may create a magepack error.

2.3.3 composer extra patches config
[text] "extra": {
"magento-force": "override",
"composer-exit-on-patch-failure": true,
"patches": {
"magento/magento2-base": {
"[Performance] Fix missing shims and phtml files with mage-init directives (https://github.com/magento/magento2/commit/db43c11c6830465b764ede32abb7262258e5f574)": "patches/composer/M233/github-pr-4721-base.diff",
"Refactor JavaScript mixins module https://github.com/magento/magento2/pull/25587": "patches/composer/M233/github-pr-25587-base.diff"
},
"magento/module-braintree": {
"[Performance] Fix missing shims and phtml files with mage-init directives (https://github.com/magento/magento2/commit/db43c11c6830465b764ede32abb7262258e5f574)": "patches/composer/M233/github-pr-4721-braintree.diff"
},
"magento/module-catalog": {
"[Performance] Fix missing shims and phtml files with mage-init directives (https://github.com/magento/magento2/commit/db43c11c6830465b764ede32abb7262258e5f574)": "patches/composer/M233/github-pr-4721-catalog.diff"
},
"magento/module-customer": {
"[Performance] Fix missing shims and phtml files with mage-init directives (https://github.com/magento/magento2/commit/db43c11c6830465b764ede32abb7262258e5f574)": "patches/composer/M233/github-pr-4721-customer.diff"
},
"magento/module-msrp": {
"[Performance] Fix missing shims and phtml files with mage-init directives (https://github.com/magento/magento2/commit/db43c11c6830465b764ede32abb7262258e5f574)": "patches/composer/M233/github-pr-4721-msrp.diff"
},
"magento/module-paypal": {
"[Performance] Fix missing shims and phtml files with mage-init directives (https://github.com/magento/magento2/commit/db43c11c6830465b764ede32abb7262258e5f574)": "patches/composer/M233/github-pr-4721-paypal.diff"
},
"magento/module-theme": {
"[Performance] Fix missing shims and phtml files with mage-init directives (https://github.com/magento/magento2/commit/db43c11c6830465b764ede32abb7262258e5f574)": "patches/composer/M233/github-pr-4721-theme.diff",
"fix_baler_jquery_cookie": "https://gist.github.com/tdgroot/f95c398c565d9bbb83e0a650cdf67617/raw/69ee2d001ff509d25d1875743e417d914e20fd85/fix_baler_jquery_cookie.patch"
}
}
},
[/text]
2.3.4, 2.3.5 composer extra patches config
[text] "extra": {
"magento-force": "override",
"composer-exit-on-patch-failure": true,
"patches": {
"magento/magento2-base": {
"Refactor JavaScript mixins module https://github.com/magento/magento2/pull/25587": "patches/composer/M234/github-pr-25587-base.diff"
}
}
},
[/text]

Now run composer update Magento 2 will be patched and we are good to go.

Let’s get ready to bundle

Magepack needs to analyse pages from your Magento 2 store to determine the Javascript files your store is using and how they can be bundled. It saves this information in a configuration file called magepack.config.js. The magepack config file is generated by analysing three different type of pages from your Magento 2 store, a cms page i.e. the home page, a category page and a product page. This is done using the magepack generate command and supplying three store urls.

magepack generate --cms-url="http://magento2.gaiterjones.com/" --category-url="http://magento2.gaiterjones.com/en/buy-x-get-y.html" --product-url="http://magento2.gaiterjones.com/en/affirm-water-bottle.html"

Run this command in the root folder of your Magento 2 installation to create the magepack.config.js file. It’s worth noting that you could run this generate command from any system, and just copy the generated config file to your Magento 2 server.

If you take a look at magepack.config.js you will see it contains references to all the javascript required to load Magento pages. Below is an example from a product page.

example product section from magepack.config.js
[code language=”javascript”] name: ‘product’,
modules: {
‘Magento_Catalog/js/price-utils’: ‘Magento_Catalog/js/price-utils’,
‘Magento_Catalog/js/price-box’: ‘Magento_Catalog/js/price-box’,
‘Magento_Wishlist/js/add-to-wishlist’: ‘Magento_Wishlist/js/add-to-wishlist’,
‘Magento_Cookie/js/require-cookie’: ‘Magento_Cookie/js/require-cookie’,
‘Magento_Swatches/js/configurable-customer-data’: ‘Magento_Swatches/js/configurable-customer-data’,
‘Magento_Review/js/error-placement’: ‘Magento_Review/js/error-placement’,
‘Magento_Review/js/process-reviews’: ‘Magento_Review/js/process-reviews’,
‘Elgentos_LargeConfigProducts/js/swatch-renderer-mixin’: ‘Elgentos_LargeConfigProducts/js/swatch-renderer-mixin’,
‘text!Magento_Theme/templates/breadcrumbs.html’: ‘Magento_Theme/templates/breadcrumbs.html’,
‘magnifier/magnifier’: ‘magnifier/magnifier’,
‘magnifier/magnify’: ‘magnifier/magnify’,
‘Magento_Catalog/js/gallery’: ‘Magento_Catalog/js/gallery’,
‘Magento_ProductVideo/js/load-player’: ‘Magento_ProductVideo/js/load-player’,
‘Magento_ProductVideo/js/fotorama-add-video-events’: ‘Magento_ProductVideo/js/fotorama-add-video-events’,
‘Magento_Theme/js/model/breadcrumb-list’: ‘Magento_Theme/js/model/breadcrumb-list’,
‘Magento_Theme/js/view/breadcrumbs’: ‘Magento_Theme/js/view/breadcrumbs’,
‘Magento_Theme/js/view/add-home-breadcrumb’: ‘Magento_Theme/js/view/add-home-breadcrumb’,
‘Magento_Catalog/js/product/breadcrumbs’: ‘Magento_Catalog/js/product/breadcrumbs’,
‘jquery/jquery.parsequery’: ‘jquery/jquery.parsequery’,
‘Magento_ConfigurableProduct/js/options-updater’: ‘Magento_ConfigurableProduct/js/options-updater’,
‘Magento_Review/js/validate-review’: ‘Magento_Review/js/validate-review’,
‘Magento_Swatches/js/swatch-renderer’: ‘Magento_Swatches/js/swatch-renderer’,
‘Magento_Catalog/product/view/validation’: ‘Magento_Catalog/product/view/validation’,
‘Magento_Catalog/js/product/view/product-ids’: ‘Magento_Catalog/js/product/view/product-ids’,
‘Magento_Catalog/js/product/view/product-ids-resolver’: ‘Magento_Catalog/js/product/view/product-ids-resolver’,
‘Magento_Catalog/js/catalog-add-to-cart’: ‘Magento_Catalog/js/catalog-add-to-cart’,
‘Magento_Catalog/js/validate-product’: ‘Magento_Catalog/js/validate-product’,
‘Magento_Catalog/js/product/view/provider’: ‘Magento_Catalog/js/product/view/provider’,
‘text!mage/gallery/gallery.html’: ‘mage/gallery/gallery.html’,
‘text!Magento_InstantPurchase/template/confirmation.html’: ‘Magento_InstantPurchase/template/confirmation.html’,
‘Magento_InstantPurchase/js/view/instant-purchase’: ‘Magento_InstantPurchase/js/view/instant-purchase’,
‘Magento_Review/js/view/review’: ‘Magento_Review/js/view/review’,
‘fotorama/fotorama’: ‘fotorama/fotorama’,
‘mage/gallery/gallery’: ‘mage/gallery/gallery’,
‘text!Magento_InstantPurchase/template/instant-purchase.html’: ‘Magento_InstantPurchase/template/instant-purchase.html’
}
},
[/code]

All that remains now is for us to create the bundle files and deploy them for all our store views and themes. This is simply done with the magepack bundle command which you can execute from the Magento installation root folder. If you are running in development mode, deploy frontend static files first.

magepack bundle

Magepacl bundle command
Magepack bundle command

Finally enable Magepack Javascript bundling in admin :

Stores – Configuration – Advanced – Developer – Javascript Settings

Enable Magepack bundling in admin
Enable Magepack bundling in admin

If you are in production mode these options will be hidden in admin. To enable Magepack Javascript bundling from the command line use

bin/magento config:set dev/js/enable_magepack_js_bundling 1

Note that you should also enable the other Javascript optimisation options here including minfy javascript files and move js code to the bottom of the page – but don’t enable the default bundling!

MagePack Javascript bundling should now be enabled. To check it’s working go to a Magento 2 product page and look at the source code, do a search for “bundle” and you should see the magepack javascript bundles

Now refresh the page and have a look at your network log

After bundling there are only 7 js requests on the product page
After bundling there are only 7 js requests on the product page

Instead of loading 194 Javascript files, the product page now loads 7, Magepack has bundled all the Javascript into two main bundle files.

I guess it’s now time to look at the PageSpeed Lighthouse performance reports for your optimised Magento 2 pages. If you are using the Chrome browser simply run a Lighthouse report from the DevTools page. You can also use Googles PageSpeed insights tool at https://developers.google.com/speed/pagespeed/insights/

This is the improvement I saw in a live production Magento 2 site

Performance results before and after bundling
Performance results before and after bundling

If you don’t see a big improvement remember there are a lot of other factors taken into Lighthouse performance reports. Work through the report and try to find out where you can make further improvements.

Deployment in production

Whenever you flush your sites static files you will need to remember to run magepack bundle again. In production mode you should add this to your deployment process

bin/magento setup:upgrade
bin/magento setup:di:compile
bin/magento setup:static-content:deploy en_GB --area adminhtml
bin/magento setup:static-content:deploy en_GB --area frontend --theme MY/Theme -f
magepack bundle
bin/magento cache:clean

Top Tip – if for any reason you want to generate the magepack.config.js bundle config again remember to disable the Magepack module first!

Testing and Troubleshooting

You should test your store thoroughly to make sure there are no Javascript problems caused by the bundling process. Magepack cannot always 100% bundle all the Javascript required by some pages. Check your web browser console for errors. If you find some features of your store are not working, try and identify if the code was included in the magepack.config.js file. Try removing the code from the bundle and test again.

Selective Bundling

If you do not want to bundle at checkout, or any other specific pages take a look at the isEnabled method in the Block\BundlesLoader class. This method determines whether the magepack module is enabled and if bundling should be activated. Simply detect any page here i.e. checkout pages and return false to disable bundling at checkout.


        \Magento\Framework\App\Request\Http $request,
        ....
        $this->request = $request;
        ....
public function isEnabled()
{
        if ($this->request->getFullActionName() == 'checkout_index_index') {

            // disable for checkout
            //
            return false;
        }

...

Magepack for Magento Cloud

You need to have magepack.config in repo, add this to magento.app.yaml:

echo "\n================================== Install and configure Magepack - Start ==============================="
unset NPM_CONFIG_PREFIX
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.35.1/install.sh | bash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh"  ] && \. "$NVM_DIR/nvm.sh"
nvm install --lts=dubnium
npm install -g magepack
echo "\n================================== create bundle ==============================="
magepack bundle
echo "\n================================== Install and configure Magepack - END ================================="

Thanks to Solteq for this information.

Magepack is pretty new with updates being made regularly, be sure to check out the projects GitHub page for new issues.

References

https://github.com/magesuite/magepack
https://github.com/magesuite/magepack-magento
https://www.integer-net.com/magento-2-javascript-bundling-integer-net-2/
https://gist.github.com/tdgroot/f95c398c565d9bbb83e0a650cdf67617
https://github.com/integer-net/magento2-requirejs-bundling
https://developers.google.com/speed/pagespeed/module

PageSpeed optimisation in Magento 1

]]> https://blog.gaiterjones.com/magento-2-javascript-bundling-with-magepack/feed/ 37 Magento 2.2.X Development and Upgrades https://blog.gaiterjones.com/magento-2-2-x-development-and-upgrades/ https://blog.gaiterjones.com/magento-2-2-x-development-and-upgrades/#respond Thu, 03 May 2018 08:55:14 +0000 http://blog.gaiterjones.com/?p=1883 Continue reading]]> Today (2nd May 2018) Magento released Magento CE 2.2.4. Recently the Magento Devs have been pretty busy pushing out 2.2.x updates and Magento 2 is maturing nicely. I am still getting to grips with Magento 2 and don’t expect to release a live site for at least another year so it makes sense to develop with the latest version of Magento. My Magento 2.2.x development environment is built on Docker and installing new Magento 2 dev shops is a breeze:

  1. git clone https://github.com/gaiterjones/docker-magento2-2
  2. cd docker-magento2-2/magento
  3. edit .env
  4. docker-compose build
  5. docker-compose up -d
  6. http://magento2.dev.com

Upgrading Magento is always a scary experience – Magento 2 is even more terrifying so I recommend you get used to doing updates with your dev systems. If you are manually upgrading from a previous 2.x version, you can use composer within the docker container:

  1. Take a backup
  2. Enter maintenance mode
  3. edit compose.json
    1. change version
  4. flush redis
    1. redis-cli flushall in the redis containers
  5. composer update
  6. rm -rf var/cache/* var/page_cache/* var/generation/* var/di/*
  7. chmod +x bin/magento
  8. php bin/magento cache:flush
  9. php bin/magento setup:upgrade
  10. php bin/magento indexer:reindex

If the upgrade breaks any sample data reinstall with php -f bin/magento sampledata:deploy

I regularly update my Magento 2 docker containers with each version update check them out at https://github.com/gaiterjones/docker-magento2-2

]]> https://blog.gaiterjones.com/magento-2-2-x-development-and-upgrades/feed/ 0 Magento 2 theme blank SASS parent and child setup and installation https://blog.gaiterjones.com/magento2-theme-blank-sass-parent-and-child-setup-and-installation/ https://blog.gaiterjones.com/magento2-theme-blank-sass-parent-and-child-setup-and-installation/#respond Mon, 03 Jul 2017 13:04:59 +0000 http://blog.gaiterjones.com/?p=1712 Continue reading]]> With CE 1.9 Magento introduced the SASS stylesheet language to aid frontend development. With Magento 2.x for reasons best known to the developers they switched from SASS to LESS. At the end of the day both LESS and SASS are compiled into good old CSS but if your stylesheet language of preference is SASS or you are new to Magento and are looking for a good starting block to build a new theme from then it is well worth taking a look at SnowDogApps theme blank sass project.

The Less to Sass Community Project is officially supported by Magento and Theme Blank SASS is the result of this project – a (production ready) Magento 2 blank theme converted to SASS.

Together with the accompanying Fronttools pack you get the development tools and extendable SASS based blank theme you need to get you started building your own custom theme.

Why use theme blank SASS ?

  • If you are coming from Magento 1.9 then you already are familiar with SASS
  • Magento 2.x may possibly move to SASS in the future
  • LUMA is not designed to be extended as a blank theme
  • Essential development tools such as browsersync are included

How to install theme blank SASS parent theme and Fronttools

You can be up and running with theme blank SASS and fronttools in a few minutes. Here is a step by step theme blank sass and frontools installation guide tested on my Magento 2.x docker containers.

I am installing the parent theme and fronttools directly into my docker Magento 2 php-apache container with a bash script :

curl -o- http://gaiterjones.com/dropbox/magento2/theme-blank-sass/install-theme-blank-sass.sh | bash
 
script to install theme-blank-sass and fronttools

[bash] #!/bin/bash
# GAITERJONES
# blog.gaiterjones.com
# script to install theme-blank-sass and fronttools
# for docker service https://github.com/gaiterjones/docker-magento2
#
set -e
# variables
RED=’\033[0;31m’
NC=’\033[0m’

# CHANGE THIS
MAGENTO_DIR=’/var/www/dev/magento2′
NVM_HOME=’/var/www’

# start
printf "${RED}Installing theme-blank-sass into Magento Root: ${MAGENTO_DIR}…\n\n${NC}"
cd ${MAGENTO_DIR}
composer require snowdog/theme-blank-sass
composer require snowdog/frontools
${MAGENTO_DIR}/bin/magento setup:upgrade
${MAGENTO_DIR}/bin/magento cache:clean

printf "${RED}Installing nvm…\n${NC}"
cd /tmp
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.1/install.sh | bash
export NVM_DIR=${NVM_HOME}/.nvm
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
nvm install –lts
nvm use –lts
printf "${RED}Installing gulp…\n${NC}"
npm install -g gulp-cli

printf "${RED}Installing frontools…\n${NC}"
cd ${MAGENTO_DIR}/vendor/snowdog/frontools
npm install
gulp setup
curl -o "${MAGENTO_DIR}/dev/tools/frontools/config/themes.json" http://gaiterjones.com/dropbox/magento2/theme-blank-sass/browser-sync.json
curl -o "${MAGENTO_DIR}/dev/tools/frontools/config/themes.json" http://gaiterjones.com/dropbox/magento2/theme-blank-sass/themes-blank-sass-parent.json
curl -o "${MAGENTO_DIR}/vendor/snowdog/theme-blank-sass/web/images/logo.svg" http://gaiterjones.com/dropbox/magento2/theme-blank-sass/logo-theme-blank-sass.svg

printf "${RED}Generating theme-blank-sass styles…\n${NC}"
gulp styles

printf "${RED}Configuring Snowdog/blank theme.\n${NC}"
THEME_ID="$(n98-magerun2.phar dev:theme:list –format=csv \
| grep ‘Snowdog/blank’ | cut -d, -f1)" \
; test -n "${THEME_ID}" \
&& n98-magerun2.phar config:set design/theme/theme_id "${THEME_ID}"
${MAGENTO_DIR}/bin/magento cache:clean

# done
printf "${RED}Installation complete.\n\n${NC}"
[/bash]

Here are the script commands broken down into some more detail :

Set the install environment variables

cd ${MAGENTO_DIR}
MAGENTO_DIR='/var/www/dev/magento2'
NVM_HOME='/var/www'

MAGENTO_DIR is the path to the magento root and NVM_HOME the path to the NVM installation home folder – this is the home folder for the user that installs NVM

Install the theme and tools with composer, upgrade Magento and refresh caches.

composer require snowdog/theme-blank-sassinstall theme blank sass
composer require snowdog/frontoolsinstall fronttools

${MAGENTO_DIR}/bin/magento setup:upgradeupdate Magento
${MAGENTO_DIR}/bin/magento cache:cleanrefresh caches

Install NVM and gulp-cli

cd /tmp
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.1/install.sh | bash
export NVM_DIR=${NVM_HOME}/.nvm
 [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
nvm install --lts
nvm use --lts
npm install -g gulp-cli

Install frontools.

cd ${MAGENTO_DIR}/vendor/snowdog/frontools
npm install

You run the fronttools setup with gulp setup, this will copy the project sample configuration files to ${MAGENTO_DIR}/dev/tools/frontools/config/.  You must edit themes.json, for the parent theme.

{
  "blank": {
    "src": "vendor/snowdog/theme-blank-sass",
    "dest": "pub/static/frontend/Snowdog/blank",
    "locale": ["en_GB"],
    "postcss": ["plugins.autoprefixer()"],
    "ignore": [".test"]
  }
}

The script is copying my themes and browser-sync json configuration files via curl.

gulp setup
curl -o "${MAGENTO_DIR}/dev/tools/frontools/config/themes.json" http://gaiterjones.com/dropbox/magento2/theme-blank-sass/browser-sync.json
curl -o "${MAGENTO_DIR}/dev/tools/frontools/config/themes.json" http://gaiterjones.com/dropbox/magento2/theme-blank-sass/themes-blank-sass-parent.json
gulp styles

Next I compile the SASS to CSS and install it using

gulp styles

Finally I am using n98-magerun2 to enable the new theme, and refresh the Magento caches.

THEME_ID="$(n98-magerun2.phar dev:theme:list --format=csv \
  | grep 'Snowdog/blank' | cut -d, -f1)" \
  ; test -n "${THEME_ID}" \
  && n98-magerun2.phar config:set design/theme/theme_id "${THEME_ID}"
${MAGENTO_DIR}/bin/magento cache:clean


Theme blank SASS parent theme is now installed. You can watch the install process below.

magento 2 theme blank sass install video

We used the gulp setup, and gulp styles tasks above – other useful tasks include

  • gulp clean
    • Removes /pub/static directory content.
  • gulp dev
    • Runs browserSync and inheritance, babel, styles, watch tasks.

See the fronttools project page for a full list of gulp tasks or use the gulp default task.

How to install a theme blank SASS child theme

Usual Magento best practice dictates that instead of editing the parent them we extend it with a child theme. You extend Snowdog/blank the same way as any other theme.

<theme xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Config/etc/theme.xsd">
    <title>Gaiterjones/blank</title>
    <parent>Snowdog/blank</parent>
    <media>
        <preview_image>media/preview.jpg</preview_image>
    </media>
</theme>

I tested this with my Docker environment and used the following script to extend theme blank SASS with my Gaiterjones/blank child theme.

curl -o- http://gaiterjones.com/dropbox/magento2/theme-blank-sass/child/install-child-theme-blank-sass.sh | bash

script to install theme-blank-sass child theme
[bash] #!/bin/bash
# GAITERJONES
# blog.gaiterjones.com
# script to install example theme-blank-sass child theme
#
set -e
# variables
RED=’\033[0;31m’
NC=’\033[0m’

# CHANGE THIS
MAGENTO_DIR=’/var/www/dev/magento2′
NVM_HOME=’/var/www’

# start
printf "${RED}Installing theme-blank-sass child theme into Magento Root: ${MAGENTO_DIR}…\n\n${NC}"
n98-magerun2.phar maintenance:enable
cd ${MAGENTO_DIR}/app/design/frontend
curl https://pe.terjon.es/dropbox/magento2/theme-blank-sass/child/gaiterjones-blank-child-theme-blank-sass.tar | tar x
${MAGENTO_DIR}/bin/magento setup:upgrade
cd ${MAGENTO_DIR}/vendor/snowdog/frontools

printf "${RED}Configuring Gaiterjones/blank theme.\n${NC}"
THEME_ID="$(n98-magerun2.phar dev:theme:list –format=csv \
| grep ‘Gaiterjones/blank’ | cut -d, -f1)" \
; test -n "${THEME_ID}" \
&& n98-magerun2.phar config:set design/theme/theme_id "${THEME_ID}"

printf "${RED}Installing new themes.json config…\n${NC}"
curl -o "${MAGENTO_DIR}/dev/tools/frontools/config/themes.json" https://gaiterjones.com/dropbox/magento2/theme-blank-sass/child/themes-blank-sass-parent-child.json
printf "${RED}Generating theme-blank-sass styles…\n${NC}"
export NVM_DIR=${NVM_HOME}/.nvm
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
gulp styles
${MAGENTO_DIR}/bin/magento cache:clean
n98-magerun2.phar maintenance:disable
# done
printf "${RED}Installation complete.\n\n${NC}"

[/bash]

The script extracts the Gaiterjones/blank child theme source files to app/design/frontend/Gaiterjones/blank and updates Magento. We need to also update the fronttools themes.json config file to let fronttools know where our child theme lives

{
  "blank": {
    "src": "vendor/snowdog/theme-blank-sass",
    "dest": "pub/static/frontend/Snowdog/blank",
    "locale": ["en_GB"],
    "postcss": ["plugins.autoprefixer()"],
    "ignore": [".test"]
  },
  "gaiterjones-blank": {
    "src": "app/design/frontend/Gaiterjones/blank",
    "dest": "pub/static/frontend/Gaiterjones/blank",
    "locale": ["en_GB"],
    "localeOverwrites": true,
    "parent": "blank",
    "postcss": ["plugins.autoprefixer()"]
  }  
}

Next I am activating the theme with n98-magerun2 and then I simply need to refresh the caches and run gulp styles again to create the CSS for our child theme.

gulp styles
${MAGENTO_DIR}/bin/magento cache:clean

Fronttools compiles and installs the CSS for our child theme which is now installed and ready to view.

magento 2 theme blank sass child theme install video

You can see theme blank SASS in action on my development site.

]]> https://blog.gaiterjones.com/magento2-theme-blank-sass-parent-and-child-setup-and-installation/feed/ 0 Docker Magento CE v2.4.x Development Deployment with PHP7.4, Apache2.4, REDIS, VARNISH, RabbitMQ, Scaleable https://blog.gaiterjones.com/docker-magento-2-development-deployment-php7-apache2-4-redis-varnish-scaleable/ https://blog.gaiterjones.com/docker-magento-2-development-deployment-php7-apache2-4-redis-varnish-scaleable/#comments Sun, 25 Jun 2017 13:31:05 +0000 http://blog.gaiterjones.com/?p=1694 Continue reading]]> Installing Magento 2 in a development environment can be time consuming. This is my Magento 2 development deployment for Docker which can be used to quickly create a new Magento 2 Open Source (formerly known as CE) environment, or switch between persistent development environment volumes.

Features

  • PHP 7.4 and Apache 2.4 built on the latest PHUSION docker images
  • 2 Redis servers for Session and System (including FPC)
  • MYSQL 8
  • PhpMyAdmin
  • Varnish 6 for FPC
  • RabbitMQ
  • Management container and Scale Manager for scaled service dynamic load balancing
  • composer, n98-magerun2, PHP Redis Admin, XDEBUG
  • Magento 2.4.2 (Feb 2021)

Requirements

  • Host server with min 2GB RAM
  • Docker Engine
  • Docker Compose 3.1

Base Images / Source for 2.4 deployment

Deployment

Here are the steps to deploy this Magento 2.4 development environment:

  • Clone the source file repository from, GitHub
    • 2.4
      • git clone https://github.com/gaiterjones/docker-magento2
  • Change into the Magento 2 directory
    • cd magento2
  • Edit the .env file and set the working environment for your deployment :
    • APPDOMAIN=dev.com
      • the domain name for the magento installation
    • SMTP=XXX
      • the smtp gateway for your deployment, required for Magento email.
    • MAGENTO_URL=http://magento2.dev.com
      • the Magento URL for your deployment
    • MAGENTO_REPO_USERNAME/PASSWORD
      • your Magento repo authentication details required for Magento installation
    • CONTAINERDATA=/home/docker/data/
      • Used for persistent container data
  • Edit docker-compose.yml
    • You can remove the Manager and Memcache service here if you do not want to deploy the scale manager.
    • for mysql persistent data uncomment the following from the mysql image
      • #– “${CONTAINERDATA}/${PROJECT_NAME}/mysql/dev1:/var/lib/mysql”
    • If you are using Nginx as a container reverse proxy change the ports and networks for the varnish service accordingly.
  • Generate a Varnish Secret
    • You can create a new varnish secret by running newsecret.sh in the varnish folder.
  • BUILD
    • docker-compose build
  • START
    • docker-compose up -d
      • Allow 60 seconds after starting for composer to initialise
  • Install Magento Sample Data (optional)
    • docker exec -it magento2_php-apache_1 install-sampledata
      • ignore error Can’t run this operation: deployment configuration is absent.
  • Install Magento
    • docker exec -it magento2_php-apache_1 install-magento

Script install

For a quick test, spin up a Type 2 packet.net server running Ubuntu LTS 20.04, install Docker CE and docker compose and then run

docker exec -it --user magento magento2_php-apache_1 su root -c "curl -o- https://gaiterjones.com/dropbox/magento2/docker/install.sh | bash"

 

magento 2 docker containers install video

Testing

  • frontend
      • http://magento2.dev.com
  • backend
      • http://magento2.dev.com/admin/
        • login using the credentials you set in .env
      • Configure Varnish FPC
        • Stores-> Configuration -> Advanced -> Full Page Cache
    • Test from containers with curl (optional)
      • curl -I -H “host: magento2.dev.com” magento2_php-apache_1:80
      • curl -H “host: magento2.dev.com” magento2_php-apache_1:80/healthcheck.php
    • n98-magerun2
      • docker exec -it –user magento magento2_php-apache_1 /usr/local/bin/n98-magerun2.phar

Persistent volumes

Your development deployment data will be lost when you restart the php-apache or sql containers. If you want data to persist after installation perform the following steps:

  • Tar the Magento source files in the php-apache container
    • docker exec -it –user magento magento2_php-apache_1 tar -cvf /home/data/magento2.tar /var/www/dev
    • The tar file will be saved to the docker container data folder specified in .env
  • Stop the containers
    • docker-compose down -v
  • Untar to host
    • tar -xvf magento2.tar
  • Move the /var/www/dev folder to your persistent data folder i.e. /home/docker/data/magento2/dev1
  • Uncomment the data volume in docker-compose.yml
    • #- “${CONTAINERDATA}/${PROJECT_NAME}/dev1:/var/www/dev”
  • Restart containers, Magento data is now persistent.

Scaling php-apache

You can scale the php-apache Magento service with

docker-compose scale php-apache=X

See this post for more information.

manager container

In production I prefer to use a management container for all Magento admin and cli commands. The manager container can be used as the main administration container for your production environment. You should move all cron jobs to the manager and you can also move web administration to the manager container. This will free up the php-apache container just to be used to serve frontend web requests and allows you to use the scale function. You will also notice that separating cron tasks into a separate container reduces the memory requirement of the php-apache container significantly.

To start an admin bash session in the management container use

docker exec -i-t --user magento magento2_php-apache_1 /bin/bash

References

]]> https://blog.gaiterjones.com/docker-magento-2-development-deployment-php7-apache2-4-redis-varnish-scaleable/feed/ 18 Docker Mono Host Magento 2 Service Scaling and Dynamic Load Balancing with VARNISH https://blog.gaiterjones.com/docker-mono-host-magento-2-service-scaling-and-dynamic-load-balancing-with-varnish/ https://blog.gaiterjones.com/docker-mono-host-magento-2-service-scaling-and-dynamic-load-balancing-with-varnish/#comments Sun, 18 Jun 2017 16:30:30 +0000 http://blog.gaiterjones.com/?p=1674 Continue reading]]> As I started building Docker service container images for Magento 2 CE development I got interested in the concept of dynamic service load balancing on a single host. Having already created a dynamic load balancer using NGINX I wanted to do the same for Magento 2 using Varnish.

My Magento 2 service container images include REDIS services for the Magento session and system cache and a Varnish service for the full page cache. Varnish is an HTTP accelerator designed for content-heavy dynamic web sites and is the recommended full page cache system for Magento 2.

Magento 2 Docker Network

Varnish is configured via the Varnish Configuration Language (VCL) config file. By configuring multiple backend servers in VCL and grouping them into a director we can create a Varnish load balancer.

Varnish Magento 2 load balancing VCL
[text]
backend magento2_php_apache_1 {
.host = "x.x.x.2";
.port = "80";
.probe = {.request =
"GET /healthcheck.php HTTP/1.1" "Host: magento2.gaiterjones.com" "Connection: close" "Accept: text/html";.timeout = 1s;.interval = 30s;.window = 10;.threshold = 8;}
}
backend magento2_php_apache_2 {
.host = "x.x.x.8";
.port = "80";
.probe = {.request =
"GET /healthcheck.php HTTP/1.1" "Host: magento2.gaiterjones.com" "Connection: close" "Accept: text/html";.timeout = 1s;.interval = 30s;.window = 10;.threshold = 8;}
}
backend magento2_php_apache_3 {
.host = "x.x.x.10";
.port = "80";
.probe = {.request =
"GET /healthcheck.php HTTP/1.1" "Host: magento2.gaiterjones.com" "Connection: close" "Accept: text/html";.timeout = 1s;.interval = 30s;.window = 10;.threshold = 8;}
}
backend magento2_php_apache_4 {
.host = "x.x.x.11";
.port = "80";
.probe = {.request =
"GET /healthcheck.php HTTP/1.1" "Host: magento2.gaiterjones.com" "Connection: close" "Accept: text/html";.timeout = 1s;.interval = 30s;.window = 10;.threshold = 8;}
}
backend magento2_php_apache_5 {
.host = "x.x.x.9";
.port = "80";
.probe = {.request =
"GET /healthcheck.php HTTP/1.1" "Host: magento2.gaiterjones.com" "Connection: close" "Accept: text/html";.timeout = 1s;.interval = 30s;.window = 10;.threshold = 8;}
}
sub vcl_init {
new cluster1 = directors.round_robin();
cluster1.add_backend(magento2_php_apache_1);
cluster1.add_backend(magento2_php_apache_2);
cluster1.add_backend(magento2_php_apache_3);
cluster1.add_backend(magento2_php_apache_4);
cluster1.add_backend(magento2_php_apache_5);
}
[/text]

Scale Manager

I use a service container called php-apache to run the Magento core code. Docker compose allows you to scale containers on a single host, so for example with the command

docker-compose scale php-apache=5

I can quickly scale the Magento2 php-apache service up to 5 containers, that is 5 instances of the php-apache service running the Magento2 core php code and apache web server.

The scale manager service monitors docker processes and detects when the php-apache service is scaled up or down. When a change is detected it dynamically creates the VCL configuration required to tell Varnish about the changes and applies these to the Varnish server.

Magento 2 Scale Demo

Magento 2 Scale Demo

In the video below I start with one running instance of the Magento 2 service, I then scale the service up to 5 and using the scale demo page (which is not cached by Varnish) on my Magento 2 dev site to analyse the scale manager output data and display the running Magento 2 service containers.

As I refresh the page the Magento 2 container rendering the page is highlighted in yellow. you can see how Varnish round robins between the 5 available containers. Finally I scale the service back down and manually update the Varnish server config via the command line php script on the manager.

Another way to see the round robin connection distribution from varnish is to load up a phpinfo() page, you will see the SERVER_ADDR variable changing with each container that processes the request.

docker scale service dynamic load balancing with nginx demo video

Would you actually run 5 Magento 2 php-apache services? Good question! For true resilience and load balancing you would need multiple hosts and a Docker swarm. Multiple service containers on a single host do at least provide a simple level of redundancy and host server load balancing, I would definitely consider using it in production especially with a multi core server.

Many thanks to Fabrizio Balliano for the inspiration and assistance.

See this post for instructions on how to deploy this Magento 2 development environment.

]]> https://blog.gaiterjones.com/docker-mono-host-magento-2-service-scaling-and-dynamic-load-balancing-with-varnish/feed/ 1