This post was last updated 3 years 7 months 1 day ago, some of the information contained here may no longer be actual and any referenced software versions may have been updated!

TL;DR

Migrating the Magento 2 catalog search engine to Smile ElasticSuite will resolve pretty much all the issues you might be experiencing with Magento 2 native ElasticSearch catalog search so go ahead and Jump to the ElasticSuite installation.

If you are new to ElasticSearch or want to find out how to customise ElasticSearch in Magento 2 read on!

Magento Catalog Search

Up until version 2.3 the default catalog search engine for Magento used the MySql Magento database. Using MySql for search was adequate but it lacked the features and scalability of enterprise search solutions. In version 2.3 Magento built in support for ElasticSearch as the catalog search engine and announced in 2019 that MySql search would be deprecated. As of version 2.4 in July 2020 the MySql catalog search engine was removed completely from Magento 2.

Native support for ElasticSearch was good news for Merchants, Elasticsearch is a java based open-source, RESTful, distributed search and analytics engine built on Apache Lucene. Since its release in 2010, Elasticsearch has quickly become the most popular search engine.

It’s worth mentioning that ElasticSearch in Magento 2 is not just used for user full text search queries, the catalog search engine is responsible for returning all catalog queries including category products and filtered navigation product lists.

For customers ElasticSearch should provide faster and more relevant search experiences – the problem for merchants is that out of the box Magento 2 ElasticSearch just doesn’t do this – catalog search results and relevance have a tendency to be extremely poor. The built in search struggles to provide accurate and relevant results for simple queries such as SKUs.

Whilst MySql catalog search had some admin options to refine search results, there are no options available to customise catalog search with ElasticSearch. ElasticSearch is a great search engine but the native Magento 2 catalog full text search implementation is very disappointing.

Let’s look at ways to customise ElasticSearch catalog search in Magento using your own module to improve some areas of search relevance.

Simple SKU Search

Poor search results or search relevance with native Magento ElasticSearch is very apparent when searching for SKUs. Let’s look at a simple SKU search for one of the sample products provided in the Magento 2 sample data.

Article MH03 is a range of Hoodies. Searching for ‘MH03’ correctly returns all 16 products. But what if you want to search for MH03-XL?

Refined SKU Search for Magento 2 sample data

Here we see that 112 items are returned when in fact only the first 3 were 100% matches for the search term. Native search really struggles with search terms containing special characters such as the hyphen commonly used in SKUs resulting in extremely poor search results. To look at why we are seeing so many results returned we need to look at the relevance score of the search results.

Customise Elastic Search

To capture the data returned by an ElasticSearch frontend full text search query we need to create a debug plugin for Magento\Elasticsearch\SearchAdapter\ResponseFactory that will let us analyse the search data and log it.

<type name="Magento\Elasticsearch\SearchAdapter\ResponseFactory">
<plugin disabled="false" sortOrder="2" name="gaiterjones_elasticsearch_queryresultsdebug" type="Gaiterjones\Elasticsearch\Plugin\QueryResultsDebug"/>
</type>

beforeCreate plugin for SearchAdapterResponseFactory

[code language=”php”] public function beforeCreate(\Magento\Elasticsearch\SearchAdapter\ResponseFactory $subject, $result)
{
if($this->debug)
{
if(!is_array($result) || empty($result)) {return false;}

$scores=array();

foreach ($result[‘documents’] as $rawDocument) {
$this->logger->debug(‘ELASTIC_SEARCH_QUERY_RESULTS_DEBUG’,$rawDocument);
array_push($scores,$rawDocument[‘_score’]);
}

if (count($result[‘documents’]) > 0)
{
$_debug=array(
‘results’ => count($result[‘documents’]),
‘scores’ => $scores,
‘min_relevance_score’ => $this->configuration->getMinScore(),
‘min_score’ => min($scores),
‘max_score’ => max($scores)
);

$this->logger->debug(‘ELASTIC_SEARCH_QUERY_RESULTS_DEBUG’,$_debug);
}
}
}
[/code]

The plugin dumps the search data to the debug log in var/log, this allows us to look more closely at the ElasticSearch results for the MH03-XL SKU full text search :

Array
(
    [results] => 112
    [scores] => Array
        (
            [0] => 40.57959
            [1] => 40.57959
            [2] => 40.57959
            [3] => 29.099976
            [4] => 12.694372
            SNIP...
            [108] => 8.300152
            [109] => 8.300152
            [110] => 7.9659967
            [111] => 7.9659967
        )

    [min_relevance_score] => 1
    [min_score] => 7.9659967
    [max_score] => 40.57959
)

The debug shows the ElasticSearch rawdocument score for the 112 search results, you can see that the score value for the search ranges from 7.9 to 40.5 with the most relevant results having a higher score. If we were to define a minimum relevance score of 40 the search results would be much more accurate.

We can do this with another plugin :

<type name="Magento\Elasticsearch\Elasticsearch5\SearchAdapter\Mapper">
<plugin disabled="false" sortOrder="1" name="gaiterjones_elasticsearch_searchadapter_mapperplugin" type="Gaiterjones\Elasticsearch\Plugin\Elasticsearch5\SearchAdapter\MapperPlugin"/>
</type>

aroundBuildQuery plugin for SearchAdapterMapperPlugin

[code language=”php”] public function aroundBuildQuery(
Mapper $subject,
callable $proceed,
RequestInterface $request
) {
$searchQuery = $proceed($request);

if ($request->getName() === ‘quick_search_container’) {
$searchQuery[‘body’][‘min_score’] = $this->configuration->getMinScore();
}

return $searchQuery;
}
[/code]

Here we set a min_score value for the search query. Setting this to 40 would return just three results for the MH03-XL SKU search.

SKU Search for Magento 2 sample products with min_score value

This looks much better, we can improve the relevance of the search results by filtering out results that have a low ElasticSearch score. The tricky part here is deciding what the minimum score value should be – it can be difficult to find a value that works well for different search queries.

Another useful ElasticSearch customisation is changing the ngram values when indexing the catalog. The ngram tokenizer helps break search terms up into smaller words. We can change the ngram value with another plugin

<type name="Magento\Elasticsearch\Model\Adapter\Index\Builder">
<plugin disabled="false" sortOrder="3" name="gaiterjones_elasticsearch_indexbuilder" type="Gaiterjones\Elasticsearch\Plugin\IndexBuilder"></plugin>
</type>

plugin afterBuild method for IndexBuilder

[code language=”php”] public function afterBuild(\Magento\Elasticsearch\Model\Adapter\Index\Builder $subject, $result)
{
$likeToken = $this->getLikeTokenizer();
$result[‘analysis’][‘tokenizer’] = $likeToken;
$result[‘analysis’][‘filter’][‘trigrams_filter’] = [
‘type’ => ‘ngram’,
‘min_gram’ => 3,
‘max_gram’ => 3
];
$result[‘analysis’][‘analyzer’][‘my_analyzer’] = [
‘type’ => ‘custom’,
‘tokenizer’ => ‘standard’,
‘filter’ => [
‘lowercase’, ‘trigrams_filter’
] ];
return $result;
}

protected function getLikeTokenizer()
{
$tokenizer = [
‘default_tokenizer’ => [
‘type’ => ‘ngram’
],
];
return $tokenizer;
}
[/code]

In this case we are increasing the ngram value to 3. This will take effect at the next catalog search full text reindex.

To test these customisation values for yourself you can download the full module here https://github.com/gaiterjones/Magento2_Gaiterjones_Elasticsearch

The module adds a configuration value to Stores -> Configuration -> Catalog -> Search where you can set the minimum score value for Elastic Search results.

Configure search minimum score value in admin

It’s a real shame that ElasticSearch customisation options such as these are not built into Magento 2 by default to help Merchants improve the search experience. ElasticSearch is new to me, and will be to a lot of merchants and Magento devs upgrading to Magento 2.4. It’s a very complex system to understand and although we can tweak some values as shown to improve results this is not a great solution to the problem.

If like me you are still not happy with the native Magento 2 ElasticSearch catalog search results the absolute best solution I have found is to migrate to Smile ElasticSuite

Simply put, installing the Smile ElasticSuite modules and changing catalog search to ElasticSuite will immediately give you almost perfect search results. ElasticSuite is very simple to install and works out of the box improving search results and search relevance.

Install Smile ElasticSuite

Here are the steps required to install ElasticSuite.

Note that ElasticSuite includes it’s own custom layered navigation, if you are already using third party layered navigation modules you will need to disable these first before installing elasticsuite.

You will need to choose the correct ElasticSuite version for the version of Magento 2 you are using. Here are the options for Magento 2.3.x and 2.4

Magento 2.3.3 -> 2.3.5
- ElasticSuite 2.8.x : composer require smile/elasticsuite ~2.8.0
Magento 2.3.5 +
- ElasticSuite 2.9.x : composer require smile/elasticsuite ~2.9.0
Magento 2.4.0
- ElasticSuite 2.10.x : composer require smile/elasticsuite ~2.10.0

After installing the modules run setup:upgrade and then configure your ElasticSearch server hostname.

bin/magento s:up
bin/magento config:set -l smile_elasticsuite_core_base_settings/es_client/servers elasticsearch_server_hostname:9200
bin/magento s:up
bin/magento index:reindex

To change your catalog search engine to ElasticSuite navigate to Stores -> Configuration -> Catalog -> Search and select ElasticSuite as the new catalog search engine.

Configure ElasticSuite as Catalog Search Engine

Refresh caches and the ElasticSuite catalog search engine should now be setup and working – congratulations – Magento 2 full text catalog search just got a whole lot better!

if you see the following error in the frontend, simply run the indexer again.

Exception #0 (LogicException): catalog_product index does not exist yet. Make sure everything is reindexed.

ElasticSuite has some great features :

A new search input form with automatic and relevant search suggestions in a dropdown list
Layered navigation with price sliders and advanced filters
Automatic redirect to product when search returns a single matching product
Automatic spell checked search terms
Smart virtual product categories
Search analysis in admin

You will notice straight away when searching for SKUs that ElasticSuite returns more relevant results than native search.

Using the SKU search example you can search for all or part of the SKU with or without the hyphen and accurate search results will be returned. Notice below the search for MH03 XL without the hyphen returns the correct results

The redirect to product feature when a single matching product is found in search is really useful taking the customer directly to the relevant product.

The search analysis in admin is a great feature allowing you to see how search is being utilised by your customers and which search terms lead to conversions.

For more information on ElasticSuite features and configuration consult the ElasticSuite Wiki or visit the website.

Many thanks to the Smile team for making this module freely available to the Magento 2 community.

Acknowledgements

Smile ElasticSuite
Moore Digital Magento 2 Elasticsearch Relevance
Sandip – ngram plugin

M	T	W	T	F	S	S
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31