To do this you configure an alias account in Gmail settings configured with the credentials of your mail service. This is useful if you want to consolidate email accounts or if you have a private email server and want to use Gmail to send email via this server using your various private email address domains.

At the start of April 2020 Google rolled out a security update that affected mail delivery using third party accounts. All emails sent via the alias account would not deliver, bouncing back with the error TLS Negotiation failed, the certificate doesn’t match the host.

TLS Negotiation failed, the certificate doesn’t match the host.

Google appears to be enforcing a new email encryption policy for secureTLS connections including validating that the host name on the mail server TLS certificate matches the canonical hostname (MX record) of the third party mail account.

If the host name does not match you can no longer use an encrypted TLS connection in Gmail to send email via your (or your ISP’s) mail servers.

For example, if your MX record resolves to mail.domain.com but the TLS certificate presented is for smtp.domain.com then Gmail will not connect to your mail server. For some users the only option to get mail working again is to revert to an unencrypted connection – strange that Google even allow that!

Google have updated the help article for Send emails from a different address or alias to include a section about this problem.

Google also no longer accept self signed certificates for TLS mail connections.

I use an EXIM4 docker container for my private mail relay, and use Gmail as the hub for my email send/receive. To workaround this problem I created a docker Certbot container and issued new LetsEncrypt TLS certificates for all my private mail domains used with Gmail as well as the primary TLS certificate for my Mail server.

I can confirm this resolves the problem and third party provider email sending via Gmail is now working again.

For anyone using Exim4 the way to configure Exim to use multiple TLS certificates is to dynamically match them to your mail domain, I did this using

tls_privatekey=${if exists{/etc/exim4/tls/exim.key.${tls_sni}}{/etc/exim4/tls/exim.key.${tls_sni}}{/etc/exim4/tls/exim.key.mail.defaultdomain.com}}
tls_certificate=${if exists{/etc/exim4/tls/exim.crt.${tls_sni}}{/etc/exim4/tls/exim.crt.${tls_sni}}{/etc/exim4/tls/exim.crt.mail.defaultdomain.com}}

Thousands of people have been affected by this. Considering the amount of people working from home or struggling to work at all during the Corona Virus pandemic its really bad timing by Google to implement new email security policies that are service affecting for a lot of users.

1. git clone https://github.com/gaiterjones/docker-magento2-2
2. cd docker-magento2-2/magento
3. edit .env
4. docker-compose build
5. docker-compose up -d
6. http://magento2.dev.com

Upgrading Magento is always a scary experience – Magento 2 is even more terrifying so I recommend you get used to doing updates with your dev systems. If you are manually upgrading from a previous 2.x version, you can use composer within the docker container:

1. Take a backup
2. Enter maintenance mode
3. edit compose.json
   1. change version
4. flush redis
   1. redis-cli flushall in the redis containers
5. composer update
6. rm -rf var/cache/* var/page_cache/* var/generation/* var/di/*
7. chmod +x bin/magento
8. php bin/magento cache:flush
9. php bin/magento setup:upgrade
10. php bin/magento indexer:reindex

If the upgrade breaks any sample data reinstall with php -f bin/magento sampledata:deploy

I regularly update my Magento 2 docker containers with each version update check them out at https://github.com/gaiterjones/docker-magento2-2

If you are building skills with my PHP code here are a few notes on using the ASK CLI to do simple skill invocation tests using a Docker compose ASK CLI nodejs container.

Assuming you are familiar with using Docker and docker-compose here are the steps to testing your skill from the command line :

1. git clone https://github.com/gaiterjones/docker-amazon-ask-cli
2. cd docker-amazon-ask-cli
3. docker-compose build
4. docker-compose up -d
5. ./init.sh
   1. Initialise ask with your amazon developer credentials, you will be prompted to create a default profile and then to paste an authorisation url into your browser. The url will take you to an Amazon authorisation page where you need to login with your developer login. You will then receive an authorisation code which you can paste into the init window to create your cli-config credentials file.
   2. 
6. Test skill with skill simulation
   1. ./ask.sh amzn1.ask.skill.123 “ask my skill to do something” where amzn1.ask.skill.123 is the id of your skill and “ask my skill to do something” is the invocation term you want to test with your skill – i.e. what you would ask Alexa to do to open and use your skill.
   2. 

Check out ask.sh for the ASK CLI syntax being used. You can use this directly from the container with docker exec -it dockeramazonaskcli_nodejs_1 /bin/bash

The credentials will be saved in cli-config when the container is stopped.

You can do a whole lot more with the ASK CLI, check out the documentation here https://developer.amazon.com/docs/smapi/ask-cli-command-reference.html

What is OpenFaaS?

OpenFaaS lets you build and distribute “serverless” functions in a cloud computing environment. The “function” could be simple application logic that you program – or execution of a binary to perform an action. The term serverless is a slight misnomer as you still require a server, or indeed many servers, but by deploying your function into the cloud you don’t have to worry about scaling or managing physical server resources yourself – which is pretty cool.

OpenFaaS works by packaging your function up into a docker container customised to work with OpenFaaS. The OpenFaaS framework stack handles the creation and deployment of the container allowing you to execute your function with a simple command line or http request.

OpenFaaS doesn’t really care how you create your function as long as the container behaves in the way the framework and the function watchdog expects. This means whilst OpenFaaS ships with support for languages like Python and NODE.js it should work with just about anything you throw at it which means you can package up just about anything as a serverless function.

I’m developing with good old PHP so the first thing I did when I decided to take a closer look at OpenFaaS was to google “open faas php” to see how I could create serverless functions written in PHP. My search led me to a php template for OpenFaaS developed by Minh-Quan Tran. The template tells OpenFaaS how to build PHP5 or PHP7 containers to run your PHP code as OpenFaaS functions. After trying out the template there were a couple of issues with it I wanted to address, so I forked it and made a few changes.

If you want to deploy OpenFaaS PHP functions here’s a step by step guide that will get you from Zero to Hero in just a few minutes.

Install the OpenFaaS Stack and CLI

Assuming you already have Docker installed, the first thing you need to do is initialise a Docker swarm and deploy the OpenFaaS stack.

docker swarm init --advertise-addr $(hostname -i)

Change into a suitable working folder and install the OpenFaaS stack by cloning it from github and executing the deploy script.

git clone https://github.com/openfaas/faas && \
  cd faas && \
  ./deploy_stack.sh

Test that the stack is working by invoking the echoit example function that echoes back any input data you send it.

curl 127.0.0.1:8080/function/func_echoit -d 'hello world'

Now that OpenFaaS is working you need to install the OpenFaaS cli framework (faas-cli) that allows you to build, deploy and invoke your own functions easily from the command line.

curl -sL https://cli.openfaas.com | sudo sh

Next install the OpenFaaS PHP template

faas-cli template pull https://github.com/gaiterjones/openfaas-template-php

2017/11/19 09:52:30 Attempting to expand templates from master.zip
2017/11/19 09:52:30 Fetched 2 template(s) :  from https://github.com/gaiterjones/openfaas-template-php
2017/11/19 09:52:30 Cleaning up zip file...

Build and Deploy a PHP OpenFaaS function

Once the template is installed we can test it by creating our first PHP function. The OpenFaaS command line syntax for creating a new function is faas-cli new functionname –lang languagename. The PHP template uses the language name php for PHP7 functions and php5 for PHP5 functions. We will create a PHP7 function called func_php1 using the code in the template.

faas-cli new func_php1 --lang php

Folder: func_php1 created.
  ___                   _____           ____
 / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
 \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
      |_|


Function created in folder: func_php1
Stack file written: func_php1.yml

You will notice a new stack file func_php1.yml and function folder is created. For my tests using an Ubuntu virtual machine I had to edit func_php1.yml and change the hostname localhost to 127.0.0.1 as shown below.

provider:
  name: faas
  gateway: http://127.0.0.1:8080

functions:
  func_php1:
    lang: php
    handler: ./func_php1
    image: func_php1

In the func_php1 folder you will see three files

composer.json
Handler.php
php-extension.sh

Handler.php contains the entrypoint for our OpenFaaS PHP function and looks like this

<?php

class Handler
{
    public function handle(string $_data): void {
        $_version = phpversion();
        echo 'PHP Version : '. $_version ."\n". 'Data : '. $_data. "\n\n";
    }
}

You can see the PHP method Handler::handle() is going to echo the PHP version, and whatever data is supplied to it from STDIN.

Build the service container for the function with

faas-cli build -f func_php1.yml

Once the container has been built deploy it with

faas-cli deploy -f func_php1.yml

Deploying: func_php1.
No existing function to remove
Deployed.
URL: http://127.0.0.1:8080/function/func_php1

200 OK

And that’s it, our PHP OpenFaaS function is ready to test we can invoke it from faas-cli

faas-cli invoke -f func_php1.yml func_php1

Or use curl

curl 127.0.0.1:8080/function/func_php1 -d 'hello world'

PHP Version : 7.1.11
Data : hello world

You can also access all functions via the nifty OpenFaaS UI at http://yourhost/:8080/ui/

Build and Deploy a GEO IP OpenFaaS PHP function

Now let’s deploy a function with a little more functionality…

One API I use a lot is GEOIP2 by MaxMind it’s very useful for getting geographic data from an IP address. So let’s build an OpenFaaS PHP function to do just that.

Create a new PHP7 function called func_geoip

faas-cli new func_geoip --lang php

Before building the function we need to add our PHP code and dependencies. First let’s add the PHP dependencies to the functions composer file by editing func_geoip/composer.json

{
    "name": "function-php7",
    "description": "OpenFaaS GEOIP PHP function",
    "type": "project",
    "license": "MIT",
    "require": {
        "php": "^7.0",
        "maxmind-db/reader": "~1.0",
        "geoip2/geoip2": "~2.0"
    }
}

The geo ip data is read from a free GEOLITE2 database so we need to make that database available to our function. We can add data to our container using the func_geoip/php-extension.sh shell file which is executed when we build the template container.

echo "Installing function dependencies"
cd /usr/local/faas/function
curl -o GeoLite2-City.tar.gz http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
tar -xvzf GeoLite2-City.tar.gz --strip 1

Now add the geo ip lookup code to our PHP Handler class by editing func_geoip/Handler.php

Build and deploy the function

faas-cli build -f func_geoip.yml
faas-cli deploy -f func_geoip.yml

Now we can test it with

curl 127.0.0.1:8080/function/func_geoip -d '128.101.101.101'

The function returns the JSON formatted array of Geo data for the requested IP address

{"ip":"128.101.101.101","country":"United States","isocode":"US","region":"Minnesota","city":"Minneapolis","postcode":"55414","latitude":44.9759,"longitude":-93.2166,"googlemap":"https:\/\/maps.google.com\/?q=44.9759,-93.2166"}

FaaS in Production

We can easily access function data from other PHP applications using php curl to post the input data and return the output array. In production you need to consider how you will secure access to the function using some form of frontend proxy authentication.

Whether you will deploy simple php code as serverless functions in this way depends very much on the type of applications you are developing and the infrastructure available to you. I can
certainly think of a lot of my commonly used PHP library functions and classes that would become very useful deployed as OpenFaaS serverless functions.

References

• OpenFaaS
• OpenFaaS Community
• OpenFaaS CLI
• PHP Template
• Docker Swarm Tutorial

Tested on

• Ubuntu Server 17.10
• Docker 17.09.0-ce

The Less to Sass Community Project is officially supported by Magento and Theme Blank SASS is the result of this project – a (production ready) Magento 2 blank theme converted to SASS.

Together with the accompanying Fronttools pack you get the development tools and extendable SASS based blank theme you need to get you started building your own custom theme.

Why use theme blank SASS ?

• If you are coming from Magento 1.9 then you already are familiar with SASS
• Magento 2.x may possibly move to SASS in the future
• LUMA is not designed to be extended as a blank theme
• Essential development tools such as browsersync are included

How to install theme blank SASS parent theme and Fronttools

You can be up and running with theme blank SASS and fronttools in a few minutes. Here is a step by step theme blank sass and frontools installation guide tested on my Magento 2.x docker containers.

I am installing the parent theme and fronttools directly into my docker Magento 2 php-apache container with a bash script :

curl -o- http://gaiterjones.com/dropbox/magento2/theme-blank-sass/install-theme-blank-sass.sh | bash
 
script to install theme-blank-sass and fronttools

Here are the script commands broken down into some more detail :

Set the install environment variables

cd ${MAGENTO_DIR}
MAGENTO_DIR='/var/www/dev/magento2'
NVM_HOME='/var/www'

MAGENTO_DIR is the path to the magento root and NVM_HOME the path to the NVM installation home folder – this is the home folder for the user that installs NVM

Install the theme and tools with composer, upgrade Magento and refresh caches.

composer require snowdog/theme-blank-sassinstall theme blank sass
composer require snowdog/frontoolsinstall fronttools

${MAGENTO_DIR}/bin/magento setup:upgradeupdate Magento
${MAGENTO_DIR}/bin/magento cache:cleanrefresh caches

Install NVM and gulp-cli

cd /tmp
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.1/install.sh | bash
export NVM_DIR=${NVM_HOME}/.nvm
 [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
nvm install --lts
nvm use --lts
npm install -g gulp-cli

Install frontools.

cd ${MAGENTO_DIR}/vendor/snowdog/frontools
npm install

You run the fronttools setup with gulp setup, this will copy the project sample configuration files to ${MAGENTO_DIR}/dev/tools/frontools/config/.  You must edit themes.json, for the parent theme.

{
  "blank": {
    "src": "vendor/snowdog/theme-blank-sass",
    "dest": "pub/static/frontend/Snowdog/blank",
    "locale": ["en_GB"],
    "postcss": ["plugins.autoprefixer()"],
    "ignore": [".test"]
  }
}

The script is copying my themes and browser-sync json configuration files via curl.

gulp setup
curl -o "${MAGENTO_DIR}/dev/tools/frontools/config/themes.json" http://gaiterjones.com/dropbox/magento2/theme-blank-sass/browser-sync.json
curl -o "${MAGENTO_DIR}/dev/tools/frontools/config/themes.json" http://gaiterjones.com/dropbox/magento2/theme-blank-sass/themes-blank-sass-parent.json
gulp styles

Next I compile the SASS to CSS and install it using

gulp styles

Finally I am using n98-magerun2 to enable the new theme, and refresh the Magento caches.

THEME_ID="$(n98-magerun2.phar dev:theme:list --format=csv \
  | grep 'Snowdog/blank' | cut -d, -f1)" \
  ; test -n "${THEME_ID}" \
  && n98-magerun2.phar config:set design/theme/theme_id "${THEME_ID}"
${MAGENTO_DIR}/bin/magento cache:clean

Theme blank SASS parent theme is now installed. You can watch the install process below.

We used the gulp setup, and gulp styles tasks above – other useful tasks include

• gulp clean
  • Removes /pub/static directory content.
• gulp dev
  • Runs browserSync and inheritance, babel, styles, watch tasks.

See the fronttools project page for a full list of gulp tasks or use the gulp default task.

How to install a theme blank SASS child theme

Usual Magento best practice dictates that instead of editing the parent them we extend it with a child theme. You extend Snowdog/blank the same way as any other theme.

<theme xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Config/etc/theme.xsd">
    <title>Gaiterjones/blank</title>
    <parent>Snowdog/blank</parent>
    <media>
        <preview_image>media/preview.jpg</preview_image>
    </media>
</theme>

I tested this with my Docker environment and used the following script to extend theme blank SASS with my Gaiterjones/blank child theme.

curl -o- http://gaiterjones.com/dropbox/magento2/theme-blank-sass/child/install-child-theme-blank-sass.sh | bash

The script extracts the Gaiterjones/blank child theme source files to app/design/frontend/Gaiterjones/blank and updates Magento. We need to also update the fronttools themes.json config file to let fronttools know where our child theme lives

{
  "blank": {
    "src": "vendor/snowdog/theme-blank-sass",
    "dest": "pub/static/frontend/Snowdog/blank",
    "locale": ["en_GB"],
    "postcss": ["plugins.autoprefixer()"],
    "ignore": [".test"]
  },
  "gaiterjones-blank": {
    "src": "app/design/frontend/Gaiterjones/blank",
    "dest": "pub/static/frontend/Gaiterjones/blank",
    "locale": ["en_GB"],
    "localeOverwrites": true,
    "parent": "blank",
    "postcss": ["plugins.autoprefixer()"]
  }  
}

Next I am activating the theme with n98-magerun2 and then I simply need to refresh the caches and run gulp styles again to create the CSS for our child theme.

gulp styles
${MAGENTO_DIR}/bin/magento cache:clean

Fronttools compiles and installs the CSS for our child theme which is now installed and ready to view.

You can see theme blank SASS in action on my development site.

Features

• PHP 7.4 and Apache 2.4 built on the latest PHUSION docker images
• 2 Redis servers for Session and System (including FPC)
• MYSQL 8
• PhpMyAdmin
• Varnish 6 for FPC
• RabbitMQ
• Management container and Scale Manager for scaled service dynamic load balancing
• composer, n98-magerun2, PHP Redis Admin, XDEBUG
• Magento 2.4.2 (Feb 2021)

Requirements

• Host server with min 2GB RAM
• Docker Engine
• Docker Compose 3.1

Base Images / Source for 2.4 deployment

• gaiterjones/magento2 for 2.4
  • https://github.com/gaiterjones/docker-magento2/tree/master/magento2/ubuntu/magento240-build
  • PHUSION
    • https://github.com/gaiterjones/docker-magento2/tree/master/magento2/ubuntu/phusion1001-apache2-php7-4-build

Deployment

Here are the steps to deploy this Magento 2.4 development environment:

• Clone the source file repository from, GitHub
  • 2.4
    • git clone https://github.com/gaiterjones/docker-magento2
• Change into the Magento 2 directory
  • cd magento2
• Edit the .env file and set the working environment for your deployment :
  • APPDOMAIN=dev.com
    • the domain name for the magento installation
  • SMTP=XXX
    • the smtp gateway for your deployment, required for Magento email.
  • MAGENTO_URL=http://magento2.dev.com
    • the Magento URL for your deployment
  • MAGENTO_REPO_USERNAME/PASSWORD
    • your Magento repo authentication details required for Magento installation
  • CONTAINERDATA=/home/docker/data/
    • Used for persistent container data
• Edit docker-compose.yml
  • You can remove the Manager and Memcache service here if you do not want to deploy the scale manager.
  • for mysql persistent data uncomment the following from the mysql image
    • #– “${CONTAINERDATA}/${PROJECT_NAME}/mysql/dev1:/var/lib/mysql”
  • If you are using Nginx as a container reverse proxy change the ports and networks for the varnish service accordingly.
• Generate a Varnish Secret
  • You can create a new varnish secret by running newsecret.sh in the varnish folder.
• BUILD
  • docker-compose build
• START
  • docker-compose up -d
    • Allow 60 seconds after starting for composer to initialise
• Install Magento Sample Data (optional)
  • docker exec -it magento2_php-apache_1 install-sampledata
    • ignore error Can’t run this operation: deployment configuration is absent.
• Install Magento
  • docker exec -it magento2_php-apache_1 install-magento

Script install

For a quick test, spin up a Type 2 packet.net server running Ubuntu LTS 20.04, install Docker CE and docker compose and then run

docker exec -it --user magento magento2_php-apache_1 su root -c "curl -o- https://gaiterjones.com/dropbox/magento2/docker/install.sh | bash"

Testing

• frontend
  • • http://magento2.dev.com
  • 
• backend
  • • http://magento2.dev.com/admin/
      • login using the credentials you set in .env
    • Configure Varnish FPC
      • Stores-> Configuration -> Advanced -> Full Page Cache
  • 
  • Test from containers with curl (optional)
    • curl -I -H “host: magento2.dev.com” magento2_php-apache_1:80
    • curl -H “host: magento2.dev.com” magento2_php-apache_1:80/healthcheck.php
  • n98-magerun2
    • docker exec -it –user magento magento2_php-apache_1 /usr/local/bin/n98-magerun2.phar

Persistent volumes

Your development deployment data will be lost when you restart the php-apache or sql containers. If you want data to persist after installation perform the following steps:

• Tar the Magento source files in the php-apache container
  • docker exec -it –user magento magento2_php-apache_1 tar -cvf /home/data/magento2.tar /var/www/dev
  • The tar file will be saved to the docker container data folder specified in .env
• Stop the containers
  • docker-compose down -v
• Untar to host
  • tar -xvf magento2.tar
• Move the /var/www/dev folder to your persistent data folder i.e. /home/docker/data/magento2/dev1
• Uncomment the data volume in docker-compose.yml
  • #- “${CONTAINERDATA}/${PROJECT_NAME}/dev1:/var/www/dev”
• Restart containers, Magento data is now persistent.

Scaling php-apache

You can scale the php-apache Magento service with

docker-compose scale php-apache=X

See this post for more information.

manager container

In production I prefer to use a management container for all Magento admin and cli commands. The manager container can be used as the main administration container for your production environment. You should move all cron jobs to the manager and you can also move web administration to the manager container. This will free up the php-apache container just to be used to serve frontend web requests and allows you to use the scale function. You will also notice that separating cron tasks into a separate container reduces the memory requirement of the php-apache container significantly.

To start an admin bash session in the management container use

References

• Fabrizio Balliano
• Alex Cheng

My Magento 2 service container images include REDIS services for the Magento session and system cache and a Varnish service for the full page cache. Varnish is an HTTP accelerator designed for content-heavy dynamic web sites and is the recommended full page cache system for Magento 2.

Varnish is configured via the Varnish Configuration Language (VCL) config file. By configuring multiple backend servers in VCL and grouping them into a director we can create a Varnish load balancer.

[text]
backend magento2_php_apache_1 {
.host = "x.x.x.2";
.port = "80";
.probe = {.request =
"GET /healthcheck.php HTTP/1.1" "Host: magento2.gaiterjones.com" "Connection: close" "Accept: text/html";.timeout = 1s;.interval = 30s;.window = 10;.threshold = 8;}
}
backend magento2_php_apache_2 {
.host = "x.x.x.8";
.port = "80";
.probe = {.request =
"GET /healthcheck.php HTTP/1.1" "Host: magento2.gaiterjones.com" "Connection: close" "Accept: text/html";.timeout = 1s;.interval = 30s;.window = 10;.threshold = 8;}
}
backend magento2_php_apache_3 {
.host = "x.x.x.10";
.port = "80";
.probe = {.request =
"GET /healthcheck.php HTTP/1.1" "Host: magento2.gaiterjones.com" "Connection: close" "Accept: text/html";.timeout = 1s;.interval = 30s;.window = 10;.threshold = 8;}
}
backend magento2_php_apache_4 {
.host = "x.x.x.11";
.port = "80";
.probe = {.request =
"GET /healthcheck.php HTTP/1.1" "Host: magento2.gaiterjones.com" "Connection: close" "Accept: text/html";.timeout = 1s;.interval = 30s;.window = 10;.threshold = 8;}
}
backend magento2_php_apache_5 {
.host = "x.x.x.9";
.port = "80";
.probe = {.request =
"GET /healthcheck.php HTTP/1.1" "Host: magento2.gaiterjones.com" "Connection: close" "Accept: text/html";.timeout = 1s;.interval = 30s;.window = 10;.threshold = 8;}
}
sub vcl_init {
new cluster1 = directors.round_robin();
cluster1.add_backend(magento2_php_apache_1);
cluster1.add_backend(magento2_php_apache_2);
cluster1.add_backend(magento2_php_apache_3);
cluster1.add_backend(magento2_php_apache_4);
cluster1.add_backend(magento2_php_apache_5);
}
[/text]

Scale Manager

I use a service container called php-apache to run the Magento core code. Docker compose allows you to scale containers on a single host, so for example with the command

docker-compose scale php-apache=5

I can quickly scale the Magento2 php-apache service up to 5 containers, that is 5 instances of the php-apache service running the Magento2 core php code and apache web server.

The scale manager service monitors docker processes and detects when the php-apache service is scaled up or down. When a change is detected it dynamically creates the VCL configuration required to tell Varnish about the changes and applies these to the Varnish server.

Magento 2 Scale Demo

In the video below I start with one running instance of the Magento 2 service, I then scale the service up to 5 and using the scale demo page (which is not cached by Varnish) on my Magento 2 dev site to analyse the scale manager output data and display the running Magento 2 service containers.

As I refresh the page the Magento 2 container rendering the page is highlighted in yellow. you can see how Varnish round robins between the 5 available containers. Finally I scale the service back down and manually update the Varnish server config via the command line php script on the manager.

Another way to see the round robin connection distribution from varnish is to load up a phpinfo() page, you will see the SERVER_ADDR variable changing with each container that processes the request.

Would you actually run 5 Magento 2 php-apache services? Good question! For true resilience and load balancing you would need multiple hosts and a Docker swarm. Multiple service containers on a single host do at least provide a simple level of redundancy and host server load balancing, I would definitely consider using it in production especially with a multi core server.

Many thanks to Fabrizio Balliano for the inspiration and assistance.

See this post for instructions on how to deploy this Magento 2 development environment.

For example if I have a web application service called php-apache

docker-compose scale php-apache=10

Will start another 9 containers running the php-apache service giving me 10 containers running the same php-apache web application service.

Of course if you want to have real resilience and load sharing Docker swarm is the solution so the question is, are there any benefits from scaling a service in mulitple containers on a single host?

Well, yes, because you can still load balance the traffic to your application and achieve a basic level of container resilience.

Scaled services will only work for applications that use private ports as multiple Docker containers cannot share the same public port. I use an NGINX service container as a reverse proxy to my web application services with NGINX listening on the host public http port, proxying traffic to the service containers on their private internal container http ports.

This scenario can scale easily and there are some documented examples that describe how to load balance Docker scaled web application services, however all the examples I found require manual configuration and a restart of the load balancer to update it when service containers are scaled up or down.

I wanted to create a dynamic scaled service where the load balancer or proxy is dynamically updated when services are scaled up or down.

NGINX as a reverse proxy will also load balance when multiple upstream hosts are configured for a website but the default configuration file must be manually edited, saved and NGINX restarted before the changes take affect.

A bit of googling around uncovered a module that allows you to dynamically update the NGINX upstream host configuration. I compiled the module into my existing NGINX service container and tested it and was able to dynamically add and remove upstream hosts, perfect for dynamically scaling Docker services!

Scaledemo Application

To demonstrate how Docker Mono Host Service Scaling and Dynamic Load Balancing with NGINX works I put together a group of containers to simulate a scaled web application :

• manager
  • A php service container that acts as the manager for the scaled services project.
• nginx
  • An NGINX container compiled with the ngx_http_dyups dynamic upstream module.
• php-apache
  • A php7, Apache2 service container that will run my demo ‘Bean Counter’ web application.
• memcached
  • Cache server used by the manager to store Docker data, and by the web application to store the ‘Bean Counter’ data.

Manager

Dynamic updates to NGINX are carried out by the manager service container. It pulls Docker system information from the Docker host via a cron job every 60 seconds, parses the data for running service containers that match the configured project web application service container (php-apache) and generates an NGINX upstream host configuration.

The manager compares the upstream host configuration running on NGINX with the generated configuration from Docker. If they differ it updates NGINX via php-curl with the new configuration. The parsed manager Docker data is stored on the memcache server so that it is available to other services.

When I scale the php-apache service up or down NGINX is dynamically configured with the changes at the next manager update, or heartbeat. I can also manually force the update via a cli command on the manager.

Demo

Watch the video below to see the scale demo in action.

In the video I start the containers with docker compose, scale php-apache up to 10 instances and check the dynamic update and round robin load balancing via the demo web application that parses the manager data from memcache and also increments a counter.

I then scale it up again to 20 instances and manually update the nginx configuration via the manager. You can see NGINX load balancing the web application with round robin requests to each of the 20 service containers as I click the refresh button to increment the bean counter and reload the page. The container that served the page is highlighted in yellow.

Finally I scale the service back down to 2 instances and stop the containers.

You can build the Docker containers and try out the demo yourself using the docker-compose files from GitHub.

• Clone the project files
  • git clone https://github.com/gaiterjones/docker-scaledemo
• Build the containers
  • cd scaledemo
  • docker-compose build
• Make sure there are no other service using port 80 on your host and start the containers
  • docker-compose up -d
• Point your browser at http://scaledemo.dev.com (which should resolve to your Docker host) to see the demo page
• Scale the php-apache container
  • docker-compose scale php-apache=5

To manually update NGINX from the scale manager

• Connect to the manager container
  •  docker exec -t -i scaledemo_manager_1 /bin/bash
    • or docker exec magento2_manager_1 php cron.php scalemanager
• run the scalemanager php script
  • php cron.php update scalemanager

Install Docker

Installing Docker is a no brainer

curl -sSL get.docker.com | sh
sudo usermod your-username -aG docker
reboot

Install docker-compose

Docker-compose makes building, starting and stopping Docker containers really simple.

apt-get -y install python-pip
pip install docker-compose

Build the container

Pull the container build files from github and run docker-compose build to build the container image.

git clone https://github.com/gaiterjones/docker-rpi-fhem
cd docker-rpi-fhem
docker-compose build

Start the container

docker-compose up -d

Connect to FHEM

Enter the following url into your browser to connect to FHEM.

http://my.pi:8803/fhem

Update FHEM by entering with the update command.
Restart FHEM by entering the shutdown restart command.

FHEM is now running and up to date.

You can start configuring it or copy your existing FHEM config or editing the fhem.cfg file in ./fhem. This is a Docker volume on the host mapped to /opt/fhem in the container. Docker host volumes will persist when you bring down the container (docker-compose down) or clear the volume caches (docker-compose down -v) so you will not lose your configuration or logs when the container restarts or is rebuilt.

Customising

Take a look at docker-compose.yml

# RPI
# FHEM CONTAINER
#
version: "3"

services:
    server:
        build: ./build/
        hostname: fhem
        domainname: home.com
        privileged: true
        ports:
          - "8083:8083"
          - "7072:7072"
        expose:
            - 8083
            - 7072
        volumes:
            - ./fhem:/opt/fhem
            #- /dev/ttyUSB0:/dev/ttyUSB0
        networks:
            - server
        restart: always
        healthcheck:
           test: ["CMD-SHELL", "echo 'QUIT' | nc -w 5 localhost 7072 > /dev/null 2>&1 && echo 'FHEM OK'"]
           interval: 15m
           timeout: 10s
           retries: 3
networks:
    server:

You can see we are configuring global ports 8083 and 7072 on the host as well as exposing them on the container. I have an SSL certificate installed on my Pi and use Nginx as a frontend proxy to my Docker containers, this lets me connect to FHEM using SSL

https://my.pi/fhem

To remove the global host ports delete or comment out the ports config leaving just the expose config. Nginx can be configured to proxy to fhem using the following configuration:

# FHEM
location /fhem {
  proxy_pass http://fhem_container_1:8083/fhem;
}

The container runs in privileged mode which gives it access to the hosts hardware and devices. If you are using external USB devices connected to the Pi you must add them as a volume in the container i.e.

– /dev/ttyUSB0:/dev/ttyUSB0

Make sure the fhem user running in the container has the same user id as your host user, in this case uid 1001.

	 	 
user@raspberrypi:/dev# id your-user-name	 	 
uid=1001(your-user-name) gid=1001(your-user-name) groups=1001(your-user-name),27(sudo),996(docker)	 	 

This will ensure there are no permission problems when the container tries to access devices on the host.

Maintaining servers whether they are physical or virtual is a real pain in the neck, as time goes by you develop apps, you install more and more software and your simple php development platform starts to become quite complex. When it comes to a major OS update e.g. Ubuntu 14LTS to 16LTS where mysql and php versions are changing then keeping all your (now live) websites, blogs and apps running can become a real challenge.

In an ideal world you would have multiple servers designed specifically for the apps they are running, in the real world when we are talking about personal websites, blogs and forums everything ends up running on the same server. The longer you postpone an operating system upgrade the harder it gets to upgrade, and whilst some people live by the ‘if it’s running don’t break it’ theory when developing apps you want to make sure you are staying up to date with best practice and are aware what is being deprecated or no longer supported. Again the longer you ignore updates to PHP the harder it will be to update your PHP apps!

It’s a bit of a catch 22 situation, you need new server hardware, or an operating system but you can’t just upgrade in one go because too many apps will be affected resulting in a lot of downtime whilst you fix all the broken code and you can’t migrate to a new server because you know some of your apps won’t run.

As soon as you build a new server the process starts again and the next upgrade becomes just as challenging.

What is the solution? – containerisation.

containerisation with docker

Docker containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in.

By moving each of my apps and server services into a container customised for the requirements of that app, be it a specific PHP or MySQL version or operating system I containerised (virtualised) all my server applications: php, mail, cloud storage, vpn, webapps, blogs, forums, Magento etc. into new zero administration servers which I wil never have to worry about upgrading!

All the old apps you know are not compatibile with the latest version of PHP (i.e. Magento 1.x) can stay running in a container customised for their requirements. The docker host server can now run a very minimal operating system, ideally with just Docker installed making future upgrades pretty uncomplicated.

Because all your apps are in containers moving to a new server is as simple as moving the container configuration and data files to the new server and starting the containers, you no longer have to worry about upgrade compatibility issues as your application is running in an container image that in theory will never change!

The golden rule to follow is to ensure that you always treat your containers as volatile storage and never store live data in them, or commit configurations that are not included in your configuration files.

By separating your application data into persistent docker volumes located in logically named folders basic housekeeping tasks like log management and backups are now also simplified. With all app data located in data folders, your container (application) data can be easily backed up. Log files are also easy to manage because they are consolidated into Docker’s central logging engine, all your log data can easily be centrally stored and analysed.

In practice Docker just works, the containers start and stop in seconds, and in some cases your apps will run much faster. There is a lot of work in configuring the container to replicate the environment you want, but once your Docker file is written it becomes a good template for future apps.

It’s easy to get carried away containerising everything so you have to monitor the host server resources you are using, I ended up with over 40 running containers which maxed out the RAM in a 4GB Linode resulting in containers crashing because they suddenly ran out of memory. I upgraded to an 8GB Linode where the ram usage now averages at just over 4GB for 40 running containers leaving me room for expansion.

I recommend building your containers using docker compose. Docker compose lets you keep your container configuration in one folder and provides an easy command line interface to start, stop and build containers. Here for example is the docker composer file for my Exim4 mail server container.

Here you see the docker compose file creates two services, Exim4 and Spam Assassin. The file also configures health checks and networking. More about building containers such as this Exim4 mail server with spam assassin will follow in another post.