It can be a headache to find solutions to run multiple local environments simultaneously without port conflicts when using tools like Docker. This is where Reward can be a very useful tool.
The need - fixing slow syncing #
In this first section, we will go through the technical difficulties Magento 2 developers meet and why it is hard to resolve the architectural problems of Docker on Windows and macOS, even for big fishes like Microsoft.
The architecture of Docker on Windows and macOS #
Generally speaking, we can say development standardization for most software becomes challenging if we want to make sure developers can use whatever operating systems they prefer. Ever since software development methodologies existed, system engineers have struggled with this problem.
In chronological order, these are the main stages of standardization for local software development:
- Without virtualization – the developer installs the entire toolchain to his/her development machine and copies the production server’s configuration to his local environment. – This is inflexible and hard to automate.
- Virtualizing whole systems (Virtualbox, et al) – more flexible, but wastes computer resources.
- Lightweight virtualization – flexible with low resource requirements.
This is how we get to one of the hottest buzzwords of recent years: “Docker”. It is a great tool providing the last station, i.e., Application Containerization.
But Docker has brought different difficulties to fight against. To put it very simply, Docker works by placing the software and its whole runtime environment (like PHP, Python, etc.) and dependencies like common libraries in an isolated package, called a container. From these containers, all system calls (requesting a service from the OS’s kernel) are secure and separated. Thus, this approach also ensures applications running in the container will behave the same everywhere and that they will run safely, regardless of the environment, and without disturbing the rest of the software on the same machine.
To reach this, Docker uses two important features of the Linux kernel: Namespaces (for isolation) and CGroups (for resource control).
Question: if Docker requires a Linux kernel to run, how can Docker operate on machines with Windows or macOS operating systems?
The answer is simple: with virtualization. Docker Desktop runs a lightweight Linux-based operating system (called Moby) on Windows and macOS.
This is a wonderful solution and now quite mature, but it has a huge flaw in its design. The speed of direct file mounting from the host system (i.e., our Windows or macOS) to a guest system (moby) is terribly slow, especially with many small files. The problem is that files are physically stored on the host operating system’s file system, and the guest operating system sees them as something like a network share. In other words, performing any file operation will go through a virtual “network”. This operation builds up a whole TCP/IP stack request for every single file read/write – a huge overload.
Of course, using Linux operating systems as a host solves this problem as there is no need for an intermediate operating system. Docker attaches (mounts) the shared folder directly to the container without additional overhead.
This is a problem for which there is no ideal solution to date. (WSL2 is Microsoft’s workaround to this problem, for further reading about the architecture, you can find a great article here: Introducing the Docker Desktop WSL 2 Backend – Docker Blog.
However, if the directory and files you want to use are right there in your Docker virtual machine, you can get pretty great performance.
This is where file synchronization tools can come in handy. With them, we have an option to have code on the guest operating system’s file system and keep it in sync with the host’s file system. In this case, we will duplicate files, but the network overhead will be “deferred”. Nowadays, as we have hard drives with hundreds of gigabytes of storage that’s something reasonable to sacrifice for the performance boost, just so long as you don’t have to duplicate hundreds of gigabytes of unnecessary data.
For example, with Magento, you might want to sync only the code and mount the media files directly, because these are larger files that don’t impact the application performance directly.
As you can see, the problem here is already starting to get quite complex, especially if keeping in mind we want to use a tool to make it work similarly on Linux, Windows, and macOS.
Multiple projects - multiple needs #
Working only on a single project, it is relatively easy to maintain the project’s environment. If we find an error or opportunity for improvement in the environment, we update the configuration in this single project and the problem will be eliminated. If a dependent software’s version needs to be upgraded, we upgrade it and are good to go. However, if we start working on multiple projects, it is worth starting to manage these configurations more dynamically.
Take a Magento 2.4 project as an example. It consists of the following components:
- webserver (Nginx 1.18)
- PHP (PHP-fpm 7.4)
- database server (MariaDB 10.4)
- in-memory cache (Redis 5)
- search engine (Elasticsearch 7.9)
(and still, we are not close to the end of this list…)
Suddenly we have to start working on a new project, but the client, unfortunately, has an outdated version of Magento (2.2). The required technology stack for Magento 2.2 looks like this:
- webserver (Nginx 1.16)
- PHP (PHP-fpm 7.1) – this Magento will not work with PHP 7.4
- database server (MariaDB 10.0) – will not work with MariaDB 10.4
- in-memory cache (Redis 3) – maybe it works with Redis 5, maybe not
- search engine (Elasticsearch 6) – likely won’t work with Elasticsearch 7
(and so on)
To overcome the situation, docker-compose can help by creating a file (docker-compose.yml) to describe your environment’s needs. You can define what (docker) services you want to run but this docker-compose.yml file is not dynamic, and it’s pretty hard to standardize it if we want to use it for both Windows and Linux environments at the same time. And what will handle the file synchronization for Windows?
Another topic that comes to mind: what if an environment configuration option works so well that we want to distribute it into all of our projects?
Multiple project types - exponential needs #
If the previous topics weren’t enough, imagine the possibility that, in addition to the previous two Magento projects, we should start working on a WordPress or Shopware project – these would need newer versions of newer services and other settings.
How can we handle so many needs at once? This is where Reward comes in.
What is Reward and how to use it? #
From the readme description: Reward is a Swiss Army knife CLI utility for orchestrating Docker-based development environments. Reward makes it possible to run multiple local environments simultaneously without port conflicts by utilizing common services proxying requests to the correct environment’s containers.
What does that mean? #
Reward will generate docker-compose files dynamically for use with docker-compose vs. the developer on a per-project basis.
How can Reward help me? #
- Reward manages common services to solve port conflicts and provide awesome development tools (Traefik for HTTP proxying, Mailhog for caching outgoing mails, Portainer to view what’s happening in Docker, and many more)
- Reward manages local Root Certificate Authority (to work with valid HTTPS certificates)
- Reward helps developers work around manual editing of /etc/hosts file
- Reward manages dynamic docker-compose files (based on the project or operating systems needs for best performance)
And so on and so on.
How does Reward work? #
Behind the scenes, Reward uses Golang’s powerful templating system, and a .env file in the project directory.
With this file, you can simply configure the parameters each project needs, like the following:
PROJECT_TYPE=magento2,PHP_VERSION=7.2,MARIADB_VERSION=10.4.
Reward will read the contents of this file and generate docker-compose files in real time and, based on its templates, run docker-compose and bring up the development environment.
Is Reward easy to use? #
For developers with Docker or Magento command-line utility experience, Reward will be rather easy to manage.
For example, installation of Reward in macOS is 2 commands – Reward takes care of the rest.
Install Reward #
$ brew install rewardenv/tap/reward
$ reward install
Initialize an empty Wordpress #
# Create an empty directory
$ mkdir wp-project
$ cd wp-project
# Initialize the environment
$ reward env-init wp --environment-type wordpress
$ reward bootstrap
Initializing WordPress from ‘zero to hero’ using Reward is easy, check out the gif:
Initialize and empty Magento 2 #
# Create an empty directory
$ mkdir wp-project
$ cd wp-project
# Initialize the environment
$ reward env-init m2
$ reward bootstrap
# You can also install magento sample data and disable the 2FactorAuth plugin
$ reward bootstrap --disable-tfa --with-sampledata
If you want to run or install a Magento 2 project with Reward that’s pretty similar:
Where can I learn more about Reward? #
Reward has detailed documentation with information, use-cases, examples, FAQs and much more.
Check it out here: Welcome to Reward’s documentation!
This post was originally published on ITG Commerce - Pioneers of Ecommerce Blog.
