Concern Worldwide - site rebuild
Case descriptionConcern.net was built using decoupled Drupal 8 and React.js. From the end users’ perspective (aka “donors”), it focuses on a flawless user experience navigating the website as well as a smooth donation journey. For stakeholders it is a tool to build brand awareness and trust in addition to generating, analysing and reporting on donation campaigns. Stakeholders benefit from a robust and comprehensive content creation system where full control of the system is in the hands of editors.
In order to achieve the project requirements and goals, we first built Falcon (https://github.com/systemseed/falcon), an open source distribution for charities. Falcon is a decoupled distribution: it is powered by Drupal 8 on the backend and React.js on the frontend. We took the best of two tech worlds to build a powerful and customisable CMS for charities. Each Concern Worldwide region (currently Ireland and the UK) run their own Falcon installation.
On the backend, Drupal 8 is used as a content hub and API gateway. Out-of-the box Drupal capabilities provide secure and powerful content management tools for relatively low effort. The previous version of the website was running Drupal 7 and was migrated using the Migrate module. Core REST API features were extended by REST Entity Recursive, a module that we built, to expose all necessary content, settings, and metadata to the frontend. To integrate Drupal Commerce with decoupled donation forms, we also developed Commerce Decoupled Checkout and included it in the Falcon distribution.
Case goals and resultsThe client had ambitious goals for the project:
- Redesign and rebuild the old website that was powered by Drupal 7
- Build a smooth and simplified donation workflow
- Increase the average donation amount as well as the number of donors
- Increase brand awareness and trust
- Migrate all content from the old system to the new one
- Allow content editors to be in charge of content, page templates, URLs and SEO
- Integrate with analytics systems for thorough analysis of campaign results
- Integrate with A/B testing tools for optimisation of conversion rates
- Allow both Concern regions to benefit from new and existing features simultaneously but also to create separate content independently
- Connect the appeals and donations part of the system to their internal CRM Microsoft Dynamics
Three months after launch, the Data Analyst at Concern Worldwide has reported the following numbers:
- Traffic increase by 70%
- Total donations increase by 16%
- Checkout conversion rate has increased by 70%
The project prioritized the complete re-thinking of user journeys and an entire re-design of the frontend for increased ease of navigation and a more appealing look & feel of the website. It was not just another “Drupal 8 upgrade”. Technically it was a completely new project powered by a new stack of technologies. The only common element between the old and new versions was using Drupal as the system backend. The content creation experience centred around the Paragraphs module plus some cool tweaks including an image preview of each paragraph type which gave content editors the desired control over the creation of flexible page layouts.
During the content migration from the old to the new system, the content was mapped in a way that avoided the “long text with html” type of content in favor of different types of paragraphs, matching the structure & design of the new website.
As mentioned previously Drupal runs the system backend and is used for content curation and handling the commerce side of donations. For seamless content fetching via API we developed a module called REST Entity Recursive. This lets us use comprehensive Drupal content / URLs management systems - the automatic generation of URLs from template, redirects, etc. The great editorial experience for metatags management comes from a popular community module called Metatag, which again we patched to allow for a better decoupled experience.
In our focus on analytics and A/B testing tools, stakeholders benefited from a detailed overview of donation funnels in Google Analytics with the ability to filter by campaign, date, donation type and many other criteria. It is a fully-fledged E-Commerce integration. Having plugged in the A/B testing tool called VWO, the client is capable of creating multiple versions of landing pages for their campaigns, observing which of them has the better conversion rate and eventually switching to the most efficient one.
The project has multiple donation journeys such as one-off donation, recurring donation, upgrading the existing recurring donation amount and several others. Depending on the donation type, the donation form changes, showing users only the fields necessary to complete the donation of the given type. This helps to increase the conversion rate of donors by asking them to complete donations in as few steps as possible.
Towards the end of the project we spun up a second copy for an additional region (https://concern.org.uk). In doing this, the client received a clone of the system with the ability to independently deploy region-specific features on top of it along with their own database for any region-specific content. Even though different regions may have different campaigns and content, some of the content is actually shared between regions, so that when one region changes the content for certain pages, it updates the other region as well. We call this feature “content sharing”.
Yet another important achievement highlighted by the client is what they call “instant feature deploys”. In the past it could take several hours or even days to deploy new features for stability reasons. Now we have a mature Continuous Deployment system where any feature can go live any time in a matter of minutes. First it goes through a pipeline of automated testing and quality checks and then if all checks have passed, it gets automatically deployed to the production environment.
ChallengesThe project was full of exciting challenges! Let us highlight the most interesting ones:
Even though there are several PAAS/IAAS players on the market offering hosting for decoupled systems matching the requirements, we had experienced poor quality of service and uptime with many of them. This resulted in a monumental decision to move the project to our own Kubernetes cluster on AWS.
# Uptime guarantee
Our aim was to give the client a 99.99% uptime guarantee as well as extensive hosting customisation options should they need it. We used Terraform and Kubernetes on AWS for running the project workload across multiple servers in different regions. This helps reduce the workload for each server, because Kubernetes is quite smart as to where to send a request depending on the current load on each of the servers. In case of a huge traffic spike, adding several more servers to the cluster is a matter of few minutes and fully automated. Additionally, having each server in a different region makes the site resilient to outages, because Kubernetes routes traffic to servers in other regions dynamically.
# Full editorial control over page templates / content / URLs / SEO in a decoupled Drupal world
Concern Worldwide wanted to be able to create complex blocks (i.e. DOM elements) on each page regardless of the page type and be in charge of page templates, URL structure, redirects and so on. To solve this challenge we based the content building experience on Paragraphs and built a module called REST Entity Recursive to glue the streamlined Drupal editorial experience to the frontend application. For better visual reference between Paragraphs and the resulting components we provided preview images for each of the paragraph types.
# Decoupled Drupal Commerce
By the time we started, there were no solutions for decoupled commerce, neither were there any payment gateways ready for working with decoupled solutions. Therefore, we developed a new module called Commerce Decoupled Checkout to provide integration with Commerce module via the API. Moreover, we built three modules for payment gateways (for Paypal, Global Payments and Direct Debits) to accomplish the desired user experience around payments on the donation forms.
# Progressively responsive design
Some paragraphs (blocks) look completely different depending on the device type of the end users. We used the React.js framework to develop those different blocks and toggle their display based on viewport width and device type.
# Content migration from the old to the new site
The content structure changed dramatically between the old and new sites. The old content was long text with html, while the new system had all content divided into reusable paragraphs of different types. To resolve this challenge, we performed a very thorough mapping and data conversion exercise together with the client. Thanks to “Migrate” and “Migrate HTML to Paragraphs” modules we were able to provide a really robust migration experience and successfully moved thousands of pages in a largely automated fashion. During this exercise we submitted multiple patches to the existing open source solutions as none of them completely fit our use case.
# Content sharing between regions
Technically each region has their own copy of the project, which means they have their own database and their own version of the code. However, some of the content needs to be shared between regions, while other content needs to be entirely independent. We came up with a solution which allows sending selected pages (nodes) between different websites and resolve the potential URL / reference conflicts automatically.
# Reusable components
It often happens on long-term projects that designers come and go throughout, which can affect the look & feel of the site resulting in discrepancies with spacings, colours, behaviours and so on. To fix this problem of too many proverbial cooks in the design kitchen, our team developed a set of all React.js elements to build pages in a style guide hosted on a separate domain. It includes all buttons with their states, form inputs, colours, icons, typography, grid system and many more elements. Then, we built every block (paragraph of a certain type) based on those components. When that was done, the blocks were finally connected to the backend. So now if someone will ever change the padding of a button, it will change all over the website. The same goes for any other element in the system.
# Test coverage
While automated testing of Drupal websites is not something new, having to figure out what test approaches and frameworks to use to cover a decoupled project built in two different programming languages was an interesting journey. After some debates internally and conversations with the client, we decided to base test coverage on the Codeception framework. We leverage their API tests to cover API endpoints; end-to-end (acceptance) tests to cover the most critical user journeys of the frontend application, and finally, functional / unit testing to cover business logic on Drupal level.
# Minimal Viable Product
Who said that challenges can only be technical? Concern Worldwide had and still has a lot of ambitions for the project and it was a huge challenge to decide what should go into the initial project launch and what could be done later. This is especially true given the fact that the old site was in constant development over the past 7 years and nobody wanted a noticeable regression of its features. Thanks to the special relationship between the SystemSeed and Concern Worldwide teams allowing for very close collaboration, many open discussions around the business value were facilitated going into each epic, and we were able to launch the project well within the expectations of senior stakeholders.
Community contributionsProjects / modules:
- This project kicked off our Drupal / React.js boilerplate for a quick dev start with a set of best practices for testing https://github.com/systemseed/drupal_reactjs_boilerplate
- In order to fulfill all of the client’s requirements, we built Falcon (https://github.com/systemseed/falcon), an open source distribution for charities.
- We built a module for proper replacement of absolute links in the content for frontend app delivery: https://www.drupal.org/project/rest_absolute_urls
- https://www.drupal.org/project/rest_entity_recursive is an add-on to the Drupal core’s REST API for advanced REST experience for content with deep nested structure.
- https://www.drupal.org/project/consumers_token is another module which provides a replacement token depending on the consumer that makes the request to Drupal.
- We built https://www.drupal.org/project/commerce_decoupled_checkout for a decoupled Commerce experience.
- https://www.drupal.org/project/fieldable_path was our initial solution for decoupled routing (we moved to REST Entity Recursive later on).
- https://www.drupal.org/project/commerce_globalpayments to support Global Payments gateway (formerly known as Realex).
- https://www.drupal.org/sandbox/duozersk/3005285 to support Direct Debit payments.
- https://github.com/systemseed/commerce_paypal_ec for a nice Paypal Express Checkout experience
We submitted countless patches to the Drupal ecosystem. There are too many to list them all, but we’d like to state a few for reference. Here are our contributions to Migrate HTML to Paragraphs module which we used for content migration to Drupal 8:
Public presentations based on the project’s development:
- Decoupled software (https://slides.com/spleshka/decoupled-software)
- Thoughtful development & automation (https://slides.com/spleshka/thoughtful-automation)
- Kubernetes, KOPS and Helm (https://slides.com/spleshka/thoughtful-development-and-automation-5)
- Headless Drupal for multiple frontends (https://slides.com/spleshka/serve-them-all)
Why should this case win the splash awards?The Concern Worldwide project is a rare combination of collaborations across many different stakeholders and goals. Concern make an invaluable contribution to society by serving people in need all over the globe. Every member of the SystemSeed team contributed a piece of their soul towards perfecting the website and meeting the lofty goals and aggressive timelines.
Additionally we at SystemSeed have contributed to open source code more than we have ever done before, as we feel that this work will be extremely valuable for the Drupal community in gaining traction with the decoupled market as well as for people in the third sector looking for open source tools to help them accomplish their mission. The enormous effort of everyone involved in the myriad contributions to this project make it unique from a technical and social perspective and would thus be deserving of consideration for the Splash Award.