Michael DinersteinSenior Software Engineer with a Focus on Full Stack Web Applications & Datamd@missionmike.dev|Oregon, U.S.

View PDF

The Real Deal (TRD) runs a high-traffic real estate news website https://therealdeal.com. This is a Next.js website powered by headless WordPress in the back-end. It is hosted in AWS using custom-build Docker images in Elastic Container Registry (ECR), deployed to Elastic Kubernetes Service (EKS).

Challenge: Pod Resources

The back-end WordPress installation is home to over 100,000 articles and tens of thousands of tags. Dozens of writers and editors can access the site to create and organize content.

One challenge we faced was to allocate the correct amount of resources per Kubernetes pod, as well as to ensure the correct min/max number of pods are available for WordPress. With some light trail & error, we were able to iron this out and keep the site running smoothly for back-end editors and staff as well as front-end users.

Challenge: Checking Work

Initially this project was outsourced to a contractor. At the time, I was working with the internal team, focused on a different project.

When it was nearly time to be delivered, I checked the state of the work out of curiosity. I had hoped to learn a new technique or two from the developers. I was surprised, however, to find out that the bulk of this project didn't appear to follow healthy coding standards.

Based on what I found, I was nervous about the success of this project. This work was meant to be a milestone transition for the company's digital footprint: it combined seven CMS platforms for various real estate markets into a single powerful website to serve millions of visitors.

Fearing a potential botched launch, I brought up my concerns to management and I presented a plan to help get the project back on track. I recommended we:

• Follow React and Next.js best practices.
• Add tests with Jest in the front-end and PHPUnit in the back-end to cover critical functionality to start, then backfill additional coverage post-launch.
• Improve TypeScript usage and remove loose "any" definitions.
• Follow lint rules and guidelines with ESLint in front-end, and WordPress Coding Standards in the back-end.
• Organize code and files for clarity in front-end and back-end repositories.
• Implement a continuous integration (CI) pipeline to ensure all new pull requests don't break existing functionality.

While we experienced a bit of a setback to the expected launch timeline, I was much more confident in the launch of this project after we fixed the codebase. The code quality and development improvements greatly aided our future development and maintenance on this project

TRD News and TRD Pro were headless WordPress installations with several services.

The back-end includes:

• MySQL
• PHP with linting, formatting and testing.
• PostgreSQL for additional data stores.
• Redis for object caching.
• Kubectl sidecar to simulate Kubernetes integration.

The front-end includes:

• Next.js with TypeScript + React
• Jest, ESLint, Prettier

These services needed to be spun up and coexisting within local development environments in order for the team to effectively develop in a realistic setting. These complex setups were accomplished via VS Code Dev Containers, using docker-in-docker to orchestrate the services.

The TRD Data platforms used an even larger set of services, with some unique challenges:

• PostgreSQL for data storage, including a pre-seeded consistent local database.
• Airflow with pre-seeded connections and variables.
• Python with Black Formatter, Pylint, and Pytest.

In addition to using VS Code Dev Containers, we ensured that every collection of services was easy to spin up, and could be spun up within the context of a GitHub Actions workflow, in order to automate our CI processes in GitHub as easily as we could locally.

What good is an address if it's not accurate?

This was a problem we faced at TRD. It was simple enough to acquire the address as represented by a website or service where the data was sourced. However, we discovered that the source wasn't always accurate - there'd be a misspelling, or invalid street number variation.

To combat this, we set out to build a proprietary address matching system with geocoding solution.

First, we decided on the spec, and an acceptable level of accuracy.

Then, we put our heads together to plan the process and scaffold a concept.

From there, I worked to implement the code and logic, relying on Google Maps API as a source of truth for formatted addresses and geolocation data.

When I first joined TRD, I quickly noticed that there was a lack of automated code quality checks in the repositories I worked in.

I made it a priority to ensure that the team learned how important and helpful it is to automate things such as formatting, linting, and testing.

These practices were adopted org-wide, and led to a more robust and safe development environment, with safer and more streamlined deployment processes.

Cache Invalidation

The front-end Next.js setup is served behind CloudFront, so we had to implement robust caching and invalidation settings at multiple levels: Redis Object Cache, Next.js SSR, and CloudFront CDN.

We needed to maintain a balance of aggressive caching to save on AWS resources, while ensuring that the latest content was always available to visitors. To accomplish this, I led the effort to plan ideal cache settings for different sections of the website, as well as to architect a plan to trigger a cache-clearing mechanism to keep all content up-to-date when a post is edited in the back-end.

You can read more about this challenge on my blog here.

Image Assets and CloudFront Costs

The news website https://therealdeal.com/ houses hundreds of thousands of images, with several sizes for each image - millions of image assets. The cost of serving these images from the CloudFront CDN via AWS was very high, especially when certain articles would gain traction and go viral.

To remedy this, I proposed we serve image assets from a separate static domain, using Cloudflare's lower-cost CDN services to serve the image assets. This offloading of images saved thousands of dollars in monthly overhead.

I set up numerous automated processes for TRD to improve the development and deployment process for multiple products.

For the TRD Pro data platform and TRD News, I helped implement the following in CI for frontend and backend:

• Prettier
• ESLint
• Jest with coverage reports
• Build (ensure a successful npm run build process)
• PHPUnit
• PHPLint with WordPress Coding Standards (WPCS)

For TRD Data ingestion and scraping platforms, I utilized the following to set up CI processes:

• Pylint and Black Formatter
• Pytest

For TRD News and Pro, I helped the QA team by setting up the configuration to allow the test suite to run within the context of GitHub Actions workflows, using a Dockerized Selenium Webdriver setup with multiple browsers in Selenium Hub.

I was responsible for designing and maintaining several landing pages and marketing websites for the Recording Connection.