Safe Practices for Keeping Your Site From Exploding

I’ll start by saying that DigiSavvy is a Pantheon Agency partner. This article discusses how I use Pantheon for our projects. No affiliate links or anything crazy like that; just my own observations and opinions on running a smooth site.

Why am I writing this in the first place?

I’m not selling Pantheon here; their platform sells itself just fine without any help from me. What I want to discuss are practices that help website owners maintain security and sanity when running tests and updates with their website. I’m of the belief that you shouldn’t be monkeying around with your website environment. You either hire a dev ops person who specializes in that stuff or you host with a provider who manages that aspect of your platform. Those are just my two pennies.

Tinkering with your website

Tinkering with your site is fun, we love doing it, and if you’re running a WP website, there are a lot of great plugins to install and test. That’s all a part of the WordPress ecosystem and a big selling point for the popular Content Management System (CMS).

I was consulting with a friend at the SBDC who runs a website with good traffic, that covers Pasadena events and culture.

He was flummoxed because he’s experienced several hosting issues stemming from a malware attack against his site. While most of his issues were resolved, we did do a sweep of his plugins, filesystem, and database.

The one thing we noted were the presence of inactive performance monitoring plugins. In particular Query Monitor. Not a big deal, but his hosting support installed the plugin during their investigation to determine if a plugin was causing additional resource related issues. Support didn’t remove the plugin. That was odd to me; what’s more was the fact that support installed something on their customer’s site in the first place. To me, that’s a no-no. I say that because there are additional tools support staff could have run to determine what was going on. New Relic is a great tool for identifying performance related issues, for instance. In short, support should not be installing random plugins on your site, regardless of their intent. I won’t get into the support conversation my client shared with me either. It was an eleven-page-long exchange of “do this, do that, remove this plugin” nonsense. It was, as I like to call it: “shotgun troubleshooting.” I also won’t go into the mess of files named with extensions like “.bak” or prefixed with “copy_1-13-2017_”. No. Just no!!! Stop that already!

Tinker responsibly

At a minimum, you should have a staging site to test your changes on. At. A. Minimum! Ideally, your staging environment is a like-for-like clone of your live environment; with the same specs, same caching, and the same server configuration. The reason for that is so that your changes on staging will work the same way on your live site. There should be parity between these environments.

Adding predictability in your testing will help you to breathe a little easier.

Beyond mere staging

When I worked as a storage administrator for a large insurance corporation, our development teams had dev, test, and production. I don’t know what they used for version control, but no changes were made in production (aka the live application or website). Changes were pushed from staging to production. Development happened on dev and testing is where more refined tests were run.

Changes were pushed from staging to production. Development happened on dev and testing is where more refined tests were run and without interrupting the development of the project or causing issues and instability in production. Filesystems were tightly locked down, only the user assigned for version changes had access to change files within the filesystem; again, those changes came from pushing from dev to test and test to prod.

I recommend a similar approach in managing your website. But isn’t that overkill? How important is that website to you? For my clients, their website is their lifeline. If it goes down, they’re in a world of pain. Downtime means lost revenues. If you’re reading this, you can probably relate.

Dev, Test, and Live

When I’m building out projects, I make use of dev and test the most. Immediate changes I’m actively working on go to dev. Test is where the refined changes that are ready for review go; that’s the site I share with clients.

Siteground and WP Engine have decent staging environment setups. While both offer the ability to sync changes back and forth from live to staging, that feature is gimmicky and error-prone. I’ve been burned by sites going down with each of those providers.

If you’re an enterprise customer, or on a VPS Pagely will provide staging. But even Pagely doesn’t think staging site syncing is safe, or they believe that it’s problematic.

Pantheon has this figured out, and it’s really the “crowning achievement” of their platform in my opinion. Sure, you can sync from dev to test, or dev to live even and back from Live to test and dev until your eyes bleed. But syncing isn’t the point. Providing a sane practice for managing your website IS the point and having a three step environment to test and review changes is a good practice.

What if I don’t want to interrupt the dev site with an awesome feature I’m working on?

Pantheon has a little feature called multi-dev. At first, I didn’t understand what the fuck it did beyond what creating a new git branch would do. Why did I need that feature? Josh Koenig, founder and head of product at Pantheon, performed a demo using multi-dev. People, I thought it was the coolest. Yes, for developers, it’s sorta like creating a feature branch for that feature you’re working on. In addition to that you get a whole new website environment to test on, so you then have dev, test, live AND your-feature-branch environment! You can then merge that multi-dev environment back into dev easy peasy.

Mutlidev isn’t something the casual site user will use, but it’s a wonderful tool for developers.

That’s all well and good, but what is a non-techie to do who just wants the thing to work and be able to do work on their site?

This one-hundred times. Pantheon’s setup isn’t the most user-friendly for non-techie users. For all of the developer resources on hand, it can seem like making it easy to use and customize like a cPanel based site wasn’t a consideration for Pantheon. I think about this a lot. Because I do believe when we build a website for a client they should be empowered to make updates to their site as they see fit. That said, I do think that a website should be “hard to break,” as Billy Jack Erickson says.

I have a number of clients on Pantheon these days, and all of them require training on how to manage updates because updates don’t get run on Live. Try it. You can’t. That ability is disabled, so it is necessary to apply updates in dev and then push to test, and finally to live. They get updated locally and pushed to dev; or changes can be pushed from dev (to do that, dev must be set to SFTP in Pantheon).

That process can be cumbersome for many users. That said, the value is that there is a solid and hardened process for applying updates and that process helps prevent outages.  One of my recent client site launches was for Urban Southern. I think Meg and Regina, there, would say they aren’t web developers, but they’ve pushed code using git principles. My heart swells with pride when I see that.

That process of updates encourages stability and testing, and I can’t tell you how I would make the system different or more friendly while lessening or eliminating the ability to shoot yourself in the foot, so to speak.

The need for—and value of— testable and versionable updates

Just about any developer I know agrees that testing and version control are a big part of ensuring a smooth development process. I’m not doing a lot with unit testing at this point, but version control is an essential part of what I do.

Being able to test changes in a non-destructive way to your live site is a key, being able to roll back changes intelligently just makes sense. Now, does that mean that I think all users should go and learn Git? Kinda sorta, I do think that would be a good thing to pick up.

Testing isn’t just about testing your updates on a random server either. You ideally want to test your changes that are the same as one another, one thing that reliably works on the dev server should work on the production server.

The cost of support when you have full control of your site

So, if we go through all of this rigamarole, what are we gaining? Certainly, having a setup like this isn’t cheap. In fact, it can seem like there are a few too many unnecessary steps just to install a single plugin. Perhaps there are a few too many steps here, but I don’t know how one gets around them honestly. What I will say is this:

When you invest in a website management platform, not just hosting, you also invest in your website—and business’s—stability. If your business is your website, what is that worth to you? I’m willing to bet it’s worth a lot more than ten bucks, thirty bucks, or one-hundred bucks per month.

Not having a solid platform in place and not having adequate support has a cost, too. Time, money, and peace of mind. In the example of my friend, from earlier, shotgun troubleshooting can often get the job done, but it’s careless and often causes other unforeseen issues. The more control one has over their server increases the chances to bring that server down; not only that but if your site is hosted on a shared hosting account, your site can be brought down by a “noisy neighbor.”

The pitfalls of shoddy support and legacy hosting environments

Shared hosting is hot garbage, and everyone knows it, or if they don’t, they should. Again, that’s just my opinion. Shared hosting is inexpensive and easy to get something up and running. Clusters, VPSeses aren’t much better either. Once your website hits that critical point where uptime truly matters, then you should consider moving your site to a dedicated or containerized config. A “noisy neighbor” can bring down an entire server, sending hundreds of websites offline.

One client that I migrated last year was on a server that was ten years old, support said. Now, let’s not look past the fact that this server has been running for ten years. It’s partly impressive and slightly “WTHF!!?”

The average Pantheon server age is around 50 days. We prefer a fresh machine to having to reboot one for a kernel update. — Zack Rosen, CEO & Co-Founder Pantheon

Where possible, you want servers that haven’t been sitting around not getting their updates on time; and certainly, you wouldn’t want one of your sites getting taken down for “regularly scheduled maintenance.” I have some clients with sites that go down on a particular day of the week each month due to such maintenance, and I’ve urged them to consider moving.

A note about backups

When I was a backup administrator the saying went:

Your backups are only as good as their restores. Always test your restores. — A person who didn’t test their restores

Sarbanes-Oxley regulations dictated that particular financial backups be tested quarterly for their ability to be restored. I think that’s a good idea. If you utilize backups from your hosting provider, it’s a good idea to periodically check the quality of those backups and test a restore. Doing so quarterly is a good idea. WP Engine does an excellent job with backups, Siteground typically does a good job with their backups, too. Obviously, Pantheon does as well.

It is my opinion that you shouldn’t use plugins like Backupbuddy or similar plugins that you install on the site to perform backups. Those plugins utilize your website resources for running backups and can cause performance issues, leave large orphaned backup files behind, etc. Sync, VaultPress, or Blogvault are all services that run external of your website and take a minimal amount of resources to run; I’ve had excellent experiences with Blogvault personally, and I use them on my maintenance clients.

In short…

I believe modern hosting considerations should address these points:

• Multiple environments to run your site, updates, and tests.
• Those environments should be a like-for-like copy of your live environment.
• Disable ability to run updates on the live site. Never run updates live, push them instead from a tested environment.
• The support that doesn’t do “shotgun troubleshooting.”
• Provides modern tools to monitor site performance like New Relic, like Pantheon does.
• Maintains a platform with fresh servers ready to go and can run updates that don’t require downtime for your website.
• Provides around the clock support.
• Provide regular testable snapshots/backups
• Testable backups and restores
• Provides containerized architecture to ensure performance and mobility of the platform to scale as needed.
• Limits end-user freedom to install software or to make filesystem changes. Web hosting should be opinionated, especially managed platforms.

What are your thoughts on hosting these days? The landscape has changed considerably over the last few years. Comment below, I’d love to chat! I love Pantheon and I didn’t even get into their Terminus CLI tool for managing upstream updates. It’s life!!! Anyway, peace out!