Server upgrades this week

Some customers have had some disruption to Lamplight over the last few days. If you have been affected, we’re really sorry for the inconvenience this has caused. This post explains a bit about what’s happened.

The programming language that Lamplight uses is called php. The short explanation is that it was time to upgrade php and this upgrade caused some unforseen issues. While we test pretty extensively in different ways before any update, and had made most of the changes necessary for Lamplight to run on the new version of php, we missed some.

The update to php was a big one, from version 7.4 to version 8. Computer software is being continually updated, for security, performance or functional reasons. Old versions become obsolete as newer versions come in – so php 7.4 will only receive security support and updates to the end of November this year.

Generally these updates are incremental, and the move from one to the next is fairly easy to make. However, major version updates contain changes that can break existing software, and so broader changes to Lamplight are required.

One of the big changes in php 8 is that it becomes a lot more demanding when things aren’t quite right. So for example, one of the issues we’ve seen this week has been caused when personal settings use a deleted data view for the profile header strip. In version 7.4, php would grumble a bit about this but carry on anyway. In version 8, it has a full on lie-down tantrum in the middle of the supermarket and refuse to carry on. And so in these circumstances, the profile wouldn’t load.

Another example we’ve seen was caused by the difference between the number 0, an empty piece of text, and a single space. Again in the previous version this was overlooked. In the new version, in the wrong place, it just stops.

In the long run, this is a good thing. As a generalisation, a stricter programming language makes for more correct software. But this week we’ve had to identify and resolve these apparently minor and fairly hard-to-spot issues as they’ve become apparent. These ones that were left were in some out-of-the-way corners of Lamplight, only kicking in in particular combinations of circumstances.

Before we make any changes to Lamplight we run a series of automated tests to verify that the components are doing what they are supposed to. As of this morning there are 7,132 of these tests, that make 23,475 checks. Every one has to pass before we make the changes. We also use a “robot browser”, for want of a better phrase, that uses Lamplight as you do – it creates profiles and activity records and running reports and various other things, checking that what it sees when it does so is what it expects. In the past few weeks these tests have helped us to identify most of the changes we’ve needed to make to the 325,000 lines of code that make up Lamplight to enable it to run on php 8. Here it’s adding a work record – you can see it using Lamplight and then checking the results.

An example of the automated testing we run using Ghost Inspector – ghost using the machine?

The errors we’ve seen this week slipped through this net. So we didn’t have a test to check profiles loaded while using a deleted data view, for example. We’ll continue to add to the set of tests that we use to increase our confidence when changes are made.

Fortunately, major changes to php are fairly infrequent. php version 5 was released in July 2004; php 7 in 2015, and php 8 in 2020. So while there will be incremental updates, we’ve got a few more years at least before the next one.

We think we have now addressed most of the issues, and again, we’re sorry for the disruption this week. And thank you for your patience while we’ve worked through it all – you can imagine it’s been quite busy at Lamplight this week. I hope that understanding a bit what’s going on is helpful.