The eagle-eyed among you will have spotted that we’ve temporarily removed a few of the more advanced features of the website. Most notably, the “Most active articles” list on the side of the homepage. The less eagle-eyed will have also spotted that in recent months LR has gradually been getting slower sometimes to load, reaching a point in the last couple of weeks where the site has even begun throwing errors occasionally when it tries to load a page.
It’ll probably not surprise anyone to learn that these two things are related. In essence, LR has a problem – albeit a good one to have:
We have an ever-increasing number of readers making an ever increasing number of valuable comments.
Why is this a problem?
Weirdly, this is a problem because while we all want the internet to be a space full of long articles featuring lots of intelligent debate, in reality that isn’t what most of the internet is actually like.
The vast majority of sites which support comments, therefore, operate on the assumption that they are of low value. That they are something they can outsource to Facebook or another third party and ignore, or which they will mostly delete when they start to impact speed because they are spam.
Indeed over the last five years or so, this has even subtly driven the way the architecture of many web-based Content Management Systems is designed, leading to more of a focus on ‘core’ functionality and social / third party integration than on optimization of heavy comment load.
For most people that isn’t a problem, but here in LR Towers it presents a unique one – because our continued focus on longform writing and your continued dedication to writing good comments (and lots of them) places a strain on our server resources that increasingly the very architecture of those resources isn’t set up to deal with.
Just as a rough example, in the last two years we have had as many comments as in the first seven years of the site. Indeed if the current trend continues, we will be on course to have about 30,000 comments this year alone – excluding spam comments.
Which is a problem – albeit a nice one to have (and a reason why our volunteer moderators, LBM and Malcolm, deserve an infinite amount of free pints)!
Why is it an issue now?
Dealing with the increasing issues this is causing is perhaps the main reason why you haven’t seen many posts by me personally over the last few months. I’ve actually got a big Uber piece in the works, but right now my “LR time” has largely needed to be focused on trying to find new ways to make sure things run smoothly for you. This has meant gradually bringing in multiple layers of caching, building on increased server resources which the launch of the magazine has helped subsidize.
It’s just that now we’re hitting the point where we’ll need to make some larger changes behind the scenes to ensure things run smoothly for some time to come.
What can we do about it?
The good news is that we’ve got several things in the works that should help, and one way readers can too:
A new site design
For a few months now I’ve been working in the background on a new site design. This is nothing revolutionary and we’re due one anyway. But for us every little efficiency matters now, and it’s built to avoid making certain large queries in certain places which then slow everything else down. This’ll bring with it some restructuring of content, but also an opportunity for us to highlight older content, and content series (such as those on rail studies, and our various industry interviews with senior people) better as well. So it’s a win-win really.
Creating dedicated archive and events sites
In the shorter term, you’ll soon notice that we’ll fully split some of our older posts, and our events site, onto slightly different domains – archive.londonreconnections.com and events.londonreconnections.com. These won’t change in functionality (although commenting will be closed on archive). It’s just that three smaller sites will run faster than one large one.
The reader one: encourage your firms to subscribe or work with us
Of course the simple truth is that all the above is necessary because, ultimately, LR is a volunteer effort. We don’t get paid to write, and we’re fine with that. We love what we do.
Server resources don’t come for free, however, and whilst launching the magazine has helped ensure that meeting those costs is less of a pull on our own pockets than it was before, as this shows there is always more server resource we would pay for if we had the choice. And the less we’re focused on that, the more time we have to write. This is, ultimately, why we’re always keen to work with companies to tell stories, where appropriate, run adverts in the magazine or even just sell corporate print or digital subscriptions.
That last one is certainly something with which our readers can help. If you think that a subscription to LR would be of interest to people within your company, or that your company has a good story we can help them tell, then do remember to mention it when the subject comes up. Corporate subscription contact details can be found in our shop and we’ve got a site that explains our partner model here for those who would like to collaborate with us on written pieces.
Why don’t you start a forum! It’d help.
Sorry, but that’s not what we do. And trust me – it really wouldn’t.
Our overriding ethos has always been that we’re about sharing good information and allowing discussion around that information. Forums aren’t that.
There are also several other places on the web that already fulfill that role well (you will find plenty of LR people on District Dave and Skyscraper City).
As someone who has managed forum communities before, I can also tell you that Forums are insanely resource-intensive to manage with the size of the community that, from our comment base, we know we’d attract. Put simply, if we had to run a forum then we wouldn’t have time to write. And the writing is what we’re here, and what we love, to do.
The conclusion: don’t panic, we’re not!
Like I say, there are good problems to have and bad ones. This is definitely one of the good ones!
Luckily, from an LR perspective, doing this stuff is my “day job,” so it’s just a matter of pushing through and implementing the things we need to implement to keep things running smoothly and indeed make them run smoother.
Just bear with us a bit while we do that, and my apologies if it means you see the odd bit of functionality sacrificed for speed in the short term. Wherever possible, that’ll only be a temporary thing and efficiencies we introduce elsewhere will let us put some of the more resource-intensive functions back into play again in future. As well as bring you some new ones.
All the best, and thanks as always for helping make LR what it is – and stay tuned today for the second part of Pedantic’s look at the Kent Rail Study.
John Bull
April, 2017
JB: I had noticed both the sidebar and some time-outs over the weekend, was planning to drop you a line about it.
On a style point: an infinite amount of free pints, shouldn’t that be an “infinite number”, as a pint is a discrete quantity? It sounds better… I’m in favour of pints either way!
😉
The loss of the active articles list is a major one to my mind and based on the way I use the site – most recent posts don’t go back far enough if I check the site once a day. I hope a new site can restore this functionality soon
What Stuart said – echoed here.
However, a question:
Archived articles means comments are closed. OPK. But … what happens if some pertinent comment or “new” insight arrives relevant to one of those articles?
Should there be an “archive-updates file that is reserved for such, so that said insights/comments do show somewhere?
Or not?
The bit of code to do that is actually a really good example of the problem we’ve got – because to do it natively using CMS functions (which is what it’s set up to do) puts a massive load on that page thanks to the relative sizes of the posts and comments tables.
Yet at the same time, I fully appreciate that for many regular commentors it is a very useful piece of functionality.
So it’s certainly something I’d like to restore. But then the question perhaps becomes whether we should instead create, say, the option of registering a user account and then having a dashboard to which you can bookmark posts.
If we do that though, then we’d need to do it in such a way as to prevent random account registration (which is a major issue on free-to-register accounts). So perhaps then we’d have to say that it’s either something people can do as subscribers, or alternatively pay a tiny annual fee for (Say, £5) just to kill off the RAR.
But then that ITSELF brings with it a level of maintenance etc. etc!
@Stuart
You might like to know that there is an RSS feed – https://www.londonreconnections.com/comments/feed/ – that you can use with the likes of Digg Reader or a browser extension to keep up-to-date with the latest comments on here.
I mostly read the comments that way, and you won’t miss a single one.
It is somehow pleasing (in a reflective way, not pleasing that you are under stress) that this most valuable of resources on travel should be going through its own London Bridge-style crisis. Pain now for something much better soon. Probably less than a seven year programme, though
ACTUALLY… that’s a really good point Briantist – maybe I can pull in the XML feed! Will have a look.
Archived articles means comments are closed. OPK. But … what happens if some pertinent comment or “new” insight arrives relevant to one of those articles?
The archive is going to be anything older than two years at the moment. So frankly if it’s that old then it’s either not worth reopening or needs a new article anyway – even if that article was effectively just the new info and a pointer to the archived article.
Indeed that’s partly why I was soft-launching the industry news stuff – because, expanded, that acts as a way of saying “hey, look, this is stuff happening elsewhere that you may wish to comment on, but which isn’t on our writing radar.”
Same principle with the reading lists, really.
I’m with Stuart here (JB’s comments noted) – the loss of this function really is a retrograde step. I relied entirely on the list and the times that comments were posted. That way I only had to remember the last time I looked at the site to see all that was new. As I write this there are 7 comments on this topic listed and 2 on the Kent Route Study. Have there been any others on any other article since I last looked at the site? I have no idea.
I liked it the way it was.
@JohnBull
I understand the reluctance to run the site as a full forum. But there are many things from fora that you could take “inspiration” from, e.g.
Threading (for those that want it)
Ability to sort and filter comments, or comments made by certain people
Hyperlinking to other comments
Private Messages
Also, have you thought about moving the site to a platform like Reddit, which has all the features I mentioned above and would cost very little?
Just a thought – have you looked at becoming a publisher on Medium? Removes all the headache of running your own server and Medium was set up to host longform content.
John has said this a temporary change, so don’t panic guys.
The cacheing affects my personal settings (I like the Dark theme, and the top bar to be fixed, for example).
I find that a popular new article gets cached on some CDN with the default settings — I can get around it though, by appending ?nocache or any arbitrary POST string to the URL, so I don’t mind.
If you’re undergoing a redesign, however, it’s probably a fantastic idea to handle themes and other user settings client-side, via cookies & javascript.
How can one not leave a comment on an article about there being too many comments??!! 😉
@JB
I’m sure you know from some of the comments I have made the kind of work I do. The activity on the site will make active content management difficult.
I’m going to tell a story from 15 years ago, but it’s just as relevant now. I once worked with a customer converting their corporate web site to new technology. One of their business people tried to force me to implement a page which showed every item they sold. I refused, on the grounds that this dynamically-built list (which changed daily) would have huge impact on the servers building it. My quote was “it’s called a web page for a reason. If it was normal to return a book, it would be called a web book”.
How is this relevant? Well, look back on some of the recent active articles.
Kent Route Study part 1 – 342 comments
Crossrail’s Transition to being a Tube – 346 comments
How London’s Dream of Rail Devolution Died – 478 comments
Taking the latter article, those of us who visit daily will have seen all but the last few comments. Those who are new to the site will want to read the article. Ignoring pictures, on the last article I quote, word count for the article is 4713, word count for the comments is 66221. So every time regulars go to a comment on this article, they load the article they have read and 14 times the numbers of words to read what is quite likely to be the last comment.
So you will have worked out where I am going. Can you paginate the comments?
I too would have preferred to keep the Active Articles list rather than the Recent Comments. If there has been a flurry of comments on one article, quite recent comments on other articles can disappear from Recent Comments within a few hours. But as long as at least one comment has been made recently on an article, it would still show up in the Active list.
I too would have preferred to keep the Active Articles list rather than the Recent Comments.
It wasn’t an either/or. They work in very different ways on a database level.
So you will have worked out where I am going. Can you paginate the comments?
It’s an option. As I said in the post, I’m looking at ways to keep the behaviour but change the way it happens on an architectural level – pagination, for example, won’t help if WordPress still interacts with the tables in the same way, but filters post-facto.
pagination, for example, won’t help if WordPress still interacts with the tables in the same way, but filters post-facto.
It depends. If you can tie down your server poor responses to searching long articles, then fair enough. If it’s down to interactive user activity retrieving books rather than pages, then pagination will help.
Speeding up the ‘active articles’
Keep a little html file that is included (assuming PHP) in the page, and displays only the most recent active n articles. When a comment is posted, update the html file to reflect the change: this way the SQL queries to find recent active articles will only be required when a comment is added, not every time the page is loaded as it has effectively been cached until a change is needed. Most recent posts would be handleable in a similar manner.
@JB
A few years back I think you did an article analysing the London Reconnections readership using web data. I tried searching for it but it appears the search facility may have succumbed to the above problems. Don’t suppose you’d consider a follow-up to compare with what the readership is now?
@Snowy
Is this the article of which you speak? https://www.londonreconnections.com/2011/site-news-who-do-you-think-you-lr/
Unfortunately the graphs don’t appear when I view it. LBM
As a lay user of this site, I don’t pretend to be able to second guess the technical problems that have led to these changes, but they do now have one very serious problem. It is now not possible to see *all* the threads which are currently live, as today, where two threads (this one and the Kent one) have accumulated enough very recent posts to crowd out any trace of other live threads which just haven’t accumulated any posts in the last day or so. How do we know what is sinking below the horizon?
@LBM
Yes that was it! I was trying to remember the title pun, but hadn’t got it quite right. Same for me sadly, no graphics.
(At the time I was living in Auckland & rather enjoyed being 1 of few readers then in NZ on the map, I suspect times have changed, there’s even an Auckland transport site which has just redesigned its web page due to increased readership! Clearly rising public transport interest is a worldwide phenomenon)
@Snowy: I remember the old Perth suburban units arriving in Auckland. What an improvement that was over the old loco hauled trains with then (1993) 50 year old carriages! It is quite amazing to see that even Auckland is finally embracing public transport after 60 years of “the car is king”…
@Moosealot
Yes, that would work, and would reduce the server load a lot. Though I’m sure JB will have already have thought about what really needs to be created dynamically and what can be statically included when serving pages. Though I guess it depends what WordPress will let you do as to whether he can implement it.
Is a major part of the problem the trend towards ‘everything must be dynamically served’ – even when a static image would do.
I would also agree with pagination of comments. (And that long requested ‘take me to first unread comment’ feature.)
When using an iPad, some suitable navigation shortcuts would be nice…
@Graham H. Yes, this is the point I was making in my comment of 5 April. I did have a Bright Idea (one step down from a Cunning Plan) that I could search the main list of topics. It didn’t work for two reasons. One was that the list is date-based, latest at the top so Southern Heights (Light Railway) comment on ‘Making the Grade (Separation): The Bermondsey Diveunder’ isn’t visible as the Bermondsey site itself is too old; the second is that even if the article itself is featured it doesn’t show the most recent comment under the title – eg The Kent Route Study Part 1 shows the most recent comment as by PoP on 21 March whereas in fact if you go into it, it includes the comment by Southern Heights (Light Railway) today at 11:46
@Littlejohn -yes, you put it more clearly.(As it now is,the site relies on our memories of what’s current; in my view I’d rather sacrifice the “Archive” and leave it to be accessed via a tab, if that were possible).
@Graham H: +1
@JB
Hmmm, I see the issue now. Making big assumptions about the design and an articles table and a comments table, storing the date/time of the latest comment with its id to the articles table solves the issue. But that assumes that Word Press gives you that flexibility. The functionality which appears to be being used (from what you are saying) is not compatible with an active site with a large archive. A traditional database solution on a large system would use either the trick I’ve suggested, or programmatically ignore all but the most recent comments on each article. I’ve seen both techniques adopted. Purists dislike my solution as it involves derived data, but they usually aren’t the ones struggling to maintain a system when it fails to perform.
For the last few weeks the site has completely failed to render on Chrome.
Shows stripes of black, white and grey at the top of the page, then nothing.
Dave Russell: it works for me on Google Chrome, version 57.0.2987.133 under Windows 10.
I also use Chrome on Windows 10 and the site has worked for the most part recently. However, there was a weird glitch this morning where the page would constantly scroll down without me having any control of it.
Malcolm : OK, thanks, it’s my problem then.
Okay, thanks, it’s my problem then.
Perversely my posting at 13:18 gave a 504, but had in fact ‘taken’.
When I first posted the above (first sentence) I also got a 504 , but it appears not to have ‘taken’
For now I’m giving up on this site.
Oh come on Dave, what’s a few server errors between friends?
I’ve been getting server errors on every post for days now!!!
Just as I expect this one to do as well!
Just for the record, I also had a 504 error when I made my post earlier.
I have always struggled to keep on top of comments only looking at my phone as obviously there is no sidebar, but have tried Briantists suggestion of an RSS feed, something I have never previously known what it was for. Sadly, Feedly is confusing me so I will have to struggle on the old way, unless anyone has any other suggestions?
Actually I’ve just pushed a shedload of speed stuff live which should hopefully have helped enough for me to restore the activity box. we’ll see how it goes…
Wonderful to see ‘Most Active Articles’ back with the helpful time-stamp.
Thank you.
Odd, got Error 520 indicating error with LR server.
But when I clicked back button, comment above was posted.
I note that site no longer remembers my ID.
Good to see Most Active Articles back. I had noticed that during the interregnum once the Recent Comments list was full with only one or two articles covered it was only those that were commented on. It will be interesting to see if comments now cover a wider range of topics.
Also glad to see Most Active Articles back.
The site seems to be loading faster now (Firefox 52.0.2 / XP)(no comments about XP please, needed to run various useful programs that won’t run on 7 or 10)
@RayL
https://support.microsoft.com/en-us/help/15078/windows-make-older-programs-compatible
Thanks for your hard work JB.
A quick question – is it now necessary to re-enter name and E Mail address every time I post a comment? The site doesn’t seem to be saving my details since the change to “https” earlier today.
WW
Me too!
Also the hang-ups on posting are still occurring with a 520 error.
I suppose it will take more time & effort to sort it eventually…..
@GT/WW -and also here. On the other hand the return of active articles is most welcome for those of us who didn’t keep records of what was live.
Delighted to see “Most Active articles” back.
Baffled however as to why “From The Archive” appears twice.
@Nameless – JB is aware of the “From the Archive” problem and will fix it.
Still getting “not received” errors, which then claim that it has posted.
As I said, OK for us, but it will do our public face no good, unless it’s fixed – which is not going to be easy (?)
@Briantist 11th April at 20.48
Thanks for your suggestion but I should have used the word ‘load’ rather than the word ‘run’ . If the program won’t load then compatibility tweaks aren’t relevant. Examples of XP-yes/Win7-No include a program from 1994 that contains 721 addresses in a format unique to that program (it originally ran on Windows 3.1).
@RayL
I know this a little off-topic, but for that situation yon can still move up to Windows 10 for safety and speed and then run https://www.virtualbox.org/ to run the few things that you can’t get to run. Virtual Box is free and very good.