stevegrant

I turned it off as I suspect that might have been having a negative impact on the database (bearing in mind that we have more than 2.6 million posts on here from as far back as 2008) - will have a play around with it over the next couple of weeks to see how much it actually gets used. It might be that it gets limited to subscribed users or something along those lines.

I think that took about 10 days, which is fairly ridiculous Yeah, I'm under no illusions here - it's entirely possible that the problems we've had will return just as quickly as they've vanished. I'm around 95% certain that the core problem is the database schema and how everything fits together, because the database uses the InnoDB engine but doesn't utilise some of the things InnoDB is especially good at (e.g. foreign keys linking tables together, etc), which makes the database quite inefficient. I suspect there are more writes/updates to the database on every post than should be necessary because of that inefficiency, which then causes a logjam and it spirals out of control. When I put the issue(s) to the Invision developers, they suggested ramping up the memory being allocated to the database, which frankly is a bullshit suggestion. A relatively simple forum structure shouldn't need multiple GBs of RAM being thrown at it in order to simply function at the most basic level. Basically they've engineered it badly and the "best" way to fix it from their perspective is to make the database so powerful that the inefficiencies aren't exposed, because otherwise they might have to admit they've fucked up somewhere along the line. There's been a new version (4.5) available as of around 4 weeks or so ago, but they're also onto incremental version 4.5.3 already, so there's clearly still a load of issues which I'm going to wait for them to resolve before we upgrade here - frankly, I've no idea how long the upgrade process might take. Knowing how long it took to convert from the old software, and how long that re-indexing process takes, it could be a matter of days...

Whisper it quietly, but I've done nothing to the setup for the last few weeks - it's just a weird fluctuation that I can't really explain. There's nothing that seems to actually cause the bad spells, it's not a spike in traffic or anything like that, there's no specific user action that I can see that's taking place, etc. Summary: dunno.

I had a load of email notifications this morning from a monitoring system telling me the site was up and down fairly regularly in the space of an hour or so. In all honesty, I've not really done much to have made it more stable over the last week or so, other than starting the re-indexing process which takes at least a week - it's currently 86% of the way through

Brentford in a similar situation too, having had a disappointing defeat yesterday at Birmingham. They were also pretty underwhelming - albeit with a weakened XI due to international call-ups - last week against Wycombe in the first round. I don't see it being a thriller

The weirdest thing is that the load balancer is effectively just a firewall at the moment because we only need one web server right now, so I would have thought traffic just automatically passes straight through, there's no logic for the load balancer to consider. Auto-scaling is in place if it's needed, as that's just a prudent way of setting things up so I don't need to manually change things when traffic increases significantly, as it does at half-time and full-time during games.

We're using an AWS MariaDB instance - not one of the most powerful ones, but it should be plenty for what this place does. In fact, the day/night of the Man City game during the restart when we set our new record for most people online simultaneously, we were actually on a smaller DB instance than we have now and it held firm. There's no real correlation between traffic and performance. I don't think I have the option to do that, but will have a poke around to see if it's possible.

Wonder if that might be as a result of the huge threads in that forum. In theory it shouldn't make any difference as it should only be looking at the topic itself, not any of the posts inside, but the way some of this site works under the hood, I wouldn't be surprised if there's some horrendous query that's slowing it down

Must admit I'm not 100% sure where I'd isolate any of those issues, tbh.

If Harrogate get through, the game will be at Doncaster anyway, Harrogate's new pitch won't be ready.

They didn't pick Ollie Watkins or Said Benrahma for today's first round game against Wycombe on the basis that they're expecting them both to leave so it's fairly pointless trying to work on a system when they're not likely to be a part of the team in the coming weeks. Anyone else who missed out today was on international duty.

phpBB still exists, but it's pretty basic these days. Absolutely zero appetite to move to another new platform, to be honest. We'd identified this one a few years ago but the issue was always having the time to do the migration. Their conversion process is not quick (it took three days running it on my fairly powerful Macbook rather than on a web server), especially as the old database was already around 3GB with more than 2m posts, so I don't really want to have to go through that sort of pain again I'm currently running the forum's re-indexing process, to see if something happened at one stage a while ago that screwed up the internal indexing. Worth noting that this is a code-based index rather than the indexing you'd ordinarily apply to the database itself, so god knows whether that'll make any difference or not, but that process was started on Thursday and is currently around 35% of the way through I'd say the last few days have been "stable" but not express pace. Looking at the latency numbers on the load balancer, it's averaging around 1 second, which is still too high really. But it's at least usable for now...

Third Round Bradford or Lincoln v Liverpool Bristol City or Northampton v Burton or Aston Villa Morecambe or Oldham v Newcastle or Blackburn Reading or Luton v Manchester United Derby or Preston v Brighton or Portsmouth Ipswich or Fulham v Rochdale or Sheffield Wednesday Manchester City v Bournemouth or Crystal Palace Millwall or Cheltenham v Burnley or Sheffield United Chelsea v Middlesbrough or Barnsley Wolves or Stoke v Gillingham or Coventry Leicester v Arsenal West Brom or Harrogate v SOUTHAMPTON or Brentford Newport or Cambridge v Oxford or Watford Fleetwood or Port Vale v Everton or Salford Leyton Orient or Plymouth v Tottenham West Ham or Charlton v Leeds or Hull

Second Round Burnley v Sheffield United Burton v Aston Villa Bradford v Lincoln Leeds v Hull Everton v Salford Wolves v Stoke Rochdale v Sheffield Wednesday West Brom v Harrogate Middlesbrough v Barnsley Derby v Preston Newcastle v Blackburn Morecambe v Oldham Fleetwood v Port Vale West Ham v Charlton Ipswich v Fulham Oxford v Watford Bournemouth v Crystal Palace Leyton Orient v Plymouth SOUTHAMPTON v Brentford Bristol City v Northampton Reading v Luton Millwall v Cheltenham Brighton v Portsmouth Newport v Cambridge Gillingham v Coventry

The reason for the lack of an update was basically because I was so fucking fed up of just reading a constant stream of whinging from people. One user who I attempted to help many years ago when he wanted a "fresh start" with a new username (but managed to out himself inside about 6 hours) and has been one of the more constant presences in this thread won't be troubling us again. The issue appears to be a particular inefficiency within the database (for those that understand this sort of thing, some of the key tables are denormalised and the indexing is a mess) - looking in depth at the structure of it, I'm amazed they've managed to make a viable business out of something that is basically a creaking pile of crap with a shiny skin that makes it look modern. There is a new version due out soon, so I'm hoping they've fixed some of these issues in that release, but it seems at the moment that we basically get three or four days of acceptable performance followed by a week of carnage. I've got some DB experts looking at it at the moment, but the concern is that if we make changes to the database itself (adding indexes, partitions on large tables, etc), that might have negative knock-on effects to other tables and big queries that are run often.

Kayne McLaggon scored a consolation for Barry Town United in their 5-1 Europa League qualifier defeat this evening against the mighty Faroese side FK Runavik.

To clarify, the 15% figure for non-league is for the first two weeks, then it goes up to 30%. There's also the slight caveat that the 15%/30% is based on the minimum required capacity for ground grading at that level of football. Taking Sholing as an example, they are in Step 4 of the National League System (i.e. the non-league pyramid, where the National League is Step 1), and the minimum capacity for grounds at that level is 1,350. 15% of that is 200, so that'll be their capacity for Stage 1 of the re-opening. Stage 2, from 31st August onwards, that capacity raises to 30%, so they'd then be allowed 400 in.

It can be today for you if you like? Getting pretty fucked off with the snide comments from people, to be honest. Absolutely nothing, that's what's so confusing. I've got an AWS specialist company looking at the database as I'm pretty sure the bottleneck is all there (the server CPU is constantly very low, so it's not a load issue), but ultimately the problem is actually likely to be in the software itself as there are a couple of queries that are probably run on every page load that are causing the issue. It'll just be a case of changing some settings in the database that will mitigate the issues until they're (hopefully) fixed in the next version of the forum software - that's supposedly due fairly soon.

They can take promotion and cup-related bonuses out of the equation, but the usual win and appearance fees would still be included, I think.

Pretty sure there is an instalment plan - it's just not being run by an external finance company where there could be potential legal issues with pro-rata refunds.

Playing around with some of the caching options - there are some really weird and seemingly inconsequential settings that, having looked at the database logs, might be doing a bit more than expected...