Site running in 'limp' mode following major failure

Forum News, Announcements and any issues you're having with the forum

Moderator: myglaren

Online
User avatar
xantia_v6
Forum Admin Team
Posts: 5093
Joined: 09 Nov 2005, 23:03
Location: Christchurch, New Zealand
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 162

Site running in 'limp' mode following major failure

Post by xantia_v6 » 10 Oct 2017, 20:21

On Sunday evening this site was moved to a new hardware node to fix some long term and short term performance problems.
On Monday night the server suffered a major disk corruption which (as far as we know) was due to some problem introduced when the server was moved (this was done by the hosting company, not me).

After spending some hours trying to recover the disk, it was deemed necessary to start again with a clean server installation, reverting to the database backup taken on Monday morning. This means that all posts made during Monday have been lost and are not coming back.

The forum is now running again, but some features, particularly image hosting, have not been enabled yet, because it takes many hours to upload the image files from the backup.

So you are free to start using the forum again, and we hope to have all the bells and whistles back by the end of tomorrow. Anything posted from now on will be kept in the database, we are not planning to lose anything more.

The BXC site and wiki will be restored tomorrow.
1x

Hell Razor5543
NOT Alistair or Simon
Posts: 7960
Joined: 01 Apr 2012, 09:47
Location: Reading
My Cars: C5 Mk1 VTR estate, Icelandic Grey (I think she is called "Bluebird")
x 426

Re: Site running in 'limp' mode following major failure

Post by Hell Razor5543 » 10 Oct 2017, 21:00

I have noticed that the bookmark tab (up at the top, alongside the 'your posts, tec.' tabs) is missing.
0x
James
ex BX 1.9
ex Xantia 2.0HDi SX
ex Xantia 2.0HDi LX

C5 VTR HDi estate, 76518762
Out amongst the stars, looking for a world of my own!

Online
User avatar
xantia_v6
Forum Admin Team
Posts: 5093
Joined: 09 Nov 2005, 23:03
Location: Christchurch, New Zealand
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 162

Re: Site running in 'limp' mode following major failure

Post by xantia_v6 » 10 Oct 2017, 21:03

Hell Razor5543 wrote:
10 Oct 2017, 21:00
I have noticed that the bookmark tab (up at the top, alongside the 'your posts, tec.' tabs) is missing.

Thanks for pointing that out. I had missed it, but it should be part of the file restore that is pending on the upload of a 7GB backup file via an ADSL link...
0x

User avatar
NewcastleFalcon
Posts: 5597
Joined: 25 Feb 2009, 11:40
Location:
My Cars:
x 242

Re: Site running in 'limp' mode following major failure

Post by NewcastleFalcon » 10 Oct 2017, 21:45

Pleased FCF hasn't disappeared, the messages started with the mild connection is not secure,as previously posted up, then at one stage turned into frenchcarforum.co.uk...URL does not exist!

Hats off to you chaps who voluntarily sort this stuff out.

Maybe when such crises occur when from a user point of view the site just disappears... some reassurring message like you used to get on the Telly...."Normal Service will be resumed as soon as possible!" could be put up. If thats a problem with the website being inaccessible, maybe a message on a FCF twitter/FB platform which members could consult if all attempts to connect to the FCF itself fail.

Course it dosen't happen very often but you are left wondering whether its your own machinery/internet or a problem at the FCF end.

Regards Neil
0x
Current favourite playlist pickled egg jukebox-morse code songs

Online
User avatar
xantia_v6
Forum Admin Team
Posts: 5093
Joined: 09 Nov 2005, 23:03
Location: Christchurch, New Zealand
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 162

Re: Site running in 'limp' mode following major failure

Post by xantia_v6 » 10 Oct 2017, 21:57

We did previously have a backup/test server that we could use to give a test-card message, but it was decommissioned a few months ago, as it was costing us money whether used or not.

I will try to implement something else for next time, but this is only the second time in 3 years that the site has gone down for more than a couple of hours, and the previous time was a software bug that also affected our backup site.
0x

User avatar
Paul-R
Posts: 2467
Joined: 07 May 2009, 16:24
Location: Wirral, NW England; Vaucluse 84, France
My Cars: 2015 1.6 Blue HDi 120 Peugeot 308 Active SW
2013 2.0 HDi 163 C5 Exclusive Tourer
2003 2.0 HDi 110 C5 Exclusive Estate (Gone)
2001 2.0 HDi 90 Xsara Estate (Gone)
x 121

Re: Site running in 'limp' mode following major failure

Post by Paul-R » 10 Oct 2017, 23:27

Oh dear, my sympathies. Obviously the forum doesn't like its new home! :)
0x
As I get older I think a lot about the hereafter - I go into a room and then wonder what I'm here after.

Inside every old person is a young person wondering what the hell happened.

"Trying is the first step towards failure" ~ Homer J Simpson​

User avatar
Zelandeth
Donor 2016
Posts: 2571
Joined: 17 Nov 2014, 00:36
Location: Milton Keynes
My Cars: 1989 Saab 900i Auto
1996 Citroen Xantia Activa
1975 AC Model-70
1985 Sinclair C5
1993 Lada Riva 1.5E Estate
x 170

Re: Site running in 'limp' mode following major failure

Post by Zelandeth » 11 Oct 2017, 01:26

Sorry to hear you've had problems, glad to see we're nearly back on our feet though. Hats off to those involved.

Always the way isn't it with hardware like this...

I've only been peripherally involved in running anything like this once, and it was for something like ten years run on truly ancient freely obtained "life expired" ex-corporate hardware out of someone's basement (no, really!) without a hitch. Sure, we had the occasional hard drive physically die, but the RAID array and backup system always saw to that with no fuss.

Then we moved to all new, warrantied hardware and moved to a proper co-location facility. Was a complete and utter pain...we had power supplies fail, we had more hard discs fail than we could count, and the remote management console at one point developed a truly maddening habit of going unresponsive - necessitating a three hour round trip to the facility to figure out what the heck it was doing. Usually approximately two minutes before someone got there it would spring back to life again as though nothing happened.

I was barely even involved, but that server took years off my life I reckon! We were pretty convinced that the thing was quite simply possessed by the time we saw the back of it. ...or more likely it had some unspecified and hard to pin down intermittent motherboard fault. Though we never did tie that up with the mortality rate of the hard drives.

I seem to recall that for the last couple of years before the project was finally canned as the mainstream Amiga usergroups died away it returned to its ancient basement server and the barely used new one was sold on to some unsuspecting big company...wonder if their IT guys had as much "fun" with it as we did...

Fingers crossed this isn't setting the scene here!
0x
Current fleet:
89 Saab 900i Auto. 1996 Citroen Xantia 2.0T Activa. 85 Sinclair C5. 93 Lada Riva 1.5EFi Estate, 75 AC Model 70.

User avatar
CitroJim
A very naughty boy
Posts: 41305
Joined: 30 Apr 2005, 23:33
Location: North Bucks
My Cars: Sally the white Saxo, Jodie the white Ka (AKA the Dagenham Dustbin), Pixie the Sliver Pixo, a Giant TCR Bike and a Giant TCR Pro Bike.
x 923

Re: Site running in 'limp' mode following major failure

Post by CitroJim » 11 Oct 2017, 07:51

Mike, a tremendous effort to get the forum this far in such a short time from 'bare metal' =D>

Well done! We are most fortunate to have you at the technical helm :)
0x
Jim

Founder and only member of the Ford Ka Preservation Society...

Please feel free to follow me on Facebook and Strava :)

Hell Razor5543
NOT Alistair or Simon
Posts: 7960
Joined: 01 Apr 2012, 09:47
Location: Reading
My Cars: C5 Mk1 VTR estate, Icelandic Grey (I think she is called "Bluebird")
x 426

Re: Site running in 'limp' mode following major failure

Post by Hell Razor5543 » 11 Oct 2017, 08:30

There are still issues. I just experienced a spell where I was told I did not have permission to access anything on the site. I have also noticed that the 'like' function has alignment problems. For example, on THIS post I see the '0', 'X' and the heart next to each other in that sequence, while on the other posts I see the heart and the '0' next to each other, and the 'X' below them.

Oh, and before I appear to be complaining, thanks for getting the forum back up and running. I was getting withdrawal symptoms! :D
0x
James
ex BX 1.9
ex Xantia 2.0HDi SX
ex Xantia 2.0HDi LX

C5 VTR HDi estate, 76518762
Out amongst the stars, looking for a world of my own!

Online
User avatar
xantia_v6
Forum Admin Team
Posts: 5093
Joined: 09 Nov 2005, 23:03
Location: Christchurch, New Zealand
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 162

Re: Site running in 'limp' mode following major failure

Post by xantia_v6 » 11 Oct 2017, 08:36

I am aware that there are still some issues, as I did not have access to all of the backup files yesterday, the site was put on-line with a cobbled together mix of old debug files that happened to be on my PC.

I have now applied the full backups from Monday morning, so most things should be back in order, but there are still some configuration issues and some peripheral software issues to sort out.

To resolve the formatting issues, please press ctrl-f5 or clear your browser cache.

I think that all of the images and pictures should be back now. If no-one reports any missing, then I will re-enable attachment uploads later today.
0x

Hell Razor5543
NOT Alistair or Simon
Posts: 7960
Joined: 01 Apr 2012, 09:47
Location: Reading
My Cars: C5 Mk1 VTR estate, Icelandic Grey (I think she is called "Bluebird")
x 426

Re: Site running in 'limp' mode following major failure

Post by Hell Razor5543 » 11 Oct 2017, 08:38

Erm, there may be issues with the pictures as well. :( The following link has 5 pictures, but the first and last are from different posts;

viewtopic.php?f=3&t=58722&p=552722#p552722

The bookmarks are back, so progress is being made. :D
0x
James
ex BX 1.9
ex Xantia 2.0HDi SX
ex Xantia 2.0HDi LX

C5 VTR HDi estate, 76518762
Out amongst the stars, looking for a world of my own!

Online
User avatar
xantia_v6
Forum Admin Team
Posts: 5093
Joined: 09 Nov 2005, 23:03
Location: Christchurch, New Zealand
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 162

Re: Site running in 'limp' mode following major failure

Post by xantia_v6 » 11 Oct 2017, 08:47

I thought that I had disabled attachment uploads, but apparently not, so the post mentioned above was made before the file restoration was complete and 2 of the attachments had the same name as restored files. I have deleted the attachments from that post, and will check for any other attachments added since Monday.
0x

Online
User avatar
xantia_v6
Forum Admin Team
Posts: 5093
Joined: 09 Nov 2005, 23:03
Location: Christchurch, New Zealand
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 162

Re: Site running in 'limp' mode following major failure

Post by xantia_v6 » 11 Oct 2017, 08:52

The emails sent by the forum do not currently have signed headers, so some mail systems may consider them to be spam. I will correct this as soon as I can.
0x

Peter.N.
Moderating Team
Posts: 8749
Joined: 02 Apr 2005, 16:11
Location: Charmouth,Dorset
My Cars: Currently:
Peugeot 406 LX Hdi estate, 01, blue
Peugeot 406 GLX Hdi estate, 01, blue

In the past: 3, CX td Safaris and about 7, XM td estates. Lovely cars.
x 103

Re: Site running in 'limp' mode following major failure

Post by Peter.N. » 11 Oct 2017, 10:10

Welcome back! and thanks for all your hard work.

Peter
0x

User avatar
ragger
Posts: 158
Joined: 20 Jan 2011, 23:54
Location: Darlington
My Cars:
x 4

Re: Site running in 'limp' mode following major failure

Post by ragger » 11 Oct 2017, 13:37

Very big thanks to Mike and everyone else who works so hard to maintain this wonderful resource.
1x
Regards

Rob

C4 Exclusive - For Sale!
Xantia V6 Exclusive (the Beast!)
C1 Feel (it might be new but it is very practical for local zipping about)
C5 Estate
***********