Site running in 'limp' mode following major failure

Forum News, Announcements and any issues you're having with the forum

Moderator: myglaren

Post Reply
User avatar
xantia_v6
Forum Admin Team
Posts: 9052
Joined: 09 Nov 2005, 22:03
Location: France or NewZealand
Lexia Available: Yes
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 825

Site running in 'limp' mode following major failure

Post by xantia_v6 »

On Sunday evening this site was moved to a new hardware node to fix some long term and short term performance problems.
On Monday night the server suffered a major disk corruption which (as far as we know) was due to some problem introduced when the server was moved (this was done by the hosting company, not me).

After spending some hours trying to recover the disk, it was deemed necessary to start again with a clean server installation, reverting to the database backup taken on Monday morning. This means that all posts made during Monday have been lost and are not coming back.

The forum is now running again, but some features, particularly image hosting, have not been enabled yet, because it takes many hours to upload the image files from the backup.

So you are free to start using the forum again, and we hope to have all the bells and whistles back by the end of tomorrow. Anything posted from now on will be kept in the database, we are not planning to lose anything more.

The BXC site and wiki will be restored tomorrow.
Hell Razor5543
Donor 2023
Posts: 13727
Joined: 01 Apr 2012, 09:47
Location: Reading
My Cars: C5 Mk2 VTX+ estate.
x 2993

Re: Site running in 'limp' mode following major failure

Post by Hell Razor5543 »

I have noticed that the bookmark tab (up at the top, alongside the 'your posts, tec.' tabs) is missing.
James
ex BX 1.9
ex Xantia 2.0HDi SX
ex Xantia 2.0HDi LX
Ex C5 2.0HDi VTR
Ex C5 2.0HDi VTR

C5 2.2HDi VTX+
Yes, I am paranoid, but am I paranoid ENOUGH?
Out amongst the stars, looking for a world of my own!
User avatar
xantia_v6
Forum Admin Team
Posts: 9052
Joined: 09 Nov 2005, 22:03
Location: France or NewZealand
Lexia Available: Yes
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 825

Re: Site running in 'limp' mode following major failure

Post by xantia_v6 »

Hell Razor5543 wrote: 10 Oct 2017, 21:00 I have noticed that the bookmark tab (up at the top, alongside the 'your posts, tec.' tabs) is missing.

Thanks for pointing that out. I had missed it, but it should be part of the file restore that is pending on the upload of a 7GB backup file via an ADSL link...
User avatar
NewcastleFalcon
Posts: 24563
Joined: 25 Feb 2009, 10:40
Location:
My Cars:
x 6866

Re: Site running in 'limp' mode following major failure

Post by NewcastleFalcon »

Pleased FCF hasn't disappeared, the messages started with the mild connection is not secure,as previously posted up, then at one stage turned into frenchcarforum.co.uk...URL does not exist!

Hats off to you chaps who voluntarily sort this stuff out.

Maybe when such crises occur when from a user point of view the site just disappears... some reassurring message like you used to get on the Telly...."Normal Service will be resumed as soon as possible!" could be put up. If thats a problem with the website being inaccessible, maybe a message on a FCF twitter/FB platform which members could consult if all attempts to connect to the FCF itself fail.

Course it dosen't happen very often but you are left wondering whether its your own machinery/internet or a problem at the FCF end.

Regards Neil
Only One AA Box left
687 Trinity, Jersey
User avatar
xantia_v6
Forum Admin Team
Posts: 9052
Joined: 09 Nov 2005, 22:03
Location: France or NewZealand
Lexia Available: Yes
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 825

Re: Site running in 'limp' mode following major failure

Post by xantia_v6 »

We did previously have a backup/test server that we could use to give a test-card message, but it was decommissioned a few months ago, as it was costing us money whether used or not.

I will try to implement something else for next time, but this is only the second time in 3 years that the site has gone down for more than a couple of hours, and the previous time was a software bug that also affected our backup site.
User avatar
Paul-R
Donor 2023
Posts: 6916
Joined: 07 May 2009, 16:24
Location: Wirral, NW England; Vaucluse 84, France
Lexia Available: Yes
My Cars: 2015 1.6 Blue HDi 120 Peugeot 308 Active SW
2013 2.0 HDi 163 C5 Exclusive Tourer
2003 2.0 HDi 110 C5 Exclusive Estate (Gone)
2001 2.0 HDi 90 Xsara Estate (Gone)
x 1369

Re: Site running in 'limp' mode following major failure

Post by Paul-R »

Oh dear, my sympathies. Obviously the forum doesn't like its new home! :)
As I get older I think a lot about the hereafter - I go into a room and then wonder what I'm here after.

Inside every old person is a young person wondering what the hell happened.

"Trying is the first step towards failure" ~ Homer J Simpson​
User avatar
Zelandeth
Donor 2024
Posts: 4731
Joined: 16 Nov 2014, 23:36
Location: Milton Keynes
My Cars: 2006 Peugeot Partner Escapade 1.6HDi.
1988 Renault 25 Monaco 2.0i.
1985 Sinclair C5.
1984 Trabant 601S.
1975 Rover 3500.
1973 AC Model-70.
x 1403
Contact:

Re: Site running in 'limp' mode following major failure

Post by Zelandeth »

Sorry to hear you've had problems, glad to see we're nearly back on our feet though. Hats off to those involved.

Always the way isn't it with hardware like this...

I've only been peripherally involved in running anything like this once, and it was for something like ten years run on truly ancient freely obtained "life expired" ex-corporate hardware out of someone's basement (no, really!) without a hitch. Sure, we had the occasional hard drive physically die, but the RAID array and backup system always saw to that with no fuss.

Then we moved to all new, warrantied hardware and moved to a proper co-location facility. Was a complete and utter pain...we had power supplies fail, we had more hard discs fail than we could count, and the remote management console at one point developed a truly maddening habit of going unresponsive - necessitating a three hour round trip to the facility to figure out what the heck it was doing. Usually approximately two minutes before someone got there it would spring back to life again as though nothing happened.

I was barely even involved, but that server took years off my life I reckon! We were pretty convinced that the thing was quite simply possessed by the time we saw the back of it. ...or more likely it had some unspecified and hard to pin down intermittent motherboard fault. Though we never did tie that up with the mortality rate of the hard drives.

I seem to recall that for the last couple of years before the project was finally canned as the mainstream Amiga usergroups died away it returned to its ancient basement server and the barely used new one was sold on to some unsuspecting big company...wonder if their IT guys had as much "fun" with it as we did...

Fingers crossed this isn't setting the scene here!
Current fleet:
06 Peugeot Partner Escapade 1.6HDi, 88 Renault 25 Monaco, 85 Sinclair C5, 84 Trabant 601S, 75 Rover 3500, 73 AC Model 70.
User avatar
CitroJim
A very naughty boy
Posts: 49526
Joined: 30 Apr 2005, 23:33
Location: Paggers
My Cars: Bluebell the AX, Polly the C3 Picasso, Pix the Nissan Pixo, Propel the duathlon bike, TCR Pro the road bike and Fuji the TT bike...
x 6160
Contact:

Re: Site running in 'limp' mode following major failure

Post by CitroJim »

Mike, a tremendous effort to get the forum this far in such a short time from 'bare metal' =D>

Well done! We are most fortunate to have you at the technical helm :)
Jim

Runner, cyclist, time triallist, duathlete, Citroen AX fan and the CCC Citroenian 'From A to Z' Columnist...
Hell Razor5543
Donor 2023
Posts: 13727
Joined: 01 Apr 2012, 09:47
Location: Reading
My Cars: C5 Mk2 VTX+ estate.
x 2993

Re: Site running in 'limp' mode following major failure

Post by Hell Razor5543 »

There are still issues. I just experienced a spell where I was told I did not have permission to access anything on the site. I have also noticed that the 'like' function has alignment problems. For example, on THIS post I see the '0', 'X' and the heart next to each other in that sequence, while on the other posts I see the heart and the '0' next to each other, and the 'X' below them.

Oh, and before I appear to be complaining, thanks for getting the forum back up and running. I was getting withdrawal symptoms! :D
James
ex BX 1.9
ex Xantia 2.0HDi SX
ex Xantia 2.0HDi LX
Ex C5 2.0HDi VTR
Ex C5 2.0HDi VTR

C5 2.2HDi VTX+
Yes, I am paranoid, but am I paranoid ENOUGH?
Out amongst the stars, looking for a world of my own!
User avatar
xantia_v6
Forum Admin Team
Posts: 9052
Joined: 09 Nov 2005, 22:03
Location: France or NewZealand
Lexia Available: Yes
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 825

Re: Site running in 'limp' mode following major failure

Post by xantia_v6 »

I am aware that there are still some issues, as I did not have access to all of the backup files yesterday, the site was put on-line with a cobbled together mix of old debug files that happened to be on my PC.

I have now applied the full backups from Monday morning, so most things should be back in order, but there are still some configuration issues and some peripheral software issues to sort out.

To resolve the formatting issues, please press ctrl-f5 or clear your browser cache.

I think that all of the images and pictures should be back now. If no-one reports any missing, then I will re-enable attachment uploads later today.
Hell Razor5543
Donor 2023
Posts: 13727
Joined: 01 Apr 2012, 09:47
Location: Reading
My Cars: C5 Mk2 VTX+ estate.
x 2993

Re: Site running in 'limp' mode following major failure

Post by Hell Razor5543 »

Erm, there may be issues with the pictures as well. :( The following link has 5 pictures, but the first and last are from different posts;

viewtopic.php?f=3&t=58722&p=552722#p552722

The bookmarks are back, so progress is being made. :D
James
ex BX 1.9
ex Xantia 2.0HDi SX
ex Xantia 2.0HDi LX
Ex C5 2.0HDi VTR
Ex C5 2.0HDi VTR

C5 2.2HDi VTX+
Yes, I am paranoid, but am I paranoid ENOUGH?
Out amongst the stars, looking for a world of my own!
User avatar
xantia_v6
Forum Admin Team
Posts: 9052
Joined: 09 Nov 2005, 22:03
Location: France or NewZealand
Lexia Available: Yes
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 825

Re: Site running in 'limp' mode following major failure

Post by xantia_v6 »

I thought that I had disabled attachment uploads, but apparently not, so the post mentioned above was made before the file restoration was complete and 2 of the attachments had the same name as restored files. I have deleted the attachments from that post, and will check for any other attachments added since Monday.
User avatar
xantia_v6
Forum Admin Team
Posts: 9052
Joined: 09 Nov 2005, 22:03
Location: France or NewZealand
Lexia Available: Yes
My Cars: -
1997 Citroen Xantia V6 (France)
1999 Citroen XM V6 ES9 (France)
2011 Peugeot 308 CC THP 155 (NZ)
1975 Jaguar XJ-S pre-HE (NZ)
x 825

Re: Site running in 'limp' mode following major failure

Post by xantia_v6 »

The emails sent by the forum do not currently have signed headers, so some mail systems may consider them to be spam. I will correct this as soon as I can.
Peter.N.
Moderating Team
Posts: 11563
Joined: 02 Apr 2005, 16:11
Location: Charmouth,Dorset
My Cars: Currently:

C5 X7 VTR + Satnav Hdi estate Silver
C5 X7 VTR + Hdi Estate 2008 Red

In the past: 3, CX td Safaris and about 7, XM td estates. Lovely cars.
x 1199

Re: Site running in 'limp' mode following major failure

Post by Peter.N. »

Welcome back! and thanks for all your hard work.

Peter
User avatar
ragger
Posts: 88
Joined: 20 Jan 2011, 22:54
Location: Darlington
My Cars:
x 9

Re: Site running in 'limp' mode following major failure

Post by ragger »

Very big thanks to Mike and everyone else who works so hard to maintain this wonderful resource.
Regards

Rob

C4 Exclusive - For Sale!
Xantia V6 Exclusive (the Beast!)
C1 Feel (it might be new but it is very practical for local zipping about)
C5 Estate
***********
Post Reply