The Joys of Admin

I'm admin on a group on LinkedIn called French Connections – 65,621 members at the time of writing.  The other day I got a join request to review.  The profile was a little bit thin, and only 15 connections, so I sent a message… 

But read for yourself.  

Monday Jack Whittaker sent the following message at 12:47 PM

You are both members of French Connections on LinkedIn

Hi Charlie!   Thanks for applying to join the French Connections group! I have a slight problem - we like members to have some sort of connection to Francophonie (it helps to keep spammers at bay), and unfortunately you don't seem to have one. Probably because you haven't added it to your profile yet, or possibly I just missed it!   Perhaps you live in France? Drive a Peugeot? Love French cuisine?   Please reply and tell me why you would like to join, and if it’s appropriate, add it to your profile. Do please let me know within a few days or I’ll assume you aren’t interested. I look forward to hearing from you   Regards Jack Whittaker Moderator

• Today Charlie O sent the following messages at 3:30 PM 

You are totally wrong in your assumptions.   first of all i am Belgian and Flemish which speaks french and 12 other languages   secondly i lived in nice france for 5 years even got married to my first wife in Nice   But your reaction as a moderator says it alll   Forget about me and be less biased next time. little bonafide Ape.   I know more about france and the culture it stand for then you ever will. You  missed a golden opportunity and cant even write french I bet.   SO, dear wanker: au revoir, Le Petit Pois Un grand salut du LION Flament, which would gobble up little minded Brits or Yankess for breakfast. You must have nothing but 'faux' wannabees trying to understand what I did a long time ago in your group.  I was willing to help you learn about what you claim interest you. Learn to do proper research and get an education  upgrade in social communication on this type of platform you badly need dear FOOL.  Carpe Diem,  I rest my case Dear Moderator you are now challenged and I will make sure the public at large knows how biased you are ....

And you do not dictate what goes on or comes off my profile dear fool!

 <Charlie O sends me a link to his brother’s Linkedin profile for no obvious reason – to spare his brother's blushes, I’ve removed it from this transcript>

my brother would slpa you for saying what you said to me, shall I prosecute you in a court room?

Dumb fool

Moderate that, Mr Le Professuer de math qui est devenue un venduer de petit pois!

• Jack Whittaker sent the following message at 3:34 PM

Jack Whittaker 3:34 PM

Hi Charlie None of that stuff appears in your profile Any one of those things would have got you welcomed to the group without question - but I'm not psychic So - let's assume you are having a bad day - would you still like to join or have I offended you too much?

• Charlie O sent the following messages at 3:35 PM

People like me are protecting what you can find out about me for good reasons as is proven yet again!!!

Lets assume you dont know google exsist A simple charlie o would have answered all your dumb questions

learn to do your job as a moderator you dumb ape

now leave me be and stop wasting my time with your spam pre created paragraphs

No one wil tell me what to put on a profile Unlike you I dont need this linkedin app or site to achiev what I achieved in lifeI was trying to help your group You dumb fool which means I am the good kind of apple you kicked in the balls, bravo fool! ILL GLADLY TAKE SOME TIME NOW TO DOCUMENT THIS ON SOCIAL MEDIA AND MAKE YOUR ATITUDE GO VIRAL

Lets see how you deal with that as a moderator and my internet foot print is huge and I know how to SEO the right way ...

Skills which will garantee your group gets seen as it needs to be seen a bunch of wankers from the UK trying to make french clients or contacts!!!

No smart come backs I see you are a coward as well?

Last but no0t least you should never have been made a moderator ... you haven't got the skills for the role!

lets see where this ape works  and if i have him as client if so your company will be booted off the books

sql dba admin what a joke

Do you even speak french ape?  <enough to know how to spell "vendeur">

Brexit YES voter I bet ....

grow up fool!

• Jack Whittaker sent the following message at 3:49 PM

You seem to be slightly upset Charlie All because I asked a man who lives in Sheffield if he has any connection to francophonie I'm curious - why do you feel that I insulted you?

Sadly, Charlie O never replied

I blocked him from the group, of course – well, obviously

But I took his advice and googled his name.  I found his Facebook account – and it’s chock full of adverts for all sorts of crap.  Remember this line in my initial message to him?  “We like members to have some sort of connection to Francophonie (it helps to keep spammers at bay)”.  Turns out he IS a spammer!

Activity Group not synchronizing

There was an activity group failover in the night.  Some sort of glitch – glitch happens.  The Primary flicked over to the Secondary and the Secondary became the Primary, no problem.
The Old Primary became the Secondary
BUT…

Databases on the New Secondary are marked as Not Synchronizing

The error log says: A connection timeout has occurred on a previously established connection to availability replica with id [5FA3CBDB-5504-43E2-8697-2C713110307B].  Either a networking or a firewall issue exists or the availability replica has transitioned to the resolving role.

Well, the replica has certainly transitioned.  Other instances on the nodes were unaffected

A bit of googling finds this article.
Microsoft recommends restarting service on secondary replica.
The problem occurs because of an internal race condition.  What on earth does that mean?  Ah - it's a bit like a deadlock.  

And another article.
Resolve Secondary Replica Database Not Synchronizing You cannot ONLINE a secondary replica availability group database Since a secondary replica availability group database is really in an ongoing restoring state, it cannot be set Online like the primary replica. Remember it is already in the Online role, according to the DMVs.

Solution Restart the SQL Server instance hosting the secondary replica. This will initiate recovery on all databases including availability group databases, and so long as SQL Server can access all the database files, the availability group database affected should recover successfully and resume synchronization with the primary replica. 
OK, that all seems pretty clear.

Next question - can we do it now?  In the working day?  
Well, if we don't, we have lost our Availability group resilience - that's not a Good Thing

If you look at the picture above, we have an estimated recovery time of 0 seconds.  I've added a few extra columns to my Dashboard - there's a list of them here.  Now that seems too good to be true.  It's been down for a few hours, are you seriously expecting me to believe that there is no data waiting to go across to the Secondary?  Pshaw! I say to you.
One of my colleagues thinks that Dave has mentioned this issue before – but he isn’t sure.  Dave is on holiday - because of course he is.  Whenever you need someone who knows stuff, Sod's Law mandates that they are not around.  

So we are a bit scared of restarting the services on the secondary – is it going to impact on the primary as it catches up with the backlog of stuff to be transferred? 

We decide to wait until everyone goes home for the night. 

I stop the agent service.  

I stop the SQL Server service

I start the SQL Server service.  

I start the Agent service

I go and have a look at the Dashboard

The two small databases have already sorted themselves out.  

The two big ones estimate 20 minutes and 10 minutes to recover, but it quickly becomes obvious that this is an over-estimate.  The actual time taken to recover was about 3 mins for the larger, 1 min for the smaller.  There doesn't appear to have been any impact on the Primary, but it fixed itself before I could check.  

When Dave returns from his hols he confirms that restarting the services on the secondary will fix this; it takes seconds he says.    He reports that it sometimes happens when he manually fails over e.g. to apply patches.  Out of hours, of course. 

So with hindsight, I would feel much more confident about restarting the service on the Secondary node, and do it in the working day rather than run the risk of losing data.