How DAC Mode Works In Exchange 2010 SP1…

Exchange admins are a bit confused when it comes to DAC mode. Though most of the them know that it has to be used in most scenarios when deploying Exchange 2010 SP1, they are not sure as to what the function of DAC mode is. DAC stands for Datacentre Activation Coordination and it works using…

Exchange admins are a bit confused when it comes to DAC mode. Though most of the them know that it has to be used in most scenarios when deploying Exchange 2010 SP1, they are not sure as to what the function of DAC mode is.

DAC stands for Datacentre Activation Coordination and it works using a new protocol named Datacentre Activation Coordination Protocol (DACP). With Exchange 2010 SP1, all kind of DAG deployments can use DAC mode, even one that has just two DAG members. DAC mode was designed to avoid the “split brain syndrome” that will occur in one particular scenario of datacentre switchover. Let me explain the scenario with the help of an example. (The images used are from TechNet video presented by Scott Schnoll). I have two datacentres with two DAG members in each datacentre. The primary datacentre has the file share witness (hub server) and everything works as normal.

Two DCs in normal operation

My primary datacentre goes down, let’s say due to a power outage. Both my DAG members and the hub server hosting FSW goes offline. Along with that, my WAN connection goes down as well. This will cause my entire DAG to fail, as there are only 2 DAG members online out of the five in the cluster (4 mailbox servers and one HUB server hosting FSW).

Primary DC fails

In this scenario, I try to restore the DAG in the secondary datacentre by using an alternate witness server (hub server in 2nd datacentre). Both mailbox servers in the 2nd datacentre comes online as there are now 3 members online (out of 5). Everything works as expected and I am happy.

Let’s say that the power in my primary datacentre gets restored and both my mailbox servers and primary witness server comes online. The servers will come online before the network devices (which is how it is usually). Now, my WAN connection is still down and Exchange servers in both datacentres are up and running. My secondary datacentre has already been activated to serve the clients manually. But, since there are three servers online (out of the 5) in the primary datacentre as well (2 mailbox and 1 hub), the mailbox servers will try to activate databases as well (or will it?). This will lead to both DCs hosting same databases, which cannot and should not happen. DAC mode was developed to make sure that this doesn’t happen.

Split brain syndrome

Now, how does DAC mode make sure that the servers in my primary datacentre doesn’t mount databases when the power is restored? When DAC mode is enabled, it changes the way Active Manager works. When a DAG member gets restarted or when the Exchange Replication service is restarted (when DAC mode is enabled), it sets the DACP bit (in MSExchangeRepl.exe) to zero (which means that it can’t mount any database). Active Manager can only mount a database in only two conditions.

  1. If it can communicate with all other DAG members, in both datacentres (which won’t be possible in our case as the WAN link is down).
  2. If it can communicate with atleast one DAG member which has DACP bit set to one (which is a DAG member that is serving clients). In our case, these are the servers in the secondary DC. But, as the WAN connection is down, this won’t be possible.

Hence, our mailbox servers in the primary datacentre won’t mount databases, as the DACP bit will be set to zero, while the servers in the secondary datacentre will continue to work.

DACP bits

This is the only scenario for which DAC mode was designed – to make sure that DAG members won’t mount databases when power gets restored in the primary datacentre, while secondary datacentre is “active” and WAN connection is down.

Once the WAN connection is restored, DAG members in primary datacentre can communicate with all other DAG members and see that the databases are already mounted in the secondary DC & they won’t mount any database, which is the normal behaviour.

Hope this clear the confusion around DAC mode.

36 Comments

  1. Thanks for the explanation!! It has been very useful to understand what happened last weekend as we had to shutdown all our Exchange 2010 SP1 environment (2 sites with 3 mailbox server in each site).

    1. Rajith Jose Enchiparambil says:

      Glad to help Miguel.

  2. Syed Wasil Uddin says:

    Hi,
    I want to study all the possible scenarios for DAG. Can you Assist i just name the types. i will study it myself, if its easy to share its links i will highly appreciate.
    You did a great work its appreciable.
    Thanks in Advance.

    1. Rajith Jose Enchiparambil says:

      Hi Syed,

      It’s not as easy as listing and learning. Different business requirements lead to different designs of DAG.
      I think it is better to start with TechNet.

      Thanks

  3. Thiyagarajan says:

    Hi Rajit, Thanks for this article. I have small clarity here. Initially the primary centre having active mailboxes and due to the power failure the secondary mailboxes act on active node on the secondary data centre.
    Now the question is, when the power failure happened, DACP set to zero (0) from 1 due to replication failed to another DAG member @ secondary data centre ? If i misunderstood, please clarify…

  4. Ravi Chandra says:

    Very clear explanation. when i read about DAC mode in Technet, its very confusing.
    Now i am clear about DAC mode. Thanks very much for the good article !!!

    1. Rajith Jose Enchiparambil says:

      Thanks Ravi.

      1. Raghavendra Jadhav says:

        HatsOff Ranjit!

        Nice Article on DAC

        1. Rajith Jose Enchiparambil says:

          Thanks Raghavendra

  5. Amit Rawat says:

    Rajith,
    Thank you so much to make it easy. Keep writing such a nice tech article.

    1. Rajith Enchiparambil says:

      Thanks Amit

  6. Hi Rajith,

    Please suggest how can I configure dag in mulitple sites with users hosted in both location.

    Site A
    1. CAS/HT1
    2 MBX1
    Site B
    1. CAS/HT1
    2. MBX2

    1. Rajith Enchiparambil says:

      You add MBX1 and 2 as DAG members. As DAGs can span Ad sites, things are fine

  7. very nice….explained lovely….really cleared my funda of DAc mode….very helpful….gud work bro.

  8. Hi Rajith. Good Explanation however had one doubt. As you said

    “Active Manager can only mount a database in only two conditions.”. The second condition

    If it can communicate with at least one DAG member which has DACP bit set to one (which is a DAG member that is serving clients)

    Now once the WAN link is restored, DAG members in primary site will be able to contact the DAG member which has a DACP bit set to one.
    So in that case, Would the DAG member on the primary site mount the database?

    Thanks

    Vilas

    1. Amit Rawat says:

      No.. if DAG knows that there is bit 1 enabled for other DAG members then it understands.. Databases are mounted there. So it will not mount database in primary DC and this is how DAG works. Problem only occur when split brain situation happen due to power outage + WAN network and after sometime Power restore but WAN takes time.

  9. Awesome!! Good explanation.

    1. Rajith Enchiparambil says:

      Thanks Ezakial.

  10. One confusion, at one place you said : “Active Manager can only mount a database in only two conditions.”
    1.If it can communicate with all other DAG members, in both datacentres (which won’t be possible in our case as the WAN link is down).
    2.If it can communicate with atleast one DAG member which has DACP bit set to one (which is a DAG member that is serving clients). In our case, these are the servers in the secondary DC. But, as the WAN connection is down, this won’t be possible.

    and at last you said :
    Once the WAN connection is restored, DAG members in primary datacentre can communicate with all other DAG members and see that the databases are already mounted in the secondary DC & they won’t mount any database.

    Please understand me.

    1. Rajith Enchiparambil says:

      Where are you getting confused Annie?

  11. Ravinder Gill says:

    Great Rajith, far better article then on Microsoft site…very easy and itself explanable article. Thanks a lot.

    1. Rajith Enchiparambil says:

      Thanks Ravinder.

  12. Impressively simple yet technical explaination. Congratulations, thank you!

    1. Rajith Enchiparambil says:

      Thanks Hernan.

  13. kochumon Nair says:

    good explanation, thanks for posting this. :)

    1. Rajith Enchiparambil says:

      Thanks Kochumon.

    1. Rajith Enchiparambil says:

      My pleasure Anant.

  14. Great job! Very nicely explained.

    1. Rajith Enchiparambil says:

      Thanks Vik.

  15. Akash Phoenix says:

    A great article!!! Very clear description. Wherever the pics are from but they fit perfect to this article!!! Will surely add this link in my blog.

    1. Rajith Enchiparambil says:

      Thanks Akash.

  16. uh60flyinlow says:

    Outstanding Sir. I understand it, but I’ve been searching for a way to explain it to the “smarter” people who sit a few IT seats higher. This should bring it down to their level. Thanks a million. :)

    1. Rajith Enchiparambil says:

      Glad to help.

  17. Good Work. Found it really informative.

    Thanks :)

Leave a Reply

Your email address will not be published. Required fields are marked *