my personal blog about systemcenter

All posts in Disaster Recovery

Protecting your secrets, one more step to remember

Categories: Active Directory, AD, Backup, Disaster Recovery, Password, TSM
Comments Off on Protecting your secrets, one more step to remember

If you are using hosted backup with TSM there is one more step to cover when people leave the org

The protection for many hosted backups are

Protection against “rouge” TSM Administrator

Client Side Encryption

Protection against “rouge” Backup Administrator

Node ID

Node Password (separation of duties one for password one for encryption)

And the last one is the issue here as its often not rotated, default TSM is 90 days but looking at different hosted TSM password is often set to no expire

This is not a TSM problem but a problem with password rotation

In the perfect world, the NodeID password and the encryption is not known by the same person, but then nodeid / password / secret is in registry so an AD admin can access this

Scenario

TSM BA Client installed on demodc01.stackdemo.dk

clip_image002

Starting the TSM client , prompting for Node Password on first backup

clip_image004

Ready for Action

clip_image006

Starting the first backup , prompts for encryption key , and after a short while the backup is completed

clip_image008

On a rouge server, outside of the environment we install the TSM BA Client and reuse the nodeID and password from the disgruntled backup admin

clip_image010

Adding the nodeid and nodepassword

clip_image012

And we restore a dummy file to see that’s its working, and is prompted for the encryption key

dsmc q b “{DEMODC01\SystemState\NULL\System State\SystemState}\ntds.dit” -sub=y

clip_image014

If we can’t remember where ntds.dit is located we can search for it

rest “{DEMODC01\SystemState\NULL\System State\SystemState}\\DEMODC01\C$|\WINDOWS\ntds\*” C:\EVILDC\ -sub=y

clip_image016

And we can restore the files

clip_image018

And we now have something we can attack , if we boot up in a winPE enviroment we can follow the procedure for system state and have a working domain controller

clip_image019

If the attacker had access to the domain controller aka disgruntled former employee the password and encryption is available on the source node in registry , since TSM used both the password and the encryption to access TSM server and backup/restore data it needs to be stored somewhere that the service can access

It’s very hard to protect anything from a domain admin even with the assume breach state of mind

clip_image021

So, we can logon without getting prompted for credentials/encryption

So what can we do

First off , prevent people from being disgruntled

And since we can’t control human nature change the password on the nodes, either scheduled or when high privilege staff leaves or both, and again the default for a TSM node is that it will be changed

clip_image023

Single Node example, log on the TSM , change password

clip_image025

Something old Something New

clip_image026

And Success , and password change can be scripted so cycling the password shouldn’t be a big issue

clip_image028

And our EvilDC can’t access TSM anymore and everything is back to normal

Deploying Data Protection Manager in a dedicated domain

Categories: Active Directory, Data Protection Manager, Disaster Recovery, DPM, Hyper-V
Comments Off on Deploying Data Protection Manager in a dedicated domain

Data Protection and the ability recover data is key to keeping your job and your company alive.

The demo setup thats is going to be used in this post are

  • PROTECTDC01 Domain Controller in the PROTECT Forest
  • PROTECTDC02 Domain Controller in the PROTECT Forest
  • PROTECTDDPM01 Data Protection Manager Server in the PROTECT Forest
  • FABRICDC01 Domain Controller in the FABRIC Forest
  • FABRICDC02 Domain Controller in the FABRIC Forest
  • FABRICHV01-04 Hyper-V HyperConverged Instal
  • FABRICHVC01 Hyper-V Cluster with member FABRICHV01-04
  • WORKLOAD01-05 Virtual Workload in the FABRIC Hyper-V Cluster

As this is a test enviroment everything are stuck on one box.

For the real world deployment the FABRIC and PROTECT domain must be seperated , the whole point in this post will be if you for some reason get compromised in your FABRIC domain , you will still have access to the PROTECT domain and maintain the ability to recover your data.

This also means that in a larger enviroment you can easier seperate the roles so one team wont have access to both source and target of backup data

We do in the example log in interative on the fabric domain , so if the host is compromised before agent install the protect domain is going down the same path , so there is still some work to be done but beats having everything in one domain.

image

On the PROTECT domain setup DNS forwarders to the FABRIC domain

image

And in Reverse to get name resolution up and running up between the two forests

image

Setting up the trust

image

Setting up the trust

image

for this test forest-wide is used , tighter security can be used with selective authentication

image

On the 4 Hyper-V Hosts we add the DPM account from the protect domain


image

We then add the DPM agent to all Hyper-V hosts and run the

SetDPMServer –dpmservername protectdpm01.protect.azurestack.coffee  , this connects the Hyper-V host to the remote DPM server

image

On the data protection manager , we use Attach Agents

image

And we add the 4 Hyper-V hosts manually

image

And we now have all 4 servers

image

use credentials in the fabric domain or the dpm account to attach the agent


image

Sucess

image

Create a protection group browse to the VM’s and add them

And we can now backup from a dedicated domain from the Fabric domain



DPM Virtual Machine Recovery Hyper-V Clustering

Categories: Broken, Data Protection Manager, Disaster Recovery, DPM, Hyper-V, Virtual Machine Manager, VMM
Comments Off on DPM Virtual Machine Recovery Hyper-V Clustering

One task needed for planning for disaster recovery with DPM is the naming of the VM from a clustering point of view.

If all VM’s are created through VMM then will be prefixed with SCVMM but in this example we have created one VM from Cluster Manager and one from VMM

The scenario for this type of recovery is a “dead” cluster or VM’s that was deleted by mistake.

Being able to recovery directly would be preferred but thats not currently a option

image

This leaves us with VMMTESTB prefixed from a naming point of view with SCMMM and one without when selecting the VM’s from DPM

image

And the view from Cluster Manager

image

And after creating the protection group we are ready to test a recovery

image

And our view from VMM

image

If we need to recover the VM to the orignal source we can just select recover

image

And we can recover to the original instance

image

And to provoke a error

image

We kill the VM from both cluster and hyper-v

image

If we then tries to recover a VM to its Original Instance

image

The recovery dies as we dont have the origianal VM

image

So we need to create a empty VM

image

Point it to a dummy location ,the name needs to match the original VM name

image

And no VHD attached

image

In VMM we can not see the new VMTESTA and the VMTESTA that we delted , this scenario is if the cluster dies and needs to be rebuild but VMM isnt “gone”

image

We can then restore the VM directly

image

We then have the temp clustered VM , we need to delete that

image

And also on the Hyper-V Host , this can also be done from VMM to avoid having to open the settings to find the restore VM

image

And Configure Role

image

And select Virtual Machine

image

And Enable it for HA

image

And last clean up if the VM wasnt deleted from VMM , remove the VM that have missing state

A

Had a question today about the ability to recover the SQL master database if the SQL server wasnt operational.

To set the scene for the converation we did the following

Image

Installed the Data Protection Manager on the SQL 2012 server , created a brand new fresh and shiny protection group , added the SQL instance

Image

Created short term recovery this was after all just a test

Image

And a backup every hour kept for the last 5 days

 

Image

And as always with Data Protection Manager Sucess is an option :)

Image

And a few minutes later we now have a full backup of the SQL server

Image

And lets try to add the human factor

Image

Twice

Image

And after that we have a non functional SQL Server

 

Image

And as we can see the reason if lack of master databases

 

Image

DPM to the rescue , enter recovery select the master database

Image

Option One , Recover to original instance (otherwise restore to folder and copy the files manually)

 

Image

And leave database operational as we are not recovering from additional logs

 

Image

Verify server/files

Image

And 35 seconds later the database is recovered without incident

 

Image

Recovery is complete , SQL instance is restarted and we can fire the blogger that deleted the master database

 

Marcel van der Berg posted about the new Hyper-V recovery Manager

http://up2v.nl/2013/04/08/disaster-recovery-of-on-premise-hyper-v-virtual-machines-using-windows-azure-recovery-services/

 

Read his brilliant post and then

If you would like to be considered for this program please complete the Microsoft survey located here. Thank you in advance for your responses! We will only be contacting those of you that have been accepted to participate.

Go Click NOW Smiley