Sunday, 19 February 2017

Data replication using PowerShell Script

Data replication using PowerShell Script + Create log files for your scripts

The below article is focused on data replication from one source to destination - Not a folder synchronization script. It's always been a question to me how to replicate the content of one folder to another for a various purpose. This article also included logging option in Powershell, So we can follow, and see what is happening while the script is executing and copying the data or if the script runs into errors is very useful. Please follow the below article to know how can we achieve this using PowerShell.

Below cmdlets would be the input for the script. We need to define the source location, destination location and log file path for the script.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
###################################
# Source Location Details
#-------------------
$Source = "\\Server2012\ContentFolder"

# Destination Details
#-------------------
$Destination = "D:\ContentFolder"

# Log File Location
#-------------------
$Folderpath = "D:\FolderSync"

We have to declare few variable such as Log file name, error handling variables etc.

1
2
3
4
5
6
7
8
9
####################################
# Declaration of Variables
$Date1 = Get-Date -UFormat "%m-%d-%y"
$time = Get-Date -Format g
$Logname = "SyncLog.log"
$FilePath = "$Folderpath"+ "\$Logname"
$Errormap = $null
$errorcheck  = ''
$error.Clear()

The next part will be Create\Modify the Log file based on the availability of log and based on the size of the log - If the log file size is more than 1 MB (We can modify the size) log file will be replaced with a new log file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
###################################

# Setting Log File

    If ((Test-Path -Path $Folderpath))
    {
            if ((Test-Path -Path $FilePath))
            {
 
                $filesize = Get-ChildItem $FilePath ; if ($filesize.Length -gt 100kb ) {Remove-Item $filesize.FullName -Force; New-Item $logfolder\$Logname -ItemType file -Force} 
                
                else {}
             }
            else
            {
                New-Item $FilePath -ItemType file -Force
            }
    }
    else
    {

        New-Item $Folderpath -Type directory -Force
        New-Item $FilePath -ItemType file -Force

    }

Checking the source location is accessible or not from the destination location - The script execution will exit in case if the source not accessible.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Connection to Source

If ((Test-Path -Path $Source))
    {
        Add-Content -Path $FilePath -Value "$time Connection to $Source established Successfully`n"  
    }
else
    {
        Add-Content -Path $FilePath -Value "$time Error : Connection to $Source Failed - Script execution stopped`n"
        exit
    }

The System.Security.Cryptography namespace provides cryptographic services, including secure encoding and decoding of data, as well as many other operations, such as hashing, random number generation, and message authentication. For more information. Refer the below link for more details.

https://msdn.microsoft.com/en-us/library/system.security.cryptography(v=vs.110).aspx

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
##################################
function Get-FileMD5 {
    $error.Clear()
    Param([string]$file)
    $mode = [System.IO.FileMode]("open")
    $access = [System.IO.FileAccess]("Read")
    $md5 = New-Object System.Security.Cryptography.MD5CryptoServiceProvider
    $fs = New-Object System.IO.FileStream($file,$mode,$access)
    $Hash = $md5.ComputeHash($fs)
    $fs.Close()
    [string]$Hash = $Hash
    Return $Hash
}
Add-Content -Path $FilePath -Value "$time Data Replication Started`n"

Below part is replicating data and writing the respective logs into log file


 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
##################################

function Copy-LatestFile{
    $error.Clear()
    Param($File1,$File2,[switch]$whatif)
    $File1Date = get-Item $File1 | foreach-Object{$_.LastWriteTimeUTC}
    $File2Date = get-Item $File2 | foreach-Object{$_.LastWriteTimeUTC}
    if($File1Date -gt $File2Date)
    {
        #Write-Host "$File1 is Newer... Copying..." -ForegroundColor Green
        Add-Content -Path $FilePath -Value "$time $File1 is Newer. Copying...`n"
        if($whatif){Copy-Item -path $File1 -dest $File2 -force -whatif}
        else{Copy-Item -path $File1 -dest $File2 -force}
        $errormap = $error[0]
        if($error[0] -eq $null) {} else {Add-Content -Path $FilePath -Value "$time Error $errormap"; $errorcheck = "1"}
    }

}

# Getting Files/Folders from Source and Destination
$SrcEntries = Get-ChildItem $Source -Recurse
$DesEntries = Get-ChildItem $Destination -Recurse

# Parsing the folders and Files from Collections
$Srcfolders = $SrcEntries | Where-Object{$_.PSIsContainer}
$SrcFiles = $SrcEntries | Where-Object{!$_.PSIsContainer}
$Desfolders = $DesEntries | Where-Object{$_.PSIsContainer}
$DesFiles = $DesEntries | Where-Object{!$_.PSIsContainer}

foreach($folder in $Srcfolders)
{
    $error.Clear()
    $SrcFolderPath = $source -replace "\\","\\" -replace "\:","\:"
    $DesFolder = $folder.Fullname -replace $SrcFolderPath,$Destination
    if(!(test-path $DesFolder))
    {
        Add-Content -Path $FilePath -Value "$time Folder $DesFolder Missing. Creating it!`n"
        new-Item $DesFolder -type Directory | out-Null
        $errormap = $error[0]
        if($error[0] -eq $null) {} else {Add-Content -Path $FilePath -Value "$time Error  $errormap"; $errorcheck = "1"}
    }
}

foreach($entry in $SrcFiles)
{
    $error.Clear()
    $SrcFullname = $entry.fullname
    $SrcName = $entry.Name
    $SrcFilePath = $Source -replace "\\","\\" -replace "\:","\:"
    $DesFile = $SrcFullname -replace $SrcFilePath,$Destination
    if(test-Path $Desfile)
    {
        $SrcMD5 = Get-FileMD5 $SrcFullname
        $DesMD5 = Get-FileMD5 $DesFile
        If(Compare-Object $srcMD5 $desMD5)
        {

            Add-Content -Path $FilePath -Value "$time The Files MD5's are Different... Checking Write`n"
            Copy-LatestFile $SrcFullname $DesFile
            $errormap = $error[0]
            if($error[0] -eq $null) {} else {Add-Content -Path $FilePath -Value "$time Error  $errormap"; $errorcheck = "1"}
        }
    }
    else
    {
        $error.Clear()
        Add-Content -Path $FilePath -Value "$time $Desfile Missing... Copying from $SrcFullname`n"
        copy-Item -path $SrcFullName -dest $DesFile -force
        $errormap = $error[0]
        if($errormap -eq $null) {} else {Add-Content -Path $FilePath -Value "$time Error  $errormap"; $errorcheck = "1"}
    }
}

# Checking for Files that are in the Destinatino, but not in Source
foreach($entry in $DesFiles)
{
    $error.Clear()
    $DesFullname = $entry.fullname
    $DesName = $entry.Name
    $DesFilePath = $Destination -replace "\\","\\" -replace "\:","\:"
    $SrcFile = $DesFullname -replace $DesFilePath,$Source
    if(!(Test-Path $SrcFile))
    {

        Add-Content -Path $FilePath -Value "$time $SrcFile is deleted from Source... Deleting it from Destination`n"
        Remove-Item -path $DesFullname -Recurse -force
        $errormap = $error[0]
        if($error[0] -eq $null) {} else {Add-Content -Path $FilePath -Value "$time Error  $errormap"; $errorcheck = "1"}
    }
}
$error.Clear()
$Source = Get-ChildItem $Destination -Recurse
$dirs = $Source | Where { (gci $_.fullName).count -eq 0 } | select -expandproperty FullName
$dirs | Foreach-Object { Remove-Item $_ }
if ($errorcheck -ne '1') { Add-Content -Path $FilePath -Value "$time Data Replication completed successfully`n"}
else {Add-Content -Path $FilePath -Value "$time Data Replication completed with some error - Please check the log for details`n"}

###################################

Combine all the script together and run.

Click here to download the script from Technet

Posted by