I recently came across a task in which I had to download all the files in lists from sharepoint farm, targeting one single web application but it had close to 16 site collections and hundreds of sub sites in them.
Solution:
To be exact we were doing Source code migration from TFS 2013 to Azure DevOps Server. The main problem is the project portal which is built in SharePoint and SharePoint portal is deprecated in Azure DevOps Server, and is replaced by Wiki Pages. How do we do that, go no further I have pre-built solution for you which even address the permissions on site collection along the way.
Before you Start:
- Make sure you have farm admin permissions
- Make sure you have site collection administrator permissions for all related site collections ( Even though you don’t have, I have automated this step as well, you just need to un-comment below line and put your account in global variables, which are at the very end of the actual script.
#Set-SPSite -Identity $Site.Url -SecondaryOwnerAlias $Global:SecondaryOwnerAliasAccount
3. Enough Disk Space to download all the files, belief me I have seen large videos checked in to Project Portal.
4. Time, it may take considerable time to download all the files, depending on the size and volume of the files and number of site collections and sub sites.
5. Use List filter to adjust what you want to download, I went with Document Libraries and skipped all other lists which are either system or not intended to download.
6. Run the script in powershell on the SharePoint Application Server with elevated privileges
Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue
#Function to Download All Files from a SharePoint Folder
Function Download-SPFolder($SPFolderURL, $LocalFolderPath, $currentWeb)
{
Try {
#Get the Source SharePoint Folder
$SPFolder = $currentWeb.GetFolder($SPFolderURL)
$LocalFolderPath = Join-Path $LocalFolderPath $SPFolder.Name
#Ensure the destination local folder exists!
If (!(Test-Path -path $LocalFolderPath))
{
#If it doesn't exists, Create
$LocalFolder = New-Item $LocalFolderPath -type directory
}
#Loop through each file in the folder and download it to Destination
ForEach ($File in $SPFolder.Files)
{
#Download the file
$Data = $File.OpenBinary()
$FilePath= Join-Path $LocalFolderPath $File.Name
[System.IO.File]::WriteAllBytes($FilePath, $data)
Write-host -f Green "`tDownloaded File:"$File.ServerRelativeURL
}
#Process the Sub Folders & Recursively call the function
ForEach ($SubFolder in $SPFolder.SubFolders)
{
If($SubFolder.Name -ne "Forms") #Leave "Forms" Folder
{
#Call the function Recursively
Download-SPFolder $SubFolder $LocalFolderPath
}
}
}
Catch {
Write-host -f Red "Error Downloading Document Library:" $_.Exception.Message
}
}
#Main Function
Function Export-SPLibraries()
{
Try {
#Get the Web Application
$WebApp = Get-SPWebApplication $WebAppURL
#Get all Site collections from the web application
$SitesCollection = $WebApp.Sites
#Enumerate all site collections in the web application
Foreach($Site in $SitesCollection)
{
Try {
#Get Site Collection URL, Owner, Content Database Details
Write-host -f Cyan "Working : " $Site.URL
#Get the Source Web
$global:Web = Get-SPWeb $Site.Url
#Set site Owner
#Set-SPSite -Identity $Site.Url -SecondaryOwnerAlias $Global:SecondaryOwnerAliasAccount
$LocalFolderPathWithCurrentSite = $Global:DownloadToLocalFolderPath+$Site.RootWeb.Title
Write-host -f White "About to emplty LocalFOlderPath : " $LocalFolderPathWithCurrentSite
#Delete any existing files and folders in the download location
#If (Test-Path $LocalFolderPathWithCurrentSite) {Get-ChildItem -Path $LocalFolderPathWithCurrentSite -Recurse| ForEach-object {Remove-item -Recurse -path $_.FullName }}
#Iterate through each website insde rootwebsite
ForEach($Web in $global:Web.Site.AllWebs)
{
Try {
Write-host -f Yellow "Wokring on: " $Web
#Array to Skip System Libraries
$SystemLibraries =@("Pages", "Converted Forms", "Master Page Gallery", "Customized Reports",
"Form Templates", "List Template Gallery", "Theme Gallery", "Reporting Templates",
"Site Collection Documents", "Site Collection Images", "Site Pages", "Solution Gallery",
"Style Library", "Web Part Gallery","Site Assets", "wfpub")
#Get all document libraries - Exclude Hidden and System Libraries
#$LibraryCollection = $Web.lists | Where-Object { ($_.BaseType -eq "DocumentLibrary") -and ($_.hidden -eq $false) -and ($SystemLibraries -notcontains $_.Title)}
$LibraryCollection = $Web.lists | Where-Object { ($_.BaseType -eq "DocumentLibrary") -and ($_.hidden -eq $false) -and ($SystemLibraries -notcontains $_.Title)}
#Iterate through each list and export
ForEach($Library in $LibraryCollection)
{
try{
Write-host -f magenta "Downloading Document Library:" $Library.Title
#Call the function to download the document library
Download-SPFolder -SPFolderURL $Library.RootFolder.Url -LocalFolderPath $LocalFolderPathWithCurrentSite\$Web.Name -currentWeb $Web
}
catch
{
Write-host -f Red "Error Downloading the files:" $_.Exception.Message
}
}
}
catch{
Write-host -f Red "Error web accessing:" $_.Exception.Message
}
}
}
catch{
Write-host -f Red "Error Site Collection accessing:" $_.Exception.Message
}
}
Write-host "*** Download Completed ***"
}
Catch {
Write-host -f Red "Error Downloading Document Library:" $_.Exception.Message
}
}
#region Runtime-Variables
$Global:WebApplication = "http://WebApplicationURL"
$Global:DownloadToLocalFolderPath = "C:\DownloadedSharepointDocs\"
$Global:SecondaryOwnerAliasAccount = "domain\FarmAdminUser"
#endregion Runtime-Variables
#Call the Function to download all document libraries from a site
Export-SPLibraries
Good luck!