There is little doubt the SharePoint community is excited for SharePoint 2013. With 60% of users in a recent SharePoint survey saying they want to upgrade in the next year, the anticipation is building to a climax.
One feature that has garnered a lot of buzz, and some confusion, is the new Shredded Storage feature and what impact it will have on binary large objects (BLOBs) storage. In this two-part series, we’ll delve deeper into Shredded Storage, explore its impact on SharePoint 2013 and address the myths about the new feature and Remote BLOB Storage.
Before we plunge into the details it makes sense to define what Shredded Storage is and is not. The term “shredding” has created a lot of confusion. Shredded Storage does not refer to file shredding, which is the secure deletion of files by overwriting the data multiple times. StoragePoint and many other storage-related products include this feature. Instead, SharePoint 2013’s Shredded Storage is an attempt by Microsoft to reduce the I/O impact when saving versions of a document or file by “shredding” it into smaller pieces and reassembling it when someone needs to access it.
Microsoft developed shredded storage to address an issue that resulted from how document edits or versions are stored in SharePoint and to reduce the number of transaction logs. The result – significantly improved I/O (network and disk) and reduced CPU overhead when storing incremental changes to documents. The SharePoint product team has done a fantastic job addressing what has been a long-standing complaint in the way files are stored in previous versions of SharePoint.
Unfortunately, by addressing one issue, Microsoft introduced another that results in a performance decrease for the uploading and downloading of files in SharePoint.
A Brief History Lesson
With the introduction of SharePoint Team Services and subsequently SharePoint Portal Server 2003, Microsoft moved away from using the Web Storage System and settled on using SQL Server exclusively for the storage of BLOBs. BLOBs are immutable objects when stored in SharePoint. This means that BLOBs are created and deleted but never updated. When editing existing documents in SharePoint, edits result in new BLOBs being created. These new BLOBs are full copies including the edits and not incremental changes.
This means that if you maintain 10 versions of a 1MB document you would end up with approximately 10MB in total storage requirement excluding metadata. While this wasn’t the most efficient use of storage it was the simplest way to handle versions without introducing complexity. Unfortunately SharePoint has a tendency to spin off new versions of documents even if only metadata has changed. With the introduction of SharePoint and Office 2010 Microsoft optimized communication between the Office client and the SharePoint server by implementing a file synchronization protocol named Cobalt.
What Does Cobalt Mean?
I am not going to cover Cobalt in nauseating detail as the topic has been covered quite well by Bill Baer in his post about Shredded Storage. What you should know is that Cobalt allowed the Office client to send incremental changes rather than the entire file to the server when a document was saved. The incremental changes were then reassembled on the server and saved as a complete file. Shredded Storage in SharePoint 2013 extends the saving of incremental changes to the database where only file changes are stored rather than entire copies of the file.
The goal of Cobalt and subsequently Shredded Storage is to reduce the network bandwidth utilization (client changes sent to the SharePoint server) and the I/O operations (SharePoint Server sends file to database) that result from incremental changes to documents. In fact Shredded Storage significantly improves I/O operations while reducing CPU overhead when saving incremental changes to documents. What isn’t immediately apparent is the negative impact to I/O operations for the inserting of new files and downloading of existing files.
The Test Results
The table below compares SharePoint 2010 and SharePoint 2013 upload and download times on the same document set. Our lab testing confirms that SharePoint 2013 uploads and downloads are slower — and in some cases significantly slower — than SharePoint 2010. This is a direct result of Shredded Storage. The overhead involved in determining how to split a document into smaller pieces and store those smaller pieces definitely has an impact on the performance for uploads and downloads.
|Upload (speed in milliseconds)||Download (speed in milliseconds)|
|Scenarios||File Name||File Type||File Size (KB)||SP2010 (A)||SP2013 (B)||Difference (A-B)||SP2010 ( C)||SP 2013 (D)||Delta (C-D)|
|5||AE_1 MB Word Doc 2010||DOCX||1,082||0.35||0.66||-0.30||0.03||0.10||-0.07|
|7||AG_1 mb image||JPG||1,210||0.20||0.64||-0.43||0.03||0.07||-0.04|
|10||AJ_2mb TIF Image||TIF||2,579||0.32||1.01||-0.69||0.05||0.12||-0.07|
|13||AM_3_6mb PDF Sample||3,690||0.49||1.47||-0.98||0.07||0.21||-0.13|
|14||AN_4 MB PDF||4,078||0.50||1.60||-1.10||0.08||0.19||-0.11|
|15||AO_Corporate Presentation 2007||PPT||4,248||0.49||1.69||-1.21||0.08||0.20||-0.11|
|16||AP_Analyst Briefing – 2008||PPT||4,434||0.54||1.68||-1.15||0.08||0.18||-0.10|
|17||AQ_4_5 mb Video||MOV||4,627||0.46||1.77||-1.31||0.10||0.19||-0.09|
|18||AR_4_5 mb wmv video||WMV||4,680||0.51||1.81||-1.30||0.24||0.18||0.06|
|19||AS_Internet Safety Presentation||PPT||4,839||0.42||1.84||-1.42||0.24||0.20||0.05|
|24||AX_Corporate Slide Deck_April 2009||PPT||5,936||0.53||2.29||-1.76||0.18||0.27||-0.09|
|26||AZ_7mb Excel File||XLSX||7,415||0.68||0.89||-0.22||0.33||0.28||0.05|
|29||BC_11_7mb Excel File||XLSX||11,974||0.92||1.25||-0.33||0.73||0.32||0.41|
|30||BD_14_5 MB PDF||14,861||1.24||5.08||-3.85||0.73||1.10||-0.38|
|31||BE_26 MB XLSX||XLSX||26,557||2.24||2.79||-0.54||1.31||2.29||-0.97|
|33||BG_33_1 MB WORD 2010 Doc||DOCX||33,947||2.53||3.24||-0.71||0.62||3.02||-2.40|
|35||BI_55 MB XLSX||XLSX||56,356||4.18||4.90||-0.72||2.29||5.48||-3.19|
|36||BJ_70 MB WORD 2010 Doc||DOCX||71,694||5.39||5.99||-0.59||6.55||6.36||0.19|
|37||BK_100 MB XLSX||XLSX||103,108||8.85||8.16||0.69||5.83||7.86||-2.03|
|38||BL_103 MB WORD 2010 Doc||DOCX||105,411||7.78||7.91||-0.13||4.84||7.20||-2.36|
|40||BN_190 mb Word Doc 2003||DOC||195,899||14.65||12.01||2.63||14.30||8.22||6.08|
|41||BO_250 mb Movie||MOV||255,454||20.30||15.70||4.60||20.35||10.22||10.14|
|42||BP_382 mb Word Doc 2003||DOC||391,739||29.25||26.70||2.55||12.22||16.76||-4.55|
One Size Does Not Fit All
The test results above led us to examine the configuration options for Shredded Storage to determine if we could mitigate the negative impact on uploads and downloads. Unfortunately your options are limited. Contrary to other blog posts on the topic, Shredded Storage cannot be disabled. You actually had the option to disable shredding in the SharePoint 2013 beta but that option was eliminated in the RTM build. The only remaining option is changing the default shred or “chunk” size that files will split into when they are stored.
For me the decision to disable shredding is a bit nearsighted. Not all organizations use SharePoint for document collaboration where content is being updated/edited in large quantities. I would even argue that while some organizations do have collaboration sites where lots of editing occurs, they almost certainly have other sites where documents are simply uploaded and downloaded without edits or new versions being created.
A common example is document imaging where PDF/TIFF images are stored within SharePoint. Those images never change. Or, how about a document center that contains tens of thousands of published documents that are being read rather than updated? What’s more, Shredded Storage provides little value for these scenarios. It is true that even with versioning disabled the I/O between the client, SharePoint Server, and database Server will be optimized. However you will not reduce overall storage requirements.
Unfortunately you are relegated to living with Shredded Storage in hopes that Microsoft will provide, at a minimum the ability to disable the feature. An even better would be an option to control Shredded Storage at the site or site collection level for added flexibility.
Solving one problem by introducing another significant problem is going to make for some unhappy campers who are already struggling to keep up with the explosive growth of their SharePoint content.
In part 2, we will address using RBS with Shredded Storage, including debunking myths, reviewing how RBS functions with Shredding Storage, and discussing best practices for optimizing RBS.