How to download files from a SharePoint Online Document Library

As SharePoint Online captures more and more market share people are going to start doing more and more with it.

One of the core challenges that I’m noticing is the struggle for developers in transitioning from server side code to using the client side object model (CSOM). There are a lot of ways to accomplish and this is just one example in many. This example assumes that the source document library has a column called Status. It will download files from that document library where the Status is equal to New and after download all of these files it will then update the status of those files to Downloaded.

Reversing a string in Python with Performance Benchmarking

Recently I’ve been putting a lot of time into brushing up my Python knowledge for big data projects and operations. For a task that’s so simple I’ve found reversing a string can be quite enlightening for how to better “think in Python”.

As an example, here’s how I would have previously reversed a string before really digging into the Python language:

Quite less than ideal!

After digging a bit further I learnt about slice notation in Python – and it’s amazing. This StackOverflow question serves as a good primer, here’s the core answer from that question for future reference as well:

There is also the step value, which can be used with any of the above:

The key point to remember is that the :end value represents the first value that is not in the selected slice. So, the difference beween end and start is the number of elements selected (if step is 1, the default).

The other feature is that start or end may be a negative number, which means it counts from the end of the array instead of the beginning. So:

So what does this mean for reversing a string? Actually, quite a lot – that means we can now take our original statement above and re-write it simply as:

But how about performance? Python is commonly used on big data after all. Let’s introduce a couple of other ways to accomplish this task in Python and then benchmark them against each other:

Another Python specific approach for this task would be the reversed() function

We could also use a generator function, one of the more friendly ways of writing this as it reads as it performs – join all the i-th-s elements of the string where i goes from the len of the string to zero.

We could also use a deque object:

But what about performance? Using timeit let’s quickly benchmark these against each other:

Original function

In [250]: %timeit reverse_codingo(“reverse”)
100000 loops, best of 3: 3.61 µs per loop

Slice Operator

In [243]: %timeit “reverse”[::-1]
1000000 loops, best of 3: 310 ns per loop

Reverse Function

In [241]: %timeit reverse(“reverse”)
100000 loops, best of 3: 3.98 µs per loop

Deque Reverse Function

In [244]: %timeit reverse_deque(“reverse”)
100000 loops, best of 3: 2.61 µs per loop

The clear performance winner in speed appears to be the slice operator, also the one which is syntactically the most simple once you’re familiar with it. Definitely an operator worth learning more about and including in the toolkit!

InfoPath Forms stop working after applying cumulative update: ‘Requested registry access is not allowed’

After recently applying the July 2015 SharePoint 2013 Cumulative update we discovered that Infopath forms or pages referring them had stopped loading throwing a generic “something went wrong” error to the screen with a correlation id. After taking a look into the logs we found the following against the Infopath form services requests:

Requested registry access is not allowed

This appears to be a known issue within previous SharePoint cumulative updates (previously documented in the March cumulative update). There’s good writeups about this issue on Todd Klint’s always useful blog and on the accepted answer to this MSDN question. The problem appears to be the same in the July 2015 update – SharePoint updates the registry removing permissions for service accounts causing problems with Infopath and other areas (in our particular case BDS stopped working as well).

To resolve the issue you need to re-grant permissions for the user WSS_WPG to the registry group HKLM/Software/Office Server/15.0. After doing this power cycle the server and everything will function as expected again.

An error occurred. Detailed message: 1 conflict prevents checkout when doing a GIT pull request in Visual Studio

I’ve noticed something interesting happening in Visual Studio 15 when I move between environments and try to do a pull request.

I’m able to see the incoming commits queue as well as the content within them but when I go to do a pull request I’ll receive the error:

An error occurred. Detailed message: 1 conflict prevents checkout

Although this is usually quite an easy problem to resolve for some reason Visual Studio isn’t showing the diff tool and allowing me to resolve the conflicts.

Thankfully as we’re using git for our version control backend this is quite easy to resolve in the backend.

To overcome this first open a new command line window for the repository where the conflict is.

First navigate to the synchronization page which shows you incoming and outgoing commits. In the window that appears click Actions and then Open in Command Prompt.

Open GIT command prompt window

You then want to perform a git status command to confirm that you have outstanding changes to pull from the current branch.

GIT Status command prompt example

And then do a git pull to manually pulldown the latest version of the branch to your local repository. In order to do successfully perform a pull request you will also need to enter alternative access credentials. If you don’t already have these (or know them) then logging into your Visual Studio Online page, managing your profile (top right) and setting them on Security will create the credentials you need whenever using git from the console window.


A SharePoint Site Collection has disappeared or been deleted, how do I restore it?

So you come in of a morning and you have a whole host of e-mails from users complaining that content isn’t working for them. You visit the main page, no issues, but then you go to visit a site in another collection… and you receive a dreaded 404 error:

This error (HTTP 404 Not Found) means that Internet Explorer was able to connect to the website, but the page you wanted was not found. It’s possible that the webpage is temporarily unavailable. Alternatively, the website might have changed or removed the webpage.

Checking the recycle bin shows that the web is still active, and so it appears that there’s something wrong with the Site Collection.

Although not present in the front end thankfully there’s a cmdlet to allow you to quickly view deleted collections and then you can look to restoring them.

Which will allow you to view the deleted collections. Click here for documentation.

Once you have a list of your collections you can then restore them using

Here’s an example we had with a client –

We identified that the services site collection had gone rogue and couldn’t be found anywhere. We’ve then run Get-SPDeletedSite to see if by chance it had been deleted, which by chance it had and showed in the results:

Example of Get-SPDeletedSite

Thankfully this means that the solution is easy – using

Restored the site immediately (without any downtime or ISS recycling being necessary) and makes it available again to users:

Example of Restore-SPDeletedSite

The interesting problem is always how a collection was deleted in the first place. Interestingly enough in this particular case it looks like a user had received the automated e-mail asking them to confirm whether a SharePoint Web site is still in use, much like the following:

From: SharePoint
Sent: Wednesday, 8 July 2015 12:00 AM
To: Administrator
Subject: Confirm SharePoint Web site in use

Please follow the link below to your SharePoint Web site to confirm that it is still in use.

If the site is not being used, please go to http://sharepoint/services/_layouts/settings.aspx, and select “Delete This Site” to remove the Web site.

You will receive reminders of this until you confirm the site is in use, or delete it.

The user following this link and then confirming that they would like the web (in this case not a web but a collection) deleted the collection. A good lesson in always revisiting who is an administrator of what – in this case the user managed to delete a collection without  central admin!

SharePoint deployment scope, solution and features

SharePoint Features

SharePoint Features can be scoped to the Farm, Web Application, Site Collection, and Web Site level depending on the purpose of the feature. The Feature scope is determined by the setting of the Scope attribute in the Feature element defined in the feature.xml file.

A sample Feature element tag is given below:

<Feature Id="F62C96CF-79FD-44be-8882-E9BFBD199184">
Title="Feature Title"
Description="Feature Description"

Web Site scoped Feature (Scope=”Web”):

A Web Site scoped Feature is one that can be activated only at the individual Web site level. List templates, list instances, custom actions, event receivers, etc are the some common elements for web site scoped features. Web Site scoped features can be activated by using:

Run the following STSADM command:

Site Collection scoped Feature (Scope=”Site”):

A Site Collection scoped Feature is one that can be activated at the site collection level and contains items that apply to the site collection as a whole (for example, content types that are shared across the site collection), as well as items that can be activated per site (for example, list instances, etc). Site Collection scoped features can be activated by:

Run the following STSADM command:

Web Application scoped Feature (Scope=”WebApplication”):

A Web Application scoped Feature is one that can be activated at the web application level and contains items like administrative web application links, delegate control registrations, feature/site template association, document form registrations, etc. A farm Feature can contain, for example, links to /_layouts pages and files, /_admin pages, and other elements. Web Applicationscoped features can be activated by using:

Run the following STSADM command:

Farm scoped Feature (Scope=”Farm”):

A Farm scoped Feature can be activated at the server farm level. A farm Feature contains a number of elements that are critical for implementing applications and logic anywhere within a deployment. A farm Feature can contain, for example, links to /_layouts pages and files, /_admin pages, and other elements. Farm scoped features can be activated by using:

Run the following STSADM command:

SharePoint Solutions

SharePoint solutions are either deployed globally or targeted to a particular web application. The decision of which is made automatically by the SharePoint Solution framework depending on the contents of the solution manifest. Exception to this rule are Sandbox solutions which are managed on the site collection level.

Globally Deployed Solutions

When a solution is deployed globally, the assembly DLL file will go and sit under windows\assembly folder. All SharePoint application pools, including Central Administration’s, are recycled automatically. This can be good and bad. This is good because any GAC installed DLL that has been upgraded needs to be reloaded. This can be bad though with regards to the availability of your entire SharePoint Farm.

Web Application Targeted Solutions

When a web application targeted solution is deployed or retracted, the assembly DLL file will go and sit under the inetpub\websitename\bin folder. Only the application pools of the targeted web applications are recycled. When deploying and retracting a web application targeted solution, deploy or retract it only to those web applications that will use it … thus preventing unnecessary recycling of application pools.

Sandbox Solutions

Sandbox solutions are deployed to the site collection. This new mechanism has been introduced in SharePoint 2010 to provided more isolation between deployed components. Sandbox Solutions deployment does not require application pool recycle and does not allow deploying DLLs into the GAC, as everything is stored in content database where site collection resides. These solution also provide much more restrictive execution model and limited access to SharePoint API.

PowerShell Code to deploy solutions (WSP)

PowerShell to deploy the WSP to the GAC on server and deploy to all URLS

PowerShell to deploy the WSP to the GAC on server and deploy only to 1 specific web application (not in bin folder)

PowerShell to deploy to specific webapplication at bin folder on server

PowerShell to deploy to all the webapplication at their bin folders on server

Bitcoin Wallets: What are the differences between them?

Satoshi Nakamoto first published their paper on Bitcoin back in 2009, but it didn’t reach real mainstream popularity until recently. Only the users who understand the highly technical mechanics of Bitcoin invested their time in the technology during 2009-13 intermission period. In February of 2013 Bitcoin, and soon cryptocurrencies in general, finally entered the public consciousness. Despite being a four year old technology, many of the core features of the Bitcoin protocol were not polished in any capacity, let alone ready for mass public consumption. The Bitcoin wallet is an excellent example of one such component. Since that first popularity explosion, much expertise has gone into designing and refining the Bitcoin wallet. Today we will be taking a look at the differences in functionality of the three types of wallets as well as comparing their security and usefulness. These types of wallets suit users depending on the way people wish to access their funds, as well as who they want to entrust the long-term security of their accounts.

What is a Bitcoin wallet?

For a majority of Bitcoin users, the first cryptocoin interface they encounter is the wallet. Much like the physical wallet you keep in your back pocket, your Bitcoin wallet serves to store currency that you mine, purchase, or receive from others. Your digital wallet functions in a very similar way to its real life counterpart – however,there is one key difference: Cryptocurrencies are exchanged using addresses not unlike email, with the exception that your Bitcoin addresses are generated randomly on your behalf. Your wallet can hold more Bitcoin addresses then you could ever use, and each address functions as a separate place for you to send and receive money. Think of each address as a separate compartment in your wallet, or as the various different bank cards you own that each have their own individual balances. These addresses are where all transactions are directed, whether they’re yours, or run by the companies you use to collectively hold your money.

Offline Wallets

In the beginning of Bitcoin, there were offline wallets. These wallets operate in the “traditional” wallet style – that is, with all of your wallet data stored only locally on your own personal machine and not in need of a constant internet connection. You do need to be on the internet to receive and send your currency, but you don’t always have to be online. Initial Bitcoin commerce wasn’t possible without at least a basic wallet, and the offline wallet was the simplest solution for early adopters to code up and release to the public, including the one Nakamoto published. As such, offline wallet apps are readily available on both the desktop and mobile platforms. It should also be noted that any damage to your wallet data can quickly render it corrupted, which means your keys (and your cash) are unrecoverable without a backup. This limitation drives fear into the heart of many Bitcoin enthusiasts, some of which can barely keep track of their physical keys, let alone their digital ones. As a result, it’s important to keep many copies of your wallet information secure and safely stored away, even if just on a USB drive behind a safe.

Online Wallets

When hosted, or online, wallets entered the scene, they changed the entire Bitcoin wallet paradigm. Instead of keeping all your wallet information stored locally, a sensible online wallet stores an encrypted copy of your wallet data on their servers with a password you choose. When you want to access your wallet, your data is downloaded and decrypted locally. After you have completed your transactions, the hosted wallet uploads your encrypted wallet to back servers the hosted wallet controls. This relieves the user of the responsibility of backing up their wallet as well as “syncing” their wallet across their devices — all while keeping the users keys protected from abuse on the company’s servers by only doing encryption and decryption operations on the “client side”, aka locally on your machine. Online wallets are most popular on web browser centric and mobile devices, but the nature of this type of wallet makes easy access possible on any networked system. IT also ensure you don’t have to run your computer all day as a node to keep up with transaction activity on the Bitcoin network, and can let your provider handle all the trouble. However, if anything were to happen to the company you use to store your currency, then you’d be out of luck unless they offer a backup service to store copies of your wallet offline.

Deterministic Wallets

A third type of wallet exists exclusive of the other two: The deterministic wallet. The previous two types of wallet operate as variations of each other, where the online and offline wallets manage the same wallet data that is located either on your machine or is shifted to someone else’s. On the other hand, the deterministic wallet generates master root key (consumable by humans in the form of a twelve word passphrase) and hands it to the user. The root key is the only information the user is required to keep track of. Once the root key in input, the deterministic wallet uses advanced cryptographic algorithms to derive new private keys, or addresses, from the original root key. Because the wallet is creating new addresses using the same formula with the same initial conditions each time, all addresses are said to be predetermined from the root key. This handy fact allows you the option to store your wallet backup securely in your brain (via memorization) without the need for the key to exist digitally anywhere — all while still being able to access funds in any of the derived addresses. The portability of this system grants deterministic wallets secure interfaces on your desktop, your mobile phone, and your web browser. This of course still depends on you remembering  a very unique kind of password that may be difficult to recall, or if written down can pose a security threat.

Which should you choose?

Since the invention of the Bitcoin protocol users and developers alike have worked to shape the function and feel of cryptocoin wallets. Today wallets are more robust, packed full of features, and offer security that we couldn’t imagine during the infancy of Bitcoin. Wallets come in many forms and serve many purposes. We’ll share with you next how most popular wallets available compare, and what you can do to pick the one that suits you.