How to transfer large files using Windows File Sharing

When copying large files (e.g. over 10Mb) that are stored in a Windows file share across an unreliable link there is no mechanism with a standard copy and paste that will deal with failures.  If a transfer fails you will need to start all over again.

Robocopy, a tool built into Windows 7 or downloadable for earlier version of Windows as part of the Windows Server 2003 Resource Kit Tools, has a flag (/z) which will cause it to automatically retry downloading a file that failed.

This will help for downloading a lot of files from a single location, but not particularly with large files.

However, you can use 7-Zip to create a compressed file that is split into multiple files of a specified size.  This can be used to split a single file into multiple chunks that you can then have robocopy download and retry any failures automatically.

For example, to download all the files in a share \serverSplitLargeFile to the current working directory use the following command:
robocopy \serverSplitLargeFile . /x
This is the closest thing I have found to the resume function in ftp which will pick up close to where you left off for any failed downloads.

A further useful step is to create an MD5 sum of the source file which can then be used to verify the downloaded file, after it has been recombined and unzipped by 7-Zip.  My preferred tool for this is winmd5sum, where you can create or compare MD5 sums using a right click.  You’ll need this installed at the source computer (so you can create the MD5 sum) and the destination (so you can compare it).  I always create a text file containing the MD5 sum at the location where I store the source.

How to transfer large files using Windows File Sharing

Word 2010 Template with Auto-updating fields

It is nice to have consistent looking documents, and Word templates can help you achieve this. For my team, I’d like all our documents to include our company logo and department name at the top of the page. I’d also like the footer to include details about the document:

  • Who last edited the document
  • When the document was last edited
  • Where the document is currently saved – this is useful when you find a useful printed document, and want to print more copies or make an update.

Word includes a feature where you can add fields to a document, including all the above. These fields can be added from the Quick Parts > Fields item from the Insert tab. They are represented by the following field codes:

  • {LASTSAVEDBY}
  • {SAVEDATE}
  • {FILENAME -p} – the -p includes the full path.

So I created a few templates that included these in the footer.

However, Microsoft disabled automatic updating of these fields since Word 2002. According to the MS knowledge base article on this, this was to improve performance. Great, but now the functionality as most people would expect is broken.

So any template that you create which includes any fields will only update those fields if you select them and hit F9 on your keyboard. Who does that? So if you use these fields in a template, they will only ever show the defaults, or at best the details about the template, not the files created from it.

Luckily, Microsoft give a workaround in the knowledge base article in the form of a macro that will automatically run when the document is opened, updating all the fields. This works pretty well, with one small problem.

As soon as the macro runs (when the document is opened), the document is marked as unsaved. So when the user closes the document they will be asked if they want to save their changes. No-one thinks about this prompt do they? They always click Yes, because the last time they didn’t they lost hours of work.

So when the person who was just reading the document clicks Yes, the document is saved. The next time the document is opened the macro runs and updates the fields. They now show who last read the document, and when they closed it.  That isn’t much use.

Happily, there is a document property that implies whether the document has been edited or not.  ActiveDocument.saved.  When this property is true, it means that the document has been saved since the last edit.  When it is false, it means it has not been saved since the last edit.  So by setting ActiveDocument.saved to true after running the field update script, Word no longer sees the need to pop up the save dialog on closing the document.  However, if after the macro has run, the viewer makes a change to the document, ActiveDocument.saved gets set to false and the prompt to save comes back.

I’m pretty pleased with this, so I present to you the updated macro:

Sub AutoOpen()
 '
 ' AutoOpen Macro - runs on open updating 
 ' all the fields in the document.
 '
    Dim aStory As Range
    Dim aField As Field

    ' Check that document is not in Protected View before
    ' doing anything
    If Application.ActiveProtectedViewWindow Is Nothing Then

        For Each aStory In ActiveDocument.StoryRanges
            For Each aField In aStory.Fields
                aField.Update
            Next aField
        Next aStory

        ' set document as unchanged (prevents save
        ' prompt on close) further changes will set
        ' this back to false restoring the prompt
        ActiveDocument.Saved = True

    End If
End Sub

Add this to your template, and now any fields in documents created from this template will be updated when the file opens.

Why not do this when the document is saved

Initially I thought all the fields should be updated on save (as have others). However, the save function would have to run after the document was saved in order to pick up the values of who saved, when and where, and then you would need to save again to update those fields. It makes more sense to update on open, and this will also pick up when the file has been moved.

Thanks to jJack on Stack Overflow for pointing me in the direction of ActiveDocument.saved.

Update

I found that in Word 2010, when the above code was added to Normal.dotm, opening a document in Protected View (e.g. documents downloaded from the Internet or in Outlook attachments) triggered VBscript error 4248: This command is not available because no document is open.

This was happening as the ActiveDocument object wasn’t accessible for protected documents – and it seems that the AutoOpen macro in Normal.dotm runs when any document is opened.  Microsoft have a blog entry on detecting Protected View in macros, which helped me update the above code to not run when a document is opened in Protected Mode.  If you aren’t adding the above macro to Normal.dotm, you don’t need the If statement around the rest of the macro, but it doesn’t hurt either.

Word 2010 Template with Auto-updating fields

Montessori Mashup

From the list of Montessori Schools registered with the Montessori Schools Association and listed on the Montessori St Nicholas Charity website, with a little text processing, and the help of Google Fusion Tables, I created the below map of Montessori Schools.

https://www.google.com/fusiontables/embedviz?q=select+col4+from+1niWxiDXEY0ceIRgJwo7_zCTVTHffdVFvt3BXTCM&viz=MAP&h=false&lat=54.66264091468062&lng=-362.90120251875&t=1&z=6&l=col4

Each node on the map is clickable, and provides a link to more details from the above site, as well as whether the school has acheived any accreditation from MSA.

Montessori Mashup

VSAT Gotchas

In some locations, the only option for an internet connection is to use some form of satellite link. Like any wireless medium these can be quite tricky to troubleshoot. Here are some factors I have experienced or learnt about that can cause problems for a VSAT connection.

Obstruction to the line of sight to the satellite

Obstructive Mango Tree in Makeni, Sierra Leone
Obstructive Mango Tree in Makeni, Sierra Leone

Anything that gets in between the VSAT dish and the satellite can weaken or obliterate the signal. Obviously when you initially set up a VSAT dish you find the clearest possible view of the sky, but environments have a tendancy to change. Buildings are erected, or extra floors are built. Trees grow taller or grow extra foliage. I’ve experienced this with mango trees in a couple of locations. Mango trees seem to be common in locations where VSAT is the only connection option!

RF interferance

VSAT operates in several frequency ranges, the most common being:

Different things can create interference in each of these bands. Notably, WiMax operates in the 2 – 11 GHz, which overlaps C Band. C Band installations within 50 miles of a WiMax network can potentially be effected. Nice one IEEE! I have also observed local welding work to completely obliterate a Ku band signal. When the work was completed the connection came back.

Rain Fade

Ku band is notorious for being affected by water droplets in the atmosphere – commonly referred to asrain fade – although this can affect any frequency above 11 GHz. This is now usually mitigated against by changes to the transmission power at the satellite and earth stations, but can still sometimes cause problems. Build up of moisture in the feed horn of a VSAT dish can degrade the signal enough to break the connection – if the moisture is removed, or once it evaporates, the connection is reestablished.

Physical disruption

Pointing a dish at a small object 35,786 km away is a fine art. Moving the dish by just 1 degree results in pointing at something more than 600 km away from the satellite! Small knocks to the dish can result in loss of signal, but the dish still looks like it is pointing in the right direction. That is why you spend a lot of money and effort on building secure mounts for your dishes, and put up signs suggesting that getting near the dish is dangerous. Leaving dead birds lying around the dish can be helpful! Talking of birds, they have been known to build nests in VSAT dishes – they generally come with some spikes to discourage this. I’ve seen a 1.5m Maribou stork perching on a VSAT dish. Cabling can also get damaged (and is expensive and awkward to replace). Again, investment in proper cable runs is worthwhile. Wind blowing a loose cable against a wall, especially over an edge, can result in damage to the cable.

Lightning strikes

Like any metal outdoor object pointed at the sky, lightning strikes are a risk. Generally you won’t be supplied VSAT equipment without some form of lightning protection, but adequate earthing/grounding is usually left up to you. Are lightning rods a good idea? I’m not sure, and haven’t been involved in any installations that include them.

VSAT Gotchas

Can you work with the Chaos Monkey?

At the Nethope Conference, one of the better plenary sessions was by Joe Baguley of VMware.  One of the things he mentioned in his talk that resonated for me was something that Netflix had developed called the Chaos Monkey.

The Chaos Monkey is a programme that Netflix run on their systems that randomly shuts down processes and services.  The idea is that the world is a chaotic place, and at some point one of your processes or services will shut down.  The chaos monkey simulates this, forcing everyone to design systems that can handle this or that part failing.

This seems to be a particularly important concept to grasp, particularly when building on platforms that market themselves as extremely resiliant.

At Christian Aid, I don’t think we need to build our own chaos monkeys. In our international environment, we are frequently interrupted by chaotic events, from giant signs falling on VSAT dishes (Abuja, 2008) to seemingly random VPN outages caused by ISP config errors (Port Au Prince, Dhaka, Delhi, La Paz, all to often recently).  Whilst these are a proper pain in the derrier, we must learn from them, and take this learning to build more resilant infrastructure, but also organisational processes that can handle everything from Earthquakes to SAN failure taking out our email system

The Chaos Monkey teaches us to expect the unexpected.

Read more:

Can you work with the Chaos Monkey?

Grameen’s Community Knowledge Worker programme – an I4D case study

Grameen Community Knowledge WorkersAt this years Nethope conference, which I attended last week, the stand out for me was a presentation of a project run by Grameen – their Community Knowledge Worker programme in Uganda.

The project employs Community Knowledge Workers who live in communities in Uganda.  These CKWs, many of whom were previously Agricultural Extension Workers, are “trusted neighbours” who can be consulted on a variety of issues that are deemed useful for smallholder farmers in Uganda.  More details about the CKW programme can be found on the Grameen website.

IDEOS phone - an $80 smart phone!At the presentation I attended, Grameen showcased the technology they are using to support their CKWs.  Each CKW is supplied with an IDEOS android mobile phone – these retail for $80 in Kenya.  The phones have three Grameen authored apps installed.

The first app, CKW Search, is a searchable repository of information that the CKW can consult through a very simple menu.  This information is stored locally on the phone, but is updated automatically when a 3G signal is available.  Each query is logged in the system with GPS coordinates, and this information is sent back to Grameen when a 3G signal is available.

The second app, CKW Survey, is a simple forms based app the CKW can use to capture images, video and text, as well as fill out surveys – typically the CKW will survey farmers who use their services.  Again the data is stored locally on the phone until a 3G signal is available.

The final app, CKW Pulse, is the hub through which Grameen can communicate with the CKWs.  This can be used to message an individual CKW or a group.  Each CKW can monitor their own performance based on the work they have done with the other two apps.  CKWs can also log support calls through CKW Pulse.

In addition to these applications, the CKWs can supplement their income by selling airtime on the phones, and selling phone charging services using the solar chargers they have for chargind the IDEOS phones.

At the backend, Grameen are using Salesforce to collect the data, and have a live dashboard where various aspects of the CKW service can be monitored.  The survey application is based on the Open Data Kit – a free open-source set of tools built to make survey building and data collection quick and easy.

This project is a great showcase of what can be achieved by joining up widespread mobile phone coverage, low cost smart phones, online database systems, and a well trained local workforce.

There are a number of opportunities that Christian Aid could take advantage of here.

  • Simple surveys on touch screen phones are a great way to collect baseline data and aggregate it quickly, before and after other initiatives have been carried out.
  • The ability to collect images, audio and video and send them quickly through 3G networks means that collecting stories about our work is easier and more immediate than ever.
  • The Pulse application could be a useful way to immediately keep in touch with partner organisations, reminding them about deadlines, contacting them with specific messages, or allowing them to see the results of M&E activities and understand their performance.
  • The Search application is a simple way of making different kinds of information available offline.  This could be useful for Christian Aid staff, as well as in programme work that employs the “trusted neighbour” model or similar.

For more information on the Grameen CKW project I suggest the following reading:

Grameen’s Community Knowledge Worker programme – an I4D case study

Tip of the Day: Create your own Admin Console in MMC

Fed up of logging on to a server as admin to manage Active Directory etc?  Found that you can’t run Active Directory Users & Computers as another user?

Why not create an Admin Console that includes all the Snap Ins you use, and can be Run As an admin user during a regular session?

Here’s how.

Start > Run > mmc

runmmc

MMC opens in edit mode

MMCEdit

File > Add/Remove Snap in

Click Add button

Add the different snap ins you normally use – ADUC, DNS, Exchange, Event Log Viewer are some of mine.

addremovesnap

Ok that, then save this MMC somewhere.

Now the tricky bit – running that MMC as a different user.

Create a shortcut to the MSC file:

shortcuts

Right click on the shortcut, and select Properties.

On the shortcut tab, click Advanced button.

credentials

Ok out of the shortcut properties.

Now when you double click the Shortcut it will ask you for credentials – you are in!

runas

Note – you may need to install Admin Tools to get the different Snap Ins – see this Microsoft document on how to install different tools.

Some of the snap-ins allow you to change the context – for example Event Viewer – you can change the server it is getting the logs for.  You can also set up multiple instances of some snap-ins – again, very useful if you want quick access to event logs on multiple servers.

Tip of the Day: Create your own Admin Console in MMC