Thursday, April 13, 2017

Sample SOAP payload for a SharePoint Remote Event Receiver

This post is mostly about showing what an actual Remote Event Receiver message looks like but there are some of my musings too because if there's something I love more than coding it is ranting about minor annoyances with the tools we use for doing it. If you can't wait to see the actual sample - go straight ahead, it's waiting for you at the end. For the most patient ones - let the rant begin!

First, a bit of context - I've joined new team working on a super complex project and as a result they are not letting me anywhere near the actual code for a second month now. What do they have me do, then? What could be the most inconsequential thing for a noob to do that would still have them learn about the solution? You guessed right - unit tests! So I want to test a Remote Event Receiver (RER for short) and I though, well, surely there must be a sample SOAP message sent by a RER somewhere around teh interwebz. Disappointingly, if there is such a thing it is very well hidden as I couldn't find it and had to generate my own. It wasn't as simple as one might think so I'm sharing the fruit of this labour with the world.

Now is the time to rant a bit about the frustrations regarding SharePoint development. I've ranted before about how hard it is to find anything meaningful online because of the ubiquity of posts explaining trivial stuff. Seriously, who needs a million how-tos about adding a RER to an app? You just right-click on the project in Visual Studio and follow the wizard. One thing I haven't moaned about before though is the inept way Microsoft is presenting SharePoint app development. And no, I'm not even referring to the ham-fisted rebranding from "apps" to "add-ins", which left the community calling them "apps" while MS insist that they are "add-ins", in the same way that MS insisted for years that JavaScript is actually called EcmaScript.


What is SharePoint app development anyway?

If you try to teach yourself SharePoint 2013/2016 through online resources, or even if you attend a course or read a book they'll have you believe that before you even write a hello world app you need to understand about SharePoint-hosted vs Provider hosted, you need to know what's the app web and host web, the remote web app, etc. and before you even start coding you need to configure your farm for apps, which is decidedly non-trivial.

I'd say screw all this and don't confuse people with the complexities of deployed apps from day one - it's needlessly obscure and fails to drive home the main point of the SP 2013/2016 app architecture: from now on, you won't be writing applications that get deployed into SharePoint, you'll be building ASP.NET apps that talk to SharePoint through the client library.
That's what Microsoft doesn't want you to know and maybe even fail to admit it to themselves but seriously, does anyone actually build apps to be deployed through an app catalog? Almost all projects I've worked on lately involve writing code in an ASP.NET application that talks to SharePoint through the CSOM. Yeah, I get it, MS built all this app catalog infrastructure and now want to shove it down our throats so they keep talking about provider hosted and app web and whatnot when you just want to connect to SharePoint and update some files.

So, if there is a beginner SharePoint developer reading this - fear not, it's not as complicated as MS make it sound. Nowhere is this more obvious than when we bring Remote Event Receivers (RER) into the picture. If you go by MS sources alone you'll start pulling your hair in despair, expecting weeks of work just to get the damn thing to work - now you don't just have the usual moving parts and security config, you also have a service that needs yet another way to obtain a SharePoint context! Well, guess what - you don't need any of that - you just create your service that implements the correct contract, call a few lines of CSOM code to register your services as a RER and you're good to go; no apps, no app-only policy configuration, no nothing. Now is the time to give a big shout out to Dan Budimir, who went against the grain at MS and wrote a guide about this very thing:
https://blogs.msdn.microsoft.com/boodablog/2015/01/12/i-thought-i-needed-a-sharepoint-provider-hosted-app-for-that-part-1/


How do we capture the SOAP message?

When I came to be in need of a recorded SOAP payload from a RER my first thought was - hey, we already have this functionality in IIS through Failed Request Tracing, which can be used to record all traffic, including requests that didn't actually fail. However, it has a size limit and for most RER request bodies it actually cuts the ending. Therefore, Fiddler had to come to the rescue - here is the whole process if you want to capture some requests yourself:
If all this sounds too complicated (which it is) - just make use of the few that I prepared, you should be able to adapt them if they don't cover the event you need.
Below you'll find the full text dumps for ItemDeleted (the smallest one) and ItemUpdated and here are links for two more events:
ItemAdded
ItemFileMoved (that's the event that gets fired when you rename a file)


ItemDeleted

<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
   <s:Body>
      <ProcessOneWayEvent xmlns="http://schemas.microsoft.com/sharepoint/remoteapp/">
         <properties xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
            <AppEventProperties i:nil="true" />
            <ContextToken />
            <CorrelationId>5bccd79d-32db-e06a-0000-077828386b67</CorrelationId>
            <CultureLCID>1033</CultureLCID>
            <EntityInstanceEventProperties i:nil="true" />
            <ErrorCode />
            <ErrorMessage />
            <EventType>ItemDeleted</EventType>
            <ItemEventProperties>
               <AfterProperties xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
               <AfterUrl i:nil="true" />
               <BeforeProperties xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays" />
               <BeforeUrl>Docs/test11.txt</BeforeUrl>
               <CurrentUserId>22</CurrentUserId>
               <ExternalNotificationMessage i:nil="true" />
               <IsBackgroundSave>false</IsBackgroundSave>
               <ListId>6980e5dd-d847-4667-8792-1e2741d6cf18</ListId>
               <ListItemId>7</ListItemId>
               <ListTitle>Docs</ListTitle>
               <UserDisplayName>Kiryazov, S. (Stefan)</UserDisplayName>
               <UserLoginName>i:0#.w|europe\d-hy32co</UserLoginName>
               <Versionless>false</Versionless>
               <WebUrl>http://sharepoint.server/sites/sitecolurl</WebUrl>
            </ItemEventProperties>
            <ListEventProperties i:nil="true" />
            <SecurityEventProperties i:nil="true" />
            <UICultureLCID>1033</UICultureLCID>
            <WebEventProperties i:nil="true" />
         </properties>
      </ProcessOneWayEvent>
   </s:Body>
</s:Envelope>

ItemUpdated

<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
  <s:Body>
    <ProcessOneWayEvent xmlns="http://schemas.microsoft.com/sharepoint/remoteapp/">
      <properties xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <AppEventProperties i:nil="true" />
        <ContextToken />
        <CorrelationId>00000000-0000-0000-0000-000000000000</CorrelationId>
        <CultureLCID>1033</CultureLCID>
        <EntityInstanceEventProperties i:nil="true" />
        <ErrorCode />
        <ErrorMessage />
        <EventType>ItemUpdated</EventType>
        <ItemEventProperties>
          <AfterProperties xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
            <a:KeyValueOfstringanyType>
              <a:Key>vti_folderitemcount</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">0</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_nexttolasttimemodified</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:dateTime">2017-02-01T08:30:41</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_candeleteversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">true</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_foldersubfolderitemcount</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">0</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_canmaybeedit</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">true</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>ContentTypeId</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">0x010100AE33CE7DE32F484490EEC317F89BE311</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_sourcecontrolcookie</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">fp_internal</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_filesize</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">1376</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_modifiedby</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">i:0#.w|domain\username</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_setuppathversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">15</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_level</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">1</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_timecreated</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:dateTime">2017-02-01T08:30:41</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_charset</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">utf-8</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_parserversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">15.0.0.4867</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_sourcecontrolversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">V1.0</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_author</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">i:0#.w|domain\username</a:Value>
            </a:KeyValueOfstringanyType>
          </AfterProperties>
          <AfterUrl>Docs/soap-list1.xml</AfterUrl>
          <BeforeProperties xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
            <a:KeyValueOfstringanyType>
              <a:Key>vti_metadatanextbsn</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">100</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_backgroundsave</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">0</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_replid</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">rid:{2824DA8B-7E05-4687-A6D3-E5293D4439FC}</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_level</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">1</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>ContentTypeId</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">0x010100AE33CE7DE32F484490EEC317F89BE311</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_setuppathversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">15</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_rtag</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">rt:2824DA8B-7E05-4687-A6D3-E5293D4439FC@00000000002</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_sourcecontrolversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">V1.0</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_sourcecontrolcookie</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">fp_internal</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_filesize</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">1376</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_modifiedby</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">i:0#.w|domain\username</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_metainfoversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">2</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_internalversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">513</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_nextbsn</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">162</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_parserversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">15.0.0.4867</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_docstoreversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">2</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_author</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">i:0#.w|domain\username</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_foldersubfolderitemcount</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">0</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_contentversionisdirty</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">false</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_timecreated</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:dateTime">2017-02-01T08:30:41</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_contenttag</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">{2824DA8B-7E05-4687-A6D3-E5293D4439FC},2,2</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_docstoretype</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">0</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_canmaybeedit</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">true</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_charset</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">utf-8</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_folderitemcount</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">0</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_timelastmodified</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:dateTime">2017-02-01T08:55:24</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_parentid</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">{733E0BCD-E0B4-45DA-8492-FDE35C98DD22}</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_candeleteversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">true</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_nexttolasttimemodified</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:dateTime">2017-02-01T08:30:41</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_contentversion</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">2</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_etag</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:string">"{2824DA8B-7E05-4687-A6D3-E5293D4439FC},2"</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_timelastwritten</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:dateTime">2017-02-01T08:55:24</a:Value>
            </a:KeyValueOfstringanyType>
            <a:KeyValueOfstringanyType>
              <a:Key>vti_streamschema</a:Key>
              <a:Value xmlns:b="http://www.w3.org/2001/XMLSchema" i:type="b:int">66</a:Value>
            </a:KeyValueOfstringanyType>
          </BeforeProperties>
          <BeforeUrl>Docs/soap-list1.xml</BeforeUrl>
          <CurrentUserId>22</CurrentUserId>
          <ExternalNotificationMessage i:nil="true" />
          <IsBackgroundSave>false</IsBackgroundSave>
          <ListId>6980e5dd-d847-4667-8792-1e2741d6cf18</ListId>
          <ListItemId>2</ListItemId>
          <ListTitle>Docs</ListTitle>
          <UserDisplayName>Kiryazov, S. (Stefan)</UserDisplayName>
          <UserLoginName>i:0#.w|europe\d-hy32co</UserLoginName>
          <Versionless>false</Versionless>
          <WebUrl>http://sharepoint.server/sites/ipp-content-stef</WebUrl>
        </ItemEventProperties>
        <ListEventProperties i:nil="true" />
        <SecurityEventProperties i:nil="true" />
        <UICultureLCID>1033</UICultureLCID>
        <WebEventProperties i:nil="true" />
      </properties>
    </ProcessOneWayEvent>
  </s:Body>
</s:Envelope>


Tuesday, March 7, 2017

SaveBinaryDirect giving you HTTP 500? Fear not!

I managed to get my code to work on the Friday afternoon and opened my celebratory beer with this warm sense of accomplishment. Life was good. For a while. Monday quickly rolled around as it always does and I found myself looking at a broken application - not a great start of the week; the code gremlins had once again sabotaged my work. My call to SaveBinaryDirect was crashing and a bit of debugging revealed that SharePoint was returning HTTP 500 without any further details (System.Net.WebException: The remote server returned an error: (500) Internal Server Error). What could it be?

Spolier alert: the maximum number of minor versions was being exceeded.

Now let's enjoy the story of my troubleshooting efforts even though I spoiled it. Some stories are worth reading even if you know how they end.

First thing I tried was to upload a different file - no problem with that. Double-checked if the file was checked-out - it wasn't. It was time to dig in the ULS.

If you have verbose logging on CSOM calls are actually easy to troubleshoot, you can see all the log entries for a given request more or less in sequence and if you use UlsViewer, and I don't see why you wouldn't, you can filter for a specific request.

I compared the log entries for the successful and unsuccessful uploads and the only difference was these two lines:
SQL return value 154 as HR 0x8007009a.
Translated store error 0x8007009a to 0x00210088

Now that's positively cryptic and google was of little help. Fortunately, it downed on me to conduct another test before spending too much time on this - I tried to upload the file the regular way, with a FileCreationInformation and batching as opposed to SaveBinaryDirect. Much to my delight, this time SharePoint was kind enough to tell me exactly what it didn't like about my request:
Microsoft.SharePoint.Client.ServerException: Minor version limit exceeded.
You must first publish a major version before editing this document.

In my case the file was being uploaded every 5 minutes, which exhausted the maximum number of 511 minor versions in just about 42 hours, which may or may not have something to do with life, the universe and everything.

I did as advised and now my solution publishes a major version by checking out and back in. The major versions can go up to 400,000 which will give me more than 3 years with an update every 5 minutes - much better though still not ideal, I guess I'll have to timestamp the file in order to avoid all these issues.

I also simplified the story of forcing the major version which is not exactly trivial - one needs to take into account situations like the file being left checked out, both by the same user and by others. If you behave I'll give you the code for this, too.

Thursday, October 27, 2016

What is time?

Тhe intricacies of System.DateTime - а .NET essay in three parts


Part 1: You're doing it wrong - don't use DateTime.Now


I was tempted to make this the shortest blog post I've ever written: Always use DateTime.UtcNow instead of DateTime.Now! It would be a magnificently prosaic follow-up to a deep-sounding title.
The takeaway point is really that simple. However, developers have inquiring minds and would demand to know more. Therefore, I did the exact opposite - wrote way more than anyone would want to read on the topic :-) so, if you're only after the reasoning behind the above recommendation - head straight to paragraph 4. If, on the other hand, you'd like to generally enlighten yourself about how dates are handled by humans in general and in .NET programming in particular then pour yourself a cup of coffee and plough straight through.


First, let's get one thing out of the way - dates are awkward. I don't mean data types representing dates - just dates in general. We live on an awkwardly shaped object that revolves around its parent star at slightly irregular intervals and to make matters worse - it's populated by beings obsessed with measuring time precisely*; we have roughly 365 1/4 days to the year and even if we ignore the "roughly 1/4" part of it we're still stuck with 365, which is an unfortunate number - we can't have months of equal numbers of days (unless having five months of 73 days sounds appealing to you) yet 360 is so close. If only we had 360 days in the year we would have had neat half years, third years, quarter years, 6-th years, 8-th years, 10-th, 15-th, 24-th... and 12 neat months of 30 days each. Alas, it was not meant to be - this, in my opinion, is one of the strongest arguments against a benevolent creator of the universe - what kind of god would do this to us? Apparently he didn't design this place with developers in mind.


Before developers entered the picture though there were all kinds of folk who liked order and measuring things precisely and this bugged the hell out of all of them. How could we divide time in sensible small-scale units for everyday use that stack nicely into larger units and eventually - into the cycles of our home planet? Civilizations experimented with all kinds of arrangements until we settled on the present one with weeks and months. We owe the concept of the month to the romans, but theirs were initially quite different than ours - they had just 10, with the winter not counting in the calendar at all and had three kinda-sorta-weeks of uneven length per month; the famous Mayan calendar that hit its own y2k** problem in 2012 on the other hand had two parallel cycles of 260 and 365 days - try converting that to System.DateTime! Our modern day calendar may be somewhat saner but it still has its leap seconds, daylight saving time and other weirdness.


I hope that you're now convinced that dates are awkward and messy, which may or may not alleviate the stress you're experiencing when troubleshooting programming issues related to dates. Indeed, some of these issues are due to the convoluted nature of our calendar and cannot be helped; others though can be easily resolved with this little hack - use UtcNow instead of Now. What's the difference? Well, DateTime.Now returns the current time in whatever timezone the computer it's executing on runs while UtcNow converts the current time to the UTC time zone. "Why would I care about UTC time zone, I live in San Francisco", some of you might be thinking. It doesn't matter where you're based, where your customers are or where the server is located - UTC is a fine choice of universal time that will be the same on every computer where the code might be executed. Why is this important, then? Consider the following scenario:
You are building a web app that allows users to manage documents. You have a functionality that shows users the documents they've opened in the last day. You might have a stored procedure like this one:
SELECT * FROM Documents WHERE DateOpened > DateAdd(DAY, -1, getutcdate()) 

  AND
DateOpened < getutcdate()
Nice and simple. And when you open a document, you run:
UPDATE Documents SET DateOpened = @d WHERE DocumentId=@id

and you pass DateTime.Now as the value of the @d parameter. You've done similar things, haven't you, retrieving the data for the last X days or hours? Maybe not just like that, maybe you use Entity Framework instead of stored procedures - it doesn't matter, the concept is the same. The bug is the same, too. See, if your SQL server resides in a different timezone than your webserver this is not going to work; in the best case it will show old documents (when the SQL server is behind), or it might not work at all if you're showing docs for the last hour instead of day.
In case you're thinking that all your code is going to run on the same server I regret to inform you that you're still doing it wrong and are just desperately trying to find an excuse to not change your wicked ways. The situations when it's safe to use DateTime.Now are a tiny minority and in most cases you can't be sure that you're in one of them at the time of writing the code and no, you're not going to come back and change it if the requirements change. Just quit your mumbling and get into the habit of always using DateTime.UtcNow instead of DateTime.Now. You can even go further. Do you have a coding standards document in your company? If you do - go ahead and edit it right now, mandating that all developers do it. If anyone objects - tell them that they are wrong; let Sheldon Cooper be your inspiration for dealing with ignoramuses who have yet to see the light of day. 


* - contrary to Aristotle's advice that "it is the mark of an educated mind to rest satisfied with the degree of precision which the nature of the subject admits"
** - many think of the y2k bug as an embarrassment for software developers; "duh", the general population thinks, "didn't these programmers figure out that the year won't fit in two digits, aren't they supposed to be like super smart - this sounds like a pretty dumb mistake to me". However, it's actually a compliment - the programs that the industry pioneers cobbled together to fix some problem there and then ended up solving millions of other problems for decades on end. The y2k panic is a hats off for software built to last.

Saturday, April 6, 2013

Processing XML with .NET while preserving your sanity


XML is facing quite a bit of criticism, raging from accusations of being too complex to just being on of Micrsoft's attempts to take over the world (it's actually an open W3C standard but nevermind). The general consensus is however that it does what is says on the tin, which is to transfer data between applications, allowing humans to take a peak in-between. The focus of this post is the last part involving human beings (stupid humans, always making programming more difficult than it needs to be!)

Consider this piece of XML:
<?xml version="1.0" encoding="UTF-8"?>
<xmldata>
  <element>value</element>
</xmldata>
Nice and readable, isn't it? Well, most times at least. Now look at it without the new lines and identation:
<?xml version="1.0" encoding="UTF-8"?><xmldata><element></element></xmldata>
Much less so, indeed!
Everyone who has had their fair share of XML programming has faced this situation - for parsers the two XML documents are identical, however when we need to take a look at our data when debugging we get confused by the ugly, unformatted XML. This is made worse by the .NET Framework's tendency to produce the latter variety of XML by default.

There are, of course, simple ways to make our lives easier by formatting XML the way we want it - it's just less obvious than it should, so I decided to put this post together and shed some light on a few tricks and subtleties.

Dumping XML to a file & XmlWriter

It's a natural scenario to grab an XML file, do some processing, and save it. Here is a trivial piece of code that does that:
XmlDocument xml = new XmlDocument();
xml.Load(INPUT_PATH);
// ... XML processing code
xml.Save(OUTPUT_PATH);
This of course can easily be adapted to save the XML to a database, over the network or wherever else we need it to be. Most times it would appear to work fine, until you try it on our simple input. Here is what we get:
<?xml version="1.0" encoding="utf-8"?>
<xmldata>
  <element>
  </element>
</xmldata>
Looks almost the same - but not quite, the closing tag is on a new line! Curiously though, when there is an actual value between the tags we don't get a new line added to it, so it might be pretty hard to spot in a complex XML document with most elements containing values. That's actually what caused the bug I was troubleshooting when I got the inspiration for this post - it might look like it's not a big deal but what's between the opening and closing tag is our value and we just had a new line inserted into it - for most applications a new line is quite different than an empty string!
So then, what do we do about it?

Using XmlWriter

We have a class in the .NET Framework that's meant to give us more control over XML-exporting operations - XmlWriter. Here is the simplest possible way to use it, without supplying any explicit settings:
XmlDocument xml = new XmlDocument();
xml.Load(INPUT_PATH);
// ... XML processing code
XmlWriter writer = XmlWriter.Create(OUTPUT_PATH);
xml.Save(writer);
writer.Close();
And the result:
<?xml version="1.0" encoding="utf-8"?><xmldata><element></element></xmldata>
... i.e. exactly what we were trying to escape from. Looks like the XmlWriter is of not much use by itself when it comes to formatting XML for human-readability.

XmlWriterSettings.Indent

Luckily, there is the Indent property in XmlWriterSettings which is false by default but can very easily bet set to true:
// ... XML processing code
XmlWriterSettings writerSettings = new XmlWriterSettings();
writerSettings.Indent = true;
XmlWriter writer = XmlWriter.Create(OUTPUT_PATH, writerSettings);
xml.Save(writer);
Leading to, finally, a well-formatted XML output!

XmlWriter with it's settings object is nice and all, but there is one caveat that you could hit before getting to the Indent property - there is also the NewLineHandling property, which you might be tricked into thinking would achieve our goal. In fact, it only affects the new lines within actual values between tags and doesn't apply to the new lines in the markup.

Juggling with XML formats in-memory

But wait a second, this neat solution relies on XmlWriter, which - in this example, at least - only writes to a file. What if we don't actually need to write the file to the FS and we'd rather have a string, byte array or some other in-memory structure? We have a few ways to achieve this using the same XmlWriter-based logic, presented here from most to least atrocious.
One option is to just save the file and read it back like a text or binary file - this will load the well-formatted XML in memory as string/bytes/whatever. In case this solution looks attractive, I have some advice for you - don't do it. Even with an SSD drive read/write operations are expensive, and also an unnecesarry risk - the HDD might be full, we might not have access to the folder, etc. I can think of only one situation where this 'solution' would be advisable - if you actually need the physical files, e.g. for debugging or logging purposes.
Another way to harness XmlWriter for this task is to combine it with .NET's flexible, polymorphic stream architecture and create your XmlWriter around a MemoryStream. Now if using something called MemoryStream in order to just sanitize your XML doesn't sound like an overkill to you then I guess I can't argue further - if you haven't been featured on TheDailyWTF you probably will soon be.
But fear not - there is another way. Enter LINQ...

XElement/XDocument

This solution doesn't actually use LINQ itself - it just taps on the XElement-based infrastructure that LINQ to XML uses to objectify XML documents. Apart from making it possible to use LINQ on XML, these classes also use some of the more recent .NET framework additions, like object initialization and anonymous types, in order to make dealing with XML in .NET less cumbersome.
Here is how to beautify an XML document in-memory and assign it to a string:
XDocument xDoc = XDocument.Load(INPUT_PATH);
// ... XML processing code
String xmlString = xDoc.ToString();
And here is what we get:
<xmldata>
  <element></element>
</xmldata>
Almost but not quite - it's missing the XML declaration (<?xml version...). Here is how add it - we just need to change the last line to:
String xmlString = xDoc.Declaration.ToString() + Environment.NewLine + xDoc.ToString();
And voila - we have a well-formatted XML in-memory, in two lines!

Further notes on XML and strings in .NET

But why did we need to change that line in order to add the XML declaration, why doesn't it get included automatically? Upon further observation, we can see that there is the XDocument.Save(String path) method, which saves the document to disk and does include the declaration - so it starts to look like an unintentional omission to not include it in ToString()?

As it turns out, not only is there a reason for that, but there are in fact at least two good reasons to implement ToString() in such a way, and each of them reveals something interesting about the way .NET handles XML and strings in general

Handling chunks of XML in-memory

The traditional XmlElement-based approach is a traditional DOM implementation - it builds a tree of the document in memory, and deals with all pieces of XML as documents - even if it's a single element, a dummy XML document object will be created around it. That's not the case with XElement - there an XElement instance can represent just one XML element without any context.
This point of view is taken further by considering the XML declaration to not be a part of the document - it's just a header of the .xml file format to mark the content as valid XML, to indicate the version and encoding (although you need to know what's the encoding in order to read the header that gives you the encoding but nevermind). That's why you only get the XML declaration inserted when you save the thing to a file - before that it's just a document in memory that holds a piece of XML markup.

String encoding in .NET

First, a quick refresher on encodings, as I suspect that developers that work for the western market only don't deal with them in-depth on a daily basis. Encodings are ways to map characters to sequences of bits so that they can be stored in binary media. Everyone knows about ASCII, which assigns one byte per character and fits just the Latin alphabet and a bunch of funny symbols, and Un In the average .NET developer's practice, encodings are used explicitly in order to convert strings to bytes and vice versa. Let's extend our example in order to get that neat XML in a byte array, e.g. to be sent over a socket:
XDocument xDoc = XDocument.Load(INPUT_PATH);
String xmlString = xDoc.Declaration.ToString() + Environment.NewLine + xDoc.ToString();
byte[] xmlBytes = Encoding.UTF8.GetBytes(xmlString);
Now, we can of course call our good friends XmlWriter and MemoryStream but as was demonstrated we have a better way to deal with the task at hand - the only change from the previous example is the addition of the last line that uses the UTF8 encoding to convert the sequence of symbols that is represented by our string to a sequence of bytes. I bolded this because it's crucial for my next point - to understand the situation here, we need to think of strings abstractly. When we have a string object it is of course just a point to a region in memory that is filled with bytes but that's of no concern to our encoding object - it only looks at the sequence of charters, regardless of how they are represented in memory. It then generates the matching bytes for each character to give us our byte array - that's it.

OK, but what does this have to do with XML and the reason why XDocument doesn't include the XML declaration? Well, it's the same principle - XDocument is a pure soul without a body (i.e. a physical file), and it treats the encoding as a bodily concern - it's a mere physical representation of the information in the XML document. That's why the pure information contained in the XDocument object shouldn't contain the XML header, and with it - the encoding, when in fact it isn't associated with any encoding at all.

.NET does store strings as bytes in memory, so there is one more encoding operation going on all the time - the mapping of our sequence of symbols to the bytes in the managed heap. For this purpose, CLR uses UTF-16 - hence the 2 byte size of the char datatype and 2 bytes per symbol for string. What is a little bit confusing at first is that there is no specific UTF-16 encoding option, although we have UTF7, UTF8, UTF32 and Unicode - which is not even an encoding but the overall standard that covers them all. The System.Text.Encoding.Unicode encoding is in fact UTF-16, which is a glimpse into how in the .NET architects' team they assume UTF-16 to be the default encoding.

For more details and fun into how strings, and objects in general, are stored in memory in .NET you can always count on the guru Jon Skeet: http://msmvps.com/blogs/jon_skeet/archive/2011/04/05/of-memory-and-strings.aspx

Bonus: The code

Here is a simple Visual Studio 2010 solution that demonstrates all the code in this post for download

Also, in case you don't like downloading files from strangers - here is the same solution shared on CodePlex:
https://xmldemo.codeplex.com/

Friday, September 21, 2012

InfoPath - The URN specified in the XML template file does not match the URN specified in the form

You cheeky developer you, been monkeying around with your XSN files, haven't you ;-)

There comes a time in every SharePoint developer's life when he has to deal with some InfoPath. That's often a frustrating experience - especially when inheriting a middle-sized to large project. InfoPath probably does what it's supposed to do well - and this is to provide static forms for users to fill. If you swerve from the road well travelled however and try to build something more custom some opportunities (as we call problems nowadays) are likely to emerge, which get aggravated by the limited resources on the more specific scenarios.

One of those specific cases is when you need to manage form templates on the fly, programmatically. That's not particularly hard - after all InfoPath form templates are just CAB files containing XML, so you can dynamically unwrap them, do the changes needed and then put them back together with MakeCab or similar tool. Still, the process is not that hard to botch, and this "The URN specified in the XML template file does not match the URN specified in the form" error is one of the possible outcomes of getting it wrong.

This one is kind of self-descriptive, although it might not be exactly intuitive for InfoPath newbies. See, the URN (Uniform Resource Name) something like an identifier for the form template and is referenced on two different places throughout the files that comprise the XSN package and naturally, it needs to be the same in both. So, if you are spewing InfoPath form templates on the fly, with different names, you need to take care to edit the URN in sync - otherwise you end up with this error. You can also leave these alone altogether and not edit them - I haven't noticed any adverse consequences so far.

First, in minifest.xsf:

<?xml version="1.0" encoding="UTF-8"?>
<!--
This file is automatically created and modified by Microsoft InfoPath.
Changes made to the file outside of InfoPath might be lost if the form template is modified in InfoPath.
-->
<xsf:xDocumentClass solutionFormatVersion="2.0.0.0" solutionVersion="1.0.0.46" productVersion="14.0.0" name="urn:schemas-microsoft-com:office:infopath:your_form_name:-myXSD-2011-02-09T17-25-42" xmlns:xsf="http://schemas.microsoft.com/office/infopath/2003/solutionDefinition" xmlns:xsf2="http://schemas.microsoft.com/office/infopath/2006/solutionDefinition/extensions" ...

Then, in template.xml:

<?xml version="1.0" encoding="UTF-8"?>

<?mso-infoPathSolution name="urn:schemas-microsoft-com:office:infopath:your_form_name:-myXSD-2011-02-09T17-25-42" solutionVersion="1.0.0.46" productVersion="14.0.0" PIVersion="1.0.0.0" ?>
<?mso-application progid="InfoPath.Document" versionProgid="InfoPath.Document.2"?>
<?mso-infoPath-file-attachment-present?>
<my:myFields ...


Friday, March 23, 2012

Smart phones, dumb people

The onset of spring reminds me of many things - beers in parks, the project that was due to finish in February, etc. - but the most relevant one is that I've actually been owning my HTC Desire Z for almost a year. My, how does time fly when you have a smartphone! Remember how tedious the longer trips to the bathroom were back in the day, reading those shampoo labels over an over again, waiting for the relief of the long awaited splash sound? And now you just can't get enough of those moments, replying to your facebook messages in the privacy of the smelly cubicle; and before you know it it's time to wipe, but you still look at the screen while you do. Mobile computing has indeed changed our lives profoundly.

I remember those days when I first fell in love with my latop, thinking that no other gadget will ever get nearly as close to my heart. And I still love it dearly - but let's face it - no matter how mainstream geek culture has become it will always be considred weird to walk out of the toilet with a laptop. Still, I must admit, smartphones have a long way to go - although Android is quite good for what it's worth, it doesn't come anywhere near a full featured OS and my limited exposure to WP7 and iOS has clearly indicated that they have a different but at least equally annoying sets of drawbacks. This is to be expected - one needs to make efficient use of the limited resources of the small device so some cuts have to be made and in the same time the millions of morons that use it around the world should be prevented from from ruining it by downloading dodgy stuff.

My smartphone has anyway found a unique place in my digital life and is inching its way towards my heart. Nevertheless, ask the general population what they think about smartphones - most will be like "Oh, those zombies staring at them iphones constantly, I wonder how they don't get run over by buses! Mine phone shows me the time and takes calls and that's all I need". It's natural for people to not feel needs that have never been fulfilled - you never miss something that you've never had. I'm confident that these are going to be a dying breed soon, though. In fact, it's already happening, at least in London - the overall UK smarphone penetration is at the top of the range for Europe at 40% but this includes rural areas with poor 3G coverage where it makes little sense to own one; in London the tedium of the trips in the tube and the congested traffic has forced most people to kill time by tapping their fingers on various screens. And in Singapore it has already happened, with hefty 90% - gosh, don't they have old people there, I thought it's a civilized country but apperently they euthanize them at 50.

Now tablets are braced to be (or already are?) the next big thing. When the first iPads came out they looked kind of neat although I couldn't quite see their niche. iPads, I thought, are only good for use in cramped seats, and iPad owners never travel in cramped seats. I was willing to try one but as someone that types with more than 60 wpm, I see the physical keyboard as a very important part of the human-device interface, so the urge was not that strong. Then the Android-based tablets came out and although they are still in their infancy I'll be damned if I buy a gadget with storage that I can't browse with my shell but should rather sync through iTunes, one of the shittiest applications every to be shoved down the users' throats. Immature as they are, the tablet wars have already provided for some hilarity with the countless legal battles that Apple and Samsung are locked in, cultimating to the one that invokes 2001: A Space Odyssey and shall from now on be known as the Kubrick defense (after the Chewbacca defence). I another twist in this story, the supposedly game-chaning new screen of the iPad and its heart, the A4 system-on-a-chip (i.e. CPU + GPU) are both manufatcured by Samsong. Go figure! The only thing that's for certain is that the Samsung tablets are not likely to be technologically inferior to the iPads. Now throw in an open OS and lower price and I can see them getting popular quite soon.

Still, I'm not entirely sold just yet. Apart from the cramped seats, which I do use often, I have limited use of a super big phone that doesn't fit into a pocket, which is what tablets pretty much are if you run iOS, or even android, on them. Now if only this thing could have a real OS on it and still last that long on battery!

I kind of like Microsoft's strategy here - they include tablet support for the full-featured OSs and leave Windows Phone just for, well, phones. Subsequently, vendors of Windows-based tablets are trying to fit a real computer into the form factor, rather than going bottom-up (i.e. enlarging a touchscreen phone). An example is the Samsung Series 7 Slate PC, that has a real i5 processor and comes from their laptop, rather than mobile phone division. That's a gadget I'd love to use so much that the 3:30 hours of battery life will simply not be enough. Now there is a lot of hype about Windows 8 and how tablet-friendly it's going to be and I must admit that I'm kind of succumbing to it, if someone manages to put it on something equally powerful that lasts for more than 7 hours on battery I'll be the first to buy one (online - not queueing like an apple addict).

Wednesday, September 28, 2011

.NET is more open than Java

There has always been this stereotype that Microsoft are sworn enemies of open source and Java is uber cross-platform. Well, guess what, that's not true and the proof came in April 2010. I know that this is about a century ago in software years but for some reason the news have gone past me and I'd hate to not comment on that - be it regretably late.

The story goes as follows - after acquiring Sun (and the Java technology along with it), Oracle decided to sue Google over their use of Java on the Android OS (source), basically redefining 'cross-platform' to mean 'running on any platform that Oracle can make money out of'. The fact that Java had a chance to become the de facto universall development environment for mobile devices doesn't matter.

In the same time Microsoft have this thing called Community Promise, which basically means that they are OK with you implementing a .NET environment on any platoform, and more specifally - it means that they won't pull the same stunt about .NET on Android (source, and yeah, .NET development on Android is possible - http://android.xamarin.com/ ).

Now let's get in the time machine and travel back to present day. It's already October 2011 and in the meantime Android has become the dominant OS for smartphones, and Novell have been acquired by Attachmate and along with it - the Mono and Mono for Android team (Ximian, the company that originally developed Mono was acquired by Novell), followed immediately by Attachmate dumping the Mono and MonoDroid teams - a truly disturbing development for the .NET and mobile development communities. However, a large part of the team that originally created Mono now formed Xamarin - a company that is committed to supporting and advancing this family of products, and they have reached an agreement with Attachmate to be the official stewards of these projects.

So, Mono, MonoTouch and Mono for Android, after changing hands three times are now once again in the hands of a small, dedicated team, which is of course very promising. We'll only need to see who will get to buy Xamarin next...

I really hope for one thing, though - that it's not going to be Microsoft or Google, or for that matter - any other major player in the smartphone game, as this will lead to Apple immediately blocking MonoTouch on iOS. For the ones unfamiliar with MonoTouch - this is a product that lets you compile .NET applications for iOS, and with HTML5's future being uncertain before 2022 - that's pretty much the only way to run anything that was not orignially written on Objective C on the iPhone and other iOS devices. Now MonoTouch is not a true .NET implementation - it's not CLR, it doesn't JIT and execute intermediate language code - it just compiles .NET to native code.

There were times that the future of MonoTouch was quite murky - there was the infamous point 3.3.1 in the Apple developer agreement that stipulated that iPhone apps must be originally developed in Objective C, i.e. the trick that MonoTouch does was officially banned. But then the ailing Jobbs was apparently struck by a dose of benevolence and decided to let MonoTouch do its thing.

The implications of both developments - now you can develop in .NET on every major mobile platform! Of course it's not as easy as it sounds - as Miguel de Icaza (the mastermind behind Mono) points out himself, you can only really reuse the business logic, you'll need to implement the front-end differently for every platform. Still, you'll do it with the same tools, and will follow roughly the same practiceses.

The topic of cross-platform mobile development with .NET is one that has captured my interest and I will be posting more about it in the future, so if there's anything that you are particularly interested in, or have something interesting to share please make full use of the comments.