Rewriting FullTextQuery within CoreResultsWebPart

Recently I found that SharePoint advanced search used query parts like Title like ‘%querystring%’ when dealing with contains operators. When dealing with documents that had long titles this didn’t yield the expected search results and with further testing I found that using CONTAINS(Title,’”querystring”’) gave me all the results I wanted.

Problem was that the advanced search was out-of-the-box and no way to configure this (as far as I know) so I needed a way to rewrite the FullTextQuery that was being generated by SharePoint without too much custom code and changing the behavior of the CoreResultsWebPart.

Enter: Reflection. I love this tool, but it should be used sparingly in large solutions as there is a performance hit.

Anyway, basically what I did was extend the CoreResultsWebPart and override the SetPropertiesOnHiddenObject method. This method initializes a private field within CoreResultsWebPart of the type SearchResultHiddenObject and this class contains the FullTextQuery propery which we want to modify at runtime.

So, here’s the quick and dirty code (should be cleaned up before production use):

private string queryPattern = @"(?<Title>Title[\w\s]+[^%]%(?<Query>[\w\s\d]+[^%]*)%')";

protected override void SetPropertiesOnHiddenObject() { base.SetPropertiesOnHiddenObject(); try { Type t = this.GetType(); FieldInfo hiddenObjectField = t.BaseType.GetField("srho", BindingFlags.NonPublic | BindingFlags.Instance); if (hiddenObjectField != null) { object hiddenObject = hiddenObjectField.GetValue(this); Type hiddenObjectType = hiddenObject.GetType(); PropertyInfo fullTextQuery = hiddenObjectType.GetProperty("FullTextQuery", BindingFlags.Public | BindingFlags.Instance); string query = fullTextQuery.GetValue(hiddenObject, null) as string; if (query != null) { Match m = Regex.Match(query, queryPattern); if (m.Success) { string queryString = m.Groups["Query"].Value; query = Regex.Replace(query, queryPattern, "CONTAINS(Title,'\"$2\"')", RegexOptions.IgnoreCase); fullTextQuery.SetValue(hiddenObject, query, null); } } } } catch (Exception ex) { HttpContext.Current.Response.Write("Error: " + ex); } }

‘As you can see I use a regular expression to match and replace the query, but you can replace this with however you wish to modify the query.


Replace the out-of-the-box CoreResultsWebPart with this webpart and you’ll retain all the normal functionality except you gain a little more power over the actual query that is used. This works perfectly on SharePoint 2007; I haven’t tested it on SharePoint 2010 and it might not be needed there.

SharePoint FBA & Reactive Extensions

Some of the dangers with new toys is to overuse them, I fear I might be doing that with reactive extensions but oh well.. It’s all about the learningSmile

Today I did a test to do Form Based Authentication with SharePoint using reactive extensions. This is the first attempt, it can probably be made more streamlined as I figure out more about Rx.

Oh, apologize for the syntax coloring in the code, I really need to find some better CSS for this…

I have two classes: One for doing the FBA (which also holds the cookie jar as we will get a cookie once the FBA is successful and need to pass this along to the next web service call) and one for doing web service calls.

public class FormsBasedAuthentication {
    public static CookieContainer CookieJar = new CookieContainer();
    private const string AuthEnvelope = @"<?xml version=""1.0"" encoding=""utf-8""?>
                <soap:Envelope xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"" xmlns:xsd=""http://www.w3.org/2001/XMLSchema"" xmlns:soap=""http://schemas.xmlsoap.org/soap/envelope/"">
                  <soap:Body>
                    <Login xmlns=""http://schemas.microsoft.com/sharepoint/soap/"">
                      <username>{0}</username>
                      <password>{1}</password>
                    </Login>
                  </soap:Body>
                </soap:Envelope>";

    public static IObservable<string> Authenticate(string server, string username, string password) {
        Uri authService = new Uri(string.Format("{0}/_vti_bin/authentication.asmx", server));
        HttpWebRequest authRequest = WebRequest.Create(authService) as HttpWebRequest;
        authRequest.CookieContainer = CookieJar;
        authRequest.Headers["SOAPAction"] = "http://schemas.microsoft.com/sharepoint/soap/Login";
        authRequest.ContentType = "text/xml; charset=utf-8";
        authRequest.Method = "POST";
        return (from request in Observable.FromAsyncPattern<Stream>(authRequest.BeginGetRequestStream, authRequest.EndGetRequestStream)().Do(stream => {
            UTF8Encoding encoding = new UTF8Encoding();
            string envelope = string.Format(AuthEnvelope, username, password);
            byte[] bytes = encoding.GetBytes(envelope);
            stream.Write(bytes, 0, bytes.Length);
            stream.Close();
        })
                from response in Observable.FromAsyncPattern<WebResponse>(authRequest.BeginGetResponse, authRequest.EndGetResponse)()
                select HandleResponse(response, authRequest));
    }

    private static string HandleResponse(WebResponse response, HttpWebRequest request) {
        var httpResponse = response as HttpWebResponse;
        if (httpResponse != null){
            var result = XDocument.Load(response.GetResponseStream());
            var code = result.Descendants(XName.Get("ErrorCode", "http://schemas.microsoft.com/sharepoint/soap/")).FirstOrDefault();
            if (code != null)
                return code.Value;
        }
        return "ErrorOccured";
    }

}

This will return an observable that will return the authentication result – if the login was successful, the published value will be NoError.

Next we have our web service class:

public class BaseWebService {
       private static string soapEnvelope =
           @"<soap:Envelope
                           xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
                           xmlns:xsd='http://www.w3.org/2001/XMLSchema'
                           xmlns:soap='http://schemas.xmlsoap.org/soap/envelope/'>
                   <soap:Body>{0}</soap:Body></soap:Envelope>";

       private XDocument soapEnvelopeXml;
       public string BaseUrl { get; set; }
       #region Helper methods
       private XDocument CreateSoapEnvelope(string content) {
           var envelope = string.Format(soapEnvelope, content);
           XDocument soapEnvelopeXml = XDocument.Parse(envelope);

           return soapEnvelopeXml;
       }

       private HttpWebRequest CreateWebRequest(string url, string action) {
           Uri serviceUrl = new Uri(url, UriKind.Absolute);
           var webRequest = WebRequest.Create(serviceUrl) as HttpWebRequest;
           webRequest.Headers["SOAPAction"] = action;
           webRequest.ContentType = "text/xml;charset=\"utf-8\"";
           webRequest.Accept = "text/xml";
           webRequest.Method = "POST";
           webRequest.CookieContainer = FormsBasedAuthentication.CookieJar;
           return webRequest;
       }
       #endregion

       public IObservable<XDocument> CallService<T>(string url, string action, string soapContent) {
           soapEnvelopeXml = CreateSoapEnvelope(soapContent);
           HttpWebRequest webRequest = CreateWebRequest(url, action);
           var call = (from request in Observable.FromAsyncPattern<Stream>(webRequest.BeginGetRequestStream, webRequest.EndGetRequestStream)().Do(stream => {
                                                                                                                                                       soapEnvelopeXml.Save(stream);
                                                                                                                                                       stream.Close();
                                                                                                                                                   })
                       from response in Observable.FromAsyncPattern<WebResponse>(webRequest.BeginGetResponse, webRequest.EndGetResponse)()
                       select Handle(response));
           return call;
       }
       private XDocument Handle(WebResponse response){
           var doc = XDocument.Load(response.GetResponseStream());
           return doc;
       }
   }

This produces an observable which will publish a XDocument with the result of the web service call. As you can see, we are using our cookie jar from the FBA class (see the CreateWebRequest method) so that our authentication token gets passed along.

To put this together and do an authenticated call to the Lists.asmx web service in SharePoint, we need two small methods.

private static void GetList(string authStatus){
            if (authStatus == "NoError"){
                BaseWebService webService = new BaseWebService();
                webService.CallService<XDocument>(“<url>/_vti_bin/Lists.asmx”, ActionGetList, string.Format(SoapGetList, "Pages")).Subscribe(Handle);
            }
        }

This method gets the published value from the FBA class and checks that we can proceed with our next call.

Oh, must not forget the SoapAction and content of the envelope:

private const string ActionGetList = "http://schemas.microsoft.com/sharepoint/soap/GetList";
private const string SoapGetList = @"<GetList xmlns='http://schemas.microsoft.com/sharepoint/soap/' xmlns:i='http://www.w3.org/2001/XMLSchema-instance'><listName>{0}</listName></GetList>";

 

private static void Handle(XDocument doc){
            Console.WriteLine(doc.ToString());
        }

And this one handles the result from the call to Lists.asmx (not doing much at this point).

So, to start it all off, we need to get our FBA observable set up:

var auth = FormsBasedAuthentication.Authenticate("<url to sharepoint site>", "<user>", "<password>");

Next we start it off:

auth.Do(GetList).Start();

What this does is that our FBA observable will call GetList for every published value that the FBA observable gives us (which should be just one) and to fire the entire process off we turn the ignition and .Start() Smile

This is all then done asynchronously.

SharePhone continues on

Just wanted to inform that I’ll be adding Forms Based Authentication as well as Basic Authentication modes as options in SharePhone so that the library can be used for those that has these kinds of authentication methods configured for their SharePoint sites.

tp_WebPartTypeID de-mystified

When fiddling around with my tool to change assembly references in the content database, I needed to make use of the tp_WebPartTypeID field.

This field is actually a MD5 hash of the assembly fullname and the class fullname separated by a pipe (|).

SharePoint uses this to look up which webpart a page uses and so if you wish to switch out one webpart for another, you switch out this MD5 hash for your new webpart.

There is no way of doing a reverse-lookup using only the MD5 hash, what I did in my tool was to load in all the assemblies thats in use and then generating type ID’s for all the webparts then matching it up with the database values.

Here’s a quick method for generating this ID:

 public static Guid GenerateTypeId(string assembly, Type type) {
            string data = string.Format("{0}|{1}", assembly, type.FullName);
            byte[] dataBytes = Encoding.Unicode.GetBytes(data);
 
            MD5 hashAlgorithm = new MD5CryptoServiceProvider();
            byte[] hash = hashAlgorithm.ComputeHash(dataBytes);
            Guid hashGuid = new Guid(hash);
            return hashGuid;
        }

How to read webpart properties from the database

This is a brief post explaining how you could go about reading the tp_AllUsersProperties and the tp_PerUserProperties fields in the SharePoint AllWebParts database table.

There are two methods used to serialize this data:

  • Using ObjectStateFormatter
  • Compressed XML

How you determine which of the two is being used is simple: check the two first bytes – if they are 1 and 5, you’re dealing with compressed XML.
These numbers represent major and minor version of the serialization type used (as far as I know).

Dealing with standard binary serialized properties, you can do something like this:

                ObjectStateFormatter formatter = new ObjectStateFormatter();
                MemoryStream inputStream = new MemoryStream(data);
                while ((inputStream.Position < inputStream.Length) && (data[(int)((IntPtr)inputStream.Position)] != 0xff)) {
                    inputStream.ReadByte();
                }
                return (object[])formatter.Deserialize(inputStream);

Saving this back is just a matter of calling the Serialize(..) method instead and updating the database.
The deserialized object is an array of property names and values. Most of the property names will be integers as SharePoint saves some space by assigning static property names to integer values (such as Description is 2, Title is 4 etc). You can find these static strings in the WebPartNameTable class in Microsoft.SharePoint.dll. 

As far as compressed XML is concerned, you’ll need to implement a XmlReader that handles the compression. The easiest here is to just open up Reflector and take a look at Microsoft.SharePoint.dll and its CompressedXmlReader class.

This is not a 100% verified approach. Using binary serialization is pretty straightforward and works fine when reading and writing to the database, but the compressed XML can be a bit more tricky.
So far, I’ve only made use of the binary serialization when it comes to updating the database as that covered my needs. Re-compressing the XML and serializing that back to the database is not something I’ve looked into yet.

Issues with RichHtmlField and upgrading

Well, its not really an issue if the site column is defined properly. I ran into a case with a customer where they wanted to upgrade their SharePoint 2007 solution to 2010 and during this upgrade, we found that RichHtmlField controls only displayed plain text instead of rendered HTML for custom site columns that had Type=”HTML”.

The problem was that these site columns were defined without specifying the RichText=”True” or RichTextMode=”FullHtml” attributes originally.
Adding these values turned horrible HTML into pretty text and all was well with the world.

Upgrading webparts with new codebase

While upgrading a SharePoint 2007 site to SharePoint 2010, we decided we’d take the opportunity to clean up our project (organize the namespaces better, cut down on number of projects in the solution, etc). Being that its a internal test project, the risk was rather low. The problem with doing this is that SharePoint keeps references of assembly fullnames and types in the content database so that if you change the name of your dll’s and/or namespaces in your code, webparts will stop working on existing content.

Assembly redirects will only get you so far and only work if the assembly name stays the same.
In our case, we had OldAssembly.One.dll, OldAssembly.Two.dll, etc which got merged into NewCleanAssembly.dll.

So I wrote a utility to remap assemblies in the content database. Basically it allows you to load in old assemblies, view which of the classes are referenced by SharePoint and remap them to new assemblies.
Also, it will go into the properties of all webpart instances and remap old values to corresponding new ones in the new assembly.

It does this by generating new type IDs for the new assemblies and classes and deserializing the webpart properties, serializing it back after fixing the values and then write the whole thing back to the database.
The three fields that are updated are: tp_WebPartTypeID, tp_AllUsersProperties and tp_PerUserProperties in the AllWebParts table.

(The remaps in the above screenshot is for demonstration only, obviously we didn’t remap SharePoint’s own assemblies)

Now, this is highly unsupported and I wouldn’t recommend you do it in a production environment. Microsoft does not support any modifications done directly in the SharePoint database as far as I know.

… but it worked perfectly 🙂

I might post the tool on codeplex if anyone is interested.

User profiles and search in SharePhone

So, version 1.0.3 is out and contains a couple of new features.

User profiles

ClientContext now offers two methods for dealing with user profiles, GetUserProfile and UpdateUserProfile.

Here’s a quick example of their usage:

 

ctx.GetUserProfile(@"sharepointdev\testuser", (UserProfile profile) => {
    profile.Title = "Test user";
    ctx.UpdateUserProfile(profile, null);
});

The UserProfile class only contains a couple of quick access properties so far, to access all of them you need to access the Properties property and read your values there. Here’s an example how:

from property in profile.Properties where property.Name == "PreferredName" select property

You can also use a simple utility method when you want to update a property:

profile.UpdateSingleValueProperty("PreferredName", "Santa");

Keep in mind that updating a user profile is subject to access restrictions set by the site administrator (some properties can only be updated by administrator, others by the owner of the profile).

Search

Again, the ClientContext class has been expanded to include some search functionality. This is the built-in SharePoint search (not sure how this will work with FAST in SharePoint 2010).

ctx.SearchProvider.KeywordSearch("Test", (SearchResult result) => {
    int numberOfHits = result.TotalAvailable;
    //access the hits via result.Results
});

You can also do advanced search via the Search(..) method and the QueryPacket class.

QueryPacket query = new QueryPacket();
query.EnableStemming = true;
query.QueryType = QueryType.MSSQLFT;
query.Query = "select * from Scope() where FREETEXT('test')";
ctx.SearchProvider.Search(query, (SearchResult result) => { });

Reading/writing list items using SharePhone

Ok, so, a quick example how to load in all items in a list and update a property, then save it back to SharePoint:

list.Items.ItemsLoaded += (object se, EventArgs ev) => {
    BaseItem item = list.Items[0];
    item.Title = "NewTitle";
    item.Update();
    list.Items.Update((object obj, EventArgs ea) => {
        
    });
};
list.Items.LoadAllItems();

There’s a few things going on here. As this works asynchronously, we add a event handler which will run when the items have been loaded into the list then we take the first item and updates the title before saving it back.

Whenever updating an item, be sure to call Update() on the item afterwards to tag it for updating. This is yet again a way to ensure that only required data is transferred, hopefully making the payload smaller. When all updates are ready, call the Update() method on the item collection (list.Items.Update(.. )).

The latter update method requires a callback delegate which will be called when the update is complete – this may change in future releases.

Remember, these properties can be databound to your WP7 controls.

If you don’t want to load the entire item collection, you can specify which items you want loaded using a CAML query.

CamlExpression titleQuery = ExpressionFactory.CreateExpression("Title", SharePhone.Query.Expression.FieldType.Text, SharePhone.Query.Expression.Operator.Eq, "NewTitle");
list.GetItemsByQuery(titleQuery, 0, (BaseItemList<BaseItem> items) => {
    if (items.Count > 0) {
        //query successfull
    }
});

The CAML builder classes in SharePhone lets you build queries rather easily, like for example using JOINs:

CamlExpression titleQuery = ExpressionFactory.CreateExpression("Title", SharePhone.Query.Expression.FieldType.Text, SharePhone.Query.Expression.Operator.Eq, "NewTitle");
CamlExpression dateFilter = ExpressionFactory.CreateExpression("PublishedDate", SharePhone.Query.Expression.FieldType.DateTimeWithTimeIncluded, SharePhone.Query.Expression.Operator.Geq, DateTime.Now);
CamlExpression query = ExpressionFactory.JoinExpression(SharePhone.Query.Expression.Conjunction.And, new CamlExpression[] { titleQuery, dateFilter });
list.GetItemsByQuery(query, 0, (BaseItemList<BaseItem> items) => {
    if (items.Count > 0) {
        //query successfull
    }
});

Strong typed list item classes in SharePhone

Usually when dealing with SharePoint through its object model, you get a SPListItem object which can be annoying to work with, especially when you want to databind in Silverlight. This is why I went for a strong typed version in SharePhone and hopefully it will be easier to use for non-SharePoint developers or those just starting out with SharePoint.

A SharePhone list item starts with the BaseItem class. When creating your own classes, you need to extend BaseItem. This class contains basic list item properties which are standard for all SharePoint list items (such as Title, Name, Created date, modified date, author, etc). If you only want to work with these basic properties, then you can use BaseItem as is and not implement your own class.

Let’s say you’ve created a content type which contains the text field MyField.

Create a new class, let’s call it MyContentType:

public class MyContentType : BaseItem{

}

Then we add our MyField property:

public class MyContentType : BaseItem{
    public string MyField { get; set; }
}

However, for SharePhone to know how to map this property to fields in your SharePoint list, you need to tag the property with some information.

public class MyContentType : BaseItem{
    [CoreField(FieldName="MyFieldName", Importance=FieldImportance.Optional, PopulateRule=CorePopulateRule.Sharepoint)]
    public string MyField { get; set; }
}

The CoreField attribute tells the SharePhone library a few key pieces of information:

  • FieldName specifies the internal name of the field in SharePoint
  • Importance is either Optional or Required, specifying whether or not the field requires a value or not.
  • PopulateRule states how the property should be treated

    SharePoint says the value of the property comes from SharePoint and allows for saving the value back.

    SharePointReadOnly says the value should be read from SharePoint but never written back.

    None states that this property should be ignored when reading/writing to SharePoint.

As BaseItem implements INotifyPropertyChanged, you can also support this in your custom properties like so:

private string myField;
[CoreField(FieldName = "MyFieldName", Importance = FieldImportance.Optional, PopulateRule = CorePopulateRule.Sharepoint)]
public string MyField {
    get {
        return myField;
    }
    set {
        myField = value;
        OnPropertyChanged("MyField");
    }
}

(I had a transparent way of dealing with this before so that custom properties would automagically support this, but due to WP7 lacking the System.Reflection.Emit namespace, I had to remove this feature – hopefully I can add it back in at some point).

Now, to use our new class, we make use of our generic methods in SharePhone:

web.Lists.GetListByName<MyContentType>("TestList", (SPList<MyContentType> list) => {
    string myText = list.Items[0].MyField; 
});

Remember, to get the items you first need to load them into the collection using list.Items.LoadAllItems();

I was going to show an example of reading and writing data from/to SharePoint but I’ll save that for the next post so this one doesn’t turn into a lengthy read.