Rewriting FullTextQuery within CoreResultsWebPart

Recently I found that SharePoint advanced search used query parts like Title like ‘%querystring%’ when dealing with contains operators. When dealing with documents that had long titles this didn’t yield the expected search results and with further testing I found that using CONTAINS(Title,’”querystring”’) gave me all the results I wanted.

Problem was that the advanced search was out-of-the-box and no way to configure this (as far as I know) so I needed a way to rewrite the FullTextQuery that was being generated by SharePoint without too much custom code and changing the behavior of the CoreResultsWebPart.

Enter: Reflection. I love this tool, but it should be used sparingly in large solutions as there is a performance hit.

Anyway, basically what I did was extend the CoreResultsWebPart and override the SetPropertiesOnHiddenObject method. This method initializes a private field within CoreResultsWebPart of the type SearchResultHiddenObject and this class contains the FullTextQuery propery which we want to modify at runtime.

So, here’s the quick and dirty code (should be cleaned up before production use):

private string queryPattern = @"(?<Title>Title[\w\s]+[^%]%(?<Query>[\w\s\d]+[^%]*)%')";

protected override void SetPropertiesOnHiddenObject() { base.SetPropertiesOnHiddenObject(); try { Type t = this.GetType(); FieldInfo hiddenObjectField = t.BaseType.GetField("srho", BindingFlags.NonPublic | BindingFlags.Instance); if (hiddenObjectField != null) { object hiddenObject = hiddenObjectField.GetValue(this); Type hiddenObjectType = hiddenObject.GetType(); PropertyInfo fullTextQuery = hiddenObjectType.GetProperty("FullTextQuery", BindingFlags.Public | BindingFlags.Instance); string query = fullTextQuery.GetValue(hiddenObject, null) as string; if (query != null) { Match m = Regex.Match(query, queryPattern); if (m.Success) { string queryString = m.Groups["Query"].Value; query = Regex.Replace(query, queryPattern, "CONTAINS(Title,'\"$2\"')", RegexOptions.IgnoreCase); fullTextQuery.SetValue(hiddenObject, query, null); } } } } catch (Exception ex) { HttpContext.Current.Response.Write("Error: " + ex); } }

‘As you can see I use a regular expression to match and replace the query, but you can replace this with however you wish to modify the query.

Replace the out-of-the-box CoreResultsWebPart with this webpart and you’ll retain all the normal functionality except you gain a little more power over the actual query that is used. This works perfectly on SharePoint 2007; I haven’t tested it on SharePoint 2010 and it might not be needed there.

tp_WebPartTypeID de-mystified

When fiddling around with my tool to change assembly references in the content database, I needed to make use of the tp_WebPartTypeID field.

This field is actually a MD5 hash of the assembly fullname and the class fullname separated by a pipe (|).

SharePoint uses this to look up which webpart a page uses and so if you wish to switch out one webpart for another, you switch out this MD5 hash for your new webpart.

There is no way of doing a reverse-lookup using only the MD5 hash, what I did in my tool was to load in all the assemblies thats in use and then generating type ID’s for all the webparts then matching it up with the database values.

Here’s a quick method for generating this ID:

 public static Guid GenerateTypeId(string assembly, Type type) {
            string data = string.Format("{0}|{1}", assembly, type.FullName);
            byte[] dataBytes = Encoding.Unicode.GetBytes(data);
            MD5 hashAlgorithm = new MD5CryptoServiceProvider();
            byte[] hash = hashAlgorithm.ComputeHash(dataBytes);
            Guid hashGuid = new Guid(hash);
            return hashGuid;

How to read webpart properties from the database

This is a brief post explaining how you could go about reading the tp_AllUsersProperties and the tp_PerUserProperties fields in the SharePoint AllWebParts database table.

There are two methods used to serialize this data:

  • Using ObjectStateFormatter
  • Compressed XML

How you determine which of the two is being used is simple: check the two first bytes – if they are 1 and 5, you’re dealing with compressed XML.
These numbers represent major and minor version of the serialization type used (as far as I know).

Dealing with standard binary serialized properties, you can do something like this:

                ObjectStateFormatter formatter = new ObjectStateFormatter();
                MemoryStream inputStream = new MemoryStream(data);
                while ((inputStream.Position < inputStream.Length) && (data[(int)((IntPtr)inputStream.Position)] != 0xff)) {
                return (object[])formatter.Deserialize(inputStream);

Saving this back is just a matter of calling the Serialize(..) method instead and updating the database.
The deserialized object is an array of property names and values. Most of the property names will be integers as SharePoint saves some space by assigning static property names to integer values (such as Description is 2, Title is 4 etc). You can find these static strings in the WebPartNameTable class in Microsoft.SharePoint.dll. 

As far as compressed XML is concerned, you’ll need to implement a XmlReader that handles the compression. The easiest here is to just open up Reflector and take a look at Microsoft.SharePoint.dll and its CompressedXmlReader class.

This is not a 100% verified approach. Using binary serialization is pretty straightforward and works fine when reading and writing to the database, but the compressed XML can be a bit more tricky.
So far, I’ve only made use of the binary serialization when it comes to updating the database as that covered my needs. Re-compressing the XML and serializing that back to the database is not something I’ve looked into yet.

Issues with RichHtmlField and upgrading

Well, its not really an issue if the site column is defined properly. I ran into a case with a customer where they wanted to upgrade their SharePoint 2007 solution to 2010 and during this upgrade, we found that RichHtmlField controls only displayed plain text instead of rendered HTML for custom site columns that had Type=”HTML”.

The problem was that these site columns were defined without specifying the RichText=”True” or RichTextMode=”FullHtml” attributes originally.
Adding these values turned horrible HTML into pretty text and all was well with the world.

Upgrading webparts with new codebase

While upgrading a SharePoint 2007 site to SharePoint 2010, we decided we’d take the opportunity to clean up our project (organize the namespaces better, cut down on number of projects in the solution, etc). Being that its a internal test project, the risk was rather low. The problem with doing this is that SharePoint keeps references of assembly fullnames and types in the content database so that if you change the name of your dll’s and/or namespaces in your code, webparts will stop working on existing content.

Assembly redirects will only get you so far and only work if the assembly name stays the same.
In our case, we had OldAssembly.One.dll, OldAssembly.Two.dll, etc which got merged into NewCleanAssembly.dll.

So I wrote a utility to remap assemblies in the content database. Basically it allows you to load in old assemblies, view which of the classes are referenced by SharePoint and remap them to new assemblies.
Also, it will go into the properties of all webpart instances and remap old values to corresponding new ones in the new assembly.

It does this by generating new type IDs for the new assemblies and classes and deserializing the webpart properties, serializing it back after fixing the values and then write the whole thing back to the database.
The three fields that are updated are: tp_WebPartTypeID, tp_AllUsersProperties and tp_PerUserProperties in the AllWebParts table.

(The remaps in the above screenshot is for demonstration only, obviously we didn’t remap SharePoint’s own assemblies)

Now, this is highly unsupported and I wouldn’t recommend you do it in a production environment. Microsoft does not support any modifications done directly in the SharePoint database as far as I know.

… but it worked perfectly 🙂

I might post the tool on codeplex if anyone is interested.

User profiles and search in SharePhone

So, version 1.0.3 is out and contains a couple of new features.

User profiles

ClientContext now offers two methods for dealing with user profiles, GetUserProfile and UpdateUserProfile.

Here’s a quick example of their usage:


ctx.GetUserProfile(@"sharepointdev\testuser", (UserProfile profile) => {
    profile.Title = "Test user";
    ctx.UpdateUserProfile(profile, null);

The UserProfile class only contains a couple of quick access properties so far, to access all of them you need to access the Properties property and read your values there. Here’s an example how:

from property in profile.Properties where property.Name == "PreferredName" select property

You can also use a simple utility method when you want to update a property:

profile.UpdateSingleValueProperty("PreferredName", "Santa");

Keep in mind that updating a user profile is subject to access restrictions set by the site administrator (some properties can only be updated by administrator, others by the owner of the profile).


Again, the ClientContext class has been expanded to include some search functionality. This is the built-in SharePoint search (not sure how this will work with FAST in SharePoint 2010).

ctx.SearchProvider.KeywordSearch("Test", (SearchResult result) => {
    int numberOfHits = result.TotalAvailable;
    //access the hits via result.Results

You can also do advanced search via the Search(..) method and the QueryPacket class.

QueryPacket query = new QueryPacket();
query.EnableStemming = true;
query.QueryType = QueryType.MSSQLFT;
query.Query = "select * from Scope() where FREETEXT('test')";
ctx.SearchProvider.Search(query, (SearchResult result) => { });


Today I released a basic version of SharePhone, my WP7 library for working with SharePoint sites. Binaries and source code can be found here: Codeplex

I’m currently porting this code from a more functional Silverlight project so I apologize if the source is a bit messy.

Anyway, the current release lets you open up any SharePoint 2007/2010 web and its sub webs and work with their lists and list items.

Here’s a short example how:

First we instantiate our context and supply the URL for the root web.

ClientContext ctx = new ClientContext("http://sharepointdev");
//supply credentials if needed
ctx.Settings.Credentials = new CredentialSettings { Domain = "<domain>", UserName = "<user>", Password = "<password>" };

Then we get our root web object (remember, Silverlight requires any web service call to be made asynchronously which is why this is the only method available in this library).

ctx.GetRootWeb((object s, SPWebLoadCompletedEventArgs result) => {
    SPWeb rootWeb = result.Webs[0];               

Next, let’s load all the lists available at the root web:

web.Lists.LoadCompleted += (object sender, SPListLoadCompletedEventArgs e) => {
    Deployment.Current.Dispatcher.BeginInvoke(() => {
        list1.ItemsSource = e.Lists;

list1 is a ListBox and it’s defined like this in XAML:

            <ListBox Name="list1">
                        <TextBlock Text="{Binding Title}" Style="{StaticResource PhoneTextNormalStyle}"/>

And we have our lists neatly listed out on our phone, ready to continue working with them. sample

Most of the operations on collections, such as sub webs on a web or lists or even list items in a list requires you first to load the collection like in the previous example (web.Lists.LoadAll()). The reason why I don’t load this data on first call is because I want to keep the data traffic at a minimum. Basically you load what you need and nothing more.

Next post I’ll show an example how to read list items, update the data and post them back to SharePoint using strong typed custom classes.