Rewriting FullTextQuery within CoreResultsWebPart

Recently I found that SharePoint advanced search used query parts like Title like ‘%querystring%’ when dealing with contains operators. When dealing with documents that had long titles this didn’t yield the expected search results and with further testing I found that using CONTAINS(Title,’”querystring”’) gave me all the results I wanted.

Problem was that the advanced search was out-of-the-box and no way to configure this (as far as I know) so I needed a way to rewrite the FullTextQuery that was being generated by SharePoint without too much custom code and changing the behavior of the CoreResultsWebPart.

Enter: Reflection. I love this tool, but it should be used sparingly in large solutions as there is a performance hit.

Anyway, basically what I did was extend the CoreResultsWebPart and override the SetPropertiesOnHiddenObject method. This method initializes a private field within CoreResultsWebPart of the type SearchResultHiddenObject and this class contains the FullTextQuery propery which we want to modify at runtime.

So, here’s the quick and dirty code (should be cleaned up before production use):

private string queryPattern = @"(?<Title>Title[\w\s]+[^%]%(?<Query>[\w\s\d]+[^%]*)%')";

protected override void SetPropertiesOnHiddenObject() { base.SetPropertiesOnHiddenObject(); try { Type t = this.GetType(); FieldInfo hiddenObjectField = t.BaseType.GetField("srho", BindingFlags.NonPublic | BindingFlags.Instance); if (hiddenObjectField != null) { object hiddenObject = hiddenObjectField.GetValue(this); Type hiddenObjectType = hiddenObject.GetType(); PropertyInfo fullTextQuery = hiddenObjectType.GetProperty("FullTextQuery", BindingFlags.Public | BindingFlags.Instance); string query = fullTextQuery.GetValue(hiddenObject, null) as string; if (query != null) { Match m = Regex.Match(query, queryPattern); if (m.Success) { string queryString = m.Groups["Query"].Value; query = Regex.Replace(query, queryPattern, "CONTAINS(Title,'\"$2\"')", RegexOptions.IgnoreCase); fullTextQuery.SetValue(hiddenObject, query, null); } } } } catch (Exception ex) { HttpContext.Current.Response.Write("Error: " + ex); } }

‘As you can see I use a regular expression to match and replace the query, but you can replace this with however you wish to modify the query.

Replace the out-of-the-box CoreResultsWebPart with this webpart and you’ll retain all the normal functionality except you gain a little more power over the actual query that is used. This works perfectly on SharePoint 2007; I haven’t tested it on SharePoint 2010 and it might not be needed there.


SharePoint FBA & Reactive Extensions

Some of the dangers with new toys is to overuse them, I fear I might be doing that with reactive extensions but oh well.. It’s all about the learningSmile

Today I did a test to do Form Based Authentication with SharePoint using reactive extensions. This is the first attempt, it can probably be made more streamlined as I figure out more about Rx.

Oh, apologize for the syntax coloring in the code, I really need to find some better CSS for this…

I have two classes: One for doing the FBA (which also holds the cookie jar as we will get a cookie once the FBA is successful and need to pass this along to the next web service call) and one for doing web service calls.

public class FormsBasedAuthentication {
    public static CookieContainer CookieJar = new CookieContainer();
    private const string AuthEnvelope = @"<?xml version=""1.0"" encoding=""utf-8""?>
                <soap:Envelope xmlns:xsi="""" xmlns:xsd="""" xmlns:soap="""">
                    <Login xmlns="""">

    public static IObservable<string> Authenticate(string server, string username, string password) {
        Uri authService = new Uri(string.Format("{0}/_vti_bin/authentication.asmx", server));
        HttpWebRequest authRequest = WebRequest.Create(authService) as HttpWebRequest;
        authRequest.CookieContainer = CookieJar;
        authRequest.Headers["SOAPAction"] = "";
        authRequest.ContentType = "text/xml; charset=utf-8";
        authRequest.Method = "POST";
        return (from request in Observable.FromAsyncPattern<Stream>(authRequest.BeginGetRequestStream, authRequest.EndGetRequestStream)().Do(stream => {
            UTF8Encoding encoding = new UTF8Encoding();
            string envelope = string.Format(AuthEnvelope, username, password);
            byte[] bytes = encoding.GetBytes(envelope);
            stream.Write(bytes, 0, bytes.Length);
                from response in Observable.FromAsyncPattern<WebResponse>(authRequest.BeginGetResponse, authRequest.EndGetResponse)()
                select HandleResponse(response, authRequest));

    private static string HandleResponse(WebResponse response, HttpWebRequest request) {
        var httpResponse = response as HttpWebResponse;
        if (httpResponse != null){
            var result = XDocument.Load(response.GetResponseStream());
            var code = result.Descendants(XName.Get("ErrorCode", "")).FirstOrDefault();
            if (code != null)
                return code.Value;
        return "ErrorOccured";


This will return an observable that will return the authentication result – if the login was successful, the published value will be NoError.

Next we have our web service class:

public class BaseWebService {
       private static string soapEnvelope =

       private XDocument soapEnvelopeXml;
       public string BaseUrl { get; set; }
       #region Helper methods
       private XDocument CreateSoapEnvelope(string content) {
           var envelope = string.Format(soapEnvelope, content);
           XDocument soapEnvelopeXml = XDocument.Parse(envelope);

           return soapEnvelopeXml;

       private HttpWebRequest CreateWebRequest(string url, string action) {
           Uri serviceUrl = new Uri(url, UriKind.Absolute);
           var webRequest = WebRequest.Create(serviceUrl) as HttpWebRequest;
           webRequest.Headers["SOAPAction"] = action;
           webRequest.ContentType = "text/xml;charset=\"utf-8\"";
           webRequest.Accept = "text/xml";
           webRequest.Method = "POST";
           webRequest.CookieContainer = FormsBasedAuthentication.CookieJar;
           return webRequest;

       public IObservable<XDocument> CallService<T>(string url, string action, string soapContent) {
           soapEnvelopeXml = CreateSoapEnvelope(soapContent);
           HttpWebRequest webRequest = CreateWebRequest(url, action);
           var call = (from request in Observable.FromAsyncPattern<Stream>(webRequest.BeginGetRequestStream, webRequest.EndGetRequestStream)().Do(stream => {
                       from response in Observable.FromAsyncPattern<WebResponse>(webRequest.BeginGetResponse, webRequest.EndGetResponse)()
                       select Handle(response));
           return call;
       private XDocument Handle(WebResponse response){
           var doc = XDocument.Load(response.GetResponseStream());
           return doc;

This produces an observable which will publish a XDocument with the result of the web service call. As you can see, we are using our cookie jar from the FBA class (see the CreateWebRequest method) so that our authentication token gets passed along.

To put this together and do an authenticated call to the Lists.asmx web service in SharePoint, we need two small methods.

private static void GetList(string authStatus){
            if (authStatus == "NoError"){
                BaseWebService webService = new BaseWebService();
                webService.CallService<XDocument>(“<url>/_vti_bin/Lists.asmx”, ActionGetList, string.Format(SoapGetList, "Pages")).Subscribe(Handle);

This method gets the published value from the FBA class and checks that we can proceed with our next call.

Oh, must not forget the SoapAction and content of the envelope:

private const string ActionGetList = "";
private const string SoapGetList = @"<GetList xmlns='' xmlns:i=''><listName>{0}</listName></GetList>";


private static void Handle(XDocument doc){

And this one handles the result from the call to Lists.asmx (not doing much at this point).

So, to start it all off, we need to get our FBA observable set up:

var auth = FormsBasedAuthentication.Authenticate("<url to sharepoint site>", "<user>", "<password>");

Next we start it off:


What this does is that our FBA observable will call GetList for every published value that the FBA observable gives us (which should be just one) and to fire the entire process off we turn the ignition and .Start() Smile

This is all then done asynchronously.

SharePhone continues on

Just wanted to inform that I’ll be adding Forms Based Authentication as well as Basic Authentication modes as options in SharePhone so that the library can be used for those that has these kinds of authentication methods configured for their SharePoint sites.

tp_WebPartTypeID de-mystified

When fiddling around with my tool to change assembly references in the content database, I needed to make use of the tp_WebPartTypeID field.

This field is actually a MD5 hash of the assembly fullname and the class fullname separated by a pipe (|).

SharePoint uses this to look up which webpart a page uses and so if you wish to switch out one webpart for another, you switch out this MD5 hash for your new webpart.

There is no way of doing a reverse-lookup using only the MD5 hash, what I did in my tool was to load in all the assemblies thats in use and then generating type ID’s for all the webparts then matching it up with the database values.

Here’s a quick method for generating this ID:

 public static Guid GenerateTypeId(string assembly, Type type) {
            string data = string.Format("{0}|{1}", assembly, type.FullName);
            byte[] dataBytes = Encoding.Unicode.GetBytes(data);
            MD5 hashAlgorithm = new MD5CryptoServiceProvider();
            byte[] hash = hashAlgorithm.ComputeHash(dataBytes);
            Guid hashGuid = new Guid(hash);
            return hashGuid;

How to read webpart properties from the database

This is a brief post explaining how you could go about reading the tp_AllUsersProperties and the tp_PerUserProperties fields in the SharePoint AllWebParts database table.

There are two methods used to serialize this data:

  • Using ObjectStateFormatter
  • Compressed XML

How you determine which of the two is being used is simple: check the two first bytes – if they are 1 and 5, you’re dealing with compressed XML.
These numbers represent major and minor version of the serialization type used (as far as I know).

Dealing with standard binary serialized properties, you can do something like this:

                ObjectStateFormatter formatter = new ObjectStateFormatter();
                MemoryStream inputStream = new MemoryStream(data);
                while ((inputStream.Position < inputStream.Length) && (data[(int)((IntPtr)inputStream.Position)] != 0xff)) {
                return (object[])formatter.Deserialize(inputStream);

Saving this back is just a matter of calling the Serialize(..) method instead and updating the database.
The deserialized object is an array of property names and values. Most of the property names will be integers as SharePoint saves some space by assigning static property names to integer values (such as Description is 2, Title is 4 etc). You can find these static strings in the WebPartNameTable class in Microsoft.SharePoint.dll. 

As far as compressed XML is concerned, you’ll need to implement a XmlReader that handles the compression. The easiest here is to just open up Reflector and take a look at Microsoft.SharePoint.dll and its CompressedXmlReader class.

This is not a 100% verified approach. Using binary serialization is pretty straightforward and works fine when reading and writing to the database, but the compressed XML can be a bit more tricky.
So far, I’ve only made use of the binary serialization when it comes to updating the database as that covered my needs. Re-compressing the XML and serializing that back to the database is not something I’ve looked into yet.

Issues with RichHtmlField and upgrading

Well, its not really an issue if the site column is defined properly. I ran into a case with a customer where they wanted to upgrade their SharePoint 2007 solution to 2010 and during this upgrade, we found that RichHtmlField controls only displayed plain text instead of rendered HTML for custom site columns that had Type=”HTML”.

The problem was that these site columns were defined without specifying the RichText=”True” or RichTextMode=”FullHtml” attributes originally.
Adding these values turned horrible HTML into pretty text and all was well with the world.

Upgrading webparts with new codebase

While upgrading a SharePoint 2007 site to SharePoint 2010, we decided we’d take the opportunity to clean up our project (organize the namespaces better, cut down on number of projects in the solution, etc). Being that its a internal test project, the risk was rather low. The problem with doing this is that SharePoint keeps references of assembly fullnames and types in the content database so that if you change the name of your dll’s and/or namespaces in your code, webparts will stop working on existing content.

Assembly redirects will only get you so far and only work if the assembly name stays the same.
In our case, we had OldAssembly.One.dll, OldAssembly.Two.dll, etc which got merged into NewCleanAssembly.dll.

So I wrote a utility to remap assemblies in the content database. Basically it allows you to load in old assemblies, view which of the classes are referenced by SharePoint and remap them to new assemblies.
Also, it will go into the properties of all webpart instances and remap old values to corresponding new ones in the new assembly.

It does this by generating new type IDs for the new assemblies and classes and deserializing the webpart properties, serializing it back after fixing the values and then write the whole thing back to the database.
The three fields that are updated are: tp_WebPartTypeID, tp_AllUsersProperties and tp_PerUserProperties in the AllWebParts table.

(The remaps in the above screenshot is for demonstration only, obviously we didn’t remap SharePoint’s own assemblies)

Now, this is highly unsupported and I wouldn’t recommend you do it in a production environment. Microsoft does not support any modifications done directly in the SharePoint database as far as I know.

… but it worked perfectly 🙂

I might post the tool on codeplex if anyone is interested.