<CharlieDigital/> Programming, Politics, and uhh…pineapples


Invoking Custom WCF Services in SharePoint with Claims

Posted by Charles Chen

In SharePoint, if you host a custom WCF service in a claims-enabled application, the authentication via NTLM is actually quite tricky if you are attempting to invoke it from a console application, for example.

There are various articles and Stackoverflow entries on using System.ServiceModel.Description.ClientCredentials on either the ChannelFactory or the client instance, but all of these did not work in the sense that on the server side, SPContext.Current.Web.CurrentUser was null and ServiceSecurityContext.Current.IsAnonymous returned true.

It seems like it should be possible to invoke the service authenticating through NTLM as if the user were accessing it through the web site.

In fact, it is possible, but it involves some manual HTTP requests to get this to work without doing some Windows Identity Foundation programming and consequently setting up tons of infrastructure to get what seems like a relatively simple and straightforward scenario to work.

The first step is to actually manually retrieve the FedAuth token:

/// <summary>
///     Gets a claims based authentication token by logging in through the NTLM endpoint.
/// </summary>
/// <returns>The FedAuth token required to connect and authenticate the session.</returns>
private string GetAuthToken()
    string authToken = string.Empty;

    CredentialCache credentialCache = new CredentialCache();
    credentialCache.Add(new Uri(_portalBaseUrl), "NTLM", new NetworkCredential(_username, _password, _domain));

    HttpWebRequest request = WebRequest.Create(string.Format("{0}/_windows/default.aspx?ReturnUrl=%2f_layouts%2fAuthenticate.aspx%3fSource%3d%252F&Source=%2F ", _portalBaseUrl)) as HttpWebRequest;
    request.Credentials = credentialCache;
    request.AllowAutoRedirect = false;
    request.PreAuthenticate = true;

    // SharePoint doesn't like it if you don't include these (403 Forbidden)?
    request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko";
    request.Accept = "text/html, application/xhtml+xml, */*";

    HttpWebResponse response = request.GetResponse() as HttpWebResponse;

    authToken = response.Headers["Set-Cookie"];

    return authToken;

There are three keys here:

  1. The first is that AllowAutoRedirect must be false or you will get an error that you are getting too many redirects.  What seems to happen is that the cookies are not set correctly when using auto redirect so the chain will continue until an exception is thrown.  In Fiddler, you will see this as a long cycle of requests and redirects.
  2. The second is that the URL must be the NTLM authentication endpoint (/_windows...") as any other URL will return a 302 and for that, you will need to set AllowAutoRedirect to true.
  3. The third is that it seems as if SharePoint really doesn't like it when the user agent and accept headers are not included.  I tried it later without these and it seemed to work, but I could not get it to work without them (403 unauthorized) initially.

Once you have the FedAuth token, you are able to basically impersonate the user.  To do so, you will need to include a cookie in your HTTP header request:

// Get the FedAuth cookie
var authToken = GetAuthToken();

// Create the connection artifacts.            
EndpointAddress endpointAddress = new EndpointAddress(endpointUrl);
BasicHttpBinding binding = new BasicHttpBinding();            

ChannelFactory<ISomeService> channelFactory = 
    new ChannelFactory<ISomeService>(binding, endpointAddress);

// Initiate the client proxy using the connection and binding information.
ISomeService client = channelFactory.CreateChannel();

using (new OperationContextScope((IContextChannel) client))
    // Set the authentication cookie on the outgoing WCF request.
    WebOperationContext.Current.OutgoingRequest.Headers.Add("Cookie", authToken);


The key is to add the header on the outgoing request before making your service API calls.

With this, you should see that you are able to invoke SharePoint hosted custom WCF service calls in claims-based web applications with NTLM authentication.

Filed under: .Net, SharePoint, WCF No Comments

SharePoint 2010 Service Applications Thoughts

Posted by Charles Chen

I can't say that I can add much more to the tutorials and articles already out there on the Net on this topic, but I'd like to point to the definitive example of how to implement one: http://msdn.microsoft.com/en-us/library/gg193964.aspx

David Taylor's blog also has a good 10 part walkthrough with a bit more explanation; however, I would caution that the example he provides is broken as heck and was part of the problem that it took me a good part of 3-4 days to get something so simple to work right.

One bit of knowledge that I'd like to impart is that you will -- more than likely, 99.99% guaranteed -- run into errors with WCF.  In these cases, the errors will:

  • Not show up in any level of detail in the SharePoint logs
  • Not show up at all in the system event logs
  • Not show up in the IIS logs with any level of detail

You'll want to use SvcConfigEditor to enable tracing and of course, SvcTraceViewer to view your logs.  Part of my problem was that I had an error in the configuration file for the service and part of the problem was that I had an error in the VirtualPath of the service in my service application.  I was only able to track these two down through the usage of the aforementioned WCF utilities so keep them handy!

Filed under: Dev, SharePoint, WCF No Comments

WCF Data Services Cheat Sheet

Posted by Charles Chen

I was working with some SharePoint REST queries and couldn't find a good list of the operations supported by the REST API.  Namely, I was trying to figure out if it supported a "contains" operation (it does using indexof).

I found a very nicely put together set of tables which shows the LINQ queries, the corresponding URL, and the SQL produced by the query.

This is the result of my tiny research about translating LINQ to REST into oData URI Conventions and finally into T-SQL. Event though I knew what I needed on the T-SQL side, I was not sure which LINQ statement would create the respective SQL query. So, as an experimental test bed I created a little console application and used Netflix oData Service to see how LINQ to REST gets translated to URIs

Very useful indeed.

On a related note, for those that are working with the REST APIs for SharePoint, I've found that you can actually reduce your network traffic when using $expand by taking only the fields that you're interested in for the expanded item as well. For example:


The downside is that this will barf out invalid JSON if you have a null value for the expanded field (in this case, if the task is not assigned to any user).  Here's a sample output:

            "Title":"Action 1",
            "Path":"/PWA/ts1/Lists/Tasks/Milestone 1",
            "PriorityValue":"(2) Normal",
            "StatusValue":"Not Started",
               "Name":"Charles Chen",
                  "value":"An error occurred while processing this request."

You can see that the second item in the "results" array is where the task was not assigned to a user.  The response is abruptly cut off with the JSON being completely broken at that point and in an invalid state.  I would have expected that the result would just set "AssignedTo":null

A bummer, but it doesn't detract from my love for the REST APIs 😀

Filed under: Awesome, SharePoint, WCF 1 Comment

Best Search Hit?

Posted by Charles Chen

Might be one of the best ones in the four years that I've had this blog.  I feel your pain dude, but I don't know the answer...sorry bro.

Filed under: lulz, WCF No Comments

WCF Load Balancing : An End To End Example For NetTcpBinding

Posted by Charles Chen

Recently, I worked on prototyping WCF load balancing for our product, FirstPoint. I built a simple example to test the configuration and behavior of load balancing. Since there aren't many end-to-end examples of WCF load balancing on the web, I'm hoping this will be useful (since the Microsoft documentation on this is basically non-existent...well, for NetTcpBindings at least).

I will assume that you are already familiar with network load balancing and configuring the NLB on Windows Server 2003.

Here's a screenshot of how I've configured my network load balancer:

You'll notice that I've configured a specific port range for my service. Also, take note of the cluster address: The two servers that make up the cluster are FPDEV1 ( and FPDEV2 (

I've defined a simple service interface which simply echoes the message with a server name:

using System.ServiceModel;

namespace WcfLoadBalancingSample.Server.Contracts {
    /// <summary>
    /// A simple service contract definition which defines an echo operation.
    /// </summary>
    [ServiceContract(SessionMode = SessionMode.Allowed)]
    public interface IEchoService {

        /// <summary>
        /// Echoes the specified message.
        /// </summary>
        /// <param name="message">The message.</param>
        /// <returns>The input message with a server identifier attached.</returns>
        string Echo(string message);

And here is the implementation:

using System;
using WcfLoadBalancingSample.Server.Contracts;

namespace WcfLoadBalancingSample.Server.Services {
    /// <summary>
    /// Implements the <c>IEchoService</c>.
    /// </summary>
    public class EchoService : IEchoService {
        #region IEchoService Members

        /// <summary>
        /// Echoes the specified message.
        /// </summary>
        /// <param name="message">The message.</param>
        /// <returns>
        /// The input message with a server identifier attached.
        /// </returns>
        public string Echo(string message) {
            return string.Format("[{0}] {1}", Environment.MachineName, message);


As you can see, I've simply prepended the machine name to each response so that we can track which server we are connecting to. Here is the main program which hosts the service:

using System;
using System.ServiceModel;
using WcfLoadBalancingSample.Server.Services;

namespace WcfLoadBalancingSample.Server {
    internal class Program {
        private static void Main(string[] args) {
            var program = new Program();

        public void Run() {           
            ServiceHost serviceHost = null;

            try {
                serviceHost = new ServiceHost(typeof (EchoService));

                Console.Out.WriteLine("Service ready.");
                Console.Out.WriteLine("Press any key to terminate.");
            catch (Exception exception) {
            finally {
                if (serviceHost != null) {

This should be deployed on your each of your two (or more) servers in your cluster. The magic comes next in the configuration file:

<?xml version="1.0" encoding="utf-8" ?>
            <service name="WcfLoadBalancingSample.Server.Services.EchoService"
                <!-- Service Endpoints -->
                    The configured endpoint address is the IP address of the load balanced
                    cluster.  The port number has been mapped to the cluster.
                    address ="net.tcp://"
                A simple custom binding.  Note the leaseTimeout setting.
                This is referenced in the SDK documentation, but not described
                in any way as to how you're supposed to configure it.
                <binding name="DefaultCustomBinding">
                    <windowsStreamSecurity protectionLevel="None"/>
                        <connectionPoolSettings leaseTimeout="00:00:01"/>
                <behavior name="WcfLoadBalancingSample.Server.Services.EchoServiceBehavior">
                    <serviceMetadata httpGetEnabled="True"
                    <serviceDebug includeExceptionDetailInFaults="True" />

The most important part of this configuration file is the setting for the lease timeout. The SDK documentation references this setting, but does not go into detail on how it's configured. Note that it seems that you can only set the least timeout value in configuration using a custom binding.

There is one particularly important thing to note here: I've set the timeout to 1 second for demonstration purposes only. The SDK documentation mentions that this value should be set to 30 seconds. As we'll see later, I have the client creating a new connection every 2.5 seconds (on purpose so we can test connectivity). What I found was that with the default setting (5 minutes, according to the SDK), if you initially connect to server A and server A goes down, it will not automatically connect to server B. I assume this is because of the lease timeout value (it throws an exception).

Also note that the endpoint is configured using the IP address of the node balancing cluster and not the individual servers.

The client side of it is pretty straight forward as well:

using System;
using System.Runtime.Remoting.Messaging;
using System.Threading;
using WcfLoadBalancingSample.Client.LoadBalancedService;

namespace WcfLoadBalancingSample.Client {
    public delegate void EchoDelegate();

    public class EchoEventArgs : EventArgs {
        public EchoEventArgs(string message) {
            Message = message;

        public string Message { get; set; }

    internal class Program {
        private static void Main(string[] args) {
            var program = new Program();

        public void Run() {
            // Start the server.
            Console.Out.WriteLine("Starting client...(press any key to exit)");

            // Start second thread to ping the server.
            EchoDelegate echoDelegate = EchoAsync;
            echoDelegate.BeginInvoke(EchoAsycCompleted, null);


        /// <summary>
        /// Asynchronous execution of the call to the server.
        /// </summary>
        private void EchoAsync() {
            try {
                int count = 0;
                while (true) {
                    using (var serviceClient = new EchoServiceClient()) {
                        // Call the echo service.
                        string result = serviceClient.Echo(



            catch (Exception exception) {

        /// <summary>
        /// Handles the completion of the thread.
        /// </summary>
        /// <param name="result">The result.</param>
        private void EchoAsycCompleted(IAsyncResult result) {
            var r = (AsyncResult) result;
            var e = (EchoDelegate) r.AsyncDelegate;


The only thing to note is that I used a delegate to spin off a second thread to make the call to the server. Adding a reference to the server generates the following configuration file:

<?xml version="1.0" encoding="utf-8" ?>
                <binding name="CustomBinding_IEchoService">
                    <security mode="Transport">
                        <transport clientCredentialType="Windows" 
                                   protectionLevel="None" />
                        <message clientCredentialType="Windows" />
                    <userPrincipalName value="Administrator@fpdev.com" />
                    <servicePrincipalName value="" />
                    <dns value="" />

Note that on this side, the binding is generated as a netTcpBinding.

And that's it! Deploy the service to your clustered servers and start the services and then start the client. If your cluster has a bias towards one server, you will see that the messages will come from that server. To test that it actually is load balancing, simply stop the service on that server and you should see that the output will be coming from the other server. Here's an example:

You can see that after I stop the service on FPDEV1 at the third echo message, it flips over to FPDEV2! Awesome!

Here's the full project (VS2008): WcfLoadBalancingSample.zip (16.89 KB)

Filed under: .Net, WCF 3 Comments

WCF Win32Exception / AuthenticationException

Posted by Charles Chen

I recently came across a weird error with a parnter that we're working with. He was getting SSPI errors at the the client side. I had seen these previously, but only when configuring SharePoint and our product (FirstPoint) for Kerberos.

After turning on Kerberos logging, it was clear that was clear that the client and service were trying to use Kerberos and failing with the error KRB_ERR_RESPONSE_TOO_BIG. Now it's normal for the client and service to negotiate the authentication scheme and fall back onto NTLM, but it seemed that in this case, WCF was not falling back to NTLM due to the error type.

It was pretty baffling since we did have any issues using the same configuration in our test environments.

Fortunately, I came across a listing about this issue. The solution is to simply add an invalid SPN to the service and client, which forces the usage of NTLM.

<endpoint address="net.tcp://myserver/myservice"
	binding="netTcpBinding" bindingConfiguration="DefaultTcpBinding"
		<!--/// The value is left intentionally blank ///--->
		<servicePrincipalName value=""/>

Lesson learned!

Filed under: WCF No Comments

Fatal Execution Engine Failure(0x7927e03e) and WCF

Posted by Charles Chen

An interesting issue cropped up during development this week.  It centered around a mysterious error that was happening in our web applications and causing IIS to crash...hard. 

Peeking in the event log, I found the following error:

Event Type:	Error
Event Source: .NET Runtime
Event Category: None
Event ID: 1023
Date: 8/8/2008
Time: 7:07:28 PM
User: N/A
Computer: ZORCH6
.NET Runtime version 2.0.50727.1433 - Fatal Execution Engine Error (79FFEE24) (80131506)

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

It was mind boggling even more so because the app worked on some environments and not others.  We chased various options around and none of them seemed to solve the issue.  Further confounding the issue was that IE would throw up an authentication dialog instead of outright generating an error on the AJAX call where the error was originating from.  This lead us in circles fiddling with the domain controller and permissions for a while, too.

There was no concrete info anywhere in the system which explained why we were getting this error.  Then I came across a post in the Microsoft Connect site which mentioned this issue.  I wasn't completely sure that this was the same as our issue, since we weren't using IEnumerable<T> in our contracts, but it was a lead.  So I built a small sample project to model our service calls.

It turns out that there's something borked with WCF data contract serialization in IIS when running in Windows 2003 environments.

In our case, we did not have IEnumerable<T> in our contracts (well, not explicitly anyways).  However, we had a few methods which took Collection<T> as an argument.  This worked all fine and dandy when sending a normal collection like so:

Collection<Person> c1 = new Collection<Person>();

Person thuy = new Person("Thuy", "Ha", "UT");


Collection<Person> c2 = new Collection<Person>();


However, using this approach breaks it (I repeat, this does not work):

Person chuck = new Person("Charles", "Chen", "NJ");
Person[] a3 = new Person[1];
a3[0] = chuck;

client.ProcessPeople(new Collection<Person>(a3));

To work around this, you can use an intermediate step like so (of course, this is only necessary if you have data coming from an array to begin with):

Person chuck = new Person("Charles", "Chen", "NJ");
Person[] a3 = new Person[1];
a3[0] = chuck;

List<Person> l3 = new List<Person>();

client.ProcessPeople(new Collection<Person>(l3));

In any case, what further confounded the issue is that it worked in an XP environment but did not work in a server 2003 environment.  Glad to have this issue behind me :-D; hopefully, you won't waste as much time fiddling with this as I did.

Filed under: .Net, WCF No Comments

Deleting ExtensionData From JavaScript

Posted by Charles Chen

WCF DataContracts include an ExtensionData property which is a bit troublesome if you are trying to send a modified object back up to the service if it has no properties in Javascript.

So I wrote a little piece of Javascript to clean it up:

function DeleteExtensionData(obj) {
var keys = Object.keys(obj);

keys.each(function(key) {
if(!Object.isFunction(obj[key])) {
if(obj[key] instanceof Object) {

if(key == "ExtensionData") {
delete obj[key];

It will recusively delete all ExtensionData properties from the object.  You can call as soon as you get the result from a completed AJAX request or you can call before sending an object parameter to a service.

Note that it uses constructs from prototype.

If you want to get fancy, you can also write a custom serializer.

Filed under: WCF No Comments

IE7, AJAX, And 400 “Bad Request”

Posted by Charles Chen

I spent bout two hours today tring to figure out why I kept getting a weird error while using prototype to call a WCF service.

There are various other resources on the web regarding how WCF has a few quirks with error handling using the default endpoints, however, none of these scenarios were applicable to me since the JavaScript call worked fine in FireFox.  To complicate things even further, it actually all worked fine in IE7 as well with one caveat: it only worked on the first load of the IE7 process.

After that, any refreshing of the page would return 400/"Bad Request" errors, mysteriously.  Just to make sure that it wasn't isolated to prototype, I also tried jQuery as well.  Still nothing.

Well, as it turns out, that setting the AJAX request content type to application/json wasn't enough; I had to set it to application/json; charset=utf-8.

So if you're encountering weird issues with any AJAX library, IE7, and WCF and/or ASP.NET, be sure to check your content type setting.

Update 6/20/2008: bah!  I woke up this morning and it's not working again!  Oddly, it works fine if I have Fiddler running and as soon as I shut off Fiddler, it stops working.  To complicate matters even more, it's only happening on methods where I have post content.

Update 6/22/2008: after further investigation, I'm concluding that it's related to SharePoint.  To isolate the different components, I created a custom WCF service and a simple runtime which models our current custom WCF runtime.  When I used a plain HTML page or even an ASPX page with a script manager on it, prototype AJAX requests worked fine.  Okay, so it wasn't how we were hosting the service and it wasn't an issue with a collision between ASP.NET AJAX and prototype.  I then hooked a SharePoint layout page up to the same service and boom, it all breaks :-S

I've broken it out into the following scenarios:

  1. Using FireFox, the requests always work.  No matter how many times I refresh the page, the AJAX request always works fine.
  2. Using IE7, it will always work on the first load of the IE process.
    1. If I hit CTRL+F5, it will continue to work.
    2. If at any point, I hit F5, it will fail and even CTRL+F5 will not fix it.
    3. If I have Fiddler running, it will always work, even if it entered a failure state after hitting F5! (So Fiddler must be doing something to the HTTP message??). As far as I can tell, the only difference between the Fiddler request and the native request, from viewing the WireShark trace, is the "Authorization" header and the fact that the failed request has a "Content-length" header value of 0 (manually setting the header doesn't work either).  I verified that prototype was not somehow mangling the POST body content by writing a trace of the XHR request body right before and right after the request is sent (both came out okay).

One of the really weird things about this error is that it cannot be observed when I have Fiddler running; somehow, Fiddler "fixes" the issue.  I had to hook up WireShark on the server and watch the raw TCP messages to finally see that on the unsuccessful attempts, the POST content to my WCF service was empty.  The method required one parameter, so the WCF serializer threw a formatter excpetion when it received an empty message:

The server encountered an error processing the request. The exception message is 'Error in deserializing body of request message for operation 'GetRoutes'. The OperationFormatter could not deserialize any information from the Message because the Message is empty (IsEmpty = true).'. See server logs for more details. The exception stack trace is:at System.ServiceModel.Dispatcher.PrimitiveOperationFormatter.DeserializeRequest(Message message, Object[] parameters) at System.ServiceModel.Dispatcher.DemultiplexingDispatchMessageFormatter.DeserializeRequest(Message message, Object[] parameters) at System.ServiceModel.Dispatcher.UriTemplateDispatchFormatter.DeserializeRequest(Message message, Object[] parameters) at System.ServiceModel.Dispatcher.CompositeDispatchFormatter.DeserializeRequest(Message message, Object[] parameters) at System.ServiceModel.Dispatcher.DispatchOperationRuntime.DeserializeInputs(MessageRpc& rpc) at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc) at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc) at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)

I've tried playing around with various cache avoidance strategies to no avail (setting the "Cache-Control" header, setting the "If-Modified-Since" header, setting the "Pragma" header, addin randomness to the POST URL) so for now, I'm just going to have to give up on this.  I've confirme the error using three different OS environments (XP SP2 (desktop), XP64 SP2 (dev), and 2003 R2 SP2 (VM)), all running IE 7.0.5730.11.

I know there are various articles out there regarding IE issues with XHR and how the ordering of the open(), onreadystatechange, and send() operations must be called; however, I verified in the prototype source (lines 1213-1223) that the correct order of operations are being performed.

Thinking that perhaps some Microsoft mojo was behind all of this, I gave the Sys.Net.WebRequest a shot.  This also did not yield any postive results.  Oddly, ASP.NET AJAX to web service calls all work fine (we had some existing AJAX components which were calling to a web service, but the goal was to transition away from writing these "empty" proxy services and call directly to the WCF service).

If any SharePoint development team members or program managers are reading this, please look into it!  I'm not sure if it affects IIS hosted WCF services at this point, but a sample project can be downloaded from here: WcfAjaxWonkiness.zip.  Create a layout page which calls the service and add it to SharePoint and observe it blowing up.

Filed under: Dev, SharePoint, WCF 3 Comments

WCF, Prototype, And Awesome-sauce

Posted by Charles Chen

One of the most awesome features of WCF in .NET 3.5 is support for REST-style AJAX requests to the services.

Rick Strahl has an excellent post on how to get it up and running with jQuery.

Just some notes on working with it myself which I hope can help clarify some some issues:

  • When using POST and the postBody option with Ajax.Request, the format must be {"argument-name":"value"}.  I struggled with this for a while using single quotes and no quotes.  For example: {arg:'hello'}, {'arg':'hello'}, {arg:"hello"} will all result in 400 "Bad Request" errors.
  • The contentType parameter must be set to "application/json".
  • Be sure to decorate the service contract using the WebInvokeAttribute with ResponseFormat and RequestFormat set to WebMessageFormat.Json.

You can download a sample project from here: WcfAjaxSample.zip (3.83 KB)

Filed under: .Net, WCF No Comments