<CharlieDigital/> Programming, Politics, and uhh…pineapples


An Alternate Meaning for FOCKED

Posted by Charles Chen

Eric Brechner came up with one of my favorite acronyms of all time in software development: FOCKED.


I want to add an alternate: Failure to Orchestrate Collective Knowledge Effectively for Delivery.

Successful delivery of software requires that different members of the team come together and understand the goals that have to be achieved and the priorities of those goals.

It's as simple as communicating to the team on a regular basis (no more than once a week, but maybe at least once a month):

  • where we are,
  • where do we want to go,
  • when do we have to get there,
  • how are we getting there,
  • who's driving

It can make the whole process of delivery of software much less stressful and maybe more successful simply by aligning all of the stakeholders periodically.

Hey, maybe you learned this in some fancy MBA class or something, but I'm starting to appreciate -- more and more -- that the real secret to successful delivery of software is driving the successful collaboration and communication of people and alignment of all pieces to a strategy, vision, or goal.  Having a bunch of smart, capable people doesn't help you much if no one knows what's going on.

Filed under: DevLife, Rants No Comments

The Importance of Scope (and How To Ship It!)

Posted by Charles Chen

One of the lessons I've been mulling about the past few weeks is the importance of scope when delivering software.

Delivery of software can be thought of as a balancing act between three buckets:

  1. Time - this is the schedule and how much time you have to complete the work.
  2. Money - this includes everything from spending on more resources, better resources, tooling support, and so on
  3. Requirements - this defines what you're building

These are the three basic buckets that constrain the scope of what can be done and they react to each other in different ways.  If there are more requirements or the requirements are inflexible, then it necessitates more time or money or both.  If both requirements and time are inflexible, then more money will be required to achieve the goals within those limits.  If money is constrained (less resources), then you must allow more time to deliver the requirements or trim the requirements.

Having been in consulting and in software development, each project has different priorities on which is more important and these priorities drive the sizing, cost, and pace of the project.

But in software development, I think one thing that I think many folks -- even experienced individuals -- get wrong is the fixation and inflexibility on requirements.  I think that requirements are really much more fluid in software development projects as compared to contract-driven consulting projects.  The reason is simple: in software development, the assumption is that there will always be another version and there will always be another point release; this is exactly what roadmaps are for.

Plan a solid roadmap and if you don't get a particular feature or capability in this release, you and your customers should have a good idea of which release it will be in down the road.

Some tend to lose sight of this and think that all features must be shipped in a constrained timeline.  I think this is a losing proposition that forces a team to compromise on quality by artificially forcing requirements into a release when the reality is that there are often really critical features that must be delivered and there are nice to have features that would be great if they could be delivered.  Teams and leaders without discipline and a focus on the criteria of success will have a difficult time discerning the two and lump nice-to-haves right alongside the critical work items.  This is a recipe for failed projects, missed deadlines, and poor quality.

The reality is that software rarely comes out fully baked unless you're NASA launching a space mission worth billions of dollars and you only get one shot; most teams are not under such otherworldly constraints.  There will always be something that could be better or easier to use or some missing feature discovered along the way or some new idea.  The trick for teams that succeed is being about to create boundaries on what is needed now.

Apple shipped 7 versions of iOS before adding support for third party keyboards and NFC.

NPR's first mobile site was terrible (I don't have a screenshot of it, unfortunately), but at least they shipped it and their audience could use it and now they've evolved it to be a clean, responsive web site.

Here's what the an early version of Facebook looked like:

Courtesy of Wikipedia

Courtesy of Wikipedia

And Amazon:

Back when it was just a bookstore

Microsoft shipped Azure without support for virtual machines until 2013.

But what if instead of new features, we're talking about bugs or design flaw?  There is an important calculus at play here and I think that one way to think about it is like this: if there is a bug or design flaw that is preventing increased growth in userbase or revenue, then that takes precedent over any other bug or design flaw that is in the shipped system.  Think about it this way: if you ship four versions of the software with the bug or design flaw, chances are, you can probably ship a fifth without addressing it (of course, this is not always the case, especially if the flaw is related to security).  But if a bug or design flaw is stopping adoption or holding back revenue, then that flaw automatically becomes the most critical focus for a release.

The point is that in software product development, more often than not, the winning strategy isn't to get it perfect (as much as Steve Jobs would have you believe that he got it perfect each time, the fact that there was a next release meant that it was intrinsically not perfect -- there was always something to improve or add); it's to get it out and ship it and acknowledge that there will be a future release to add features or other improvements.  This really allows the team to focus on what's critical now and get it out the door and on time.

To that end, roadmaps are important as a communication tool and a lever for controlling scope because it gives customers visibility and a sense of certainty that while feature X is missing in this release, in 3 months, it'll be in the next point release.  It's important because it helps manage the requirements bucket; without a roadmap, the tendency of the team and customers -- in my observations -- will be to assume that every requirement is critical.  It's a purely psychological notion because the lack of the roadmap makes it difficult to allow the team to offload some ideas and lesser requirements so that the team can focus on the requirements that are truly necessary to ship it.  Without the concrete notion of The Next Release, the feeling will be that everything must be crammed into this release.

Ultimately, I think that for software development teams to successfully ship software -- given the typical constraints of time, money, and requirements -- it's important to be able to take a critical eye to the requirements and really be able to categorize and define the scope of what is critical versus what is nice to have.  A clear roadmap is an important tool to help teams organize work and thoughts as well as communicate intent to customers.


Thoughts on Burnout

Posted by Charles Chen

I was reading an NPR piece on worker burnout and some different tactics taken by different companies to deal with it and came across a very nice, concise definition:

Christina Maslach is a professor at the University of California, Berkeley, whose four decades of research on the subject helped popularize the term "burnout." Maslach says it's a loose term that encompasses a combination of work overload, lack of autonomy and reward and social and moral discord at work.

This sentence very concisely summarizes the key drivers of burnout and the factors at play are not as simple as "too much work".

The article also brings up an interesting observation (well, it's just the next few paragraphs):

Most burnout stems from interpersonal strife, but most employers see the solution as time off, she says.

If companies really want to know what's causing burnout in their workplace, Maslach says, they shouldn't just mandate more time off. They should assess the core problem, then design solutions to mitigate those issues.

"When it's time off, I mean, that might be time away from work," Maslach says. "Maybe you're addressing issues of exhaustion, but it's not really addressing what may be the problems at work."

Ultimately, a company, a project, a product -- it is the effort of many individual humans who must come together to fulfill a common goal.  And when humans are involved, conflict is sure to arise.  Obviously, you can still get things done when not all of your parts are in harmony, but isn't it much more enjoyable when they are?

I hardly consider myself an expert, but in my own experience, I've found that it's a good idea to work to reinforce those relationships between the people that comprise the team through team activities.  A common one is eating together with one another or occasionally taking the whole office to lunch or dinner. It is especially important for management to be involved because it shows that the employees are valued as people and not just as fungible parts of a machine.

Andrew Fitzgerald comments in that NPR article:

One day I got called into the boss's office. I was thinking to myself "Shoot! What does this guy have on me now? They called me in just to tell me that they thought I was doing a good job and that they appreciate my work ethic. I didn't make a lot of money. The work was kind of tedious and repetitive but I could not tell you how good that made me feel. A little positive feedback from the higher ups goes a long way.

At IC, the development team is in a unique position because all of us work remotely and travel to Irvine.  So we end up spending quite a bit of time together eating meals, going to the shooting range, kayaking on the weekends, and I'm planning on taking the team to an indoor climbing facility as well (I try to keep things fresh).  I also try to make sure that everyone is taken care of; there is nothing I won't do from picking up lunch, driving a co-worker to a train station, picking up fruit for everyone to share, and so on.  Not just because I manage them, but because I like and respect these guys as people first and foremost.

Even at a basic level, we sit together in the office and chit-chat from time to time about random things and watch random videos after we've been hacking away for 8 or 9 hours. When we are on site, not one member of the development team leaves before the others.  Not because anyone is forced to stay, and not because we have some unspoken code about such actions or that we would shame anyone that did, but I think because we all feel that we are in this together and that truly, we have a common goal to achieve as a team.

And that is an important point, in my opinion, because too often, how leaders fail is by not aligning all of the cogs of the machinery towards a common goal.  Most of the time, that simply involves clear and open communication about expectations, company goals, and an understanding of the priorities of the company or the team.

Like a train with two engines heading in opposite directions, failure by team leads to align the members of a team to a goal or failure by management to communicate expectations and priorities seems to lead to inaction, indecision, and conflict when team members are trying to pull in opposite directions.  Ultimately, this just help feed into worker burnout.


A Common Sense Way to Improve Cloud Backup

Posted by Charles Chen

In the wake of the Apple iCloud debacle, there has been a lot of discussion on what Apple has done wrong, what it could do better, and how this could have been prevented.

This is not a blog post about 2-factor authentication or proper implementation of authentication channels or how Apple should be more open in their dealings with the security community, but something more basic and common sense: give users more granular control on what gets backed up.

You will see in many discussions and comments to articles that there is quite a bit of "victim shaming".

Example pulled from Business Insider

Example pulled from Business Insider

But I think that this is quite unfair and I postulate that an average smartphone user has no idea that their photos are being synced to the cloud.  It is far more likely that users had no idea that these photos and videos were synced to the cloud in the first place and even if they had an abstract idea that it was (for example, you take a photo on your phone and you can see it on your desktop later), they had no concrete idea of the implications (those photos are now resident in the cloud as opposed to transient).

It is easy to imagine that such things are obvious and should be trivially easy to configure and control to the end users, but I think that this is a poor assumption to make by anyone that is technically savvy; people like my mother and wife really have no idea about these things.  My guess is that Jennifer Lawrence and Kate Upton simply had no idea that their photos and videos were sitting resident in the cloud and even if they did, they probably couldn't figure out how to get rid of them.

Some have said that this is not the fault of the OS makers or app makers.  Google Photos, for example, gives you a very clear screen when you launch it for the first time asking you if you'd like to sync the files to the cloud.  But one problem is that users may not actually read these things before agreeing.  The other is that even after a user agrees, if the user decides that she wishes to change her mind, the setting is turned off from a screen that is three levels deep (launch Photos, click Menu, click Settings, click Auto Backup).  While this is very obvious to some, to many -- like my mother -- this is an absolute mystery.  She has no idea that it's syncing her photos and has no idea how to turn it off.

I think that there are many common sense solutions that can be implemented outside of the security measures implemented above to give users more granular control over their content.

Give Periodic Notifications to Update Privacy Settings

One simple idea is that say every three months, the phone prompts you with a notification in your notification bar:

Simple mockup of notification

Simple mockup of notification

This would allow users to periodically be reminded that things like automatic sync are on and that they have the option of turning them off.  The user is free to ignore it, but it would give them at least a reminder that "Hey, I'm sending your stuff to the cloud, are you OK with that?  Do you want to review your settings?"

Make Synchronization Explicit

One of the problems I have with Google Photos is that it's all or nothing by default.  There isn't a middle ground that allows me to sync some of my photos as a default.

The user experience paradigm here would be much like that of Facebook where you can post photos by selecting them from your album to explicitly and with fine grain control what gets sent to the cloud.  Likewise, iCloud and Google Photos would do well to allow a middle ground that gives users more fine grained control over what gets sent to the cloud instead of ON and OFF.

In discussions, some have said that this would present too high a burden on end users, but it seems to work fine for Facebook and I think that it would be relatively easy to implement in an easy to use manner:

Example of a notification UI that would allow more fine grained control.

Simple mockup of a notification with controls inline to quickly sync all, explicitly choose photos to sync, or don't sync anything.

If the user selects "Sync All", then all 20 new photos are synced to the cloud (be that iCloud, Dropbox, Google Drive, etc).  If the user selects "Choose", the user is given a screen that allows the user to explicitly pick the ones to sync.  The pick screen should prompt the user to "Ignore unselected items for future backup?" when selection is complete so that any unselected photos are simply ignored next time.  If the user selects "Don't Sync", then do nothing.

A simple design like this still gives the user access to the convenience of cloud backups while giving them explicit, fine-grained control and acknowledgement that their data will be stored in the cloud.

Closing Thoughts

The victim shaming is simply not warranted; whether these individuals should or should not have taken these compromising photos and videos is not the right question to ask.  The right question to ask is whether Apple or Google should be automatically syncing them to a resident cloud storage without finer grained controls and explicit consent.


Invoking Custom WCF Services in SharePoint with Claims

Posted by Charles Chen

In SharePoint, if you host a custom WCF service in a claims-enabled application, the authentication via NTLM is actually quite tricky if you are attempting to invoke it from a console application, for example.

There are various articles and Stackoverflow entries on using System.ServiceModel.Description.ClientCredentials on either the ChannelFactory or the client instance, but all of these did not work in the sense that on the server side, SPContext.Current.Web.CurrentUser was null and ServiceSecurityContext.Current.IsAnonymous returned true.

It seems like it should be possible to invoke the service authenticating through NTLM as if the user were accessing it through the web site.

In fact, it is possible, but it involves some manual HTTP requests to get this to work without doing some Windows Identity Foundation programming and consequently setting up tons of infrastructure to get what seems like a relatively simple and straightforward scenario to work.

The first step is to actually manually retrieve the FedAuth token:

/// <summary>
///     Gets a claims based authentication token by logging in through the NTLM endpoint.
/// </summary>
/// <returns>The FedAuth token required to connect and authenticate the session.</returns>
private string GetAuthToken()
    string authToken = string.Empty;

    CredentialCache credentialCache = new CredentialCache();
    credentialCache.Add(new Uri(_portalBaseUrl), "NTLM", new NetworkCredential(_username, _password, _domain));

    HttpWebRequest request = WebRequest.Create(string.Format("{0}/_windows/default.aspx?ReturnUrl=%2f_layouts%2fAuthenticate.aspx%3fSource%3d%252F&Source=%2F ", _portalBaseUrl)) as HttpWebRequest;
    request.Credentials = credentialCache;
    request.AllowAutoRedirect = false;
    request.PreAuthenticate = true;

    // SharePoint doesn't like it if you don't include these (403 Forbidden)?
    request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko";
    request.Accept = "text/html, application/xhtml+xml, */*";

    HttpWebResponse response = request.GetResponse() as HttpWebResponse;

    authToken = response.Headers["Set-Cookie"];

    return authToken;

There are three keys here:

  1. The first is that AllowAutoRedirect must be false or you will get an error that you are getting too many redirects.  What seems to happen is that the cookies are not set correctly when using auto redirect so the chain will continue until an exception is thrown.  In Fiddler, you will see this as a long cycle of requests and redirects.
  2. The second is that the URL must be the NTLM authentication endpoint (/_windows...") as any other URL will return a 302 and for that, you will need to set AllowAutoRedirect to true.
  3. The third is that it seems as if SharePoint really doesn't like it when the user agent and accept headers are not included.  I tried it later without these and it seemed to work, but I could not get it to work without them (403 unauthorized) initially.

Once you have the FedAuth token, you are able to basically impersonate the user.  To do so, you will need to include a cookie in your HTTP header request:

// Get the FedAuth cookie
var authToken = GetAuthToken();

// Create the connection artifacts.            
EndpointAddress endpointAddress = new EndpointAddress(endpointUrl);
BasicHttpBinding binding = new BasicHttpBinding();            

ChannelFactory<ISomeService> channelFactory = 
    new ChannelFactory<ISomeService>(binding, endpointAddress);

// Initiate the client proxy using the connection and binding information.
ISomeService client = channelFactory.CreateChannel();

using (new OperationContextScope((IContextChannel) client))
    // Set the authentication cookie on the outgoing WCF request.
    WebOperationContext.Current.OutgoingRequest.Headers.Add("Cookie", authToken);


The key is to add the header on the outgoing request before making your service API calls.

With this, you should see that you are able to invoke SharePoint hosted custom WCF service calls in claims-based web applications with NTLM authentication.

Filed under: .Net, SharePoint, WCF No Comments

Defining a “Release Lead”

Posted by Charles Chen

Came across a great definition while reading up on release 4.0 of WordPress:

A release lead, if anyone is curious, determines all important parameters for a release, like schedule, deadlines, which feature plugins are merged, and more generally, scope and goals. They take point when it comes to meetings, shepherding contributions, announcement posts, and updates. A release lead is a connector and facilitator, identifying bottlenecks and friction wherever they may be and at the service of the developers and plugin teams that are aiming to have something in a given release, and be in frequent communication with them.

The release lead should should follow what’s being committed, and set the tone for prioritizing and gardening the milestone on Trac. Given the constraint of time in hitting a date, help with prioritization and ensuring good communication lines are two of the most valuable things a lead can contribute.

The last five release leads were lead developers, but that’s not a requirement, nor is being a committer. I always thought of my “code reviewer” and “committer” hats as being separate, additional responsibilities. (Helen, of course, also wears these same hats.) Regardless: the release lead has the final call on all important decisions related to the release.

I particularly like the term "gardening the milestone" as this is a very suitable analogy to what the responsibilities of a release lead involves as it's very much an art like pruning a rose bush or a hedge and keeping a garden looking sharp.

Filed under: Dev 1 Comment

Adventures in Poorly Written APIs

Posted by Charles Chen

I'm working with a library that I have been fighting with for the better part of three days now to try to get it to work.

The previous version of this library was actually very well written from an API perspective and well documented.  The new version?  Not so much.

But from this, I take away two lessons in API design.

Lesson 1: Don't Use Dictionaries in Public APIs -- ESPECIALLY for Required Data

The first mistake the API designers made in this case was exposing a "dictionary" in the public API.

This itself may not be all bad if the API is highly flexible and can accept many forms of requests, but if a public dictionary interface is required, then at least have the good judgement to provide a wrapper to populate those dictionaries.

But even the lack of usability is not as bad was having required data inputs passed to your service as dictionary entries.  In other words, the service call won't succeed without some sort of magical recipe of just the right metadata keys and values.  If it's a required piece of metadata, why not surface it as an actual parameter to the function?

Exposing dictionaries in public APIs is a terrible design idea which is as bad as APIs that require you to pass in XML strings (understood if it's a must, but have the courtesy to provide downstream callers an API to serialize objects into that XML and don't put required data into open ended inputs).  They should be used infrequently and preferably wrapped by an abstraction layer.

Lesson 2: Don't Require Your Caller to Keep Passing You Data You Handed Back

In this particular API, there is an initialization handshake with the server whereby large bits of system data are passed back to the caller.

For whatever reason, instead of simply handing a token back to the server, all (or part?? -- again, it's a magical incantation of sorts) of this metadata must be exchanged with the server to complete a request!

Why not provide a session key?  A caller token?  Something that the caller can use to represent this collection of system data instead of the full set of system data?

More to come, I'm sure of it, but this has been an interesting adventure in poorly written APIs.

Filed under: Dev, Rants No Comments

Mind Your Matches

Posted by Charles Chen

I recently had to track down a performance issue with one of our Cypher queries that was taking an obscenely long amount of time to run (15 seconds!) given the simplistic nature of the query.

Can you spot the problem?

MATCH (dist:Distribution), (user:User)
MATCH (docType:DocumentType)
WHERE dist.Uid = '...'
   AND user.Uid = '...'
   AND docType.Uid = '...'

It was not easy to track down as this is an entirely valid query that gives the exact result desired.

However, that first MATCH will load every (!) Distribution and User in the graph into memory.

MATCH (dist:Distribution), (user:User), (docType:DocumentType)
WHERE dist.Uid = '...'
   AND user.Uid = '...'
   AND docType.Uid = '...'

It was an easy fix, as you can see, but also an easy mistake to make!

Filed under: Neo4j 2 Comments

FluentNHibernate and SQL Date Generation

Posted by Charles Chen

So you'd like your SQL entry to have a system generated date/time, eh?

Here is a sample table:

CREATE TABLE [dbo].[AuditLog] (
	EventOffsetUtc datetimeoffset(7) DEFAULT(SYSDATETIMEOFFSET()) NOT NULL,
	EntityContextUid uniqueidentifier,
	EntityContextName nvarchar(256),
	EntityContextType varchar(128),
	UserLogin nvarchar(128),
	EventName varchar(128),
	AppContext varchar(64),
	EventData nvarchar(max),

To spare you hours dealing with this error:

   SqlDateTime overflow. Must be between
   1/1/1753 12:00:00 AM and 12/31/9999
   11:59:59 PM.

What you need to do is to use the following mapping for your date/time columns:

Map(a => a.EventUtc).Column("EventUtc")
Map(a => a.EventOffsetUtc).Column("EventOffsetUtc")

Special thanks to this Stackoverflow thread.


What Alan Watts Can Teach Us About Leadership

Posted by Charles Chen

I was listening to a talk by Alan Watts and found one bit of advice that really connected to what I've learned about leading others.

The principle is that any time you -- as it were -- voluntarily let up control.

In other words, cease to cling to yourself; you have an excess of power because you are wasting energy all the time in self defense.

Trying to manage things, trying to force things to conform to your will.

The moment you stop doing that, that wasted energy is available.

Therefore you are in that sense -- having that energy available --  you are one with the divine principle; you have the energy.

When you are trying, however, to act as if you were god, that is to say you don't trust anybody and you are the dictator and you have to keep everybody in line you lose the divine energy because what you are simply doing is defending yourself.

One mistake that I've been guilty of is to try to force things to conform to my will on various projects (I still do it to varying degrees!).  It is usually with the best of intentions -- for a cleaner framework, a better product, a more efficient process -- but at the same time, it is true that a lot of energy is spent wasted in doing so.

What is the alternative, then?

I think Watts is right that a level of trust has to exist that the team around you can help you achieve your project goals.  Instead of expending the energy in controlling the members of the team, spend the energy in building that trust through training, mentorship, guidance, and giving up not just control, but responsibility.

Sometimes that trust will be unwarranted, but sometimes, that trust will pay itself back many-fold.