<CharlieDigital/> Programming, Politics, and uhh…pineapples

28Mar/150

5 Pitfalls to Software Project Failure

Posted by Charles Chen

Poorly Controlled Scope

Scope is enemy number 1; it is the amorphous blob that threatens to consume and grow until it is an uncontrollable monster, swallowing all of your carefully planned man hours.

Increases in scope are often the result of failure to manage the customer and expectations.  In any given project, there are only so many levers that can be used to control the successful delivery and it is up to the skilled project manager or client interface to toggle these levers of team size, timelines, requirements, and so on.

The worst is when growth of scope originates from within the team as it is a form of cancer that only causes teams to compromise on quality to meet timelines promised to the customer.  You see, when scope creep originates from the customer, there is a certain expectation that of course, costs will increase or timelines will need to be shifted.  After all, they are asking you to do more than was initially agreed upon.  But when new scope originates from the team itself, the customer will not readily accept this delay.

The cost of scope increases is often not well accounted for.  A change that takes a developer 2 days to make will cause ripples that force test teams to adjust their scripts, documentation teams to update their documents, and possibly trigger expensive regression testing.

Smart teams and leaders will understand that these can be controlled, in many cases, by simply creating a roadmap and understanding that desired features and capabilities that don't fit into existing timelines can be added to "v.next".

(Over) Reliance on Manual Effort

To a certain extent, software engineering requires raw manpower to execute large projects that require many hundreds of thousands of lines of code and lots of moving parts.

But within the lifecycle of a project, there are many activities that can be simplified by the use of automation.  Teams must judiciously balance the cost and effort of the automation versus the savings gained, but more often than not, even a little bit of automation is better than none.  It's crazy to think that it was once the case that all phone calls were manually routed between parties.

switchboard

Can you imagine if we never evolved past this?

Nowadays, the idea seems crazy!  Imagine if the billions of people on this Earth were to rely on the same processes to connect phone calls today!

Testing is a great example where failure to automate creates a bottleneck to progress.  It increases the cost of changes and bug fixes because it increases the cost of regression testing.  Make the regression testing virtually free and the cost of introducing changes (whether small scope creep for critical bug fixes) is decreased dramatically.

Technologies like Selenium WebDriver and Visual Studio's built in tooling make it possible to achieve significant gains in productivity when it comes to testing.  Don't let excuses hold your team back.

are-you-too-busy-to-improve2

Author's depiction of trying to convince test teams to automate

One skilled test automation engineer is worth her weight in gold!

Poor Communication and Collaboration

Strong and open channels of communication are critical for the success of projects, especially so when some or all of the resources are remote.

The flow of information and feedback from the customer to the design and engineering teams must be swift and clear so that expectations are known and any roadblocks can be communicated back.  Engineering teams will often have insights into the challenges and nuances of a customer's input and it can be dangerous to agree to timelines or make promises without clearly engaging the teams executing the implementation. Ideas that seem simple on paper or in concept can require massive engineering changes or sacrifices to achieve and not properly estimating this work is a common pitfall.

Demarco and Lister's Peopleware offers excellent insight into how to foster better communication and collaboration between teams.

Often, one of the simplest solutions is to simply talk to each other instead of using emails, chat messages, and worst of all: assumption ("Oh, I thought you already knew that"; we've all heard that one before!).  Get in front of a whiteboard and draw out ideas, deadlines, goals, and so on.  Go out to eat lunch together.  Plan team activities that engage everyone.  Make sure that everyone is on the same page on a professional level as well as a personal level.

Not Keeping Your Eyes on the Prize

It's easy for a team to get distracted and lose their focus on the goals of the project and the conditions of victory.

It is therefore critical that teams focus on a goal-oriented approach to the delivery of software projects.  This is a mind-set that scales up from daily scrums to weekly reviews and so on.  Even a short coffee break can be used to re-orient a wandering team member towards the goal posts.  Small, daily victories can help teams build momentum and continuously align towards the long term milestones.

It's important that individuals and teams know, at any given time, what is expected of them and what the priorities of the project are.  This allows individuals to make decisions autonomously and with little managerial overhead as they understand how to align themselves with the goals of the project and team.  Clear communication of goals allows any misunderstandings to surface early by pinning expectations to milestones -- be they simply daily ones, weekly ones, or project level milestones.

Teams and leaders that are poor at communication and collaboration will often lose their focus on the prize because there is a lack of understanding about shifting goals and priorities; there is a dependence on assumption instead of clearly aligning all parties to a set of well-defined conditions of victory.  These anti-leaders will focus on the tasks instead of the goals; it should be the other way around - focus on the goals and derive your tasks from them.

Unwillingness to Compromise

Teams must always be ready to compromise because this is the real world where timelines and successful delivery of usable software matters, but people also have families and life outside of work.  Unplanned circumstances arise that challenge the best laid blueprints.

If it is discovered that a feature will negatively impact performance of the system in the current architecture, compromise must be made on either the feature or the timelines to ensure that the desired capability can be delivered as usable software.

If unforeseen circumstances eat into the project timelines, compromise must be made to clearly redefine the scope and conditions of victory.

This is the real-world; man-hours are not unlimited and an unwillingness to compromise when necessary leads to poor quality as a team pushes to make up time.

In many cases, it is a bitter pill to swallow as it may mean telling a customer that a feature must be delayed or built into the next release, but I find that more often than not, openness and clearly communicating these issues as early as reasonable is productive and allows for rational decision making.

14Mar/150

Adding Support for Azure AD Login (O365) to MVC Apps

Posted by Charles Chen

I spent the day toying around with ASP.NET MVC 5 web applications and authentication.  I won't cover the step-by-step as there are plenty of blogs that have it covered.

It seems that online, most examples and tutorials show you either how to use your organizational Azure AD account or social identity providers but not both.

I wanted to be able to log in using Facebook, Google, and/or the organizational account I use to connect to Office 365.

This requires that you select Individual User Accounts when prompted to change the authentication mode (whereas most tutorials have you select "Organization Accounts"):

mvc-use-individual-account

This will give you the baseline needed to add the social login providers (more on that later).

To enable Windows Azure AD, you will need to first login into Azure and add an application to your default AD domain.  In the management portal:

  1. Click on ACTIVE DIRECTORY in the left nav
  2. Click the directory
  3. Click the APPLICATIONS link at the top
  4. Now at the bottom, click ADD to add a new application
  5. Select Add an application my organization is developing
  6. Enter an arbitrary name and click next
  7. Now in the App properties screen, you will need to enter your login URL (e.g. https://localhost:4465/Account/Login) and for the APP ID URI, you cannot use "localhost".  You should use your Azure account info like: https://myazure.onmicrosoft.com/MyApp.  The "MyApp" part is arbitrary, but the bolded text must match your directory identifier.

Most importantly, once you've created it, you need to click on the CONFIGURE link at the top and turn on the setting APPLICATION IS MULTI-TENANT:

mvc-multi-tenant

If you fail to turn this on, the logins are limited to the users that are in your Azure AD instance only; you will not be able to log on with accounts you use to connect to Office 365.  You'll get an error like this:

Error: AADSTS50020: User account ‘jdoe@myo365domain.com’ from external identity provider ‘https://sts.windows.net/1234567e-b123-4123-9112-912345678e51/’ is not supported for application ‘2123456f-b123-4123-9123-4123456789e5'. The account needs to be added as an external user in the tenant. Please sign out and sign in again with an Azure Active Directory user account.

An important note is that if you used "localhost" in step 7, the UI will not allow you to save the settings with an error "The App ID URI is not available. The App ID URI must be from a verified domain within your organization's directory."

Once you've enabled this, we're ready to make the code changes required.

First, you will need to install the OpenId package from nuget using the following command:

install-package microsoft.owin.security.openidconnect

Next, in the default Startup.Auth.cs file generated by the project template, you will need to add some additional code.

First, add this line:

app.SetDefaultSignInAsAuthenticationType(CookieAuthenticationDefaults.AuthenticationType);

Then, add this:

app.UseOpenIdConnectAuthentication(new OpenIdConnectAuthenticationOptions
{
    ClientId = "138C1130-4B29-4101-9C84-D8E0D34D222A",
    Authority = "https://login.windows.net/common",
    PostLogoutRedirectUri = "https://localhost:44301/",                
    Description = new AuthenticationDescription
    {
        AuthenticationType = "OpenIdConnect",
        Caption = "Azure OpenId  Connect"
    },
    TokenValidationParameters = new TokenValidationParameters
    {
        // If you don't add this, you get IDX10205
        ValidateIssuer = false   
    }
});

There are two very important notes.  The first is that the Authority must have the /common path and not your Azure AD *.onmicrosoft.com path.

The second note is that you must add the TokenValidationParameters and set ValidateIssuer to false.

If you don't set this to false, you'll get the following 500 error after you successfully authenticate against Azure AD with your organizational O365 account:

IDX10205: Issuer validation failed. Issuer: ‘https://sts.windows.net/F92E09B4-DDD1-40A1-AE24-D51528361FEC/’. Did not match: validationParameters.ValidIssuer: ‘null’ or validationParameters.ValidIssuers: ‘https://sts.windows.net/{tenantid}/’

I think that this is a hack and to be honest, I'm not quite certain of the consequences of not validating the issuer, but it seems that there aren't many answers on the web for this scenario yet.  Looking at the source code where the exception originates, you'll see the method that generates it:

public static string ValidateIssuer(string issuer, SecurityToken securityToken, TokenValidationParameters validationParameters)
{
    if (validationParameters == null)
    {
        throw new ArgumentNullException("validationParameters");
    }
    
    if (!validationParameters.ValidateIssuer)
    {
        return issuer;
    }
    
    if (string.IsNullOrWhiteSpace(issuer))
    {
        throw new SecurityTokenInvalidIssuerException(string.Format(CultureInfo.InvariantCulture, ErrorMessages.IDX10211));
    }
    
    // Throw if all possible places to validate against are null or empty
    if (string.IsNullOrWhiteSpace(validationParameters.ValidIssuer) && (validationParameters.ValidIssuers == null))
    {
        throw new SecurityTokenInvalidIssuerException(string.Format(CultureInfo.InvariantCulture, ErrorMessages.IDX10204));
    }
    
    if (string.Equals(validationParameters.ValidIssuer, issuer, StringComparison.Ordinal))
    {
        return issuer;
    }
    
    if (null != validationParameters.ValidIssuers)
    {
        foreach (string str in validationParameters.ValidIssuers)
        {
            if (string.Equals(str, issuer, StringComparison.Ordinal))
            {
                return issuer;
            }
        }
    }
    
    throw new SecurityTokenInvalidIssuerException(
        string.Format(CultureInfo.InvariantCulture, ErrorMessages.IDX10205, issuer, validationParameters.ValidIssuer ?? "null", Utility.SerializeAsSingleCommaDelimitedString(validationParameters.ValidIssuers)));
}

We're simply short circuiting the process.  It's clear that there is no matching issuer, but it's not quite clear to me yet where/how to configure that.

So what about the other social IdP's?  It's important to note that for Google, not only do you have to create a new client ID in the Google Developer Console, but you also need to enable the Google+ API:

mvc-google-api

You'll just get a bunch of useless error messages if you don't enable the API.

If you manage to get it all working, you should see the following options in the login screen:

mvc-azure

And when you click it, you should be able to log in using the same organizational credentials that you use to connect to Office 365:

mvc-login-azure

Filed under: .Net, MVC No Comments
23Feb/15Off

One of the More Creative Ways to Advertise Career Opportunities

Posted by Charles Chen

Soundcloud Console

As seen on Soundcloud.com as I was examining why the page wasn't loading...

Filed under: Uncategorized No Comments
16Jan/15Off

Adding The Google Test to Your Interviews

Posted by Charles Chen

On a message board, I read a thread where a poster -- a research scientist -- was describing how he ended up becoming the defacto IT guy in his department simply because of his superior Google skills and willingness to Google for and apply solutions to fix issues for his colleagues.

This is something I've personally never been asked to do in an interview nor have I thought to ask others when I interview them, but it seems that being able to quickly Google and sift through results quickly to separate the wheat from the chaff is a skill that is supremely underrated in today's world of software engineering.

The fact is that developers and technology specialists today need to deal with so many technologies and understand deep nuances, Google is often the only way that any of us can get anything done, especially with obscure errors and what not that Microsoft and SharePoint loooove to throw at you.

In fact, I'm quite surprised that I've never been asked to do a Google search speed and accuracy test.

How would one design such a test to be effective at measuring a candidate's speed and accuracy at using Google?  Should the topics be relevant to the candidates job domain?  Or should it be more generic?  Should it test a candidate's knowledge of Google's advanced features?

11Nov/14Off

Indoor Rock Climbing – Try It!

Posted by Charles Chen

One of my recently discovered activities that I'm falling in love with is indoor rock climbing (though I suppose I may try outdoor rock climbing and bouldering one day, too).

In a weird way, it's the ultimate thinking person's type of sport that is physically demanding, but also mentally challenging as well.  Climbers like to talk in terminology like "problems", "projects", and "solutions" and it's entirely accurate and applicable way to describe what climbing is all about.  If you walk into a bouldering area in a gym, you will see climbers just sitting around, planning routes, thinking about how to position their bodies to make the right move and attacking routes over and over again.  Difficult routes demand that you plan and think about how you can make your way up a vertical face while expending the least amount of energy.

It's odd, but I also think that it's a very "romantic" (or "bromantic"?) activity because you'll have the most fun climbing with someone else.  There is a lot of communication and trust involved when one person is controlling the safety and well-being of another person suspended 40 feet in the air.  For that same reason, it's a great team building activity for companies because to climb, you need to be able to work together, communicate, and have trust in your partners.

To get started, you can look up Google Maps and find some nearby rock climbing gyms and just call and take a class.  I took my first class at Rockreation in Costa Mesa, CA where you had to schedule ahead and the classes are far more formalized, but there are also places like Rockville back home in NJ, where the classes are much more informal and you can just show up and take a short intro class.

Most intro classes will teach you the basic elements of indoor climbing:

  • Using harnesses and shoes
  • Basic double-figure-8 knot tying
  • Belaying
  • Basic safety including verbal commands and communications.

But in looking through some videos, I've found that there is LOT more to learn and I've developed an even deeper appreciation for it. Take a look for yourself:

Five Fundamentals of Indoor Rock Climbing

How to Grip Indoor Climbing Holds

Footwork for Climbing

Five Advanced Bouldering Techniques

What I hope that you can get from this is that there is a real art to this that is beautiful to watch in action.  In that last video, the Bat Hang at 1:45  is a thing of beauty.  Seasoned climbers make it look easy, but it really takes a lot of practice, experience, and creativity to move around like Cliff Simanski does in the video.

Charlotte and Sandra working a wall.

Charlotte and Sandra working a wall.

I've also learned that I've been wrecking my forearms because I've basically been muscling my way up the walls with my upper body strength alone.  A strong grip and upper body are certainly beneficial for climbing, but you need far more than that to advance in the sport.

In a sense, rock climbing has a lot in common with dance or gymnastics: it demands creative body movement, flexibility, balance, body awareness, and spatial awareness (maybe even more so because your life is on the line in some cases).

It's a great activity for kids of all ages (Charlotte is 3.5 years old) to enjoy.

4Nov/14Off

An Alternate Meaning for FOCKED

Posted by Charles Chen

Eric Brechner came up with one of my favorite acronyms of all time in software development: FOCKED.

focked

I want to add an alternate: Failure to Orchestrate Collective Knowledge Effectively for Delivery.

Successful delivery of software requires that different members of the team come together and understand the goals that have to be achieved and the priorities of those goals.

It's as simple as communicating to the team on a regular basis (no more than once a week, but maybe at least once a month):

  • where we are,
  • where do we want to go,
  • when do we have to get there,
  • how are we getting there,
  • who's driving

It can make the whole process of delivery of software much less stressful and maybe more successful simply by aligning all of the stakeholders periodically.

Hey, maybe you learned this in some fancy MBA class or something, but I'm starting to appreciate -- more and more -- that the real secret to successful delivery of software is driving the successful collaboration and communication of people and alignment of all pieces to a strategy, vision, or goal.  Having a bunch of smart, capable people doesn't help you much if no one knows what's going on.

Filed under: DevLife, Rants No Comments
16Oct/14Off

The Importance of Scope (and How To Ship It!)

Posted by Charles Chen

One of the lessons I've been mulling about the past few weeks is the importance of scope when delivering software.

Delivery of software can be thought of as a balancing act between three buckets:

  1. Time - this is the schedule and how much time you have to complete the work.
  2. Money - this includes everything from spending on more resources, better resources, tooling support, and so on
  3. Requirements - this defines what you're building

These are the three basic buckets that constrain the scope of what can be done and they react to each other in different ways.  If there are more requirements or the requirements are inflexible, then it necessitates more time or money or both.  If both requirements and time are inflexible, then more money will be required to achieve the goals within those limits.  If money is constrained (less resources), then you must allow more time to deliver the requirements or trim the requirements.

Having been in consulting and in software development, each project has different priorities on which is more important and these priorities drive the sizing, cost, and pace of the project.

But in software development, I think one thing that I think many folks -- even experienced individuals -- get wrong is the fixation and inflexibility on requirements.  I think that requirements are really much more fluid in software development projects as compared to contract-driven consulting projects.  The reason is simple: in software development, the assumption is that there will always be another version and there will always be another point release; this is exactly what roadmaps are for.

Plan a solid roadmap and if you don't get a particular feature or capability in this release, you and your customers should have a good idea of which release it will be in down the road.

Some tend to lose sight of this and think that all features must be shipped in a constrained timeline.  I think this is a losing proposition that forces a team to compromise on quality by artificially forcing requirements into a release when the reality is that there are often really critical features that must be delivered and there are nice to have features that would be great if they could be delivered.  Teams and leaders without discipline and a focus on the criteria of success will have a difficult time discerning the two and lump nice-to-haves right alongside the critical work items.  This is a recipe for failed projects, missed deadlines, and poor quality.

The reality is that software rarely comes out fully baked unless you're NASA launching a space mission worth billions of dollars and you only get one shot; most teams are not under such otherworldly constraints.  There will always be something that could be better or easier to use or some missing feature discovered along the way or some new idea.  The trick for teams that succeed is being about to create boundaries on what is needed now.

Apple shipped 7 versions of iOS before adding support for third party keyboards and NFC.

NPR's first mobile site was terrible (I don't have a screenshot of it, unfortunately), but at least they shipped it and their audience could use it and now they've evolved it to be a clean, responsive web site.

Here's what the an early version of Facebook looked like:

Courtesy of Wikipedia

Courtesy of Wikipedia

And Amazon:

Back when it was just a bookstore

Microsoft shipped Azure without support for virtual machines until 2013.

But what if instead of new features, we're talking about bugs or design flaw?  There is an important calculus at play here and I think that one way to think about it is like this: if there is a bug or design flaw that is preventing increased growth in userbase or revenue, then that takes precedent over any other bug or design flaw that is in the shipped system.  Think about it this way: if you ship four versions of the software with the bug or design flaw, chances are, you can probably ship a fifth without addressing it (of course, this is not always the case, especially if the flaw is related to security).  But if a bug or design flaw is stopping adoption or holding back revenue, then that flaw automatically becomes the most critical focus for a release.

The point is that in software product development, more often than not, the winning strategy isn't to get it perfect (as much as Steve Jobs would have you believe that he got it perfect each time, the fact that there was a next release meant that it was intrinsically not perfect -- there was always something to improve or add); it's to get it out and ship it and acknowledge that there will be a future release to add features or other improvements.  This really allows the team to focus on what's critical now and get it out the door and on time.

To that end, roadmaps are important as a communication tool and a lever for controlling scope because it gives customers visibility and a sense of certainty that while feature X is missing in this release, in 3 months, it'll be in the next point release.  It's important because it helps manage the requirements bucket; without a roadmap, the tendency of the team and customers -- in my observations -- will be to assume that every requirement is critical.  It's a purely psychological notion because the lack of the roadmap makes it difficult to allow the team to offload some ideas and lesser requirements so that the team can focus on the requirements that are truly necessary to ship it.  Without the concrete notion of The Next Release, the feeling will be that everything must be crammed into this release.

Ultimately, I think that for software development teams to successfully ship software -- given the typical constraints of time, money, and requirements -- it's important to be able to take a critical eye to the requirements and really be able to categorize and define the scope of what is critical versus what is nice to have.  A clear roadmap is an important tool to help teams organize work and thoughts as well as communicate intent to customers.

30Sep/14Off

Thoughts on Burnout

Posted by Charles Chen

I was reading an NPR piece on worker burnout and some different tactics taken by different companies to deal with it and came across a very nice, concise definition:

Christina Maslach is a professor at the University of California, Berkeley, whose four decades of research on the subject helped popularize the term "burnout." Maslach says it's a loose term that encompasses a combination of work overload, lack of autonomy and reward and social and moral discord at work.

This sentence very concisely summarizes the key drivers of burnout and the factors at play are not as simple as "too much work".

The article also brings up an interesting observation (well, it's just the next few paragraphs):

Most burnout stems from interpersonal strife, but most employers see the solution as time off, she says.

If companies really want to know what's causing burnout in their workplace, Maslach says, they shouldn't just mandate more time off. They should assess the core problem, then design solutions to mitigate those issues.

"When it's time off, I mean, that might be time away from work," Maslach says. "Maybe you're addressing issues of exhaustion, but it's not really addressing what may be the problems at work."

Ultimately, a company, a project, a product -- it is the effort of many individual humans who must come together to fulfill a common goal.  And when humans are involved, conflict is sure to arise.  Obviously, you can still get things done when not all of your parts are in harmony, but isn't it much more enjoyable when they are?

I hardly consider myself an expert, but in my own experience, I've found that it's a good idea to work to reinforce those relationships between the people that comprise the team through team activities.  A common one is eating together with one another or occasionally taking the whole office to lunch or dinner. It is especially important for management to be involved because it shows that the employees are valued as people and not just as fungible parts of a machine.

Andrew Fitzgerald comments in that NPR article:

One day I got called into the boss's office. I was thinking to myself "Shoot! What does this guy have on me now? They called me in just to tell me that they thought I was doing a good job and that they appreciate my work ethic. I didn't make a lot of money. The work was kind of tedious and repetitive but I could not tell you how good that made me feel. A little positive feedback from the higher ups goes a long way.

At IC, the development team is in a unique position because all of us work remotely and travel to Irvine.  So we end up spending quite a bit of time together eating meals, going to the shooting range, kayaking on the weekends, and I'm planning on taking the team to an indoor climbing facility as well (I try to keep things fresh).  I also try to make sure that everyone is taken care of; there is nothing I won't do from picking up lunch, driving a co-worker to a train station, picking up fruit for everyone to share, and so on.  Not just because I manage them, but because I like and respect these guys as people first and foremost.

Even at a basic level, we sit together in the office and chit-chat from time to time about random things and watch random videos after we've been hacking away for 8 or 9 hours. When we are on site, not one member of the development team leaves before the others.  Not because anyone is forced to stay, and not because we have some unspoken code about such actions or that we would shame anyone that did, but I think because we all feel that we are in this together and that truly, we have a common goal to achieve as a team.

And that is an important point, in my opinion, because too often, how leaders fail is by not aligning all of the cogs of the machinery towards a common goal.  Most of the time, that simply involves clear and open communication about expectations, company goals, and an understanding of the priorities of the company or the team.

Like a train with two engines heading in opposite directions, failure by team leads to align the members of a team to a goal or failure by management to communicate expectations and priorities seems to lead to inaction, indecision, and conflict when team members are trying to pull in opposite directions.  Ultimately, this just help feed into worker burnout.

3Sep/14Off

A Common Sense Way to Improve Cloud Backup

Posted by Charles Chen

In the wake of the Apple iCloud debacle, there has been a lot of discussion on what Apple has done wrong, what it could do better, and how this could have been prevented.

This is not a blog post about 2-factor authentication or proper implementation of authentication channels or how Apple should be more open in their dealings with the security community, but something more basic and common sense: give users more granular control on what gets backed up.

You will see in many discussions and comments to articles that there is quite a bit of "victim shaming".

Example pulled from Business Insider

Example pulled from Business Insider

But I think that this is quite unfair and I postulate that an average smartphone user has no idea that their photos are being synced to the cloud.  It is far more likely that users had no idea that these photos and videos were synced to the cloud in the first place and even if they had an abstract idea that it was (for example, you take a photo on your phone and you can see it on your desktop later), they had no concrete idea of the implications (those photos are now resident in the cloud as opposed to transient).

It is easy to imagine that such things are obvious and should be trivially easy to configure and control to the end users, but I think that this is a poor assumption to make by anyone that is technically savvy; people like my mother and wife really have no idea about these things.  My guess is that Jennifer Lawrence and Kate Upton simply had no idea that their photos and videos were sitting resident in the cloud and even if they did, they probably couldn't figure out how to get rid of them.

Some have said that this is not the fault of the OS makers or app makers.  Google Photos, for example, gives you a very clear screen when you launch it for the first time asking you if you'd like to sync the files to the cloud.  But one problem is that users may not actually read these things before agreeing.  The other is that even after a user agrees, if the user decides that she wishes to change her mind, the setting is turned off from a screen that is three levels deep (launch Photos, click Menu, click Settings, click Auto Backup).  While this is very obvious to some, to many -- like my mother -- this is an absolute mystery.  She has no idea that it's syncing her photos and has no idea how to turn it off.

I think that there are many common sense solutions that can be implemented outside of the security measures implemented above to give users more granular control over their content.

Give Periodic Notifications to Update Privacy Settings

One simple idea is that say every three months, the phone prompts you with a notification in your notification bar:

Simple mockup of notification

Simple mockup of notification

This would allow users to periodically be reminded that things like automatic sync are on and that they have the option of turning them off.  The user is free to ignore it, but it would give them at least a reminder that "Hey, I'm sending your stuff to the cloud, are you OK with that?  Do you want to review your settings?"

Make Synchronization Explicit

One of the problems I have with Google Photos is that it's all or nothing by default.  There isn't a middle ground that allows me to sync some of my photos as a default.

The user experience paradigm here would be much like that of Facebook where you can post photos by selecting them from your album to explicitly and with fine grain control what gets sent to the cloud.  Likewise, iCloud and Google Photos would do well to allow a middle ground that gives users more fine grained control over what gets sent to the cloud instead of ON and OFF.

In discussions, some have said that this would present too high a burden on end users, but it seems to work fine for Facebook and I think that it would be relatively easy to implement in an easy to use manner:

Example of a notification UI that would allow more fine grained control.

Simple mockup of a notification with controls inline to quickly sync all, explicitly choose photos to sync, or don't sync anything.

If the user selects "Sync All", then all 20 new photos are synced to the cloud (be that iCloud, Dropbox, Google Drive, etc).  If the user selects "Choose", the user is given a screen that allows the user to explicitly pick the ones to sync.  The pick screen should prompt the user to "Ignore unselected items for future backup?" when selection is complete so that any unselected photos are simply ignored next time.  If the user selects "Don't Sync", then do nothing.

A simple design like this still gives the user access to the convenience of cloud backups while giving them explicit, fine-grained control and acknowledgement that their data will be stored in the cloud.

Closing Thoughts

The victim shaming is simply not warranted; whether these individuals should or should not have taken these compromising photos and videos is not the right question to ask.  The right question to ask is whether Apple or Google should be automatically syncing them to a resident cloud storage without finer grained controls and explicit consent.

12Aug/14Off

Invoking Custom WCF Services in SharePoint with Claims

Posted by Charles Chen

In SharePoint, if you host a custom WCF service in a claims-enabled application, the authentication via NTLM is actually quite tricky if you are attempting to invoke it from a console application, for example.

There are various articles and Stackoverflow entries on using System.ServiceModel.Description.ClientCredentials on either the ChannelFactory or the client instance, but all of these did not work in the sense that on the server side, SPContext.Current.Web.CurrentUser was null and ServiceSecurityContext.Current.IsAnonymous returned true.

It seems like it should be possible to invoke the service authenticating through NTLM as if the user were accessing it through the web site.

In fact, it is possible, but it involves some manual HTTP requests to get this to work without doing some Windows Identity Foundation programming and consequently setting up tons of infrastructure to get what seems like a relatively simple and straightforward scenario to work.

The first step is to actually manually retrieve the FedAuth token:

/// <summary>
///     Gets a claims based authentication token by logging in through the NTLM endpoint.
/// </summary>
/// <returns>The FedAuth token required to connect and authenticate the session.</returns>
private string GetAuthToken()
{
    string authToken = string.Empty;

    CredentialCache credentialCache = new CredentialCache();
    credentialCache.Add(new Uri(_portalBaseUrl), "NTLM", new NetworkCredential(_username, _password, _domain));

    HttpWebRequest request = WebRequest.Create(string.Format("{0}/_windows/default.aspx?ReturnUrl=%2f_layouts%2fAuthenticate.aspx%3fSource%3d%252F&Source=%2F ", _portalBaseUrl)) as HttpWebRequest;
    request.Credentials = credentialCache;
    request.AllowAutoRedirect = false;
    request.PreAuthenticate = true;

    // SharePoint doesn't like it if you don't include these (403 Forbidden)?
    request.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko";
    request.Accept = "text/html, application/xhtml+xml, */*";

    HttpWebResponse response = request.GetResponse() as HttpWebResponse;

    authToken = response.Headers["Set-Cookie"];

    return authToken;
}

There are three keys here:

  1. The first is that AllowAutoRedirect must be false or you will get an error that you are getting too many redirects.  What seems to happen is that the cookies are not set correctly when using auto redirect so the chain will continue until an exception is thrown.  In Fiddler, you will see this as a long cycle of requests and redirects.
  2. The second is that the URL must be the NTLM authentication endpoint (/_windows...") as any other URL will return a 302 and for that, you will need to set AllowAutoRedirect to true.
  3. The third is that it seems as if SharePoint really doesn't like it when the user agent and accept headers are not included.  I tried it later without these and it seemed to work, but I could not get it to work without them (403 unauthorized) initially.

Once you have the FedAuth token, you are able to basically impersonate the user.  To do so, you will need to include a cookie in your HTTP header request:

// Get the FedAuth cookie
var authToken = GetAuthToken();

// Create the connection artifacts.            
EndpointAddress endpointAddress = new EndpointAddress(endpointUrl);
BasicHttpBinding binding = new BasicHttpBinding();            

ChannelFactory<ISomeService> channelFactory = 
    new ChannelFactory<ISomeService>(binding, endpointAddress);

// Initiate the client proxy using the connection and binding information.
ISomeService client = channelFactory.CreateChannel();

using (new OperationContextScope((IContextChannel) client))
{
    // Set the authentication cookie on the outgoing WCF request.
    WebOperationContext.Current.OutgoingRequest.Headers.Add("Cookie", authToken);

    // YOUR API CALLS HERE    
}

The key is to add the header on the outgoing request before making your service API calls.

With this, you should see that you are able to invoke SharePoint hosted custom WCF service calls in claims-based web applications with NTLM authentication.

Filed under: .Net, SharePoint, WCF No Comments