<CharlieDigital /> Programming, Politics, and uhh…pineapples

3Jun/130

Understanding Billing for Amazon EBS Provisioned IOPs

Posted by Charles Chen

I've been experimenting with Amazon EC2 and SharePoint to better understand how an enterprise architecture would be built on top of it as a platform and, as a part of this exercise, better understand the cost structure.

One key number that escaped me during my initial review of the pricing structure is the one highlighted below:

provisioned-iops-month

Now before we get into the details of what to watch for here, I should preface and say that for use cases that demand high performance disk I/O, provisioned IOPS is probably well worth it and, in fact, I definitely foresee us using this in select scenarios (i.e. high I/O database server instance).

In the screen cap below, you can see that I've got three provisioned IOPS volumes configured with 2000, 2000, and 1000 IOPS respectively (note that maximum IOPS on a volume is a multiple of the size of the volume):

disks

The question is what does "$0.10 per provisioned IOPS-month" actually mean?

Well, here's the skinny from the Amazon web site:

For example, if you provision a volume with 1000 IOPS, and keep this volume for 15 days in a 30 day month, then in the Virginia Region, you would be charged $50 for the IOPS that you provision ($0.10 per provisioned IOPS-Month * 1000 IOPS Provisioned * 15 days/30).

So in my case, for a 30 day month, it would work out to be (2000 + 2000 + 1000) * 0.10 * 1 = $500/mo. for the provisioned IOPS volumes.

This is something to keep your eye on as if you are using EC2 instances with provisioned IOPS volumes sporadically, you may want to take snapshots or create an AMI and discard the volumes when not in use.  Comparatively speaking, snapshot data is much friendlier on the wallet when your system isn't under active use.

Tagged as: No Comments
4Dec/12Off

Thoughts on Successful Team Management

Posted by Charles Chen

Reflecting on the past year and a half, I've come to some conclusions on how development teams can be successful in delivering software.  I don't think any of them are major or relevatory, but I think each team and each project has a different take on the same ideas.  Here are mine:

Iterate, Quickly

During our active build phase, we release a new build every week and each build is functional software that incrementally incorporates requirements and defect fixes from previous builds.  This has given us the benefit of allowing our customer to preview the software and help us identify flaws or areas requiring additional clarity in the requirements continuously and early in the process.

Automate

It might sound insane, but it is possible to release weekly builds because our solution incorporates a heavy dose of automation where it counts on many levels.

  • We've removed raw SQL from the equation, relying purely on FluentNHibernate and NHibernate to automatically generate our schema
  • We've invested in building tools to automate the export and re-import of configuration and data, allowing us to easily and quickly reset our development environments entirely with standard test data (bonus: the same tool allows us to move configuration and data from environment to environment)
  • We've invested in idiot-proofing our installs so that they are boiled down to a few scripts
  • We've built automated build scripts that package everything up neatly and even FTPs a build to our integration and test environments
  • Our domain data model is 90% generated automatically from content schemas (SharePoint content types) which we have to create anyways.

Because of the automation, tasks which would otherwise be expensive are cheap to execute.

It also cuts down on mistakes and missed steps.

Meet Infrequently

Our team is 100% geographically dispersed with developers and team members in Vietnam, Mexico, Virginia, New Jersey and California.

But relatively speaking, we meet very infrequently.  Two development team meetings a week: one at the start of the week -- our "A" meeting -- and one towards the end of the week -- our "B" meeting.  We use the "A" meeting to discuss our deliverables for the week and the "B" meeting to discuss the outcome of our weekly sprint walkthroughs, any adjustments that need to be made, and so on.

We also use these sessions as show-and-tell to let everyone see the progress and changes being made by different team members as well as to inform of upcoming breaking changes and mitigating actions required downstream.

Otherwise, developers are encouraged to have long spans of uninterrupted work time instead of constantly being pulled into meetings.  One-on-one sessions and communications occur as necessary, but this recipe has been very successful in minimizing the amount of time the team spends huddled up in conference calls and gives everyone more time to solve problems.

Meet with a Purpose

Every meeting should have an agenda and an outcome (an action, decision, or an issue for followup).  Demand a bullet-listed agenda when someone sends you a meeting request and provide one if you need to schedule a meeting.  Ensure that the goal and outcome of the meeting is clear for all parties and schedule new meetings to resolve items not on the agenda or do not contribute to the outcome.

Additionally, create a record of every meeting.  Who attended?  What was covered?  What was not covered?  What was left open?  What are the action items?  Ensure that this record is easily accessible (wiki or forum system is perfect for recording these details) and email a copy to all participants and other relevant parties to ensure that everyone has the same understanding of what was just discussed.  This basic task can often clear up misunderstandings before they become systemic issues.  I take the burden on myself to record and followup with major bullet points from the meetings and it's saved my butt many times when following up with customers.

This is the simple art of not running a terrible meeting.

Lead by Example

A bit of career advice for those with a passion for software development: never remove yourself from the process of creation.

I have witnessed it as the career ladder moves individuals up and up, further from the pure act of creation that is software development.  For those of us who feel invigorated when we solve a difficult programming task, for those of us who feel a great rush of exhilaration when a machinery of thousands of lines of code executes in harmony, it is our burden to tinker, to learn, and to create.

When you "ascend" from developer to architect or team lead or such, never leave your passion for creation behind; authority flows naturally from understanding, knowledge, and mastery -- not just a title.

I was inspired to reflect on this by an interview with James Dyson in Wired:

Wired: Now that Dyson is a sprawling, multinational corporation, how do you keep the spirit of innovation alive?

Dyson: We try to make the corporation like the garage. We don’t have technicians; our engineers and scientists actually go and build their own prototypes and test the rigs themselves. And the reason we do that—and I don’t force people to do that, by the way, they want to do it—is that when you’re building the prototype, you start to really understand how it’s made and what it might do and where its weaknesses might be. If you merely hand a drawing to somebody and say, “Would you make this, please?” and in two weeks he comes back with it and you hand it to someone else who does the test, you’re not experiencing it. You’re not understanding it. You’re not feeling it. Our engineers and scientists love doing that.

As a team lead, never just be a middle man with the developers and requirements; be an active participant in the process.  Work on the hard problems.  Understand the creation process and understand the challenges of the team from the bottom up and build your authority from your ability to innovate and solve problems.

If you watch shows like Iron Chef or Chopped, every one of the judges and every one of the Iron Chefs can be considered the vanguard of their craft and it is from there that their authority flows.  You would not watch Iron Chef if all the Iron Chefs did was design the menu and then watch their team cook.  You would not trust the judges on Chopped if they weren't great chefs in their own right that understood the ingredients, the techniques, and the skill required to pull off a dish.

The better you understand the system, the better you understand your team, the more effective you will be in understanding the root cause of an issue and how to route defects within the team.

Push Your Team Incrementally

As a young developer, I always found great satisfaction in solving new problems and new challenges.  I think it's important that throughout the process, you push your team members and give them tasks that challenge their knowledge and abilities to push them just a little bit.

Of course, there will be plenty of mundane code and defect fixing, but don't box in your team members intellectually.  Understand their capabilities individually and push them to try things that are just beyond their current level of capability, understanding, and comfort zone.  This will keep them engaged and improve their skills to boot.

Invest in Code Quailty, Especially Early On

It's a lot easier to write good code earlier on in the process than it is to come in and try to refactor later on.  Additionally, code written early on tends to be reused more often and patterns and solutions are copied by other developers later on.  So early in the process, it is important to keep code quality in mind and correct bad behaviors since the most influential code will be written earlier in the process rather than later.

What this means is that detailed code reviews are far more important at the beginning than at any other time in the project.  If you can correct bad programming practices or show a developer a better, more modular way of writing a piece of functionality early on, she will carry that knowledge throughout the project.

We rarely do code reviews now (1 year in) as I focused heavily on working one-on-one with developers as they joined the team to ensure that the code was up to standards.  I frequently rejected code and asked developers to rewrite whole pieces of functionality to help them understand why one design was better than another.

Put Your Best Developers on Your Least "Visible" Code

What this boils down to is the framework level components.  Your best developers should be working on the deepest innards of the system that power what the rest of the developers do at the presentation and business logic layers.  This code will be the most invoked and the most reused so it is important that it is:

Do not waste your best developer's time with defect fixes (unless there is sufficient bandwidth), even if they can do it better than anyone else on the team because it will throw off the balance of the team (your more junior developers might not be able to fix a low level defect as quickly, but there are many design issues and higher priority defects that they cannot solve effectively yet).

Document, Document, Document

Early on in our process, I had to decide between a wiki system or a Word document for our documentation.  Because of the fast, iterative nature of the project, I decided to use a wiki/forum system as it was more flexible and -- in a sense -- more "visible".

While our formal documentation is trailing, it is easy to assemble it from our weekly sprint guides which document every new feature with screenshots, details, and examples.

But at any given time, our customer and our delivery partner can load up the wiki and see exactly when we delivered a feature, how to install and configure the feature, how to use the feature, and so on.  By putting it all out there in lock-step with the weekly delivery, it is easy to ensure that the entire team is aware of the status of the project and progress being made and allows test teams to progress iteratively so that by the end of the project, most features have been tested for weeks.

Mid ways through the project, we moved from "status" focused meetings to "demo" focused meetings where we would do a weekly writeup and walkthrough of what changed, what was added, and what was fixed.  It also allowed for open forums for test and business teams to ask questions and get clarifications.

This approach has allows the customer to see progress and the customer will never be surprised at the end of the project as they will have seen the progress and documentation update on a weekly basis.


So far, we have done well with these basic guiding principles.

I'm sure I will revise and add to my own lessons learned as the project continues, but I think that this is a good starting point!

26Aug/10Off

Event Receivers on Content Types

Posted by Charles Chen

Adding this to the category of things-that-I-didn't-know-but-would-have-made-a-lot-of-stuff-I-previously-wrote-much-more-elegant-and-awesome.

You should add it to yours, too!

As a quick summary, it's common knowledge (well, amongst SharePoint developers at least) that you can associate event receivers with a list template type.  However, an interviewer recently brought to light that one can also associate an event receiver directly with a content type.  This is immensely useful for anyone building custom solutions on SharePoint, especially if you make heavy usage of content types in your design.

Here's an example using the same basic content type:

<?xml version="1.0" encoding="utf-8" ?>
<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
  <Field DisplayName="Model Code"
    Name="Model_Code"
    StaticName="Model_Code"
    ID="{F0000000-0000-0000-0000-000000000001}"
    Type="Integer"
    SourceID="http://schemas.someusedcarinventory.com"
    Group="My Custom Columns"/>
  <Field DisplayName="VIN"
    Name="VIN"
    StaticName="VIN"
    ID="{F0000000-0000-0000-0000-000000000002}"
    Type="Text"
    SourceID="http://schemas.someusedcarinventory.com"
    Group="My Custom Columns"/>
  <Field DisplayName="Make"
    Name="Make"
    StaticName="Make"
    ID="{F0000000-0000-0000-0000-00000000003}"
    Type="Text"
    SourceID="http://schemas.someusedcarinventory.com"
    Group="My Custom Columns"/>
  <ContentType Name="Vehicle"
    ID="0x0100FC000000000000000000000000000001"
    Description="Used car inventory"
    Group="My Custom Content Types" >
    <FieldRefs>
      <FieldRef ID="{c042a256-787d-4a6f-8a8a-cf6ab767f12d}" Name="ContentType" />
      <FieldRef ID="{fa564e0f-0c70-4ab9-b863-0177e6ddd247}" Name="Title" 
        Required="TRUE" ShowInNewForm="TRUE" ShowInEditForm="TRUE" />
      <FieldRef ID="{F0000000-0000-0000-0000-000000000001}" Name="Model_Code"/>
      <FieldRef ID="{F0000000-0000-0000-0000-000000000002}" Name="VIN"/>
      <FieldRef ID="{F0000000-0000-0000-0000-000000000003}" Name="Make"/>
    </FieldRefs>
    <XmlDocuments>
      <XmlDocument NamespaceURI="http://schemas.microsoft.com/sharepoint/events">
        <spe:Receivers xmlns:spe="http://schemas.microsoft.com/sharepoint/events">
          <Receiver>
            <Name>VehicleCreatedHandler</Name>
            <Type>ItemCreated</Type>
            <SequenceNumber>1</SequenceNumber>
            <Assembly>My.Library, Version=1.0.0.0, Culture=neutral, 
                PublicKeyToken=c5168cbbbb64acf7</Assembly>
            <Class>My.Library.EventReceivers.VehicleCreatedHandler</Class>
            <Data />
            <Filter />
          </Receiver>
        </spe:Receivers>
      </XmlDocument>
    </XmlDocuments>
  </ContentType>  
</Elements>

Oddly enough, I was trying to figure out what the Filter element does when I stumbled upon the stackoverflow posting by accident. As it stands, I still can't figure out what the element is and what XML is valid for it. Perhaps some CAML query filter?

30Dec/09Off

C# and ASP.NET Syntax Highlighting in Trac

Posted by Charles Chen

Well, spent the good amount of time trying to figure this out. See the configuration info below from my trac.ini file.

[mimeviewer]
max_preview_size = 262144
mime_map = text/x-dylan:dylan,text/x-idl:ice,text/x-ada:ads:adb,
php_path = php
pygments_default_style = trac
pygments_modes = text/x-csharp:csharp:7,text/plain:aspx-cs:7
tab_width = 4
treat_as_binary = application/octet-stream,application/pdf,application/postscript,application/rtf

Oh yeah, it helps if you actually install Pygments, too.

Filed under: Dev, Self Note No Comments
7Dec/09Off

Host Headers, SharePoint (2010), and Access Denied (401)

Posted by Charles Chen

I've been working on setting up a VM to play around with SharePoint 2010 and kept running into a weird issue where I would be prompted for my credentials repeatedly and I was denied access to the site entirely, even when using an administrator account.


I had set up my SharePoint web application using a host header; for example, beta.dev.com.


Meanwhile, I was able to access the same site from my host environment without any issue.  What gives?  Turns out that there is a slight registry tweak you have to make when working with host headers and IIS.


Phil Harding has a good writeup with links to the Microsoft KBs and additional details.

10Oct/06Off

Self Reminder

Posted by Charles Chen

I've been meaning to put together a series of articles on how to build a system to support software automation (where automation is not supported natively) after completing what I would say is my most significant project of my career a few months ago (it was indeed, quite awesome to see in action.

What's the motivation for this? There are somethings that you simply cannot do via APIs that require automation of the UI. For example, what if you want to write an engine that will browse a site list and grab screenshots of the pages? What if you need to generate content on the server in Excel, Word, or PowerPoint (pre-2007)? What if you need to adjust and auto-publish a large number of project plans in Microsoft Project Professional?

These types of actions require an automation engine and a supporting framework to allow for manipulation of the UI and some clever tricks to take care of the exceptional cases since they were written to be used in interactive sessions. While I obviously cannot release any of the code I wrote previously line for line, I would like to discuss the strategies and some code snippets and use new code to cover this topic which I found to be lacking when I myself was searching for a way to do this.

As a summary, I think I'm going to break it down into a few chapters:

  • An Introduction: Automating Internet Explorer. As an example, we'll start by looking at the foundation of automating any of the Windows applications using .Net.  The ideas here apply not only to IE, but to almost any Windows application that exposes an API.
  • Simulating User Input. We'll continue the example by examining how to deal with cases where the application is expecting user input.  If so inclined, we could disregard the API completely and simply use this methodology throughout.
  • UI Mapped Input Sequences. We'll discuss how to take the example a bit further by examining how to trap UI elements and send the proper key sequences to handle them.

I think this will be a great series (once I actually get around to putting it together) as it will cover a whole host of technologies from Spring.Net to log4net to Enterprise Library.

Filed under: Self Note No Comments
23Feb/06Off

WinFX February CTP

Posted by Charles Chen

This is definitely something I'm going to have to download and play around with in the next few weeks.

Filed under: .Net, Self Note No Comments
30Sep/05Off

DHTML Games

Posted by Charles Chen

Finally figured out how to fix some of the bugs and problems I
had with jsTetris last night as I was laying down to sleep (I hate
that).

Each list element needs to have a property, State, which takes one of two values:

  • 0 when the list element is empty (no pieces in it) or the list element is holding an active block (a block that can still move).
  • 1 when the list element is holding a fixed block (a block that can no longer move).

One of the big problems I had previously was that it was difficult to
calculate when a piece could still move.  The method I was using
was too inefficient and took too many lines of code.  It tried to
examine each of the target blocks to see if it contained an inactive
block.  This should make it easier by performing some simple math
instead.

Each block of each piece must be examined in the direction of the move
and simply do a sum between each block and the list element in the
target block.  If the sum of the State is 0, then we can move the block.  If the sum is anything other than 0, then the block cannot be moved anymore.

Once a block stops moving, before the next buffered piece is moved (may
have to cancel timer), we sum across the bottom most row up 4 rows and
see if the sum in that row divided by the number of columns is exactly
1.  If so, this means that every list element in that row is
holding a fixed block and we can move get rid of that row.  Once
we've checked 4 rows, we can then move all of the remaining straggling
pieces down before reactivating the timer on the buffered piece.

Seems like a plan.

I was also thinking about doing two other games last night. 

One of them would be based off of pipe dreams.  Remember that game
where you have to move around blocks which contained pipes as this
green ooze started flowing?  I think it can be done with .gif
animations if each of the animations takes precisely the same amount of
time.  There are a total of 7 different pieces.  6 of those
pieces would need two sets of animations and one of them, a cross
shaped piece, would need 4 animations.  This brings the total
number of pieces up to 16.  The idea would be to cache each of
these images and swap them out as the flow of the ooze moves through
the pipes.  This one should be interesting.

The other game would be based off of a game that I had on my TI-83 back
in my highschool days which is kind of along the lines of Dr. Mario or
Tetris Attack, except without falling pieces.  Basically, you get
a random m*n board, which is completely filled with
pieces (maybe 5-6 different types).  When you have three or more
adjacent pieces, those pieces disappear.  And "loose" pieces fall
into the space left by those pieces and can cause "combos".  The
objective is to clear the entire board with as high a score as possible
(factoring in time and combos).  Like Tetris Attack (I don't
completely remember the mechanics of Dr. Mario), you can manipulate two
adjacent blocks in the grid at a time.

Neither of these seem too difficult and sound like fun.

Oh yeah, I'm also working on a ASP.Net Beta 2 based version of Chinese chess or Xiangqi
The idea is to use Atlas to allow two players to play in real time,
with the moves and chat being relayed using calls to web services from
the client.  So far, I have the board and objects laid out. 
I still have to figure out how I want to implement the game rules and
how complex to make it in terms of supporting multiple games at
once.  I'm also waiting for my Beta 2 hosting account from ServerIntellect, my webhost.

That's it for now.

Filed under: Self Note No Comments