Real World Agile - Finding Some Rhythm

It's been a few weeks since my last post - November seems to have come and gone and I was knocked sideways a bit by a nasty cold and a couple of weeks of unplanned work. We're still practicing Agile, though (or at least trying out best) and I wanted make an observation about how things are going in relation to rhythm. This is a bit of a ramble, but I wanted to get my thoughts out before too many weeks had gone by.

In music, the beat is key - it doesn't matter if it's four-on-the-floor dance music or experimental jazz, without rhythm it's just very hard to follow along. Rhythm provides predictability and stability as well as a routine. It is a pattern that makes it possible for anyone to follow along and jump in - even in the middle of a iteration.

We've been experimenting with one-week iterations, just to get our routine and rhythm down. Each week starts with a retrospective from the last week, moves into planning and then execution until the end of the sprint. In between we complete stories, deal with business as usual support and maintenance and start it all over again the next week.

The pattern is simple, but getting a consistent beat for the iterations has not been. We've had sickness, unplanned work, PTO and holidays to disrupt the cadence. Our velocity has been highly variable (which is not unexpected when first starting Agile) and the focus of our stories has been less defined than I'd like.

I think, however, that we are starting to hit our stride. The repetition, despite the variability, has brought predictability to the week, if not the results. We're finally starting to see a flattening of the velocity line and the way we approach the work is starting to gel.

The one thing we've noticed is that one week iterations seem a bit short for us - especially with a three-hour retrospective and planning block in the middle of it. Starting in January, we're going to experiment with a two-week iteration making the three hour block much more tolerable since it only occurs every other week.

Agile is definitely a process worth keeping, but it's hard work and not a silver bullet - especially if you're in an organization pulling you in several directions at once. It's something that takes time and commitment. Having a champion to drive the process and team buy-in is key. I've found that, so far, focusing on the basics and making small incremental improvements week-after-week has been the key to keeping the momentum. I look forward to finishing out the year and seeing our improvements in the next one!

Real World Agile - Week 3: How I Learned to Stop Worrying and Love the Story Board

Ever have one of those weeks where you were only working on one project, no support or maintenance calls came in and all your customers and stakeholders got exactly what they wanted?

Me neither.

Business As Usual
Without any system to handle customer requests, they just come in as they pop up. One week it's a feature on a product another week, a different feature and maybe in that same week a different customer has a feature she wants added on a different system altogether. Meanwhile, we are also handling bug fixes and regular maintenance requests, answering email and going to meetings.

In this chaotic system estimating how long a feature or product will take is difficult to impossible as we succumb to the tyranny of the urgent or whomever is most vocal and are left with a trail of half finished features, frustrated customers and only the most urgent (but not necessarily most important) things done.

Enter the story board
Under our current Agile experiment, things are a bit different. We still have the requests coming in week after week, we still have requests to handle, bugs to fix and meetings to attend; but we have something else as well - the story board.

If our previous take-it-as-it-comes system was chaotic, the story board is order. It is deceptive in its simplicity: a white board with rows of sticky notes. Green sticky notes on the left representing our stories, or major objectives, for the week and next to each story, a row of yellow sticky notes representing the tasks needed to complete the story.

The board is an artifact of our weekly planning session. We look at all the things that need to get done and choose up to ten stories for the week. But there's a catch - we've allocated a number of points to each story depending on how big or complex we thing it is. We can only choose stories that add up to the amount of points we completed last week; that's our budget. That's the general idea anyway - it's a bit more complicated than that since we're new at this and still trying to figure out what our weekly budget, or velocity, is.

There is one more thing that makes this system work. We have an agreement to only complete the stories on the board. New feature requests can come in during the week, but they won't be started until they get on the board, which could happen during the next planning session. The product owners decide what the most important features are for the week, prioritize the backlog and pick out as many stories as fit the budget.

The payoff: getting stuff done
The story board doesn't change the amount of things we need to get done, but it does help us get the most import things (as defined by the product owners) done. During the week we can see tasks and stories moving into the done column. The developers know that new requests will not interrupt the flow of the current work and requesters know that if it's important enough we'll be working on the story in less than a week. That means we're becoming more predictable, which managers and product owners like, even if they don't necessarily like waiting.

There is another intangible element to this that can't be understated - being energized by success. Seeing tasks and stories move into the done column is a good feeling. The visible indicator that we're actually accomplishing what we said we would is an affirmation that the Agile experiment is working. This is my favorite part of Agile so far - the cats are being herded.

Final thoughts
The story board isn't a silver bullet; we still have challenges. For instance, we've had to plan without the product owners present, making our best guess as to the priorities for the week. We also had to move our planning session a day later for one of the iterations, which throws off the rhythm of the week and made a bit of a hash of the point budget.

Along with the story board, we also use software to track story and task progress. The reason for this is we have a bit of a distributed team. Ideally, we'd all be able to sit where we could see the board and collaborate on the stories face to face. Unfortunately, we have one developer in a different time zone and some organizational distribution that makes true co-location impossible. Software is a good solution, but really isn't a substitute for a white board and sticky notes.

Even with these challenges, the story board has been a huge success in my mind. It gives me focus for the week, helps me prioritize my work and breaks big features down into bite size chunks.

Does this resonate with your experience? What are the challenges you've had with planning and completing stories and how did you overcome them? Share your thoughts - I'd love to hear them and I'm sure others would as well.

Real World Agile - Week Two - The Retrospective

In my last post, I wrote about our first week of using Agile planning and delivery techniques to better deliver value to our customers quicker. I mentioned a few of the things we started doing, a few we chose to forgo and where we met successes and challenges. One of the key disciplines that we did this iteration that we didn't do last time was the retrospective.

The retrospective is the time the team spends examining the last iteration, release or project to find what is working well, what are areas of improvement. The retrospective exposes strengths and weakness in the team's processes, interpersonal dynamics, technologies and out of it comes a continuous improvement plan. This week was my first official retrospective and in the spirit of retrospectives, I thought I'd do a retrospective on the retrospective.

The Process
Before I dive in to the specifics of our retrospective, I want to share the process that I'm trying to follow for our meetings. Much of the process comes Agile Retrospectives: Making Good Teams Great by Esther Derby and Diana Larsen.

It's actually a very simple progression:

1. Set the stage
2. Gather data
3. Analyze the data
4. Decide what to do
5. Close the retrospective

This is how brains are wired to make decisions, so it makes sense that the meeting would be structured this way. However, I found that it isn't quite as easy as it looks. We're problem solvers and we want to jump right into solving and deciding without the thorough data gathering and analysis needed.

This is where the book is really helpful for me. At least half the book is dedicated to activities a retrospective leader can use in each portion of the retrospective. The activities provide focus and keep the meeting both on topic and progressing toward its desired end. This is not easy to do and it is both an art and a skill. I'm hoping to improve as a leader the more I practice - having ideas and inspiration helps.

Our Focus
Since we had just come from Agile training, I wanted to focus on getting better at what we had learned and find the areas where we could reinforce our strengths and improve on our weaknesses. I was surprised at how well this set our agenda and direction for the coming weeks and led to some concrete improvement experiments which are already paying off.

Setting the Stage
In a retrospective, it's important to get everyone engaged early as the input of all team members is vital for success. This is particularly challenging for us as we have a member of the team, Erwin, who is not co-located and must participate via web conferencing.

After introducing the meeting's focus and going over the agenda, I asked each team member to share their thoughts from the Agile training or, for Erwin, his thoughts on our first iteration. This led to a frank discussion of the training which was valuable for me, especially since their experiences were much different than mine. We spent about ten minutes on this activity.

Gathering the Data
Once we finished the stage setting exercise, it was time to move into data gathering. It is counterintuitive, but most of the meeting is spent in the data gathering and analysis portions. Since our focus was our implementation of Agile, I had the team list every Agile practice they could think of. This seemed like a fairly non-threatening activity that allowed us to throw a bunch of terms and phrases on the board and also pull out what things most "stuck" from our training. This took five or ten minutes before we ran out of things to add.

Areas of Improvement:

• I should have limited the activity to five minutes and time-boxed it. This keeps things moving along and helps compel people to contribute because of artificial urgency.
• I should introduce each activity and its purpose. Team members need to know why they are doing something and what the intended outcome is.

This activity was foundational for the next one, which was to vote on those terms on the board (actually a list in a Word document we were sharing online) that were most important to focus on for the week. Everyone had four votes to use as they desired - four on everything, one spread across four items or any combination they wanted. My goal was to identify those areas most important to the team and begin to narrow down where our improvement would be focused for the week.

I placed asterisks next to each item a person voted for until we had all voted. We now had several ideas with one or more vote. Looking at the list we noticed that we had about five items with two or more votes, so we decided to take those as our focal areas. This activity took about five minutes.

Areas of Improvement:

• Ideally I would use sticky dots on a flip chart or marker on a board, but having a remote member makes this challenging. The phone works but it would be better to find creative ways of engaging everyone.

Analyzing the Data
Now we had some data to look at and I wanted to go deeper so that we could find which area would give us the most value as a focus of improvement. We took our five items and put them on the board in a radar graph which looked like the spokes of a wheel. I asked each team member rate the items from zero to ten in two areas: how are we doing in this area and how important is this area to our success for the coming week. Everyone placed a green dot on the spokes for the former and a red dot on the latter. Obviously, Erwin couldn't do this, so he put a colored number next to each item in the shared Word document. This took about five minutes.

Looking at the board, it was clear that two areas were most prominent: communication and defining "done done". I asked the team which item we should focus on first and we all agreed that communication should be first. I then moved on to my next activity which was designed to give us improvement actions for the next iteration. This was a rookie mistake.

Areas of Improvement:

• I used the radar graph to move us into the decision-making activity. This kept the meeting moving toward its intended conclusion, but missed an opportunity to go deeper on the topic of communication. I should have spent time asking questions about why communication was most prominent, where were we falling short and communication techniques were working. This would have been useful in exposing healthy or harmful team dynamics, areas where the team felt we were lacking and given us a much stronger improvement plan.
• Be sure to take time to go deep and really analyze the data. There is a healthy balance between keeping the meeting moving and making it truly valuable. Don't sacrifice the latter for the former.

Deciding What To Do
The next activity was a brainstorming activity. I liked this one because I could easily time box it, it could be done individually and out of it we could gather an actionable list of items. I set a few ground rules (aim for quantity, don't filter, don't judge, be wild and creative), gave everyone ten minutes and told them to write as many ideas about communication as they could. Ten minutes seemed long, but I wanted to give people time to think and write (or type).

At the end of the ten minutes I gathered up all the ideas on a shared spreadsheet so we could look at them and begin filtering. I didn't anticipate how difficult it would be to come up with filters, but the team did a great job coming up with a few such as: provides value to team and is no (or little) work (e.g. simple to implement).

It was difficult to filter our list but in the end after a discussion we came up with four things we could do this week to improve communication: use IM (we're mostly an email shop), look into better co-location of team members, a weekly informal lunch and end-of-day check-outs. After two days, I can say we've implemented two of the four (IM and check-outs) and it has definitely improved communication. This activity took about twenty minutes.

Areas of Improvement

• The brainstorming and filtering activity needed to be better focused. Introduce what we're doing, why we're doing and what we hope to get out of it
• Brainstorming activities need to be focused. I should have said "brainstorm ways we can improve communication". It was implicit, but should have been explicit

Closing the Retrospective
Closing is important. It allows the team to commit to the decisions made, celebrate accomplishments and do a mini-retrospective on the retrospective. That last item was very important for me. I am fortunate to have members on the team candid enough to let me know what can be improved and some of the notes above came out of those comments.

Conclusion
Retrospectives are hard. They are hard to do well and hard work for the participants. For a first go I felt this went very well, but there are obviously areas of improvement to be made. I think practice and feedback will improve the retrospectives as we continue to do them. This applies both to me as a leader and participants. The real value of the retrospective, however, isn't in the meeting itself but the results it achieves. I'm looking forward to the continuous improvement that comes out of this regular meeting.

Real World Agile - Week One

Two weeks ago, I had the privilege to participate in an Agile Planning and Delivery training with James Shore and Diana Larsen. This week we had the chance to put that training into practice. While the training was very hands-on and interactive (we actually did four iterations and built working software the last three days), it was also in an artificial environment. In the real world, things didn't quite fit into the neat patterns we practiced in class. Here's I present what we did, what we didn't do, what worked and what needs improvement after week one.

TL;DR Version

• Task Board and Test Driven Development (TDD) are Very Good Things.
• Software tracking tools, not as useful as a rolling white board and post-it notes.
• Early practice shows promise and we'll continue to build on what we've learned.

Our Early Attempts

There's a lot more to Agile than the ceremonies and tools - or maybe a lot less. In the past we focused on story planning, iterations and a software tool to help us keep track of what was going on, but all we were really doing was cramming a waterfall methodology into one week sprints. Needless to say, that didn't work. We didn't gain much from daily stand up meetings, putting all the stories, tasks and estimations into a tracking tool or having a scrum master. We still ended up trying to cram too much scope into too little time with the added complexity of a new methodology run by an inexperienced coach. It could have gone better.

What We Did This Time

Fewer Stories So Could Confidently Commit

One of the key points that I brought away from our training was to deliver value with each story. Even if the value is small, it's still an incremental gain for our customers. Rather than plan a project up front and stuff all the stories into iterations several weeks into the future, plan the week's stories and only commit what can actually get done.

That last point is worth reiterating - only commit to what can actually get done. Completing stories energizes the team and gives us a real sense of accomplishment toward our end goal. We had previously guessed wrongly and over-committed and got farther and farther behind with our stories. This week, we took a look at all the possible stories, prioritized them and only committed to a handful (no more than 6-10 is a good rule). The rest went into the backlog.

What we knew intuitively, but failed to account for in our previous attempts was a buffer for technical debt, regular support and maintenance of other software and administrative overhead (meetings, email, interruptions, etc.). This time we left plenty of buffer and committed to paying down technical debt if there was buffer time we could use.

Test Driven Development (TDD)

I had previously advocated Test Driven Development to the team, but it was new, seemingly complicated and I wasn't very good at it myself. I was familiar with the concept and had done it myself, but didn't do a good job at explaining the technique or the value to the team. Luckily the training spent a fair amount of time on this concept with real-world examples and practice. And as fortune would have it, this week an excellent online meetup, "Test Driven Development Demystified", occurred and we could all listen in to help reinforce our learning. With that baseline established, we are committing to TDD as a regular practice for our coding.

One challenge we face with TDD is that we have piles of legacy code. We chose not to try to do any retro-testing of legacy code, but rather write unit tests for all new code and test old code as we end up working with it. Part of paying down our technical debt will be refactoring old code with tests, but only as the system and time demands. Our rule is "make the code better than it was before". Small improvements add up and eventually we will be fully covered.

What We Didn't Do

Demonstrations

We are committing to one day a week (we chose Wednesday) as our demonstration, retrospective and planning day but since we only had three days before base-lining our iteration schedule we chose to skip the demonstrations this week. Next week will be a different story.

Retrospective

For the same reasons we didn't do any demonstrations, we skipped a formal retrospective. Instead, we chose to talk about what we learned in the training and commit to a few key concepts (TDD and story planing being the most prominent). We also used this time to estimate stories and add stories them to our task board.

Use a Software Tracking Tool

Speaking of a task board, we chose to skip a software tracking tool for our iteration planning and use sticky notes and a rolling white board instead. The board is visible to everyone in a communal area and it's very nice to interact with something tangible for planning and doing stories and tasks. The board did eventually get translated into the software tool, but that is only because we have a developer who isn't co-located and we need a way of sharing the task board across time zones.

What Worked

The Task Board

Of all the things that are really working this week, the physical task board has to be the winner. Prominently displayed stories and tasks (completed and in progress) is a great reminder of what we're working on and what we got accomplished. Planning using the board was also much nicer as we could move stories around in priority order and drop them into the iteration much quicker than fumbling with a software tool. This is definitely a keeper and I'm actually excited about planning next week's stories.

Daily Stand Ups

We went back to a five minute stand up - one minute per team member. We can check in, get a level set for the day and move into planning and clearing road blocks immediately afterward. We are using Scrum's "did, doing, impediments, confidence" questions which is working for us. When it becomes tedious or boring, we'll switch it up to something that works for us.

Areas of Improvement

TDD

We're all practicing TDD now, but I didn't feel I was able to really get in and write the tests I wanted to for the week. I saw plenty of room for refactoring, but a couple of in-progress stories that had hard deadlines meant I only had time to do some incremental improvements to code, rather than create fully tested objects. That being said, we spent a lot of time learning the process and tools, so it wasn't a failure, just an area for improvement.

Planning

Our planning was a bit rushed this week. Impending deadlines and being short-handed due to illness and planned PTO meant we had to crash through our planning session. We were missing our primary product owner since he was at a conference, so we did the best we could with the people we had. Like TDD, not a total failure but I'm confident we will be better next week.

Summary

Agile in a classroom and Agile in the real world are two different animals. In the classroom there aren't meetings, email and support calls to take and everything works (almost) according to the book. As a way of building software, it's incredibly useful, but we're going to have to slowly incorporate all the best practices as we're able to absorb them. I would say the first week was a huge success at small improvements - each of which will add up to better efficiency, better code and better value for our customers. It might just take a few months to get there.

Additional Resources

Test Driven Development

• MXUnit - This is a unit testing framework for ColdFusion. If you do CF, I highly recommend it.
• James Shore's Let's Play TDD - Jim goes through an actual Java software project from scratch using TDD. You get to see it built up warts and all. It's incredibly useful to see it in action.
• Test Driven Development Demystified - this online meetup was an excellent one hour presentation on TDD and Katas. Like the Let's Play, a demonstration of TDD happened live (it's now recorded and available).
  Slides and Code

Agile

There are a ton of resources on Agile out there. Here are a couple to get you started.

• James Shore's Blog
• Diana Larsen's Blog

What Do You Think?
Do you have a question or comment about Agile? Have you tried it or are currently doing it? I'd love to hear your experiences and what worked and didn't work for you. I'd also like to know if you have additional resources I should add to my list. Thanks!

And So It Begins...World Cup Knockout Rounds!

This is a decidedly off topic post, but given Team USA's game against Ghana tomorrow, I just can't help myself. The knockout rounds are upon us and we've got a great chance to advance quite far coming out of the group stages on top. And so here I offer my video support for the home team.

Weezer - Represent



Video Montage of Crowd Reactions to Victory Over Algeria



We'll be watching tomorrow - I hope to be celebrating!

Using WS Security for SOAP Requests in ColdFusion

UPDATE: I've posted the source on GitHub: https://github.com/anthony-id/cfWSAuthenticator

Late last year, I had need to use a third party API that required WS Authentication. I had the WSDLs, I could get it to work in SOAP UI but when I tried to get it to work in my code it just wasn't happening. It turns out that WS Authentication works out of the box with Axis 2, but ColdFusion creates all its web services with Axis 1, so I had to create the client myself.

After quite a bit of research, it turns out this is quite easy using a couple of Java libraries and ColdFusion's "CreateObject" function. Here's how I got it to work:

First download the following libraries:

xmlsec-1.4.2.jar - http://www.aleksey.com/xmlsec/

An updated version is available, but this is the one I have working

wss4j-1.5.8.jar - http://ws.apache.org/wss4j/
This is the library that does most of the work - Rampart is the portion that Axis 2 uses for this, but we needed to just utilize the signing portions for our purposes.

The libraries need to go to [cf_home]/lib or be added to your classpath. A server restart may be necessary for ColdFusion to be able to see them.


Now the jars are available for use. I created two functions to get these to work - one to do the signing and one to send the request. I have a component in which these live, but you could do the whole thing inline if you wanted to.

The one downside to this is that you'll need to construct the SOAP requests manually. This isn't hard if you use something like SOAP UI - I just grab the XML from there and modify as necessary. I place the SOAP request inside a cfxml block.

<cffunction name="addWSAuthentication" access="public" output="false" hint="I sign SOAP envelope using WS Authentication">
<cfargument name="soapEnvelope" type="string" required="true">
<cfargument name="username" type="string" required="true">
<cfargument name="password" type="string" required="false">
<cfscript>
// Create Java Objects from xmlsec and wss4j
var WSConstants = CreateObject("Java","org.apache.ws.security.WSConstants");
var msg = CreateObject("Java","org.apache.ws.security.message.WSSAddUsernameToken");
// Get Soap Envlope document for Java processing
var soapEnv = arguments.soapEnvelope;
var env = soapEnv.getDocumentElement();
var e = "";
// Set Password type to TEXT (default is DIGEST)
msg.setPasswordType(WSConstants.PASSWORD_TEXT);
// Create WS-Security SOAP header using the build method from WSAddUsernameToken
e = msg.build(env.GetOwnerDocument(),arguments.username,arguments.password);
// Add the Nonce and Created elements
msg.addNonce(e);
msg.addCreated(e);
// Return the secure xml object
return soapEnv;
</cfscript>
</cffunction>

This function sends the soap request and returns the result. The endpoint and SOAP Action can be found in the WSDL or by using SOAP UI. In SOAP UI, look at the raw post, the SOAP Action is in the headers. Also note the additional headers in the cfhttpparams - these are necessary to make the WS Authentication work.

<cffunction name="sendSoapRequest" access="public" output="false" hint="I send the SOAP request off retrun the SOAP response">
<cfargument name="endpoint" type="string" required="true">
<cfargument name="soapEnvelope" type="any" required="true">
<cfargument name="soapAction" type="string" required="false" default="">
<cfset result="">
<cfset soapenv="">

<cfhttp url="#arguments.endpoint#" method="POST" result="result">
<cfhttpparam type="Header" name="Accept-Encoding" value="deflate;q=0">
<cfhttpparam type="Header" name="TE" value="deflate;q=0">
<cfhttpparam type="header" name="SOAPAction" value="">
<cfhttpparam type="xml" value="#toString(xmlParse(arguments.soapEnvelope))#">

<cfreturn>
</cfreturn>
</cffunction>

Hopefully this will save you some time if you ever have to do WS Security in ColdFusion. It took me a couple of weeks of trial and error to get this working, but I'm now using it for two different third party APIs without a hitch.

Acknowledgments
The following were critical in helping me get this solved:
Ben Nadel's Blog - Listing All Classes In A Jar File Using ColdFusion And CFZip
Steven Erat's Blog - Workaround for CFHTTP and Compressed HTTP Response from IIS

CF.Objective() Pecha Kucha Presentaion

Here is my Pecha Kucha presentation from CF.Objective().

What is Pecha Kucha? In a sentence, it's a talk about anything you're interested in that lasts six minutes and forty seconds. Why 6:40? Because that's 20 slides for 20 seconds each that automatically advance. Ignite is based on Pecha Kucha with the primary difference being those slides advance every 15 seconds.

I hope to get the slides up here as well. Until then, enjoy and let me know what you think. I had a great time presenting and hope to get a chance to do it again!

Head over to Bob Silverberg's Blog to see all the PK talks.

All the talks in succession are also posted on YouTube as a playlist.