Monthly Archives: July 2010



Five mistakes of a Scrum Product Owner that can make your project fail

by Nils

Some professions and positions are amazingly clear: The dentist fixes teeth, the architect plans houses, the journalist reports what happened. Being a scrum product owner is different – not because it’s more difficult than writing a great story, but because what is takes for a PO to succeed depends on so many different factors: the company culture, the type of project, the management style of the CEO, and last but not least the dynamics of the team that develops the product. Nevertheless, there are some mistakes that can jeopardize about any project, no matter how unique the circumstances. In the following paragraphs, we would like to share five of them that can make your project fail – or thrive, if you make sure to get this right from the beginning.

1 – “We’ll figure this out later” (Lack of product goal and vision)

Sometimes a project starts, but not every detail is clear. That’s okay – the beauty of Scrum is its adaptability to uncertainties. However, as time progresses and core structures are being built, the lack of a clear vision becomes more and more of a problem. It’s not the problem of the team – it’s the problem of the product owner, whose responsibility is to define the product goal and vision in cooperation with the client and communicate it to the team. This can be a tedious process, especially if the client is unclear about his goals or has difficulty articulating them. A good product owner is able to assist the client and take the lead when necessary to come up with a product vision that is comprehensive, thought-through and makes clear to the team where the project is heading.

2 – “Everything is important!” (Lack of product priorities)

If product vision is lacking, prioritizing the product- and sprint-backlog becomes very tricky. If the product vision is clear, but there’s no agreement on which factors determine the business value for the client, prioritization is almost impossible. Priorities should be based on product vision and business value – if either is unknown, the product owner has not worked hard enough to sort this out. The consequence are backlog priorities based on what “feels right” rather than solid arguments, difficulties explaining backlog priorities to the team, as well as difficulties defending the backlog against questionable changes requested by the client. If client and PO agree on what the project is about right at the beginning, a lot of questions and conflicts that may arise later can be settled quickly by reiterating the goal and vision of the product.

3 – “I don’t care how you get there, just do it!” (Lack of clarity on development path)

Another consequence of lacking product goal, vision and/or priorities is that deciding on the best way of how to structure the development of a project becomes unnecessarily difficult. In other words: If the WHY and the WHERE are not clear, the HOW won’t be clear either. Here’s an example: There are three big features to be built – a forum, a chat, and a calendar widget. If they are seen as three separate and independent chunks of work, starting with either of them is okay. In the end, they have the same business value to the client. However, if the product vision is to create an online community that connects people with like-minded individuals around the world, the calendar should come last – because it promotes local connectes rather than global ones. In this case, a clear product vision helps prioritizing in a way that the business values might not be able to.

4 – “Who wrote this story?” (Lack of physical or intellectual presence)

Do you have your own office room? Maybe even on the top floor with a great view, far above the developer cubicles? Nice – but probably not very good for your project. Being physically and intellectually present and avaible to the team is crucial to keep product development on track throughout the sprint. If the PO is separated from the team, the risk increases that questions remain unanswered during the sprint, resulting in stories that the team considers “done”, but the PO doesn’t accept. The more availble the PO is during the sprint, the more precisely the outcome will match his and the client’s expectations. Also, by being confronted with the questions that the team has during a sprint, the PO can improve his specifications for the upcoming sprint and make sure future stories are more clear. For the same reason, the product owner must physically attend the sprint planning meeting and the review meeting – plus the retrospective and daily scrum meetings, if possible. That doesn’t mean he has to talk during all of them – but he should be there to listen, if nothing else. PO attendance at these meetings cannot be delegated to the scrum master or another team member.

5 – “Let me ask the client about this” (Lack of decision-making power)

Clients can be a pain: Some don’t know what they want and expect the PO to invent their product. Others who know what they want control every move and forget why they hired a PO in the first place. Having a product owner who is not given sufficient authority and autonomy by his client or CEO is almost as inefficient has not having a product owner at all: If the PO cannot make decisions when fundamental questions arise during the sprint, it is likely that someone within the team will start filling in and taking on the role of the product owner. The consequence is similar to what happens when a PO is not physically or intellectually present: stories are misunderstood, questions remain unclear, and results won’t match the expectations of client and PO. In another scenario, CEO or client are taking over the role of the PO and start micromanaging the project and its team. The result: The PO’s knowledge and understanding of the product remain underutilized, and his ability to relay the product vision to the team is corrupted. Furthermore, the quality of product management decreases, because neither CEO nor client are able to be as present as a PO should and could be.

Conclusion

Obviously, there’s a lot more that can go wrong. However, starting with the PO when it comes to improving scrum performance is not a bad idea. In fact, I believe that most problems in a scrum project are symptoms of bad management and leadership rather than a lack of skills or discipline within the team. Therefore, doing whatever it takes to prevent or correct these five issues is worth the effort either way.

Note: This essay was inspired by Monica Yap’s presentation “Product Owner Anti Patterns” given at the Scrum Gathering Shanghai on April 19-20, 2010.

Posted in Scrum 1 comment(s)

Looking for a new scrum master

by Thomas

In scrum, the scrum master is an essential role. A scrum team that does not have a scrum master should better call itself a team working together on one project, rather than a scrum team. The role of the scrum master is clear, but how would you decide about who is going to be the scrum master in a team? What would be the best way according to scrum?

Recently in one of our teams we were looking for a new scrum master. There were basically two options.

Let the manager decide

When we started with scrum in our company more than two years ago, the team’s manager made the final decision about who is going to be the scrum master in the team. This is a clear decision, and the manager would have the possibility to install a scrum master whom he knows and whom he trusts.

No, let the team decide

But, wait, what does the scrum master have to do with the manager? The scrum master is a team member and is fully dedicated to the team, the scrum master serves the team.

That’s why we were thinking of letting the team members decide who will become the team’s new scrum master, rather than keeping them outside of the loop. This has several advantages:

  • The team acts as a team
  • Team members concentrate on team decisions
  • Team members learn to self-manage themselves
  • Team members understand that their voices are important

Since the team members are actively involved in the decision, it is more likely that they accept the result than if somebody from outside of the team appoints the scrum master.

Having a vote

After listening to the two different options, the team eventually decided to go with the second option. We scheduled a meeting for the election and we even prepared ballot papers with the names of the team members listed. Before the vote, we discussed again about the tasks and duties of “the perfect” scrum master and we also agreed that the new scrum master should prove themselves during a probation time of five sprints (almost four months in our case).

During the vote, each team member had one main vote and one second vote. The second vote would have helped us in case two or more team members received the same number of main votes. The election was held anonymously (which is very easy to achieve with a room big enough). We counted the votes for each team member and finally had a new scrum master for one of our teams.

Conclusion

I am satisfied with the process of our election and how the team discussed this matter. Luckily, the team was already very experienced and also had several good candidates for the new scrum master.

Concluding, I have to say that although it is not a big deal to have an election for the scrum master, I also did not find any sources that described this as a common practice. I was curious to try that out and fortunately the team was open for this opportunity and very curious, too.

Posted in Development, Scrum, TNC 2 comment(s)
Tagged

New possibilities offered by smart deployment scripts

by Mauro

In the past year we were using several methods to deploy our PHP code to the production machines. Now finally we have a deployment method which seems to be very stable and gives us many advantages.

Deployment methods that we’ve tried

The main problems during the deployments have always been the filesystem-related caches. We have two of those:

  • The higher level filesystem cache gets generated by our PHP framework. Its gathering a lot of configs from the whole filesystem, compiles them into PHP files and then stores them in some dedicated places.
  • On the lower level there is APC, a PHP module which does opcode caching. For performance reasons we’ve set the “file_stat” option to 0. This means that APC, once it cached a compiled PHP file, will ignore every change that is done to it.

When we first built the deployment scripts for the new environment we often had the problem that we didn’t know how to clear the filesystem cache safely.

Let’s assume we first clear the filesystem caches, then deploy the PHP code. During the time it takes to deploy PHP the filesystem caches already start getting regenerated which means that we end up with an inconsistent cache because it is partly generated by the old code and partly generated by the new one.

deployment1

On the other hand, let’s assume we first deploy code and then clear the filesystem caches. This would mean that after the deployment is finished we do have a clean site with consistent caches, but during the time of the deployment nobody knows what exactly is going to happen because there is already new code running with the old caches.

deployment2

We also do not want to rely on APC and just assume that it really caches every single file. If we could rely on the fact that it cached every single file, we could simply deploy the new code and then clear APC. But if we deploy twice in a short time span we can’t be sure that each of the PHP nodes already cached each of the PHP files.

Furthermore, none of the above described deployment methods allows us to do cache warming before we throw the code into the wild. On nightly deployments that’s no problem, unfortunately we often ran into the situation that we had to fix critical bugs and deploy during daytime.

A first step into the right direction

After going through many problems due to the above described issues, we decided that we can’t deploy on a machine while it is in production use. Fortunately we have quite a lot of machines in our PHP cluster, which allowed us to split them half-half. For this i need to say that we run Nginx in front of the environment, which then uses FastCGI to connect back to the PHP servers. In this setup it didn’t take much to make the Nginx temporarily only use the first half of the backend servers and let the deployment script deploy to the second half. Then swap the Nginx to use the second half for prod and make the deployment deploy to the first half. That way we could deploy cleanly and the above described problems were solved.

deployment3

The only problem with this solution was that we have many caches to clean and rsyncing the new code can take up to a minute, so altogether we divided the power of the cluster by two for around 5 minutes. Additionally that half which received the new code first, had to run the whole site in the second phase and regenerate its caches at the same time. During peak times we couldn’t afford that loss of computing power and we knew we had to find a better solution.

Final solution

I think we can say that we now, finally, found a solution which doesn’t have any disadvantages to the above described ones and it also solved all the described problems.

The trick is that we are versioning the directories on the PHP servers where our code is stored in. When the Nginx does the FastCGI request back to the PHP cluster, it always passes the absolute path of the PHP file it wants to have processed as part of the FastCGI header. On the Nginx it’s very simple to change the prefix for all those PHP files and then send all FastCGI requests simply to another version of the code on the backend servers while all the different versions of code on the PHP cluster can coexist in different directories. This solves the problem of the file-caching APC, because the absolute path of the same file in two different versions of code is different, because they reside in different document roots. The problem of the framework caches is also solved, because all the framework cache directories are specific to each version of the code and also reside together with the code in the versioned directory.

deployment4

I show an example snippet of our Nginx config to try to make this whole thing a little clearer:

  set $code_version 13;

  location ~* ((.*).php(.*)) {
    fastcgi_intercept_errors on;
    error_page 404 = @404; 

    set $script $uri;
    set $path_info "";

    if ($uri ~ "^(.+\.php)(/.+)") {
      set $script $1;
      set $path_info $2;
    }

    include /etc/nginx/fastcgi_params;
    fastcgi_param  SCRIPT_FILENAME  /srv/www/vhosts/$code_version.code/web$script;
    fastcgi_param SCRIPT_NAME $script;
    fastcgi_param REQUEST_URI $uri;

    fastcgi_pass   backend.cgi;
  }

Now the deployment script will simply deploy the new code into a new directory on all the PHP servers and then we request certain PHP files on each of the backend nodes to warm the cache. As final step it will change the line “set $code_version” to the new version and tell the Nginx to reload its config, without any user interruption and without crazy high load due to cache regeneration.

New possibilities

Since we now have multiple versions of PHP code on each of the backend nodes we can switch between them in seconds. Simply by editing the version number which is saved in the nginx config via a script, and then tell the Nginx to reload the conf. This allows us to, just in case we deploy something that causes problems, roll back without any bigger service interruption.

The coolest new possibility that we have now is that we can compare the efficiency of different code versions life in the production systems. We deploy two versions of the code into each of the backend blades into different directories without making the Nginx use the new versions. Lets say one version is 51 and the other one is 52. then we simply create a filesystem symlink with the name 53, on half of the nodes it points to 51 and on the other half it points to 52. Then we make the Nginx use the version 53, which then means that half of the nodes will run one version and the other half runs the second version. Once we want to switch everything to one of those two versions we simply make the Nginx switch to use version 52 or 51.

First half node:

rsid-a-20:/srv/www/vhosts # ls -lha
total 19M
drwxr-xr-x 71 user users 4.0K Feb 10 11:55 .
drwxr-xr-x  7 root   root  4.0K Dec  7 04:40 ..
drwxr-xr-x 14 wwwrun www   4.0K Feb  4 09:31 51.code
drwxr-xr-x 14 wwwrun www   4.0K Feb  4 09:31 52.code
lrwxrwxrwx  1 root   root    28 Feb  4 09:33 53.code -> /srv/www/vhosts/51.code

Second half node:

rsid-a-20:/srv/www/vhosts # ls -lha
total 19M
drwxr-xr-x 71 user users 4.0K Feb 10 11:55 .
drwxr-xr-x  7 root   root  4.0K Dec  7 04:40 ..
drwxr-xr-x 14 wwwrun www   4.0K Feb  4 09:31 51.code
drwxr-xr-x 14 wwwrun www   4.0K Feb  4 09:31 52.code
lrwxrwxrwx  1 root   root    28 Feb  4 09:33 53.code -> /srv/www/vhosts/52.code

That way we can compare multiple different versions of code while they are running on different backend nodes on prod and live monitor if one of them has some efficiency/load problems.

Posted in General 2 comment(s)

Join Our Team in Shanghai

Now hiring PHP Developer in Shanghai