14.0 Literature Survey 14.1 Introduction Large software companies use several methodologies for

14.0 Literature Survey
14.1 Introduction
Large software companies use several methodologies for software development. One such popular methodology in use is the waterfall method. The waterfall development methodology proceeds linearly through the phases of requirements analysis, design, implementation, testing (validation), integration and maintenance (Royce, 1998). The serial approach to product development has a few disadvantages, notable of which are, the inability to make mid-course corrections and the lengthy period of development before the product can reach the end user. Some companies have successfully adopted a more iterative methodology called the agile development methodology. The agile methodology allows the company to build the smallest possible useful part and makes it available it to its users (instead of building the whole product), who provide feedback iteratively on its usefulness. Agile development is an evolutionary conversation in which incremental steps lead to feedback that allows requirements to be tested and adjusted. However, with its heavy emphasis on customer interaction, self-organizing teams, verbal-communication over-written-documentation, prototyping and requirements flexibility, it may pose a challenge to implement the agile methodology in large software companies.
This research will focus on a few areas that will help in implementing agile methodologies in large software companies. First, it will provide the large company context to the agile manifesto. Second, it will identify the impediments to implementing agile methodology in a large software company. Finally, it will identify best practices and areas where agile can be tuned for adoption by large software companies.
Agile software development is a methodology that prescribes four focal values, namely, individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation and responding to change over following a plan (Abrahamsson, Salo, Ronkainen, & Warsta, 2002).
14.2 Individuals and Interactions
Among the focal values of the agile methodology, the most important value that an organization needs to adopt is in the realm of people factors (Boehm & Turner, 2003). In existing agile practices this manifests itself in co-located teams that have excellent communications between them and the central theme is team spirit (Abrahamsson et al., 2002). In his study about agile methodologies and its impact on people, Boehm (2003) argues that software engineering is done “of the people, by the people, and for the people” (p. 4). It could be loosely translated as people organizing themselves in groups to develop what other people, within or outside the organization need. He goes on to add that in an agile culture, people are like craftsmen who feel empowered to solve problems. Agile practitioner Rick Dove highlights the importance of increased interaction between people as a key success factor to an organization being agile (Dove, 2001). He succinctly states that training doesn’t bring as much diversity of thought as better collaboration among employees does. Boehm (2003), however, cautions that agile methods rely heavily on communication through tacit, interpersonal knowledge for its success and hence relying completely on tacit knowledge is like performing without a safety net. In defining best practices for adaptive systems, the author (Meso, 2006), presents the argument that organizations that use heavy process in the product development cycle should look to adopting agile practices if they are to produce high-quality software in a cost effective and timely manner. In adopting agile the emphasis is on people and their interactions rather than heavy process.
14.3 Working Software
The second focal value that agile practitioners advocate is emphasis on working software over comprehensive documentation. Abrahamsson (2002) emphasizes the need to lessen the documentation down to a minimum by keeping the code simple, straightforward and self-documenting. The objective of self-documenting code is to enable the reader to understand the source code, its design and function without having prior knowledge of the system. This is achieved through the use of a set of defined conventions for naming software programs, functions and variables (Schach, 2002). Boehm (2003), points out that in traditional waterfall process documentation could become heavy as the process requires every foreseeable situation to be documented. For example, a user manual would have to provide guidance to users on every possible error or exception that a user might encounter. In his book titled Running and Agile Software Development Project, the author emphasizes creating simple code that is clear and empowers the target reader with enough detail to change, update, or develop it further with ease (Holcombe, 2008). Citing the limited utility of comprehensive documentation, the book Agile Project Management: Creating Innovative Products, argues that documentation at best can provide only 15 to 25% of the understanding (Highsmith, 2004). Highsmith (2004) characterizes documentation as providing only content and not context. They key takeaway from these readings is agile software development does not separate code and documentation as two separate artifacts and that code should be self-documenting.
14.4 Customer Collaboration
Customer collaboration is the single most impactful agile value that brings the development team and the customer together (Hanssen & Fægri, 2006). In a longitudinal case study, Hannssen (2006) studied a software company that adopted agile development methodology and presented his findings showing prerequisites, benefits, costs and risks in a software product setting. He states that the close customer cooperation had a highly motivating effect on the developers’ of the company. Further he observes that, the developers’ confidence has increased as a result of continuously meeting customer expectations. He concludes that meaningful engagement of customers is an imperative for innovation in product development. Another research (Molokken-Ostvold & Furulund, 2007) that studied the relationship between customer collaboration and software project overruns concludes that good collaboration with customers, facilitated by frequent communication, was associated with projects that experienced a lesser magnitude of effort overruns.
14.5 Responding to Change
Benjamin Disraeli, former British Prime Minister, said “change is inevitable, change is constant” (Lugon & Secker-Walker, 2001). Responding to change is the fourth focal value of agile development. Agile development is built on a different change model from that of traditional development models and it assumes incremental change and continuous improvement (Kelly, 2008). To a company that is adopting agile methodologies; incremental change means that requirements are going to change based on frequent feedback and hence will evolve as the development progresses. As a result of this evolution, improvement based on user feedback is continuous, unlike traditional models where improvement happens as a separate development cycle, i.e. post initial development. It is expected that agile development groups are competent enough to understand the change and have the authority to make changes as needed during the development lifecycle. A{{}} review paper (Misra, Kumar, & Kumar, 2008) that provides a conceptual framework illustrating the relationships between the different predictor variables and agile software development success, suggests that change is considered expensive in plan-driven methods. The author provides empirical analysis that show agile methodologies thriving in high change environments. The benefit of the high change behavior is it removes the element of surprise when the end user receives the final product as it has been refined constantly to meet the end users’ requirements. In another review that investigates the benefits and limitations of agile methods, the authors’ define agility as a means to strip away as much of the heaviness as possible to promote quick response to changing environments and changing user requirements (Dybå & Dingsøyr, 2008).
14.6 Impediments to Implementing Agile in Large Software Companies
Hierarchical & Bureaucratic Organization
An organization’s culture could be the first impediment towards adopting agility. A study conducted by (Iivari, 2007) states that large organizations tend to have a hierarchical culture that is oriented toward security and order. The emphasis in such organizations is on control, stability and efficiency through the following of strict regulations or processes. Managers of these organizations may have to manage larger teams and this eventually leads to less direct control of their teams and more indirect control in the form of reports. Eventually this leads to layers between the accountable manager and the developers. A case study of a financial organization that develops large-scale, complex software using agile methodologies validates that deep hierarchical organizational structure results in unresponsive environments with high inertia (Cao, Mohan, Xu, & Ramesh, 2004). It is to be noted that there are no agile principles that deal with organization structure and hence attempting to adopt agile methodologies in an organization with hierarchical structure and bureaucratic culture may not produce the desired result.
Management Alignment
Adopting agile methodologies is a substantial change and agile implementers may have to deal with the reality that not everyone is willing or able to change (Benefield, 2008). In a retrospective look at her experiences in implementing agile methodologies at Yahoo!, Benefield (2008) found that 85% of the employees at Yahoo! were happy with the current state of development and not receptive to change when she first started the implementation. Getting the management to align with the change could be challenging especially in large companies where there are several layers of management. Benefield (2008) found that she would have gained more acceptance from the management had she been able to educate them on the agile values early on in the process. In an attempt to cultivate adoption from the ground-up in a grassroots fashion, Yahoo! adopted agile methodologies only for certain projects and this lead to a few problems. Some of the managers from projects that were not adopting agile felt left out and hence became negative proponents of the program. Also, the decision to train only a few managers on agile methodologies led to a lack of understanding across the rest of the organization. This resulted in resistance to its adoption and eventually dismissal of the methodology. An expected outcome of adopting agile methodologies is teams that are more self-organized and require less management supervision. The management will have to be trained to manage the transition from a traditional command and control model to a decentralized model (McDonald & Welland, 2005). This is further reinforced by the book Agile Adoption Patterns – A Roadmap to Organizational Success which studies the progress of agile adoption by a large software company. The author, Elssamadisy (2008) suggests that a key success factor for agile adoption is the training of both the development team and the management on agile methodologies (Elssamadisy, 2008).
Dilution of Core Agile Values
Implementing agile methodologies require discipline and if core values are diluted or modified there is a danger that agile may not produce the desired result. Benefield (2008) found that one of impediments to implementing agile is dilution of the core agile values. She recounts that at one point during the implementation, many teams were claiming they were using agile techniques, but were actually dropping key elements such as the daily meeting. Another core value that could be diluted is the equal participation of all team members. Agile espouses the importance of cohesive teams, the empowerment of these teams, and the collective ownership and self-evaluation of work by the team. However, this value could be diluted when individual members of a team demonstrate a tendency not to work as hard as they could or should (McAvoy & Butler, 2009). The phenomenon is termed “social loafing” and occurs because the team provides a degree of anonymity and hence the individual feels their lack of work will be hidden from evaluation within the overall output of the team.
Access to Customers
Customer involvement and collaboration is a key factor for successful adoption of agile methodologies. However, in reality, access to customers is often difficult. This is due to a variety of reasons. In large projects, the customers may be from a variety of domains and hence challenging to access all of them at one time. In a distributed project the developers may be spread across several locations and hence without direct access to the customer. Sometimes accessible customers are often not the end users of the system. There might also be occasions where the software being developed maybe a product for which the end user is still unknown. In a study at IBM, that discusses the positive and negative forces that affect the adoption of agile software development in a project, the authors’ highlight the non-availability of an onsite customer as a major disadvantage (Grossman, Bergin, Leip, Merritt, & Gotel, 2004). Despite having a product manager representing several different stakeholder groups, the development team struggled without interaction with the customer.
Scalability of Agile Methodologies
Critics of the agile methodology have questioned its scalability. They point out that with larger teams, the number of communication lines that have to be maintained can reduce the effectiveness of agile practices (Turk, France, & Rumpe, 2002). In his book Agile Software Development – The Cooperative Game, Cockburn addresses the issue of agile development not scaling for large organizations (Cockburn, 2007). He states that a well-structured, 10-programmer team using agile methodology may be able to solve a larger problem than a 30-person team using a larger methodology. Another view on scaling agile teams is expressed by Highsmith in his book. His recommendation is to avoid scaling the project based on hierarchy, control, documents and ceremony as this will lead to compliance activities to dominate delivery as each level in the hierarchy justifies their existence (Highsmith, 2004).
14.7 Tuning Agile Values to Work in Large Software Companies
Flat Technology Organization
In his book titled Organization Patterns, Coplien (2006) discussed how large organizations tend to build deep hierarchical structure which results in unresponsive environments. Though agile methodologies do not dictate an organization structure, it demands increased and frequent communication between stakeholders. Coplien (2006) adds that a slim and flat organization is expected to be more responsive due to the reduced layers of management. Iivari (2007), in his study, states that for organizations to successfully adopt agile development they require a flat organization with lighter process and less bureaucracy. He adds that a flat development organization empowers developers to make technical decisions and promotes decentralized development-oriented decision-making. He cautions that the empowerment should be controlled in an elegant manner through the committed use of standard interfaces and design patterns.
Another aspect of an agile software organization is the technical leadership skills. An experimental study conducted in Norway suggests that software professionals in technical leadership roles provided better time estimates for software projects and were better equipped to make quick technical decisions than those leaders with a non-technical background (Moløkken & Jørgensen, 2005). The study also attributes the performance of non-technical leaders to their lack in depth knowledge in technical details. A key take away from the study is that human resource management differs in agile software development and hence the leadership requires people with technical knowledge and experience.
Design Patterns and Framework Development
An aspect that large software companies are challenged with while adopting agile methodologies is upfront architecture and design. Traditional software development methodologies consider upfront architecture and design as essential for the successful implementation of a software project. On the contrary, there is an emphasis on iterative architecture and design while developing using the agile methodologies. To facilitate iterative development without the full architectural design, agile uses the concept of a metaphor to describe the basic elements and relationships of an application (Beck & Andres, 2004). A method to achieve iterative design but yet have a solid foundation to build on is to identify design patterns within the product domain and build a framework for each of the design patterns (Gamma, Helm, Johnson, & Vlissides, 1995). Gamma (1995) recommends that once the framework has been built for a pattern, the application functionality can be built iteratively on top of the framework with relative ease and speed.
Strong Product Management Organization
There is a need for continuous customer engagement in agile development projects and direct access to customers is not always available. The problem could be overcome by having a strong product management team that will act as proxy to the real customers. An agile project treats dates and budgets as fixed and hence scope is the only variable. A core responsibility of the product manager is to determine the scope of features and the depth of each feature that is required in the product. The scope can be determined only with extensive customer input throughout the project. There is also the need to set the scope priorities up-front, and continually refine them throughout the project. Refinement keeps the priorities synchronized with the changing view of dependencies, complexity and the ability to execute, as the project progresses. Product managers are also expected to meet with the development team frequently and iterate over requirements, negotiate and modify them as required. These meetings could also be used to make decisions on development options. The practice of frequent meetings with the product manager led to better project outcomes when compared to projects that had infrequent customer engagement (Glazer, 2001).
Creating an Agile Community
Margaret Mead, an American cultural anthropologist, said “Never doubt that a small group of thoughtful citizens can change the world. Indeed, it is the only thing that ever has” (Sergiovanni, 1996). Organizational change takes time and in the middle of change, the change itself is not evident and probably doesn’t move as fast as organizations would like it to (Silva & Doss, 2007). Adopting agile requires perseverance and continued reinforcement. Further, the adoption of agile exposes several organization issues and they have to be resolved by a collective group of engaged and committed people. An agile community of like-minded people who can provide support and mentoring for projects and the organization will go a long way in ensuring successful adoption. This community should evolve into an influential body of change agents, which routinely meets to address team or organizational initiatives in agile projects across the enterprise.
Code Review Culture
In an agile environment, the development team is expected to make frequent high quality releases to its end users. In traditional processes quality is ensured by a quality assurance team. However, the serial nature of quality assurance after development is completed is rather heavy in an agile environment. The concept of a peer code review will help in such situations. Peer code review or pair programming is where two programmers develop software side by side and review each other’s code. An empirical analysis to study the costs and benefits of pair programming found that for a development-time cost of about 15%, pair programming improves design quality, reduces defects, reduces staffing risk, enhances technical skills, improves team communications and is considered more enjoyable at statistically significant levels (Cockburn & Williams, 2001). Although pair programming is seen as a good practice, it is not considered to be realistic in all situations (Elssamadisy, 2001). Allowing for certain functions to be done using pair programming has yielded significant benefit in a financial organization (Cao et al., 2004). The study by Cao (2004) states that certain functions in the development cycle such as analysis, design, test case development, and unit testing were done in pairs. However, developers had the option of coding in pairs or solo. The level of pairing was customized to suit to the nature of the developers and the nature of the tasks that they perform.
14.8 Conclusion
Large software companies can produce high-quality software in a cost effective and timely manner using agile software development if they change a few things about the organization. Changing the organization culture is the first step towards agility. Lighter and less bureaucratic organizations will be able to successfully adopt agile development methodologies. It is also important to have the right level of technical leadership at the management layers. A manager should have enough technical knowledge to make the right decisions. The management should be fully aligned with the effort to adopt agile methodologies and relevant stakeholders should be trained on agile methodologies. To sustain and grow the knowledge within the organization it is also recommended that an agile community of like-minded agile practitioners is created. Continued management engagement with the agile methodology is a requisite for successful adoption. In the process of adoption there may be attempts to dilute the core values of agile methodologies. It is important to stay on course and ensure that core values are implemented without any divergence. The agile community within the organization should govern the adoption efforts and ensure compliance. A strong product management team should act as the bridge between the customer and the development team. This team should ensure that customer feedback is captured; determine the scope and priorities are refined as the project progresses. Finally, there are several software best practices that can help in agile adoption. Design patterns help identify common business patterns and allow for creation of software frameworks that abstract that pattern. This helps reduce development time when incremental functionality is needed to build on top of that pattern. Pair programming creates a culture of peer code reviews, which has helped improve the quality of code and reduce the development time.
15.0 References and Bibliography
Abrahamsson, P., Salo, O., Ronkainen, J., & Warsta, J. (2002). Agile software development methods. Relatório Técnico, Finlândia,
Beck, K., & Andres, C. (2004). Extreme programming explained: Embrace change Addison-Wesley Professional.
Benefield, G. (2008). Rolling out agile in a large enterprise. Hawaii International Conference on System Sciences, Proceedings of the 41st Annual, 461.
Boehm, B., & Turner, R. (2003). People factors in software management: Lessons from comparing agile and plan-driven methods. CrossTalk,
Cao, L., Mohan, K., Xu, P., & Ramesh, B. (2004). How extreme does extreme programming have to be? adapting XP practices to large-scale projects.
Cockburn, A. (2007). Agile software development: The cooperative game Addison-Wesley.
Cockburn, A., & Williams, L. (2001). The costs and benefits of pair programming. Extreme Programming Examined, , 223-248.
Coplien, J. (2006). Organizational patterns. Enterprise Information Systems VI, , 43-52.
Dove, R. (2001). Response ability: The language, structure, and culture of the agile enterprise Wiley.
Dybå, T., & Dingsøyr, T. (2008). Empirical studies of agile software development: A systematic review. Information and Software Technology, 50(9-10), 833-859.
Elssamadisy, A. (2001). XP on a large Project–A developer’s view. Proceedings of the XP Universe Conference,
Elssamadisy, A. (2008). Agile adoption patterns: A roadmap to organizational success Addison-Wesley Professional.
Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object-oriented software Addison-wesley Reading, MA.
Glazer, H. (2001). Dispelling the process myth: Having a process does not mean sacrificing agility or creativity. Crosstalk,
Grossman, F., Bergin, J., Leip, D., Merritt, S., & Gotel, O. (2004). One XP experience: Introducing agile (XP) software development into a culture that is willing but not ready. Proceedings of the 2004 Conference of the Centre for Advanced Studies on Collaborative Research, 242-254.
Hanssen, G. K., & Fægri, T. E. (2006). Agile customer engagement: A longitudinal qualitative case study. Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering, 164-173.
Highsmith, J. (2004). Agile project management: Creating innovative products Addison Wesley Longman Publishing Co., Inc. Redwood City, CA, USA.
Holcombe, W. M. L. (2008). Running an agile software development project. Hoboken, N.J.: Wiley.
Kelly, A. (2008). Changing software development : Learning to become agile. Chichester, , GBR: John Wiley & Sons, Ltd.
Lugon, M., & Secker-Walker, J. (2001). Advancing clinical governance RSM Press.
McAvoy, J., & Butler, T. (2009). The dilution of effort in self-evaluating development teams: Agile loafing.
McDonald, A., & Welland, R. (2005). Agile web engineering (AWE) process: Perceptions within a fortune 500 financial services company. Journal of Web Engineering, 4(4), 283.
Meso, P. (2006). Agile software development: Adaptive systems principles and best practices. Information Systems Management, , 19-30.
Misra, S. C., Kumar, V., & Kumar, U. (2008). Success factors of agile software development.
Moløkken, K., & Jørgensen, M. (2005). Expert estimation of web-development projects: Are software professionals in technical roles more optimistic than those in non-technical roles? Empirical Software Engineering, 10(1), 7-30.
Molokken-Ostvold, K., & Furulund, K. M. (2007). The relationship between customer collaboration and software project overruns. AGILE 2007, 72-83.
Royce, W. (1998). Software project management: A unified framework Addison-Wesley Reading, MA.
Schach, S. R. (2002). Object-oriented and classical software engineering McGraw-Hill New York NY.
Sergiovanni, T. J. (1996). Leadership for the schoolhouse Jossey-Bass Publishers.
Silva, K., & Doss, C. (2007). The growth of an agile coach community at a fortune 200 company.
Turk, D., France, R., & Rumpe, B. (2002). Limitations of agile software processes. Third International Conference on eXtreme Programming and Agile Processes in Software Engineering, 43-46.