Agile in Value Delivery

Agile in End to End Value Delivery 

Agile has made development faster and have frequent deployments. Business and IT operations needs stability with speed.  To bridge this gap agile need to stretch itself beyond development and deployment. Business and Development team together need to consider software as a value.

A software value chain is a series of software definition, development, deployment process and operation process.   A series of align process and tools need to work along and synchronized way to provide continuous flow of value to the customer or end user. Processes and tools need to work around the value to provide simplified development and deployment to remove unnecessary handoff and step, also accelerate the delivery of the value by implementing lean and agile principals and practices. Can agile principle be defined to operation also?

Lean and Agile have many things in common each are focused on value delivery

Legacy Dev Ops Model

Teams and Roles

Product Management : The team is responsible for define the functionalities required in the product. In tandem with the Business Operation the team strategies the requirement for the Development Team or Product Vendor for  Product Changes (using the defined Change Control  Process ).  Product Vendors have Product Owners who interact with the team and add these changes to the product log for the Agile team to define and plan Sprints

Tools Used :  Requirement Analysis and Gathering tools, Requirement Tracking and DMS tool  eg Rational, Visio, XLS, Clearcase  

Development Team: Plan and execute the sprints and publish Sprint plan for the deployment, Burn Charts for management. Weekly releases to the staging system for the UAT team to test . A group of sprint releases or one sprint release can be part of the release plan published by the Product owner to the Production and Business Operation.

Tools Used :  Version Control (SVN, CVS,GIT, Visual SourceSafe, Starteam ), ALM tools (SwiftALM, Wizible, Collabnet, Microsoft Team foundation), Continuous integration (Jenkins, Hudson )

UAT /SIT Team: The team is responsible for testing the release in the staging system for integration and system and when done provide UAT pass approval for the release team to prepare release build/ patch  for  Product team

Tools Used : Testpro, Testing Tools/ Simulators, Load runner

Release team : Is responsible for the SCM activity and release activity for both Staging and Production system. Depending on the published release plan and date the team will provide signed build and release note once a go is provided  by the SIT team in the staging server. The team publishes to the stake holders for  downtime approval.

Tools used : Continuous integration (Jenkins, Hudson ),Cruise Control,  ALM tools and PLM tools

Production Support : On downtime approval the team deploys the build or patch on the production system and runs and acceptance test with the business operation. A go-live is declared on successful deployment else it is rolled back with a failure issues logged in the incident system with severity  

Tools : Ticketing or incident management tools ( Bugzilla, Jira, Quality Center, fogbuz), PLM tools like redmine. SwiftPLM, etc

Business Operation: Responsible to provide go-live testing and also provide product changes required for Business Needs

Tools:  The product, Incident Management Tools, PLM tools to track changes status.   

Agile Process in Delivering Value in Operations

Agile team starts with a value delivery (Value Release Team). The Release is planned as a value to be delivered with the whole team. The story board is build on the value and the product log and service catalogue is build on the story board. This is the take away for the agile development   team, release team, business ops, testing team, infrastructure team and product support team.

Agile Value Model

Read More

The Java NIO advantage

The challenge was to create a SIP User Agent with support for 50 Calls per second. The current implementation used UDP and java io api’s.  The implementation worked seamlessly for 10CPS from SIPP once we reached above 10CPS there was packet loss in the UDP Datagram socket. We change the code to support worker threads but still the loss was there. This prompted us to look for using other methods like queuing or   JMS.  I came across a article published by Jakob Jenkov on IO V/s NIO http://java.dzone.com/articles/java-nio-vs-io  and the tutorial provided by http://tutorials.jenkov.com/java-nio/index.html

Java IO V/s Java NIO

The main difference between Java IO and Java NIO is

IO	NIO
Stream oriented	Buffer oriented
Blocking IO	Non blocking IO
	Selectors

The biggest difference is Java IO is stream based and NIO is buffer oriented. Java io being stream oriented can read one byte at a time . They are not cached. Hence we need to cache it in a buffer first.  Java NIO's being buffer oriented the approach different. Data is read into a buffer from which it can be processed later. You can move forth and back in the buffer as you need to. This gives you a bit more flexibility during processing. However, you also need to check if the buffer contains all the data you need in order to fully process it and, you need to make sure that when reading more data into the buffer, you do not overwrite data in the buffer you have not yet processed.

Java io is blocking which means the thread that read () or write () block until there is some data to read or write. When large chunk of messages are pumped into a UDP socket by the time worker thread reads the stream it is blocked which cause packet loss

Java IO UDP Datagram Server

UDPServer.Java

private   final int MAX_PACKET_SIZE = 2048;

private   final byte[] MAX_PACKET_ARRAY = new byte[MAX_PACKET_SIZE];

public void run()

                {

                                try

                                {

                  String value = new String();

                  rxSocket = new DatagramSocket(8080, InetAddress.getByName(localhost));

                  DatagramPacket incomingPacket = new DatagramPacket(MAX_PACKET_ARRAY,                                            MAX_PACKET_ARRAY.length);

                                               

                  while (true)

                  {

                      rxSocket.receive(incomingPacket);

                      if(incomingPacket.getLength()>0){

                        value = new String(incomingPacket.getData(), 0, incomingPacket.getLength());

                        logger.info("Message from client - "+value);

                        ProcessClientMessages pi=new ProcessClientMessages(value,sippclientthread,serverThread,serverinfoThread,incomingPacket.getAddress().getHostAddress(),incomingPacket.getPort());

                        tpes.submit(pi);

                      } 

                    }

                }catch (Exception ex) {

                                                ex.printStackTrace();

                                                logger.error("Callconnector Exception:" ,ex);

                                }

                }

The above program is to recieve the UDP packet using Java IO.

  

Java NIO UDP Server

UDPServer.Java

static int i=0;

public void run()

                {

                                try

                                {

                      String value = new String();

                          DatagramChannel channel = DatagramChannel.open( );

                          DatagramSocket socket = channel.socket( );

                          SocketAddress address = new InetSocketAddress("8080");

                          socket.bind(address);

                          ByteBuffer buffer = ByteBuffer.allocateDirect(65507);

                          byte byteArray[]=new byte[4096];

                  while (true)

                  {

                    SocketAddress client = channel.receive(buffer);

                            buffer.flip();

                            ipport=GeneralManipulation.getHostPort(client);

                            String fromIp=ipport[0];

                            String fromport=ipport[1];

                            while (buffer.hasRemaining( ))

                            { 

                                byteArray[i]=buffer.get();  

                                i=i+1;

                            }

                            i=0;

                            buffer.clear( );

                            value=new String(byteArray);

                    logger.info("str from client - "+value);

                    ProcessClientMessages pi=new ProcessClientMessages(value,sippclientthread,serverThread,serverinfoThread,fromIp,Integer.parseInt(fromport));

                    tpes.submit(pi);

                  }

                }catch (Exception ex) {

                                                ex.printStackTrace();

                                                logger.error("Callconnector Exception:" ,ex);

                                }

                }

The above is the code with java nio channel api that recieves messages on port 8080 and puts it in to the buffer and gets value from it and processes it. ByteBuffer : we allocate size to the bytebuffer and the channel which gets value will be in the buffer., (channel.receive(buffer);) then we iterate the buffer and get value from it.

Summary

NIO helps you to manage multiple channels using a single thread. If you need to connect multiple connections simultaneously using a single thread to manage all outbound connections implementing NIO server is probably an advantage  

Read More

Unix Command Sheet (Cheat Sheet)

Sysinfo : This command is used to display system information i.e cpu, memory, etc

Memory and Swap : These command give information regard physical memory and swap area
Disks, Filesystems and Devices: These commands display disk information, file system etc.
Networking  : These commands are used to display and configuring network parameters

Crash Dump : To configure display and use crash dump data 

Performance Monitoring and Diagnostics : The commands List, Monitor and trace processes



You can download the entire Cheat Sheet from the link

Read More

Tools for Managing Software Development

Software Engineering is a complex process, from modeling and design to code generation, project management, testing, deployment, change management and beyond, tools play a very important role and have become an essential part of managing Software development Process.  Tools allow repetitive, well-defined actions to be automated, reducing the cognitive load on the software engineer who is then free to concentrate on the creative aspects of the process. Tools are often design to support a particular software engineering method thus imposing a specific process. This makes adaptability to a particular tool very difficult since every organization has its own development process. It is either difficult or impossible for companies to move from set processes. We were posed with the same problem. Most of the tools were either ALM based or PLM based and the requirement was to have a single tool to match all our processes. The requirement of the tool was to fulfill the below requirement    

Software Development Tool features / aspects

Requirement Management

Defect Management and Defect Tracking

Source Code Management and Control

Code Review Process and Code Coverage

Design and Case Tool (Use Case etc)

Release and Continuous Integration (Build Automation)

Test Management and Automation

Quality Checklist

Management DashBoard

Resource Planning  and Timesheet Management

Project Management (Planning, Tracking , Cost management )

Breakdown of a typical Software Engineering Methods and Tools

Courtesy  : http://www.computer.org/portal/web/swebok/html/ch10

Typical Product Company tools and Status Data Flow 

Holistic Tool across Organization

 The  tools needs to integrate with the available below to make it a holistic tool

Continuous Integration

Link all work items to tangible, working software in the form of builds. This enables teams to measure progress in terms of working software and to identify problems sooner.

Defect Management

Triage defects and include them in the planning process to balance new work and maintenance. Leverage existing automated workflows and defect reporting, while adding the practice of agile project management.

 IDE

View and update work items directly from your IDE. Stories, defects, tasks and tests can be tracked, updated and closed without ever needing to leave the IDE.

Test Management

Monitor the quality of your projects more easily by making the latest test results visible to dashboard. Teams can leverage existing test plans and test reporting, while creating new tests based on current user stories/backlog items.

Automation Testing Tool

Source Code Management

View code changes for a story or defect quickly by integrating tool with your source code manager. This makes it easier to track down the source of a defect and perform code reviews.

Portfolio and Project Management

Transition from traditional tools and project management practices without making everyone take a “leap of faith”. Integrate with  existing project management tools to avoid the pains of duplicate entry.

Requirements Management

Leverage your existing requirements management tool by creating a two-way flow with the took that keeps your requirements documentation synchronized and traceable while adding the practice of agile project management.

Customer Relationship Management

Link support cases to agile work items. This link enables development teams to understand real customer needs and for support desk to track engineering work through to delivery.

Approach Using Open Source

Build a holistic product using open source technologies to meet the requirement. This will need to have a team of developer and leads (Toolsmiths ) who will use open source tools and modify to cater to our needs

Toolsmith 

Continuous Integration

Tracking Tool (Defect/ FR ) With Workflow 

IDE

Requirement Management

Test Management and Testing Tool

Read More

Agile Project Reporting and Metrics

“Agile processes are generative, not prescriptive. Processes need to evolve as needed, not be prescribed up-front. A prescriptive approach generates complex and complicated processes whereas a generative approach begins with a set of simple processes and adds others as they are needed.”

- Jim Highsmith, author of Agile Project Management

Agile recognizes that most effect software processes cannot be defined up-front but it is a continuous process. Traditional ALM process focuses on defining and enforcing processes and very little in identifying and delivering customers need.  Although they are aim to improve consistency and quality they often come at the expense of productivity and delivery. Traditional ALM has also used monolithic heavyweight tools, caused fragile builds and ineffective hand-offs between development, testing, deployment, and operations. Agile ALM frees companies from heavyweight processes and tools. Using continuous builds the customer is delivered regularly with working software which allow the customer to gauge the product which will be delivered and provide corrective measure beforehand.  While traditional ALM applied tools to enforce pre-defined standardized process, Agile ALM is the application of tools to support people and the processes that best suit the way they actually work when creating great software.  

A typical product development company has issue management process (Ticket Handling for Bugs/defects), Change/Feature Management process and new customer implementation process. In a product organization the PMO is a centralized resource for all the IT project teams to monitor project deliver with a wide range of projects with defined goals and timeframes. The PMO is responsible for (1) resource planning, including staff competencies, assignment, and capacity planning; (2) metrics and reporting to management (3) Definition and standardization project implementation and delivery processes, guidelines, training, and other project support services. Project managers need to  consistently communicate to and report on their projects across business units and development methodologies to the PMO so the PMO can present a single dashboard across  breadth of projects within a COO’s control, it is not surprising that the PMO is often heavy-handed, imposing a strict approach to metrics and guidance across all projects.

Among the above responsibilities the  greatest impact on Aglie project team is

1.       Providing dashboard to all stakeholders

2.       Providing metrics for process , project deliveries etc.

3.       Defining standard development practice which will support  and can be easily mapped to traditional project reporting and practice

Agile processes must participate within PMO infrastructure. This requires the Agile team willingness to conform to (or at least externally) traditional project management practices. Not only must the Agile project adapt the PMO must as well This will benefit from Agile projects’ ability to accurately measure the status of a project and adapt to changing requirements

Legacy PMO tracks project on the basis of project schedule, budget, issues and risk. This data is largely fed through project management tools (Ms Project etc). The PMO tracks time and cost estimate from project tasks dependencies among these task across the phase of the project life cycle.  Agile teams must communicate effectively, by using language and metrics that the PMO can appreciate. Agile team report on velocity (rate at which team delivers a story), burn charts(trend of remaining effort),test coverage and running test features. This do not effective help reporting in time, budget and risk analysis. Hence Agile team need to identify and bridge this reporting gap.

Sample  Agile Reporting

  

Release in Agile ALM mapped in Project Plan

Release Backlog 

Tradition Project Plan

 Mapping User Stories to Project Plan

Metrics such as earned value analysis (EVA) and cost to complete are certainly valuable to all stakeholders.  

Key Earned Value Management Terms Defined

(note: The term value in EVM does not refer to business value. The term refers to value as expressed by the project or program budget.)

Earned Value Term	Definition
Planned Value (PV)	The value of the work planned to be accomplished based on the budget (in dollars or hours)
Earned Value (EV)	The integrated value of work actually accomplished based on the budget (in dollars or hours)
Actual Cost (AC)	Actual Cost incurred for that increment of work
Budget At Complete (BAC)	The budget assigned to complete the work
Estimate To Complete (ETC)	The forecasted amount to complete remaining work (in dollars or hours) based on past performance.
Estimate At Complete (EAC)	The forecasted total amount for all work in the project plan based on past performance.

Earned Value Metrics

Metric	Formula	Metric Analysis
Planned Value	BAC * Planned Percent Complete	The planned value indicates how much value was planned to have been generated by a particular milestone or point in time.
Earned Value	BAC * Actual Percent Complete	The earned value indicates how much value has actually been generated at a particular milestone or point in time.
Cost Performance Index (CPI)	EV/AC	This metric indicates how many cents have been "earned" out of every dollar spent. It measures cost efficiencies.
Schedule Performance Index (SPI)	PV/AC	This metric measures schedule efficiency. It indicates how fast you are progressing against the rate of progress planned.
ETC	(BAC - EV)/CPI	This metric is the forecast amount to complete the remaining work.
EAC	BAC / CPI Or AC + ETC	Forecasted cost for the total planned work.

Agile story points can be used as the measure of work planned and work performed, providing the basis for all of the traditional earned value calculations. Percent complete can be derived by dividing the number of complete sprints by the total number of planned sprints. Actual percent complete can be derived by dividing the number of story points completed by the total number of story points planned for a release. To calculate the initial baseline in Agile 5 data points are required

1. The number of planned sprints in the release backlog
2. The length of each sprint in calendar days
3. The number of story points planned for the release backlog
4. The budget planned for the release backlog
5. Start date of the project

 To calculate EVM in Agile

1.    Number of sprints completed: This measures the expected percent complete when divided by the total number of sprints planned.

2.    Net Story points added (this can be a negative number if story points are removed from the release.) This reflects the net change in work planned for the release, in effect re baselining the release. Actual Cost to date At each boundary we are measuring results to date, therefore we use the cumulative actual cost as of that sprint boundary.

3.    Cumulative story points completed This measures the total amount of work completed for the release as of that sprint boundary.

Item	Definition
Budget At Complete (BAC)	The planned amount you expect to spend
Actual Cost (AC)	The actual cost to complete the work
PRSP	Planned Release Story Points for the release. Story points are defined at the Product Backlog level.
Expected Percent Complete (EPC)	Current Sprint(n) / Total planned Sprints
Actual Percent Complete (APC)	Story points completed / Total planned Story points
Planned Value (PV)	PV = BAC * EPC
Earned Value (EV)	EV = BAC * APC
Cost Performance Index (CPI)	CPI = EV / AC
Schedule Performance Index	CSPI = EV/PV

Agile Dashboard for tradition PMO

Team Dashboard using  TASK Board

In agile it is a common practice to maintain task board on walls to visualize the project status.  A visual process management system like dashboard (Task Board) which tells what to produce, when to produce it and how to produce

The project status as a dashboard.

Defect Metrics as per Release BackLog

 

Read More