Monday, August 19, 2013

The Java NIO advantage



The challenge was to create a SIP User Agent with support for 50 Calls per second. The current implementation used UDP and java io api’s.  The implementation worked seamlessly for 10CPS from SIPP once we reached above 10CPS there was packet loss in the UDP Datagram socket. We change the code to support worker threads but still the loss was there. This prompted us to look for using other methods like queuing or   JMS.  I came across a article published by Jakob Jenkov on IO V/s NIO http://java.dzone.com/articles/java-nio-vs-io  and the tutorial provided by http://tutorials.jenkov.com/java-nio/index.html
Java IO V/s Java NIO

The main difference between Java IO and Java NIO is
IO
NIO
Stream oriented
Buffer oriented
Blocking IO
Non blocking IO

Selectors

The biggest difference is Java IO is stream based and NIO is buffer oriented. Java io being stream oriented can read one byte at a time . They are not cached. Hence we need to cache it in a buffer first.  Java NIO's being buffer oriented the approach different. Data is read into a buffer from which it can be processed later. You can move forth and back in the buffer as you need to. This gives you a bit more flexibility during processing. However, you also need to check if the buffer contains all the data you need in order to fully process it and, you need to make sure that when reading more data into the buffer, you do not overwrite data in the buffer you have not yet processed.
Java io is blocking which means the thread that read () or write () block until there is some data to read or write. When large chunk of messages are pumped into a UDP socket by the time worker thread reads the stream it is blocked which cause packet loss

Java IO UDP Datagram Server
UDPServer.Java

private   final int MAX_PACKET_SIZE = 2048;
private   final byte[] MAX_PACKET_ARRAY = new byte[MAX_PACKET_SIZE];
public void run()
                {
                                try
                                {
                  String value = new String();
                  rxSocket = new DatagramSocket(8080, InetAddress.getByName(localhost));
                  DatagramPacket incomingPacket = new DatagramPacket(MAX_PACKET_ARRAY,                                            MAX_PACKET_ARRAY.length);
                                               
                  while (true)
                  {
                      rxSocket.receive(incomingPacket);
                      if(incomingPacket.getLength()>0){
                        value = new String(incomingPacket.getData(), 0, incomingPacket.getLength());
                        logger.info("Message from client - "+value);
                        ProcessClientMessages pi=new ProcessClientMessages(value,sippclientthread,serverThread,serverinfoThread,incomingPacket.getAddress().getHostAddress(),incomingPacket.getPort());
                        tpes.submit(pi);
                      } 
                    }
                }catch (Exception ex) {
                                                ex.printStackTrace();
                                                logger.error("Callconnector Exception:" ,ex);
                                }
                }

The above program is to recieve the UDP packet using Java IO.


  
Java NIO UDP Server
UDPServer.Java

static int i=0;
public void run()
                {
                                try
                                {
                      String value = new String();
                          DatagramChannel channel = DatagramChannel.open( );
                          DatagramSocket socket = channel.socket( );
                          SocketAddress address = new InetSocketAddress("8080");
                          socket.bind(address);
                          ByteBuffer buffer = ByteBuffer.allocateDirect(65507);
                          byte byteArray[]=new byte[4096];
                  while (true)
                  {
                    SocketAddress client = channel.receive(buffer);
                            buffer.flip();
                            ipport=GeneralManipulation.getHostPort(client);
                            String fromIp=ipport[0];
                            String fromport=ipport[1];
                            while (buffer.hasRemaining( ))
                            { 
                                byteArray[i]=buffer.get();  
                                i=i+1;
                            }
                            i=0;
                            buffer.clear( );
                            value=new String(byteArray);
                    logger.info("str from client - "+value);
                    ProcessClientMessages pi=new ProcessClientMessages(value,sippclientthread,serverThread,serverinfoThread,fromIp,Integer.parseInt(fromport));
                    tpes.submit(pi);
                  }
                }catch (Exception ex) {
                                                ex.printStackTrace();
                                                logger.error("Callconnector Exception:" ,ex);
                                }
                }
The above is the code with java nio channel api that recieves messages on port 8080 and puts it in to the buffer and gets value from it and processes it. ByteBuffer : we allocate size to the bytebuffer and the channel which gets value will be in the buffer., (channel.receive(buffer);) then we iterate the buffer and get value from it.

Summary

NIO helps you to manage multiple channels using a single thread. If you need to connect multiple connections simultaneously using a single thread to manage all outbound connections implementing NIO server is probably an advantage  

Thursday, June 6, 2013

Unix Command Sheet (Cheat Sheet)

Sysinfo : This command is used to display system information i.e cpu, memory, etc







Memory and Swap : These command give information regard physical memory and swap area







Disks, Filesystems and Devices: These commands display disk information, file system etc.








Networking  : These commands are used to display and configuring network parameters


















































































Crash Dump : To configure display and use crash dump data 

























Performance Monitoring and Diagnostics : The commands List, Monitor and trace processes



You can download the entire Cheat Sheet from the link



















Tuesday, May 28, 2013

Tools for Managing Software Development

Software Engineering is a complex process, from modeling and design to code generation, project management, testing, deployment, change management and beyond, tools play a very important role and have become an essential part of managing Software development Process.  Tools allow repetitive, well-defined actions to be automated, reducing the cognitive load on the software engineer who is then free to concentrate on the creative aspects of the process. Tools are often design to support a particular software engineering method thus imposing a specific process. This makes adaptability to a particular tool very difficult since every organization has its own development process. It is either difficult or impossible for companies to move from set processes. We were posed with the same problem. Most of the tools were either ALM based or PLM based and the requirement was to have a single tool to match all our processes. The requirement of the tool was to fulfill the below requirement    

Software Development Tool features / aspects
Requirement Management 
Defect Management and Defect Tracking 
Source Code Management and Control
Code Review Process and Code Coverage 
Design and Case Tool (Use Case etc)
Release and Continuous Integration (Build Automation)
Test Management and Automation 
Quality Checklist 
Management DashBoard 
Resource Planning  and Timesheet Management
Project Management (Planning, Tracking , Cost management )

Breakdown of a typical Software Engineering Methods and Tools


Typical Product Company tools and Status Data Flow 
Holistic Tool across Organization

 The  tools needs to integrate with the available below to make it a holistic tool
Continuous Integration
Link all work items to tangible, working software in the form of builds. This enables teams to measure progress in terms of working software and to identify problems sooner.
Defect Management
Triage defects and include them in the planning process to balance new work and maintenance. Leverage existing automated workflows and defect reporting, while adding the practice of agile project management.


 IDE
View and update work items directly from your IDE. Stories, defects, tasks and tests can be tracked, updated and closed without ever needing to leave the IDE.


Test Management
Monitor the quality of your projects more easily by making the latest test results visible to dashboard. Teams can leverage existing test plans and test reporting, while creating new tests based on current user stories/backlog items.


Automation Testing Tool


Source Code Management
View code changes for a story or defect quickly by integrating tool with your source code manager. This makes it easier to track down the source of a defect and perform code reviews.


Portfolio and Project Management
Transition from traditional tools and project management practices without making everyone take a “leap of faith”. Integrate with  existing project management tools to avoid the pains of duplicate entry.


Requirements Management
Leverage your existing requirements management tool by creating a two-way flow with the took that keeps your requirements documentation synchronized and traceable while adding the practice of agile project management.


Customer Relationship Management
Link support cases to agile work items. This link enables development teams to understand real customer needs and for support desk to track engineering work through to delivery.


Approach Using Open Source
Build a holistic product using open source technologies to meet the requirement. This will need to have a team of developer and leads (Toolsmiths ) who will use open source tools and modify to cater to our needs

Toolsmith 


Continuous Integration


Tracking Tool (Defect/ FR ) With Workflow 


IDE

Requirement Management


Test Management and Testing Tool





Sunday, May 5, 2013

Agile Project Reporting and Metrics


“Agile processes are generative, not prescriptive. Processes need to evolve as needed, not be prescribed up-front. A prescriptive approach generates complex and complicated processes whereas a generative approach begins with a set of simple processes and adds others as they are needed.”
- Jim Highsmith, author of Agile Project Management
Agile recognizes that most effect software processes cannot be defined up-front but it is a continuous process. Traditional ALM process focuses on defining and enforcing processes and very little in identifying and delivering customers need.  Although they are aim to improve consistency and quality they often come at the expense of productivity and delivery. Traditional ALM has also used monolithic heavyweight tools, caused fragile builds and ineffective hand-offs between development, testing, deployment, and operations. Agile ALM frees companies from heavyweight processes and tools. Using continuous builds the customer is delivered regularly with working software which allow the customer to gauge the product which will be delivered and provide corrective measure beforehand.  While traditional ALM applied tools to enforce pre-defined standardized process, Agile ALM is the application of tools to support people and the processes that best suit the way they actually work when creating great software.  
A typical product development company has issue management process (Ticket Handling for Bugs/defects), Change/Feature Management process and new customer implementation process. In a product organization the PMO is a centralized resource for all the IT project teams to monitor project deliver with a wide range of projects with defined goals and timeframes. The PMO is responsible for (1) resource planning, including staff competencies, assignment, and capacity planning; (2) metrics and reporting to management (3) Definition and standardization project implementation and delivery processes, guidelines, training, and other project support services. Project managers need to  consistently communicate to and report on their projects across business units and development methodologies to the PMO so the PMO can present a single dashboard across  breadth of projects within a COO’s control, it is not surprising that the PMO is often heavy-handed, imposing a strict approach to metrics and guidance across all projects.
Among the above responsibilities the  greatest impact on Aglie project team is
1.       Providing dashboard to all stakeholders
2.       Providing metrics for process , project deliveries etc.
3.       Defining standard development practice which will support  and can be easily mapped to traditional project reporting and practice
Agile processes must participate within PMO infrastructure. This requires the Agile team willingness to conform to (or at least externally) traditional project management practices. Not only must the Agile project adapt the PMO must as well This will benefit from Agile projects’ ability to accurately measure the status of a project and adapt to changing requirements

Legacy PMO tracks project on the basis of project schedule, budget, issues and risk. This data is largely fed through project management tools (Ms Project etc). The PMO tracks time and cost estimate from project tasks dependencies among these task across the phase of the project life cycle.  Agile teams must communicate effectively, by using language and metrics that the PMO can appreciate. Agile team report on velocity (rate at which team delivers a story), burn charts(trend of remaining effort),test coverage and running test features. This do not effective help reporting in time, budget and risk analysis. Hence Agile team need to identify and bridge this reporting gap.

Sample  Agile Reporting
  





Release in Agile ALM mapped in Project Plan
Release Backlog 

Tradition Project Plan

 Mapping User Stories to Project Plan


Metrics such as earned value analysis (EVA) and cost to complete are certainly valuable to all stakeholders.  
Key Earned Value Management Terms Defined
(note: The term value in EVM does not refer to business value. The term refers to value as expressed by the project or program budget.)

Earned Value Term
Definition
Planned Value (PV)
The value of the work planned to be accomplished based on the budget (in dollars or hours)
Earned Value (EV)
The integrated value of work actually accomplished based on the budget (in dollars or hours)
Actual Cost (AC)
Actual Cost incurred for that increment of work
Budget At Complete (BAC)
The budget assigned to complete the work
Estimate To Complete (ETC)
The forecasted amount to complete remaining work (in dollars or hours) based on past performance.
Estimate At Complete (EAC)
The forecasted total amount for all work in the project plan based on past performance.

Earned Value Metrics
Metric
Formula
Metric Analysis
Planned Value
BAC * Planned Percent Complete
The planned value indicates how much value was planned to have been generated by a particular milestone or point in time.
Earned Value
BAC * Actual Percent Complete
The earned value indicates how much value has actually been generated at a particular milestone or point in time.
Cost Performance Index (CPI)
EV/AC
This metric indicates how many cents have been "earned" out of every dollar spent. It measures cost efficiencies.
Schedule Performance Index (SPI)
PV/AC
This metric measures schedule efficiency. It indicates how fast you are progressing against the rate of progress planned.
ETC
(BAC - EV)/CPI
This metric is the forecast amount to complete the remaining work.
EAC
BAC / CPI
Or
AC + ETC
Forecasted cost for the total planned work.


Agile story points can be used as the measure of work planned and work performed, providing the basis for all of the traditional earned value calculations. Percent complete can be derived by dividing the number of complete sprints by the total number of planned sprints. Actual percent complete can be derived by dividing the number of story points completed by the total number of story points planned for a release. To calculate the initial baseline in Agile 5 data points are required
  1. The number of planned sprints in the release backlog
  2. The length of each sprint in calendar days
  3. The number of story points planned for the release backlog
  4. The budget planned for the release backlog
  5. Start date of the project

 To calculate EVM in Agile
1.    Number of sprints completed: This measures the expected percent complete when divided by the total number of sprints planned.
2.    Net Story points added (this can be a negative number if story points are removed from the release.) This reflects the net change in work planned for the release, in effect re baselining the release. Actual Cost to date At each boundary we are measuring results to date, therefore we use the cumulative actual cost as of that sprint boundary.
3.    Cumulative story points completed This measures the total amount of work completed for the release as of that sprint boundary.

Item
Definition
Budget At Complete (BAC)
The planned amount you expect to spend
Actual Cost (AC)
The actual cost to complete the work
PRSP
Planned Release Story Points for the release. Story points are defined at the Product Backlog level.
Expected Percent Complete (EPC)
Current Sprint(n) / Total planned Sprints
Actual Percent Complete (APC)
Story points completed / Total planned Story points
Planned Value (PV)
PV = BAC * EPC
Earned Value (EV)
EV = BAC * APC
Cost Performance Index (CPI)
CPI = EV / AC
Schedule Performance Index
CSPI = EV/PV


Agile Dashboard for tradition PMO


Team Dashboard using  TASK Board
In agile it is a common practice to maintain task board on walls to visualize the project status.  A visual process management system like dashboard (Task Board) which tells what to produce, when to produce it and how to produce
The project status as a dashboard.






Defect Metrics as per Release BackLog

 

Tuesday, April 23, 2013

BI and Predictable analysis using Open Source BI Tool



Gartner defines business intelligence (BI) as an umbrella term that spans the people, processes and applications/tools to organize information, enable access to it and analyze it to improve decisions and manage performance. Today’s banking world is changing; it is changing as customer is changing dynamically. For years banks have relied on internal insight they have on their customer to manage risk fraud and also decide on product development, marketing strategy and customer communication.  Banking has also changed from personalized to virtual banking where customers are more relying on various online channels like ATM, Internet etc rather than visiting tellers or relationship managers. However the availability of vast pool of structured and unstructured “BIG DATA” allows for real time analysis and decision making that can help save money and also increase revenue.
“There were 5 exabytes of information created by the entire world between the dawn of civilization and 2003. Now that same amount is created every two days.”
– Eric Schmidt, CEO/Google
With majority of the Tier 1 and Tier 2 Banks have fully implemented IT system  there is vast pool of customer and transactional data is available in the central system for analysis. BIG Data has become a new buzz word for most of the Banking Product Companies who are seeing depleting revenues in CBS sales. There is lot of white papers available on the web with reference to new products, tool for BI and DW with new jargon for Big Data. There are also lots of white paper for comparative analysis of Open Source and Proprietary system sponsored by Big Players with emphasis on why not use Open Source. Open source solutions has always been an earning opportunity to the big proprietary players and failure to large bank that have tried implementing open source solutions themselves, but have always used product built by proprietary player using open source.  Lack of proper documentation and highly technical implementation meant only for the Geeks are one of the major reasons for not acceptance of open source solutions.
BI open source components available are
1.       Reporting : Use of open report systems like Jasper, BIRT etc
2.       MULTIDIMENSIONAL ANALYSIS (OLAP): Use of  Jpivot/Mondrian, JPalo/Mondrian, JPXMLA to provide multi dimensional cube and cube charts These tool allows the multidimensional analysis through flexible and user-friendly OLAP engines. Users can monitor the data on different detail levels and from different perspectives, through drill-down, drill-across, slice-and-dice, drill-through processes.


Charts :  JFreeChart  allows to develop single ready-to-use graphical widgets (such as histograms, pie graphs, bar graphs, area graphs, scatter diagrams, line graphs, bubble graphs, dispersion graphs) and interactive ones (temporal sliders, add/delete series), to be used separately, by choosing the properties to be included into the reports for a richer data view.



Data Mining and ETL : Tools like WEKA, QBE engine  and Talend  allows advanced data analysis, thanks to Data Mining processes aiming to find out hidden information patterns among a great amount of data.

Combining these tools together Banks can built dashboards, interactive cockpits,  KPI, Adhoc Reporting


There are many open source BI framework solution available

Open source free products

·         Eclipse BIRT Project
·         RapidMiner
·         SpagoBI
·         R
·         KNIME
·         TACTIC

Open source commercial products

·         Jaspersoft: Reporting, Dashboards, Data Analysis, and Data Integration
·         Palo (OLAP database): OLAP Server, Worksheet Server and ETL Server
·         Pentaho: Reporting, analysis, dashboard, data mining and workflow capabilities
·         TACTIC: Reporting, analysis, dashboard, data mining and integration, workflow capabilities

Proprietary free products

·         InetSoft
·         MicroStrategy
·         MicroStrategy Reporting Suite
·         QlikTech
·         QlikView

 

And Large range of  Proprietary products

·         ActiveReports & ActiveReports Server
·         Actuate Corporation
·         Advanced Business Solutions
·         ApeSoft
·         Birst
·         BOARD
·         Business Analyze
·         Cloud9 Analytics
·         ComArch
·         Data Applied
·         Decision Support Panel
·         Dimensional Insight
·         Grapheur, implementing the reactive business intelligence (RBI) approach[2]
·         IBM Cognos
·         IDV Solutions Visual Fusion
·         InetSoft
·         Informatica
·         Information Builders
·         InfoZoom
·         Intellicus
·         Izenda
·         Jackbe
·         Jolicharts
·         JReport (from Jinfonet Software)
·         Klipfolio Dashboard
·         Lavastorm
·         LIONsolver
·         List & Label
·         LogiXML
·         Microsoft
·         SQL Server Reporting Services
·         SQL Server Analysis Services
·         PerformancePoint Server 2007
·         Proclarity
·         Microsoft BI
·         MicroStrategy
·         myDIALS
·         Oracle
·         Hyperion Solutions Corporation
·         Panorama Software
·         Pentaho
·         Pervasive DataRush
·         PRELYTIS
·         ProcessEdge
·         QlikView
·         Business Objects
·         OutlookSoft
·         SAS
·         Siebel Systems
·         Spotfire (now Tibco)
·         Super Office Analyze
·         Sybase IQ
·         Tableau Software
·         TARGIT Business Intelligence
·         Teradata
·         Thomson Data Analyzer
·         XLCubed
·         Zoho Reports (as part of the Zoho Office Suite

SpagoBI is the only entirely Open Source Business Intelligence suite. It covers all the analytical areas of Business Intelligence projects, with innovative themes and engines. SpagoBI offers a wide range of analytical tools, as follows.
SpagoBI analytical engines full list, by Module and Analytical Area.


REPORTING
4 engines: JasperReport, BIRT, Accessible report, BO
SpagoBI allows to realize structured reports, using structured information views (e.g. lists, tables, crosstabs, reports) and to export them using several formats (HTML, PDF, XLS, XML, TXT, CSV, RTF).
·         View example

MULTIDIMENSIONALANALYSIS(OLAP)
3 engines: Jpivot/Mondrian, JPalo/Mondrian, JPXMLA
SpagoBI allows the multidimensional analysis through flexible and user-friendly OLAP engines. Users can monitor the data on different detail levels and from different perspectives, through drill-down, drill-across, slice-and-dice, drill-through processes.
·         View example

CHARTS
3 engines: JFreeChart, HChart, ExtChart
SpagoBI offers a specific graphics engine, based on JFreeChart, which allows to develop single ready-to-use graphical widgets (such as histograms, pie graphs, bar graphs, area graphs, scatter diagrams, line graphs, bubble graphs, dispersion graphs) and interactive ones (temporal sliders, add/delete series), to be used separately, by choosing the properties to be included into the reports for a richer data view.
·         View example

KPI
1 engine: KPI
SpagoBI offers all the necessary tools to create, manage, view and browse KPI hierarchical models, through different methods, calculation rules, thresholds and alarm rules.
·         View example

INTERACTIVE COCKPITS
2 engines: composed document, in-memory (1)
SpagoBI allows users to aggregate several documents into a single view and to interact with them in an dynamic and intuitive way.
·         View example

AD-HOC REPORTING
1 engine: Worksheet
SpagoBI allows end-users to freely create their own multi-sheet reports, including simple tables, cross-tables and different chart types in the document layout.
·         View example n.1
·         View example n.2
·         View example n.3
·         View example n.4

LOCATION INTELLIGENCE
2 engines: GEO, GIS
SpagoBI offers two geographical engines allowing to set run-time connections between geographical data and business data stored in the data warehouse:
·         a GEO engine, which uses a static catalogue in order to display data, allowing users to dynamically re-aggregate the information, according to geographical hierarchies (ex. nation, country, city). This engine can also be used out of the geographical "context" in the strict sense of the word: the indicators distribution can be displayed through any graphical structure (ex. process flow charts, topological schemes of hardware infrastructures)
·         a GIS engine, which interacts with real spatial systems, according to the WFS/WMS scheme.

FREE INQUIRY (Driven Data Selection)
2 engines: QbE, Smart Filter
SpagoBI offers a QbE (Query by Example) engine, which is suitable for those cases in which the free inquiry of data and the extraction of data are more important than their graphical structure and structural layout. Users can build queries through an entirely graphical and web-based interface. They can also execute queries, check the results, export them, save them for future use and generate reporting templates.
·         View example

DATA MINING
2 engines: Weka, R (1)
SpagoBI allows advanced data analysis, thanks to Data Mining processes aiming to find out hidden information patterns among a great amount of data.
·         View example

REAL TIME DASHBOARDS AND CONSOLE
2 engines: Dash, Console
SpagoBI offers a specific engine allowing to produce real-time monitoring consoles, to be used in Business, applicative or BAM processes.

·         View example

COLLABORATION
1 engine: Analytical Dossier
SpagoBI offers a specific engine allowing to automatically create organized report dossiers, enriched with the attending notes and information, posted by users to comment data. A collaborative workflow manages the dossier components to be sent to users in order to receive their notes.

·         View example

OFFICE AUTOMATION
1 engine: Office
SpagoBI offers a specific engine for the publication of personal documents in BI environments, realized through common Office tools (Open Office or MS Office).

ETL
1 engine: Talend
SpagoBI allows to load data into the data warehouse and manage it. SpagoBI ETL engine integrates the open source product TOS (Talend Open Studio).
·         View example

MOBILE
4 engines: Table, Chart, Cockpit, KPI (1)
SpagoBI offers specialized components based on the common touch-screen interaction paradigm, to efficiently combine BI with mobility.
·         View example n.1
·         View example n.2
·         Watch the video

EXTERNAL PROCESSES
1 engine: CommonJ
SpagoBI offers a component that allows to manage customized processes, running in the background and/or starting at a scheduled time.
·         View example

MASTER DATA MANAGEMENT
1 engine: Registry (QbE)
SpagoBI offers a component that allows users to take advantage of write-back functionalities on the database. It allows them to modify the table data through a highly intuitive user interface, whose behavior can be set by means of simple configuration parameters, using pre-defined models available within the platform.
·         View example