Friday, January 2, 2015

VMware & Cloud

VMware is the market leader in virtualization space. I was reading about different product offering from VMware and it's usage. VMware has many products and some of the product's functionality overlapping each other, however VMware is better than Oracle for product overlapping  !. 

Some of my note are given below for reference.

What is a cloud ? Since you have cloud available, why clutter your brain ? You can store some of the brain memory is external private cloud and access it from your home and outside :) 

I will not try to define cloud, Refer to  National Institute of Standards and Technologies (NIST)'s cloud definition. A Cloud can be private, hybrid, community or public. A cloud should have On-Demand Self-Service, Broad Network Access, Resource Pooling,Rapid Elasticity and Measured Services. Virtualization is an enabling technology for cloud computing. Cloud has different service models such as IAAS (Infrastructure as a service), PAAS (Platform as a service) and SAAS (Software as a service) 

Software as Service (Saas) means you don't need to install software in your laptop. My question is will it make my laptop lighter :)  

vCenter is what manages vSphere virtual infrastructure hosts, virtual machines, and virtual resources (CPU, memory, storage, and network). You create software based VMClusters using Vcenter. You can VMotion across different VMClusters. VMWare admin uses Vcenter to manager your entire VMWare infrastructure. Some companies have seperate Vcenter for production and non-production environment for security reasons. You should be able to see and configure multiple datacenter in single Vcenter console. You can view utilization of your hosts (CPU, Memory etc) in Vcenter. 

vCloud Director (VCD) is at a higher level in the cloud infrastructure. vCloud Director (vCD) can traverse multiple virtual infrastructures and further abstracts the virtual infrastructure resources into software-defined storage, networking, security, and availability. vCloud Director manages one or more vCenter instances for your private cloud.

To create a cloud you’ll need both vCenter and vCD, however most organization has only vCenter.VCD communicate to Vcenter to create abstration. 

vSphere is a software suite that has many software components like vCenter, ESXi, vSphere client and so on. vSphere is not a software that you can download and install. This is a combination of different products. ESXi is the virtualization server - type 1 hypervisor. All virtual machines or Guest OS are installed on ESXi server. vSphere client is used to access vCenter Server and used for manage ESXi servers.

vRealize Orchestrator (VCO) provide workflow and automation capabilities. If you are using VCO, then you may not need vRealize Automation (formerly vCloud Automation Center).

Not sure why VMware renamed vcloud to vRealize ? Is it a good idea to build your own cloud or buy ? The cloud providers like Amazon EC2 is not cheap. It is very expensive. I created an account earlier using my credit card. Created couple of VM's and installed few Middleware software on it. I realized that Amazon was charging my credit card. Some how I noticed it lately ! 

Public cloud is a good idea if you need to ramp up and down your infrastructure unpredictably or need elasticity on your existing data center based on consumer traffic. 

More to follow .......

Wednesday, December 31, 2014

Storage

Question: What is your most expensive line item for IT?
Answer: Storage!

I was reading about storage this week and sharing some of my notes here:

Only 10 % of data are structured (database). Other 90 % of data are unstructured (Email, Xrays images etc). 

As per the projections by IDC, storage has 300 x growths from 2005 to 2020. 80 % of data never used after 90 days! Most of the data stored in Tapes! Tape is the lowest cost storage medium. Tape storage cost less than $ 0.15/GB however high performance disk cost $5-$10/GB! Disk cost 26 times more than Tape solution!

How to achieve scalability, performance and flexibility for your storage infrastructure?  It is very important that you handle storage infrastructure appropriately. You may have to ensure that the storage infrastructure is optimized to reduce the cost, provision storage without delays and automate all repetitive tasks.

Goals for storage optimization:
  • Remove unused storage
  • Reduce storage costs.
  • Provision storage by on demand (Thin provisioning ? ) 
  • Simplify and automate storage management. 
  • Increase storage capacity
  • Remediating storage, reclaiming storage from an application
  • Creating chargeback reporting on an application (Is this possible ??)
 NetApp,EMC, Borcade, IBM are the major players in storage market.
It is amazing to me that the basic concept of RAID (Redundant array of inexpensive disks) technology hasn't changed since last 25 years. RAID has different levels RAID 0 to RAID 6.  The purpose of the RAID technology is to combine individual drives. LUN (Logical unit number) is created from the RAID set by partitioning the available capacity in to small units. 

High end storage provide Active-Active arrays. Mid range storage provide active - passive arrays. EMC Symmetrix is Active - Active arrays which include Symmetrix Virtual Matrix (VMAX) series. EMC VNX storage is active - passive arrays. 

Before looking ways to optimize storage infrastructure and looking at the emerging technologies, let us identify the fundamental concepts of storage. 
Following are major types of storage:
  • SAN
  • NAS
What is the difference between NAS and SAN? What are the protocol for accessing storage from a computer (to and from) - SCSI or NFS or CIFS? How the storage is connected to a system? - Storage device is directly attached or network attached? What kind of cabling? Ethernet or Fiber optic?

NAS (Network Attached storage) - Single Storage device. Multiple file servers are not needed.  Uses Ethernet or TCP/IP for connections or data transfer. NFS/CIFS (NFS - Network file system, CIFS - Common internet file system) are the main I/O protocol. NFS or CIFS are the network file services.  NAS appears as shared folder for computer. NAS evolved from direct attached storage (DAS). It is not possible to share the file using portable storage ! NAS enable user to share the files over an IP network. NAS can be used by Linux or Microsoft windows. 

NAS device has it's own software and hardware including operating system. It perform I/O much better than normal operating system. Files could be moved across multiple NAS devices without interruptions.   NAS don't need to create a new network (like SAN), it can use existing IP network. 

SAN - Local network of multiple storage devices. It is a high speed network that makes connections between storage devices and servers. Uses FibreChannel for interconnects. SAN is suitable for large enterprises. It can contain high-performance disk arrays. SCSI is the main I/O protocol. SAN appears to computer as its own storage. SAN provide great performance than NAS. Example: EMC Clariion (currently discontinued, replaced by VNX and VNXe storage area network (SAN)

SAN Carries data between servers using Fiber Channel network ( usually 2/4/8 and 16-gigabit per second rates)

SAN has virtualization: 

1) Block level storage virtualization (Aggregate LUNs and enable provisioning of virtual storage) 
2) Virtual SAN (VSAN)

SAN has following components:

  • Drive enclosure - any number of drivers. Usually each drive enclosure will have more than SAN drive enclosure has more than one drive. Expand based demand. Type of drive could be SATA. Fiber channel or Solid state
  • Controllers & Cache - it is a server with its own operating system. That connects Drive.
  • Switch network - Fiber
  • Host bus adapter - Similar to computer network card. Fancy name for "Fiber optic network card"
  • HBA Driver Servers.
 RAID - Redundant array of independent disks. Data virtualization technology that combines multiple disk drive components in to a logical unit.
Connecting storage to Computer:
  • SCSI - Small computer system interface connects printers and scanners to computer. 
  • iSCSI(Internet Small Computer System Interface ) - It is a IP based standard for interconnecting
  • Fibre channel
  •  NFS - Network file system protocol. Allow user to access files from network as though it is on the local disk.It has a server and client. Standard file sharing protocol. Latest NFSV4 .
  •  The Common Internet File System (CIFS) - Standard way that computer users share files across corporate intranets and the Internet.
  • SMB Remote File Protocol (Including SMB 3.0) - Derived from CIFS.
Storage Devices:  
  • DAS - Directly attached storage. Example: USB device or Dell's MD1220
  • RAID, Optical storage or Tape.
 SAN backbones - Network switching platforms.
Example: Brocade DCX 8 GPS
Storage Standards:
  • Storage Networking Industry Association (SNIA) – Storage industry
  • Cloud Data Management Interface (CDMI) - the functional interface that applications will use to create, retrieve, update, and delete data elements from the Cloud
Emerging storage technologies:
  • SDS - Software defined storage
  • Pure-Play Cloud Storage: AWS Storage Gateway - Backup local storage to cloud based storage.
  • Open Stack-Based Storage Clouds: Rackspace Cloud Block Storage
  • RRAM Revolution - New technology for RAM and DRAM  
Storage provisioning (Steps).
  • Assign LUNs (Logical Unit Numbers)
  • verify that data storage and data recovery routes
  • create alternate routes in case of a partial failure
  • Manage storage capacity for expansion
  • Test the storage to ensure that users can use it
Storage thin provisioning relies on on-demand allocation of blocks of data versus the traditional method of allocating all the blocks up front. This optimize utilization of storage. 

Automated Storage provisioning (Options)

Use threshold to allocate more storage automatically (or send notification to SAN administrator to allocate more storage)

1. Policy driven storage provisioning
2. Application threshold level provisioning.

Applications available for automating storage - EMC Automated Resource Manager software (ARM) ?  


Virtual Machine (VM) - A Virtual machine a template which consists of  set of files that can be moved and copied. All files that make up a VM is stored in a single directory. 


Vmware ESXi has a cluster file system called  Vitual Machine File system (VMFS) 

Thursday, December 25, 2014

IT Automation

I live in Ashburn, VA. Up to 70% of the world's internet traffic passes through my county (Loudoun) on a daily basis !!

I can see lot of data center around here, however couldn't see any cars parked in those buildings ? Why ? Looks like, a data center need only a security guard !! (security also can be done remotely !)

Automation play a big role in Data center. Any task that is repeatable should be automated.  There are several tools available to create workflow and orchestration of process and tasks. Example: Microsoft System Center 2012 R2 suite, BMC Bladelogic etc.

The first step of any automation is to understand the manual steps that an IT administrator/resource performs to do the repeatable task. You document the steps as run book for that process. Next step is to identify the time savings and business efficiency associated  with automating those tasks.  

The objective of IT automation is to increase speed of IT services delivery and management thereby achieve agility and accelerate application delivery and support. 

Use Self Service portals for achieving IT automation.  Empower end users to obtain IT compute to do their work. Simplify IT and reduce bureaucracy. 

Governance is very important for IT automation. In order to achieve automation, you don't want your company to compromise security, control on the IT infra resources or  IT hardware/software license management and optimization. 

IT Automation is all about people, process and technology. Automation can't be achieved by implementing couple of tools. The business process changes/optimization require to achieve real automation. 

The collaboration and buy-in from different organizations within IT is a must. Automation is about optimizing and making every one's life easier. Resources can concentrate on real IT work instead of doing repetitive boring work such as provisioning and configuring a software using same methodology for multiple customers. It is all about removing manual process and optimizing the process of install, configure, manage and deploy hardware and software. 

IT Automation have three major parts:  All should be managed by software with Self service portals. 


  • Compute - Manage compute infrastructure. Virtualized servers for internal or cloud . Use self-service portals for obtaining, managing and supporting IT infrastructure. N
  • Network - Software based networking. Use Self Service Portals for obtaining, managing and supporting network. 
  • Storage - Software based Storage. Use Self Service Portals for obtaining, managing and supporting storage. 
Some of the commonly used tools are listed below: 

Deployment: 
Jenkins

Build Automation 
Apache Ant
Apache Maven

Software configuration management 
SVN 

Continues integration 
Chef

Agile
Scrum 

bug reports:

Bugzilla 



Sunday, April 27, 2014

Agility in Network layer

Most of the companies still using traditional layer 2 architecture for networking. In a network, the hardware switch is a device that redirects data messages at the layer 2 level. It uses MAC address to determine the destinations.  Of course layer 4 and layer 7 is available from CISCO and other companies for web applications configurations. Can we use software instead for hardware for network routing? Why not bring agility in to networking layer? IT has the requirement to scale networking layer rapidly to meet the workload demands. Also will network layer is a single point of failure?
 
We create HA in Web, app and Database layer, but how about networking layer? Can network layer handles fail over without an admin doing manual work? We can provision hosts/VMs in matter of minutes; however it may take two to three weeks to configure a network layer? Why not bring the agile provisioning capability at network layer? Network virtualization and ability to create virtual networks helps organizations to speed up the development and deployment process.
 
How to achieve network virtualization with agility along scalability, performance and high resource efficiency?  With current technologies, will it possible to create “Network VMs”? Will Network virtualization solutions such as VMware NSX or CISCO virtual networking products are matured for IT industry? The key questions is - can it deliver agility in network layer along with auto failover, auto provisioning, auto resource utilization and performance.

Saturday, March 15, 2014

What is next after SOA ?


Companies are still in the process of implementing SOA. Some organizations using SOA as an IT strategy that organizes discrete functions contained in enterprise applications in to interoperable, standards-based services.
 
How successful are they in implementing SOA?
 
The whole idea about SOA is consumer decoupled from producer, business process decomposed in to  re-usable services and we should be combining re-usable services in to business processes, there  by achieve re- usability, loose coupling, create service in asynchronously with standard service  contract. Also implement discoverability for services that has well defined interface  definitions, use standard data format across organization and use standard Wire and Security protocols.
 
 
I don't think many organizations are achieving the above goals! How many organizations can claim achieving IT agility by implementing SOA?
 
 
Some people believe that by using SOA software, you achieve SOA for the organization!. You don't need a vendor product for implementing SOA for your organization.
 
OK. What is next? I have no doubt that IT Industry is moving towards COTS products. Currently nobody write an accounting or a payroll application. We just buy it from off the shelf, however since the business process are not standard across different IT organizations, we still need custom development software/web applications.  
 
 
Solution: SAAS - within organization?
 
The key to success is, how can we create re-usable applications/code base/services that can use within the organization which allow customization based on the business need?.
 
How can we implement SAAS within the organization without complexity of heavy vendor specific SOA products ? Why can't we use simple JDK/simple containers for writing applications? Use POJO instead of EJBs!.
 
I was having conversation with my friends (Jaswant and Nathan) this morning about some of these trends - Currently industry is getting matured in IAAS (Infrastructure as a service). We have many cloud providers that provide infrastructure such as Amazon, Rackspace, CSC, Terremark etc. Organizations have VMWare  or open source based hypervisors in-house that enable us to create Virtual images as needed and create/update/move/delete hosts based on the requirements. It is still cheaper (financially) to use an in house data center instead of using external vendor provide data center for long term !.
 
 
We have the technology available today to standardize, automate compute resources including Middleware and database complemented by storage and networking capabilities and provide it based on the customer demand.
 
We can self-provision Hosts, Operating Systems, Middleware, Database and Storage using a Web- based graphical user interface that serves as an IT operations management console for entire IT.
 
What is missing in this space is an integration layer or workflow based system that authorize and approve the creation of compute resources for the organization for internally or externally hosted data center.
 
We need a software distribution model in which re-usable applications are hosted internally and made available to departments within IT to use it, customize it, and enhance it based on their requirements. Ideally we should have this from an external vendor (externally hosted). We have few companies in this space such as salesforce, however industry don't have enough on demand computing software delivery models.
 
Let us work towards implementing a SAAS model within the organization !.
 
 

Sunday, December 22, 2013

Some spiritual thoughts!

As you might have seen in many places, "Happiness is not a destination, it is a way of life" !. You get happiness by leading an effective life. So the question is how can we lead an effective life? . One of the way is to look is at the people you consider who successfully leading a good life. Will money help?

You need money for food, shelter, transportation, enjoyment and other basic necessities. Any extra money you have may not bring happiness (You agree?) 

If you ask the richest person in the area, he or she still needs more money to do more things. Collecting more money is a never ending process. So money will not bring happiness or satisfaction.

I always thought about creating a start-up business (I did it once and still thinking!) however will establishing a business and bringing additional money and comfort bring you happiness ? 

If your family is not happy then you can't be happy in home. If your team members are not happy then you can't be happy in office!. 

How to lead an effective life ?
The first question is are you independent or are you depend on others to live?  Any job you do is to serve others and we must need support from family and community. Without family or community one can not survive, however the question is - Are you blaming others for your life misfortunes? I became a software engineer because of my mom!. I can't live a good life because of my wife/husband?  Example: wife/husband is irritating me so I can't relax in my home!  

One must be truly independent, don't blame others for your life. You decide what to do in your life and how to lead your life. 

I recently read a book "The 7 Habits of Highly Effective People: Powerful Lessons in Personal Change" by 
Stephen R. Covey. He explains that all problems that are experienced by individuals fall into one of three categories, which are direct control, indirect control, or no control.

The problems that are classified under direct control are the problems that involve our own behavior. The problems classified as indirect control and no control encompasses problems that we can do nothing about. We always worry about the issues that are outside our control.

Even though issues arise outside your control, the response to the events are your choice. 

I read about one incident where a group of girls and boys visited a library. As they were settling around a table, one cockroach suddenly dropped on one of the girls. The girl got panic and shouted. The cockroach jumped on to another girl. As they were all shouting panicking the librarian approached. He calmly took the cockroach using a towel and placed in the trash bag.

You can't control the incidents in your life; however you can control your reaction to the incidents. 

I read an article by Louis V. Gerstner. He is former CEO of IBM and the author of book "Who Says Elephants Can't Dance?"  As per, Louis, You need to think about this in your early career - "About 10 years from now, what do I want to spend my time on?". He planned for his retirement. He retired from IBM as CEO at age of 61 and is still active and pursuing his dreams.

How many of you (including myself) thought about what we should do after 10 years? or during the retirement ? We flow at the direction of the wind. Wherever wind takes us we go!.What I will be doing after 10 years ? or What I want to do after 10 years?. I am going to give a serious thoughts and work towards the end goal. 

You decide your future. You are independent. There are things that you can't control such as a war or natural calamities such as tornado; however what you can control is your response.













Technology Corner (an article published for KAGW)

By Ravi Saraswathi and Jaswant Singh authors of the book “Oracle SOA BPEL Process Manager 11gR1 – A Hands-on Tutorial” ISBN: 978-1-84968-898-7. It is the comprehensive hands-on industry-leading-practice guide to deliver real-world SOA composite applications using the Oracle SOA Suite BPEL Process Manager platform. Ravi blogs at www.ravisaraswathi.com. Jaswant blogs at techblog.baghel.com. Ravi was also the editor of KAGW 2007.

 


Technology Trends – Cloud Computing:

In this edition of the technology corner, we are introducing and discussing one of the leading technology trends called cloud computing. It doesn’t matter whether you are an IT or non-IT professional, business owner and/or simple home user; cloud computing is transforming your day-to-day life and business activities. In this article, one will learn about cloud computing and how it is changing our day-to-day digital life.




What is cloud computing? We all consume software services for our day-to-day life and business activities. These software runs on the platform and platforms are created using infrastructure components such as network, compute and storage. In traditional setup, we own the software, platform & infrastructure and responsible for maintenance. In other words, we spend for initial setup and spend for the refresh and operations for the life. The cloud computing offers an option to consume software, platform and/or infrastructure services as pay-per-use model like utility bills. We only pay for the time we used these services without incurring the initial setup and maintenance cost. The building blocks for software services are software, platform and infrastructure. The cloud computing primary delivery models are aligned with the building blocks of the software services. These delivery models are Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS) and Infrastructure-as-a-Service (IaaS).

So What? I am an IT professional. How does it change my life? I am a business owner. What benefits it has for me? I am not an IT professional. Why do I care? I am a general user at home? What it offers to me? 

Why cloud computing? Ease of setup, access, automation, flexibility and utilization; managed by industry experts that let you focus on your core competencies; and no setup cost.

Cloud computing is transforming life for everyone. For business owner, it provides opportunities to transform the capital expense into operational expense for their IT needs and focus on your business core competencies. For IT professional, it provides an opportunity to reduce the risk and time to deliver IT services for business function but poses a challenge as well to learn to deliver business function utilizing cloud computing. For non-IT professional, you can expect better availability, accessibility and standardization for the services you are using to run your day-to-day business activities.

In this article, we will discuss in detail the benefits for general user at home. We are assuming that general user these days are using some kind of computing,  communication and entertainment platform such as smart phone, smart television, tablet, laptop and/or desktop with internet access. For your surprise, all of us are already using Software-as-a-Service for our modern life style. Some of us are even consuming Platform-as-a-Service and Infrastructure-as-a-Service to run our personal and professional life.  

Many of us at some point in our life lost personal documents, contacts, photos, music and/or videos due to hardware issues with computers at home and not able to access them while away from home.  Imagine your desktop and/or laptop CPU, RAM and Disk sizes can be adjusted as needed in few moments and you are able to access and stop/start/restart from anywhere and you are guaranteed to not lose the your data with built-in recovery.  It is all possible with cloud computing when you are utilizing Infrastructure-as-a-Service to provide desktop services. 

You can quickly create your personal blog, web sites and web applications with fraction of cost leveraging Platform-as-a-Service of Cloud Computing. 

We all are using Email, Contacts, Calendar, Photos, Videos, Movies, Music, Documents and/or Drive as Software-as-a-Services from various service providers to run our day-to-day personal activities. The cloud computing delivery of these services enables easy collaboration & sharing and access from anywhere using a smart phone, smart television, tablet, laptop or desktop with internet. One needs to select a service provider based on their brand value, reputation and service offering as most of us are consuming the freemium services. Freemium is a business model by which a proprietary product or service is provided free of charge, but money is earned by either advertisements or charge for advanced features. The leading brands for these services are Goolge, Amazon, Apple, Yahoo, MSN, and Aol. Imagine your email, contacts, movies, music and/or documents are available to us anywhere we go. It is all possible with cloud computing. Most of us have not used CDs and DVDs for Music & Movies for several years already and in near future we will not find CDs and DVDs similar to Audio or Video Tapes.

Cloud Computing is enabling us to not worry about losing and/or upgrading a software, smart phone, smart television, tablet, laptop and/or desktop. Our critical data and services are always available and accessible from anywhere.

Tuesday, June 25, 2013

My SOA book co-authored with Jaswant - Oracle SOA BPEL Process Manager 11gR1 – A Hands-on Tutorial is published and available. 


ISBN-13: 9781849688987
Publisher: Packt Publishing

Book is available from the following Websites:

What you will learn from this book

Learn BPEL language and create basic and complex BPEL processes using JDeveloper and Oracle SOA Suite BPEL Process Manager Platform
  • Follow step-by-step instructions for creating and configuring adaptors, human workflow, and advanced BPEL orchestration techniques
  • Practise deploying, testing, debugging, tuning, error handling, and troubleshooting Oracle SOA Suite Platform and BPEL processes
  • Integrate BPEL Process manager with Oracle Service Bus, Business Rules Engine, Business Activity Monitoring (BAM), and Single-Sign-On (SSO)
  • Learn how to achieve scalability, high availability, and failover-failback capabilities to deliver Business Continuity and Disaster Recovery (BCDR) objectives
  • Discover the evolution and future of BPEL and compare BPEL with BPMN and other leading process technologies









In Detail
BPEL, Business Process Execution Language is the definitive standard in writing and defining actions within business processes. Oracle BPEL Process Manager R1 is Oracle’s latest offering, providing you with a complete end-to-end platform for the creation, implementation, and management of your BPEL business processes that are so important to your service-oriented architecture.

"Oracle SOA BPEL Process Manager 11gR1 – A Hands-on Tutorial" is your guide to BPEL design and development, SOA Suite platform troubleshooting, and engineering in a detailed step-by-step guide working real-world examples and case studies.

Using industry-leading practices you will start by creating your first BPEL process and move onto configuring your processes, then invoking, orchestrating, and testing them. You will then learn how to use architect and design services using BPEL, performance tuning, integration, and security, as well as high availability, troubleshooting, and modeling for the future.

"Oracle SOA BPEL Process Manager 11gR1 – A Hands-on Tutorial" is your complete hands-on guide to Oracle SOA BPEL Process Manager 11g.

What you will learn - Details
  • Learn BPEL language and create basic and complex BPEL processes using JDeveloper and Oracle SOA Suite BPEL Process Manager Platform
  • Follow step-by-step instructions for creating and configuring adaptors, human workflow, and advanced BPEL orchestration techniques
  • Practise deploying, testing, debugging, tuning, error handling, and troubleshooting Oracle SOA Suite Platform and BPEL processes
  • Integrate BPEL Process manager with Oracle Service Bus, Business Rules Engine, Business Activity Monitoring (BAM), and Single-Sign-On (SSO)
  • Learn how to achieve scalability, high availability, and failover-failback capabilities to deliver Business Continuity and Disaster Recovery (BCDR) objectives
  • Discover the evolution and future of BPEL and compare BPEL with BPMN and other leading process technologies
Approach
This hands-on, example-driven guide is a practical getting started tutorial with plenty of step-by-step instructions for beginner to intermediate level readers working with BPEL PM in Oracle SOA Suite

Who this book is written for
Written for SOA developers, administrators, architects, and engineers who want to get started with Oracle BPEL PM 11g. No previous experience with BPEL PM is required, but an understanding of SOA and web services is assumed

Table of Contents

Preface
Chapter 1: Creating Basic BPEL Processes
Chapter 2: Configuring BPEL Processes
Chapter 3: Invoking a BPEL Process
Chapter 4: Orchestrating BPEL Services
Chapter 5: Test and Troubleshoot SOA Composites
Chapter 6: Architect and Design Services Using BPEL
Chapter 7: Performance Tuning – Systems Running BPEL Processes
Chapter 8: Integrating the BPEL Process Manager with Service Bus, Registry, and SOA Deployment
Chapter 9: Securing a BPEL Process
Chapter 10: Architecting High Availability for Business Services
Chapter 11: The Future of Process Modeling
Chapter 12: Troubleshooting Techniques
Index
 http://www.amazon.com/dp/1849688982/?tag=packtpubli-20


SOA Implementation

Enabling or implementing Service Oriented Architecture for your organization has nothing to do with implementing SOA Suite software! You enable re-use of your applications by implementing the loose coupling (Web services) and achieve application agility. You can say that you are successful with implementing SOA when you enable reuse of applications or services. 
 
I see many companies purchase SOA Suite and implement few new applications then claims that they implemented SOA! Achieving interoperability and loose coupling with standard well defined re-usable interface is important for any SOA implementation.

The views of SOA may be different for business, IT Architect, Developer or an IT manager, however just deploying few applications in SOA Suite will not enable SOA for your organization.  

SOA Suite enable loose coupling by creating the web services from the monolithic systems to enable HTTP access of web apps/services.  It doesn’t matter what language, operating system or protocol being used for your applications or legacy systems.

The bottom line for SOA implementation is implementation of Web services. In any organization, the port number 80 (HTTPS – 443) of all webservers is open! So any organization can access the web services published by any other organization. Of course you will have appropriate authentication and authorization built as part of the web services.

Web services provide a interoperable messaging framework. WSDL (Web service description language) provide a description of web services. WSDL answers the following questions about the web services. 

What (what this service does), How (This service gets invoked - HTTP, SMTP, Queue) & Where (where is this service located?) The WSDL contains Types (XML definitions usually you put import XML schema), Messages, Port Types, Binding and Services.

 

Monday, June 24, 2013

SOA Suite Internals


Question: Which Operating system environment SOA Suite runs well ? 

Answer: PowerPoint !. 


SOA Suite application (Deployed on top of WebLogic) consists of the following modules/engines (Yes, I consider SOA Suite is an application deployed on top of WebLogic) 

  • Mediator 
  • BPEL
  • Human Task
  • Business Rules
  • UMS (User Messaging Service) 
  • BPMN (You need BPM Suite) 


Mediator
Mediator will do the routing. It is a software based router! It can do dynamic routing or content based routing. Mediator can filter content from the SOAP header information. The SOAP does not get in to composite application if we have mediator to do the validation. Mediator does VETRO Pattern (Validate, Enrich, Transform, Routing, and Operate). Mediator is part of SCA.

Do we need to validate the XML document before sending to BPELs? It may be a good idea to check the document based on the content or syntax.

Basically, Mediator does the following:

  • Validate XML Schema
  • Filter Data
  • Transform XML data
  • Route data from Service client to target services
  • Manage event as a subscriber and publisher


Every event will have a name and an XML document associated with it. Mediator can be configured to listen for events from clients. The message exchange can be one-way or request-response. You can add error handling with Mediator such re-try logic.

We can assign threads for each of SOA Suite service engine. I don't think that the service engine threads relates to WebLogic threads. WebLogic threads are allocated dynamically by work manager. It is a kind of self-tuning.


Mediator have ParallelWorkerThreadCount, ParallelMaxRowsRetrieved,Parallelthreadsleep etc. It is better to keep the default values instead of changing it without knowing the full impact.

BPEL (Business Process Execution Language) Engine

BPEL is a markup language to orchestrate services. It is designed to integrate web services. BPEL processes are executable and can run over long time. The entire language looks like flow charts. It is a programming language built on top of XML specifications.  


 User Messaging Service (UMS)

Enable communication between Users and deployed applications. You can send email notification from composite applications to users using UMS. UMS Can be invoked as a web service or using RMI from an EJB. All messages will be stored in a database. The UMS has a server and an EMail Driver. On Enterprise Manager Console, select "User Messaging Service" from the left navigation to configure UMS. SMTP Server name and protocol can be added to the configurations. You can invoke UMS from your BPEL to send and receive the messages from users.  

Dehydration store 

Dehydration store is couple of Table in Data base created as part of RCA Script for storing transaction data.  
 In SOA Suite, some of the transactions may be running in days or weeks! The container may restart during this period. Dehydration store is required to store these transactions and restart when the container comes back to original state. 

Dehydration makes sense only on Asynchronous transactions. Assume that you have wait time defined between BPEL activities and you shut down the SOA Suite during the wait period. The Dehydration store will ensure that the wait time is honored when your start the SOA Suite container.


LDAP integration with SOA Suite: 
SOA Suite has embedded LDAP. You can add external LDAP as well. JXplorer  software can use to connect to LDAP directory. Other available LDAP browsers are openLdap browser and ldapbrowser.

UDDI:

According to the UDDI model, each organization should have separate UDDI registries that publish the available web services. The UDDI is a scam!. I couldn't find any organization that has successfully implemented run time UDDI for heavy transaction business critical applications for production environment.  


Composite application:

Composite applications are web services!

How do we group together all web services? You can use composite application. It enables course grained services that composed of different components. This is an end product of SOA Suite developed components.  All composite application will have a composite.xml. All ear file will have an application.xml. These are called deployment descriptors.

SCA define collection of speciations. Service component architecture (SCA) specification provides a way for developing an assembled SOA application.

You can combine all the application components such as EJBs, Mediator, Services, Adaptors etc. in to the single file called as composite application.

Usually the naming convention of composite application is SCA_<Composite Name>_1.0.jar. If a new version of application is deployed then the entire request will go to new version. The new version will becomes default composite application. The enterprise manager console indicates the default application with a dot. You can mark the old version as retired. Any active instance will finish its run, however new request will go to new default version of the composite application.

Always deploy the SOA composite applications as bundle. Use SAR (SOA Archive). You can use ant-sca-package script for creating the SOA Archive, or use JDeveloper.
Other option is to use sca_package scripts from WLST. The main advantage of SAR file is, all composite application gets deployed or none. The container will roll back all the deployments in case one of the composite application fails to deploy.

SOAP Message

A typical SOAP Message consists of the following: 

<SOA:Envelope>
<SOAP: Header>
</SOAP:Header>
<SOAP:Body>
<SOAP: Part name="desc">
<xml>
<SOAP: part name="pic">
<xml>
</soap:Body>
</soap:Envelope>



Monday, May 27, 2013

My "SOA BPEL Process Manager" book is to publish soon

http://www.packtpub.com/oracle-soa-bpel-pm-11g-r1-tutorial/book
(Authored by me and my friend Jaswant)

 http://www.packtpub.com/oracle-soa-bpel-pm-11g-r1-tutorial/book

 

Saturday, November 24, 2012

HP Pavalion DV6 -Windows 7 Wireless connectivity issue

I used to loose internet connetion once in a while with my new HP Pavalion DV6 -Windows 7 laptop. Found a solution today !.

Go to Control Panel  or windows 7 search > Device Manager > locate the wireless network adapter in the list, right click and choose Properties. On the last tab, Power Management, uncheck the box for "Allow computer to turn off this device to save power"

Hope this helps some one !.

Saturday, September 8, 2012

Hadoop - my notes !!


Hadoop is for parallel processing. Use for distributed processing of large data across clusters. It brings the ability to cheaply process large amounts of data, Example: 10 to 100 gigabytes or above.  It can use for ETL. It is a platform for data storage. Hadoop is written basically on java; however it uses some of the native libraries as well.  Instead of using costly data warehouse software, you can use Hadoop! It has ability to process non-structured data as well.
  • Map Reduce - It is a process engine and a distributed computing framework. It will distribute the large file in to small separate files. Processes large jobs in parallel across many nodes and combines the results. Reading around 4 Terabyte data may take some time as reading speed usual low as 55 Mega Bytes per second. 
  • Apache Hive – SQL-like language and metadata repository
  • Apache Pig – High level language for expressing data analysis programs
  • Apache HBase – Hadoop database for random, real-time read/write access
  • Apache Mahout – Library of machine learning algorithms for Apache Hadoop
  • Apache Oozie – Server-based workflow engine for Hadoop activities
  • Apache Zookeeper – Highly reliable distributed coordination service
  • Apache Sqoop – Integrating Hadoop with RDBMS
  • Apache Flume – Distributed service for collecting and aggregating log and event data
  • Fuse-DFS – Module within Hadoop for mounting HDFS as a traditional file system
  • Hue – Browser-based desktop interface for interacting with Hadoop. GUI (Provide authentication, workflow etc)
  • HDFS - Distributed, scalable, fault tolerant file system.  High bandwidth clustered storage - Distributed file system. Store the files in blocks and store the data across the cluster. The default block size is 64M and it can increase up to 128M. There can be many nodes (e.g. 10 or 1000)
It stores three replicas for data. So if one node goes down then it should not be a problem. Hadoop uses the internal algorithm to locate the block of the data. First node is selected randomly.
Fair scheduler - Priority pool and non priority pool.
Interface to HDFS: (How to get data in out of Hadoop).Use command line. Use commands to export and import files from regular file system to Hadoop.
Scheduling - OOZE.  No rollback available in Hadoop !!

Webmethod

 Webmethod is not a product; it contains a group of products.  Will webmethod better than Oracle SOA Suite and Service bus for creating services ?? Webmethod do not support BPEL. Software AG believes BPMN is the future. !!
Main Webmethod containers:
  • Webmethod integration Server - It is a Tomcat Container. Provide similar functionality as Oracle SOA or Service bus.  Webmethod DSP (Dynamic Server Pages) is similar to JSP. Integration server provide transformation and translation service. Development is done on the server itself! It has lock and unlock functionalities similar to check in and checkout.
Example of a service is "shipping", "billing" etc. Service is different from business process. Service is similar to java class. 
One of available service is “Flow Service” - GUI available such as "repeat", "Flow", "Sequence (Arrange and group together)", "loop", etc for transformation and translation.
  • Webmethod broker - Messaging platform. Publish subscribe. (No programming) use Designer for change. Write trigger service to subscribe it from broker.
  • My WebMethod Server - Portal container. It manages all Webmethod products. Have default Portlets for many WebMethod products. My WebMethod server is responsible for administering all WebMethod products. A Single GUI to administer all WebMethod products.
  • Central site is similar to Registry and repository.
  • Webmethod designer IDE is based on Eclipse - Can create Portlets and BPM.  To create web services use development tool: Webmethod Developer.
One of the advantage of WebMethod is it automatically create a script and image file during the install which allow administrators to install further installs from the original installation.
Software AG market Webmethod product. Broker is written in C language. Other Webmethod products are written in java. Broker clustering is complicated. My WebMethod server is used for administering Integration server and other products. Designer is used for creating Portlets.
Designer is nothing but an eclipse IDE with ads on.
Webmethod uses Terracotta or coherence for clustering!  I am not sure how to monitor of the services in WebMethods. All we could see the audit feature. Trading network can use for monitoring all the services. It is a B2B component.  
Software AG Webmethod BPM Suite consists of the following:
  • Webmethod integration server, MWS, Analytical Engine. Webmethod Designer, Rules Engine is an added product with Webmethod 8.
Central site and ARIS process governance (Work flow engine used between ARIS and Integration server) product are separate products. Central site and ARIS process governance is used for syncing process models between ARIS and Integration server. The license is included as part of the BPM Suite licensing.
Integration server has a process engine. MWS has a task engine. Prediction engine and Web service data collector are also included, Webmethod broker. Mash Zone is a separate product for creating dashboards (www.mashzone.com).
Governance product is not part of BPM Suite. Software AG partner with FICO and providing a rule/decision engine as part of Webmethod 8.
 ARIS 7.2 new version has the feature of import and export functionality.
IT has a scheduler available to pick up the files. Use fully qualified name of the service for scheduling. All the services in Webmethod integration server is web service. You can create WSDL file and invoke it. 
By using adaptors, integration server is used to integrate multiple systems. E.g. Integrate with mainframe, database, JMS, SAP etc.
Webmethod can pull file from another server or accept the file via http port.  Integration server can be in DMZ, but recommended way is to have one integration server in DMZ and configure reverse HTTP Gateway server configuration and have one Integration server in Secured network.