BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation

BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation - Opis

Control-M is one of the most widely used enterprise class batch workload automation platform. With a strong knowledge of Control-M, you will be able to use the tool to meet ever growing batch needs. There has been no book that can guide you to implement and manage this powerful tool successfully... until now. With this book you will quickly master Control-M and be able to call yourself a Control-M specialist!BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation will lead you into the world of Control-M and guide you to implement and maintain a Control-M environment successfully. By mastering this workload automation tool, you will see new opportunities opening up before you.With this book you will be able to take away and put into practice knowledge from every aspect of Control-M ñ implementation, administration, design and management of Control-M job flows, and more importantly how to move into workload automation and let batch processing utilize the cloud.You will start off with batch processing and workload automation, and then get an understanding of how Control-M meets these needs. Then we will look more in depth at the technical details of Control-M, and finally look at how to work with it to meet critical business needs. Throughout the book, you will learn important concepts and features, as well as learn from the Author's experience, accumulated over many years. By the end of the book you will be set up to work efficiently with this tool and also understand how to utilize the latest features of Control-M. Spis treści:
BMC (...) więcej Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation
Table of Contents
BMC Control-M 7: A Journey from Traditional Batch Scheduling to Workload Automation
Credits
About the Author
Acknowledgement
About the Reviewers
www.PacktPub.com
Support files, eBooks, discount offers and more
Why Subscribe?
Free Access for Packt account holders
Instant Updates on New Packt Books
Preface
What this book covers
Who this book is for
Conventions
Reader feedback
Customer support
Errata
Piracy
Questions
1. Get to Know the Concept
Introduce batch processing
The history of batch processing
Batch processing versus interactive processing
Time-based batch- and event-driven batch
Is this the end for batch processing?
Running batch processing tasks
Automating batch processing
Basic elements of a job
What to trigger
When to trigger (Jobs scheduling criteria)
Dependencies (Job's predecessors and dependents)
More advanced features of scheduling tools
Ability to generate notifications for specified events
Ability to handle an external event-driven batch
Intelligent scheduling decision-making based on predefined conditions
Security features
Additional reporting, auditing, and history tracking features
Centralized enterprise scheduling
Challenges in today's batch processing
Processing time
Batch window length
Batch monitoring and management
Cross-time zone scheduling
Resource utilization
Maintenance and troubleshooting
Reporting
Reacting to changes
The solution
Processing time and resource utilization
Batch monitoring and management
Cross-time zone scheduling
Maintenance and troubleshooting
Reporting
Reacting to changes
From batch scheduling to workload automation
Batch scheduling: Static scheduling
The Workload Automation concept
Dynamic batch processing with virtualization technology and Cloud computing
Integration with real-time system, workload reusability
Summary
2. Exploring Control-M
Control-M overview
Control-M road map
Key features
Supported platforms
The Control-M way
Control-M job
Job conditions
Resources
Submitting jobs
Post processing
From the user's perspective - Control-M/Enterprise Manager
Control-M Enterprise Manager GUI Client
Control-M Desktop
Control-M Configuration Manager
Reporting Facility
Control-M's Optional Features
Control-M Control Modules
Control-M/Forecast and BMC Batch Impact Manager
Control-M/Forecast
BMC Batch Impact Manager
BMC Batch Discovery
Control-M Architecture and Components
Control-M/Enterprise Manager
Control-M/Enterprise Manager Server Components
Naming Service
Control-M Configuration Server
Control-M/Enterprise Manager Configuration Agent
GUI Server
Gateway process (GTW)
Global Alert Server (GAS)
Global Condition Server (GCS)
Control-M Web Server
Control-M/Server
Control-M/Server processes
SU: Supervisor
SL: Job Selector
TR: Job Tracker
NS: Agent Communication Process
CE: New Day and EM Communication Process
CS: Server Process
LG: Logger Process
WD: Watchdog Process
RT: Internal Communication Router
CA: Configuration Agent
Control-M/Agent
AG: Agent Listener, Request Handler
AT: Agent Tracker
AR: Agent Router Process
UT: Utility Process
Agentless Technology
Control-M/Control Modules
How do Organizations Work With Control-M?
Where to Start?
General Product information
Official Education and Certification
Getting a Job in Control-M
Summary
3. Building the Control-M Infrastructure
Three ages to workload automation
Stone age
Iron age
Golden age
Planning the Batch environment
Control-M sizing consideration
Total number of batch jobs run per day
Total number of job execution hosts
Number of datacenters
Amount of concurrent GUI users
Use Control-M/Agent or go Agentless
Production, development, and testing
Control-M high availability requirements
Control-M in a clustered environment
Control-M/Server mirroring and failover
Control-M/Server database mirroring
Control-M/Server failover
Control-M node group
High availability by virtualization technology
Pre-installation technical considerations
Environment compatibility
Choices of database
System configuration requirements
Linux Kernel parameters
Shared memory
Semaphores
User limits
Other requirements
Storage space related considerations for Control-M
Firewall requirements
Between Control-M/Enterprise Manager Clients and Server Components
Between Control-M/Enterprise Manager Server Components and Control-M/Server
Between Control-M/Server and Control-M/Agent
Agentless remote hosts
Database
Last things to make sure of before the installation starts
Installation
Install Control-M/Enterprise manager server components
Download and execute the check_req script
Create a Linux user and allocate space for Control-M/EM
Configuring the system to meet installation requirements
Preparing the installation media
Installation
Post-installation tasks
Install Control-M/Enterprise manager clients
Preparing the installation media
Installation
Post-installation tasks
Installing Control-M/Server
Installation in Linux environment
Pre-installation
Installation
Post-installation tasks
Installation in a Windows environment
Pre-installation tasks
Installation
Post-installation tasks
Installing Control-M/Agent
Installation in Linux environment
Pre-installation tasks
Installation
Post-installation tasks
Installation in a Windows environment
Summary
4. Creating and Managing Batch Flows with Control-M GUI
The Control-M way continued
Contents of a job definition
"What" #1: job type
"What" #2: task type
"Who" #1 owner of the job
"Who" #2 author of the job
"Where" #1 job's execution host
"Where" #2 storing job definitions
Datacenter/Table/Job
Application/Group/Job
"When" #1 job's scheduling date
Defining a job's scheduling date
Calendars
Rule-Based Calendar (RBC)
Retro job
"When" #2 time frame for job submission
"When" #3 cyclic jobs
"When" #4 manual confirmation jobs
"When" #5 job condition
"When" #6 resource and job priority
Quantitative resource
Control resource
"When" #7 time zone
What happens right after the job's execution is completed?
PostProc
Step
Autoedit facility
Autoedit variables
Autoedit expressions and functions
Lifecycle of a job
Write/Load, Upload/Download, Order/Force, and Hold
State of a job
New Day Procedure (NDP)
Active job ordering
Active job cleaning
Control-M Date and Odate
User Daily
Working with Control-M Desktop and EM GUI Client
Control-M Desktop the Workspace
Control-M/EM GUI client Active ViewPoint
Defining and running jobs
Creating the first job Hello World!
Write, Upload, and Order the job
Write
Upload
Order
Monitor and Control the Job
Job Sysout
Rerun a Job
Job Log
Job Statistics
Modifying and rerunning the job
Modifying the static job definition
Modifying the active job instance
A more complicated job flow
Defining SMART table, application, and group
Building cyclic jobs
Utilizing the Autoedit facility
Job submission variables
User-defined Variables
System Variables
Linking jobs with job conditions
Defining Global Conditions
Deciding the Global Condition pre-fix
Registering the Global Condition pre-fix
Creating calendars
Adding job post-processing and job steps
Post-processing
Job steps
Working with Resources
Quantitative Resource
Control Resources
Having a "Start" job
Summary
5. Administrating the Control-M Infrastructure
Additional component installations
Installation of BIM and Forecast
Installation
Post-installation tasks
Configuring BIM web interface
Installation of Control Modules
Pre-installation considerations
Installation Control-M for database
Installation Control Module for Advanced File Transfer
Interactive installation
Silent installation
Installation Control-M Business Process Integration Suite
Post-installation tasks
Importing CM-specific job editing forms
Installing CM utility add-ons into the CCM and Control-M/EM server
Expanding and updating the batch environment
Ongoing installation of Control-M/Agents and Control Modules
Installing multiple Control-M/Agents on the same host
Defining Agentless remote hosts
Unix/Linux remote host (using SSH)
Windows remote host (using WMI)
Applying Control-M fix packs and patches
When to apply fix packs and patches
How to apply fix packs and patches
Fix pack and patch installations in our environment
Installing additional Control-M GUI clients
Frequent administration tasks
Stop/start components
Manually stop/start components
Control-M/EM server components
Control-M/Server components
Control-M/Agent components
Configuring automatic startup script
Defining additional GUI users and groups
Authorization of configuration items
Active tab
Tables and calendars
Prerequisite conditions and global conditions
Quantitative and control resources
Owners
Privileges
Member of
Customizing Control-M GUI
Control-M/EM GUI
Control-M Desktop
Summary
6. Advanced Batch Scheduling and Management
Importing existing batch processing tasks
Importing CRON jobs into our environment
For Host ctm-demo-linux-01
For Host ctm-demo-linux-02
Enhance the file processing batch flow
Control-M filewatch
Technical background
Invoking methods
Filewatch rules
Defining filewatch job
Adding the job
Defining filewatch rules
SMART table level autoedit variable
Advanced file transfer
Technical background
Verifying destination file size after transfer
Verifying checksum
Restarting from the point of failure
Encryption and compression
Pre and post (transfer) actions
Filewatcher
Implementing AFT jobs
Creating an AFT account
Defining the AFT job
Control-M for database
Technical background
Implementing Database CM jobs
Creating a database connection account
Defining Database CM Job
Advanced batch management
ViewPoints
Viewing jobs in Active ViewPoint
Find feature
Dynamic filter
Print
Performing job actions in Active ViewPoint
Delete/Undelete
Kill
Force OK/Force OK with no post processing
Why/Enhanced Why
Bypass
Branch Menus and Neighborhood
Critical Path
The Time Machine Archived ViewPoint
Creating ViewPoint
Hierarchy
Collection and default filter
Forecasting
Job scheduling plan forecasting
Forecast ViewPoint
Managing batch flows as services
Defining services
Monitoring services
Control-M reporting facility
Type of reports
Creating a report
Automated reporting
The emreportcli utility
Reporting job
Summary
7. Beyond Everyday Administration
GUI alternative command-line utilities
Control-M/Server utilities
Control-M/Agent utilities
Securing the environment
User authentication: External authentication
Configuring LDAP parameters
Converting existing GUI users to authenticate with LDAP
Associating EM user groups with LDAP groups
User privilege: Control-M/Server security
Defining group-level security
Defining user-level security
Enabling Control-M/Server security
Job ordering and submission: User exit
Job execution: Control-M/Agent security
Security for Windows Control-M/Agents
Security for Unix/Linux Control-M/Agents
Control-M/Server utility authorizations
Inter-component communication firewall
Between Control-M/EM server components and GUI clients
Between Control-M/Server and Agents
Inter-component Communication SSL
Implementing SSL
Auditing
Enabling and configuring auditing
Producing auditing report
Control-M mirroring and failover
Pre-implementation tasks
Installing and configuring the secondary Control-M/Server
Configuring Control-M/Agents
Testing the secondary Control-M/Server
Initializing mirroring and failover
Switching to mirroring and failover
Switching over to mirroring
Switching over to failover
Recovering from mirroring and failover
Recovering from mirroring
Recovering from failover
Perfecting Control-M
Housekeeping
Active environment-related housekeeping
Statistic average calculation
Historical statistic average cleaning
Job condition cleaning
Exporting Control-M/Server Log (IOALOG/ctmlog)
Database-related housekeeping
Control-M/Server database statistics calculation
Control-M/Server health check
Control-M/Server database hot backup
Control-M/EM data backup
Filesystem-related housekeeping
Component status checking
NDP tuning
Things happening during NDP
Removing old ctmlog
Removing old job statistic information
Sending Sysout cleanup trigger
Deleting conditions
After active job cleaning and ordering
Shortening NDP
Removing old job statistic information outside NDP
Ordering jobs outside NDP
Other configurations items
Control-M/EM: MaxOldDay and MaxOldTotal
Control-M/EM: Default AverageTime
Control-M/Server: New Day Time
Control-M/Server: Simple Mail Transfer Protocol parameters
Control-M/Server: shout destination tables
Summary
8. Road to Workload Automation
Integrating Control-M with business processes
Building the environment
Interacting with BPI interfaces
Technical background
Defining an account
Triggering job ordering
Creating a project in soapUI
Sending SOAP request to the web services
Taking parallel processing to the next level
Merging the two file processing job flows
Enabling parallel processing
Adding table ID into condition names
Modifying the filesystem directory structure and job scripts
Updating the quantitative resource
Implementing control module for BPI jobs
Technical background
Defining accounts
Web service account
Message queue account
Creating jobs
Web service job
Message queue jobs
Updating the quantitative resource
End-to-end testing
Managing batch jobs as workloads
Running jobs in node groups
Technical background
Creating node groups
Making necessary changes to the environment and jobs
The big picture
Making changes to jobs and scripts
Making changes to quantitative resources
Putting into action
Defining and managing workloads
Technical background
Defining workloads
Putting into action
Into the Cloud
Technical background
Defining accounts
Defining jobs
Modifying the file processing job flow
Defining CM for Cloud jobs
End-to-end testing
Summary
Index O autorze: Qiang Ding (Melbourne, Australia) has been working within the Control-M space for more than a quarter of his life. During his early days at BMC Software, Qiang resolved countless number of critical technical issues for Control-M customers around the world from Fortune 500 companies to government organizations. In recent years, Qiang has travelled hundreds thousands of miles around Australia and the North AP area to help many organizations to design, manage, and optimize their batch workload automation environment and to extend his passion to others by delivering Control-M trainings to end users and BMC Partners. Currently Qiang is temporary living in Sydney and working on a enterprise wide Control-M migration and Consolidation project for a major Australian bank. He enjoys working with other experts in the field and is constantly involved in finding ways for making improvements to the batch environment that he works on. mniej