Concurrent Processing Server

Concurrent Processing Server

There are reporting programs and data updating programs that need to run either periodically, or on an ad hoc basis. These programs, which run in the background while users continue to work on other tasks are run using the Concurrent Processing architecture. Concurrent Processing is an Oracle E-Business Suite feature that allows these non–interactive and potentially long-running functions to be executed efficiently alongside interactive operations on a specialized server, the Concurrent Processing Server.
Processes that run on the Concurrent Processing server are called Concurrent Requests.
When you submit such a request, either through HTML-based or Forms-based applications, a row is inserted into a database table specifying the program to be run. A Concurrent Manager then reads the applicable requests in the table, and starts the associated concurrent program.

Concurrent Manager:

• It is an executable that is registered as a program library within Oracle E-Business Suite, and which runs in its own operating system process.
• Runs operating system processes called target processes (often referred to as workers), each of which can start one concurrent program at a time.
• Can optionally run an immediate program (a concurrent program that runs in the concurrent manager’s own operating system process) eg: PL/SQL programs.
• Can be allowed to run any concurrent program, or be specialized to run certain programs.
• Operates during the days and times defined by a work shift.

Important Concurrent Managers:

• Internal Concurrent Manager (ICM): It controls the behavior of all of the other managers, and because the ICM is the boss, it must be running before any other managers can be activated. The main functions of the ICM are to start up and shutdown the individual concurrent managers as defined by their workshifts, monitors for process failure and cleans up if a failure occurs.

• Conflict Resolution Manager (CRM): It ensures that incompatible concurrent requests do not run together. However, the ICM can be configured to take over the CRM’s job to resolve incompatibilities.

• Standard Manager (SM): The SM functions to run any reports and batch jobs that have not been defined to run in any specific product manager.

Tuning the Concurrent Manager:

1. Tuning the Internal Concurrent Manager

2. Purging Concurrent Requests

3. Troubleshooting Oracle Apps performance problems

4. Adjusting the Concurrent Manager Cache Size

5. Analyzing the Oracle Apps Dictionary Tables

1. Tuning the Internal Concurrent Manager

The ICM performance is affected by the three important Oracle parameters PMON cycle, queue size, and sleep time.

• PMON cycle: This is the number of sleep cycles that the ICM waits between the time it checks for concurrent managers failures, which defaults to 20. You should change the PMON cycle to a number lower than 20 if your concurrent managers are having problems with abnormal terminations.

• Queue Size: The queue size is the number of PMON cycles that the ICM waits between checking for disabled or new concurrent managers. The default queue size of 1 PMON cycle should be used.

• Sleep Time: The sleep time parameter indicates the seconds that the ICM should wait between checking for requests that are waiting to run. The default sleep time is 60, but you can lower this number if you see you have a lot of request waiting (Pending/Normal). However, reducing this number to a very low value may cause excessive cpu utilization.

2. Purging Concurrent Requests

One important area of Concurrent Manager tuning is monitoring the space usage for the subsets within each concurrent manager. When the space in FND_CONCURRENT_PROCESSES and FND_CONCURRENT_REQUESTS exceed 50K, you can start to experience serious performance problems within your Oracle Applications. When you experience these space problems, a specific request called “Purge Concurrent Requests And/Or Manager Data” should be scheduled to run on a regular basis. This request can be configured to purge the request data from the FND tables as well as the log files and output files that accumulate on disk.

3. Adjusting the Concurrent Manager Cache Size

Concurrent manager performance can also be enhanced by increasing the manager cache size to be at least twice the number of target processes. The cache size specifies the number of requests that will be cached each time the concurrent manager reads from the FND_CONCURRENT_REQUESTS table. Increasing the cache size will boost the throughput of the managers by attempting to avoid sleep time.

4. Analyzing Oracle Apps Dictionary Tables for High Performance

It is also very important to run the request Gather Table Statistics on these tables:

• FND_CONCURRENT_PROCESSES

• FND_CONCURRENT_PROGRAMS

• FND_CONCURRENT_REQUESTS

• FND_CONCURRENT_QUEUES

Run the request “Analyze All Index Column Statistics” on the indexes of these tables. Since the APPLSYS user is the owner of these tables, so you can also just run the request Analyze Schema Statistics for APPLSYS.
To troubleshoot performance, a DBA can use various types of trace. A module trace, can be set by enabling the module’s profile option Debug Trace from within the applications. Second, to enable trace for a specific request, log in as a user with the System Administrator responsibility. Navigate to Concurrent -> Program -> Define. Query for the request that you want to enable trace. At the bottom right of the screen you can check the box Enable Trace.

Concurrent Request Life Cycle :

Phase	Activity
Pending	The request is waiting to be run
Running	The request is running
Completed	The request has finished
Inactive	The request cannot be run

Concurrent Processing Database Tables

Various database tables are employed by the concurrent processing architecture:

Table	Content
FND_CONCURRENT_REQUESTS	Details of user requests, including status, start date, and completion date
FND_CONCURRENT_PROGRAMS	Details of concurrent programs, including execution method, whether the program is constrained, and whether it must be run alone.
FND_CONCURRENT_PROCESSES	Cross-references between concurrent requests and queues, and a history of concurrent manager processes.
FND_CONCURRENT_QUEUES	Information about each of the concurrent manager queues.

Caution: Do not update these tables manually. You can (subject to your organization’s archiving requirements) periodically run the “Purge Concurrent Requests and/or manager data” program to prevent these tables growing too large.

Concurrent Processing Operations

Because the Internal Concurrent Manager controls all the other managers, it must be running before any other manager can be activated. Once the ICM has been activated, it starts a Service Manager on each node that is enabled for concurrent processing. Acting as an agent of the ICM, the Service Manager starts the concurrent managers on its node, excluding any managers that have been deactivated, or that have no current work shift. The ICM can be activated and deactivated from the operating system prompt, or Oracle Applications Manager. It can also be deactivated (but not activated) from the Administer Concurrent Managers form.
When the ICM is initiated on UNIX, the $FND_TOP/bin/startmgr program is invoked. This calls $FND_TOP/bin/batchmgr, which then:
Starts a shell process
Starts the ICM process using the command FNDLIBR, with startup parameters FND, CPMGR, and FNDCPMBR
Creates log files (std.mgr and wnnn.mgr) in $APPLCSF/$APPLLOG
Normally, startmgr is run by the user account that owns the application software (for example, applmgr). This account must have write privileges to the log and out directories where the log and output files respectively are written.
The ICM starts up a Service Manager on each node that is enabled for concurrent processing, by instructing the node’s Applications Listener (which is dedicated to Concurrent Processing) to spawn a process running the Service Manager executable (FNDSM). The Applications Listener must be configured to source the Oracle E-Business Suite environment file before FNDSM is spawned. Following startup, the Service Manager acts as an agent of the ICM to start and stop concurrent managers on that node, according to their defined work shifts.
Concurrent manager processes on a specific node can be seen by running the UNIX commands:

ps –ef | grep FNDLIBR
ps –ef | grep FNDSM

For every process that was successfully started at operating system level, the ICM inserts a row into FND_CONCURRENT_PROCESSES table. It then updates the RUNNING_PROCESSES column to reflect the actual running processes as shown in FND_CONCURRENT_QUEUES.

Viewing Concurrent Processing Output

The output from a concurrent processing job goes through several stages before being displayed to the user.
The Concurrent Processing server communicates with the database server via Oracle Net.
The log or output file associated with a concurrent request is passed back to the Report Review Agent, also known as the Web Review Agent.
The Report Review Agent passes a file containing the entire report to the Forms services.
The Forms services pass the report back to the user’s browser one page at a time.

                                  Viewing Concurrent Processing Output

Concurrent manager log files:

• O.S level:-


$INST_TOP/logs/appl/conc/log OR $APPLCSF/$APPLLOG
The following are the specific log file names.
Internal Concurrent Manager Log: *.mgr
Concurrent Manager Log: w.mgr
Transaction Manager: t.mgr
Conflict Resolution Manager: c.mgr
Request Log = l.req
Request Output = o.out

• SQL prompt:


Connect to sqlplus as apps user and run this query

SELECT ‘ICM_LOG_NAME=’ || fcp.logfile_name
FROM fnd_concurrent_processes fcp, fnd_concurrent_queues fcq
WHERE fcp.concurrent_queue_id = fcq.concurrent_queue_id
AND fcp.queue_application_id = fcq.application_id
AND fcq.manager_type = ‘0’
AND fcp.process_status_code = ‘A’;

Sample Output:


‘ICM_LOG_NAME=’||FCP.LOGFILE_NAME
————————————————————-
ICM_LOG_NAME=/applprod/comndev/admin/log/DEV_moon1/DEV_0426.mgr

• Front end:-


Log in as a user with the System Administrator responsibility.
Navigate to Concurrent -> Manager -> Administer -> (Select a manager) Processes -> Manager Log.

Location of concurrent manager output files:

$INST_TOP/logs/appl/conc/out OR $APPLCSF/$APPLOUT

Check status of the concurrent manager:

There are 3 ways to find the status of the manager:


• O.S level
ps –ef | grep FNDLIBR

• Front end:
System Administrator > Concurrent > Manager > Administer

• SQL script
SQL> @$FND_TOP/sql/afimchk.sql

Start / Stop concurrent manager:

$ADMIN_SCRIPTS_HOME
adcmctl.sh start apps/apps_pwd (To start the concurrent manager)
adcmctl.sh stop apps/apps_pwd (To stop the concurrent manager)
Individual managers can also be started from front end by navigating to System Administrator > Concurrent > Manager > Administer

Submitting a concurrent request:

Log in as a user with the System Administrator responsibility. Navigate to Concurrent -> Requests -> Submit a New Request -> Single Request -> Name: Active Users -> Submit

Finding a concurrent request:

System Administrator -> Concurrent -> Requests

You can filter the search results with the following parameters:
• Request ID: A particular concurrent request.
• Name: All concurrent requests with this name.
• Date Submitted: Date at which the request was submitted.
• Date Completed: Date at which the request got completed.
• Status: Current status of the request.
• Phase: Current phase of the request.
• Requestor: Request submitted by.

Concurrent Manager: troubleshooting

Summary of Possible Reasons and Solutions for the Problem Where All Concurrent Requests Stuck in Pending Phase

Below are several different possible solutions to the problem where concurrent requests are stuck in pending status:

1. When shutting down the concurrent manager are there any FNDLIBR processes still running at the OS level? If so, do a kill -9 on them. Then restart the concurrent manager.

2. Try Relinking $FND_TOP.

3. Rebuild the concurrent manager views. As applmgr run the following from the OS:

This is non-destructive. Concurrent Manager views can be rebuild by running the following command at the command line:

Ensure that concurrent manager is shutdown.

FNDLIBR FND FNDCPBWV apps/apps SYSADMIN 'System Administrator' SYSADMIN

restart the concurrent mgr.

4. Another possibility is that the Profile Option "Concurrent: OPS Request Partitioning" is set to OFF (by default). Unless Concurrent OPS is being used, the setting for this profile option should be NULL.

5. Yet another possibility is that System Profile Option: Concurrent Active

Requests is set to 0.

1. Log into Oracle Applications as SYSADMIN.

2. Select System Administrator responsibility.

3. Navigate to PROFILE --> SYSTEM.

4. Query for %CONC%ACTIVE%.

5. Change the profile option for Concurrent: Active Request Limit to Null (blank).

6. Exit Oracle Applications and log in again for the change to take affect.

7. Run a new concurrent request.

6. Another possibility is that Concurrent managers were brought down, while an outstanding request was still running in the background. In which case, Update the FND_CONCURRENT_REQUESTS table as follows:

sql> update fnd_concurrent_requests

set status_code='X', phase_code='C'

where status_code='T';

sql> commit;

7. The control_code for concurrent_queue_name = 'FNDCRM' is 'N' in the FND_CONCURRENT_QUEUES table, which means 'Target node/queue unavailable'. This value should be NULL (CRM is running; target and actual process amount are the same), or 'A' ('Activate concurrent manager' control status).

Set the control_code to 'A' in fnd_concurrent_queues for the Conflict Resolution Manager:

1. Logon to Oracle Applications database server as 'applmgr'.

2. Verify the Applications environment is setup correctly ($ORACLE_HOME and $ORACLE_SID).

3. Logon to SQL*Plus as 'APPS' and run the following SQL statement:

update fnd_concurrent_queues

set control_code = 'A'

where concurrent_queue_name = 'FNDCRM';

commit;

4. Verify the status of the concurrent managers through the

Concurrent -> Manager -> Administer form.

If the CRM is still not active, bounce (deactivate, activate) the Internal Concurrent Manager. This is done through the Concurrent -> Manager ->

Administer form from the 'System Administrator' responsibility. It can also be done through the CONCSUB command at the command level.

Setting the control_code to 'A' in the fnd_concurrent_queues table for the Conflict Resolution Manager indicates that this concurrent manager is to be activated with the parameter values specified through this table for this manager (MAX_PROCESSES, CACHE_SIZE, etc).

8. What is the cache size? Try increasing it then bounce the concurrent manager.

If concurrent requests are rarely prioritized and there are managers that service short-running requests, consider setting the cache size to equal at least twice the number of target processes. This increases the throughput of the concurrent manaagers by attempting to avoid any sleep time. For example, if more than one manager or worker processes the same type of requests with only a small cache size, it may be unable to process any jobs in a single processing cycle, because other processes have already run the cached requests. When this happens, it is important to note that the manager will sleep before refreshign its cache. To increase manager throughput where there are sufficient requests of the required type in the queue, increase the cache size to improve the chance of the manager finding work to process and thus avoid having to enter a sleep phase.

TIP: Ensure that the system is not resource-constrained before attempting to increase the rate of concurrent processing in this way; otherwise, these changes may actually reduce concurrent processing throughput because jobs take longer to run..

Enter the number of requests your manager remembers each time it reads which requests to run. For example, if a manager's workshift has 1 target process and a cache value of 3, it will read three requests,, and will wait until these three requests have been run before reading new requests.

In reading requests, the manager will only put requests it is allowed to run into its cache. For example, if you have defined your manager to run only Order Entry reports then the manager will put only Order Entry requests into its cache.

If you enter 1, the concurrent manager must look at its requests list each time it is ready to process another request. By setting the cache size at a higher number, the concurrent manager does not have to read its requests list each time it runs a request. However, the manager does not recognizea nay priority changes you make for a particular request if it has already read that request into its cache. Further, even if you give a higher priority to a new request, that new request must wait until the buffer isempty and the manager returns to look at the requests list. That request may have to wait a long time if you set the buffer size to a high number.

You should use cache size to tune your concurrent managers to work most efficiently for you site's needs. If your organization tends to reprioritize jobs going to a certain manager, that manager should have its buffer size set fairly low.

*Suggestion: *Enter a value of 1 when defining a manager that runs long, time-consuming jobs, and a value of 3 or 4 for managers that run small, quick jobs.

9. Please check the sleep seconds set for the concurrent manager which runs the report that is taking long time for completion. Reducing the sleep time will improve performance.

1. Check for the value set for the sleep seconds for the concurrent manager using the following path: System Administrator Responsibilty -> Concurrent -> Manager -> Define.

2. Check the processes running for the concurent manager using the following command:

ps -ef | grep 'INVLIBR'

As all concurrent programs are dealt by INVLIBR, the number of processes returned here must tally with the value specified for the processes defined for the concurrent manager.

3. Also, check for the duration for which the processes are sleeping. Reducing the value assigned for it would improve performance.

As a workaround to improve performance, you could change the priority of concurent request.

Explanation

-----------

The sleep state would be assigned to some tasks when the program running it has to handle more tasks than it has been configured to process. For example, if the number of processes has been defined as say 10, then at any point of time there is a provision to run 10 tasks. Now say, 12 tasks arrive for the program, then time slots are alloted for 10 and the rest 2 are put to sleep. So, if the concurrent program running the problem report is handling larger number of processes the sleep and run behaviour can be observed.