13 Apr 2010

Troubleshooting the BRM Pipeline Start Up

Oracle BRM 16 Comments

Issue 1
To see if all the pipelines are up, use this command: »pss | grep ifw«. The output should look like this:
$ pss | grep ifw
pin 18601 18486 0 10:48:02 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 24721 1 0 11:53:28 pts/0 01:00 bin/ifw -r ./custom/conf/dia_1_P.reg
pin 18603 18486 0 10:48:02 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18613 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18607 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18612 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18605 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18600 18486 0 10:48:02 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18618 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18616 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18611 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18614 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18486 1 0 10:48:00 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18610 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18608 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 19123 1 0 10:50:47 pts/0 00:18 bin/ifw -r ./custom/conf/rt_1_P.reg
pin 18490 18486 0 10:48:00 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18609 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18602 18486 0 10:48:02 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18617 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18615 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 19130 1 0 10:50:59 pts/0 01:29 bin/ifw -r ./custom/conf/mbi_1_P.reg
pin 18606 18486 0 10:48:03 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 18604 18486 0 10:48:02 pts/0 00:00 /opt/portal/pin/bin/dm_ifw_sync
pin 21975 1 0 11:23:18 pts/0 01:01 bin/ifw -r ./custom/conf/dcn_1_P.reg
pin 19126 1 0 10:50:53 pts/0 00:17 bin/ifw -r ./custom/conf/rt_1_S.reg

Issue 2
If you can’t start the pipeline and you get this error message: »Error: Could not start event handler, cannot proceed.», then you need to kill the event handler process. See the example that follows:
/opt/portal/pre_dia/log/process/ $ tail -f dia_1_P.out
;
Pipeline Server / Version 7.3 10095 started at 06.01.2009 16:23:11.
Copyright (c) 1998 – 2007 Oracle Corporation. All rights reserved.

Too many failed attempts at starting event handler daemon.
Error: Could not start event handler, cannot proceed.

Pipeline Server / Version 7.3 10095 finished at 06.01.2009 16:23:11.

Resolution:
$ pss | grep dia
pin 27727 1 0 15:50:21 pts/3 00:00 /opt/portal/pre_dia/bin/ifw_eventhandler64
pin 3055 4463 1 16:24:18 pts/3 00:00 grep dia
$ kill -9 27727

Issue 3
To rebalance the MBI pipeline, run the following command:
cd /opt/portal/pre_mbi/bin/
./pin_ctl snmpset rebalanceAaaGw aaa

To rebalance the DIAMETER pipeline, run the following command:
cd /opt/portal/pre_dia/bin/
./pin_ctl snmpset rebalanceAaaGw aaa

Issue 4
If you get this error message: »ERR_TAM_INIT_FAILED,« then you need to delete files in the tam directory.

bin/ifw: An error occurred during creation.
Originator : ifw.Pipelines.ExceptionPipeline.TransactionManager
Destination :
Message : ERR_TAM_INIT_FAILED
Arguments : ifw.Pipelines.ExceptionPipeline.TransactionManager
State : false
Severity : CRITICAL
Thread : 1

Resolution:
cd /opt/portal/
/tam
Backup the files to other location
rm TamTrans*.dat
rm ExceptionBinaryLogFile.dat

Issue 5
If you get this error message: »ERR_REG_LOCK_FILE_EXISTS,« then you need to delete the lock file.

*** Log not initialised.
*** Tried to write message ‘[T:1] ERR_REG_LOCK_FILE_EXISTS’
*** argument (0): ./custom/conf/rt_1_P.reg.lock

bin/ifw: An error occurred during creation.
Originator : ifw
Destination :
Message : ERR_REG_LOCK_FILE_EXISTS
Arguments : ./custom/conf/rt_1_P.reg.lock

State : false
Severity : CRITICAL
Thread : 1

Resolution:
cd /opt/portal/
/custom/conf
rm rt_1_P.reg.lock

16 Responses to “Troubleshooting the BRM Pipeline Start Up”

  1. Angshuman Mukherjee says:

    Hi,
    This blog si relly helpful and provides a good insight to Oracle BRm issues and soltuion. I have some specific questions regarding piepline configuration fil for a particular module. Can I ask the question in this blog?
    Thanks
    Angshuman

  2. Ales Pristovnik says:

    Hi Angshuman,

    You could post it here, or you could contact Goran Stumergar at goran.stumergar@tridens.si

    Best regards, Ales Pristovnik

  3. Kenneth Ducharme says:

    Hey, nice post, really well written. You should write more about this.

  4. RMV says:

    Hello ,

    I find this blog on BRM very interesting and informative. It would be nice if such posts are made more regualar.
    Also request you to include the scenerios of specific questions from others so that everybody can gain some inputs from it like the one above abt adding new modules to pipeline framework.And topics like MTA , multi threaded functionality to tune the BRM app performance.

    Just to add to troubleshooting part of it , the ifw_network_med , the network mediation pipeline can sometimes crash due to some memory segment faults when you have CDR Duplicate Checking enabled. So one way to troubleshoot it is to clean the dir that is configured to contain DuplicateData binary files eg $IFW_DATA/misc where misc is the said dir.

    Thanks & Regards
    RMV

  5. Darmowe says:

    This is like my third time coming by your site. Regularly I do not make comments on, but I have to mention that this article really pushed me to do so. Really awesome article!

  6. rafie says:

    Hi,

    I have a question about pipeline. I encountered a problem where the ifw_eventhandler64 is still running and the batch rating pipeline could not start. Do you know how to troubleshoot this issue?

    Thanks

  7. Ales Pristovnik says:

    Hi Rafie,

    You should begin with analyzing of log files.

    Best regards,
    Ales

  8. Deblina Dasgupta says:

    Hi Ales,

    I am trying to start the pipeline framework with simple.reg registry , to check if my installation is ok or not but getting the below Output file parsing error , not sure of this.

    [integrate@pindbhptest1 ifw]$ ifw -r samples/simple/simple.reg

    Pipeline Server / Version 7.4 10097 started at 10.09.2011 19:19:23.

    Copyright (c) 1998 – 2008 Oracle. All rights reserved.

    ifw: An error occurred during creation.

    Originator : ifw.Pipelines.TestPipeline.DataDescription
    Destination :
    Message : ERR_PARSE_DESCRIPTIONS
    Arguments : error `./formatDesc/Formats/Solution42/SOL42_V670_OutMap.dsc’: line 390: ERROR: failed to parse the output mapping description file

    State : false
    Severity : CRITICAL
    Thread : 2979621568

    Pipeline Server / Version 7.4 10097 finished at 10.09.2011 20:19:33.

    Please help.

    Regards
    Deblina

  9. admin says:

    Hi Deblina,

    I need more details before I can answer. Please provide error message in process log.

    Best regards,
    Ales

  10. channappa says:

    Can you/anyone please let me know what was wrong with LD_LIBRARY_PATH. We are facing issue with database connection while starting wireless.reg in LINUX Platform.

    ifw: An error occurred during creation.

    Originator : ifw.DataPool.Login.Module
    Destination :
    Message : ERR_CANNOT_OPEN_DATABASE
    Arguments : pindb
    INTEGRATE
    5A46BAEBC6C2C1C3A796C20A000E1E091066017D
    [DBNOTFOUND] No Access Module for

    We also could see the ‘liboci10g6312d.so’ is present at $PIPELINE_HOME/lib with all permissions.

    Please find LD_LIBRARY_PATH details:-
    LD_LIBRARY_PATH=/opt/ifw/lib:/home/oracle1/oracle/ product/10.2.0/db_1/lib32:/home/oracle1/oracle/pro duct/10.2.0/db_1/rdbms/lib32

  11. admin says:

    Hi,

    This error can be resolved by ensuring correct entries in the LD_LIBRARY_PATH and LD_LIBRARY_PATH_64 , as specified in the Oracle BRM installation documentation. Please set the environment variable LD_LIBRARY_PATH_64 properly; It should include Oracle/lib and also the $IFW_HOME/lib. Check if you are able to login to sqlplus with the same parameters as configured in the wireless.reg After you login into SQLPlus check for the records in the tables; If records are not present, please execute ifw_pipeline.sql.

    Best regards,
    Ales

  12. channappa says:

    Hi Ales,

    Thanks for the reply. We did check on the reply and it looks fine. Could connect to SQLPLUS from the credentails given in .reg file and entries are present in ifw_pipeline table.
    We have installed both 32 and 64 bit oracle client and giving reference of 64 bit client in the LD_LIBRARY_PATY_64.
    Can you please advise will it be causing problem or creating conflict.
    Below are the entries for your reference.

    LD_LIBRARY_PATH=$ORACLE_HOME/lib32:$ORACLE_HOME/rdbms/lib32

    LD_LIBRARY_PATH_64=$ORACLE_HOME/lib:$ORACLE_HOME/rdbms/lib:$IFW_HOME/lib

    Thanks,
    Channappa K C

  13. admin says:

    Hi Channappa,

    I need environment access for troubleshooting.

    Best regards,
    Ales

  14. Mahesh says:

    I am getting same kind of error as Ales . Can yoou let me know how you resolved it.
    integrate@pindbhptest1 ifw]$ ifw -r samples/simple/simple.reg

    Pipeline Server / Version 7.4 10097 started at 10.09.2011 19:19:23.

    Copyright (c) 1998 – 2008 Oracle. All rights reserved.

    ifw: An error occurred during creation.

    Originator : ifw.Pipelines.TestPipeline.DataDescription
    Destination :
    Message : ERR_PARSE_DESCRIPTIONS
    Arguments : error `./formatDesc/Formats/Solution42/SOL42_V670_OutMap.dsc’: line 390: ERROR: failed to parse the output mapping description file

    State : false
    Severity : CRITICAL
    Thread : 2979621568

    Pipeline Server / Version 7.4 10097 finished at 10.09.2011 20:19:33.

    Please help.

  15. Tom says:

    Hi There!, I have gone ahead and bookmarked your page on Reddit so my friends can see it too. I simply used your blog title as the title in my bookmark, as I figured if it’s good enough for you to title your blog post that, then you probably would lik…

  16. Pavan says:

    Hi All,

    This is a very usefull information. can any one plz provide me detail docs on BRM as iam beginner

Leave a Reply