Patch troubleshooting issues

PATCH TROUBLESHOOTING FOR ORACLE APPLICATIONS RELEASE 11

1.  When applying patches, where should the patch files reside for adpatch

     to find them?

    The patches can be copied anywhere as long as the environment is set correctly when

    executing adpatch. You may usually find that these are in a directory under $APPL_TOP.

2. Which drivers should be run first?  If they are run in the wrong order can

    they be run again?

    Always read the readme.txt.  In general for Release 11, patches can include three

    different drivers: "c", "d" and "g". The "C" drivers (copy) are file system drivers, these

    are used to update the file system with the new versions of certain files.  The "D"

    drivers are database drivers and they run SQL scripts that update database tables and

    objects.  The "G" (generate) drivers are also file system drivers, they generate forms

    and reports.  These drivers are run in alphabetical order c, d, then g.  If they are run

   out of order, rerun them in the correct order.

3. Where (on which tier) do you run the three different patch drivers?

   The "c" and "g" drivers are run on all tiers.  The "d" driver is only run on the database

   tier. In Applications Release 11, there can be five different tiers: concurrent processing

   server, forms server, administration server, web server, and database server.  When

   adpatch is run the answers to the configuration questions will determine the actions

   adpatch takes. The configuration questions are the same questions asked by adadmin

   and adaimgr. 

  3a.  Do you currently have files used for installing or upgrading the database installed

         in this APPL_TOP?

  3b.  Do you currently have Java and HTML files for Self-Service Applications installed

         in this APPL_TOP?

  3c.  Do you currently have Oracle Applications forms files installed in this APPL_TOP?

  3d.  Do you currently have concurrent program files installed in this APPL_TOP?

4. What is the basic process that adpatch runs with the 'c' driver?

    The "c" (copy) driver updates the file system.  Below is a portion of a patch and an

    explanation of the activities  within the patch.

               copy            fnd    admin/sql   fnddold.sql  110.4
               copy            fnd    forms/US    FNDADDTC.fmb 110.4
               libout          fnd    lib               afdict.o
               copy            fnd    lib               afdict.o110.1
               libin             fnd    lib              afdict.o
               forcecopy    fnd    media        FNDIEROR.gif
               link              fnd    bin             FNDCLOAD

4a. copy   fnd  forms/US  FNDADDTC.fmb  110.4

ADPATCH performs version checks allowing it to not overlay a higher version object

or sql files.  If the version on the file system is the same or higher it skips the copy. 

In the above example, notice the version to the right of the filename.  It uses this

version to do the compare.  The "fnd" next to the copy tells ADPATCH what applications

top to use and "forms" points adpatch to the proper directory. With the above statement

ADPATCH will copy the file FNDADDTC.fmb (from patch directory) to $FND_TOP/forms. 

The copy actually moves the existing FNDADDTC.fmb file to FNDADDTC.fmbO and then

physically copies the new file version to FNDADDTC.fmb.  The file FNDADDTC.fmbO will

remain on the file system.

4b.  libout   fnd   lib   afdict.o

Libout is a utility that removes files from the library archive.  The archive file resides in

the lib directory under the TOP directory of the application.  The name of the file is

lib<application>.a.  So the archive file for FND is located in $FND_TOP/lib and is named

libfnd.a. The actual command that libout executes is:

     ar -d libfnd.a afdict.o

This removes the object file (afdict.o) from the library file (libfnd.a).

4c. libin  fnd  lib  afdict.o

Libin is the utility that adds an object file to the library file.  The actual command that

libin executes is:

     ar -rv libfnd.a afdict.o

The object file (afdict.o) is appended to the bottom of the library file. Libin is executed

even if the copy did NOT take place because the information was already removed with

the libout command.

4d.  forcecopy    fnd  media  FNDIEROR.gif

Similar to copy, but no version checking is done. Forcecopy copies the target file

irregardless of the presence or version of the file currently on the target system.

4e.  link  fnd  bin  FNDCLOAD

The link process is similar to a compile.  It links new object files together with the other

objects files to create the executable.  The command that the above link is using:

      adrelink force=y "fnd FNDCLOAD"

The "fnd" is the top that the executable is being relinked in.  The executable FNDCLOAD

has to be in the proper case that it would normally appear in the $FND_TOP/bin directory.

5. What does adpatch actually do with a 'd' driver?

    The "d" (database) drivers alter the tables in the database. These patches can create

    tables, add indexes and alter the APPS schema. The only way to back-out a 'd' driver is

    with a database backup. Even if the executables are restored (on the file system), they

    will not work because information in the database has changed.

6. What does adpatch actually do with a 'g' driver?

     The "g" (generate) drivers generate forms, .plls (PL/SQL libraries) reports and menus.

     The following is an excerpt from a "g" driver:

                 genform      fnd    forms/US         FNDWFNOT.fmb 
                 genrep        fnd    reports            FNDCPWKM.rdf
                 genfpll        fnd    resource          JL.pll 
                 genmenu    fnd    resource/US    FNDMENU.mmb 

6a.  genform uses f60gen to generate forms.

       Syntax is: 
                 f60gen userid=APPS/APPS module=<form name>.fmb 
                 output_file=<full path of schema top>/forms/US/<form name>.fmx 
                 module_type=form batch=no compile_all=special

6b.  genrep uses rwcon60 to generate reports

       Syntax is:
     rwcon60 userid=APPS/APPS 
     source=/<full path of schema top>/reports/<report name>.rdf 
     dest=/<full path of schema top>/reports/<report name>.rdf stype=rdffile 
     dtype=rdffile logfile=<report name>.log overwrite=yes batch=yes dunit=character

6c.  genfpll uses f60gen to generate .plls (forms libraries)

       Syntax is: 
          f60gen userid= APPS/APPS module=<library name>.pll module_type=library     
          output_file=/<full path of schema top> /resource/<library name>.pll
          compile_all=Yes

6d.  genmenu uses FNDMDGEN to generate menus
       Syntax is: 
                 FNDMDGEN APPS/APPS 0 Y <language codename>
                 DB_TO_RUNTIME  <application shortname> <message filename>.msb

7.  What happens when a patch is applied where the version is lower than the

      current version?

     Higher versions contained within the patch are copied, the lower versions

     are ignored.

8.  Which patches can be backed out should there be a problem?

     It is always recommended that you perform a cold backup of a system prior

     to applying any patch.  It is also recommended that you apply patches to a

     test system first.  However, for those cases where you have not done these

     two steps; you should know that database drivers 'd' CANNOT be backed out

     without a system backup. If the "c" or "g" drivers need to be backed out,

     reverse the steps from the "c" driver and relink, then reverse the steps from

    the "g" driver. This will require you to manually review the driver files and

    create commands that will do the opposite of the process.  There is no utility

    that you can use to reverse the patch. For the section of the above patch, you

    need to do the following:

        libout  fnd    lib    afdict.o
        copy    fnd    lib    afdict.o    110 
        libin     fnd    lib    afdict.o

   Dearchive afdict.o from libfnd.a, then copy rename afdict.oO to afdict.o  and

   then archive afdict.o into the library file.

8a. libout    fnd   lib  afdict.o

      ar -dv  libfnd.a  afdict.o

8b. copy   fnd   lib  afdict.o  110

      To reverse this:

      cp afdict.o0  afdict.o

8c. libin   fnd    lib  afdict.o

      ar -rv libfnd.a afdict.o

      This will place the old object file back into the library file.

8d.  link  fnd  bin  FNDCLOAD

       Now the executable needs to be relinked.

       adrelink force=y ranlib=y "fnd FNDCLOAD

8e.  copy fnd  forms/US  FNDADDTC.fmb 110.4

       This was the set that copied the file from the patch directory to  $FND_TOP/forms. 

       To reverse it:

        cp  FNDADDTC.fmbO  FNDADDTC.fmb

       Adpatch command to generate the form:

       genform   fnd  forms   FNDADDTC.fmb

       To reverse it:

        f45gen userid=APPS/APPS module= FNDADDTC.fmb
        output_file=$FND_TOP/forms/US/ FNDADDTC.fmx 
        module_type=form batch=no compile_all=special

9a.  You notice that while running ADPATCH you encounter an error and

       you are being prompted if you want to proceed.  What are the types

       of errors and what should you do?

       There are two types of errors: Fatal in which ADPATCH halts and Non-Fatal

       errors where users is prompted if they want to proceed. If the user answer

       Yes, adpatch skips the operation it was trying to complete and notes in the

       $APPL_TOP/install/log/adpatch.log file that it was completed. This could be

       the cause of problems later on.

9b.  Should you proceed? 

        This is a very hard question. On some patches the answer is NO on others

        it does not matter. This has to be determined on a case by case basis.

10.  You answered "Yes" to proceed with the patch when it encountered a

       Non-Fatal error. How can you get ADPATCH to execute that operation again?

        The only way to get back to the same point is to restart the patch from the beginning.

11.  What log files are generated by ADPATCH and how are they named?

        The log file created by ADPATCH is named by the user when they start ADPATCH. 

       So the log filename is user defined.  If the person applying patches does not give

       it a unique name each time, the log information is appended to the end of the

       existing file.  That file is located in $APPL_TOP/admin/log and is named adpatch.log.

       This means that the ADPATCH log file could have references to many patches. Some

       patches may spawn workers and log file will be generated for each worker. These

       log files are named in the following way: 

       adwork01.log, adwork02.log, etc... and are located in $APPL_TOP/admin/log.

12. How is the environment set and what kind of errors would ADPATCH

      generate if there was an issue with the environment?

      ADPATCH should be run as APPLMGR, the owner of the APPL_TOP.  The login

      should execute the environment ile so ADPATCH should have this set. The errors

      you might encounter with an environment problem would be:

      ADPATCH will not know how to copy and will not be able to  relink. 

      In that case, you might  get messages like:

      Don't know how to make.

13. What happens if adpatch fails on a PL/SQL script?

      If adpatch fails on a PL/SQL script you can try running it from the command line. 

      If it runs, then you can  continue as if successful. You can get the syntax and

      parameters for the script from the ADPATCH.log file.  This will help you know who

      the user was who ran the  script and where the script is located.

14.  What happens if the patch fails?  You are being prompted to restart

       the patch when you start ADPATCH.

       If the patch fails you can try to restart the patch.  The patch will start from

       where it failed.  The other option will be to start over.  There are two questions

      ADPATCH prompts for when restating a patch.

       Do you want to continue with the previous session?
       Are you sure you don't want to continue with the previous install?

       The default is to restart the patch, so if you want to decline this option they

       will have to answer  "NO" and then "YES".

15.  You have decided to start all over and resubmit ADPATCH from the

       beginning, but you get the error that FND_INSTALL_PROCESSES already

      exits. How is FND_INSTALL_PROCESSES related and why does ADPATCH

       have a problem with it?

       FND_INSTALL_PROCESSES is a table that is created by RAPID INSTALL, ADPATCH

      and ADADMIN  utilities. Normally the above utilities will remove this table after

      they complete. This table might exist if you are running any of the above processes. 

      So, if you are also applying  another application patch at the same time, the table

      might exist for the other patch. Dropping the FND_INSTALL_PROCESSES table should

      only be done if you are NOT in the middle of running ADADMIN or applying another

      patch. Consider this scenario:  You are running the install and you get instructions to

      obtain a patch (usually this would be an FND patch) and you attempt to use ADPATCH

      to install the patch. If you  were to drop  FND_INSTALL_PROCESSES at this point, you

      would have to start your install over.  However, you could backup FND_INSTALL_PROCESSES

      and the restart files.  Then run adpatch to install the patch.  Then restore

      FND_INSTALL_PROCESSES and the restart files, and continue the installation.

      If you want to drop the table, you will need to login as the owner of the FND tables.

      Normally this is APPLSYS. Then type the following:

                drop table fnd_install_processes;

16.  ADPATCH has failed and it indicated there was an error with a worker. 

      What steps can be taken to investigate this problem?

        When ADPATCH is using workers, it (ADPATCH) keeps track of what jobs the

        workers are doing with a table in the database called FND_INSTALL_PROCESSES.

        Look at the bottom of the log file associated with the worker that failed

       (i.e. adwork01.log or adwork02.log  or adwork03.log)

        There is a utility ADCTRL that can be used to update FND_INSTALL_PROCESSES. 

        This utility will allow you to reset a flag in FND_INSTALL_PROCESSES.  This will allow

        ADPATCH to restart the failed step.  ADCTRL has a menu with 7 options.

       ( 8th option is Hidden and it is to skip a worker )

        Choose option one on the ADCTRL menu to review the worker status. Other

        workers will wait for the failed  worker.

        Choose option two on the menu to fix the worker.  Place in the worker number

        and hit return.

        Check the status of worker again using option one.  The status should have

        changed to "fixed restart."

        Try restarting the patch again.  (This assumes you have fixed the problem that

        caused the worker to fail, increased rollback segments, max extents, etc...)

17.  ADPATCH continues to fail and you have successfully completed the sql

       statement from the command line. Is there any way to direct the patch

       to skip this sql and continue?

        Insert the following at the very top of the script that failed:

             exit ;

        Go into ADCTRL and set the worker that failed to fixed/restart (as  above).

        Then restart ADPATCH.

18.  Your patch is erroring with an Undefined Symbol.  What does this mean and 

       what can you do to resolve this issue?

        The error generally looks like this:

        Can't relink APVOID.ABI

        An undefined symbol means ADPATCH cannot find a reference to a symbol in

        the archive file (ex. $FND_TOP/lib/libfnd.a) which was archived from the .o file

        contained in the patch library directory.  The undefined  symbols that it cannot

        fine are listed following the error message. You can try the nm command searching

        the library for something that references that symbol:

        nm *.o |grep I afmall   (afmall in this case is the undefined symbol)

        The object files are binary.  Use the following command to put the libfnd.a file

        in a text version that you can read:

        nm libfnd.a > tmp

        Use VI to edit the .tmp file and search for the undefined symbol.  When  you view

        the file you will notice that each .o is listed and there are a large number of objects.

        A symbol will only be listed once in each object file (.o). The .o file is listed at the

        top and the columns are listed below.

        The name column will contain the names of the symbols.  The Shndx column will

        contain the text UNDEF or it will contain a number.  The number is referencing

        what the symbol means. Use the find command in VI to locate the symbol where

        a number appears in there Shndx column next to the symbol that is erring.  You

        shouldn't find one that is what is causing the undefined symbol. You will need to

        find the object file that defines this symbol.  If this was working and you were

        applying a patch start  with the patch.  Also path problems can cause this type of  error.

19.  You want to know if you can reapply a patch.  Will this cause problems?

        You can almost always reapply patches without a problem.  If a patch can not be

        reapplied, there will be a warning in the README.  However, problems may exist

        if you are trying to reapply a patch where you have already applied a later version

        of the same patch.  One example is applying FND Patchset of lower version,then FND

       Patchset of higher verion and then trying to reapply FND of Patchset lower version.

20.  How to apply a patch when adpatch is already running

      Applying an 11i patch when adpatch is already running