Hi Friends, This is Madhu, working as a Sr.Oracle apps dba consultant.
Webogic Admin Challenges and Issues
Madhusappsdba
What is best way to problem handling?
As a Middleware Admin there are many troubles in daily jobs, whenever a problem occurs, You must understand the problem, identify that is it a known or unknown. If it is known issue there could be workaround for that it can be done with knowledge management. As part of best practices you must maintain a issues and their resolutions as Knowledge Management Database KMDB.
Problem can be understood with the following simple Questions that make you clarity about which area, what could be the reason for the issue.
What do we know?
What do we need to know?
What should we do?
Remember that every problem there must be solution or possible alternative solutions. You need to have enough knowledge to resolve the issue with wisdom. Wisdom gives you the right path to reach right decision to solve the issue.
Problem solving strategy: Problem || Analysis Data -> Information -> Knowledge -> Wisdom || Solution
Problem vs Issue
Problem that happen accidentally due to any reasons, such as network not in sync, Database server might overloaded or anything. An issue can be transformed into problem when it is not identified in time and not resolved. A problem that need to fix immediately. The impact of the problem can be Life threatening. To make more simple, fever is an issue it can be resolved by pills, Heart Attack is a major Life threatening it is not solved immediately there could be huge loss.
To explore on middleware if you encounter more troubles more you learn. At the same time remember that your Client always wishes that business will run uninterruptedly. You give the 99.9999%
Bottom line, When the problem is not in Middleware environment then don't play the name blame game. Try to give your best for support the solution providing team.
Love issues, which make you to learn more about the subject. Enjoy with issues, troubles, you will be enlighted with your problem skills so.......... dear smart WLA rock with them!!!
1. Port Conflict Issue:
While configuring a new WebLogic instance and starting it, that might be get an issue like : "Port already in use". There could be many reasons for this one. 1. on the same machine multiple standalone instances might be running one of the instance already used that port which you have given for new configuration. 2. apache might be running with the same port. 3. middleware might be running on the same machine with same port On Solaris Operating environment we have 2 options: 1. using pfiles command netstat –na|grep <port> --> identify port in use
pfiles <pid>|grep -i sockname |grep port --> look for every java process is initialized by startWebLogic.sh or startManagedWebLogic.sh
2. Another way costly one (Third party package) to find the process that is using particular port is : lsof -i tcp:<Port_number> This Linux command lsof works for that user to where this command is executed by the user. 3. Best way is perl script using a method it will check only standard ports which are used by the system.
When I copy pasted few of the JMS Queue related configurations from existing WLS config.xml to newly created WLS Domain's config.xml. I got stuck the control at following line in the WebLogic Server logs. <Jul 28, 2008 12:08:26 PM EDT> <Notice> <Management> <BEA-140005> <Loading domain configuration from configuration repository at /path/ Sol: Verify all the lines in the config.xml file there might be missing any xml tags missing. I have missed out one line need to correct it now it is working fine.
3. Multicast Failure
while running the managed server on a remote server getting the following exception:
<Jul 31, 2008 4:16:16 AM EDT> <Error> <Cluster> <BEA-000109> <An error occurred while sending multicast message: java.net.SocketException: Socket is closed
java.net.SocketException: Socket is closed
at java.net.DatagramSocket.send(DatagramSocket.java:577)
Sol: 1. check for the ping happening between the cluster involved machines. 2. Verify the Multicast IP with the MulticastTest with multiple ips and with ports then use which is communicating the message between two participants.
AppMachine 1:
$ java utils.MulticastTest -n app1 -a 237.0.0.5 -p 9999
WebMachine 2:
$ java utils.MulticastTest -n app2 -a 237.0.0.5 -p 9999
The messages must exchange between them... otherwise check with Network team. 3. Multcast heart beat can be visible by the following command line:
While accessing the application url after Web App deployment. <Aug 19, 2008 10:58:07 AM EDT> <Error> <HTTP> <BEA-101017><[ServletContext(id=29754737,name=webapp1,context-path=/contextpath)] Root cause of ServletException. java.lang.StackOverflowError Sol: Check the classpath it is the communication gap between Servlet and an EJB 5. Editing the WLS password New password is given in boot.properties when server is in offline.
The Admin server started but while starting the managed instances got the following error: <Aug 22, 2008 8:21:54 AM EDT> <Critical> <Security> <BEA-090402> <Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.> ******************************************************** The WebLogic Server did not start up properly. Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted. ************************************************************ Sol: Resolved by editing in boot.properties, startManagedWebLogic.sh files for the new password in 'CLEARTEXT' pattern and rename ldap folder in the Managedserver instance folder present in the domain folder. While starting up the server instance boot.properties will be encrypted and re-written back. after this ldap folder will be generated in the domain directory for the Managed instance where new file will be created with instance name with extention as .ldif Apache Plug-in configuration While configuring the Apache using the instructions given in the following link :
6. Deployment Error
Module Name: apollo, Error: [HTTP:101179][HTTP] Error occurred while parsing descriptor in Web application "/domainpath/application.war" [Path="/source_code_path", URI="applicaiton.war" org.xml.sax.SAXParseException: Element type "local-path" must be followed by either attribute specifications, ">" or "/>". Sol: Check the Deployment Descriptors (here weblogic.xml) edit as per the given Line.
7. JNDI Issue
While deploying application which is having multiple JMS Queues getting as: data couldn't be saved. The exception is : Error while doSenderQueueLookUp for :QUEUEJNDIUnable to resolve 'QUEUEJNDI' Resolved Sol: Please verify the Persistance Store is connected to Connection Pool.
8. New managed server Configuration issue
Here I am with set of errors when newly configuring WebLogic 9.2 MP3 managed server and adding to a webcluster. The managed server when started first time shown the following : 1. BEA-000362 2. BEA-000386 On Solaris box: <Mar 30, 2009 2:56:49 PM EDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: There are 1 nested errors: java.net.UnknownHostException: mywebhostname.com: myhostname.com at java.net.InetAddress.getAllByName0(InetAddress.java:1128) at java.net.InetAddress.getAllByName0(InetAddress.java:1098) at java.net.InetAddress.getAllByName(InetAddress.java:1061)On On Linux box: <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.ldap.EmbeddedLDAPException: Unable to open initial replica url: http://myadminhost.com:7001/bea_wls_management_internal2/wl_management weblogic.ldap.EmbeddedLDAPException: Unable to open initial replica url: http://myadminhost.com:7001/bea_wls_management_internal2/wl_management at weblogic.ldap.EmbeddedLDAP.getInitialReplicaFromAdminServer(EmbeddedLDAP.java:1319) at weblogic.ldap.EmbeddedLDAP.start(EmbeddedLDAP.java:221) at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209) at weblogic.work.ExecuteThread.run(ExecuteThread.java:181) java.io.FileNotFoundException: Response: '500: Internal Server Error' for url: 'http://myadminhost.com:7001/bea_wls_management_internal2/wl_management' at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:472) at weblogic.ldap.EmbeddedLDAP.getInitialReplicaFromAdminServer(EmbeddedLDAP.java:1296) at weblogic.ldap.EmbeddedLDAP.start(EmbeddedLDAP.java:221) at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209) Truncated. see log file for complete stacktrace
Solution/Work around:BEA-000362 First thing you need to ensure on your configuration, double check your config.xml file assigned IP or VIP or domain name is correct or not. On the other hand we need to investigate for the network communication happening or not with the hostnames given in the config.xml. ping from managed server box to admin server hosting dns. And also check the /etc/hosts updated with the new host dns. Some times DNS table entries not updated in WAN domains may cause this issue. BEA-000386 Check for the hostname of the managed server listen-address given one and hosting dns/ip is matching or not using the WebLogic console(this could be a time taking process). Unix aware guy always prefer grep on config.xml :)
Session Replication issues on Web-Tier cluster BEA-101310 Web server failed to perform batched update for replicated sesionsohh my god!!!
BEA-000117 Received a stale replication request Whenever you encounter this error code in your logs you need to check following: 1. Network connectivity between the cluster member hosting machines. 2. Check all the member servers are alive in that cluster.
BEA-090870
Security issue while starting Admin Server
Exception:
####<Oct 28, 2010 3:38:41 AM PDT> <Error> <Security> <Unknown> <AdminServer> <[STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <1288262321492> <BEA-090870> <The realm "myrealm" failed to be loaded: weblogic.security.service.SecurityServiceException: java.lang.ExceptionInInitializerError.
weblogic.security.service.SecurityServiceException: java.lang.ExceptionInInitializerError
at weblogic.security.service.CSSWLSDelegateImpl.initializeServiceEngine(CSSWLSDelegateImpl.java:342)
The workaround is suggested as 'Please login to the Database and then run the below SQL file to execute different queries mentioned in this file:
Suppose Database User Login is “weblogic” then login to Database with this credential and run the script under
$BEA_HOME\wlserver_10.3\server\lib\rdbms_security_store_oracle.sqlThen restart your AdminServer'.
BEA-000337 - Stuck Thread** You can use wise step using jps -v|grep [instance] First step you need to take thread dump (kill -3 for Unix, CTRL+Break for Windows) make sure one of thread dump analyzer is opend use any thread dump analyzer. Check the logs where the STUCK found, Check the timing when it came first time, how much time it exceeded configured time? If that time interval seconds exceeded for more than 10 mins and threadpool doesnot have idle threads then stop that WebLogic server instances. WebLogic 9.2 MP3 Managed Server cannot start !!??!! On Red Hat Linux 2.4 the WebLogic managed Server could not start. Tried many times kill the process and start no luck! memory is in comfortable range. CPU don't have load, Network connectivity is also verified, Good condition. WebLogic server log is not writing about why it is not able to start the server instance. Finally checked uptime for that box. It is more than a year ago started, thought that rebooting machine might helps.
BEA-000388 - JVM called WLS shutdown hook. The server will force shutdown now
Issue:
The WebLogic server got shutdown with the following error stack found in the server standard out log files:
####<Sep 7, 2012 3:14:20 PM IST> <Notice> <WebLogicServer> <Ncorp-PLM-08> <Ncorp-PLM-08-AgileServer> <Thread-1> <<WLS Kernel>> <> <> <1347011060768> <BEA-000388> <JVM called WLS shutdown hook. The server will force shutdown now>
####<Sep 7, 2012 3:14:20 PM IST> <Alert> <WebLogicServer> <Ncorp-PLM-08> <Ncorp-PLM-08-AgileServer> <Thread-1> <<WLS Kernel>> <> <> <1347011060768> <BEA-000396> <Server shutdown has been requested by <WLS Kernel>>
The following could be possible Cause and we have the workaround solution as:
I have the following suggestion for you for the below error:
<Sep 7, 2012 3:14:20 PM IST> <Notice> <WebLogicServer> <Ncorp-PLM-08> <Ncorp-PLM-08-AgileServer> <Thread-1> <<WLS Kernel>> <> <> <1347011060768> <BEA-000388> <JVM called WLS shutdown hook. The server will force shutdown now>
1) It seems like some component is sending the wrong signal to the JVM and this issue is occurring. JVM monitors and catches OS signals, like: CTRL +C event, Log off event, shutdown event.When JVM catches one of the stated above signals, it shutdowns all java processes.
Please try these possible solutions of this problem:
· Specify -Xrs parameter in the JAVA startup arguments and start the admin server
· -Xrs
· Note: -Xrs is a non-standard option developed by Sun Microsystems for their HotSpot JVM. BEA JRockit continues to support this option; however the BEA JRockit non-standard option -XnoHup provides the same functionality. -Xrs reduces usage of operating-system signals by the JVM. If the JVM is run as a service (for example, the servlet engine for a web server), it can receive CTRL_LOGOFF_EVENT but should not initiate shutdown since the operating system will not actually terminate the process. To avoid possible interference such as this, the -Xrs command-line option does not install a console control handler, implying that it does not watch for or process CTRL_C_EVENT, CTRL_CLOSE_EVENT, CTRL_LOGOFF_EVENT, or CTRL_SHUTDOWN_EVENT Operation.
2) If the issue recurs again even after trying the above option then can you please apply the following JAVA_OPTION:
· Depending on the JVM version, it may be possible to get a thread dump before the process exits
· HotSpot supports the command-line option -XX:+ShowMessageBoxOnError
· The corresponding JRockit option is -Djrockit.waitonerror
· While the JVM< goes down, it may prompt the user: "Do you want to debug the problem?"
· This pauses the JVM, thereby creating an opportunity to generate a thread dump (a stack trace of every thread in the JVM), attach a debugger, or perform some other debugging activity.
BEA-160192
<BEA-160192> <While upgrading weblogic.xml, encountered "max-in-memory-sessions" param "session-param". This param is unknown and will be removed>
This eror you can see in your weblogic logs, Because the deployment descriptor weblogic.xml is not updated with the latest DTD defined for weblogic 11g.
To resolve this do setWLSEnv.sh and run the following and it will convert it to compatable to WebLogic 11g. But as this is template the basic xml will need to be amended with the 11g DTD and then build script creating the correct file out of it -
java weblogic.DDConverter -d . <<ear/warfile or webapp/ear directory>>
No comments:
Post a Comment