Oracle RAC: Node evictions

Description

Oracle RAC Cluster integrity must be maintained at all time. The eviction of a particular node can be triggered by the RAC cluster master daemon following the detection of certain failure conditions.

Sample Error

Java Application Servers connecting directly to the affected Oracle RAC node can get the following errors following a Node Eviction by Oracle RAC:

Ex: Oracle RAC installed on Linux OS and accessed through Oracle WebLogic 11g

<Warning> <JDBC> <BEA-001129> <Received exception while creating connection for pool 

"JDBCPool":

• ORA-27101: shared memory realm does not exist
• ORA-01034: ORACLE not available
• Linux-x86_64 Error: 2: No such file or directory

Possible causes

• Hardware failure (disk, defective RAM...) of the server hosting the node.
• Various scenarios triggering heartbeat failures such as Oracle DB node hang, OS kernel locks/bugs triggering kernel "panics".
• A wrongly enabled USB network device can also lead to frequent and intermittent Oracle RCA node evictions. This problem can arise since when the IP is automatically started leading to  disrupted HAIP (Highly Available Internet Protocol). The solution normally involved disabling the USB device in order to prevent this problem upfront.

 References and Case Studies

• http://www.dbas-oracle.com/2013/06/Top-4-Reasons-Node-Reboot-Node-Eviction-in-Real-Application-Cluster-RAC-Environment.html
• http://searchitchannel.techtarget.com/feature/Oracle-RAC-Debugging-node-eviction-issues

BEA-001074 - A JDBC pool connection leak was detected

Description

Error message generated by the Oracle Weblogic server when detecting a JDBC Pool connection leak.

Sample Error

<Jul 2, 2013 11:52:13 AM EEST> <Warning> <JDBC>  <AppServer> <Finalizer> <<anonymous>> <> <BEA-001074> <A JDBC pool connection leak was detected. A connection leak occurs when a connection obtained from the pool was not closed explicitly by calling close() and then was disposed by the garbage collector and returned to the connection pool. The following stack trace at create shows where the leaked connection was created.

Possible causes

• Your application code is not closing properly some JDBC connections in a finally {} clause.
• A third party API that you are using is not properly closing some JDBC connections in a finally {} clause.

 References and Case Studies

• 
  

BlackBerry Chief Admits pressure points with U.S BB 10 sales

Thorsten Heins, the president and chief executive of BlackBerry acknowledged on Tuesday that his company is still facing pressure points with its U.S sales of its new BB 10 device.

The company is asking for more time and patience from the investors in order to move forward with their long term plan of increasing their sales and competitiveness in the SmartPhone market.

Please refer to the original New York Times article for more detail.

We wish them the very best in their restructuring plan.

IBM Java core file

Description

An IBM Java core file is a snapshot generated by the IBM JVM as a result of:

• Severe events such as a java.lang.OutOfMemoryError event.
• User signal (action) sent to the JVM such as kill -3 <Java PID>.

Sample file

javacore.20130522.091935.14024730.0008.txt

NULL           ------------------------------------------------------------------------

0SECTION       TITLE subcomponent dump routine

NULL           ===============================

1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" received

1TIDATETIME    Date:                 2013/05/22 at 09:25:36

1TIFILENAME    Javacore filename:    <App Folder>/javacore.20130522.091935.14024730.0008.txt

NULL           ------------------------------------------------------------------------

0SECTION       GPINFO subcomponent dump routine

NULL           ================================

2XHOSLEVEL     OS Level         : AIX 6.1

2XHCPUS        Processors -

3XHCPUARCH       Architecture   : ppc64

3XHNUMCPUS       How Many       : 12

Possible causes

……………………………………………………………………………………………

0SECTION       THREADS subcomponent dump routine

NULL           =================================

NULL           

1XMCURTHDINFO  Current Thread Details

NULL           ----------------------

……………………………………………………………………………………………

0SECTION       CLASSES subcomponent dump routine

NULL           =================================

1CLTEXTCLLOS        Classloader summaries

……………………………………………………………………………………………

Data content & purpose

• IBM JVM thread dump analysis (thread lock contention, deadlock, thread chains, execution snapshot etc.)
• Class loader analysis (# instances, leak patterns etc.).
• JVM runtime arguments review.
• Last GC activity history.

References and Case Studies

• https://www.ibm.com/developerworks/community/groups/service/html/communityview?communityUuid=2245aa39-fa5c-4475-b891-14c205f7333c
• http://www.ibm.com/developerworks/aix/library/au-javaonaix_core.html

mysql_connect(): Can't connect to local MySQL server

Description

Problem from a PHP application connecting to a local MySQL server database.

Sample Error

PHP Warning:  mysql_connect(): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (11)

Possible causes

• Your local MySQL database is down or not responding via its local server socket IP address and port.
• Verify the physical utilization of your physical server hosting MySQL and determine if you are dealing with high CPU and/or other resource utilization. CPU saturation can also cause connection failures from client such as Apache, PHP when trying to connect to the MySQL database server socket port.

 References and Case Studies

• http://stackoverflow.com/questions/1005485/warning-mysql-connect-cant-connect-to-local-mysql-server
  
  

java.lang.NoClassDefFoundError

Description and Symptoms

Exception in thread "main" java.lang.NoClassDefFoundError is one of the most complex problems that you can face when developing Java applications. This error is thrown by the JVM at runtime when it is unable to locate a particular Java class from the current Thread context class loader.

Sample Error

java.lang.NoClassDefFoundError: Could not initialize class org.ph.javaee.training2.ClassA

       at org.ph.javaee.training2.NoClassDefFoundErrorSimulator.main(NoClassDefFoundErrorSimulator.java:32)

java.lang.ExceptionInInitializerError

       at org.ph.javaee.training2.NoClassDefFoundErrorSimulator.main(NoClassDefFoundErrorSimulator.java:23)

Caused by: java.lang.IllegalStateException: ClassA.initStaticVariable(): Internal Error!

       at org.ph.javaee.training2.ClassA.initStaticVariable(ClassA.java:37)

       at org.ph.javaee.training2.ClassA.<clinit>(ClassA.java:16)

       ... 1 more

Possible causes

• Missing Java class and/or JAR file(s) from your Java runtime classpath.
• Failure of a static variable or initializer block at class loading time.
• Packaging problem of your application or third part API causing a mixed-up of class loading between the parent and child class loaders.
• Wrong Class loader delegation model based on your application and packaging requirements.

Troubleshooting and solutions

• First identify the Java class the JVM is complaining about.
• Check for a simple root cause first: wrong Java runtime classpath.
• If the problem persists, verify if the affected Java class contains static initializers e.g. static variable and/or static block initializer. Look for any failure at class loading time.
• Once the above items are ruled out, perform an assessment of your application runtime class loader delegation model and look for any packing problem.

References and Case Studies

• http://docs.oracle.com/javase/7/docs/api/java/lang/NoClassDefFoundError.html

java.lang.NullPointerException

Description and Symptoms

 java.lang.NullPointerException is a very common Java exception that you can face when developing pure Java, Java EE applications or executing Java programs & games. Exception in thread "main" java.lang.NullPointerException has been around since early JDK days e.g. JDK 1.0.

Sample Error

java.lang.NullPointerException

       at org.ph.javaee.training1.CallerClassA.doSomething(CallerClassA.java:27)

       at org.ph.javaee.training1.NoClassDefFoundErrorSimulator.main(NoClassDefFoundErrorSimulator.java:28)

Possible Causes

• Java programming problem.
• Java software (applications, games...) bug.

Troubleshooting and solutions

• Stack trace and root cause analysis of the affected code. Extra logging and debugging is also recommended.
• Java code review and bug identification. Addition of proper NULL validation to prevent the problem at the first place.
• Patch installation of the affected Java software.

References and Case Studies

• http://docs.oracle.com/javase/7/docs/api/java/lang/NullPointerException.html