VMware vCenter Server Appliance 5.0 vpxd crashes few seconds after starting

VMware vCenter Server Appliance 5.0 vpxd crashes few seconds after starting(2030601)

Details

• When trying to start the VMware vCenter Server Appliance 5- Build 472350, the vpxd fails after a few seconds.
• In the /var/log/vmware/vpx/vpxd.log file, you see entries similar to:
  
  2012-06-28T12:07:32.583+02:00 [7FFFF3B09700 info 'Default'] [Init:842] Calling: VpxdCertificate_Load(gDB, CERTIFICATE_VMDBPATH )
  2012-06-28T12:07:32.583+02:00 [7FFFF3B09700 info 'Default'] [Init:843] Calling: VpxdVdb::Init(VpxdVdb::GetVcVdbInstId(), false, false, NULL)
  2012-06-28T12:07:32.583+02:00 [7FFFF3B09700 info 'Default'] Registry Item DB 5 value is ''
  2012-06-28T12:07:33.370+02:00 [7FFFF3B09700 error 'Default'] [Vdb::LockRepositoryHelper] SQLExecDirect failed: 57011:-438:ODBC error: (57011) - [IBM][CLI Driver][DB2/LINUXX8664] SQL0438N Application raised error or warning with diagnostic text: "Error deleting from VPX_SESSIONL". SQLSTATE=57011
  -->
  2012-06-28T12:07:33.370+02:00 [7FFFF3B09700 error 'Default'] Unable to get exclusive access to vCenter repository. Please check if another vCenter instance is running against the same database schema.
  2012-06-28T12:07:33.374+02:00 [7FFFF3B09700 error 'Default'] [Init] Init failed: VpxdVdb::Init(VpxdVdb::GetVcVdbInstId(), false, false, NULL)
  2012-06-28T12:07:33.374+02:00 [7FFFF3B09700 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...
  2012-06-28T12:07:33.374+02:00 [7FFFF3B09700 info 'Default'] Wrote uptime information
  2012-06-28T12:07:33.374+02:00 [7FFFF3B09700 error 'Default'] Alert:false@ /build/mts/release/bora-472350/bora/vpx/vpxd/util/vpxdVdb.cpp:408
  --> Backtrace:
  --> [00] rip 00007fffeed9ed14
  --> [01] rip 00007fffeecf59e2
  --> [02] rip 00007fffeec9b38b
  --> [03] rip 00007ffff58c613d
  --> [04] rip 00007ffff4ea5a40
  --> [05] rip 00007ffff4eb4e21
  --> [06] rip 00007ffff4ea2953
  --> [07] rip 00007fffed03abc6
  --> [08] rip 00007ffff4ea1c19
  -->
  2012-06-28T12:07:33.374+02:00 [7FFFF3B09700 info 'Default'] Registry Item DB 5 value is ''
  2012-06-28T12:07:34.132+02:00 [7FFFF3B09700 error 'Default'] [Vdb::LockRepositoryHelper] SQLExecDirect failed: 57011:-438:ODBC error: (57011) - [IBM][CLI Driver][DB2/LINUXX8664] SQL0438N Application raised error or warning with diagnostic text: "Error deleting from VPX_SESSIONL". SQLSTATE=57011
  -->
  2012-06-28T12:07:34.133+02:00 [7FFFF3B09700 error 'Default'] Unable to get exclusive access to vCenter repository. Please check if another vCenter instance is running against the same database schema.
  2012-06-28T12:07:34.138+02:00 [7FFFF3B09700 error 'Default'] Unhandled exception <Error[VdbError] (-24) Vdb instance is not initialized or terminated>

Solution

Note: This solution is applicable only to vCenter Server Appliance 5.0 and is not applicable to vCenter Server Appliance 5.0 Update 1, which uses the PostgresSQL database.

This issue may be caused due to a lack of adequate space on the /storage partition, a full transaction log, or due to aVPX_SESSIONLOCK table.

To resolve this issue due to core dumps filling the partition:

1. Connect to vCenter Server Appliance console as the root user. The default password is vmware.
2. Change directory to /storage/core and run the ls command to check for core files
   
   Note: If there are core dumps, delete these dumps and try to start the service again. The problem should now be solved.

To increase the transaction log size:

1. Stop the vCenter Virtual Appliance service:
   
   service vmware-vpxd stop
2. Run the following commands to connect to the database:
   
   su db2inst1
   
   db2 connect to vcdb
3. Run this command to retrieve current database configuration:
   
   db2 get db cfg for vcdb | grep LOG
4. Run this command to update the database configuration:
   
   db2 update db CFG FOR VCDB USING logprimary 16 logsecond 112 logfilsiz 8192You see an output similar to:
   
   DB20000I The UPDATE DATABASE CONFIGURATION command completed successfully. SQL1363W One or more of the parameters submitted for immediate modification were not changed dynamically. For these configuration parameters, all applications must disconnect from this database before the changes become effective.
5. Run this command to verify whether the update command has executed successfully:
   
   db2 get db cfg for vcdb | grep LOGYou see an output similar to:
   
   Catalog cache size (4KB) (CATALOGCACHE_SZ) = 300
   Log buffer size (4KB) (LOGBUFSZ) = 256
   Log file size (4KB) (LOGFILSIZ) = 8192
   Number of primary log files (LOGPRIMARY) = 16
   Number of secondary log files (LOGSECOND) = 112
6. Type exit to leave the db2 session.
    
7. Start the VMware VirtualCenter Server service and check the status with these commands:
   
   service vmware-vpxd start
   service vmware-vpxd status

To delete the VPX_SESSIONLOCK:

Caution: Engage your Database Administrator for assistance if you are unsure of how to complete these actions.

1. Connect to the DB2 database using the DB Control Center. For more information on the DB Control Center, see The VirtualCenter Server service fails to start after the service was stopped or restarted when using DB2 database (1021581).
2. Select the VPX_SESSIONLOCK table and delete all the rows in the table.
3. Restart the VMware VirtualCenter Server service:
   
   service vmware-vpxd restart

To Monitor:

1. Check that the VMWARE-VPXD service is running
2. Check the storage partition's free space
3. Check the /Storage/core directory does not contain any dump files "vmware-vpxd.xxxx"

REF: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2030601