Share the knowledge

Monday, March 30, 2015

High Disk Space is being consumed by C:\Program Files\Symantec\Symantec Endpoint Protection Manager\data\outbox\Importpackage

. Delete all files from %installlocation%\Symantec\Symantec Endpoint Protection Manager\data\outbox\ImportPackage folder. (without stopping any services)
2. Delete everything older than today's date in %installlocation%\Symantec\Symantec Endpoint Protection Manager\Inetpub\content (also without stopping any services)
3. In the Symantec Admin Console go to Admin > Servers > localhost. Right-click localhost and truncate the transaction logs.

Symantec has released new version of Symantec Endpoint Protection. English versions of Symantec Endpoint Protection 12.1.5337.5000 (RU5) is now available.
It has new content storage optimization feature:
As part of the upgrade to SEPM 12.1 RU5, the SEPM converts all of the content from full definitions to delta definitions. This process is resource intensive and may take an extended period of time. After this process is completed, the SEPM will use significantly less disk space.
In a typical enterprise setup where 30 content revisions stored, the SEPM upgrade process must reduce 55GB of full content to under 2GB of delta content. This process requires significant resources to complete and is impacted by the performance of any available CPUs, CPU cores (physical/logical/hyperthreading), memory, and disks (I/O). On a server that performs multiple roles, stores larger numbers of content, or is otherwise resource constrained, this process may take a longer duration to complete.
Refer this article to find more info: The LiveUpdate content optimization and content storage space optimization steps take a long time to complete when upgrading to Symantec Endpoint Protection Manager 12.1 RU5
http://www.symantec.com/docs/TECH224055

Thursday, March 12, 2015

vCenter Server Appliance: Troubleshooting full database partition

http://www.vcloudnine.de/vcenter-server-appliance-troubleshooting-full-database-partition/

A customer of mine had within 6 months twice a full database partition on a VMware vCenter Server Appliance. After the first outage, the customer increased the size of the partition which is mounted to /storage/db. Some months later, some days ago, the vCSA became unresponsive again. Again because of a filled up database partition. The customer increased the size of the database partition again (~ 200 GB!!) and today I had time to take a look at this nasty vCSA.

The situation

Within 2 days, the storage usage of the databse increased from 75% to 77%. First, I checked the size of the database:

vcsa:/opt/vmware/vpostgres/current/bin # /opt/vmware/vpostgres/current/bin/psql -h localhost -U vc VCDB
psql.bin (9.0.17)
Type "help" for help.

VCDB=> SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database;
  datname  |  size
-----------+---------
 template1 | 5353 kB
 template0 | 5345 kB
 postgres  | 5449 kB
 VCDB      | 2007 MB
(4 rows)

VCDB=>

vcsa:/opt/vmware/vpostgres/current/bin # /opt/vmware/vpostgres/current/bin/psql -h localhost -U vc VCDB

psql.bin (9.0.17)

Type "help" for help.

VCDB=> SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database;

datname | size

-----------+---------

template1 | 5353 kB

template0 | 5345 kB

postgres | 5449 kB

VCDB | 2007 MB

(4 rows)

VCDB=>

As you can see, the database had only 2 GB. The pg_log directory was more interesting:

vcsa:/storage/db/vpostgres # du -shc /storage/db/vpostgres/*
4.0K    /storage/db/vpostgres/PG_VERSION
2.0G    /storage/db/vpostgres/base
704K    /storage/db/vpostgres/global
47M     /storage/db/vpostgres/pg_clog
4.0K    /storage/db/vpostgres/pg_hba.conf
4.0K    /storage/db/vpostgres/pg_ident.conf
<strong>141G    /storage/db/vpostgres/pg_log</strong>
252K    /storage/db/vpostgres/pg_multixact
12K     /storage/db/vpostgres/pg_notify
324K    /storage/db/vpostgres/pg_stat_tmp
20K     /storage/db/vpostgres/pg_subtrans
4.0K    /storage/db/vpostgres/pg_tblspc
4.0K    /storage/db/vpostgres/pg_twophase
81M     /storage/db/vpostgres/pg_xlog
20K     /storage/db/vpostgres/postgresql.conf
4.0K    /storage/db/vpostgres/postmaster.opts
4.0K    /storage/db/vpostgres/postmaster.pid
0       /storage/db/vpostgres/serverlog
143G    total

vcsa:/storage/db/vpostgres # du -shc /storage/db/vpostgres/*

4.0K /storage/db/vpostgres/PG_VERSION

2.0G /storage/db/vpostgres/base

704K /storage/db/vpostgres/global

47M /storage/db/vpostgres/pg_clog

4.0K /storage/db/vpostgres/pg_hba.conf

4.0K /storage/db/vpostgres/pg_ident.conf

<strong>141G /storage/db/vpostgres/pg_log</strong>

252K /storage/db/vpostgres/pg_multixact

12K /storage/db/vpostgres/pg_notify

324K /storage/db/vpostgres/pg_stat_tmp

20K /storage/db/vpostgres/pg_subtrans

4.0K /storage/db/vpostgres/pg_tblspc

4.0K /storage/db/vpostgres/pg_twophase

81M /storage/db/vpostgres/pg_xlog

20K /storage/db/vpostgres/postgresql.conf

4.0K /storage/db/vpostgres/postmaster.opts

4.0K /storage/db/vpostgres/postmaster.pid

0 /storage/db/vpostgres/serverlog

143G total

The directory was full with log files. The log files containted only one message:

vcsa:/storage/db/vpostgres/pg_log # more postgresql-2015-03-04_090525.log
 123462 tm:2015-03-04 09:05:25.488 UTC db:VCDB pid:1527 WARNING:  there is already a transaction in progress

1 2	vcsa:/storage/db/vpostgres/pg_log # more postgresql-2015-03-04_090525.log 123462 tm:2015-03-04 09:05:25.488 UTC db:VCDB pid:1527 WARNING: there is already a transaction in progress

The solution

This led me to VMware KB2092127 (After upgrading to vCenter Server Appliance 5.5 Update 2, pg_log file reports this error: WARNING: there is already a transaction in progress). And yes, this appliance was upgraded to U2 with high probability. The solution is described in KB2092127, and is really easy to implement. Please note that this is only a workaround. There’s currently no solution, as mentioned in the article.

How to connect/interact with VCVA DB (DB2 and vPostgres)

If you need to connect/interact with the VC appliance database, for example to remove the locks of DB2 or performing an script, you can do the following after being logged in as root via SSH on the appliance:

- On VCVA 5.0 GA with DB2:

1. Turn into the db2inst user:

vcenter:/ # su db2inst1

2. Start the db2 client:

db2inst1@vcenter:/> db2

You'll see a prompt like this:

db2 =>

3. connect to the VCDB database:

db2 => connect to VCDB

(the command is like this, very literal)

4. Change to VC schema:

db2 => set schema vc

5. Perform any command you need. For example, to remove the VPX_SESSIONLOCK lines, you can do like this:

db2 => delete from VPX_SESSIONLOCK
DB20000I The SQL command completed successfully.

You can type "quit" anytime you want to exit from the db2 client, and "exit" when you want to go back to root userspace.

- On VCVA with vPostgres:

1. Connect to the database using psql:

vcenter:/ # /opt/vmware/vpostgres/1.0/bin/psql -U vc -d VCDB

You'll see a prompt like this:

psql (9.0.4)
Type "help" for help.

VCDB=>

2. Perform any command you need (selects, inserts, etc). For example, to list all tables:

VCDB=> \dt

There are a lot of new tables in 5.1, (mainly the vpx_hist_stat* ones).

To quit, just type "\q"

Changing the default VMware vCenter Server Appliance database password (2056968)

Changing the default VMware vCenter Server Appliance database password

Details

You can change the default password for the VMware vCenter Server Appliance database when you want or if the password is compromised.

Solution

To change the default:

Change the embedded database password:
1. Connect to the vCenter Server Appliance using SSH.
2. Open the embedded_db.cfg file for editing with this command:
  
  vi /etc/vmware-vpx/embedded_db.cfg
3. In the file, locate EMB_DB_PASSWORD and change the password between the single quotation marks.
Change the password for the vc and postgres database users:
1. Connect to the vPostgres database for SQL execution by running this command:
  
  /opt/vmware/vpostgres/current/bin/psql -d VCDB U postgres
2. Run these SQL statements to change the passwords for the vc and postgres users:
  
  alter user postgres with password 'new-password';
  alter user vc with password 'new-password';
3. Exit the database with this command:
  
  \q
4. Open the .pgpass file for editing by running this command:
  
  vi /root/.pgpass
5. Modify the .pgpass file with the new password as follows:
  
  localhost:5432:VCDB:postgres:new-password
  localhost:5432:postgres:postgres:new-password
  localhost:5432:VCDB:vc:new-password
Change the postgres database password:
1. To change the password for the vPostgres database by running this command:
  
  passwd postgres
2. Type the new password.
3. Retype the new password.
To update the encrypted password in the vpxd.cfg file, run this command:

/usr/sbin/vpxd -p
Enter the password when prompted.
Run this command to restart the vpxd service:

/etc/init.d/vmware-vpxd restart

vCenter Appliance – Call “EventHistoryCollector.SetLatestPageSize” for object “SessionID” on vCenter Server failed.

When using the vSphere Client to connect to the VMware vCenter Server Appliance was appearing every now and again.

Call “EventHistoryCollector.SetLatestPageSize” for object “SessionID” on vCenter Server “ServerName” failed. (unfortunately didnt take a screenshot, so here’s one I found and modified).

This issue is pretty common, and is to do with the amount of events in the database not being purged, and is covered by VMware in this KB article for windows environments.
However not so commonly covered for the vCenter Appliance which uses a progress database.

After a bit of digging around, I found the following crude solution on the VMware communities board.

So open up a console to your VCSA, login in. Run the following commands

/opt/vmware/vpostgres/1.0/bin/psql -d VCDB vc 
TRUNCATE TABLE vpx_event CASCADE;

then to exit “/q”

Here are the steps:

First of all - stop VPXD

service vmware-vxpd stop

connect to DB:

/opt/vmware/vpostgres/1.0/bin/psql -d VCDB vc

You will be prompted for the "vc" password which is not the same as the

root password.

Password is in "/etc/vmware-vpx/embedded_db.cfg" file

issue this commands:
- TRUNCATE TABLE vpx_event CASCADE;
- TRUNCATE TABLE vpx_event_arg CASCADE;
- TRUNCATE TABLE vpx_task CASCADE;
quit DB command line

issue the command "/q "

start the VPXD

service vmware-vxpd start or restart vCSA appliance

Check the size of VCBD. Now the size is only 165MB

VCDB=> SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database;
datname | size

-----------+---------
template1 | 5289 kB
template0 | 5281 kB
postgres | 5385 kB
VCDB | 165 MB
(4 rows)

References:
https://communities.vmware.com/thread/80738

http://www.educationalcentre.co.uk/vmware-5-1-vcenter-appliance-call-eventhistorycollector-setlatestpagesize-for-object-sessionid-on-vcenter-server-failed/#more-418

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2054085

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2056968

VMware vCenter Server Appliance Error: VPXD must be stopped to perform this operation.

http://www.virtualizationteam.com/management-automation/vmware-vcenter-server-appliance-error-vpxd-must-be-stopped-to-perform-this-operation.html

Error: VPXD must be stopped to perform this operation.
This error has came up while trying to change authentication to active directory authentication and kinda seen the same error when trying to change the database to an external database. For some reason the Server service stop button is shadowed as well which mean I could not stop it by the GUI interface. Although my lab setup is not ideal and it might just due to the way I set it up, though I will still document how I resolved this where the same mechanism can be used to stop and restart any other service used by the vCenter Server Appliance. Below are the steps I have followed:
1- SSH to your VMware vCenter Server Appliance using the root account.
2- Execute the following command to see the status of all the service running in the vCenter Appliance:   chkconfig
The output of all services will look something like below:
localhost:~ # chkconfig
after.local              off
apache2                  off
arpd                       off
atftpd                   off
auditd                   on
autoyast                off
chargen                    off
chargen-udp              off
cron                       on
daytime                    off
daytime-udp              off
dbus                     on
dcerpcd                  on
dhcp6r                   off
dhcp6s                   off
dhcpd                    off
discard                  off
discard-udp              off
earlysyslog              on
echo                     off
echo-udp                 off
eventlogd                on
fbset                    on
gpm                      off
haldaemon                on
haveged                  on
irq_balancer             on
kbd                      on
ldap                     on
lsassd                   off
lwiod                    on
mdadmd                   off
multipathd               off
netlogond                on
netstat                  off
network                  on
network-remotefs         on
nfs                      on
ntp                      off
pcscd                  off
powerd               off
random               on
raw                      off
rpasswdd          off
rpcbind                on
rpmconfigcheck           off
sendmail                       on
servers                           off
services                         off
setserial                        off
skeleton.compat          off
splash                            on
splash_early                on
sshd                               235
stunnel                         off
syslog                           on
syslog-collector         off
systat                          off
time                            off
time-udp                   off
uuidd                         off
vami-lighttp             235
vami-sfcb               235
vaos                          235
vmware-inventoryservice on
vmware-logbrowser        off
vmware-netdumper         off
vmware-rbd-watchdog      off
vmware-tools             on
vmware-vpostgres         on
vmware-vpxd              on
vsphere-client           on
xinetd                   off
ypbind                   off
3- Stop the required service in my case was vmware-vpxd using the following command: chkconfig service-name off (ex: chkconfig vmware-vpxd off)
4- carry out your changes
5- Start the service again using the following command: chkconfig service-name on (ex: chkconfig vmware-vpxd on)

Backing up and restoring the vCenter Server Appliance vPostgres database (2034505)

Purpose

This article provides steps to back up and restore the vCenter Server Appliance's (VCSA) vPostgres database.

Note: This article is only supported for backup and restore of the vPostgres database to the same vCenter Server Appliance. Use of image-based backup and restore is the only solution supported for performing a full, secondary appliance restore.

Resolution

Before you proceed, ensure that you have these installed:

SSH client for connecting to the vCenter Server Appliance.
WinSCP (or any SCP client) for retrieving and replacing the vPostgres database recovery file.

Backing up the embedded vPostgres database

To back up the embedded vPostgres database:

Connect to the vCenter Server Appliance via SSH. For more information, see Enable or Disable SSH Administrator Login on the VMware vCenter Server Appliance section in the vCenter Server and Host Management Guide.
When prompted, log in as the root user. The default password is vmware.
Stop the VMware vCenter Server service by running this command:

service vmware-vpxd stop
On the vCenter Server Appliance virtual machine, navigate to the vPostgres utility directory using this command:

cd /opt/vmware/vpostgres/1.0/bin
To display the vPostgres database configuration file, run this command:

cat /etc/vmware-vpx/embedded_db.cfg
To back up the vCenter Server database, run this command:

./pg_dump EMB_DB_INSTANCE -U EMB_DB_USER -Fp -c > VCDBBackupFile

Fill in the EMD_DB_INSTANCE and EMB_DB_USER from the embedded_db.cfg configuration information listed in Step 5. Fill in the VCDBBackupFile with the location and file name to generated, for example:

./pg_dump VCDB -U vc -Fp -c > /tmp/VCDBackUp

Caution: The /tmp/ directory is reset after rebooting the vCenter Server Appliance. VMware recommends that if this location is used, backup should be moved to a persistent location.

Note: If prompted, enter the EMB_DB_PASSWORD password.
Using WinSCP, connect to the vCenter Server Appliance and download the VCDBackUp file from /tmp/
Start the VMware VirtualCenter Server service by running this command:

service vmware-vpxd start

Restoring from the backend vPostgres database file

To restore from the back up vPostgres database file:

Note: Ensure that you take a snapshot of the vCenter Server Appliance virtual machine before proceeding. This allows you to restore the database from the snapshot if this restore procedure fails.

Connect to the vCenter Server Appliance via SSH. For more information, see Enable or Disable SSH Administrator Login on the VMware vCenter Server Appliance section in the vCenter Server and Host Management Guide.
Using WinSCP, connect to the vCenter Server Appliance and upload the backup copy of the VCDBackUp file into the /tmp/ directory
To display the new vPostgres database configuration file, run this command:

cat /etc/vmware-vpx/embedded_db.cfg
Navigate to the vPostgres utility directory by running this command:

cd /opt/vmware/vpostgres/1.0/bin
Stop the VMware vCenter Server service by running this command:

service vmware-vpxd stop
To restore the vCenter Server vPostgres database from backup, run this command:

PGPASSWORD='EMB_DB_PASSWORD' ./psql -d EMB_DB_INSTANCE -Upostgres -f VCDBBackupFile

Fill in the EMD_DB_INSTANCE and EMB_DB_PASSWORD from the embedded_db.cfg configuration information listed in Step 3. Fill in the VCDBBackupFile with the location and file name to be used, for example:

PGPASSWORD='g<T4EuybGsA=kG$G' ./psql -d VCDB -Upostgres -f /tmp/VCDBackUp

Note: Use single-quotes (') around the password as shown in the embedded_db.cfg configuration file.
To restart the VMware VirtualCenter Server service for the database restore to take effect, run this command:

service vmware-vpxd start