This article will show you how to recover the PSM server application when it is unavailable due to PSM server root partition full usage.

The PSM client cannot connect or login to the PSM server.   This can be the result of the PSM Linux server's root partition being full, which results in the (PSM) EMS processes stopping.   As a result the PSM client cannot connect.

To identify these symptoms, open a terminal session via SSH to the PSM Linux server, and check the status of the PSM processes and the disk space usage. 

1. Enter the command "psm-status" to verify if the four EMS processes show "running".   If not, continue to step 2 and view the disk usage.  

   Example of normal PSM status:

   Note: Some lines have been edited or removed for purposes of this document.

   The Monit daemon 5.14 uptime: 6d 10h 36m
Process 'Ems-Server-Masters'        Running
Process 'Ems-Server-Workers'        Running
Process 'Ems-Server-Discovery'      Running
Process 'Ems-Dashboard'             Running

2. Using the command "df -H" the output shows the root partition "/ "usage is at or near 100% 

   [root@psmdemo]# df -H
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       224G   224G  0G  100% /
devtmpfs        4.1G     0  4.1G   0% /dev
tmpfs           4.1G     0  4.1G   0% /dev/shm
tmpfs           4.1G  313M  3.8G   8% /run
tmpfs           4.1G     0  4.1G   0% /sys/fs/cgroup
/dev/sda2       521M  183M  339M  36% /boot
tmpfs           820M     0  820M   0% /run/user/1007
tmpfs           820M     0  820M   0% /run/user/0

To free disk space, first find out which files are taking up the majority of the space of the root partition, then delete them or archive them.  

1. Open a terminal session via SSH to the PSM Linux server.

   Use the command du -sh /* to view the usage of each folder that belongs to "/".   Continue to perform the same command for each folder and subfolder until you get to the files that are taking up the majority of space.

   For example:

   du -sh /usr/*
du -sh /usr/local/ems9001/*

2. Delete any of the files you think are unnecessary and taking up space.  These might include leftover rpm files or old logs that are no longer needed.  

    Note: PSM will purge the old logs it generated older than 60 days automatically, while in PSM older release(R5.x) that function might stop for some reason and the logs files can be flood the root partition.  The command below will locate and manually free up the space by deleting the log files older than 90 days
     
   find /usr/local/ems9001/logs/ -mtime +90 -type f -exec rm -rf {} \;

3. View the filesystem again, and make sure the root partition usage is below 75%

   [root@psmdemo ftpadmin]# df -H
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       224G   156G  68G  70% /
devtmpfs        4.1G     0  4.1G   0% /dev
tmpfs           4.1G     0  4.1G   0% /dev/shm
tmpfs           4.1G  313M  3.8G   8% /run
tmpfs           4.1G     0  4.1G   0% /sys/fs/cgroup
/dev/sda2       521M  183M  339M  36% /boot
tmpfs           820M     0  820M   0% /run/user/1007
tmpfs           820M     0  820M   0% /run/user/0

4. PSM client should be able to connect once the EMS processes are running again. 

   Issue the command "psm-restart" at the PSM Linux command line to restart the PSM services. 
    
   [root@psmdemo]# psm-restart 
Every 2.0s: monit summary                    Mon Sep 20 14:16:24 2019
The Monit daemon 5.14 uptime: 6d 11h 11m

---TRUNCATED---
Process 'Ems-Server-Masters'        Running
Process 'Ems-Server-Workers'        Running
Process 'Ems-Server-Discovery'      Running
Process 'Ems-Dashboard'             Running

5. Wait for the EMS processes to show Running.  Then press CTRL+C to exit the monitor screen.

   The PSM Client should now be able to connect.