Category Archives: Apache

30Oct/15

Error: Failed to create temporary file

Issue
======

The apache error log shows

[Wed Oct 28 23:49:30 2015] [error] [client xx.xx.xxx.xxx] ModSecurity: Input filter: Failed to create temporary file: /root/tmp/20151028-234930-VjGXSkAWQiQAA@W7VAsAAAAD-request_body-nKiZdT [hostname "www.xxxx.com"] [uri "/wp-admin/admin.php"] [unique_id "VjGXSkAWQiQAA@W7VAsAAAAD"]

The issue is because the modsecurity has no access on /root/tmp directory.

So while accessing /root/tmp it will show the error “Failed to create temporary file”

Fix
===

The issue can be fixed by adding the following lines on the configuration file of modsecurity.

# vi /usr/local/apache/conf/modsec2.conf

Add these below lines in /usr/local/apache/conf/modsec2.conf

SecUploadDir /tmp
SecTmpDir /tmp
SecDataDir /tmp
SecRequestBodyAccess On

Now you need to restart apache to make the changes effective

 service httpd restart

This will fix the issue normally

13Dec/14

Detecting WordPress Outbound Bruteforce attack

We have heard a lot about the WordPress inbound Bruteforce attacks. On checking we can see the inbound attack IP’s  may have cpanel installed and  are actually generating from some other server which is infected.

Two day before,we  got a message from a client of ours saying data center has informed them that server is infected and is generating attacks on other servers. Initially I am not able to get any of the details regarding the attack as no rogue process is running neither scan using gave me any valid clue on this attack .

I was just checking the result of tcpdump to see the what all data is being transferred from the server.

user@host ~ # tcpdump -A -i eth0 -s 1500 port not 22

While checking the results I can see something is going on and many wp-login.php entry was going on.

Sample tcpdump Output (changed domain and hostnames)

v.G....pPOST /restaurants/wp-login.php HTTP/1.0^M
Host: domain.com^M
Content-Type: application/x-www-form-urlencoded^M
Content-Length: 30^M
^M
log=admin&pwd=minedoruksay2940
06:15:22.056294 IP host5.domain.com > host6.domain.com48202: Flags [P.], seq 2779525802:2779527849, ack 2761432155, win 3216, options [nop,nop,TS val 166530731 ecr 1994475337], length 2047

I tried to stop apache and mysql ,psa, and still some process were running as www-data user and the process was some thing like the below.

www-data 1258 10.8 1.5 18327 1268 ? Ssl Dec10 129:10 /usr/bin/host 

I took the lsof result of this command and I got the culprit (account ) responsible for this attack 🙂 Thanks to lsof command to give me correct location and scripts.

Relevant output from lsof command

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
host 20636 username cwd DIR 9,2 4096 60874901 /var/www/vhosts/domain.com/site1/wordpress/wp-content/plugins/subscribe2/extension/readygraph/assets/js
host 20636 username rtd DIR 9,2 4096 2 /
host 20636 username txt REG 9,2 120240 68160132 /usr/bin/host
host 20636 username DEL REG 9,2 60817452 /var/www/vhosts/domain.com/site1/wordpress/wp-content/plugins/subscribe2/extension/readygraph/assets/js/bruteforce.so
host 20636 username mem REG 9,2 22928 23855190 /lib/libnss_dns-2.11.3.so
host 20636 username mem REG 9,2 51728 23855282 /lib/libnss_files-2.11.3.so
host 20636 username mem REG 9,2 12582912 60827148 /var/www/vhosts/domain.com/site1/wordpress/wp-content/plugins/subscribe2/extension/readygraph/assets/js/.frsdfg
host 20636 username DEL REG 9,2 60817412 /var/www/vhosts/domain.com/site1/wordpress/wp-content/plugins/subscribe2/extension/readygraph/assets/js/libworker.so
cwd : /var/www/vhosts/domain.com/site1/wordpress/wp-content/plugins/subscribe2/extension/readygraph/assets/js

The above entry from lsof means attack is being generated from this folder and scripts are located in this location.

/var/www/vhosts/domain.com/site1/wordpress/wp-content/plugins/subscribe2/extension/readygraph/assets/js/bruteforce.so

/var/www/vhosts/domain.com/site1/wordpress/wp-content/plugins/subscribe2/extension/readygraph/assets/js/.frsdfg

/var/www/vhosts/domain.com/site1/wordpress/wp-content/plugins/subscribe2/extension/readygraph/assets/js/libworker.so

Above 3 files are the main hack files in which /bruteforce.so was not present in the server at that time . This script was removed soon after the attack in initiated.

To fix this , I have removed the entire “js” folder and then kill all these process. Also asked client to remove the plugin. It will be good if we can remove the host binary (/usr/bin/host) file. If it is there, they can again come back with the attack and can kill the server reputation within few hours.

16Aug/14

pnp4nagios Installation and Configuration

Installation
—————-

PNP is an addon for the Nagios Network Monitoring System. PNP provides easy to use, easy to configure RRDTools based performance charts feeded by the performance data output of the Nagios Plugins. To install this,

1. Download the latest version of pnp4nagios source from http://sourceforge.net/projects/pnp4nagios/

2. Untar it and cd into the extracted directory.

3. Run ./configure –with-nagios-user=nagios –with-nagios-group=nagios

a. Some lines run across the screen after running this. The paths shown should be checked. If the displayed values aren’t correct you can change them calling ./configure with appropriate options( ./configure –help ).

NB:: Make sure that we have rrdtool and perl-rrdtool are installed on our server.

4. Run make; make install clean; make install-webconf; make install-config; make install-init;

Attention: After copying the configuration file for the web server you have to restart the web server (/etc/init.d/httpd restart).

A detailed installation documentation can be found at http://docs.pnp4nagios.org/pnp-0.6/install

Configuration
——————

We can configure pnp4nagios in three different modes, they are

Default Mode
Bulk Mode
Bulk Mode with NPCD

Among these, Default mode is less complicated and it will take less time to configure. To configure pnp4nagios in Default mode,

1. Enable processing of performance data in nagios.cfg. Please note that this directive might already exist in the config file. Default is “0�?.

process_performance_data=1

2. Data processing has to be disabled in the definition of every host or service whose performance data should NOT be processed. By default it isn’t enabled for any definitions. Just make sure that the particular directive isn’t explicitly set to 1.

define service {

process_perf_data 0

}

3. Set enable_environment_macros to 1 in nagios.cfg

4. Additionally the command to process performance data is to be specified in nagios.cfg

service_perfdata_command=service_perfdata

5. Starting with Nagios 3.0 it may be useful to enable processing of performance data for hosts as well. Due to changed host check logic Nagios 3 now performs regularly scheduled host checks.

host_perfdata_command=host_perfdata

6. Define the following two commands from nconf

check command name – service_perfdata

check command line – /usr/bin/perl /usr/local/pnp4nagios/libexec/process_perfdata.pl
check command name – host_perfdata

check command line – /usr/bin/perl /usr/local/pnp4nagios/libexec/process_perfdata.pl -d HOSTPERFDATA

7. Save the changes and run generate nagios configuration.
Read more details at http://docs.pnp4nagios.org/pnp-0.4/config

06Aug/14

Multiple Versions of PHP in plesk

 

By default, in latest version of plesk we will have 5.4 version of php. So if you need additional php version, you can install seperatly and those additional  version will be available in fastcgi and cgi handlers.

Installtion Steps for php 5.2.17 version

# cd /usr/local/src
# wget http://museum.php.net/php5/php-5.2.17.tar.gz
# tar -zxvf php-5.2.17.tar.gz
# cd php-5.2.17
# ./configure --with-libdir=lib64 --cache-file=./config.cache --prefix=/usr/local/php-5.2.17 --with-config-file-path=/usr/local/php-5.2.17/etc --disable-debug --with-pic --disable-rpath --with-bz2 --with-curl --with-freetype-dir=/usr/local/php-5.2.17 --with-png-dir=/usr/local/php-5.2.17 --enable-gd-native-ttf --without-gdbm --with-gettext --with-gmp --with-iconv --with-jpeg-dir=/usr/local/php-5.2.17 --with-openssl --with-pspell --with-pcre-regex --with-zlib --enable-exif --enable-ftp --enable-sockets --enable-sysvsem --enable-sysvshm --enable-sysvmsg --enable-wddx --with-kerberos --with-unixODBC=/usr --enable-shmop --enable-calendar --with-libxml-dir=/usr/local/php-5.2.17 --enable-pcntl --with-imap --with-imap-ssl --enable-mbstring --enable-mbregex --with-gd --enable-bcmath --with-xmlrpc --with-ldap --with-ldap-sasl --with-mysql=/usr --with-mysqli --with-snmp --enable-soap --with-xsl --enable-xmlreader --enable-xmlwriter --enable-pdo --with-pdo-mysql --with-pdo-pgsql --with-pear=/usr/local/php-5.2.17/pear --with-mcrypt --without-pdo-sqlite --with-config-file-scan-dir=/usr/local/php-5.2.17/php.d --enable-fastcgi
# make 
# make install

There may be some error while compiling and you need to make necessary changes in the configure as per your need / your server.

# cp /usr/local/src/php-5.2.17/php.ini-recommended /usr/local/php-5.2.17/etc/php.ini

Make necessary changes in the php.ini like timezone setting, memory limit etc.

Inorder to register this php version in plesk, we need to use the below command.

# /usr/local/psa/bin/php_handler --add -displayname "PHP 5.2.17" -path /usr/local/php-5.2.17/bin/php-cgi -phpini /usr/local/php-5.2.17/etc/php.ini -type fastcgi -id "fastcgi-5.2.17"

You will get a message like ” The new PHP handler with the id “fastcgi-5.2.17″ was successfully registered”

Installtion Steps for php 5.3.28 version

# cd /usr/local/src
# wget http://in1.php.net/distributions/php-5.3.28.tar.gz
# tar -zxvf php-5.3.28.tar.gz
# cd php-5.3.28
# ./configure --with-libdir=lib64 --cache-file=./config.cache --prefix=/usr/local/php-5.3.28 --with-config-file-path=/usr/local/php-5.3.28/etc --disable-debug --with-pic --disable-rpath --with-bz2 --with-curl --with-freetype-dir=/usr/local/php-5.3.28 --with-png-dir=/usr/local/php-5.3.28 --enable-gd-native-ttf --without-gdbm --with-gettext --with-gmp --with-iconv --with-jpeg-dir=/usr/local/php-5.3.28 --with-openssl --with-pspell --with-pcre-regex --with-zlib --enable-exif --enable-ftp --enable-sockets --enable-sysvsem --enable-sysvshm --enable-sysvmsg --enable-wddx --with-kerberos --with-unixODBC=/usr --enable-shmop --enable-calendar --with-libxml-dir=/usr/local/php-5.3.28 --enable-pcntl --with-imap --with-imap-ssl --enable-mbstring --enable-mbregex --with-gd --enable-bcmath --with-xmlrpc --with-ldap --with-ldap-sasl --with-mysql=/usr --with-mysqli --with-snmp --enable-soap --with-xsl --enable-xmlreader --enable-xmlwriter --enable-pdo --with-pdo-mysql --with-pdo-pgsql --with-pear=/usr/local/php-5.3.28/pear --with-mcrypt --without-pdo-sqlite --with-config-file-scan-dir=/usr/local/php-5.3.28/php.d --without-sqlite3 --enable-intl
# make 
# make install

There may be some error while compiling and you need to make necessary changes in the configure as per your need / your server.

# cp -a /etc/php.ini /usr/local/php-5.3.28/etc/php.ini

Make necessary changes in the php.ini like timezone setting, memory limit etc.

Inorder to register this php version in plesk, we need to use the below command.

# /usr/local/psa/bin/php_handler --add -displayname "5.3.28" -path /usr/local/php-5.3.28/bin/php-cgi -phpini /usr/local/php-5.3.28/etc/php.ini -type fastcgi -id "fastcgi-5.3.28"

You will get a message like ” The new PHP handler with the id “fastcgi-5.3.28″ was successfully registered.”

After this try to login to plesk and see if these additional php version are available in fastcgi and cgi handlers.

If all good then you are set to use all these additional php version in plesk.

 

 

 

23Jul/14

Error : “Identifier removed: couldn’t grab the accept mutex” in apache error logs

Apache was getting stopped automatically with the below logs in the Apache Error log:

[Tue Jul 22 22:29:35 2014] [alert] Child 30758 returned a Fatal error… Apache is exiting!
[Tue Jul 22 22:29:35 2014] [emerg] (43)Identifier removed: couldn’t grab the accept mutex
[Tue Jul 22 22:29:35 2014] [emerg] (43)Identifier removed: couldn’t grab the accept mutex
[Tue Jul 22 22:29:35 2014] [emerg] (43)Identifier removed: couldn’t grab the accept mutex

These type of errors occur on systems low on memory or file handlers.The AcceptMutex directives sets the method that Apache uses to serialize multiple children accepting requests on network sockets.

One suggested resolution for the error message would be to try adding the following line to /usr/local/apache/conf/httpd.conf file:

AcceptMutex fcntl

If you add this above “<IfModule prefork.c>” in tht file, you could then try running these commands to distill the change and restart Apache:

Continue reading

17Jul/14

Product images not working and giving “Warning: Creating default object from empty value” Error

Last day there was an issue for a wordpress installation. None of the product images are not working and if we take the image directly in the browser, we can see the below errors

==================
Warning: Creating default object from empty value in /home/wp-user/public_html/wp-content/plugins/shopp/core/model/Asset.php on line 123

Warning: Cannot modify header information – headers already sent by (output started at /home/wp-user/public_html/wp-content/plugins/shopp/core/model/Asset.php:123) in /home/wp-user/public_html/wp-content/plugins/shopp/core/model/Asset.php on line 199

Warning: Cannot modify header information – headers already sent by (output started at /home/wp-user/public_html/wp-content/plugins/shopp/core/model/Asset.php:123) in /home/wp-user/public_html/wp-content/plugins/shopp/core/model/Asset.php on line 200

Continue reading

09Jul/14

Find domains targeted for wordpress brute-force attack in Plesk

The following script will give an overview of all the domains and the corresponding hits to wordpress login page. By analyzing the result, you will be be able to find which all domains are facing brute-force attack.


for dom in `ls -l /var/www/vhosts/ | awk -F” ” {‘print $9’}`; do if [ -f /var/www/vhosts/$dom/statistics/logs/access_log ]; then COUNT=`grep wp-login.php /var/www/vhosts/$dom/statistics/logs/access_log |wc -l`; echo “$dom:$COUNT”;fi; done | sort -n -t “:” -k 2 -r

09Jul/14

Install Redis Daemon and Redis PHP extention on CentOS/RHEL/cPanel

Redis is an open-source, networked, in-memory, key-value data store with optional durability. It is written in ANSI C. It’s a “NoSQL” key-value data store. More precisely, it is a data structure server.

To install Redis as daemon on a CentOS/RHEL/cPanel server, do the following steps

cd /usr/local/
wget http://download.redis.io/releases/redis-2.8.12.tar.gz
tar -xvzf redis-2.8.12.tar.gz
cd redis-2.8.12
make
cp src/redis-server /usr/local/bin
cp src/redis-cli /usr/local/bin
mkdir -p /etc/redis
mkdir -p /var/redis
cp redis.conf /etc/redis/redis.conf

Open /etc/redis/redis.conf using vi edirot and set the values as follows.

daemonize yes
port 6379
bind 127.0.0.1
dir  /var/redis/
logfile  /var/log/redis.log
pidfile  /var/run/redis.pid

Now create the startup script. Create a new file as /etc/init.d/redis and add the following contents to it ( Reference https://gist.github.com/paulrosania/257849 ). And make it executable ( chmod 755 /etc/init.d/redis ).

#!/bin/sh
#
# redis - this script starts and stops the redis-server daemon
#
# chkconfig:   - 85 15 
# description:  Redis is a persistent key-value database
# processname: redis-server
# config:      /etc/redis/redis.conf
# config:      /etc/sysconfig/redis
# pidfile:     /var/run/redis.pid
 
# Source function library.
. /etc/rc.d/init.d/functions
 
# Source networking configuration.
. /etc/sysconfig/network
 
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
 
redis="/usr/local/bin/redis-server"
prog=$(basename $redis)
 
REDIS_CONF_FILE="/etc/redis/redis.conf"
 
[ -f /etc/sysconfig/redis ] && . /etc/sysconfig/redis
 
lockfile=/var/lock/subsys/redis
 
start() {
    [ -x $redis ] || exit 5
    [ -f $REDIS_CONF_FILE ] || exit 6
    echo -n $"Starting $prog: "
    daemon $redis $REDIS_CONF_FILE
    retval=$?
    echo
    [ $retval -eq 0 ] && touch $lockfile
    return $retval
}
 
stop() {
    echo -n $"Stopping $prog: "
    killproc $prog -QUIT
    retval=$?
    echo
    [ $retval -eq 0 ] && rm -f $lockfile
    return $retval
}
 
restart() {
    stop
    start
}
 
reload() {
    echo -n $"Reloading $prog: "
    killproc $redis -HUP
    RETVAL=$?
    echo
}
 
force_reload() {
    restart
}
 
rh_status() {
    status $prog
}
 
rh_status_q() {
    rh_status >/dev/null 2>&1
}
 
case "$1" in
    start)
        rh_status_q && exit 0
        $1
        ;;
    stop)
        rh_status_q || exit 0
        $1
        ;;
    restart|configtest)
        $1
        ;;
    reload)
        rh_status_q || exit 7
        $1
        ;;
    force-reload)
        force_reload
        ;;
    status)
        rh_status
        ;;
    condrestart|try-restart)
        rh_status_q || exit 0
	    ;;
    *)
        echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
        exit 2
esac

Now make sure the daemon will start after server reboot.

chkconfig --add redis
chkconfig redis on

Now we can start redis using command /etc/init.d/redis start

To make sure redis is working, run command “redis-cli ping” from commandline. If you get result “PONG”, that shows redis is working.

Now install redis PHP extension, using following command.

pecl install redis

Now open php.ini ( /usr/local/lib/php.ini on cPanel servers ) file and add the following line to it

extension=redis.so

Now restart Apache to apply the changes to php.ini file.

11Jun/14

Service Interruption while EasyApache

I have seen many people warn customers about a few minutes of downtime while running EasyApache on their servers. So they always suggest to run it is off-peak hours. Though it is not recommend to recompile such mission critical stuff during peak traffic hours ( which may put some server load as well ) it will not cause a few minutes of down-time normally. cPanel has built EasyApache Apache package intelligently, so that even though the build fails it will switch Apache/PHP back to the working versions. Also it will do only single service restart at the end, which is only defined service interruption while an EasyApache. But there are other possibilities as well, if Apache restarts during EasyApache build. Like if we restart Apache while building mod_sec module, it will throw syntax error if you have mod_sec directives defined for of of your websites. That is because Apache off-load mod_sec module just to recompile and the error remains until the so file rebuild by EasyApache. That will persists only for a few seconds and if we try to restart Apache during that time it will fail. So in such a case, we will have to force another manual start for Apache after building specific .so file.

01May/14

no certificate found for domain.com in the installed ssl data store – cPanel

Today we had this strange error in WHM while attempting to install SSL certificate for a domain. Auto-fill option worked just fine and filled each field with correct data. But clicking on “Install” button kept showing the above error message. To overcome it, I have gone to WHM >> Home » SSL/TLS » Manage SSL Hosts and “Deleted” the SSL virtualhost for the domain, for which I am attempting to install new certificate. Then tried to install the certificate again and it worked just fine. I believe the error occurred due to the conflict/missing SSL certificate details configured with the old SSLvirtualhost.