MegaCLI Megaraid useful instructions

Recently I installed a server with a Supermicro SMC2108 RAID adapter, which is actually a LSI MegaRAID SAS 9260. LSI created a command line utility called MegaCLI for Linux to manage this adapter. You can download it from their support pages. The downloaded archive contains an RPM file. I installed mc and rpm on Debian with apt-get, and then extracted the MegaCli64 binary (for x86_64) to /usr/local/sbin, and the libsysfs.so.2.0.2 from the Lib_utils RPM to /opt/lsi/3rdpartylibs/x86_64/ (that’s the location where MegaCli64 looks for this library).

Here are some useful commands:

View information about the RAID adapter

For checking the firmware version, battery back-up unit presence, installed cache memory and the capabilities of the adapter:

# MegaCli64 -AdpAllInfo -aAll

View information about the battery backup-up unit state

# MegaCli64 -AdpBbuCmd -aAll

View information about virtual disks

Useful for checking RAID level, stripe size, cache policy and RAID state:

# MegaCli64 -LDInfo -Lall -aALL

View information about physical drives

# MegaCli64 -PDList -aALL

Patrol read

Patrol read is a feature which tries to discover disk error before it is too late and data is lost. By default it is done automatically (with a delay of 168 hours between different patrol reads) and will take up to 30% of IO resources.

To see information about the patrol read state and the delay between patrol read runs:
# MegaCli64 -AdpPR -Info -aALL

To find out the current patrol read rate, execute
# MegaCli64 -AdpGetProp PatrolReadRate -aALL

To reduce patrol read resource usage to 2% in order to minimize the performance impact:
# MegaCli64 -AdpSetProp PatrolReadRate 2 -aALL

To disable automatic patrol read:
# MegaCli64 -AdpPR -Dsbl -aALL

To start a manual patrol read scan:
# MegaCli64 -AdpPR -Start -aALL

To stop a patrol read scan:
# MegaCli64 -AdpPR -Stop -aALL

You could use the above commands to run patrol read in off-peak times.

Migrate from one RAID level to another

In this example, I migrate the virtual disk 0 from RAID level 6 to RAID 5, so that the disk space of one additional disk becomes available. The second command is used to make Linux detect the new size of the RAID disk.

# /usr/local/sbin/MegaCli64 -LDRecon -Start -r5 -L0 -a0
# echo 1 > /sys/block/sda/device/rescan

Create a new RAID 5 virtual disk from a set of new hard drives

First we need to now the enclosure and slot number of the hard drives we want to use for the new RAID disk. You can find them out by the first command. Then I add a virtual disk using RAID level 5, followed by the list of drives I want to use, specified by enclosure:slot syntax.

# MegaCli64 -PDList -aALL | egrep 'Adapter|Enclosure|Slot|Inquiry'
# MegaCli64 -CfgLdAdd -r5'[252:5,252:6,252:7]' -a0

View reconstruction progress

When reconstructing a RAID array, you can check its progress with this command.
# MegaCli64 -LDRecon ShowProg L0 -a0

(replace L0 by L1 for the second virtual disk, and so on)

Configure write-cache to be disabled when battery is broken

# MegaCli64 -LDSetProp NoCachedBadBBU -LALL -aALL

Change physical disk cache policy

If your system is not connected to a UPS, you should disable the physical disk cache in order to prevent data loss.

# MegaCli -LDGetProp -DskCache -LAll -aALL

To enable it (only do this if you have a UPS and redundant power supplies):

# MegaCli -LDGetProp -DskCache -LAll -aALL

More information

http://ftzdomino.blogspot.com/2009/03/some-useful-megacli-commands.html
https://twiki.cern.ch/twiki/bin/view/FIOgroup/DiskRefPerc
http://hwraid.le-vert.net/wiki/LSIMegaRAIDSAS
http://kb.lsi.com/KnowledgebaseArticle16516.aspx

 

attr: Frederick’s Blog

 

nJoy 😉

 

Using Stunnel to Encrypt Unsecure Connections

So you’re a fully signed up ‘tin foil hatter’ and you have an old app that communicates over the internet but doesn’t use SSL?

Not to worry, there is a solution for you and its called stunnel.

Stunnel is available for both Linux and Windows, and simply put creates an SSL tunnel from one machine or server to another.

So why use stunnel? Well stunnel…

… provides an extra layer of security to an internet based application.
… secures data previously transmitted in plain text.
… protects your exploitable services by hiding them behind stunnel.

So what else can it be used for? Well…

… you could use it to fool certain websites into thinking you are in a different location.
… it can be used as a simple ‘bouncer’ for pretty much anything!

 

In my example I wanted to provide a little security to my simple ‘Munin’ monitoring solution (I’ll write something about that at a later date). Munin checks the servers status by ‘telnet’ to the munin-node running on the target server/pc, so if you are monitoring a remote server you have data about your system flying about everywhere! Also it means that there is a naked telnet port open on your server, which although you can lock it down by only allowing connections from a specific IP etc. doesn’t work to well if your on a dynamic IP and still leaves unencrypted data flying about the ether. Ok so in this case security isn’t that vital and none of the data is particularly sensitive. But if it uses TCP and you want to provide an extra layer of protection stunnel can do it.

As an added bonus stunnel is extremely easy to setup.

The rest of this post is based around a CentOS installation. But its pretty much the same on Debian/Ubuntu… just replace ‘yum’ with ‘apt-get’. The Windows install is even easier and the configs are in the same format as the Linux confs so you can easily adapt the instructions bellow!

Firstly lets install stunnel on the client machine (In the case of my example this is actually my ‘server’ machine in Munin, but for example if you wanted to secure SMTP communications this would be the machine sending the emails).

Although in this post I’ve used Munin as an example it really is easy to modify these instructions to fit any service. Simply change the ports!

Install using Yum:

yum install stunnel

(Now I cant remember exactly but I think stunnel is not on the default CentOS repos… so you can add the RPMforge repos like I tend to do: http://www.rpmrepo.org/RPMforge/Using)

Install from source:

wget http://mirror.hudecof.net/stunnel/stunnel-4.22.tar.gz
tar zxf stunnel-4.22.tar.gz
cd stunnel-4.22
./configure
make
make install

Configuration:

On CentOS the default location for the stunnel.conf is /etc/stunnel/ so open this file in your editor of choice:

vi /etc/stunnel/stunnel.conf

Lets set the following options:

#chroot = /var/run/stunnel
setuid = nobody
setgid = nobody
pid = /stunnel.pid
client = yes
debug=7

And create a service:

[tomcat]
accept = 9090
connect = 8080

Now as you can see here I’ve set the listener to 127.0.0.1 but you could set this to all interfaces or a specific one by omitting or replacing the 127.0.0.1: The ‘connect’ setting is the servers hostname or IP address and the port that the stunnel ‘server’ is listening on. Another example of using stunnel could be to direct all web requests onto another server using a secure layer:

[www]
accept = 80
connect = myserver.hotname.com:8080

You would then setup the ‘server’ stunnel to listen on 8080 and connect to the local (or even a remote!) web server.

Ok thats it for the client side for now. Lets look at the server:

Install stunnel as per the installation instructions above.

On CentOS the default location for the stunnel.conf is /etc/stunnel/ so open this file in your editor of choice:

vi /etc/stunnel/stunnel.conf

Set the following options:

cert = /etc/stunnel/stunnel.pem
chroot = /var/run/stunnel
setuid = nobody
setgid = nobody
pid = /stunnel.pid
debug=7

And create a service:

[tomcat]
accept = 9090
connect = 8080

[www]
accept = 8080
connect = myserver.hostname.com:80

With a bit of messing around with DNS or hosts files we could use stunnel to bounce our connection to any server for example www.bbc.co.uk! But that’s not really what we’re covering in this article and I’m guessing if you have your reasons for doing something like that you can figure it out on your own.

OK we’re nearly ready to start stunnel. But the observant amongst you will have noticed that stunnel.pem certificate file that we set in the server options doesn’t exist! So lets create one now!

openssl req -new -x509 -days 3650 -nodes -out /etc/stunnel/stunnel.pem -keyout /etc/stunnel/stunnel.pem

Right now lets start stunnel, on both machines simply run the following command:

/usr/sbin/stunnel /etc/stunnel/stunnel.conf

Now lets test it! If the service we’ve setup can be talked to with telnet (eg. Munin or SMTP) then we can test this very simply from the client machine:

telnet localhost:4948

You should get the following back:

Trying 127.0.0.1…
Connected to localhost (127.0.0.1).
Escape character is ‘^]’.
# munin node at mysever.hostname.com

(Now its a little confusing because ‘Connected to localhost’ is actually the response from the munin-node on the remote server!)

As you can see your telnet session has gone into stunnel locally, been transmitted from stunnel on the local machine to stunnel on the server and then from stunnel into the Munin node on the server! Magic!

Stunnel startup script:

Now you can either start stunnel every time your machine starts up manually, add it to the crontab (if you try and start stunnel again and its already running the second instance will just close, but it leaves a mess in your /var/log/secure so don’t do it to often) or use a simple startup script like this one I ( used ) to use:

#!/bin/bash
    if [ -f /var/run/stunnel/stunnel.pid ]; then
        ps aux |grep -v grep |grep $pid |grep stunnel > /dev/null
        if [ $? = 0 ]; then
            echo “Server is already running !!”
        else
            echo “Pid file exists but process not found … trying to start stunnel”
            /usr/sbin/stunnel /etc/stunnel/stunnel.conf
        fi
        rm -f /tmp/stunnelrun > /dev/null
    else
        echo “Pid file not found. Starting stunnel.”
        /usr/sbin/stunnel  /etc/stunnel/stunnel.conf
    fi

This is a very simple script I knocked up in a few mins. I later replaced it with an init script that I wrote when I had a little more time. But I’m going to post that as another article as I have a bit more to say about that (including some adaptation to make general init scripts for various programs)

To use the above script to start stunnel do the following:

cd /usr/local/sbin
vi stunnel-run

Paste in the above code.
Save and exit.

chmod +x stunnel-run

Test by doing the following:

./stunnel-run

Now add it to the crontab:

crontab -e

Insert the following line:

*/15 * * * * /usr/local/sbin/stunnel-run 2>&1 > /dev/null

Now your system will run the stunnel-run script every 15mins, checking if stunnel is running and starting stunnel if it is not running. You could simply start stunnel every 15mins as it will exit if it finds it can not use the ports its been assigned, but that leaves a mess in your secure log.

Some of you I’m sure are thinking ‘why use stunnel you could use a vpn or xyz’. Well yes, there are many ways of skinning this cat, stunnel is a simple and versatile one but not the ultimate jack of all trades. Use it where you see fit!

Change Hostname Permanently on Debian or Ubuntu

Debian based systems use the file /etc/hostname to read the hostname of the computer at boot time and set it up using the init script /etc/init.d/hostname.sh

We can edit the file /etc/hostname and change the hostname and then run:

/etc/init.d/hostname.sh start

Steps:

1. sudo vim /etc/hostname
2. Save the file with the hostname you like to set
3. sudo /etc/init.d/hostname.sh start

Ref: http://sysblogd.com/50/change-hostname-permanently-on-debian-or-ubuntu.htm

 

nJoy 😉

Moving SSL Certs from IIS to Apache

Some instructions for converting SSL certificates generated for IIS to private key, and cert files you can use on unix, or Apache for windows.

First Export your IIS certificate into a pfx file (this is something you should do anyways for backup)

  • Run mmc.exe
  • Click the ‘Console’ menu and then click ‘Add/Remove Snap-in’.
  • Click the ‘Add’ button and then choose the ‘certificates’ snap-in and click on ‘Add’.
  • Select ‘Computer Account’ then click ‘Next’.
  • Select ‘Local Computer’ and then click ‘OK’.
  • Click ‘Close’ and then click ‘OK’.
  • Expand the menu for ‘Certificates’ and click on the ‘Personal’ folder.
  • Right click on the certificate that you want to export and select ‘All tasks’ -> ‘Export’.
  • A wizard will appear. Make sure you check the box to include the private key and continue through with this wizard until you have a .PFX file.

Next run openssl to extract the private key, and the cert file.

# Export the private key file from the pfx file
openssl pkcs12 -in filename.pfx -nocerts -out key.pem
# Export the certificate file from the pfx file
openssl pkcs12 -in filename.pfx -clcerts -nokeys -out cert.pem
# This removes the passphrase from the private key so Apache won't
# prompt you for your passphase when it starts
openssl rsa -in key.pem -out server.key

Booting into Single User Mode – (Password Recovery)

Many occasions during commissioning of servers we need to reboot especially to freshen hardware re-configurations. (ain’t that still a bitch). Single user mode is there to help.

Booting into single user mode

  1. At the GRUB splash screen at boot time, press any key to enter the GRUB interactive menu.
  2. Select CentOS with the version of the kernel that you wish to boot and type ‘a' to append the line.
  3. Go to the end of the line and type single as a separate word (press the Spacebar and then type single). Press Enter to exit edit mode.
    You are in Single User mode.

Checking for email server blacklisting

Trying to understand why mail is not being sent ?

[For Sendmail]

If your  tail -f /var/log/maillog logs are showing something like this:

Aug 30 22:43:06 netman sendmail[8100]: starting daemon (8.14.4): SMTP+queueing@01:00:00
Aug 30 22:43:06 netman sm-msp-queue[8109]: starting daemon (8.14.4): queueing@01:00:00
Aug 30 22:43:11 netman sendmail[8102]: q7SIq1Kk011256: to=<david.saliba@jial.com>, ctladdr=<root@netman.lan> (0/0), delay=2+04:51:10, xdelay=00:00:05, mailer=esmtp, pri=3720580, relay=alt4.jial-smtp-in.l.gogglee.com. [XX.125.142.26], dsn=4.0.0, stat=Deferred: alt4.jial-smtp-in.l.googglee.com.: No route to host

Try telnet-ing to the IP  [XX.125.142.26] on port 25:

telnet XX.125.142.26 25

[root@netman ~]# telnet 74.125.142.27 25
Trying 74.125.142.27...
telnet: connect to address 74.125.142.27: No route to host
[root@netman ~]#

Check here to see if your server is blacklisted using this site:

http://www.mxtoolbox.com/blacklists.aspx

There all you need to do is enter your external IP address and see if that is the issue.

If not remember if you are using dynamically assigned IPs there is a good chance that’s the issue try relaying through another server.

 

Creating a router on a CentOS 6 server

Assuming you want to NAT the network on eth1 and route the traffic to eth0 this is the spell:

Create the forwarding rule:

/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

and then enable IP forwarding

 echo “1” > /proc/sys/net/ipv4/ip_forward

OR

sysctl -w net.ipv4.ip_forward=1

 

Permanent setting using /etc/sysctl.conf

If we want to make this configuration permanent the best way to do it is using the file/etc/sysctl.conf where we can add a line containing net.ipv4.ip_forward = 1

/etc/sysctl.conf: net.ipv4.ip_forward = 1

if you already have an entry net.ipv4.ip_forward with the value 0 you can change that 1.

To enable the changes made in sysctl.conf you will need to run the command:

sysctl -p /etc/sysctl.conf

On RedHat based systems this is also enabled when restarting the network service:

service network restart

 

Windows time stamping in batch files

Creating a time-stamp  in windows can be usful for the automated backups we all SHOULD be doing 🙂

 

pkzip c:\<source>\*.* c:\<target>\TempZip.zip
ren C:\<target>\TempZip.Zip c:\<target>\TempZip_%date:~-4,4%%date:~-7,2%%date:~-10,2%.zip

 

Or simply to create a directory for copying open files (Lawrence 🙂 )

set backdir=%date:~-4,4%%date:~-7,2%%date:~-10,2%

mkdir %backdir%

cd %backdir%

etc..

 

 

Common Errors and Fixes for OpenStack Compute

The Launchpad Answers site offers a place to ask and answer questions, and you can also mark questions as frequently asked questions.

Credential errors, 401, 403 forbidden errors

A 403 forbidden error is caused by missing credentials. Through current installation methods, there are basically two ways to get the novarc file. The manual method requires getting it from within a project zipfile, and the scripted method just generates novarc out of the project zip file and sources it for you. If you do the manual method through a zip file, then the following novarc alone, you end up losing the creds that are tied to the user you created with nova-manage in the steps before.

When you run nova-api the first time, it generates the certificate authority information, including openssl.cnf. If it gets started out of order, you may not be able to create your zip file. Once your CA information is available, you should be able to go back to nova-manage to create your zipfile.

You may also need to check your proxy settings to see if they are causing problems with the novarc creation.

Instance errors

Sometimes a particular instance shows “pending” or you cannot SSH to it. Sometimes the image itself is the problem. For example, when using flat manager networking, you do not have a dhcp server, and an ami-tiny image doesn’t support interface injection so you cannot connect to it. The fix for this type of problem is to use an Ubuntu image, which should obtain an IP address correctly with FlatManager network settings. To troubleshoot other possible problems with an instance, such as one that stays in a spawning state, first check your instances directory for i-ze0bnh1q dir to make sure it has the following files:

  • libvirt.xml
  • disk
  • disk-raw
  • kernel
  • ramdisk
  • console.log (Once the instance actually starts you should see a console.log.)

Check the file sizes to see if they are reasonable. If any are missing/zero/very small then nova-compute has somehow not completed download of the images from objectstore.

Also check nova-compute.log for exceptions. Sometimes they don’t show up in the console output.

Next, check the /var/log/libvirt/qemu/i-ze0bnh1q.log file to see if it exists and has any useful error messages in it.

Finally, from the instances/i-ze0bnh1q directory, try virsh create libvirt.xml and see if you get an error there.