Category: Howtos

Bash Datestamps

Posted on by admin 
        Format/result         |       Command              |          Output
------------------------------+----------------------------+------------------------------
YY-MM-DD_hh:mm:ss             | date +%F_%T                | $(date +%F_%T)
YYMMDD_hhmmss                 | date +%Y%m%d_%H%M%S        | $(date +%Y%m%d_%H%M%S)
YYMMDD_hhmmss (UTC version)   | date --utc +%Y%m%d_%H%M%SZ | $(date --utc +%Y%m%d_%H%M%SZ)
YYMMDD_hhmmss (with local TZ) | date +%Y%m%d_%H%M%S%Z      | $(date +%Y%m%d_%H%M%S%Z)
YYMMDDhhmmss                  | date +%Y%m%d%H%M%S         | $(date +%Y%m%d%H%M%S)
YYMMDDhhmmssnnnnnnnnn         | date +%Y%m%d%H%M%S%N       | $(date +%Y%m%d%H%M%S%N)
Seconds since UNIX epoch:     | date +%s                   | $(date +%s)
Nanoseconds only:             | date +%N                   | $(date +%N)
Nanoseconds since UNIX epoch: | date +%s%N                 | $(date +%s%N)
ISO8601 UTC timestamp         | date --utc +%FT%TZ         | $(date --utc +%FT%TZ)
ISO8601 Local TZ timestamp    | date +%FT%T%Z              | $(date +%FT%T%Z)


njoy ;-)

Mounting an LVM file system

Posted on by admin

So after recovering a box which had a controller failure we needed to mount the old fs to recover some stuff.

when trying to mount /dev/sdc2 we got:

mount: unknown filesystem type ‘LVM2_member’

fix is easy :

modprobe dm-mod

vgchange -ay

lvscan now yields:

ACTIVE '/dev/VolGroup/lv_root' [50.00 GiB] inherit
ACTIVE '/dev/VolGroup/lv_home' [178.46 GiB] inherit
ACTIVE '/dev/VolGroup/lv_swap' [3.94 GiB] inherit

mount /dev/VolGroup/lv_root /mnt/tempdisk/

et viola` !!

nJoy 😉

 

 

 

Backup and Restore Elastic search

Posted on by admin

While elastic search is usually run as a cluster, for the sake of this tutorial I am showing the _snapshot and _restore tools.

 

mkdir  /mnt/backups/my_backup
chmod 777 -R /mnt/backups/

Must available on all nodes.


 curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
 "type": "fs",
 "settings": {
        "location": "/mnt/backups/my_backup",
   "compress": true
    }
}'






[root@centos-base mnt]# curl -XGET 'http://localhost:9200/_snapshot/my_backup?pretty'

{
  "my_backup" : {
    "type" : "fs",
    "settings" : {
      "compress" : "true",
      "location" : "/mnt/backups/my_backup"
    }
  }
}



 curl -XGET 'http://localhost:9200/_snapshot?pretty'                                       {
  "my_backup" : {
    "type" : "fs",
    "settings" : {
      "compress" : "true",
      "location" : "/mnt/backups/my_backup"
    }
  }
}

_____________________________________________________________________________________________

changing


 curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
 "type": "fs",
 "settings": {
  "location": "/mnt/backups/my_backup",
 "compress": true,
 "verify":true
	}	
	}'


 curl -XPUT "localhost:9200/_snapshot/my_backup/snapshot_`date | tr -d " " | tr -d ":" | tr '[:upper:]' '[:lower:]' `?wait_for_completion=true&pretty"
 
 

 
_____________________________________________________________________________________________

restoring

 mkdir -p /mnt/backups/my_backup
 chmod -R 777 /mnt/backups/

 
 Create repository
 -----------------------
 
 curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
 "type": "fs",
 "settings": {
  "location": "/mnt/backups/my_backup",
 "compress": true,
 "verify":true
	}	 
	}'

	
 restore from file system
 --------------------------------
 
  curl -XPOST "localhost:9200/_snapshot/my_backup/snapshot_satapr25223454cest2015/_restore"

nJoy 😉

Sending Elasticsearch to a syslog server

Posted on by admin

yum install rsyslog -y

Add the following to rsyslog.conf on the client system

############

$ModLoad imfile
$InputFileName /var/log/elasticsearch/elasticsearch.log
$InputFileTag elasticsearch
$InputFileStateFile stat-elasticsearch
$InputFileSeverity Info
$InputFileFacility daemon
$InputRunFileMonitor
#local3.* hostname:<portnumber>

daemon.* @192.168.1.66:514


############

 

Also if you want all logs to go through to syslog server:

 

*.* @192.168.1.66

 

at the end of the file.

Issue a :

service rsyslog restart

and watch the logs flow in.

 

nJoy 😉

 

 

 

 

fixing permission denied issue with udp 512 port graylog

Posted on by admin

use iptables to pre-route NAT the udp port :

 

iptables -A PREROUTING -t nat -i eth0 -p udp –dport 514 -j REDIRECT –to-port 10515

 

This will bypass the limit in the OS to ports < 1024 to non=root users .

nJoy 😉

 

Installing sample data in elastic search

Posted on by admin

After installing elastic search it is useful for testing and training to load some sample data.

1) create mapping :

curl -XPUT http://localhost:9200/shakespeare -d '
{
 "mappings" : {
  "_default_" : {
   "properties" : {
    "speaker" : {"type": "string", "index" : "not_analyzed" },
    "play_name" : {"type": "string", "index" : "not_analyzed" },
    "line_id" : { "type" : "integer" },
    "speech_number" : { "type" : "integer" }
   }
  }
 }
}
';

2) Load the data using the bulk api:

wget "https://github.com/ropensci/elastic_data/blob/master/data/shakespeare_data.json?raw=true" -O  shakespeare.json

curl -XPUT localhost:9200/_bulk --data-binary @shakespeare.json

 

nJoy 😉

 

 

 

Pretty Json in Bash

Posted on by admin

Simply pipe through

python -m json.tool

e.g.

 

curl -u Administrator:kpassword http://172.16.3.136:8091//pools/default | python -m json.tool

 

Restoring a bucket in Couchbase

Posted on by admin

[root@mofo bin]# ./cbrestore /tmp/cbbackup/2015-01-19T220530Z/2015-01-19T220530Z-full/ -u Administrator -ppassword http://mofo.lan:8091 –bucket-source=beer-sample –bucket-destination=DAVID
[####################] 100.0% (7303/estimated 7303 msgs)
bucket: beer-sample, msgs transferred…
: total | last | per sec
byte : 2541549 | 2541549 | 3415000.2
done
[root@mofo bin]#