Unexpected Fail over warning on Couchbase servers

This is the “expected” behavior. Let me explain it, with a cluster of 3 nodes and 1 replica.

So you have started with 1 node, so in this case you have only “active documents” (no replica)

Then you add another node, and do a rebalance. Once it is done you have 50% of the active data on each node, and 50% of the replica on each node.

Let’s add a new node again, just to have a more “realistic” cluster of 3 nodes. So the node is added and cluster is rebalanced. This means now you have, as you can guess 33.33% on each node (Active and Replica)

So what you have notice is that the Rebalance is an expensive operation, since the cluster has to move data between all the nodes. (moving active and replicas).

You have a now a well balanced 3 nodes cluster.

Now you stop one node, or one node crashes… this means that some of the data are not accessible (they are still here not available, you do not lose anything).

Here you have 2 options:
– if you restart the server, nothing to do the cluster is back online entirely. (3 nodes cluster well balances)

you do a failover on the node that is off. Let’s explain this in detail.
Failover:
So what is happening here: Couchbase will do that as fast as possible to be sure all the data are available(read and write). So the only thing that is happening here is: promote the replicas to active (for the keys that were active on the node that is off now)

So what is the status now?
– all the data are accessible in read/write for the application on 2 nodes, so you have 50% of the active data on each node.
– BUT you do not have all the replicas since:
– the replicas that are on the node that is off are “not present”
– the replicas that have been promoted are not present anymore

This is why you see the message “Fail Over Warning: Rebalance required, some data is not currently replicated!” in your console.

Does it make sense?

So to be able to get back in a status that is “balanced” you need to do a rebalance.

Note: when you failover of node, this node is removed from the cluster, and to add it back you need to add it, and rebalanced. (the data that are on this server are just “ignored”)

Hope this clarify the message.

Some pointers about this:
– http://docs.couchbase.com/couchbase-manual-2.2/#couchbase-admin-tasks-failover5
– http://docs.couchbase.com/couchbase-manual-2.2/#couchbase-admin-tasks-failover-addback4

Installing vim editor.

To install vim

[root@testarossa-00-0c-29-47-8f-35 ~]# yum whatprovides vim-enhanced
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.crazynetwork.it
 * epel: fr2.rpmfind.net
 * extras: mirror.crazynetwork.it
 * updates: mirror.crazynetwork.it
2:vim-enhanced-7.0.109-7.el5.i386 : A version of the VIM editor which includes
                                  : recent enhancements.
Repo        : base
Matched from:

[root@testarossa-00-0c-29-47-8f-35 ~]# yum install vim-enhanced-7.0.109-7.el5.i386 
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.crazynetwork.it
 * epel: ftp.uni-koeln.de
 * extras: mirror.crazynetwork.it
 * updates: mirror.crazynetwork.it
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package vim-enhanced.i386 2:7.0.109-7.el5 set to be updated
--> Processing Dependency: vim-common = 2:7.0.109-7.el5 for package: vim-enhanced
--> Running transaction check
---> Package vim-common.i386 2:7.0.109-7.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package              Arch         Version                   Repository    Size
================================================================================
Installing:
 vim-enhanced         i386         2:7.0.109-7.el5           base         1.2 M
Installing for dependencies:
 vim-common           i386         2:7.0.109-7.el5           base         6.4 M

Transaction Summary
================================================================================
Install       2 Package(s)
Upgrade       0 Package(s)

Total download size: 7.7 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): vim-enhanced-7.0.109-7.el5.i386.rpm               | 1.2 MB     00:03     
(2/2): vim-common-7.0.109-7.el5.i386.rpm                 | 6.4 MB     00:18     
--------------------------------------------------------------------------------
Total                                           344 kB/s | 7.7 MB     00:22     
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : vim-common                                               1/2 
  Installing     : vim-enhanced                                             2/2 

Installed:
  vim-enhanced.i386 2:7.0.109-7.el5                                             

Dependency Installed:
  vim-common.i386 2:7.0.109-7.el5                                               

Complete!
[root@testarossa-00-0c-29-47-8f-35 ~]#

Dsabling SElinux In Centos

Sometimes and with some DB platforms especially when you are testing and want to reduce the number of variables during development, testing etc.. you don not want SELinux watching your back. While it is a must to enable SELinux in hardened production systems it can be quite a pain to handle. Sometimes it needs disabling (if for a short period), Here is how.

# Important

Changes you make to files while SELinux is disabled may give them an unexpected security label, and new files will not have a label. You may need to relabel part or all of the file system after re-enabling SELinux.

Command Line

From the command line, you can edit the /etc/sysconfig/selinux file. This file is a symlink to/etc/selinux/config. The configuration file is self-explanatory. Changing the value of SELINUX orSELINUXTYPE changes the state of SELinux and the name of the policy to be used the next time the system boots.

[root@host2a ~]# cat /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#       enforcing - SELinux security policy is enforced.
#       permissive - SELinux prints warnings instead of enforcing.
#       disabled - SELinux is fully disabled.
SELINUX=permissive
# SELINUXTYPE= type of policy in use. Possible values are:
#       targeted - Only targeted network daemons are protected.
#       strict - Full SELinux protection.
SELINUXTYPE=targeted

# SETLOCALDEFS= Check local definition changes
SETLOCALDEFS=0

At the prompt type :

echo 0 > /selinux/enforce

From the GUI

Use the following procedure to change the mode of SELinux using the GUI.

# Note

You need administrator privileges to perform this procedure.

 

  1. On the System menu, point to Administration and then click Security Level and Firewall to display the Security Level Configuration dialog box.
  1. Click the SELinux tab.
  2. In the SELinux Setting select either DisabledEnforcing or Permissive, and then click OK.
  3. If you changed from Enabled to Disabled or vice versa, you need to restart the machine for the change to take effect.

 

# Note

Changes made using this dialog box are immediately reflected in /etc/sysconfig/selinux.

Introduction to APT (advanced package Tools)

Advanced Packaging Tool

Ubuntu — and all Debian-based distros — includes the Advanced Packaging Tool (APT), which can be used to easily download and install software for the operating system. This article looks at APT, and how it is used.

Topics :
Table of Contents
Installing Software on a Computer
Installing Software on Windows
Installing Software on Linux
Repositories
Enabling Additional Repositories
Software Updates
Configuring Package Updates
GUI Front-Ends to APT
Add/Remove Programs
Synaptic Package Manager
Using APT from the Command-Line
apt-get
dpkg
wget
apt-cache

Introduction to Yum

1. Introduction

 

Yum is a tool for automating package maintenance for a network of workstations running any operating system that use the Red Hat Package Management (RPM) system for distributing packaged tools and applications. It is derived from yup, an automated package updater originally developed for Yellowdog Linux, hence its name: yum is “Yellowdog Updater, Modified”.

Yup was originally written and maintained by Dan Burcaw, Bryan Stillwell, Stephen Edie, and Troy Bengegerdes of Yellowdog Linux (an RPM-based Linux distribution that runs on Apple Macintoshes of various generation). Yum was originally written by Seth Vidal and Michael Stenner, both at Duke University at the time. Since then both Michael and Seth have moved on, Seth to working for Red Hat, where he remains the dominant force behind yum development and maintenance.

It is important to note that yum is an open source GPL project and that many people have contributed code, ideas, bug fixes and documentation. The AUTHORS list was up to 26 or so as of the time of this HOWTO snapshot; yum is a clear example of the power of open source develpmment!

Yum is a Gnu Public License (GPL) tool; it is freely available and can be used, modified, or redistributed without any fee or royalty provided that the terms of its associated license are followed.

1.1 What Yum Can Do

(more…)

Configure, make and make install. GNU configure and build systems.

In Linux installing software can be done in more than one way. Software installed on a platform is always reccommended to be installed from the repositories using the yum or apt tools. These tools have a lot of logic in them to check for package consistency, resolve dependencies , compare local version to the one being installed,  etc.

Yum and Apt will be discussed in other pages but suffice it to say they are the tools you must use (depending on your platform one will be preferred to the other. e.g. the rpm systems usually based on Red Hat redistributions called downstream distros use yum and Debian based distributions use apt as the preferred package manager subsystem.

There are occasions, rare on the average, but the more advanced the setup you are trying to deploy the more common this becomes, when you need to compile a later version from what your repository has. Now this is not a good practise according to stability buffs but it is a necessary one if you are going after security. This because repositories are compiled (or should be) with a certain amount of testing before a version is updated hence there is a natural lag behind the latest stable versions of any one given package. This especially true for packages that are heavily updated (usually due to security updates and bug fixes).

As always the two most important and valid reasons for running versions later than those in the repositories are :

  1. Security : a vulnerability becomes known that might compromise thsecurity of the service or system, and there has been a fix that has been tested.
  2. Features: new features are required that the older version in the repo does not support or was still in beta and has now been promoted to production ready.

On Compiling

(more…)

The /etc/passwd File Format

The /etc/passwd file stores essential information, which is required during login i.e. user account information. /etc/passwd is a text file, that contains a list of the system’s accounts, giving for each account some useful information like user ID, group ID, home directory, shell, etc. It should have general read permission as many utilities, like lsuse it to map user IDs to user names, but write access only for the superuser (root).

The anatomy of /etc/passwd

The /etc/passwd contains one entry per line (row) for each user (or user account) of the system. All fields are separated by a colon (:) symbol. Total seven fields as follows. It is one of the many database text files in NIX systems. Generally, passwd file entry looks as follows :

sample of passwd
A sample row from the /etc/passwd file

 

  1. Username: It is used when user logs in. It should be between 1 and 32 characters in length.
  2. Password: An x character indicates that encrypted password is stored in /etc/shadow file.
  3. User ID (UID): Each user must be assigned a user ID (UID). UID 0 (zero) is reserved for root and UIDs 1-99 are reserved for other predefined accounts. Further UID 100-999 are reserved by system for administrative and system accounts/groups.
  4. Group ID (GID): The primary group ID (stored in /etc/group file)
  5. User ID Info: The comment field. It allow you to add extra information about the users such as user’s full name, phone number etc. This field use by finger command.
  6. Home directory: The absolute path to the directory the user will be in when they log in. If this directory does not exists then users directory becomes /
  7. Command/shell: The absolute path of a command or shell (/bin/bash). Typically, this is a shell. Please note that it does not have to be a shell.

Viewing User List

/etc/passwdis only used for local users only. To see list of all users, enter:

$ less /etc/passwd

To search for a username called toro, enter:

$ grep toro /etc/passwd

/etc/passwd file permissions

The permissions on the /etc/passwd file should be read only to all users i.e. 644 (-rw-r–r–) and the owner must be root: $ ls -l /etc/passwdOutput:

-rw-r--r--. 1 root root 1563 Jul 13 11:03 /etc/passwd

Scanning through /etc/passwd file

One can read the /etc/passwdfile using the while loop and IFS separator as follows:

#!/bin/bash
# seven fields from /etc/passwd stored in $f1,f2...,$f7
#

while IFS=: read -r f1 f2 f3 f4 f5 f6 f7
do
     echo "User $f1 use $f7 shell and stores files in $f6 directory."
done < /etc/passwd

Another way to list all entries in the passwd database is using the getent utility.  This will show all user accounts, regardless of the type of name service used. For example, if both local and LDAP name service are used for user accounts, the results will include all local and LDAP users:

$ getent passwd

The /etc/shadow file

Passwords are not stored in /etc/passwd file the. It is stored in /etc/shadow file. In the good old days there was no great problem with this general read permission. Everybody could read the encrypted passwords, but the hardware was too slow to crack a well-chosen password, and moreover, the basic assumption used to be that of a friendly user-community, both assumptions really wrong today. Almost, all modern Linux / UNIX line operating systems use the shadow password suite, where /etc/passwd has asterisks (*) instead of encrypted passwords, and the encrypted passwords are in /etc/shadow which is readable only by the superuser.

The Linux Conundrum

A large company, was taking over our smaller company and they were on a trend to replace Linux and Java with MS Windows  ®  and ASP.NET.

When the CIO was asked why not go the other way since arguably our smaller company was more advanced put plainly his answer “Linux and Java guys are so hard to find! (and expensive). MS Windows ® guys are all over the place … ”

I liked the proposition that Linux guys are not easy to find, is this really so ..? (feel free to comment) GOOD !!  🙂

So now I know Linux/ Unix is niche, and better paid, but I cannot but ask myself the question why is this so. Is MS Windows ® so much easier or is Linux still growing into a user OS ? and why in the server business is ease of use given importance over customize-ability and tweak-ability.

Also is Linux in any deep way better that MS Windows ®. In my opinion the differences are more in the approach and the attitude of trust towards a single focal point i.e. MS in this case or on a community led by the benevolent dictator  Linus Torvalds . (By the way this is how he pronounces Linux.  [Linux])

I think there is a whole discussion behind this but money affairs aside how did we end up where we are with Linux being so popular and still perceived as difficult.

(more…)