Monday, 21 January 2013

MogileFS - Installation on RHEL/Cent OS Linux

MogileFS Briefly
• A scalable, Fault tolerant, High performance distributed file system
• Asynchronous Replication
• No Single Point of Failure
• Automatic file replication (3 replications Tracker Storage Node recommended)
• Better22 than RAID Host1 Host4
• Data Compression (BZIP2)
• Flat NameSpace
• Share-Nothing Tracker Storage Node
• No RAID required
• Local filesystem agnostic Host2 Host5
• Tracker client transfer (mogilefsd)
   - Replication
   - Deletion
   - Query
   - Reaper
   - Monitor
• Files are broken up and spread over the Storage Node (mogstored) HTTP and WebDAV server DBNode Storage Node
• DBNode MySQL stores the MogileFS metadata (the namespace, and which files are where) Host3 Host6• Client Library : Ruby, Perl, Java, Python, PHP…

Database creation for mogilefs: IP
# mysql -u root -p
mysql> CREATE DATABASE mogilefs;
mysql> grant all on mogilefs.* TO ‘mogile’@'localhost’ identified by ‘secret’;
mysql> grant all on mogilefs.* TO 'mogile'@'%' identified by 'secret';
mysql> flush privileges;
mysql> quit

Step 1# Download & Install MogileFS-Server
#unzip master
#cd MogileFS-Server
#perl Makefile.PL
#make test
#make install

--------------------For Your Information--------------------------------------
If you get any errors during above (make install) process it Will probably be errors telling you that a dependent module is missing. If during the 'make test' step you get the error, "t/00-startup....DBI connect('mysql','root',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at t/lib/ line 16" it can probably be ignored – it should not be necessary for MySQL to be running

Step 3# Download & Install MogileFS-Utils
#unzip master
#cd MogileFS-Utils-master
#perl Makefile.PL
#make test
#make install

Step 4# Download & Install MogileFS-Client
#unzip master
#cd perl-MogileFS-Client-master
#perl Makefile.PL
#make test
#make install

Step 5# Install perl modules

perl -MCPAN -e 'install Danga::Socket'
perl -MCPAN -e 'install Gearman::Client'
perl -MCPAN -e 'install Gearman::Server'
perl -MCPAN -e 'install Gearman::Client::Async'
perl -MCPAN -e 'install Net::Netmask'
perl -MCPAN -e 'install IO::WrapTie'
perl -MCPAN -e 'install IO::AIO'
perl -MCPAN -e 'install List::MoreUtils'
perl -MCPAN -e 'install Path::Class'
perl -MCPAN -e 'install Perlbal'
perl -MCPAN -e 'install ExtUtils::MakeMaker'
perl -MCPAN -e 'install DBD::mysql'


#cd Linux-AIO-1.9
#make test
#make install

Step 6# Database Configuration

#mogdbsetup --dbname=mogilefs --dbuser=mogile --dbpassword=secret

---------------------------------------For Your Information-----------------
The database is empty and will need a schema applied. A utility named 'mogdbsetup' to make this process simple. By default it assumes the database is located on localhost.

The mogdbsetup utility does not specify a table type by default so your tables will match the defaults for your database. In many cases this will mean that you end up with MyISAM tables. If you prefer InnoDB tables you will either need to make sure your database defaults to InnoDB or you can manually convert the tables (both of these are outside of the scope of this document but there are plenty of examples out there).

Step 7# Tracker Configuration
On each tracker server create a configuration file at /etc/mogilefs/mogilefsd.conf with the following:

db_user mogile
db_pass secret
conf_port 6001
listener_jobs 5

----------------For Your Information--------------------------------
1. db_dsn points to your database instance.
2. If you are running the database on the same machine as the storage server you can omit " and it will use the local machine.
3.db_user and db_pass should match the user and password you configured when setting up your database.

Step 8# Adding user to run mogilefsd service

The program 'mogilefsd' will not run as root so you will need to run this as a non-root user. To create a user for this, enter the following command and follow the prompts to create the "mogile" user:

# adduser mogile

Step 9# Storage Server Configuration

On each storage server, create the storage directory (make sure it has access permissions for the user you will use to run mogstored):

# mkdir /var/mogdata

Configure it:

On each storage server, create a configuration file at /etc/mogilefs/mogstored.conf with the following:


Commands for Administration of MogileFS

To add domain:
mogadm --trackers= domain add consim

To add class domain:
mogadm --trackers= class add consim classconsim

To add Host
mogadm --trackers= host add mogilestorage --ip= --port=7500 --status=alive

To add device name for host
mogadm --trackers= device add mogilestorage dev1

To delete host :
mogadm --trackers= host delete mogilefstorage

To list the host:
mogadm --trackers= host list

To list the device:
mogadm --trackers= device list

To Check:
mogadm --trackers= check

To upload:
mogupload --trackers= --domain="consim" --key="/hello" --file="/tmp/test2.txt"

To Download:

mogfetch --trackers= --domain="consim" --key="/hello" --file="/home/test2f.txt"

To Delete:
mogdelete --trackers= --domain="consim" --key="/hello"

To know the Statistics:

for all
mogstats --db_dsn="DBI:mysql:mogilefs:host=" --db_user="mogile" --db_pass="secret" --verbose --stats="all"


for device & files only
mogstats --db_dsn="DBI:mysql:mogilefs:host=" --db_user="mogile" --db_pass="secret" --verbose --stats="devices,files"

To know the file Info:
mogfileinfo --trackers= --domain=consim --key='/hello'

To rename the key name:
Usage: /usr/local/bin/mogrename --trackers=host --domain=foo --class=bar --old_key='/hello.jpg' --new_key='/bye.jpg'

Extra Informations:

1. If you want the tracker to use memcached:
#mogadm --trackers= settings set memcache_servers memchage_installed_ip:its_port

2. To use nginx as web server Refer the below

3. php driver to upload/download files via php coding

4. Script to get the host/device check through mail

mogadm --trackers=, check |tee /tmp/mogadm.check |grep OK && cat /tmp/mogadm.check | mail -s "mogadm check status"

5. To upload/download files through Web Interface


1) Create a read-only user who has SELECT privileges to your MogileFS database

    Something like:

        GRANT SELECT ON mogilefs.* TO 'mogilero'@'localhost' IDENTIFIED BY 'password';

2) tar xzvf mogilefsbrowser-1.0.0b3.tar.gz

2) cp mogilefsbrowser /var/www/html

3) Edit config.php and supply the settings for your Mogile Database and Mogile system

3) create & Set permissions on docs/logs.txt to be writable by the web server process

4) Open in a web browser

Client Configuration: IP:

Step 1# Download & Install MogileFS-Server (as done like above)
Step 2# Download & Install MogileFS-Utils  (as done like above)
Step 3# Download & Install MogileFS-Client (as done like above)
Step 4# Install perl modules (as done like above)
Step 5# Tracker Configuration (as done like above) [in this add the ip address of MySQL DB server]

[root@consim tmp]# cat /etc/mogilefs/mogilefsd.conf
db_dsn DBI:mysql:mogilefs:
db_user mogile
db_pass secret
conf_port 6001
listener_jobs 5

Step 6# Adding user to run mogilefsd service (as done like above)
Step 7# Storage Server Configuration (as done like above)

Step 9# Run in Mogilefs DB Server:

Add client Host
mogadm --trackers= host add mogilestorage1 --ip= --port=7500 --status=alive

To add device name for client host
mogadm --trackers= device add mogilestorage1 dev2

#once device name is added please create the directory In client( /var/mogdata/dev2

No comments:

Post a Comment