MogileFS Briefly
****************
• A scalable, Fault tolerant, High performance distributed file system
• Asynchronous Replication
• No Single Point of Failure
• Automatic file replication (3 replications Tracker Storage Node recommended)
• Better22 than RAID Host1 Host4
• Data Compression (BZIP2)
• Flat NameSpace
• Share-Nothing Tracker Storage Node
• No RAID required
• Local filesystem agnostic Host2 Host5
• Tracker client transfer (mogilefsd)
- Replication
- Deletion
- Query
- Reaper
- Monitor
• Files are broken up and spread over the Storage Node (mogstored) HTTP and WebDAV server DBNode Storage Node
• DBNode MySQL stores the MogileFS metadata (the namespace, and which files are where) Host3 Host6• Client Library : Ruby, Perl, Java, Python, PHP…
Installation:
*************
Database creation for mogilefs: IP 192.168.2.14
************************************************
# mysql -u root -p
mysql> CREATE DATABASE mogilefs;
mysql> grant all on mogilefs.* TO ‘mogile’@'localhost’ identified by ‘secret’;
mysql> grant all on mogilefs.* TO 'mogile'@'%' identified by 'secret';
mysql> flush privileges;
mysql> quit
Step 1# Download & Install MogileFS-Server
******************************************
#wget https://github.com/mogilefs/MogileFS-Server/archive/master.zip
#unzip master
#cd MogileFS-Server
#perl Makefile.PL
#make
#make test
#make install
--------------------For Your Information--------------------------------------
If you get any errors during above (make install) process it Will probably be errors telling you that a dependent module is missing. If during the 'make test' step you get the error, "t/00-startup....DBI connect('mysql','root',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at t/lib/mogtestlib.pl line 16" it can probably be ignored – it should not be necessary for MySQL to be running
-------------------------------------------------------------------------------
Step 3# Download & Install MogileFS-Utils
*****************************************
wget https://github.com/mogilefs/MogileFS-Utils/archive/master.zip
#unzip master
#cd MogileFS-Utils-master
#perl Makefile.PL
#make
#make test
#make install
Step 4# Download & Install MogileFS-Client
******************************************
wget https://github.com/mogilefs/perl-MogileFS-Client/archive/master.zip
#unzip master
#cd perl-MogileFS-Client-master
#perl Makefile.PL
#make
#make test
#make install
Step 5# Install perl modules
****************************
perl -MCPAN -e 'install Danga::Socket'
perl -MCPAN -e 'install Gearman::Client'
perl -MCPAN -e 'install Gearman::Server'
perl -MCPAN -e 'install Gearman::Client::Async'
perl -MCPAN -e 'install Net::Netmask'
perl -MCPAN -e 'install IO::WrapTie'
perl -MCPAN -e 'install IO::AIO'
perl -MCPAN -e 'install List::MoreUtils'
perl -MCPAN -e 'install Path::Class'
perl -MCPAN -e 'install Perlbal'
perl -MCPAN -e 'install ExtUtils::MakeMaker'
perl -MCPAN -e 'install DBD::mysql'
OR
#wget http://search.cpan.org/CPAN/authors/id/M/ML/MLEHMANN/Linux-AIO-1.9.tar.gz
#cd Linux-AIO-1.9
#make
#make test
#make install
Step 6# Database Configuration
******************************
#mogdbsetup --dbhost=mogiledb.yourdomain.com --dbname=mogilefs --dbuser=mogile --dbpassword=secret
---------------------------------------For Your Information-----------------
The database is empty and will need a schema applied. A utility named 'mogdbsetup' to make this process simple. By default it assumes the database is located on localhost.
The mogdbsetup utility does not specify a table type by default so your tables will match the defaults for your database. In many cases this will mean that you end up with MyISAM tables. If you prefer InnoDB tables you will either need to make sure your database defaults to InnoDB or you can manually convert the tables (both of these are outside of the scope of this document but there are plenty of examples out there).
-----------------------------------------------------------------------------
Step 7# Tracker Configuration
*****************************
On each tracker server create a configuration file at /etc/mogilefs/mogilefsd.conf with the following:
db_dsn DBI:mysql:mogilefs:mogiledb.yourdomain.com
db_user mogile
db_pass secret
conf_port 6001
listener_jobs 5
----------------For Your Information--------------------------------
1. db_dsn points to your database instance.
2. If you are running the database on the same machine as the storage server you can omit ":mogiledb.yourdomain.com: and it will use the local machine.
3.db_user and db_pass should match the user and password you configured when setting up your database.
--------------------------------------------------------------------
Step 8# Adding user to run mogilefsd service
********************************************
The program 'mogilefsd' will not run as root so you will need to run this as a non-root user. To create a user for this, enter the following command and follow the prompts to create the "mogile" user:
# adduser mogile
Step 9# Storage Server Configuration
************************************
On each storage server, create the storage directory (make sure it has access permissions for the user you will use to run mogstored):
# mkdir /var/mogdata
Configure it:
On each storage server, create a configuration file at /etc/mogilefs/mogstored.conf with the following:
httplisten=0.0.0.0:7500
mgmtlisten=0.0.0.0:7501
docroot=/var/mogdata
Commands for Administration of MogileFS
=======================================
To add domain:
**************
mogadm --trackers=192.168.2.14:6001 domain add consim
To add class domain:
********************
mogadm --trackers=192.168.2.14:6001 class add consim classconsim
To add Host
***********
mogadm --trackers=192.168.2.14:6001 host add mogilestorage --ip=192.168.2.14 --port=7500 --status=alive
To add device name for host
***************************
mogadm --trackers=192.168.2.14:6001 device add mogilestorage dev1
To delete host :
***************
mogadm --trackers=192.168.2.14:6001 host delete mogilefstorage
To list the host:
*****************
mogadm --trackers=192.168.2.14:6001 host list
To list the device:
*****************
mogadm --trackers=192.168.2.14:6001 device list
To Check:
*********
mogadm --trackers=192.168.2.14:6001 check
To upload:
**********
mogupload --trackers=192.168.2.14:6001 --domain="consim" --key="/hello" --file="/tmp/test2.txt"
To Download:
***********
mogfetch --trackers=192.168.2.14:6001 --domain="consim" --key="/hello" --file="/home/test2f.txt"
To Delete:
**********
mogdelete --trackers=192.168.2.14:6001 --domain="consim" --key="/hello"
To know the Statistics:
***********************
for all
-------
mogstats --db_dsn="DBI:mysql:mogilefs:host=192.168.2.14" --db_user="mogile" --db_pass="secret" --verbose --stats="all"
or
for device & files only
-----------------------
mogstats --db_dsn="DBI:mysql:mogilefs:host=192.168.2.14" --db_user="mogile" --db_pass="secret" --verbose --stats="devices,files"
To know the file Info:
*********************
mogfileinfo --trackers=192.168.2.14:6001 --domain=consim --key='/hello'
To rename the key name:
***********************
Usage: /usr/local/bin/mogrename --trackers=host --domain=foo --class=bar --old_key='/hello.jpg' --new_key='/bye.jpg'
Extra Informations:
*******************
1. If you want the tracker to use memcached:
#mogadm --trackers=192.168.1.10:6001 settings set memcache_servers memchage_installed_ip:its_port
2. To use nginx as web server Refer the below
**********************************************
http://code.google.com/p/mogilefs/wiki/nginx_webdav
3. php driver to upload/download files via php coding
*****************************************************
wget http://pecl.php.net/get/mogilefs-0.9.2.tgz
4. Script to get the host/device check through mail
***************************************************
mogadm --trackers=192.168.2.14:6001,192.168.1.10:6001 check |tee /tmp/mogadm.check |grep OK && cat /tmp/mogadm.check | mail -s "mogadm check status" mdmansoor26@gmail.com
5. To upload/download files through Web Interface
**************************************************
#wget http://mogilefs-browser.googlecode.com/files/mogilefsbrowser-1.0.0b3.tar.gz
1) Create a read-only user who has SELECT privileges to your MogileFS database
Something like:
GRANT SELECT ON mogilefs.* TO 'mogilero'@'localhost' IDENTIFIED BY 'password';
2) tar xzvf mogilefsbrowser-1.0.0b3.tar.gz
2) cp mogilefsbrowser /var/www/html
3) Edit config.php and supply the settings for your Mogile Database and Mogile system
3) create & Set permissions on docs/logs.txt to be writable by the web server process
4) Open in a web browser
Client Configuration: IP: 192.168.1.10
======================================
Step 1# Download & Install MogileFS-Server (as done like above)
Step 2# Download & Install MogileFS-Utils (as done like above)
Step 3# Download & Install MogileFS-Client (as done like above)
Step 4# Install perl modules (as done like above)
Step 5# Tracker Configuration (as done like above) [in this add the ip address of MySQL DB server]
Example:
********
[root@consim tmp]# cat /etc/mogilefs/mogilefsd.conf
db_dsn DBI:mysql:mogilefs:192.168.2.14
db_user mogile
db_pass secret
conf_port 6001
listener_jobs 5
Step 6# Adding user to run mogilefsd service (as done like above)
Step 7# Storage Server Configuration (as done like above)
Step 9# Run in Mogilefs DB Server:
**********************************
Add client Host
***************
mogadm --trackers=192.168.1.10:6001 host add mogilestorage1 --ip=192.168.1.10 --port=7500 --status=alive
To add device name for client host
**********************************
mogadm --trackers=192.168.1.10:6001 device add mogilestorage1 dev2
#once device name is added please create the directory In client(192.168.1.10)
192.168.1.10#mkdir /var/mogdata/dev2
****************
• A scalable, Fault tolerant, High performance distributed file system
• Asynchronous Replication
• No Single Point of Failure
• Automatic file replication (3 replications Tracker Storage Node recommended)
• Better22 than RAID Host1 Host4
• Data Compression (BZIP2)
• Flat NameSpace
• Share-Nothing Tracker Storage Node
• No RAID required
• Local filesystem agnostic Host2 Host5
• Tracker client transfer (mogilefsd)
- Replication
- Deletion
- Query
- Reaper
- Monitor
• Files are broken up and spread over the Storage Node (mogstored) HTTP and WebDAV server DBNode Storage Node
• DBNode MySQL stores the MogileFS metadata (the namespace, and which files are where) Host3 Host6• Client Library : Ruby, Perl, Java, Python, PHP…
Installation:
*************
Database creation for mogilefs: IP 192.168.2.14
************************************************
# mysql -u root -p
mysql> CREATE DATABASE mogilefs;
mysql> grant all on mogilefs.* TO ‘mogile’@'localhost’ identified by ‘secret’;
mysql> grant all on mogilefs.* TO 'mogile'@'%' identified by 'secret';
mysql> flush privileges;
mysql> quit
Step 1# Download & Install MogileFS-Server
******************************************
#wget https://github.com/mogilefs/MogileFS-Server/archive/master.zip
#unzip master
#cd MogileFS-Server
#perl Makefile.PL
#make
#make test
#make install
--------------------For Your Information--------------------------------------
If you get any errors during above (make install) process it Will probably be errors telling you that a dependent module is missing. If during the 'make test' step you get the error, "t/00-startup....DBI connect('mysql','root',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at t/lib/mogtestlib.pl line 16" it can probably be ignored – it should not be necessary for MySQL to be running
-------------------------------------------------------------------------------
Step 3# Download & Install MogileFS-Utils
*****************************************
wget https://github.com/mogilefs/MogileFS-Utils/archive/master.zip
#unzip master
#cd MogileFS-Utils-master
#perl Makefile.PL
#make
#make test
#make install
Step 4# Download & Install MogileFS-Client
******************************************
wget https://github.com/mogilefs/perl-MogileFS-Client/archive/master.zip
#unzip master
#cd perl-MogileFS-Client-master
#perl Makefile.PL
#make
#make test
#make install
Step 5# Install perl modules
****************************
perl -MCPAN -e 'install Danga::Socket'
perl -MCPAN -e 'install Gearman::Client'
perl -MCPAN -e 'install Gearman::Server'
perl -MCPAN -e 'install Gearman::Client::Async'
perl -MCPAN -e 'install Net::Netmask'
perl -MCPAN -e 'install IO::WrapTie'
perl -MCPAN -e 'install IO::AIO'
perl -MCPAN -e 'install List::MoreUtils'
perl -MCPAN -e 'install Path::Class'
perl -MCPAN -e 'install Perlbal'
perl -MCPAN -e 'install ExtUtils::MakeMaker'
perl -MCPAN -e 'install DBD::mysql'
OR
#wget http://search.cpan.org/CPAN/authors/id/M/ML/MLEHMANN/Linux-AIO-1.9.tar.gz
#cd Linux-AIO-1.9
#make
#make test
#make install
Step 6# Database Configuration
******************************
#mogdbsetup --dbhost=mogiledb.yourdomain.com --dbname=mogilefs --dbuser=mogile --dbpassword=secret
---------------------------------------For Your Information-----------------
The database is empty and will need a schema applied. A utility named 'mogdbsetup' to make this process simple. By default it assumes the database is located on localhost.
The mogdbsetup utility does not specify a table type by default so your tables will match the defaults for your database. In many cases this will mean that you end up with MyISAM tables. If you prefer InnoDB tables you will either need to make sure your database defaults to InnoDB or you can manually convert the tables (both of these are outside of the scope of this document but there are plenty of examples out there).
-----------------------------------------------------------------------------
Step 7# Tracker Configuration
*****************************
On each tracker server create a configuration file at /etc/mogilefs/mogilefsd.conf with the following:
db_dsn DBI:mysql:mogilefs:mogiledb.yourdomain.com
db_user mogile
db_pass secret
conf_port 6001
listener_jobs 5
----------------For Your Information--------------------------------
1. db_dsn points to your database instance.
2. If you are running the database on the same machine as the storage server you can omit ":mogiledb.yourdomain.com: and it will use the local machine.
3.db_user and db_pass should match the user and password you configured when setting up your database.
--------------------------------------------------------------------
Step 8# Adding user to run mogilefsd service
********************************************
The program 'mogilefsd' will not run as root so you will need to run this as a non-root user. To create a user for this, enter the following command and follow the prompts to create the "mogile" user:
# adduser mogile
Step 9# Storage Server Configuration
************************************
On each storage server, create the storage directory (make sure it has access permissions for the user you will use to run mogstored):
# mkdir /var/mogdata
Configure it:
On each storage server, create a configuration file at /etc/mogilefs/mogstored.conf with the following:
httplisten=0.0.0.0:7500
mgmtlisten=0.0.0.0:7501
docroot=/var/mogdata
Commands for Administration of MogileFS
=======================================
To add domain:
**************
mogadm --trackers=192.168.2.14:6001 domain add consim
To add class domain:
********************
mogadm --trackers=192.168.2.14:6001 class add consim classconsim
To add Host
***********
mogadm --trackers=192.168.2.14:6001 host add mogilestorage --ip=192.168.2.14 --port=7500 --status=alive
To add device name for host
***************************
mogadm --trackers=192.168.2.14:6001 device add mogilestorage dev1
To delete host :
***************
mogadm --trackers=192.168.2.14:6001 host delete mogilefstorage
To list the host:
*****************
mogadm --trackers=192.168.2.14:6001 host list
To list the device:
*****************
mogadm --trackers=192.168.2.14:6001 device list
To Check:
*********
mogadm --trackers=192.168.2.14:6001 check
To upload:
**********
mogupload --trackers=192.168.2.14:6001 --domain="consim" --key="/hello" --file="/tmp/test2.txt"
To Download:
***********
mogfetch --trackers=192.168.2.14:6001 --domain="consim" --key="/hello" --file="/home/test2f.txt"
To Delete:
**********
mogdelete --trackers=192.168.2.14:6001 --domain="consim" --key="/hello"
To know the Statistics:
***********************
for all
-------
mogstats --db_dsn="DBI:mysql:mogilefs:host=192.168.2.14" --db_user="mogile" --db_pass="secret" --verbose --stats="all"
or
for device & files only
-----------------------
mogstats --db_dsn="DBI:mysql:mogilefs:host=192.168.2.14" --db_user="mogile" --db_pass="secret" --verbose --stats="devices,files"
To know the file Info:
*********************
mogfileinfo --trackers=192.168.2.14:6001 --domain=consim --key='/hello'
To rename the key name:
***********************
Usage: /usr/local/bin/mogrename --trackers=host --domain=foo --class=bar --old_key='/hello.jpg' --new_key='/bye.jpg'
Extra Informations:
*******************
1. If you want the tracker to use memcached:
#mogadm --trackers=192.168.1.10:6001 settings set memcache_servers memchage_installed_ip:its_port
2. To use nginx as web server Refer the below
**********************************************
http://code.google.com/p/mogilefs/wiki/nginx_webdav
3. php driver to upload/download files via php coding
*****************************************************
wget http://pecl.php.net/get/mogilefs-0.9.2.tgz
4. Script to get the host/device check through mail
***************************************************
mogadm --trackers=192.168.2.14:6001,192.168.1.10:6001 check |tee /tmp/mogadm.check |grep OK && cat /tmp/mogadm.check | mail -s "mogadm check status" mdmansoor26@gmail.com
5. To upload/download files through Web Interface
**************************************************
#wget http://mogilefs-browser.googlecode.com/files/mogilefsbrowser-1.0.0b3.tar.gz
1) Create a read-only user who has SELECT privileges to your MogileFS database
Something like:
GRANT SELECT ON mogilefs.* TO 'mogilero'@'localhost' IDENTIFIED BY 'password';
2) tar xzvf mogilefsbrowser-1.0.0b3.tar.gz
2) cp mogilefsbrowser /var/www/html
3) Edit config.php and supply the settings for your Mogile Database and Mogile system
3) create & Set permissions on docs/logs.txt to be writable by the web server process
4) Open in a web browser
Client Configuration: IP: 192.168.1.10
======================================
Step 1# Download & Install MogileFS-Server (as done like above)
Step 2# Download & Install MogileFS-Utils (as done like above)
Step 3# Download & Install MogileFS-Client (as done like above)
Step 4# Install perl modules (as done like above)
Step 5# Tracker Configuration (as done like above) [in this add the ip address of MySQL DB server]
Example:
********
[root@consim tmp]# cat /etc/mogilefs/mogilefsd.conf
db_dsn DBI:mysql:mogilefs:192.168.2.14
db_user mogile
db_pass secret
conf_port 6001
listener_jobs 5
Step 6# Adding user to run mogilefsd service (as done like above)
Step 7# Storage Server Configuration (as done like above)
Step 9# Run in Mogilefs DB Server:
**********************************
Add client Host
***************
mogadm --trackers=192.168.1.10:6001 host add mogilestorage1 --ip=192.168.1.10 --port=7500 --status=alive
To add device name for client host
**********************************
mogadm --trackers=192.168.1.10:6001 device add mogilestorage1 dev2
#once device name is added please create the directory In client(192.168.1.10)
192.168.1.10#mkdir /var/mogdata/dev2
No comments:
Post a Comment