Exposing GIT over HTTP in Ubuntu

Continuing my GIT exploration here I will talk about how to expose GIT over HTTP in UBUNTU

Pre-requisites: -

GIT - GIT Core should be installed (Refer to Working with GIT)
Apache2 - Install it by running the following command: -

sudo apt-get install apache2

The above command will install the apache in the following Directories

1. /etc/apache2 - will contain all the configurations like ports, virtual hosts etc
2. /var/www - Contains the HTML files, CGI scripts etc

GITWEB

GITWEB is a package which can be used to expose GIT repositories over HTTP

It is nothing more than few set of files: -

1. gitweb.css
2. git-logo.png
3. git-favicon.png
4. gitweb.cgi
5. gitweb.js
6. gitweb.conf (Configuration File which usually is found at /etc/gitweb.conf)

sudo apt-get install gitweb will download all of the above files

Preparing Apache for Hosting GIT: -

1. Create a directory by name of "gitweb" under "/var/www".
2. Except gitweb.conf, copy all other files downloaded with GITWEB package to "/var/www/gitweb" directory.
3. Create a another folder by name of "/var/www/gitRepo" and configure your git client and also download the code from git Server (refer to Working with GIT) for all the branches.
4. Open /etc/gitweb.conf and modify the following highlighted Attributes: -

$GIT = "/usr/bin/git";
# path to git projects (.git)
# $projectroot = "/var/cache/git";
$projectroot = "/var/www/gitRepo";

# directory to use for temp files
$git_temp = "/tmp";

# target of the home link on top of all pages
#$home_link = $my_uri || "/";

# Description Text for your Repository
$site_name = "Project Trees";

# html text to include at home page
$home_text = "indextext.html";

# file with project list; by default, simply scan the projectroot dir.
$projects_list = $projectroot;

# stylesheet to use
$stylesheet = "/gitweb.css";

# logo to use
$logo = "/git-logo.png";

# the 'favicon'
$favicon = "/git-favicon.png";

5. Now execute the following Commands in "/etc/apache2/mods-enabled" which will enable the specific modules which are needed for this exercise: -

ln -s ../mods-available/dav.load dav.load
ln -s ../mods-available/dav_lock.load DAVLockDB
ln -s ../mods-available/dav_fs.load dev_fs.load
ln -s ../mods-available/dav_fs.conf dev_fs.conf
ln -s ../mods-available/cgid.load cgid.load
ln -s ../mods-available/cgid.conf cgid.conf
ln -s ../mods-available/rewrite.load rewrite.load

sudo a2enmod rewrite

6. Next create a virtual Hosts, that can expose your git repository

Create a git_vhosts.conf file in "/etc/apache2/sites-available" which will contain the details about the virtual Hosts.
The content of the file will look like this: -

#
# Virtual Hosts
#
# If you want to maintain multiple domains/hostnames on your
# machine you can setup VirtualHost containers for them. Most configurations
# use only name-based virtual hosts so the server doesn't need to worry about
# IP addresses. This is indicated by the asterisks in the directives below.
#
# Please see the documentation at
# <URL:http://httpd.apache.org/docs/2.2/vhosts/>
# for further details before you try to setup virtual hosts.
#
# You may use the command line option '-S' to verify your virtual host
# configuration.

#
# Use name-based virtual hosting.
#
NameVirtualHost *:801

#
# Almost any Apache directive may go into a VirtualHost container.
# The first VirtualHost section is used for all requests that do not
# match a ServerName or ServerAlias in any <VirtualHost> block.
#

<VirtualHost *:801>
    ServerName git.myrep.net
    DocumentRoot "/var/www/gitweb"
    DirectoryIndex gitweb.cgi
    SetEnv  GITWEB_CONFIG   /etc/gitweb.conf

    <Directory "/var/www/gitweb">
        Options FollowSymlinks ExecCGI
        Allow from all
        AllowOverride all
        Order allow,deny

        <Files gitweb.cgi>
            SetHandler cgi-script
        </Files>

        RewriteEngine on
        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteCond %{REQUEST_FILENAME} !-d
        RewriteRule ^.* /gitweb.cgi/$0 [L,PT]
    </Directory>

    <Directory "/opt/git/myrepo.git">
        Allow from all
    </Directory>

    # To debug rewrite rules, which were very painful to figure out. Create directories in case they don't exists
    RewriteLog /var/log/httpd/rewrite_log
    RewriteLogLevel 9

    ErrorLog /var/log/httpd/gitweb
</VirtualHost>

You need to focus on the highlighted text in the above configuration as these attributes are very much specific to the paths exposed on the individual boxes.
In case you have followed all the previous instructions line by line than you just need to change "ServerName" attribute, which should be the IP or Domain Name of your machine.

7. Now to enable the Virtual Host and execute the below command from "/etc/apache2/sites-enabled" directory

ln -s /etc/apache2/sites-available/git_vhosts.conf 002-git_vhost.conf

8. Lastly restart your Apache server by using the following set of Commands

sudo /etc/init.d/apache2 stop
sudo /etc/init.d/apache2 start

NOTE: Make sure it is a clean start and there are no Errors shown on console.

9. Now browse the URL (http://:801/gitweb.cgi?p=.git;a=tree) and you will be able to see the GIT repository.

The window which Appears would be something like the below snapshot

Next I leave to you to play around this UI.

Updating Git repository: -

As we are exposing client repository over HTTP, so naturally the question arises: -

How do you make sure that your repository is continuously updated and doesn't require any manual intervention?

There are 2 ways which can be used to do this: -

1. Create a Cron JOB in Ubuntu which updates the repository at every fixed interval
2. enable the "post-update Hook" of your GIT repository and from there execute a Shell Script which does the job for you.

As both the above the options requires a Shell Script which is capable to logging remotely and updating the repository through various SSH and GIT commands.

Normal Shell scripts are nowhere capable to doing this thing....but thanks to Linux which always have a solution to any problem.

"Expect" is a wonderful utility which helped me to develop a short and simple script which does the needful.

sudo apt-get install expect

Here is actual Script which I created and used with #2 to keep my repository in Sync with each and every commit: -

#!/bin/bash

/usr/bin/expect << EOD

spawn  ssh <username>@localhost

expect "password:"

send -- "<password>\r"

expect "$"

send -- "cd /var/www/gitRepo \r"

expect "$"

send -- "unset GIT_DIR \r"

expect "$"

send -- "git pull ssh://<username>@<IP>/<PATH> <BRANCH> \r"

expect "password:"

send -- "<password>\r"

expect "$"

expect eof

EOD

echo "rep updated"

Read More!

Configuring and Working with GIT in Ubuntu

GIT - A new name in the Arena of SCM's

Though we already have many SCM's available (Perforce,CVS, SVN, VSS etc), than why GIT?

Actually GIT offers much more than just a simple repository.

Apart from all the other standard SCM features, there are few which I liked the most: -

1. Every developer (by default) has its own private repository, which can be synced with the mainstream at periodical intervals.
2. Already Hosted over the internet (GIT-HUB) and anybody can use it, which itself makes the source code sharing such an easy task. Seems like a Social Networking of code :)
3. Local Branching is easy and Cheaper now, which was a pretty heavy and almost difficult in other SCM's
4. Faster than any other SCM's.
5. Over the Network - Works very well and far better than any other SCM, doesn't eat my network resources for each commit and checkouts are also very quick.


Visit the following liks to know more benefits of using GIT over other SCM
https://git.wiki.kernel.org/index.php/GitSvnComparison
http://whygitisbetterthanx.com/


Installing GIT: -

login as root or use sudo to run the below command

apt-get install git-core

Setting up Private repository: -

create and browse the directory which will be used as your local repository
<Path>/<code-dir>

run the following commands

git init

The above command initialize the empty GIT repository

Then clone the empty git repository to create a MATSER REPOSITORY (thats the default Branch for GIT empty repo)

git clone --bare /<Path>/<code-dir>/.git /<PATH>/myrepo.git

myrepo.git - is a Directory which will contain your master repository. All other development folks will be referring to this directory when checking out the code

That's it!!!!You have just completed the client setup and your repository can be used by dev folks

Setting up client or local repository

create and browse the directory which will be used as your local repository
<Path>/<code-dir>

run the following commands

git init

The above command initialize the empty GIT repository

git config user.email <email id>

The above will configure the repository to use the specified the email id for each commit

git remote add origin ssh://<serverIP/ domain-Name>/<Path of your GIT repo>

e.g. in our case it would be something like: -

git remote add origin ssh://<serverIP/ domain-Name>/<PATH>/myrepo.git

The above adds a master server to your repository..This would be used while you are trying to sync your local repository with main repository.

HURRAY----We are done with client too...so easy isn't it

Lets try a simple commit: -

Browse your local directory where you have set up your GIT working repository and create a file by by name "first.txt" and write something in it

Now execute the following commands from the console

git add *

Above will add all the changed or untracked files to your local repository

git commit -m "my First Commit"

Above will commit the code to your local repository

git push -u origin master

Now this will sync your local repo with the master...i.e. you just ready to upload all your changes to the master repository.
As soon as you execute the above command, it will ask for the password and here is what exactly it is doing: -

GIT mainly works on 2 protocols GIT and SSH and while doing an Sync it needs to login to the remote box and connect to the repository.

In case of ssh it asks for the password of the current user (by which you are logged into your local box), assuming that same user is already setup on remote box and have SSH privileges and also have access to git repo.

So make sure that the username (same as your local login) is created on the remote server.

Though it is not necessary that the SSH login of remote box matches your local login (you can use a different username while Syncing your changes) but just to avoid any unnecessary complexities I would have always suggested to use the same Logins.
Also while working on the real networks you will always use your network login, which anyways will be same on the remote box and local box.

Frequently used commands: -

Below is the list of GIT commands which are commonly used in our day to day life

git

Simply shows the available commands and their usage

git add "list of files to be added"

Adds the List of Files to your local repo

git commit –m "comments"

Commits the files to your local repo

git push -u origin "Branch Name"

Pushes the code to upstream branch and now other developers can checkout

git fetch ssh://<username>@&server-IP or domain-Name>/<Path of the repo on remote box>.git <branch-name>

Used to fetch the branches from the upstream branches to your local repo

git pull ssh://<username>@&server-IP or domain-Name>/<Path of the repo on remote box>.git <branch-name>

Same as Fetch but the only difference is that it tries to merge into the current branch

git checkout <branch-name>

By default every user is on "master" branch. The above command will move him to the specified Branch

git log

Shows the History of the branch on which the person is working.

git status -s

Shows the status of your local repo. Status of all committed or un-commited files

git rm -r >dir>

Used to remove the specified directory from your local repo

git reset --hard <branch-name>

Loose all your local changes and make your code in Sync with the upstream Branch.

Get more insight of GIT: -

http://book.git-scm.com
http://git-scm.com
http://gitref.org/

Read More!

Knowing your System in Linux

Recently I was in a situation where it was required to find out the software/ application and hardware details for my ubuntu10.10 and to my surprise there was no single place where i could find the list of commands which can give me complete information.

It might be because of my short experience with linux but I am sure that lot of people (like me :)) do a googling to find out the correct information.

So thought of compiling all information at one place: -



lshw - list Complete Hardware information about your system

Logging in to root and just typing "lshw" will provide the complete information about the Hardware of your system.
To get more Specific info use the following Commands

lshw -class memory
lshw -class cpu
lshw -class disk
lshw -short


cat /proc/*

Another powerful command to get the detailed information about the system

cat /proc/cpuinfo - Provides info about your CPU
cat /proc/meminfo - provides info about your Memory

sginfo - access mode page information for a SCSI (or ATAPI) device

In case this command is not available than install sg3-utils "apt-get install sg3-utils"

All information around the various arguments for this command can be found here

hdprm - Another prowerfull command for not only getting your Drive information but also can be used to tune your Drive

hdparm /dev/hda - gives the information about the drive
hdparm -Tt /dev/hda - used for getting the read and write speed of yur drive


Some more useful monitoring commands: -

iostat -c -k -p -d 2

What does it means: -

–c = Display the CPU utilization report.
–d = Display the device utilization report
–k = Displays utilization in KB/sec (use –m per sec)
–p = Displays Stats for all devices and partitions
–d = device utilization report
2 = displays the same report after every 2 seconds

mpstat -P ALL 2 5

The above command will provide the 5 reports for "average CPU utilization" for all the available processors in interval of 2 seconds

Total number of CPU's

grep core\ id /proc/cpuinfo | grep -c \ 0$ | grep ^0$ >> /dev/null && grep -c processor /proc/cpuinfo || grep core\ id /proc/cpuinfo | grep -c \ 0$

Sockets/ FD (File Descriptors)

There are n number of of powerful commands which can give the full details about the Sockets or FD's open in my Linux Box. Below are few of them

#Maximum number of open files:
cat /proc/sys/fs/file-max

#Allocated, free allocated, and max FDs:
cat /proc/sys/fs/file-nr

#Max threads:
cat /proc/sys/kernel/threads-max

#Max half-open TCP:
cat /proc/sys/net/core/somaxconn

#Upper limit for the number of sockets which can be opened by all the processes
ulimit -a

#Modify the below configuration file to impose the max number of sockets for a user
/etc/security/limits.conf

#Count of total FD's open by a process
cat /proc/<pid>/fd/ | wc -l
lsof | grep <pid> | wc -l

#Details about all FD's open by a process
lsof | grep <pid>

Read More!