Skip to main content

Posts

Showing posts from 2013

Chrome Custom URL Redirector Extension

Working as a IT professional sometimes you need to access remote systems using vpn connection. Throughout the vpn connection dns records and hosts records may be overwritten and control panel url addresses are inaccessible. To overcome this you need to use custom browser extension like greasemonkey, tempermonkey etc.
But i don't want to use these third party extensions so i wrote a simple chrome plugin which translates url addresses to ip addresses.
You need two files, manifest.json and background.js
Examples of these files are(add urls and ip addresses to urlDict variable as required!):
manifest.json: {   "name": "Custom URL Redirector",   "version": "0.1",   "description": "Checks URL and redirects as required.",   "manifest_version": 2,   "background": {     "scripts": ["background.js"]   },   "permissions":["http://*/*","https://*/*","webReques…

How to create hive external table for nutch's hbase webpage schema?

In order to query hbase table using hive, an external table should be created.
CREATE EXTERNAL TABLE webpage_hive (key string, baseUrl string, status int, prevFetchTime bigint, fetchTime bigint, fetchInterval bigint, retriesSinceFetch int, reprUrl string, content string, contentType string, protocolStatus string, modifiedTime bigint, prevModifiedTime bigint, batchId string, title string, text string, parseStatus int, signature string, prevSignature string, score int, headers map<string,string>, inlinks map<string,string>, outlinks map<string,string>, metadata map<string,string>, markers map<string,string>) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,f:bas,f:st,f:pts#b,f:ts#b,f:fi#b,f:rsf,f:rpr,f:cnt,f:typ,f:prot,f:mod#b,f:pmod#b,f:bid,p:t,p:c,p:st,p:sig,p:psig,s:s,h:,il:,ol:,mtdt:,mk:") TBLPROPERTIES ("hbase.table.name" = "webpage");
after…

Howto disable IPv6 Stack Ubuntu Linux

Sometimes network infrastructures are misconfigured to use ipv6 stack. Then you want to disable it at all. For ubuntu distribution it is easy as changing sysctl.conf and reboot your machine. Here is how to do it.

First check whether it is enabled or not:
$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
if output is 0 it is enabled and 1 it is disabled.

Adding the following rows is enough to disable ipv6 stack then restart your computer to take effect:
# vi /etc/sysctl.conf
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.all.disable_ipv6 = 1
# shutdown -r now


When your computer is up again, you can check it:
$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6


Pig Latin Error 2998 when loading data from HBaseStorage

ERROR 2998: Unhandled internal error. org/apache/hadoop/hbase/filter/WritableByteArrayComparable
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/filter/WritableByteArrayComparable
Probably your PIG_CLASSPATH does not contain hbase jars. PIG_CLASSPATH should contain them like:
export PIG_CLASSPATH=/<hbase_home>/<hbase-your_version.jar>:<hbase_home>/lib/<zookeeper-your_version.jar>:$PIG_CLASSPATH

Hive exception when executing a query: java.lang.Integer cant be mapped for this datastore

When executing query in the hive shell following error is shown.
hive> show tables; FAILED: Error in metadata: MetaException(message:Got exception: org.apache.hadoop.hive.metastore.api.MetaException javax.jdo.JDOFatalInternalException: JDBC type integer declared for field "org.apache.hadoop.hive.metastore.model.MTable.createTime" of java type java.lang.Integer cant be mapped for this datastore.
NestedThrowables: org.datanucleus.exceptions.NucleusException: JDBC type integer declared for field "org.apache.hadoop.hive.metastore.model.MTable.createTime" of java type java.lang.Integer cant be mapped for this datastore.) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
That is because of your language settings. To solve the problem export lang variable before launching hive. # export LANG=C

Saltstack as a Configuration Management Software

Salt is an infrastructure management system and takes a new approach for developing system administration software that is to get running in minutes, scalable to tens of thousands of servers. The Salt open source project was started in 2011. Salt provides a dynamic infrastructure communication bus used for orchestration, remote execution, configuration management. SaltStack is the company responsible for keeping the Salt project healthy and dynamic. Salt remains 100% open.
Installing Salt on RHEL/Centos Systems:
From the 0.9.4 release Salt has been available in EPEL repos. To enable EPEL: # rpm -Uvh http://ftp.linux.ncsu.edu/pub/epel/6/i386/epel-release-6-8.noarch.rpm
Salt is divided into two packages. Minion and the master. It is necessary only to install the suitable package for the role the machine will play. In general, there is one master and multiple minions.
On the master, install salt-master: # yum install salt-master 
On each slave node, install salt-minion: # yum install salt-…

How to install flash player on fedora linux 18

To install flash player on fedora linux 18 operating system:

32-bit installation:
# rpm -ihv http://linuxdownload.adobe.com/adobe-release/adobe-release-i386-1.0-1.noarch.rpm
# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux

Hadoop Summit 2013

Events at Hadoop Summit Tuesday, June 25 Birds of a Feather Sessions 10:00 AM – 4:00 PM

Pig   10:00am – 12:30pm
YARN   10:00am – 12:30pm
Accumulo   10:00am – 12:30pm
Verticals   10:00am – 12:30pm
Oozie   11:00am – 12:30pm
Hive   1:30pm – 4:00pm
HBase   1:30pm – 4:00pm
Ambari   1:30pm – 4:00pm
Big Data Science   1:30pm – 4:00pm

Open to all registrants of Hadoop Summit.

HBase is able to connect to ZooKeeper but the connection closes immediately

I got following error when i tried to connect hbase from hadoop nodes. Zookeeper servers runs normally but hadoop nodes can not connect remote zookeeper cluster.

ERROR: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.

Then i fixed problem copying hbase-site.xml to hadoop/conf directory.

JUDCon2013: UnitedStates

When: June 9-11, 2013
Where:Hynes Convention Center, Boston, Massachusetts
Why: Because developers need to get their hands dirty in code. Early registration is open, and you can save 40% if you register now! The ninth JUDCon - JUDCon2013: UnitedStates - will be a 3-day, 3-track extravaganza including a welcome reception, 3 tracks of valuable sessions, with 2 additional workshop tracks. And this year, one low price will get you access to CamelOne and Red Hat Exchange as well! JUDCon sessions will include: Building HTML5 applications, the JBoss Way
Peter Muir59 mins to (geo)satellite search in your Hibernate apps... 58 mins...
Emmanuel BernardRe-thinking Mobile/Web Applications, Getting it Done, Keeping it That Way
Lincoln Baxter, IIIWhat's new in Drools 6.0
Mark Proctor The activities start Sunday evening with a JUDCon, CamelOne and Red Hat Exchange reception. Then Monday and Tuesday, there will be 3 tracks of sessions and 2 workshop tracks. During dinner Monday evening, the JBoss Communi…

How to check hadoop health from commandline

from the hadoop home directory:
# ./bin/hadoop fsck /
FSCK started by root from /10.1.1.2 for path / at Mon Apr 15 14:16:20 EEST 2013 .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ..............................................................Status: HEALTHY Total size:                               6907330009 B Total dirs:               …

How to check Hbase health from commandline?

from the hbase home directory: # ./bin/hbase hbck
Version: 0.90.5
Number of Tables: 1 Number of live region servers: 3 Number of dead region servers: 0 Number of empty REGIONINFO_QUALIFIER rows in .META.: 0 Summary:   -ROOT- is okay.     Number of regions: 1     Deployed on: server1.internal:60020   .META. is okay.     Number of regions: 1     Deployed on: server2.internal:60020   webpage is okay.     Number of regions: 41     Deployed on: server3.internal:60020 server2.internal:60020 server1.internal:60020 0 inconsistencies detected. Status: OK

Open Source Load Balancing Solutions

There are plenty of open source tcp load balancing software.

HAproxy
site: http://haproxy.1wt.eu/

Linux Virtual Server
site: http://www.LinuxVirtualServer.org/

Linux Network Load Balancing
site: http://lnlb.sourceforge.net/

Distributor Load Balancer
site: http://distributor.sourceforge.net/

Balance
site:http://www.inlab.de/balance.html

Load Balancer Project
site: http://www.jmcresearch.com/projects/loadbalancer/

Pen
site: http://siag.nu/pen

Zen Load Balancer
site: http://zenloadbalancer.sourceforge.net/

Crossroads
site: http://crossroads.e-tunity.com/

Core Balance
http://core-balance.sourceforge.net/

Open Source System Logging Solutions

For large datacenters collecting of server logs is headache. There are some opensource solutions for this. Graylog2, logix, logstash, Facebook Scribe are popular ones.


Graylog2's first release is June 2010. Standart syslog pakhages are sent to the central log server. There is web tool for analyzing and searching for logs. Logs are stored in MongoDb database.
logix can be considered as a log sending tool for graylog. In standard ways, some logs may be lost while sending over network. Logix takes logs in a queue then send them to log server.
Logstash is a popular solution like graylog. Also there is a web interface for searching logs. Logstash can store logs in elasticsearch.
Scribe is written by facebook using C++ server. Facebook collects billions of logs from thousands of its servers.


Network performance testing in Linux

IPERF is a network speed testing tool. It is reliable if you have two servers, and you want to measure network speed between them.

It can be installed using yum
# yum install iperf

For testing, one iperf instance runs as a server on one server and another iperf instance runs as a client on the other server.

For running iperf as a server
# iperf -s
Server listening on TCP port 5001


And for running iperf as a client
# iperf -c 192.168.0.11
Client connecting to 192.168.0.11, TCP port 5001 TCP window size: 16.0 KByte (default) 

------------------------------------------------------------ 
[3] local 192.168.0.12 port 47326 connected with 192.168.0.11 port 5001 
[ID] Interval Transfer Bandwidth 
[3] 0.0-10.0 sec 267 MBytes 228 Mbits/sec

What is OpenID

OpenID is an open standard that allows users to be authenticated by certain co-operating sites using a third party service, eliminating the need for webmasters to provide their own ad hoc systems and allowing users to consolidate their digital identities.

Users may create accounts with their preferred OpenID identity providers, and then use those accounts as the basis for signing on to any website which accepts OpenID authentication.

Toad for Cloud Databases

Toad for Cloud Databases supplies SQL-based interface that makes it easier than ever for generating queries, migrating, browsing, and editing data, as well as create reports and tables in a common SQL view. It’s easy to load and export data from cloud databases to external data sources.
Puts NoSQL databases within your reach even if you’re an RDBMS developer or DBA Supports business decision making with fast access to cloud data Supports ANSI-standard SQL issued against cloud datasets Speeds accurate query generation through the query builder Lets you browse data and edit in-place Helps you create “tables” or their logical equivalent Supports DML SQL (UPDATE, INSERT, DELETE statements) Loads data from and exports data to a variety of external data sources, including RDBMS Provides assistance for deploying and managing Amazon EC2 instances

Linux bash string manipulation

Length of a string${#str} str="abcdefg" echo ${#str} #gives output 7
Substring extraction ${str:pos} extracts substring from $str at $pos str="abcdefg" echo ${str:2} #gives output cdefg
${str:pos:len} extracts substring from $str at $pos length of $len str="abcdefg" echo ${str:2:3} #gives output cde
Substring Removal${str#substr} removes shortest match of $substr from front of $str str="abcdefgabcdefg" echo ${str#a*c} #gives output defgabcdefg
${str##substr} removes longest match of $substr from front of $str str="abcdefgabcdefg" echo ${str##a*c} #gives output defg
${str%substr} removes shortest match of $substr from back of $str str="abcdefgabcdefg" echo ${str%b*g} #gives output abcdefga
${str%%substr} removes longest match of $substr from back of $str str="abcdefgabcdefg" echo ${str%%b*g} #gives output a

Redhat's Openshift Cloud QuickStart

Redhat's Openshift Cloud QuickStart
1. Sign Up for openshift.redhat.com 2. Install the Client Tools: $ sudo gem install rhc 3. Create a Namespace: $ rhc domain create -n <namespace> -l <login> 4. Create an Application: $ rhc app create -a <app name> -t <type> 5. Go to: http://<app name>-<namespace>.rhcloud.com

libXext.so problem when installing oracle

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-01-22_10-21-26AM. Please wait ...[oracle@orcltest database]$ Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/OraInstall2013-01-22_10-21-26AM/jdk/jre/lib/amd64/xawt/libmawt.so: libXext.so.6: cannot open shared object file: No such file or directory
at java.lang.ClassLoader$NativeLibrary.load(Native Method) at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1753) at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1649) at java.lang.Runtime.load0(Runtime.java:769)

This may be encountered when installing oracle database on a redhat linux server. That's because libXp package was not installed on the RHEL5/6 system. Just install the libXp packege.

SOLR4 how to abort running dataimport command

First you can get current dataimport status with the following url in a multicore environment: http://<solr4_server_ip>:<port_number>/solr/<core_name>/dataimport?command=status
Probably you may have seen the following result: <str name="importResponse">A command is still running...</str>
Then this command can be aborted with the following url: http://<solr4_server_ip>:<port_number>/solr/<core_name>/dataimport?command=abort

bash script include config file

With source command static variables may be included from the main bash script like:
source /etc/script.config

And content of config file may like:
server_ip="5.5.5.5"
app_version=3.5
app_list=('app1' 'app2' 'app3')