I needed to find the different file extensions inside a folder in Debian Linux. I used the following command to do it.
find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u
I found this code in this link.
I recently received a blob of data which needed some alteration to properly suit the CSV format. I found that the data file ‘xyz.dat’ contained close to 600,000 rows so writing a VBA script in Excel to format the data was not an option. I opted to do it using some plain commands in Unix. I had to attain the following purpose:
1. Remove all rows with blank lines.
2. Remove all rows that didn’t start with the number ‘1’
3. Trim the file to 1000 rows to sample the data.
I executed the following commands to attain the above objectives.
To remove all blank lines I used this command. I am basically asking grep to treat the text as ASCII text and using a regular expression to identify blank rows:
cat xyz.dat | grep -a -v -e '^[[:space:]]*$' > xyz_no_space.dat
To remove all rows that don’t start with ‘1’ I used this command. This command is similar to above command except for the regular expression to identify rows that start with ‘1’.
cat xyz_no_space.dat | grep -a '^1' > xyz_no_space_start_with_1.dat
At this point we have the data we want but for sampling purpose I need to copy first 1000 rows into another file. This command uses ‘sed’ to do it.
sed -n -e '1,1000p' xyz_no_space_start_with_1.dat > xyz_trimmed_1_1000.dat
This command basically renames the *.dat file into *.csv for convenience.
mv xyz_trimmed_1_1000.dat xyz_trimmed_1_1000.csv
A summary of all commands I executed is listed below:
cat xyz.dat | grep -a -v -e '^[[:space:]]*$' > xyz_no_space.dat
cat xyz_no_space.dat | grep -a '^1' > xyz_no_space_start_with_1.dat
sed -n -e '1,1000p' xyz_no_space_start_with_1.dat > xyz_trimmed_1_1000.dat
mv xyz_trimmed_1_1000.dat xyz_trimmed_1_1000.csv
I wanted to try out Tomcat 8 and Java 8 for one of my production servers. The following link summarizes the various steps required for installing and configuring both Java 8 and Tomcat 8 from scratch.
I use Maven for almost all Java projects I work in. I prefer using Netbeans with Maven as it has solid integration with Maven and I find it a hassle free approach of working with Maven without cluttering up your project with ‘special’ project files. However I have found out that Eclipse cannot be ignored as it has much faster performance and it needs lesser memory than Netbeans. So here are some tricks to work around the build issues I have faced with Eclipse when combined by Maven. The usual issue I faced was that I could build my project from console but I saw lots of build issues when building from inside Eclipse and lot of build errors were logged inside Eclipse ‘Problem’ view.
Refresh Project On pom.xml change
If you have changed anything in the pom.xml you should refresh your Eclipse project. It is possible Eclipse will pick up your changes from pom.xml and resolve any build / dependency issue in this step itself.
Re-generate Eclipse Project Files
If you have added a dependency in pom.xml you should ensure that the following commands are executed to regenerate the Eclipse project configuration files.
mvn eclipse:clean
This will delete the .project, .classpath and .settings folder of Eclipse.
mvn eclipse:eclipse
This will regenerate the .project, .classpath and .settings folder for your project.
Once you execute the above commands go back to your Eclipse project and refresh your project by right clicking on the project and clicking on refresh in the context menu or by pressing the ‘F5’ key after selecting the project.
In my experience so far Eclipse is able to resolve all dependence / build issue and you should have a trouble free coding experience now.
The Jenkins CI server can be integrated with BitBucket by using the concept of deployment keys. In BitBucket go to Settings > Deployment Keys. Add a new public key. Make sure you are adding an openssh compatible public key otherwise the key won’t be accepted. Once this is done you need to test the read-only cloning capability using this approach.
ssh-agent (ssh-add /home/user1/ssh_keys/my_private_key; git clone git@bitbucket.org:my_user/my_project.git)
This command basically attempts to make a clone from the BitBucket server using the private key you specified as ‘my_private_key’. Since we are using the ssh-agent command to wrap the other two commands the key is loaded, the clone command is executed and then immediately the key is unloaded. If your cloning process failed you know that something is wrong. Remember you need the private key in openssh format as well.
If the above command gave you the cloned repository, go ahead and use this private key inside the Jenkins server.
Please do refer to BitBucket documentation link has more details on the exact steps.
I was trying to setup Jenkins web app inside the Tomcat server and it continuously gave me this error:
SEVERE: Failed to initialize Jenkins
hudson.util.NoHomeDir
This I fixed by creating the folder ‘/usr/share/tomcat7/.jenkins’
mkdir /usr/share/tomcat7/.jenkins
On redeploying Jenkins I again noticed the following error.
WARNING: Failed to record boot attempts
java.io.FileNotFoundException: /usr/share/tomcat7/.jenkins/failed-boot-attempts.txt (Permission denied)
I fixed this issue by giving full access to this folder.
sudo chmod 777 /usr/share/tomcat7/.jenkins
I believe 777 is not a good approach in a production environment, but for development perspective it looks ok.
I recently had to find out if a particular package was properly installed in my Debian Linux OS as something was broken. I used the following command to do that.
dpkg --get-selections | grep -v deinstall | grep oracle
This basically lists out all the packages which contain the text ‘oracle’ in them. Once I find out the exact package name I can actually decide to remove all packages containing them.
sudo apt-get purge oracle*
sudo apt-get autoremove oracle*
I was working with a scheduler and wanted to keep track of my server time. Continuous invocation of date command wouldn’t cut it. So a little search on net lead me to this link which basically sums up the following script. This script will show the current time on the top left corner of your terminal which is good enough for my usecase.
while sleep 1;do tput sc;tput cup 0 $(($(tput cols)-11));echo -e "\e[31m`date +%r`\e[39m";tput rc;done &
I executed this code on a Cent OS system and it worked fine. I have tested this snippet in Debian as well and it worked like a charm. This won’t solve everybody’s problem but for specific situation where you just want to watch the clock of your server from a shell this is a good enough script.
I have to work with Windows, Linux and Macbook environment and with time I have no issues working with all of these environments simultaneously. What used to irk me the most was the need to physically use multiple keyboards and mouse and control my Windows laptop and Macbook. Doing VNC / Remote Desktop into another computer didn’t cut it. So I found out about Synergy project long time ago. This is a free and open source keyboard/mouse sharing solution.
Basically the computer whose keyboard/mouse you want to use becomes the Synergy server and all other laptops / computer would become Synergy client. It is very easy to setup and once it is done I can use the keyboard / mouse of of my Macbook to control my windows laptop as well as Linux laptop simultaneously.
We can use the jps command that is part of JDK to find out which Java processes are running in the current user’s session. We can further find out which network ports have been opened by a particular java process.
Use the follwing command to list all java processes:
jps -l
If you want more details about a process you can execute the following command:
jps -v | findstr <process id>
The following variant of netstat command lists all the listening as well as established socket connection in a system:
netstat -ano
This command can be further used to find out what sockets are opened by a specific process
netstat -ano | findstr <process id>