Tag: Tutorials
How to Implement Chef roles using Chef server ?
What is Role?
A role is a way to define certain patterns and processes that exist across nodes in an organization as belonging to a single job function. Each role consists of zero (or more) attributes and a run-list. Each node can have zero (or more) roles assigned to it. When a role is run against a node, the configuration details of that node are compared against the attributes of the role, and then the contents of that role’s run-list are applied to the node’s configuration details. When a chef-client runs, it merges its own attributes and run-lists with those contained within each assigned role.
How to use Roles in Chef?
- Create a Role and add the cookbooks into it.
- Assign the role into each node or bootstrap new nodes using roles
- The the run list
How to create Role?
Method 1: In Chef Server directly
> knife role create client1
&
Add the run list e.g. “recipe[nginx]” under “run_list”
Save & exit
The role will be created in Chef Server.
Example
{ "name": "client1", "description": "", "json_class": "Chef::Role", "default_attributes": { }, "override_attributes": { }, "chef_type": "role", "run_list": [ "recipe[nginx]", "recipe[phpapp::web]" ], "env_run_lists": { } }
Let’s download the role from the Chef server so we have it locally in a Chef repository.
> knife role show client1 -d -Fjson > roles/client1.json
Now, Lets bootstrap the node using knife with roles
> knife bootstrap --run-list "role[client1]" --sudo hostname
How to edit the roles in chef Server?
> knife role edit client1
Method 2: In local repo under chef-repo folder
> vi webserver.rb
example –
name "web_server" description "Role for web servers" run_list( "role[base]", "recipe{web_server]" )
& Then upload to chef server using following commands.
> knife role from file path/to/role/file
How Assigning Roles to Nodes?
> knife node list
> knife node edit node_name
This will bring up the node’s definition file, which will allow us to add a role to its run_list:
{ "name": "client1", "chef_environment": "_default", "normal": { "tags": [ ] }, "run_list": [ "recipe[nginx]" ] }
For instance, we can replace our recipe with our role in this file:
{ "name": "client1", "chef_environment": "_default", "normal": { "tags": [ ] }, "run_list": [ "role[web_server]" ] }
How to bootstrap node using role?
> knife bootstrap {{address}} --ssh-user {{user}} --ssh-password '{{password}}' --sudo --use-sudo-password --node-name node1 --run-list 'role[production]'
> knife bootstrap --run-list "role[phpapp-web]" --sudo hostname
How to run roles against nodes?
You can run chef-client on multiple nodes via knife ssh command like, To query for all nodes that have the webserver role and then use SSH to run the command sudo chef-client, enter:
> knife ssh "role:webserver" "sudo chef-client"
To find the uptime of all of web servers running Ubuntu on the Amazon EC2 platform, enter:
> knife ssh "role:web" "uptime" -x ubuntu -a ec2.public_hostname
Reference
http://docs.chef.io/roles.html
https://docs.chef.io/knife_ssh.html
https://docs.chef.io/knife_role.html
Jenkins Backup and Restore Process | Jenkins Tutorial
How to attach the jenkins build log as part of the email body?
Way 1 –
Put following lines as part of Default Content text box
${BUILD_LOG, maxLines=9999, escapeHtml=false}
this works for free style projects
Way 2 – If you want send only specific line using regex
${BUILD_LOG_REGEX, regex=”\\b(Buildfile|BUILD)\\b”, linesAfter=1}
${BUILD_LOG_REGEX, regex=”^.*?BUILD FAILED.*?$”, linesBefore=0, linesAfter=10, maxMatches=5, showTruncatedLines=false, escapeHtml=true}
${BUILD_LOG_REGEX, regex=”\\b(FC0)\\b”, linesAfter=1}
${BUILD_LOG_REGEX, regex=”^.*?BUILD FAILED.*?$”, linesBefore=0, linesAfter=10, maxMatches=5, showTruncatedLines=false, escapeHtml=true}
${BUILD_LOG_REGEX, regex=”^.*?FC0.*?$”, maxMatches=95, showTruncatedLines=false, escapeHtml=false, matchedLineHtmlStyle=true}
This will show exactly which is expected lines as part of the regEx.
More Info
http://wiki.hudson-ci.org/display/HUDSON/Email-ext+plugin
http://champasoft.blogspot.nl/2012/11/jenkins-pattern-for-e-mailing-log.html
DevOps Courses and Training | DevOps Training | Online | Classroom
In this course, we look at the necessity of Devops and how a DevOps transformation can help focus on value and streamlined delivery of your software. We will also cover concepts like Automation, tools & technology which play huge roles in DevOps success;
- Understand the need for DevOps and the problems it resolves.
- Learn about the common Infrastructure Servers, Scalability and Availability
- Implement Automated Installations and Deployments
- Understand Performance and basic Security for Infrastructure
- Implement Virtualization Concepts
- Understand the need and concepts of Monitoring and Logging
- Understand the Continuous Integration and Deployment (CI/CD)
- Learn various DevOps tools Chef, Puppets, Jenkins, Nagios, Docker, GIT, etc
- DevOps Engineer
- Build and Release Engineer,
- AppOps Engineer,
- Site Reliability Engineer
- System Administrator
- Operations Engineer
- Automation Engineer
- DevOps Engineer
- Build and Release Engineer,
- AppOps Engineer,
- Site Reliability Engineer
- System Administrator
- Operations Engineer
- Automation Engineer
- Project Managers,
- Testing Professionals,
- Software Developers and Architects,
- Basic understanding of linux/unix system concepts
- Familiarity with Command Line Interface (CLI)
- Familiarity with a Text Editor
- Experience with managing systems/applications/infrastructure or with deployments/automation
- info@scmGalaxy.com
- +91 700 483 5930
- skype – scmGalaxy
We do offer classroom training and workshop in Bangalore, Hyderabad, Pune, Mumbai and New Delhi. For more details, please send us an email to info@scmGalaxy.com
Also, you can follow this url Click Here
With Lifetime Enrollment
INR 25K
- Technical benefits: Continuous software delivery
- Technical benefits: Less complex problems to fix
- Technical benefits: Faster resolution of problems
- Business benefits: Faster delivery of features
- Business benefits: More stable operating environments
- Business benefits: More time available to add value (rather than fix/maintain)
- 30 Hours instructor led online class
- Hands on Approach – We emphasize on learning by doing.
- Life time free re-enrollment to future DevOps courses
- Life time free access to all learning materials including
- Class recordings
- Presentations
- Sample Code
- Projects
- Total Lab Infrasture in cloud and 24×7 available
- 70% of the class is consist of Lab
- Each week assignments (total 4) with personal assistance
- Two real time senario based projects with standard evaluation
- 24×7 online support to queries during and after the course completion
- 1 dedicated class for Interview preparations
- Online Quizs for each tool
- Lifetime Free access to Our Learning Portal for FreeVideos, Scripts Collection, Quiz, Interview Guide, Projects, Tutorials etc.
- Two Courses One Fee – DevOps and Build & Release which includes Chef and Puppet courses are together is being offered to our students.
- Life time Enrollment – Once you enroll, its life time enrollment. That means you can attend any number of session, Any Batch, Any time without paying another time for DevOps, Build & Release, Chef and Puppet. That means all courses, only one fees for life time.
Agenda of the training: DevOps Concept and Process
- Continous Integration and Delivery Process
- Problem That DevOps Solves
- Explore the DevOps background, approach, and best practices
- Integrate test automation with DevOps
- Implement continuous testing
- Learn how DevOps practices and principles improve software quality and efficiency
- Understand the differences between DevOps and traditional operational methodologies
- Discover the major steps required to successfully implement delivery pipelines
- DevOps Transition to a Project
- Understanding the Continuous Integration and Deployment (CI/CD)
- Implement DevOps – Organization & Culture
Agenda of the training: Tools and Technologies
DevOps Concept, Foundation, Culture, Best Practices and Learning
- Problem That DevOps Solves
- Implement DevOps – Organization & Culture
- Continous Integration and Delivery Process
- Explore the DevOps background, approach, and best practices
- Integrate test automation with DevOps
- Implement continuous testing
- Discover the major steps required to successfully implement delivery pipelines
- DevOps Transition to a Project
Operating Systems
Windows
Linux (CentOs & Ubantu)
Version Control Management Tools Using
Git
Build & Lifecycle Management Tools Using
Maven
MSBUILD
Gradle
World of Application Packaging Tools with
Nuget
YUM
APT
Virtualization and Container with
Docker
Vagrant
Static Code Analysis using
SonarQube
Test Coverage using
Jacoco
Cobertura
Incident and Task Management System
Jira
Configuration Management, Automation & Deployment Tools Using
Chef
Puppet
Continuous Integration & Delivery Using
Jenkins
Understanding the Cloud Infrastructure with
Amazon AWS EC2
Server Monitoring Tools with
Nagios
Programming and Scripting with
Bash Scripting
Ruby Fundamental
Shell Scripting (Bash) Training | Bash/Shell Scripting Course
Introduction
- Introduction
- Why Shell Scripting?
- Linux Internal
- What to Expect From This Course?
- Prerequisites
Git fundamental
- Introduction of git
- Git setup
- Basic operations in git
- Github
- Summary
A First Look At Shell Scripts
- Introduction
- Creating A Shell Script
- Demo: A Note-Taking Script
- Calling The Script
- The Shebang
- Naming Your Script
- Demo: The type Command
- Summary
Variables
- Introduction
- Demo: Variables
- Using Variables in A Script
- Using Variables: Good Habits
- Reading Input
- Debugging your Script
- Summary
If, Then, Else
- Introduction
- Demo: The If Statement
- The If Statement
- Return codes
- The Conditional Expression
- Demo: The Conditional Expression
- The Conditional Expression 2
- Arithmetic Tests
- Demo: Arithmetic Tests
- The If Statement Revisited
- And, Or, Not
- Summary
Input and Output
- Introduction
- Output: echo and printf
- Input: read revisited
- Standard Streams and Redirection
- Demo: Redirection
- Summary
Control Flow
- Introduction
- While and Until
- The Classic For Statement
- The C-Style For Statement
- Break and Continue
- The Case Statement
- && and ||
- Summary
Variables 2
- Introduction
- Integer Variables
- Arithmetic Expressions
- Arithmetic Expressions 2
- Read-only Variables
- Exporting Variables
- Arrays
- Summary
Handling Script Parameters
- Introduction
- Special Variables
- Shift
- Getopts
- Getopts: Handling Errors
- Summary
Shell Functions
- Introduction
- Shell Functions
- Shell Functions 2
- Functions: Demo
- Some Miscellaneous Remarks
- Summary
Fun with Strings
- Introduction
- Removing Part Of A String
- Search and Replace
- Setting A Default Value
- Conditional Expression Patterns
- Regular Expressions in The Conditional Expression
- End of Options
- Summary
Many Ways to Run Your Script
- Introduction
- Running your Code
- Nohup and The Background
- Exec
- At and Cron
- Set and Shopt
- Summary
Build and Release Course Training with Jenkins / Maven /Ant/ SVN & Git
Upcoming Training Dates | Training Agenda | Training Calender | FAQ | Why scmGalaxy Online Training
Mode – Online based
Email – info@scmgalaxy.com
Lab Setup Details – Prerequisites
Operating Systems – Windows 7 or Linux(RHEL 7.X or Ubantu)
JDK 1.7 or Higher
SCM Tool Clients Installed (SVN, GIT, Perforce)
Servlet container like GlassFish and Tomcat (If one wants to host Jenkins outside) – Optional
[If trainee is having any specific learning agenda to be addressed in this training apart from the table of Content as below; please email me @ info@scmgalaxy.com]
Table of Content
Continuous Integration with Jenkins
- What is Software Configuration Management
- What is Build and Release Engineering
- Introduction of Version Control Management?
- Basic usage of Subversion
- Basic usage of git
- Introduction of Build Management?
- How to write ant script?
- How to write Maven script?
- How to write MSBuild Script?
- Introduction of Release Management?
- Benefit of Continuous Integration?
- Different Continuous Integration Tools?
- Focus on Jenkins
- Installation and Setup of Jenkinks
- Configure a Job to Build the Project using SVN /GIT and ANT/Maven
- Using Build Script (Ant, Maven and MSBuild) with Jenkins
- Authentication and Authorization in Jenkins
- Lab and Assignment
- Nodes Setup in Jenkins
- Labels, Tags and Notification using Jenkins
- Empowered with Plugins in Jenkins
- 20 Popular Plugins and Their Usage
- Integrating Jenkins with Junit for automated testing
- Integrating Jenkins with HP-QC for automated testing
- Integrating Jenkins with Cobertura for test Coverage
- Integrating Jenkins with Sonar for static code analysis
- Release and Deployments using Jenkins and Nexus
- Advance Jenkins
- Jenkins Scripting
- Jenkins Command line
- Reporting and Dashboard using Jenkins
- Lab and Assignment
Build and Release Training In Bangalore | Build and Release Training in India | Build and Release Training in Hyderabad | Build and Release Training in Delhi | Build and Release Training in Pune | Build and Release Trainer In Bangalore | Build and Release Trainer in India | Build and Release Trainer in Hyderabad | Build and Release Trainer in Delhi | Build and Release Trainer in Pune
If a linux-build-server suddenly starts getting slow, what would you check?
If a linux-build-server suddenly starts getting slow, I would divide my approach / troubleshooting into 3 section as follows;
1. System Level troubleshooting
a. RAM related issues
b. Disk Space related Issues
c. Disk I/O read write issues
d. Network Hardware issues
e. Mount issues
f. Too Many process running in the machine
2. Application Level troubleshooting
a. Application is not behaving properly. Hit to Application log file OR application server log file OR web server Log file and try to understand the issues.
b. zombie process issues – Find out if any as such process which is causing the system performance issues.
c. Application Log – depends on the application installed, this can be referred and make use of the experience with the project and troubleshoot.
d. Web Server Log – we can check http, tomcat log as well.
e. Application Server Log – We can see jboss, weblogic logs to see if the application server response/receive time is the issues for slowness.
f. Memory Leak of any application – This is one of well known issues in lunux based server due to bad application coding. Many times this can be resolved either by fixing the code or rebooting. But many other solutions are there to apply.
3. Dependent Services troubleshooting
a. SMTP Response time – SMTP server is not responding faster which is causing delay in response and queue up many processes.
b. Network issues – There are many System performance issues is dependent on network or service which is depends on the network.
c. Firewall related issues
d. Antivirus related issues
Some of the useful commands for troubleshooting are..
1. df –k
2. du –sh
3. top
4. uptime
5. ps –eaf | grep
6. vmstat
7. ping
8. tail –f <logfile>
9. iostat
10.free
11.kill -9
12.mount
13.sar
14.ifconfig eth0 | enable | disable
15.traceroute
16.netstat -r
17.nslookup
18.route
There are various options available for each commands and depends on the need during the troubleshooting, right commands can be applied. To find arguments and their explanation, I usually refer google.com rather than man page.
Build and Release Training in Bangalore
Upcoming Training Dates | Training Agenda | Training Calender | FAQ | Why scmGalaxy Online Training
Click Here
Mode – Online
Lab Setup Details OR Prerequisites for personal Laptop
Operating Systems – Windows 7 or Linux (RHEL 7.X or Ubantu)
JDK 1.7 or Higher,
SCM Tool Client and server Installed (SVN, GIT, Perforce)
Send us an email to info@scmgalaxy.com with the confirmed payment receipt for acknowledgement
Free Lunch would be serve to all Participants.
Registration is based on First Come basis and only confirmed registration would be considered.
- What is Software Configuration Management
- What is Build and Release Engineering
- Introduction of Version Control Management?
- Basic usage of Subversion
- Basic usage of git
- Introduction of Build Management?
- How to write ant script?
- How to write Maven script?
- How to write MSBuild Script?
- Introduction of Release Management?
- Benefit of Continuous Integration?
- Different Continuous Integration Tools?
- Focus on Jenkins
- Installation and Setup of Jenkinks
- Configure a Job to Build the Project using SVN /GIT and ANT/Maven
- Using Build Script (Ant, Maven and MSBuild) with Jenkins
- Authentication and Authorization in Jenkins
- Lab and Assignment
- Nodes Setup in Jenkins
- Labels, Tags and Notification using Jenkins
- Empowered with Plugins in Jenkins
- 20 Popular Plugins and Their Usage
- Integrating Jenkins with Junit for automated testing
- Integrating Jenkins with HP-QC for automated testing
- Integrating Jenkins with Cobertura for test Coverage
- Integrating Jenkins with Sonar for static code analysis
- Release and Deployments using Jenkins and Nexus
- Advance Jenkins
- Jenkins Scripting
- Jenkins Command line
- Reporting and Dashboard using Jenkins
- Lab and Assignment
Build and Release Training In Bangalore | Build and Release Training in India | Build and Release Training in Hyderabad | Build and Release Training in Delhi | Build and Release Training in Pune | Build and Release Trainer In Bangalore | Build and Release Trainer in India | Build and Release Trainer in Hyderabad | Build and Release Trainer in Delhi | Build and Release Trainer in Pune
Linux Commands for User | Linux User Commands Reference
General Commands Not Used Frequently
cal – The Calender
bc – The Calculator
file – knowing the file types
cmp – Comparing two files.
comm – Common in two files
dos2unix and unix2dos – Converting between dos and unix
stat – Display file or file system status such as Access, Modify,Change, users and more
# stat <fileName> unix Command
General Commands Frequently Used
Script – recording your session
Passwd – change your password
Echo – Displaying the message
Printf – Displaying the message
who – Who are the Users
who am I – To know my user name
uname – Knowing your machine characteristics
# uname –a – To Print all info.
tty – knowing your terminal
pwd – Checking Current Directory
cd – changing the current Directory
mkdir – Making directories
rmdir – removing directories
ls – Listing Directory Contents
# ls –F – Marks executables with *, directories with / and symbolic links with @ at the end of file name.
# ls – a – Shows all filesnames including beginning with a dot.
# ls –R – Recursive display list of files and folders
# ls -1 – one filename in each line
# ls –l long listing + time of last file modification
# ls –t Sorts filenames by last modification time
# ls –u – Sorts filenames by last access time
# ls –I – Displays inode number
# ls –lc – Time of last inode modifcation
Cat – Displaying and creating Files
# cat filename – Displaying file contents
# cat –n filename – Displaying file contents with line numbers
# cat > filename – To Create a file name
More – Paging Output
# more filename – To to display the filename contents
Less – Paging Output
wc – Counting lines, Words and Characters
# -l – Option counts only the number of lines
# -w – Option counts only the words.
# -c – Options counts only the characters.
cp – copying a files
Options
# -p – Copy the files including preserve the mode, ownership and permissions.
# -R – Copying Directory Structure
gzip and gunzip – Compressing and Decompressing Files
# gzip filename
# gzip –l – To find out how much of the compression was archived.
# gzip –d filename – To Decompress a file
# gunzip filename – To Decompress a file
# gzip –R dir – To Compress Recursive
Tar – The archival Program
# -c – To create a archive
# -x – Extract files from archive
# -t – Display fils in archive
# -v – verbose
# -f – name of the arch filename .tar
# tar –cvf filename.tar filename1 filename2
# tar –xvf filename.tar
Create tar and gzip
# tar cvf – foodir | gzip > foo.tar.gz
Un-tar and gunzip
# tar -zxvf apache-activemq-5.1.0-bin.tar.gz
Un-tar and uncompress using gunzip
# gunzip -c apache-activemq-5.1.0-bin.tar.gz | tar xvf –
or alternatively:
# gunzip apache-activemq-5.1.0-bin.tar.gz | tar xvf –
File Attributes
chmod – Changing File Permission
Abbreviations Used by chmod
Category | Operation | Permission | Assigned Number |
u -user | + – Assigns Permission | r – Read permission | 4 |
g -group | – – Assigns Permission | w – Write Permission | 2 |
o – other | = – Assigns Permission | x – Execute Permission | 1 |
a –all(ugo) |
Example:
# chmod u+x filename
# chmod u+x filename
# chmod ugo+x filename
# chmod u+x filename1 filename2 filename3
# chmod a-x,go+r filename
# chmod 566 filename
# chmod 755 filename
# chmod –R 755 .
# chmod –R a+x *
Chown – Changing File Ownership
# chown username filename
More example needed.
Shell’s Wild-Cards
Wild-Card | Matches |
* | Any numbers of characters including none |
? | A single character |
[ijk] | A single character – either I,j or k |
[x-z] | A single character that is within the ASCII range of the characters x and z |
[!ijk] | A single character that is not an I,j, or k |
[!x-z] | A single character that is not within the ASCII range of the characters x and z |
Ps – Process Status
Options to ps
POSIX Options | Significance |
-f | Full listing showing the PPID of each process |
-e or –A | All Process including user and Systems |
-u usr | Process of user usr only |
-a | Processes of all users excluding processes not associated with terminal |
-l | A long listing showing memory-related information |
Example
# ps -ef
# ps –u sumit
# ps –a
# ps –e
# ps -A – ps command will report a snapshot of the current processes. To select all processes use the -A
# ps -Al – Show Long Format Output
# ps -AlF – To turn on extra full mode (it will show command line arguments passed to process):
# ps axu – Print All Process On The Server
# ps -U vivek -u vivek u – See Every Process Running As User Vivek
# ps -p 55977 -o comm= – Display The Name of PID 55977
# ps -auxf | sort -nr -k 4 | head -10 – Find Out The Top 10 Memory Consuming Process
# ps -auxf | sort -nr -k 3 | head -10 – Find Out top 10 CPU Consuming Process
# $! – Store the PID of the last background jobs
# echo $$ – To Know the PID of current Shell
# echo $SHELL – To Know the Current Shell
# System process easily identifies the ? in the TTY coloumn.
# Mechanism of Process Creation
Fork
Exec
Wait
& and nohup – Ruunning jobs in background
# nohup sort emp.lst $
nice – Job execution with low priority
Kernel decides how much processor time is required for a process based on the nice value. Possible nice value range is: -20 to 20. A process that has a nice value of -20 is very high priority. The process that has a nice value of 20 is very low priority.
Use ps axl to display the nice value of all running process as shown below.
# ps axl
# nice command is used with & operator to reduce the priority of jobs
# nice script.sh &
# ./nice-test.sh & Default the nice value of 0
# nice -10 ./nice-test.sh & Nice value is 10 Low priority.
# nice –10 ./nice-test.sh & Nice value is -10 High priority.
Kill – killing the process
# kill PID
# kill PID1 PID2 PID3
# kill $i – Killing the last background process
# kill –s KILL PID – recommended way for killing (SIGKILL)
# kill -9 PID – same as above but not recommended. (SIGKILL)
# kill -9 $$ -$$ store the PID of current shell
# kill –s KILL 0 – kill all the process including the login shell.
# kill –l – To View the list of all signal names and numbers that are available on your machine.
# kill %1 – Kill first background jobs
Jobs
List the background jobs in following fashion
[3] + running command
[1] – running command
[2] running command
bg – Convert Jobs to background
# if you have invoked a command and the prompt has not yet return, you can suspend the job by pressing Ctrl-Z. Observe that job has not been terminated yet; its onl y suspended (“stopped”). Now you can use bg command to push the current foreground job in the background.
# bg %2 – Sends second job to background
# bg %sort – Sends sort job to background
fg – bring background job to foreground
# fg – To bring most recent job to forground
# fg %1 – Bring First job to foreground
# fg %2 – Bring second job to foreground
# fg %sort – Bring sort job to foreground
at – On time execution
# at 14:08
at > script.sh
[ctrl-d]
Batch – execute when system resources are available
# batch < script.sh
Cron – Schedule and run jobs periodically
# crontab –e – To Edit the cron tab
# crontab –l – To Display the cron tab
# crontab cron.tx – cron.txt contains cron commands
# crontab –r – To Remove the cron
Format of crontab – TODO
Customizing the environment
# set – set statemtent display a complete list of all environment variable
# PATH =$PATH:/usr/xpg4/bin – Adding new value to old values
# PS1 =”C> “ – To Change the prompt
# PS1=’[$PWD] ‘ – To Change the prompt to pwd
# alias cp=”cp –I” – To Set the alias in bash
# history – To See the history
# IFS – Field Separators for commands and arguments
# !! – Repeat Previous commands
# !2 – Repeat commands 2 from history output
# !-2 – Execute the commands prior to previous one
# !v – Execture very last commands beginning withg v
# $_ – Using last arugement of previous commands
# mkdir raj
# cd $_
ln – Hard Links and Softlinks
# ln /usr/bin/perl /usr/local/bin/perl – To create a hard links from src to dest
# ls –I – To Display the node number of files
# ln –s /usr/bin/perl /usr/local/bin/perl – To Create a soft link
umask – default file and directory permission
# When you create a files and directories, the permission assigned to them depends on rge system’s default setting..
# unix has default 666 for regular files & 777 for directories
# umask –ENTER
# 022
This is an octal number which has to be subtracted from system default to obtain the actual default. This becomes 644 (666-022) for ordinary files and 755 ( 777 -022) for directories.
# User can set the umask such as umask 023.
touch – Changing the time stamp
# touch emp.lst – Create a file name called emp.lst
# touch 02161430 emp.lst – To Change the time stamp of file for MMDDhhmm format
# touch –a 02161430 emp.lst – To Change the access time stamp of file for MMDDhhmm format
# touch –m 02161430 emp.lst – To Change the modification time stamp of file for MMDDhhmm format
find – Locating files
Expression Used by find.
Expression | Use |
-inum n | Having inode number n |
-type x | if of type x can include files, directories or symbolic link |
-type f | If an ordinary file |
-perm nnn | If octal permission match nnn completely |
-links n | If having n links |
-user usname | If owner by usname |
-group gname | If owned by group gname |
-size +x[C] | |
-mtime –x | If modified in less than x days |
-newer filename | If modified after filename |
-mmin –x | If modified in less than x minutes |
-atime +x | If accessed in more than x days |
-amin +x | If accessed in more than x minutes |
-name filename | Filename |
Action | Significance |
Prints selected file on standard output | |
-ls | Executes ls –lids commands on selected files |
-exec cmd | Executes UNIX command cmd followed by {} \; |
-ok cmd | Like –exec, except that command is executed after user confirmation |
Example
Change File Permissions Recursively
# find . -type f -exec chmod 644 {} \;
# find . -type d -exec chmod 755 {} \;
# find . -name Configuration.php -exec chmod 666 {} \;
Find files modified in the last 48 hours, and in current folder and one level below
# find -maxdepth 2 -type f -mtime -2
To find all files modified in the last 24 hours (last full day) in a particular specific directory and its sub-directories:
# find /directory_path -mtime -1 –print
To find all files with regular file types only, and modified in the last 24 hours (last full day) in current directory and its sub-directories:
# find /directory_path -type f -mtime -1 –print
# find . -type f -mtime -1 –print
To find all files that are modified today only (since start of day only, i.e. 12 am), in current directory and its sub-directories:
# touch -t `date +%m%d0000` /tmp/$$
# find /tmefndr/oravl01 -type f -newer /tmp/$$
# rm /tmp/$$
To find all files in /home/user/demo directory
# find /home/user/demo -type f –print
To find all files in /home/user/demo directory with permission 777, enter:
# find /home/user/demo -type f -perm 777 –print
Apply new permission using the -exec option as follows:
# find /home/user/demo -type f -perm 777 -print -exec chmod 755 {} \;
To select directories and subdirectories use the following syntax:
# find /var/www/html -type d -perm 777 -print -exec chmod 755 {} \;
This first Linux find example searches through the root filesystem (“/”) for the file named “Chapter1”. If it finds the file, it prints the location to the screen.
# find / -name Chapter1 -type f –print
A nice thing to know is that on Linux systems and modern Unix system you no longer need the -print option at the end of the find command, so you can issue it like this:
# find / -name Chapter1 -type f
This next find command searches through the /usr and /home directories for the file named Chapter1:
# find /usr /home -name Chapter1 -type f
To search in the current directory, and all subdirectories, just use the . character to reference the current directory in your find commands, like this:
# find . -name Chapter1 -type f
This next command searches through the /usr directory for all files that begin with the letters Chapter, followed by anything else. The filename can end with any other combination of characters. It will match filenames such as Chapter, Chapter1, Chapter1.bad, Chapter-in-life, etc.:
# find /usr -name “Chapter*” -type f
This next command searches through the /usr/local directory for files that end with the extension .html. These file locations are then printed to the screen.
# find /usr/local -name “*.html” -type f
To find all directories named build under the current directory, use this command:
# find . -type d -name build
This command searches through the htdocs and cgi-bin directories for files that end with the extension .cgi. When these files are found, their permission is changed to mode 755 (rwxr-xr-x). This example shows that the find command can easily search through multiple sub-directories (htdocs, cgi-bin) at one time.
# find htdocs cgi-bin -name “*.cgi” -type f -exec chmod 755 {} \;
Find and display files last modified les than 90 days ago.
# find . -name “*” -mtime -3 -print
find everything in your home that has been modified more recently than “abc.txt”:
# find $HOME -newer ~joeuser/lastbatch.txt
For finding only files from all directories recursively
# find ./ -type f | wc -l
For finding only files from all directories recursively
# find ./ – type d | wc -l
How to Return a message when a file is not found using find command? OR
Find command return type OR
when the file is not found i want it to return some value OR
# find . -name raj.txt > raj.txt
# counter= `cat abc.txt | wc -l’
# if [$counter -gt 0]
# then
# echo “File is found”
# else
# echo “File is not found”
# fi
That’s nice, but what if I want to see the last modification time of these files, or their filesize? No problem, I just add the “ls -ld” command to my find command, like this:
find . -name “*.pl” -exec ls -ld {} \;
Count Total number of files in Directory and Subdirectory
# find . -type f | wc –l
Count Specific extention files in Directory and Subdirectory
# find . -type f -name \*.mnp |wc –l
Count only Directory
# find . -type d | wc –l
head – Displaying the beginning of a file
# head –n 3 filename
# vi `ls –t` | head –n 1` – Opens last modified file for editing
tail – Displaying the end of a file
# tail -3 filename
# tail –f filename Monitering file live
Cut
# cut –c 6-12,24-32 filename – Cutting column
# cut –d \| -f 2,3 filename – Cutting fields
# cut –d “|” –f 1,4- filename –To cut out the fields numbered 1,4,5 and 6.
Sort – Ordering a file
Sort Options
Option | Description |
-tchar | Use delimiter char to identify fields |
-k n | Sorts on nth field |
-k m,n | Start sort on mth field and end sort on nth field |
-k m.n | Start sort on nth column of mth field |
-u | Removes repeated lines |
-n | Sort numerically |
-r | Reverse sort order |
-m list | Merge sorted files in list |
-c | Checks of file sorted |
-o filename | Place output in file filename |
Examples:
# sort –t”|” –k 2 shortlist
# sort –t”|” –r –k 2 shortlist
# sort –t “|” –k 3,3 –k 2,2 shortlist
# sort –t”|” –k 5.7,5.8 shortlist
# sort numfile
# cut –d”|” –f3 filename | sort –u
# sort –m foo1 foo2 foo3
Uniq – Locate repeated and non-repeated lines
# sort dept.lst | uniq
# cut –d”|” –f3 emp.lst | sort | uniq –u ( -u selects only lines which is not repeated)
# cut –d”|” –f3 emp.lst | sort | uniq –d ( -d selects one copy of repeated lines)
# cut –d”|” –f3 emp.lst | sort | uniq –c
tr – Translating characters
# tr ‘|/’ ‘~-‘ < emp.lst | head –n 3 – To replace the | with a ~ and the / with a -.
# head –n 3 emp.lst | tr ‘[a-z]’ ‘[A-Z]’ – Change first three lines from lower to upper.
# tr –d ‘|’ < emp.lst | head –n 3 – To deleting characters
grep – Searching for a pattern
# grep “director” filename1 filename2
# grep “Rajesh Kumar” filename1
# grep –i ‘rajesh’ filename1 To Ignore case
# grep –v ‘rajesh’ filename1 > filename2 To Select all except lines containing patterns
# grep –n ‘rajesh’ filename1 To Display line numbers
# grep –c ‘rajesh’ filename Counting line containing pattern
# grep –l ‘rajesh’ *.lst -l options display only the name of files containing pattern
# grep –f pattern.lst emp.lst Taking patterns from files
Grep a file, but show several surrounding lines?
For BSD or GNU grep you can use -B num to set how many lines before the match and -A num for th number of lines after the match.
grep -B 3 -A 2 foo README.txt
If you want the same amount of lines before and after you can use -C num.
grep -C 3 foo README.txt
This will show 3 lines before and 3 lines after.
Basic Regular Expression tables
Symbols or Expressions | Matches |
* | Zero or more occurrances of the previous character |
g* | Nothing or g,gg,ggg, etc |
. | A Single Character |
.* | Nothing or any number of character |
[pqr] | A single character p, q or r |
[c1-c2] | A Single Character with ASCII range |
[1-3] | A single digit between 1 and 3 |
[^pqr] | A Single character which is not a p,q or r |
[^a-zA-Z] | A non-alphabetic character |
^pat | Pattern pat at the beginning of the line |
pat$ | Pattern pat at the end of the line |
bash$ | Bash at the end of the line |
^bash$ | Bash as the only one word in line |
^$ | Line containing nothing |
+ | Matches one or more character of previous character |
? | Matches zero or one occurrence of the previous character |
| | Delimiter for multiple pattern |
( ) | Group pattern |
Example:
# grep “[aA]g[ar][ar]wal filename
# grep “[aA]gg*[ar][ar]wal” filename
# grep “j.*Saxena” filename
# grep “^2” filename
# grep “7…$” filename
# grep “^[^2]” filename
# grep –E “[aA]gg?arwal” filename
# grep –E ‘sengupta|dasgupta’ filename
# grep –E ‘(sen|das)gupta’ filename
Editor
Awk – http://www.thegeekstuff.com/2010/01/awk-introduction-tutorial-7-awk-print-examples/
Sed
vi
# sed – The Stream Editor
# sed ‘3q’ filename == head –n 3 Quit after line number 3
# sed –n ‘1,2p’ filename Prints the first 2 lines. Must use –n with p
# sed –n ‘$p’ filename print last line
# sed –n ‘9,11p 7,9p $p’ filename
# sed –n ‘3,$ip’ filename Don’t print line 3 to the end, display only line 1 and 2
# sed –n ‘/director/p’ filename To print the lines which has pattern in filename
# sed –n ‘/dasgupta/,/saksena/p’ filename
# sed –n ‘1,/dasgupta/p’ filename
# sed –n ‘/[aA]gg*[ar][ar]wal/p’ filename
# sed –n ‘/sa[kx]s*ena/p /Gupta/p’ filename
# sed –n ‘/50…..$/p’ filename
# sed –n ‘director/w dlsit’ filename
# sed –n ‘director/w dlist /manager/w mlist /executive/w elist’ filename
# sed –n ‘1,500w foo1 501,$w foo2’ filename
# sed ‘1i\
> #include <stdio.h>\
> #include<unistd.h>
>’ foo.c >> $$ Include these include on beginning of the program
# sed ‘a\
‘ filename # insert after every line this blank line
# sed “/director/d’ filename > filename2 -n option not to be used with d
== # grep –v “director” filename > filename2
# sed –n ‘/director/!p’ filename1 > filename2
# sed ‘s/|/:/’ filename | head -2
# sed ‘s/|/:/g’ filename | head -2
# sed ‘1,3s/|/:/g’ filename
# sed ‘1,5s/director/member /’ filename
# sed ‘s/^/2’ filename
# sed ‘s/$.00/’ filename
Sed tables
Command | Description |
I,a,c | Insert, Appends and Changes text |
d | Delete lines |
10q | Quit after reading the first 10 lines |
p | Print line on standard outputs |
3,$p | Print lines 3 to the end. –n option is required |
$!p | Prints all lines except last line. –n option required |
/begin/,/end/p | Print lines enclosed between begin and end. –n option required |
q | Quit after reading uo to the address line |
Vi – Editor
IMPORTANT – YOU SHOULD USE DOUBLE QUOTES ONLY WHEN PARAMETER EVALULATION OR command substitution is embedded within command
Write about command “w”