CISC 3130 Unix System Programming Xiaolan Zhang Spring 2013 1 Unix

CISC 3130 Unix System Programming Xiaolan Zhang Spring 2013 1 Unix System Programming, Spring 2013

Outline Important course information Objective Roadmap Requirement & policy Brief history of GNU/Linux GNU/Linux Architecture Getting started Log in, simple command, shell 2 Unix System Programming, Spring 2013

You’ve written programs Lot of times, one can use existing tools to implement new functionalities Real world applications are complicated: – generate input & output, or have GUI – communicate with other program (local or remote) – use multiple processes or threads for improved interactivities – Needs to be profiled/tested to improve performance 3 Unix System Programming, Spring 2013

About this course UNIX: time-sharing operating system, consisting of kernel (program that controls and allocates system resources, file system) Essential programs: compilers, editors, commands & utilities Linux is a variation of Unix programming environment is very similar 4 Unix System Programming, Spring 2013

About this course Many levels of programming: Stronger awareness about hardware and operating system Low-level: kernel programming System programming device driver programming 5 Unix System Programming, Spring 2013 Application programming High-level: Web programming

Operating System, Kernel operating system: two different meanings the entire package consisting of central software managing a computer’s resources and all of accompanying standard software tools, such as command-line interpreters, graphical user interfaces, file utilities, and editors. central software that manages and allocates computer resources (i.e., CPU, RAM, and devices). kernel is often used as a synonym for second meaning 6 Unix System Programming, Spring 2013

What is Operating System? From app. programmer’s User –level applications System calls Operating System Physical machine interface Hardware: processor(s), main memory, disks, printers, keyboard, display, network interface, etc. point of view: – O.S. manages hardware resources – O.S. provides user programs with a simpler interface, i.e. system calls cnt read(fd, buffer,nbytes) getc() etc. We will encounter OS concepts, inevitably. 7 Unix System Programming, Spring 2013

Kernel Functionalities: Process scheduling Managing one or more central processing units (CPUs), Unix: a preemptive multitasking operating system multiple processes (i.e., running programs) can simultaneously reside in memory and each may receive use of the CPU(s). Preemptive: scheduler can preempt (or interrupt) a process, and resume its execution later to support interactive responses the processors are allowed to spend finite chunks of time (quanta, or timeslices) per process 8 Unix System Programming, Spring 2013

Kernel Functionalities: Memory management Manage physical memory (RAM) to be shared among processes in an equitable and efficient fashion Virtual memory management: Processes are isolated from one another and from the kernel, so that one process can’t read or modify the memory of another process or the kernel. Only part of a process needs to be kept in memory, thereby lowering the memory requirements of each process and allowing more processes to be held in RAM simultaneously. better CPU utilization, since it increases the likelihood that, at any moment in time, there is at least one process that the CPU(s) can execute. 9 Unix System Programming, Spring 2013

Other OS functionalities The kernel provides a file system on disk, allowing files to be created, retrieved, updated, deleted, and so on. Creation and termination of processes Peripheral device: standardizes and simplifies access to devices, arbitrates access by multiple processes to each device Networking: transmits and receives network packets on behalf of user processes. Support system call interfaces: processes can request the kernel to perform various tasks using kernel entry points known as system calls. Second part of this course: Unix system call API 10 Unix System Programming, Spring 2013

Layers in UNIX/Linux System library interface Users Standard utility POSIX 1003.2 programs System (shell, editors, call Standard library compilers) interface POSIX (open, close, read, write, fork, etc) Unix/Linux kernel (process management, memory management, file system, I/O, etc) Hardware (CPU, memory, disks, terminals, etc) 11 Unix System Programming, Spring 2013

Goal of this course Learn tools needed for develop application in GNU/Linux A working understanding about UNIX Basic commands, shell scripting GNU tools for app. development compiler, debugger, make, version control, library, testing/profiling tools System calls provided in Unix: to request services from operating system 12 Unix System Programming, Spring 2013

Roadmap: a top-down approach Get started topics – Basic concepts & useful commands vi,emacs, sed, awk Bash programming Basic GNU Tools – Compiler chain, make, debugger (gdb) Unix system calls Advanced GNU Tools – Library, gcov, gprof, version control tools 13 Unix System Programming, Spring 2013

Now let’s get started with some background information. 14 Unix System Programming, Spring 2013

Timeline of Unix/Linux, GNU 1993 Windows NT Windows 3X Dennis Ritchie PDP-11 Ken Thompson 15 Unix System Programming, Spring 2013

GNU history GNU: GNU is Not Unix Richard Stallman (author of Emacs, and many other utilities, ls, cat, , on linux) – 1983: development of a free UNIX- like operating system – Free Software Foundation (100s of Programmers) Free software: – freedom to run the program, for any purpose. – freedom to study how the program works and adapt it to your needs. – freedom to redistribute copies so you can help others. – freedom to improve the program and release your improvements to the public, so that everyone benefits. 16 Unix System Programming, Spring 2013

GPL License GNU General Public License is a free, copyleft license for software and other kinds of works “The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users.” Manual pages for commands include copyright info: COPYRIGHT Copyright 2011 Free Software Foundation, Inc. License GPLv3 : GNU GPL version 3 or later http://gnu.org/licenses/gpl.html . This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. 17 Unix System Programming, Spring 2013

Linux history Linus Torvalds – 1991: “hobby” operating system for i386-based computer, while study in Univ. of Helsinki 1996: Linux becomes a GNU software component GNU/Linux: A fairer name than Linux? “Most operating system distributions based on Linux as kernel are basically modified versions of GNU operating system. We began developing GNU in 1984, years before Linus Torvalds started to write his kernel. Our goal was to develop a complete free operating system. Of course, we did not develop all the parts ourselves—but we led the way. We developed most of the central components, forming the largest single contribution to the whole system. The basic vision was ours too. “ --- RMS 18 Unix System Programming, Spring 2013

Linux kernel versions Use “uname –a” to check system information (including kernel version). 19 Unix System Programming, Spring 2013

Linux version history 1.0: only supported single-processor i386-based computer 1.2 support for computers using processors based on the Alpha, SPARC, and MIPS architectures. 2.0: SMP support (symmetric multiple processors) and support for more types of processors 2.2: removed global spinlock, improved SMP support, support m68k and PowerPC architectures, new file systems (including read-only support for Microsoft’s NTFS) 2.4.0: support for ISA Plug and Play, USB, and PC Cards, PA-RISC processor, Bluetooth, Logical Volume Manager (LVM) version 1, RAID support, InterMezzo and ext3 file systems. 20 Unix System Programming, Spring 2013

Linux version history 2.6.0 : integration of µClinux , PAE support, support for several new lines of CPUs, integration of ALSA , support for up to 232 users , up to 229 process IDs, increased the number of device types and the number of devices of each type, improved 64-bit support, support for file systems of up to 16 terabytes, in-kernel preemption, support for the Native POSIX Thread Library (NPTL), User-mode Linux integration into the mainline kernel sources, SELinux integration into the mainline kernel sources, 3.0 : 21 July 2011. the big change was, "NOTHING. Absolutely nothing.” ".let's make sure we really make the next release not just an all new shiny number, but a good kernel too." , released near the 20th anniversary of Linux 21 Unix System Programming, Spring 2013

Understanding uname uname -a Linux storm.cis.fordham.edu 3.6.11-1.fc16.x86 64 #1 SMP Mon Dec 17 21:29:15 UTC 2012 x86 64 x86 64 x86 64 GNU/Linux Kernel name: Linux: Hostname Kernel release: 3.6.11-1.fc16.x86 64 Kernel version: #1 SMP Mon Dec 17 21:29:15 UTC 2012 Machine hardware name: x86 64 (AMD64 instruction set) Processor:x86 64 Operating system: GNU/Linux 22 Unix System Programming, Spring 2013

Unix Standardization Different implementations of Unix diverged: Different meaning for command options System calls syntax and sementatics POSIX, "Portable Operating System Interface", is a family of standards specified by IEEE for maintaining compatibility between Unix systems. C library level, shell language, system utilities and options, thread library currently IEEE Std. 1003.1-2004 POSIX for Windows: Cygwin provides a largely POSIX-compliant development and run-time environment for Microsoft Windows. 23 Unix System Programming, Spring 2013

Single Unix Specification 1990: X/Open launches XPG3 Brand. OSF/1 debuts. 1993: Novell transfers rights to "UNIX" trademark and Single UNIX Specification to X/Open. 1994: X/Open introduces Single UNIX Specification, separating the UNIX trademark from any actual code stream 1995:X/Open introduces UNIX 95 branding program for implementations of Single UNIX Specification. 1996 : Open Group forms as a merger of OSF and X/Open. 24 Unix System Programming, Spring 2013

X/Open Open Group 1997: Open Group introduces Version 2 of Single UNIX Specification, including support for realtime, threads and 64-bit and larger processors. 1998: Open Group introduces UNIX 98 family of brands, including Base, Workstation and Server. First UNIX 98 registered products shipped by Sun, IBM and NCR. 1999: Open Group and IEEE commence joint development of a revision to POSIX and the Single UNIX Specification. 2001: Version 3 of the Single UNIX Specification unites IEEE POSIX, The Open Group and the industry efforts. 25 Unix System Programming, Spring 2013

Today’s Unix Systems To be an officially Unix system, need to go through certification based on the Single Unix Specification Registered Unix systems: AIX, HP/UX, OS X, Reliant Unix, . Linux and FreeBSD do not typically certify their distributions, as the cost of certification and the rapidly changing nature of such distributions make the process too expensive to sustain. 26 Unix System Programming, Spring 2013

Outline Important course information Objective Roadmap Requirement & policy Brief history of GNU/Linux GNU/Linux Architecture Getting started Shell File system, file 27 Unix System Programming, Spring 2013

A Quick Start Terminal PuTTy, a telnet/ssh client a free and open source terminal emulator application a window in your desktop that works like old time terminal Enter your ID and password 28 Unix System Programming, Spring 2013

Your first encounter: shell Shell: a special-purpose program, command line interpreter, read commands typed by a user and execute programs in response to entered commands Many different shells: Bourne Shell (sh): oldest, 29 I/O redirection, pipelines, filename generation (globbing), variables, environment variables, command substitution, background command execution, function C Shell (csh): syntax of flow-control similar to C, command history, command-line editing, job control, aliases Korn Shell (ksh): “csh”, compatible with sh Bourne again Shell (bash): GNU’s reimplementation of Bourne shell, supports features added in C shell, and Korn shell Unix System Programming, Spring 2013

Check/Change Login Shell To check the shell you are using echo SHELL echo 0 login shell: default shell for a user, specified in /etc/passwd To change your login shell, use command chsh 30 Unix System Programming, Spring 2013

Shell: interactive mode A shell session (a dialog between user and shell) Displays a prompt character, and waits for user to type in a command line Prompt depends on shell: sh, ksh, bash: csh: % tcsh: May be customized (with current directory, host, .) 2. On input of a command line, shell extracts command name and arguments, searches for the program, and runs it. 3. When program finishes, shell continues to step 1 4. The loop continues until user types “exit” or “ctrl1. d” to end 31 Unix System Programming, Spring 2013

UNIX command line Command name and arguments: command [ [ - ] option (s) ] argument (s) ] [ option argument (s) ] [ command – Command arguments are mostly file or directory names cp prog1.cpp prog1.cpp.bak – Options: used to control behavior of the command head -20 lab1.cpp wc –w lab2.cpp // count how many words Some options come with option argument – – 32 sort –k 1 data.txt // use the first column of data.txt as the key to sort Unix System Programming, Spring 2013

The most important command !!! man ls man: displaying online manuals – Press q to quit, space to scroll down, arrow keys to 33 roll up/down Unix System Programming, Spring 2013

Correcting type mistakes Shell starts to parse command line only when Enter key is pressed Delete the whole line (line-kill): C-u Erase a character: C-h or backspace key Many more fancy functionalities: – Auto-completion: press Tab key to ask shell to auto-complete command, or path name – History (repeat command): use arrow (up and down) keys to navigate past commands – 34 Unix System Programming, Spring 2013

Shell: batch/scripting mode In batch mode, shell can interpret and execute shell scripts #!/bin/bash # count number of files/directories in curr. directory ls –l wc –l Shell constructs: variables, Loop and conditional statements I/O commands (read from keyboard, write to terminal) Function, arrays 35 Unix System Programming, Spring 2013

Outline Important course information Objective Roadmap Requirement & policy Brief history of GNU/Linux GNU/Linux Architecture Getting started Shell File systems, file, 36 Unix System Programming, Spring 2013

Unix File Files: store information a sequence of 0 or more bytes containing arbitrary information What's in a filename? Case matters, no longer limited to 14 chars Special characters such as -, spaces are allowed, but you shouldn’t use them in filename Can you think of the reason ? Dot files are hidden, i.e., normally not listed by command ls To display all files, including hidden files, use ls -a 37 Unix System Programming, Spring 2013

What’s in a file ? So far, we learnt that files are organized in a hierarchical directory structure Each file has a name, resides under a directory, is associated with some admin info (permission, owner) Contents of file: Text (ASCII) file (such as your C/C source code) Executable file (commands) A link to other files, Virtual file: /proc: a pseudo-filesystem, contains user-accessible objects on runtime state of kernel and executing processes To check the type of file: “file filename ” 38

File Viewing Commands cat: concatenate files and display on standard output (i.e., the terminal window) – cat [option] [file] [ ] means the argument is optional – cat proj1.cpp means there can be multiple arguments of this type – cat proj1.cpp proj2.cpp – cat –n proj1.cpp // display the file with line # More: file perusal filter (i.e., displaying file one screen at a time) – more proj1.cpp head, tail: display the beginning or ending lines of a file – tail -f output // display the file, append more lines as the file grows 39 Unix System Programming, Spring 2013

File manipulation commands rm: remove one or multiple files or directories – rm [option] FILE – rm temp – rm temp1 temp2 Wildcards (metacharacter) can be used in command line – Letter * matches with any string rm *.o: remove all .o files – ?: match any one character – [abc]: match with letter a or b or c rm –r: remove directories and their sub-dirs recursively rm –i : confirm with user before removing files 40 Unix System Programming, Spring 2013

File manipulation commands (2) cp: copy file or directory cp [OPTION] SOURCE DESTINATION To make a backup copy of your program before dramatic change cp proj1.cpp proj1.cpp.bak To make a backup copy of a whole directory cp –r lab1 dir lab1 dir backup -R, -r, --recursive: copy directories recursively 41 Unix System Programming, Spring 2013

File manipulation commands (3) mv: move (rename) files/directories mv [OPTION] SOURCE DEST Rename SOURCE to DEST mv proj1.cpp lab1.cpp mv [OPTION] SOURCE DIRECTORY Move SOURCE to DIRECTORY mv lab1.cpp lab2.cpp CISC3130 42 Unix System Programming, Spring 2013

Hierarchical file system Directory: a file that can hold other files Advantages of hierarchical file system: Files can have same names, as long as they are under different directories Easier for protection Organized files dev cdrom tty24 / (root) bin home staff zhang 43 Unix System Programming, Spring 2013 etc lib group passwd

Absolute pathname, path / (root) dev cdrom tty24 bin home staff etc lib passwd zhang Pathname of a file/directory: location of file/directory in the file 44 system How do you tell other where your prog. Is located ? Absolute pathname: path name specified relative to root, i.e., starting with the root (/) e.g., /home/staff/zhang Unix Programming, Spring 2013 System What’s the absolute pathname for the “passwd” file?

Home directory Every user has a home directory created for him/her When you log in, you are in your home directory In home directory, a user usually has permission to create files/directories, remove files . to refer to current user’s home directory username to refer to username’s home directory 45 Unix System Programming, Spring 2013

Current directory & Relative Pathname Tiring to specify absolute pathname each time To make life easier: working directory User can move around the file system, shell remembers where the user is (i.e., current directory) To check your current directory, use command: pwd 46 Unix System Programming, Spring 2013

Getting around in the file system To create a subdirectory: – mkdir [option] directory – cd – mkdir CISC3130 – cd CISC3130 – mkdir lab1 To remove a directory: – rmdir [option] directory – Report failure if directory is not empty Can use rm –rf to remove non-empty directory 47 Unix System Programming, Spring 2013

Command for change current directory (move around) cd [directory] [zhang@storm Work] cd [zhang@storm ] pwd /home/staff/zhang [zhang@storm ] cd Work [zhang@storm Work] pwd /home/staff/zhang/Work [zhang@storm Work] cd . [zhang@storm ] pwd /home/staff/zhang [zhang@storm ] 48 Unix System Programming, Spring 2013 48

Relative pathname Absolute pathname: specified relative to root Relative pathname: specified relative to current directory . (current directory), . (parent directory, one level up) If current directory is at /home/staff/zhang, what is the relative pathname of the file passwd? / (root) ./././etc/passwd: go one level up, go one level up, go dev cdrom 49 one level up, go to etc, passwd is there lib tty24 bin home staff Unix System Programming, Spring 2013 zhang etc passwd

Relative pathname For all commands that take file/directory name as arguments, you can use pathnames of the file/directory Example: cd /home/staff/zhang/public html pico CISC3130/index.html cd . (go up one level to parent directory) cp ./prog2.cpp prog2.cpp 50 Unix System Programming, Spring 2013

Getting around in the file system ls: list directory contents ls [OPTION] [FILE] ls: list files/directories under current directory ls –l: long listing, [zhang@storm CISC1600] ls -l total 59180 -rw-r--r-- 1 zhang staff 509952 Sep 7 13:02 3 types.ppt -rw-r--r-- 1 zhang staff 593408 Sep 14 23:38 4 computation.ppt -rw-r--r-- 1 zhang staff 1297 Sep 2 12:18 account.html -rw-r--r-- 1 zhang staff 3304448 Nov 7 18:24 ArrayVector1.ppt drwxr-xr-x 2 zhang staff 4096 Dec 8 22:36 Codes 51 Unix System Programming, Spring 2013

Long listing To get more information about each file [zhang@storm Demo] ls -al Total disc space taken in blocks (1024 Byte) total 32 drwxr-xr-x 5 zhang staff 4096 2008-01-16 16:01 . drwxr-xr-x 41 zhang staff 4096 2008-01-16 16:01 . drwxr-xr-x 2 zhang staff 4096 2008-01-16 15:55 CCodes -rw-r--r-- 1 zhang staff 38 2008-01-16 16:01 .HiddenFile -rw-r--r-- 1 zhang staff 53 2008-01-16 15:57 README drwxr-xr-x 2 zhang staff 4096 2008-01-16 15:55 SampleCodes drwxr-xr-x 4 zhang staff 4096 2008-01-16 15:56 ShellScriptes User name of the owner and its group d means Who has permission to read/write the file directory 52

Long listing explained drwxr-xr-x 4 zhang staff 4096 2008-01-16 15:56 ShellScriptes field 1 1st Character: specifies the type of the file. - normal file, d directory, s socket file, l link file next 9 characters – File Permissions field 2: specifies the number of links for that file field field field field 53 3 : specifies owner of the file 4: specifies the group of the file 5 : specifies the size of file. 6: date and time of last modification of the file field 7: File name

File permissions drwxr-xr-x 4 zhang staff 4096 2008-01-16 15:56 ShellScriptes Each file is associated with permission info. Differentiate three type of users: owner user, users from same group as owner, others Three type of access: - in the field means no permission Read (r): use “cat” to open a file to read, use “ls” to list files/directories under a directory Write (w): modify the contents of the file, create/remove files from the directory Execute (x): execute the file, or “cd” or “search” the directory for file Trying to list other’s directory [zhang@storm ] ls ./roche/ ls: cannot open directory ./roche/: Permission denied 54

User and Group Each user has unique login name (user name), and corresponding numeric ID Group id, home directory, login shell: Stored in file /etc/passwd Groups: each user can belong to multiple groups For collaboration Group name, group id, user list info stored in /etc/group Superuser ID: 0, username: root Bypass all permission checks 55 Unix System Programming, Spring 2013

More to play with who: show who is logged on who am I write: write to another user’s terminal (IMS?) which: show the full path name of a command which bash /bin/bash How does shell find a command ? Environment variable PATH stores a list of paths to search for programs: “set grep PATH” or “echo PATH”, “set” to show all variable settings PATH PATH: HOME/bin:. Built-in commands: history, set, echo, etc. mail: send email from command line mail –s “graded project” zhang proj1.cpp 56 Unix System Programming, Spring 2013

Summary Unix, a time-sharing operating system Operating system a program that sits between hardware and user Manages resources (CPU, memory, disk, network connection) and present user interface to user Unix programming environment, i.e., platform the layered view 57 Unix System Programming, Spring 2013

Summary (cont’d) PuTTy emulates a terminal Shell: a command line interface – Engage in dialog (a session with user) What’s in a command – Name, and arguments Hierarchical file system: file and directory Commands for working with file system 58 Unix System Programming, Spring 2013