hpc @ aub day 1 - computational software carpentry

thanks

my heartfelt acknowledgements to Profs. Wafic Sabra and Prof. George Turkiyyah for the invitation to come visit the most beautiful campus in the Middle East.
Mike Hamam has been a superstar in helping me get the laboratory machines ready. This workshop would not be happening without him.

overview

today we discuss the "power tools" of computational software usage and development.
tomorrow we dive into the Message Passing Interface standard, with some notes on tuning I/O.
Wednesday is an overview of the free computational tools available through the Sage toolkit, including numpy, scipy, and sympy.
Thursday we integrate some of what we've learned from each day, and dive into two application codes, Clawpack and PETSc, through their Python wrappers.

overview

each day will be split into a lecture and a laboratory

the lectures will try to grant you shallow exposure to many topics, and will be full of references for you to follow up on (all the slides are or will be available online at: http://aron.ahmadia.net/hpcataub
the laboratories are aimed at helping you put some of these tools in practice. There will be exercises corresponding with each day's lecture. The exercises will start off fairly specific, then become more open-ended as the week progresses.

today's lecture

provenance and version control
reproducibility and build systems
the command line
secure shell (ssh)

provenance

computational scientist's motivation: reproduce, reproduce, reproduce!

reproduction

scientific reproducibility

scientific provenance: the origin or source of data presented in defense of a scientific theory or argument
experimental provenance: equipment, materials, and procedure in obtaining data
computational provenance: software, input data, and environment for obtaining data

Never trust a result you can't reproduce

managing code provenance

How are changes allowed to be introduced?: change management
What do we do to patent/publish/sell this?: release management
How was this particular artifact (graph/table) produced? How do we coordinate distributed development of our software?: version management

code management

change control: formal process for proposing, verifying, and recording changes to software

computational science perspective

not as relevant unless contributing to large or commercial software engineering projects
still must verify that scientific code produces the same results after it has been changed

release management

release management: management of the process of releasing software to users or a production environment

computational science perspective:

commercial/academic scientific software tools and libraries release new versions of their code to users
a researcher who publishes results is 'releasing' their code, whether they realize it or not

release management

software versioning schemes are somewhat arbitrary, generally:

gcc-a.b.c

version numbers

a (major) - functionality changed/removed from API
b (minor) - functionality added to the API
c (patch) - a bug was found and fixed

version control

version control: management of changes to documents, sources, and other aspects a repository

computational science perspective (absolutely essential!):

aids and fosters collaboration
encourages experimentation within repository
removes needless commented out code cruft
provenance for publishing scientific results

history of version control

1972 - SCCS: commercial, introduced weave
1982 - RCS: open source, single file
1990 - CVS: open source, centralized repository
1998 - BitKeeper: commercial, distributed
2000 - Subversion: it's better than CVS!
2001 - Arch: open source, distributed
2003 - Monotone: open source, distributed, three-way merge
2005 - Hg: open source, fast, three-way merge
2005 - git: open source, very fast, many merge schemes

bitbucket.org

offers free project hosting
- so long as your code is public
- offers both hg and git for version control!
launched in 2008, acquired by Atlassian in 2010
similar to Google Code, Launchpad, github

introduction to git

clone/init (start working on a repository)
checkout - I made a mistake!
add - files to a repository
commit - changes to a repository
push - changes upstream
pull - changes downstream

Looks very similar to SVN, but introduces no abstractions because there is no central repository.

provenance

computational scientist's motivation: reproduce, reproduce, reproduce!

computational software engineer's motivation

Build, Baby Build!

a basic taxonomy of programming languages

machine language - the native language executed by central processing units on computers, represented usually in hexadecimal
assembly language - an isomorphic mapping between a machine language and a set of human understandable text mnemonics
object code, a collection of separate, named sequences of machine language and associated data and metadata

a basic taxonomy of programming languages

high level language - a programming languages designed to be abstracted from the specifics of a particular computer architecture
scripting language - a programming language that control one or more software applications

interpreted vs. compiled

a high level language can be either interpreted or compiled into machine language before being executed on a computer
- an interpreted program is indirectly executed through an interpreter, a separate program that usually runs on the same computer
- a compiled program is instead translated to machine code by a compiler before being executed directly on the computer

interpreted vs. compiled

compiler - translates a high level language to object code
linker - collects and merges object files into single executables
assembler - assembly -> machine code
archiver - objects -> libraries

building scientific software

most freely distributed scientific software is designed to be built with make
make [Feldman, 1978] - a software construction tool for building targets from their corresponding dependencies
- dependencies are frequently hierarchical (e.g. program: objects: source)
- builds can be partial (only some targets updated) or complete

building scientific software

the most popular accompanying tool is the GNU build system
- GNU build system [MacKenzie, 1991]- a suite of programming tools for portably building source-code based distributions

how make all works

look in the current directory for a Makefile, a text file containing declarations of targets, their dependencies, and a set of rules for building targets from their respective dependencies
constructs a directed acyclic graph (DAG) mapping targets to dependencies.
traverses the directed acyclic graph and, for each target, finds all its dependencies, and updates the target if it is older than any of its dependencies (out-of-date)

directed acyclic graph

targets

dependencies

the command line

"Unix is hard to learn. The process of learning it is one of multiple small epiphanies. Typically you are just on the verge of inventing some necessary tool or utility when you realize that someone else has already invented it, and built it in, and this explains some odd file or directory or command that you have noticed but never really understood before."

Neil Stephenson "In the Beginning... Was the Command Line"

the command line - ls

$ ls

the command line - ls

 $ ls

Applications      ILOG              Public            games             sandbox
Desktop           Library           Red Kawa          homebrew          sparse_mutant
Documents         Mail              Sites             hpc@aub           thesis
Downloads         Movies            bac               hpc@aub_private   tmp
Dropbox           Music             bin               junk from desktop unison
Frameworks        Pictures          blast             ksl

ls,wc - count directory contents

$ ls | wc

  29      32     237

ls, grep, wc - number of contents that start with D

$ ls | grep '^D' | wc

   4       4      36

command line tools

grep, wc, and ls are all examples of command line tools. we don't have the time to explore the full capabilities of the command line, but it is important to know of these techniques for getting more information about any tool

command line tools - getting help

the google - "tutorial $tool" is usually a good way to find a decent introduction to a unix tool
man 1 $tool brings you into a manual page for command line tools, arrow keys scroll, q quits
UNIX Power Tools by O'Reilly - solid all around reference

secure shell (ssh)

secure shell (ssh) is a network protocol for secure remote shell access, data communication, and command.
offers password access, but prefers public-key cryptography
replaces rsh, rcp, and ftp

secure shell (ssh) in 15 seconds

ssh-keygen -t rsa -C "user@aub.edu.lb"
ssh-copy-id -i ~/.ssh/id_rsa.pub user@machine
exec ssh-agent $SHELL
ssh-add ~/.ssh/id_rsa