ROOT Project Status Major developments Directions DESY 5 December 2005

ROOT Project Status Major developments Directions DESY 5 December 2005 René Brun CERN Overview, Major Developments, Directions 1

ROOT: a 10 years old project Started in January 1995 in NA49 First public presentation in November 95 Cooperation with Masa Goto/CINT in April 96 First implementation of ROOT with gAlice in December 97 (TGeant3)- VMC FNAL chooses ROOT for I/O and data analysis Announced at CHEP98 in Sept 98 in Chicago RHIC experiments follow immediately Hoffmann computing Review in 2000/2001 LCG project in 2002 (Blueprint RTAG) ROOT: LCG project in 2005 : DESY Rene Brun Overview & Major Developments 2

Applications Area Organization in LCG Phase I decisions Architects strategy Applications manager forum SPI project POOL project DESY Rene Brun SEAL project PI project User provider Simulation project consultation Overview & Major Developments ROOT Applications area meeting 3

Applications Area Organization in LCG Phase II Same organization as in Phase I. ROOT and SEAL projects merge SPI project POOL project ROOT SEAL project Simulation project consultation Applications area meeting ROOT project structured in work-packages DESY Rene Brun Overview & Major Developments 4

New ROOT team structure The work-packages SEAL Lorenzo Moneta DICT Philippe Canal MATH Lorenzo Moneta I/O & Trees Philippe Canal GEOM/VMC Andrei Gheata DESY Rene Brun BASE Fons Rademakers 2-D/3D graphics Olivier Couet PROOF Fons Rademakers Overview & Major Developments GUI Ilka Antcheva 5

ROOT version 5 pro release 4.04/02 3 May (new Users Guide) 1st dev release 5.02 28 June 2nd dev release 5.04 20 September 3rd dev release 5.06 2 Nov Pro release 5.12 15 December (New Guide) See detailed Release Notes http://root.cern.ch/root/Version50400.news.html DESY Rene Brun Overview & Major Developments 6

ROOT 2005 workshop September 2830 at CERN 115 registered participants 41 talks 9 posters See talks at ROOT web page DESY Rene Brun Overview & Major Developments 7

ROOT: large user community ROOT is LGPL DESY Rene Brun Overview & Major Developments 8

Base Work Package New infrastructure features Miscellaneous Overview, Major Developments, Directions 9

TArchiveFile and TZIPFile TArchiveFile is an abstract class that describes an archive file containing multiple sub-files, like a ZIP or TAR archive. The TZIPFile class describes a ZIP archive file containing multiple ROOT sub-files. Notice that the ROOT files should not be compressed when being added to the ZIP file, since ROOT files are normally already compressed. To create the file multi.zip do: zip –n root multi file1.root file2.root The ROOT files in an archive can be simply accessed like this: TFile *f TFile::Open("multi.zip#file2.root") or TFile *f TFile::Open("root://mymachine/multi.zip#2") A TBrowser and TChain interface will follow shortly. DESY Rene Brun Overview & Major Developments 10

Replica of a DB subset T0 local TZipFile T1 remote TZipFile che. a c d stor, a c , rootd x , p htt DESY Rene Brun Overview & Major Developments 11

SLAC’s New File Server xrootd The file server xrootd (eXtended ROOT daemon) has been developed by Andy Hanushevsky of SLAC. The server exploits a multithreaded architecture to provide high-performance file based access, focusing on scalability and fault tolerance. The server is being extensively used by BaBar, Star and Alice. The xrootd file server will in the near future replace the current daemon rootd. xrootd and its plugin facility is heavily used by PROOF DESY Rene Brun Overview & Major Developments 12

The New xrootd Client TXNetFile The new client class TXNetFile implements the xrootd protocol and is provided to open a file via the xrootd daemon. TXNetFile can detect when it talks to on old rootd daemon and return a TNetFile. To open a file via xrootd, just use the standard static method TFile::Open() as for opening via rootd. DESY Rene Brun Overview & Major Developments 13

Class TGrid (abstract interface) //--- General GRID const char *GridUrl() const const char *GetGrid() const const char *GetHost() const const char *GetUser() const const char *GetPw() const const char *GetOptions() const Int t GetPort() const //--- Catalogue Interface virtual TGridResult *Command(const char *command, Bool t interactive kFALSE, UInt t stream kFALSE) virtual TGridResult *Query(const char *path, const char *pattern, const char *conditions, const char *options) virtual TGridResult *LocateSites() virtual virtual virtual virtual virtual TGridResult *ls(const char*ldn "", Option t*options "") Bool t cd(const char*ldn "",Bool t verbose kFALSE) Bool t mkdir(const char*ldn "", Option t*options "") Bool t rmdir(const char*ldn "", Option t*options "") Bool t register(const char *lfn , const char *turl , Long t size, const char *se, const char *guid) virtual Bool t rm(const char*lfn , Option t*option "") //--- Job Submission Interface virtual TGridJob *Submit(const char *jdl) virtual TGridJDL *GetJDLGenerator() //--- Load desired plugin and setup conection to GRID static TGrid *Connect(const char *grid, const char *uid, const char *pw, const char *options) DESY Rene Brun Overview & Major Developments 14

Access to File Catalogues eg Alien FC nted e m e l p be im d l u o c rface e t n i logues le a y t t a s C e Sam File D I R G Other DESY Rene Brun Overview & Major Developments for 15

TGrid example with Alien // Connect TGrid alien TGrid::Connect(“alien://”); // Query TGridResult *res alien.Query (“/alice/cern.ch/user/p/peters/analysis/miniesd/”, ”*.root“); // List of files TList *listf res- GetFileInfoList(); // Create chain TChain chain(“Events", “session"); Chain.AddFileInfoList(listf); // Start PROOF TProof proof(“remote”); // Process your query Chain.Process(“selector.C”); DESY Rene Brun Overview & Major Developments 16

Auto Loading of Plugins Support for auto-loading libraries when an unknown class is being referenced. The auto-loading mechanism reads the files ROOTSYS/etc/system.rootmap, /.rootmap and ./.rootmap (via TEnv) to try to map the unknown class to a library. If the library is found it, and the libraries on which it depends, are loaded. The rootmap files are created with the rlibmap tool when executing "make map". Example: in an interactive session, one can do directly TLorentzVector v; without having to do gSystem- Load(“libPhysics”); DESY Rene Brun Overview & Major Developments 17

TMacro This class allows for storing a C macro in a ROOT file. In addition to being stored in a ROOT file a TMacro can be executed, edited, etc. TMacro m("Peaks.C"); created //macro m with name "Peaks" is //from file Peaks.C //macro executed with default m.Exec(); arguments m.Exec("4"); //macro executed with argument m.SaveSource("newPeaks.C"); TFile f("mymacros.root","recreate"); DESY Rene Brun Overview & Major Developments m.Write(); //macro saved to file with name 18

Using PCRE for Reg Exp’s new class TPRegexp which uses the Perl Compatible Regular Expressions library. Well know, rich, regular expression syntax. It is interfaced to TString and other class and methods now using TRegexp. TRegexp will of course stay for backward compatibility. DESY Rene Brun Overview & Major Developments 19

Dict Work Package New version of Reflex New version of rootcint rootcint CINT rootcint - Reflex - Cintex - CINT rootcint - gccxml - Reflex - CINT Adapt PyRoot to Reflex Adapt CINT to Reflex Overview, Major Developments, Directions 20

Dictionaries : root only Root meta C X.h CINT DS ROOT CINT API rootcint XDictcint.cxx DESY Rene Brun Overview & Major Developments CINT 21

Dictionaries : situation today lcgdict XDictlcg.cxx REFLEX API REFLEX DS X.xml Root meta C gccxml X.h ROOT cintex CINT API CINT DS rootcint XDictcint.cxx DESY Rene Brun Overview & Major Developments CINT 22

Dictionaries : situation in the future Python Root meta C ROOT CINT/Reflex API CINT Reflex/Cint DS rootcint -cint XDictcint.cxx rootcint -reflex X.h rootcint -gccxml DESY Rene Brun Overview & Major Developments 23

IO work-package Consolidation, Consolidation, Consolidation Support for STL collections More cases in auto schema evolution Better support for references Bitmap index Overview, Major Developments, TreeSQL Directions 24

ROOT I/O: STL Collections ROOT now supports I/O of all STL containers std::vector T std::list T std::set T std::deque T std::map K,T std::multimap K,T And implicitly (through std::deque) std::queue T std::stack T STL collections are saved in split mode Objects are split (but: NOT if pointers) Quick pre-selections on trees Interactivity: Trees can be browsed Save space (see ROOTSYS/test/bench): std::vector THit : compression 5.38 std::vector THit* : compression 3.37 DESY Rene Brun Overview & Major Developments 25

Float, double and space (1) Math operations very often require double precision, but on saving single precision is sufficient New data type: Double32 t In memory: double On disk: float or integer DESY Rene Brun Overview & Major Developments 26

Float, double and space (2) Usage (see tutorials/double32.C): Double32 t m data; // [min,max ,nbits ] No nbits,min,max: min, max: saved as float saved as int 32 bits precision explicit values or expressions of values known to Cint (e.g. “pi”) nbits present: saved as int with nbit precision higher precision than float for same persistent space DESY Rene Brun Overview & Major Developments 27

Float, double and space (3) Increase Increase precision precision Save Savespace space DESY Rene Brun Overview & Major Developments 28

File types & Access in 5.06 user Local File X.xml http Local File X.root DESY Rene Brun TFile TTreeSQL TKey/TTree TSQLServer TStreamerInfo TSQLRow TSQLResult rootd/xrootd Oracle Dcache Castor RFIO Chirp Overview & Major Developments MySQL PgSQL SapDb 29

Bitmap Indices Bitmap indices are efficient data structures for accelerating multi-dimensional queries: E.g. pT 195 AND nTracks 4 AND muonTight1cm 12.4 Supported by most commercial database management systems and data warehouses Optimized for read-only data However, because an efficient index may be as big as the data, we think that it is only appropriate for things like event meta data catalogues DESY Rene Brun Overview & Major Developments 30

Query Performance TTreeFormula vs. Bitmap Indices 1-Dimensional Queries 5-Dimensional Queries 10 100 Time [sec] 1 1 0.1 0.00001 0.0001 0.001 0.01 0.1 0.1 0.00001 1 0.0001 0.001 Query box 0.01 0.1 1 Query box 10-Dimensional Queries TTreeFormula BMI-EE BMI-RE 100 Bitmap indices 10X faster than TTreeFormula Time [sec] Time [sec] 10 10 1 0.00001 0.0001 0.001 0.01 0.1 1 Query box DESY Rene Brun Overview & Major Developments 31

Data analysis with bitmap indices Event catalogue Bitmap index Direct use by PROOF slaves to select events Ev en t li st ry e qu DESY Rene Brun Overview & Major Developments 32

Math work-package MathCore MathMore Minuit2 Smatrix Linear & Robust Fitter Splot Overview, Major Developments, Directions 33

MATH work-package : News MathCore library with basic Math functionality Basic Special and statistical functions Physics and geometry vectors MathMore library C interface to function and algorithm from GSL Minuit2 Extra math functions, Adaptive integration, derivation, root finders New OO implementation of Minuit Interface to ROOT TVirtualFitter Linear and Robust Fitter sPlot DESY Rene Brun Overview & Major Developments 34

New Math Libraries organization DESY Rene Brun Overview & Major Developments 35

MATH work-package : Plan Complete MathCore with Random numbers Adapt ROOT classes to MathCore TF1,2,3, Fitting Virtual Fitter extensions corresponding changes in ROOT fitting and roofit Fully integrate and extend new Minuit Fitting GUI Box plots, qqplots Many new tools required for LHC Physics analysis (PHYSTAT05 Oxford) DESY Rene Brun Overview & Major Developments 36

Linear Fitter (0) To fit functions linear in parameters Polynomials, hyperplanes, linear combinations of arbitrary functions TLinearFitter can be used directly or through TH1, TGraph, TGraph2D::Fit interfaces When used directly, can fit multidimensional functions DESY Rene Brun Overview & Major Developments 37

Linear Fitter (1) Special formula syntax: Linear parts separated by “ ” signs: “1 sin(x) sin(2*x) cos(3*x)” “[0] [1]*sin(x) [2]*sin(2*x) [3]*cos(3*x)” Simple to use in multidimensional case “x0 x1 exp(x2) log(x3) x4” Polynomials (pol0, pol1 ) and hyperplanes (hyp1, hyp2, ) are the fastest to compute By default, polynomials in TH1, TGraph::Fit functions now go through Linear Fitter Data to be used for fitting is not copied into the fitter DESY Rene Brun Overview & Major Developments 38

Linear Fitter (2) Advantages in separating linear and non-linear fitting: Doesn’t require setting initial parameter values The gain in speed Function Linear fitter Minuit Pol3 in TGraphErrors 1000 fits of 1000 points Average CPU time 1.95 Average CPU time 30.54 TMath::Sin(x) TMath::Sin(2*x) Average CPU time 2.39 Average CPU time 21.34 DESY Rene Brun Overview & Major Developments 39

Robust fitting (0) Least Trimmed Squares regression – extension of the TLinearFitter class Motivation: least-squares fitting is very sensitive to bad observations Robust fitter is used to fit datasets with outliers The algorithm tries to fit h points (out of N) that have the smallest sum of squared residuals DESY Rene Brun Overview & Major Developments 40

Robust fitting (1) High breakdown point smallest proportion of outliers that can cause the estimator to produce values arbitrarily far from the true parameters Graph.Fit(“pol3”, “rob 0.75”, -2, 2); DESY Rene Brun 2nd parameter – fraction h of the good points Overview & Major Developments 41

Multivariate covariance Minimum Covariance Determinant Estimator – a highly robust estimator of multivariate location and scatter Motivation: arithmetic mean and regular covariance estimator are very sensitive to bad observations Class TRobustEstimator The algorithm tries to find a subset of h observations (out of N) with the minimal covariance matrix determinant DESY Rene Brun Overview & Major Developments 42

Multivariate covariance High breakdown point Left – covariance ellipses of a 1000point dataset with 250 outliers Right – distances of points from the robust mean, calculated using robust covariance matrix Indices of outlying points can be returned DESY Rene Brun Overview & Major Developments 43

sPlot – A statistical tool to unfold data distributions Left – Projection plot cut on the likelihood ratio Excess of events – Signal? Background? Right – sPlot – no cut getting rid of background by statistical methods Signal!!! DESY Rene Brun Overview & Major Developments 44

News in TH1 and TF1 TH1: Chi2 test Mean & RMS error, skewness and kurtosis TF1: Derivatives (1st, 2nd and 3rd) Improved minimization – a combination of grid search and Brent’s method (golden section search and parabolic interpolation) DESY Rene Brun Overview & Major Developments 45

Graphics work-package zillions of micro/mini features http:// couet.home.cern.ch/couet/POW files/frame.htm reimplement (TGaxis) GL with new GUI GL for dynamic tracks GL in Pad Overview, Major Developments, Directions 46

DESY Rene Brun Overview & Major Developments 47

TImageDump Many extensions to the libAfterImage library to support line, marker and (filled) polygon drawing. Accessible via TASImage. The new class TImageDump uses TASImage and derives from TVirtualPS to allow the saving of canvases in gif, jpg, png, tiff, root –betc., image formats in batch mode: root [0] .x hsimple.C root [1] c1- Print("c1.gif"); Or to display any gif, jpg, png, tiff in a canvas, do: TCanvas *c1; TImageDump *imgdump new TImageDump("test.png"); c1- Paint(); imgdump- Close(); DESY Rene Brun Overview & Major Developments 48

GL in Pad DESY Rene Brun Overview & Major Developments 49

DESY Rene Brun Overview & Major Developments 50

GL Features : Clipping DESY Rene Brun Overview & Major Developments 51

GUI work-package zillions of micro/mini features http://antcheva.home.cern.ch/antcheva/ GUI Builder completion New Editor Widgets Fit Panel widget Overview, Major Developments, Directions 52

GUI work-package : Plan High Level Widgets Editors, Browsers, Building Blocks Widgets Combos, scroll bars, dialogs, sliders, MDI, etc Code generators TVirtualX TGX11 DESY Rene Brun TGWin32Gdk Overview & Major Developments Changing New features Bug fixes Very stable TGQt 53

DESY Rene Brun Overview & Major Developments 54

Graphics Editor Object orientation of editor design Manage GUI complexity by object editors Presents the right GUI at the right time according to the selected object in the canvas Easy-to-use Capacity for growth DESY Rene Brun Overview & Major Developments 55

Style Manager Top level interface Preview window Manage a collection of TStyle objects Create a new style Delete a selected style Import from a canvas / a C macro Export to a C macro Apply on all canvases or a selected object Activate the style editor Show the predicted results On line update or by request Placed in front of the selected canvas Style Editor DESY Rene Brun Overview & Major Developments 56

GUI Builder GUI Builder simplifies the process of designing GUIs based on the ROOT widget classes. Using Ctrl S or SaveAs dialog, users can generate C code in a macro that can be edited and executed via CINT interpreter: root [0] .x example.C // transient frame TGTransientFrame *frame2 new TGTransientFrame(gClient GetRoot(),760,590); // group frame TGGroupFrame *frame3 new TGGroupFrame(frame2,"curve"); TGRadioButton *frame4 new TGRadioButton(frame3,"gaus",10); frame3- AddFrame(frame4); frame2- SetWindowName(“Fit Panel"); frame2- MapSubwindows(); frame2- Resize(frame2- GetDefaultSize()); frame2- MapWindow(); } DESY Rene Brun Overview & Major Developments 57

GEOM work-package Support for parameterized shapes. This will reduce the geometry size in memory for certain geometries defined in G3 style. CAD geometry import Geometry builder GUI Overview, Major Developments, Directions 58

LHC detectors in ROOT TGeo DESY Rene Brun Overview & Major Developments 59

The Virtual MC User Code VMC Reconstructi on G3 G3 transport G4 G4 transport FLUKA FLUKA transport Geometrical Modeller Visualisatio n Generators DESY Rene Brun Overview & Major Developments 60

ROOT Geometry TGeant3 Used in production – native GEANT3 New: TGeant3TGeo – interface to G3 using TGeo geometry TGeant4 Used for Geant4 physics validation – G4 native geometry built after g3tog4 conversion No interface yet between G4 and ROOT geometry No modification required in the user code See presentation from Ivana Hrivnacova Few possible strategies for this implementation discussed in details at last VMC workshop Expect an interface with G4 June06 TFluka Old geometry interface using G4 geometry vis FLUGG Currently a fully validated geometry interface based on TGeo DESY Rene Brun Overview & Major Developments 61

PROOF work-package Connect/Disconnect modes Multiple queries in parallel Feedback histograms Multi-core CPUs Overview, Major Developments, Directions 62

PROOF – Parallel ROOT Facility System to export the ROOT analysis model on clusters of computers for interactive analysis of large data sets Flexible multi-tier architecture in GRID contexts adapts to cluster of clusters or wide area virtual clusters Exploit inter-independence of entries in a tree or directory to achieve basic parallelism data set split into packets assigned to worker nodes on demand DESY Rene Brun G. Ganis, ROOT05, 29 Sept 2005 Overview & Major Developments 63

PROOF – Multi-tier Architecture Optimize for data locality If not possible, remote data via (x)rootd, rfiod, dCache less important good connection ? VERY important G. Ganis, ROOT05, 29 Sept 2005 DESY Rene Brun Overview & Major Developments 64

PROOF – data analysis Normal ROOT TChain: collection of TTree TSelector: :Begin(),Process(),Terminate() Local processing TChain a("h42"); {// Define the data set a.Add(“root://oplapro62.cern.ch//tmp/dstarmb.root"); a.Add(“root://oplapro62.cern.ch//tmp/dstarp1a.root"); a.Add(“root://oplapro62.cern.ch//tmp/dstarp1b.root"); a.Add(“root://oplapro62.cern.ch//tmp/dstarp2.root"); // Process the selector a.Process("h1analysis.C"); } PROOF Same chain, same selector Remote processing {// Open PROOF TProof proof(“master”); // Process the selector a.Process("h1analysis.C"); } G. Ganis, ROOT05, 29 Sept 2005 DESY Rene Brun Overview & Major Developments 65

PROOF and Selectors Client Initialize each slave Slaves Slaves Slaves Slaves DESY Rene Brun Client Overview & Major Developments Many Trees are being processed The same code works also without PROOF (of course!) No user’s control on the order 66

PROOF – User Sandbox User’s have their own sandbox on each worker node File transfers minimized cache packages, selector File integrity: MD5 checksums, timestamps Package manager to upload files or packages binary or source PAR (PROOF Archive, like Java jar) provides ROOT-INF directory, BUILD.sh, SETUP.C to control setup in each worker TProof API to handle all this G. Ganis, ROOT05, 29 Sept 2005 DESY Rene Brun Overview & Major Developments 67

Typical query-time distribution non-blocking Blocking / Non-blocking blocking G. Ganis, ROOT05, 29 Sept 2005 DESY Rene Brun Overview & Major Developments 68

Analysis Session Example AQ1: 1s query produces a local histogram AQ2: a 10mn query submitted to PROOF1 Monday at 10h15 ROOT session on my laptop AQ3- AQ7: short queries AQ8: a 10h query submitted BQ1: browse results of AQ2 to PROOF2 BQ2: browse temporary results of AQ8 BQ3- BQ6: submit 4 10mn queries to PROOF1 CQ1: Browse results of AQ8, BQ3- BQ6 DESY Rene Brun Overview & Major Developments Monday at 16h25 ROOT session on my laptop Wednesday at 8h40 Carrot session on any web browser 69

GUI manager Allows full on-click control on everything define a new session submit a query, execute a command query editor execute macro to define or pick up a TChain browse directories with selectors online monitoring of feedback histograms browse folders with results of query retrieve, delete, archive functionality DESY Rene Brun Overview & Major Developments 70

Query processing Processing information Feedback histograms DESY Rene Brun Overview & Major Developments 71

PROOF and XROOTD XROOTD is already playing a very important role in PROOF. It will continue to play a growing role. The PROOF and XROOTD teams are cooperating to get even more from XROOTD: caching, read ahead, new XROOTD services. Still a lot to do to have a good integration of XROOTD with other services like CASTOR. DESY Rene Brun Overview & Major Developments 72

PROOF test facility Since a few weeks we have access to a dedicated farm with 32 dual processor nodes. These machines are intended for testing, not production. The nodes are slow machines (800 MHz), but are extremely useful (vital) to test our PROOF prototypes (Alice, CMS, Phobos). We expect to have many more (100) and faster machines next year. CAF for Alice and CMS. DESY Rene Brun Overview & Major Developments 73

Multi Core CPUs http://www.intel.com/technology/computing/archinnov/platform2015/ This is going to affect the evolution of ROOT in many areas DESY Rene Brun Overview & Major Developments 74

Moore’s law revisited DESY Rene Brun Overview & Major Developments 75

CPU/Node hierarchy Laptop node Local cluster GRID(s) 1- 32- ?N cpus 1000xN cpus 100x1000 nodes 100 micros 100 millis latency Disk 2005 Disk 2012 100 nanos 100 Giga 1 Tera DESY Rene Brun 100 Tera 1 Peta Overview & Major Developments 10 Peta 100 Peta 76

Many implications for ROOT More and more multi-threaded applications Will have to make many classes thread aware ACLIC compilation in parallel GL viewer could take advantage of multi cpus Fitting too I/O with threads for Read ahead Unzipping And obviously TTree queries and PROOF DESY Rene Brun Overview & Major Developments 77

Summary After 10 years of development, ROOT is widely used in HEP and elsewhere. The team has been extended with LCG2 and cooperates with many external developers. Consolidation phase for I/O and Trees Intensive developments in most packages Pushing PROOF data analysis model Pro release 5.12 (15 December) Next Pro release 6.06 (June 2006) DESY Rene Brun Overview & Major Developments 78

SEAL ROOT transition time/versions SEAL Expts S/W ROOT functionality Adiabatic changes towards the experiments SEAL functionality will be maintained as long as the experiments require DESY Rene Brun Overview & Major Developments 79