System-on-Chip Workshop, 12-14 June 2019, CERN SoCs SoCs and and

System-on-Chip Workshop, 12-14 June 2019, CERN SoCs SoCs and and Control Control System System Integration Integration Stefan Schlenker, CERN Input from and thanks to: Piotr Nikiel, Paris Moschovakos, Ben Farnham, Michael Ludwig, Viatcheslav Filimonov, Revital Kopelianski System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 1

Motivation Integration of devices into detector control system (DCS) back-end Basic principle: each Front-End (FE) hardware component should be monitored/controlled by DCS Usually four main scenarios for mainstream hardware components to be interfaced: 1. On-detector FE electronics interfaced via optical links and FPGA receivers 2. Entirely independent (signal power) monitoring/control via e.g. ELMB2/ELMB (on/off-detector) or other custom equipment 3. Off-detector power supply systems 4. Off-detector electronics crates ( ATCA in ATLAS Phase II upgrades) DCS back-end integration shall be done via o Ethernet TCP/IP as standard FE interface o Standard middleware for interfacing with SCADA system / back-end applications ATLAS has chosen OPC UA (since Phase-0 upgrades) System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 2

On-Detector Paths SoC Power supplies ELMB Receiver FE Optical Receiver Custom Custom Custom Legend: Hardware WinCC OA OPC UA client WinCC OA OPC UA client To Trigger/DAQ data handlers. Receiver FPGA(s) Controls data handler Software OPC UA server WinCC OA TCP/IP OPC UA Server DCS NIU TCP/IP Optical Links GBT-interfaced ASICs (local power) SCA SCA SCA DCS Back-End GBT- or E-links WinCC OA Project On-Detector ELMB Satellites/Hubs (remote power) Configuration OPC UA client Diagnostic OPC UA client Firmware SoC System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 3

Off-Detector Paths Example ATCA DCS Back-End Custom ASICs, FPGAs, etc. FPGA processor OPC UA Server SoC System-on-Chip Workshop, 12th-14th of June, CERN OPC UA server WinCC OA OPC UA client WinCC OA OPC UA client Configuration OPC UA client Diagnostic OPC UA client ATCA board Legend: WinCC OA OPC UA client Hardware Software (optional) SoC: IPbus core SNMP IPMC TCP/IP or UDP ATCA shelf IPMI via I2C OPC UA server WinCC OA Project IPMC IPbus WinCC OA SNMP agent on ATCN or via private-ATCN gateway Shelf Manager Firmware Stefan Schlenker, CERN 4

Why SoC? To interface hardware as done for many Ethernet-interfaced devices nowadays: with embedded Linux SoC in general is very flexible since it allows to run software and programmable logic/firmware in the same device very useful also for non-DCS users Optical Receivers for on-detector data: SoC simplifies early extraction of controls data within combined streams and easily provides network interface with TCP/IP To allow embedding of standard middleware closer to hardware o Avoids ecosystem of ad-hoc/custom protocols over Ethernet which are otherwise used o Hardware responsible/developer can concentrate on hw interface tool/library usually anyway needed, integrate with middleware – framework/tooling may be provided by central group o Can provide an abstraction layer if desired, examples: calculating calibrating a temperature value from several FPGA registers can be done in embedded software FPGA configuration may be done conveniently via client call transferring e.g. a binary bit file o DCS back-end experts may avoid dealing with hardware-specific protocols or low-level data handling, can concentrate e.g. on WinCC OA integration (tooling provided by central DCS/JCOP) System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 5

OPC Unified Architecture Industrial machine-to-machine communication protocol for interoperability Originally developed by OPC Foundation for IoT applications (keyword Industry 4.0) OO Information modeling capabilities Enhanced security, performance and scalability Supports buffering, session mgmt, pub-sub, per-connection heartbeats/timeouts, discovery Multi-platform implementation, lightweight embedding possible Commercial SDKs available with stack from OPC foundation or open source stack implementations (C, C , Java, JS, Python) for servers and clients Specifications of Information Models of other Organizations Alarms Alarms Conditions Conditions Historical Historical Access Access Programs Programs Data Data Model Model and and Services Services Transport Transport Protocol Protocol Mappings Mappings OPC OPC UA UA Data Data Model Model Modeling Modeling Rules Rules System-on-Chip Workshop, 12th-14th of June, CERN OPC UA Base Information Access Robustness Robustness Security Security Data Data Access Access Information Models Vendor Specific Model Excellent experience in ATLAS since 2012 Fully supported by JCOP Still requires expertise and effort in programming with OPC UA Provide development environment and generate OPC UA related code? Stefan Schlenker, CERN 6

quasar – Quick opcUA Server generAtion fRamework A tool for rapid C server development and more CERN-made (currently ATLAS DCS, BE-ICS, alumni) framework for model-driven creation of OPC-UA software components o generates servers, clients (Uao, UaoForPython), SCADA integration layer (fwQuasar), etc. C compiler (gcc .), boost (regex, thread, system, program-options), jre, cmake, xsd, python ( lxml, enum34, six), xerces-c, libxml2, openssl OPC UA Server Made with effort efficiency in mind (design, development, testing, deployment) quasar-based software used in LHC experiments (JCOP) as well as beyond CERN quasar can build 100% free and open source OPC-UA servers and clients Validated on different platforms, operating systems, software deployment strategies etc. Choice of OPC-UA stack used: OPC UA client OPC UA client OPC UA client UA SDK (paid license) or open62541 (free & open-source) OPC-UA server toolkit (C ) – Unified Automation Dependencies (all open source): Common XML configuration Logging namespace items and namespace utilities Server metainformation Embedded python Provided or generated components Device specific logic, partially generated Device logic Device access layer XML config file System-on-Chip Workshop, 12th-14th of June, CERN Security (X509 certificate handling) Commercial/OS toolkit Hardware Hardware Remote process 100% application developer/vendor Stefan Schlenker, CERN 7

Modus Operandi START Get understand model of target device/system Developer benefits: Design file can be created using provided XSD schema Roughly 50-90% of code can be generated User sections of Device Logic stubs are well separated, merging tool simplifies re-generation after design changes or quasar upgrades CMake based build system with pre-built toolchains for several platforms Can generate rpms or perform INSTALL ((Yocto PetaLinux) Configuration file can be created using generated XSD schema System-on-Chip Workshop, 12th-14th of June, CERN Fill/edit Design File Generate UA address space configuration module (Re-)generate Device Logic stubs and variable handling Implement/merge user sections of Device Logic Choose platform, build server test binaries Fill Configuration File Test, evaluate Device model is OK Device Logic is OK Generate SCADA types, instances, UA addressing END Stefan Schlenker, CERN 8

quasar Server Example Design Generated class diagram of an advanced quasar server for GBTSCA interfacing Authors: SCA-SW team Running quasar example server seen by OPC UA client UI System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 9

Components & Tools Archiving SQL and NoSQL archivers Platform toolchains: Linux x86 64, i686, ARM (Raspbian, YOCTO, PetaLinux, CentOS), Windows 32/64 Easy RPM generator Generated program to test full address space Documentation: in-design doc to generate auto-documentation in config schema and address space Software management: consistency checker helps using versioning system System-on-Chip Workshop, 12th-14th of June, CERN XML configuration Generated schema simple creation Embed python scripts inside server device logic Embed server project in an existing python application (no C coding) Validation tool verify design constraints Tools Design visualization: UML generator python support Logging Provides API and exchangeable back-end, component based Generated loader for object instantiation and runtime access to configuration Client Generation Generate client code from quasar-based server project, enables server chains, no OPC-UA specific code for users! CalculatedVariables WinCC OA Integration Generates WinCCOA data types, instances and addresses More to come attaching synthetic variables to existing hardware-gathered data (without writing any code) Server meta-information # Items, memory usage, thread pool size, run time Stefan Schlenker, CERN 10

Some quasar-based projects Name Description Status LArPurity ATLAS Liquid Argon calorimeter purity analyzer Production since 2015 IpBus Generic IpBus Production since 2018 ATLAS Wiener Wiener VME crates interfaced with CAN Production since 2015 TileHvMicro HV Micro, ATLAS Tile calorimeter Production since 2015 CAEN CAEN power supplies, JCOP Production since 2018 ISEG ISEG power supplies, JCOP Production since 2017 JCOP Wiener SNMP and CAN Wiener devices Development FtkSbc SBC monitoring, ATLAS FTK Production since 2017 SCA GBT-SCA for ATLAS NSW, LAr and BIS In test HvSys HVSys monitoring and controls over RS232, ATLAS TRT Production since 2017 Generic SNMP Generic SNMP Development LAr LTDB LTDB LATOME monitoring and controls Development Uao, peripheral gFEX ATLAS L1 TDQ gFEX Development embedded (Zynq) eFEX used from Python ATLAS L1 TDQ eFEX Development ATCA Shelf Manager nVent Schroff (aka Pigeon Point) PPS MIB In test ELMB ELMB Receiver Development System-on-Chip Workshop, 12th-14th of June, CERN Notes will become deprecated Uao embedded (Zynq) Stefan Schlenker, CERN 11

System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 12

quasar on SoC: Yocto/PetaLinux, native or x-compiler builds quasar-based OPC-UA servers natively built for Yocto / PetaLinux (based on Yocto) Validated for platforms/devices: o o o o Zynq 7020 dev board (PetaLinux): server for monitoring/control using SoC’s programmable logic (XADC, GPIO, .) ZU19 (Yocto): server under development for publishing ATLAS gFex status data Raspberry Pi (Yocto): made independent RPi Linux distribution with OPC-UA server qemu (Yocto) Documentation for build process is available Natively built on SoC (ATLAS MuCTPi project): CentOS 7 natively boots on the SoC (see yesterdays presentation from Panagiotis Papageorgiou) Successfully built a quasar-based server on ZCU102 o o Equipped the quasar server with hardware monitoring (2-hour effort) and added monitoring of I2C sensor on SoC Reached publishing frequency of 1.8 kHz, imposed by I2C readout rate, with marginal CPU usage Cross-compiling quasar projects Tested in multiple environments o o o RCE (Reconfigurable Computing Element, Zynq 7k) from SLAC with ArchLinux software distribution Raspberry Pi: Raspbian cross-compiler on a desktop and RPi’s sys-root Empty quasar server mem footprint even cross-compiled for Windows using Linux desktop ;-) Needs the compiler and the sysroot (rootfs ) System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 13

Pointers to quasar Project page: https://github.com/quasar-team/quasar, distributed under LGPL ecosystem (optional modules), WinCC OA integration, C OPC-UA client generation facility Mailing list: [email protected] Documentation, video tutorials, quasar papers etc.: https://github.com/quasar-team/quasar/wiki Regular developer meetings, happy to receive contributions or feedback! Support: community effort, JCOP support for DCS systems Extensive tutorial for quasar on SoC in the following contribution (by Piotr Nikiel)! System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 14

SoC / OPC UA Network Integration To other EN Devices EN Switch B Shelf ATCA SM Manager System-on-Chip Workshop, 12th-14th of June, CERN SoC SoC SoC SoC ATCA ATCAShelf Shelf1 2 Any (Any device Device) DCS Machine DAQ Machine reduuplink uplink reduuplink reduuplink EN Switch A LanDB Set: members: ATCN Switch A trusted: DCS Machine DAQ Machine uplink EN router uplink Although OPC-UA traffic can be secured, SoC devices should still be secured to mitigate risks and avoid necessity of frequent PC-like software maintenance Need for isolating SoC even within experiment network (EN) Possible solution being explored: use LanDB sets to restrict SoC access to trusted back-end machines EN Switch B Other Other Device Device Any Any Any Device Device Device Stefan Schlenker, CERN 15

Conclusions Various applications / use cases for SoCs in the context of control systems Embedding of OPC UA on SoCs allows convenient and flexible integration quasar framework available for SoC platforms which generates software based on object-oriented device models Significantly reduces software development effort on SoC (server) and back-end (client) Support and experience with quasar for Yocto/PetaLinux and native ARM CentOS, cross-compilers Network integration of SoC for controls is a challenge not to underestimate System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 16

Backup System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 17

Why SoC on ATCA Blades for slow control? Parameters to be monitored on each blade are numerous: potentially more than hundred, not all of them accessible by the IPMC, e.g. optical transceiver power, temps IPMC path via shelf manager o Entirely proprietary software on shelf manager (also embedded Linux-like OS), vendor-locked o Bottleneck of IPMC communication based on IPMI over I2C, limits monitoring scalability (ok for up to few tens of parameters continuously monitored in an entire shelf at 1Hz) o Load on shelf manager and IPMB which is not primarily done for external monitoring/control but for self-consistent management of the shelf itself SoC with Ethernet interface provides independent link to DCS SoC is often anyway foreseen for run control / configuration / DAQ monitoring tasks SoC allows flexibility on the board to get data from I2C buses or high-speed chip-tochip interfaces (e.g. via programmable logic on FPGA and SoC) which are anyway present for data or configuration handling System-on-Chip Workshop, 12th-14th of June, CERN Stefan Schlenker, CERN 18

quasar system-wide example based on simplified existing project high-throughput link with calibration data hardware 1 HAL1 OPC-UA server type “1” UaO client for “1” DAQ’s group calibration configuration software in C WinCC OA hardware 2 HAL2 hardware connectivity OPC-UA System-on-Chip Workshop, 12th-14th of June, CERN OPC-UA server type “2” UaO client for “1” UaO client for “2” OPC-UA server type “3” “peripheral server” in cascaded server conceptStefan Schlenker, CERN 19