DCS Field Test & Automation Page 1

DCS Field Test & Automation Page 1

Introduction DCS Field Test is based on a USGS utility to allow field technicians to quickly check live messages when configuring. Returns information about a platform and any message data found in a date/time range specified by the query. Accessible via link located DCS website login pages or by browsing directly to a URL: o https://dcs1.noaa.gov/Account/FieldTest o https://dcs2.noaa.gov/Account/FieldTest o https://dcs3.noaa.gov/Account/FieldTest o https://dcs4.noaa.gov/Account/FieldTest Page 2

Field Test Form Parameters o o o o DCP Address Start & End Dates Hours (Current Day Only) Login Credentials Output o Messages found in query time range. o PDT group and transmit information. Clicking ‘SUBMIT’ sends query to the server for processing. Returns PDT information and messages in the specified period range if successful. Page 3

Current Day Default field test query looks for messages received in the last hour of the current day. Start and End dates must both be set to the current day to use (show) the Hours option. Not used (hidden) when either date is not current, or when equal but not current. Selected from drop down containing values from 1 to 23 hours. Determines how far back to look, in hours, when querying on the current day. Page 4

Default Output Page 5

Day Range More advanced query looks for messages received in range of days determined by the Start and End dates. Entire 30-day archive of DCS message data is available. Dates implemented as datepicker controls, and selected from the corresponding popup calendar. Available days limited to last 30 days. Validated following calendar selections. Prevents invalid date combinations from being used. Page 6

Day Output Page 7

End DCS Field Test Page 8

Website Automation A long desired feature of DADDS has been the ability to automatically retrieve DCS messages from the website. While it has been discussed a number of times, no custom interfaces for website automation have been implemented. Possible to use free COTS software to automate retrieval of DCS message data from the website using: o cURL utility and the website PDT field test lookup tool o Python script and website stored filter/message export features Windows and Linux can be configured to automatically retrieve DCS message data using the same 3rd party software. Page 9

Why Automate? Allows for continual download of DCS message data without the need for human interaction. Redundancy. Setting up additional instances that connect to different DCS data sources is trivial. Easily expanded to store downloaded message data for analysis and/or archive purposes: o CSV or Excel data files o Database containing message data Page 10

Options Operating system independent with software for both Windows and most mainstream Linux distributions. Two reliable solutions offering differing levels of complexity, control, and output of message data. DCS Field Test via cURL command line utility and simple scripts to generate server requests for message data. DCS message export via interactive Python scripts that perform website logins, create or apply filters and export message data to Excel data files. Page 11

Field Test Most basic method to automate retrieval of message data. Access now requires an active DCS website login. Use cURL to submit properly formatted field test request to DCS servers. Output consists of – – Platform group, channel and transmit details Grid containing messages returned by query (if any) Requires additional processing to parse and store messages contained in grid output. Page 12

cURL Utility Software package consisting of a URL transfer library and command line utility. Transfers raw data via a supported network protocol, in our case, HTTPS. Command line tool retrieves or sends data, including files, using standard URL syntax. Output depends on the protocol, in our case, the response from the server is a JSON formatted object. Output can be saved to a file for further processing offline. Page 13

cURL - Windows Originally a Linux utility, cURL has been ported to both x86 and x64 versions of Windows. Windows version downloaded from the following link: o https://curl.haxx.se/windows/ MS-DOS batch or Windows PowerShell scripts used to run cURL commands that retrieve DCS message data. Windows Task Scheduler used to execute the scripts containing cURL commands to retrieve message data as necessary. Page 14

cURL - Linux Releases of the cURL utility are available for most mainstream distributions of Linux. Download from cURL version listed at the following link: o https://curl.haxx.se/download.html Be sure to choose correct version for Linux distribution being used, otherwise cURL may not work as intended. Crontab utility is used to execute a cURL command or script containing cURL commands continuously. Page 15

cURL – Hours Example The following example demonstrates how to retrieve the last 1 hours of message data from the platform FD00066A. Raw cURL output shown with no parsing performed. Page 16

cURL – Date Example The following example demonstrates how to retrieve all messages from platform FD00066A between the start and end dates provided. Note that the ‘hours’ parameter must be set to ‘0’ for this query type. Raw cURL output shown with no parsing performed. Page 17

Message Export Advanced method to automate retrieval of message data. Uses Python scripting language to fully interact with the DADDS website. Script loads a DCS website, enters/submits login information, applies a saved filter, adjusts the number of records to export, exports an Excel data file and logs out of the website. Outputs an Excel file that does not require additional effort to parse messages from the server response. Requires a DCS website account. Page 18

Python Language High level, general purpose interpreted programming language. Created to emphasize code readability to help both novice and experienced programmers write clear, logical. Designed to be highly extensible, allowing it to interface with existing applications seamlessly. Consists of small core language with large standard library of functions, classes, algorithms and data structures Page 19

Python - Windows Versions exist for both x86 and x64 Windows distributions. Most recent release 3.74 can be downloaded at: o https://www.python.org/downloads/release/python-374 Once installed scripts can be executed from command prompt the same as MS-DOS batch scripts. Requires manual installation and configuration. Does not include a system supported installation that comes with most Linux distributions. Page 20

Python – Linux Most Linux distributions incorporate Python module supported by the operating system. If none exists it can be installed using ‘apt-get’ to download necessary packages. Refer to link below for details on how to manually install the Python packages on Linux: o https://docs.python-guide.org/starting/install3/linux/ Otherwise Python interpreter should be configured and ready to execute scripts. Page 21

Python Scripts Scripts have extension of ‘.py’ and contain code that is executed by Python interpreter. Contain code with requests and responses necessary to interact with website and export message data in Excel format. Perform a number of interactive steps including: o Login to DCS website o Apply pre-defined saved filter o Increase export message limit o Export messages to Excel file o Logout of DCS website Page 22

Python Example Page 23

Page 24

Page 25

Scheduling cURL and Python scripts must be executed by a scheduling utility for on-going automated message data retrieval. The utility used to control when and how often the script executes depends on the operating system. Windows scripts can be executed by the task manager in Windows 10, or as basic scheduled tasks for older versions. https://www.windowscentral.com/how-create-automated-task-using-task-scheduler-windows-10 Linux scripts can be executed as ‘jobs’ by the Crontab time-based scheduling utility, or implemented as a full service. https://opensource.com/article/17/11/how-use-cron-linux Page 26

Notice! Poorly implemented automated systems can create ‘denial of service’ conditions leading to website outages. IT Security has final say on what is allowed on public facing systems such as the DCS website. Wary of the field test in general, even prior to use of automation. Recent login requirement added to help alleviate their concerns. BUT If outages are a problem THEY WILL PUSH FOR REMOVAL of field test. Imperative that users are good neighbors, and follow the DCS automation best practices when implementing systems, to prevent website outages and loss of the field test feature. Page 27

Best Practices - General NOT INTENDED for large organizations or users with many platforms where receive system or software is more appropriate. Use only cURL or Python tools (if possible) to ensure compatibility and prevent problems with server requests. Create a DCS website field test account for automated systems. Do not use an account that is also used for management purposes. Conflicts occur when attempting a login using an account already logged in. Account will require some human interaction. Password resets still apply! Manually login periodically to reset account password. Page 28

Best Practices - Frequency Denial of service causing outages directly related to how often field test requests occur. Some automated systems observed issuing 10-20 requests per second in response to failed request. DO NOT DO THIS! Execute scripts in-frequently as possible! Ideally only as often as platform transmits. Highest frequency allowed is 15 minutes to account for random transmissions triggered by a condition. Cool-down period used to enforce request frequency limit. Requests during cool-down period immediately rejected. Page 29

Best Practices - Implementation Time range should never be greater than one day, ideally the default ‘last hour’ option. No need to query any deeper for automated scripts running on schedule. Test as much as possible prior to deployment to ensure proper execution of script. Evaluate error handling by sending requests during cool-down after successful requests. DO NOT handle server errors and failures by immediately sending another request! Excessive requests caused by poor error handling triggers website outages. Instead back off and wait at least 5 minutes before making another attempt. If errors continue STOP script and send bug report or contact DCS support. Page 30

End Website Automation Special thanks to Appalachian State University for the python script used in the message export automation example. Page 31