{"id":12060,"date":"2024-02-17T00:19:49","date_gmt":"2024-02-16T23:19:49","guid":{"rendered":"https:\/\/www.swissfluxnet.ethz.ch\/?page_id=12060"},"modified":"2024-02-17T00:19:49","modified_gmt":"2024-02-16T23:19:49","slug":"upload-icos-header-file","status":"publish","type":"page","link":"https:\/\/swissfluxnet.ethz.ch\/index.php\/documentation\/upload-icos-header-file\/","title":{"rendered":"Upload ICOS Header File"},"content":{"rendered":"

If the variables in an ICOS file changes, e.g. the sequence of variables or when new variables were added or variables removed, it is necessary to upload a new header file via the ICOS PI area<\/a>. It is necessary to submit information via the BADM system, for more info see the BADM FAQs here<\/a>.<\/p>\n

In this example, we update the header information for the changed 10_meteo file we transfer to ICOS every day. In this file, some variables were recently removed, therefore this info has to be given to ICOS by uploading a new header file.<\/p>\n

Steps<\/h3>\n
    \n
  1. Assign a new file number before the first file with the new structure is transferred to ICOS.<\/strong>
    \nThis is necessary because the sequence of variables in the file has changed. The new file number has to be assigned in the Python script pp-icos<\/code> that converts our ETH files to the ICOS format. pp-icos<\/code> is running automatically on a virtual PC, currently only LH has access. To find out which file number is free for this logger L02<\/code>, check the table on the SFN HP     here<\/a>.<\/li>\n
  2. Download the newest file that was transferred to ICOS.
    \n<\/strong>In this example, the newest file is \\\\grasslandserver.ethz.ch\\gl-processing\\CH-DAV_Davos\\01_ICOS_TRANSFER\\10_meteo\\2020\\04\\CH-Dav_BM_20200402_L02_F03.zip<\/code>
    \nIn this step it is important to use a file that already went through the post-processing script, because during post-processing variable renaming for ICOS is done (not always, but for some variables). For example, the original LoggerNet raw data files add suffices to variable names, such as _Avg<\/code>. The _Avg<\/code> suffix is not allowed in ICOS and is therefore removed from the variable names in our raw data files before the files are uploaded to the Carbon Portal for ICOS.<\/li>\n
  3. Unzip the file.<\/strong>
    \nSince this is a ZIP file, unzip the file to get the CSV file:CH-Dav_BM_20200402_L02_F03.csv<\/code><\/li>\n
  4. Open the CSV file in a text editor.<\/strong>
    \nDo not use Excel to open the CSV file, since it may change the info contained in the file when saved via Excel. The best way is to use a text editor to open and display the file.<\/li>\n
  5. The first few lines and columns in the file looks like this:\"\"<\/li>\n
  6. Delete all rows except the first row from the file.<\/strong>
    \nThe ICOS header file must only contain the header information, all the data and also the second header row of the file are not needed.<\/li>\n
  7. The file now contains only one row (the first row) that contains the header information and should look like this:
    \n\"\"<\/li>\n
  8. Save the file following the ICOS guidelines for header files.
    \n<\/strong>The name of the header file in this example is CH-Dav_BMHEADER_202004020000_L02_F03.csv<\/code>
    \n– CH-Dav<\/code> is the official site identifier
    \n– BMHEADER<\/code> gives the info that this is a header file
    \n– 202004020000<\/code> is the start date and time from when the info in the file is valid
    \n– L02<\/code> is the ICOS logger number, i.e. the logger that records the file
    \n– F03<\/code> is the ICOS file number for this logger<\/li>\n
  9. Upload the file.<\/strong>
    \nGo to the     ICOS PI area<\/a> > Button UPLOAD DATA > Drop-down menu Select the site: CH-Dav > Drop-down menu Select data type: Header file > SELECT A FILE > Select the header file that you created > Button SUBMIT DATA<\/li>\n
  10. Register the file in the ICOS PI area.<\/strong>
    \nGo to the ICOS PI area > Button BADM ON LINE TOOL > Drop-down menu Select site: CH-Dav > DATA RECORD GROUP > FILE MODEL > Button Add new record > Enter the file information.
    \nIn this example:
    \n\"\"<\/li>\n
  11. Update the variable info in the ICOS PI area.
    \n<\/strong>Go to the ICOS PI area > Button BADM ON LINE TOOL > Drop-down menu Select site: CH-Dav > METEO MEAS. GROUP > Button Add new record > Enter the updated variable information.Some changes can easily be done with the BADM ON LINE TOOL. However, if there are many variables and for the sake of traceability, it is recommended to upload BADM info using the     BADM templates<\/a> (Excel files) and upload the files via the PI area.<\/strong><\/li>\n
  12. Update the info on the SFN HP.<\/strong>
    \n– Update the file info in the table     here<\/a>.
    \n– Update info about the variables in the file in this table     here<\/a>.<\/li>\n
  13. Now ICOS has the newest info for the meteo files.<\/strong>
    \nCheck the daily error reports if there are issues that need to be addressed.<\/li>\n<\/ol>\n

     <\/p>\n","protected":false},"excerpt":{"rendered":"

    If the variables in an ICOS file changes, e.g. the sequence of variables or when new variables were added or variables removed, it is necessary to upload a new header file via the ICOS PI area. It is necessary to submit information via the BADM system, for more info see . . . Read more<\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"parent":11817,"menu_order":34,"comment_status":"closed","ping_status":"closed","template":"template-full-width.php","meta":{"_acf_changed":false,"_lmt_disableupdate":"","_lmt_disable":"","ngg_post_thumbnail":0,"footnotes":""},"coauthors":[314],"class_list":["post-12060","page","type-page","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/swissfluxnet.ethz.ch\/index.php\/wp-json\/wp\/v2\/pages\/12060","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/swissfluxnet.ethz.ch\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/swissfluxnet.ethz.ch\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/swissfluxnet.ethz.ch\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/swissfluxnet.ethz.ch\/index.php\/wp-json\/wp\/v2\/comments?post=12060"}],"version-history":[{"count":1,"href":"https:\/\/swissfluxnet.ethz.ch\/index.php\/wp-json\/wp\/v2\/pages\/12060\/revisions"}],"predecessor-version":[{"id":12061,"href":"https:\/\/swissfluxnet.ethz.ch\/index.php\/wp-json\/wp\/v2\/pages\/12060\/revisions\/12061"}],"up":[{"embeddable":true,"href":"https:\/\/swissfluxnet.ethz.ch\/index.php\/wp-json\/wp\/v2\/pages\/11817"}],"wp:attachment":[{"href":"https:\/\/swissfluxnet.ethz.ch\/index.php\/wp-json\/wp\/v2\/media?parent=12060"}],"wp:term":[{"taxonomy":"author","embeddable":true,"href":"https:\/\/swissfluxnet.ethz.ch\/index.php\/wp-json\/wp\/v2\/coauthors?post=12060"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}