Retrieve a large number of model outputs using Globus within CyberGIS-Compute

Author(s): Admin

This notebook demonstrates how to use Globus within CyberGIS-Compute to retrieve a large number of outputs generated by a model executed on HPC, which is often needed for postprocessing work performed on CJW. A new “data transfer” job type is provided for moving data from HPC back to the CJW Jupyter environment. Under the hood, this new job type utilizes the Globus service (https://www.globus.org/) to perform a point-to-point data transfer between HPC and CJW. In this demo, we will first prepare a 60-member ensemble SUMMA mode and submit it to the XSEDE Expanse HPC for execution using the CyberGIS-Compute. When the model run is finished, we won't use the regular "download" function in the Compute SDK to retrieve the results. Instead, we submit another Globus job to the Compute, which will hand it off to the Globus scheduler and monitor the process (just like talking Slurm scheduler on HPC in the case of regular model submission). Please refer to the example notebook below for more details.

Keywords: CJW, cybergis-compute, Globus, HydroShare

Posted by

zhiyuli

CyberGIS Center @ UIUC

118 0 0

Launch on CyberGIS-Jupyter for Water

(for viewing purpose only)

Launch on CyberGIS-Jupyter for Water

LEAVE A COMMENT Cancel reply

Name and email are required. Your email will not be published.

Please provide a username.

Please provide a valid email

Save my name and email in this browser for the next time I comment.

Please input your message.

Terms of Use

The purpose of this website, https://cybergisxhub.cigi.illinois.edu/ is to provide information to You related to “CyberGISX Project” (“Project”). This Project aims to enable open access and sharing of computation- and data-intensive geospatial analytics, under Apache 2.0 license agreement. The initial version of the Project (“Original Project”), was developed by the CyberGIS Center for Advanced Digital and Spatial Studies (CyberGIS Center) and is solely owned by The Board of Trustees of the Illinois of Illinois. The Site and its content is made available by the Board of Trustees of the Illinois of Illinois (“Illinois”, “we”, “our”, “us”), on behalf of the CyberGIS Center. The Site is owned by the Illinois.

By using the Site, You agree that You have read and accept the terms of use as presented below (“Terms”). The Terms apply to You because You (“End User”) are using, contributing to or installing the Project, or otherwise accessing the Site. Please read these Terms carefully. The Terms govern Your use of the Site and any incorporated content (such as Illinois or third party software, text, data, information, or graphics) made available through the Site. By continuing to use this site, You accept these Terms.   If You are not willing to be bound by these Terms, please stop using this Site. 

1. Governing Terms:

The Terms are governed by Illinois’s web privacy notices and policy links presented here:

https://www.vpaa.uillinois.edu/resources/web_privacy and

https://www.vpaa.uillinois.edu/resources/terms_of_use/.

2. Intellectual Property:

You hereby accept that all intellectual property, including copyrights, and other proprietary rights in or related to Original Project and Site are, and will remain, the exclusive property of Illinois, whether or not specifically recognized or perfected under applicable law. As End User, You will not take any action that jeopardizes Illinois’s proprietary rights.

3. Credit and Attribution.

You agree to credit and attribute the authors and creators of Project that You use with the copyright notice or statement of credit/attribution as customarily acceptable in industry.

4. Limitations:

In no event shall Illinois or its employees be liable for any damages (including, without limitation, damages for loss of data or profit, or due to business interruption) arising out of the use or inability to use the content on Site, even if Illinois or a Illinois authorized representative has been notified orally or in writing of the possibility of such damage. Because some jurisdictions do not allow limitations on implied warranties, or limitations of liability for consequential or incidental damages, these limitations may not apply to you.

5. Modification of Site and Terms.

Illinois reserves the right to modify the Site and may discontinue, temporarily or permanently, the Site for any reason, at its sole discretion, with or without notice to You. Illinois may likewise change the Terms from time to time with or without notice to you. You agree to review the Terms periodically to ensure that you are aware of any modifications. Your continued access to the Site after the modifications have become effective shall be deemed your conclusive acceptance of the modified Terms.

6. Termination.

If you violate or fail to comply with any of these Terms or in Illinois’ sole discretion restrict or inhibit any other user from using or enjoying the Site, Illinois may terminate your access to the Site account without notice to You.

CyberGISX Project Contributor License Agreement

Version 1.0   February 9, 2021

Thank you for your interest in contributing to the CyberGISX Project. In order to contribute, you will need to provide your name and contact information and sign this CyberGISX Project Contributor License Agreement, which sets the terms and conditions of the intellectual property license granted with your contributions.

This CyberGISX Project Contributor License Agreement (“Agreement”) is by and between you, any person or entity (“You” or “Your”) and the Board of Trustees of the University of Illinois, a body corporate and politic of the State of Illinois (“Illinois”). Please read this document carefully before signing and keep a copy for your records. By signing this Agreement or making a “Contribution” to the “CyberGISX Project” as defined below (even if You do not sign), You agree to the following:

1. “CyberGISX Project” is an open-source software project that aims to enable open access and sharing of computation- and data-intensive geospatial analytics with computational reproducibility and transparency support available under Apache 2.0 license agreement: https://www.apache.org/licenses/LICENSE-2.0

2. “Contribution” means all of Your contributions of object code, source code, content, and documentation, and any modifications thereof, to the CyberGISX Project.

3. “Licensed Patents” mean patent claims licensable by You which are necessarily infringed by the making, using, selling, offering for sale, having made, import, or transfer of either Your Contribution alone or when combined with the CyberGISX Project.

4. “Representation.” You represent the following to the best of your knowledge:

a. You are at least 18 years of age and have full power and authority to enter into this Agreement and to grant the rights in and to the Contribution as set forth herein (individuals who are under 18 years of age and who wish to contribute to the CyberGISX project may not enter into this Agreement, but may contact CyberGISX at cybergis@illinois.edu to explore alternatives);

b. If your employer has rights to intellectual property that You create as part of the Contribution, You represent that You have obtained permission from Your employer to make the Contribution on behalf of that employer, or Your employer waived any rights in and to Your Contribution, or Your employer authorizes the Contribution and agrees to be bound by the terms herein by signing as an entity below;

c. That either:

1. all documentation and code in the Contribution is Your original work and includes complete details of any third-party license and any other restriction (including, but not limited to related patents and trademarks) of which You are personally aware and which are associated with any part of Your Contributions; or

2. any part of the Contribution that is not Your original creation is submitted to CyberGISX separately from any original Contribution, includes the complete details of its source and any corresponding license and any other restriction (including, but not limited to related patents, trademarks, and license agreements) of which You are personally aware, and is conspicuously marked as "Submitted on behalf of a third party: [named here]".

d. That Your Contribution does not include any viruses, worms, Trojan horses, malicious code or other harmful or destructive content;

e. The CyberGISX Project delivered under this Agreement may be subject to U.S. export control laws and may be subject to export or import regulations in other countries.  You agree to comply strictly with all such laws and regulations and acknowledge that You have the responsibility, at Your own expense, to obtain such licenses to export, re-export, or import as may be required; and

f. Your Contribution does not include any encryption technology and no government license or permission is required for the export, import, transfer or use of the Contribution.

5. Notification. Your representations are accurate, and You agree to notify Illinois of any facts or circumstances of which You become aware that would make any of Your representations inaccurate in any respect.

6. License grant to Contributions. You grant to Illinois and to recipients of the CyberGISX software (collectively, “Recipients”), a perpetual, irrevocable, non-exclusive, worldwide, royalty-free unrestricted license to use, reproduce, prepare derivative works of, publicly display, publicly perform, distribute, and sublicense the Contribution, and such derivative works, in source code and object code forms.

7. License grant to Patents.

8. No support. Except for the rights granted to Recipients above, You reserve all right, title, and interest in and to Your Contribution. You are not expected to provide support for your Contributions.

9. No Warranties. Subject to Your representations above, Your Contributions are provided on an “AS-IS” basis WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND (express or implied), including, without limitation, any implied warranty of merchantability and fitness for a particular purpose, and any warranty of non-infringement.

10. No Liability. Illinois, its trustees, directors, officers, employees, students, and agents have no liability for any infringement of any copyright, patent, or other right of third parties in connection with any Contributions, the CyberGISX Project, or CyberGISX software, and are not liable for any direct, indirect, punitive, special, incidental, consequential, or exemplary damages arising in connection with any Contribution, the CyberGISX Project, or CyberGISX software.

11. Indemnity. You agree to indemnify, defend, and hold Illinois and its subsidiaries, affiliates, officers, agents, employees, partners, licensees, and licensors harmless from any claim or demand, including but not limited to reasonable attorneys’ fees, made by any third party due to or arising out of Contributions You submit, post, transmit or otherwise make available through the CyberGISX Project, as well as Your use of the CyberGISX software; connection to the CyberGISX Project; or violation of any rights of another.

12. You agree that Illinois may assign this Agreement and CyberGISX Project to any third party without notice or consent.

13. Illinois is under no obligation to accept or include Your Contribution to the CyberGISX Project.

14. This Agreement is governed by the laws of the State of Illinois, excluding its conflict of laws provisions.

Retrieve a large number of model outputs using Globus within CyberGIS-Compute

zhiyuli

Launch on CyberGIS-Jupyter for Water

Launch on CyberGIS-Jupyter for Water

LEAVE A COMMENT Cancel reply

Redirecting to CILogon in 10 seconds.