Body
Summary
Are you familiar with the base level Linux commands and you need to access your Data. You've come to the right place, and this is Phase 2.a/3 of the self-guided practice. Within this Phase there are 3 sections in total and you can find them within the Index for the HPC self guided practice.
As always, if you have questions or concerns while going through the learning material, please don't hesitate to stop by Office Hours (Monday's 2-3pm and Thursday's 3-4pm), our HPC related workshops will be linked below or submit a ticket to arc-support@umich.edu.
"Go forth and compute!" - Dr. Charles Antonelli
Environment
High Performance Computing (HPC) on the Great Lakes cluster (non-sensitive data)
Directions
How to access, manage, transfer or restore Data:
Note: Your ability to utilize these Storage options will depend on the specific accounts to which you have access.
- /home/<uniqname>
- Every user with a login account will have access to a home directory that is 80GB in size and cannot be increased due to Advanced Research Computing’s (ARC) policy.
- Helpful commands:
- home-quota
- Outputs your Used and Available space as well as File Count
- pwd
- Outputs your “Present Working Directory”
- /scracth
- Having access to a class, department, or PI account will grant access to a shared space per root account. Scratch by default has a 10TB storage limit and a 1 million file limit.
- Please note: /scratch has an auto-purge policy that means any unaccessed data will be automatically deleted by the system after 60 days and is not backed up.
- Helpful commands:
- scratch-quota
- Outputs your Used and Available space as well as File Count within the scratch space
- Globus
- With Globus, you can easily, reliably and securely move and share data between computers and data storage systems within and between institutions.
- We offer an introductory workshop on Globus, as well as an advanced segment within our Advanced Research Computing session on the Great Lakes Cluster.
- Upcoming sessions can be found at this link.
- Session Slides can be found at this link.
- File Transfer GUI tools and Commands
Please note: Test files copied from Windows have different line endings. Use dos2unix to fix within a terminal. e.g. dos2unix mywindowsfile.txt
- Graphical User Interface Tools (GUI):
- Command Line Tools:
- Data from AWS or Google drive or Dropbox to the Cluster:
Schedule a meeting with us by sending an email to arc-support@umich.edu and we can walk through the process.
- If you are working with a PI or a Lab, you should have access to additional Storage systems depending if access has been granted.
- Requests to be added to a volume will need to be granted from the PI, Project Administrator or Lab Manager via the Resource Management Portal (RMP) or by submitting a ticket.
- Turbo: Active Storage
- Data you are currently working with
- Can be mounted on Windows, Mac or Linux devices
- If mounted to Great Lakes:
- cd /nfs/turbo/lsa-<PIuniqname>
- Data Den: Archival Storage
- Old or Raw data that is currently inactive or awaiting processing
- Can only be accessed through Globus at this time
- Brock Palen created a tool to help users manage their data for archival and restoration purposes
External resources
Links to external documentation should be added here. Please note that if you are linking to other articles within the KB, the "Related Articles" field should be used. If this section is not needed, it can be deleted.