For environmental researchers, the volume of data generated by remote sensing technologies can be massive and challenging to manage. The Joint Remote Sensing Research Program (JRSRP) is a leader in this field, involving experts from academia and government to push the boundaries of what is possible in environmental observation. Their mission is to develop methods to track environmental changes over time, and to inform decisions on land use, agriculture and environmental management.

Addressing data management challenges

Navigating the complexities of data movement is a challenge for the JRSRP, a collaboration that involves multiple institutions, including the New South Wales Department of Climate Change, Energy, the Environment and Water (NSW DCCEEW), the University of New South Wales, the University of New England, as well as The University of Queensland and the Queensland Department of Environment, Science and Innovation.

JRSRP researchers rely on a suite of remote sensing instruments—among them, Sentinel-2 satellites and terrestrial laser scanners (TLS), to collect detailed data on vegetation and land use patterns. A typical research project can generate substantial amounts of data. For example, for a recent project, Sentinel-2 satellite imagery amounts to several gigabytes (GB) of data, and TLS scans reached 5.5 GB per site.

Dr Jim Watson, Principal Scientist for remote sensing at NSW DCCEEW said the cross-institute file transfer methods they were using had limitations, so they needed to find a better way to move these large datasets around quickly and securely.

“With the assistance of AARNet, we tried FileSender, which proved to be the robust data transfer solution we required for the task,” he said.

Leveraging the high-bandwidth AARNet network

With all the JRSRP partner organisations and institutions connected to the high-speed AARNet network, AARNet’s FileSender web application provided an optimal solution. FileSender, with its ease of use and reliability, also offers end-to-end encryption and flexible access control, ensuring data security and integrity.

“FileSender worked better than I initially expected because it benefits from the AARNet network’s ability to handle a high volume of data and scale up and down on demand without any degradation in performance. Also, being able to share and receive data from collaborators without requiring them to create an account significantly simplifies the sharing process,” explained Dr Watson.

Dr Watson and his team have now used FileSender for various projects, including successfully transferring large satellite and field datasets quickly and easily from the NSW DCCEEW high-performance computing facility to the Australian Research Data Commons Nectar Research Cloud, which they use for analysis.

They are in the process of testing FileSender for the purpose of distributing field data visualisations, a task that involves transferring more than 40 GB of drone-based LiDAR (Light Detection and Ranging) data to the Nectar Cloud. With FileSender, they can complete these large data transfers in just a few hours, significantly faster than the time it took with their former methods of data transfer.

Optimising research data transfers

Adopting FileSender allows program participants to enjoy hassle-free transfers without using up the storage quotas of their corporate cloud services, since it operates independently of their internal systems. In addition, FileSender streamlines the management of user accounts by allowing the use of existing institutional login credentials, eliminating the need to create a new account for the service.

The JRSRP's integration of AARNet’s FileSender into their data movement workflow demonstrates the vital role robust and reliable digital infrastructure plays in data-intensive scientific collaborations.

Photo: A terrestrial laser scanner (TLS) taking readings at a remote site. Credit: Joint Remote Sensing Research Program.