The University of Melbourne’s eScholarship Research Centre faced moral, legal and ethical questions when it began digitizing its archival collections and making them available on the web.

The Centre felt strongly that it was their societal obligation to take their digital archival materials to the researcher, rather than holding on to the traditional archival practice that required the researcher to come to the archive.

The challenge was that the archival materials weren’t created for publication explains Associate Professor Gavan McCarthy, Director of the eScholarship Research Centre.

“The materials needed to be managed responsibly and with care, because they represent traces of the events they document rather than standing by themselves as meaningful information sources,” A/P McCarthy said.

“What’s more, archival materials are also unconstrained, often containing personal information, commercial-in-confidence information, information relating to national security and other sensitive or secret information.”

Towards a virtual reading room

Traditional archival reading rooms provide due diligence: they stipulate the terms and obligations that accompany access to privacy and rights-compromised materials, and they require a researcher to identify themselves before they access the materials.

Open web publication affords no such opportunity for these rules and conventions. Attempts by other archives to review, redact and restrict what could be disseminated and to whom, was seen as highly subjective and ultimately problematic on many levels.

“As a human-intensive activity, controlling access to the materials seemed like a poor use of the limited archival budget, and so we began to conceptualise what a virtual reading room might look like,” A/P McCarthy said.

The eScholarship Research Centre already had in place well-developed archival documentation and digital imaging processes with associated web-based discovery systems (finding aids) and the allied web-based display systems that could be used for publishable archival materials.

What they lacked was a user-activated digital-delivery system to replace the manual, ad-hoc service being used to deliver the digitised materials as an email attachment, or on an external hard drive or USB stick depending on the scale or quantity.

FileSender as a delivery system

After some investigation, the FileSender component of CloudStor became the Centre’s preferred delivery mechanism.

“This was primarily because of its ability to document the process of transfer – both sender and receiver had email documentation that could be used as a record of the transaction.

“The emails to researchers could articulate the conditions and obligations associated with the transfer, and the archive could receive confirmation that the researcher had downloaded the materials.”

So FileSender was adopted and began to be used instead of external storage. But the eScholarship Research Centre were still spending significant time responding to requests.

“We wondered if we could automate the process and have it instigated by the researcher. The productivity gains were tantalizing.”

A user-activated solution

Using the FileSender API – which was just being developed (the eScholarship Research Centre was the first user) – they were able to do just this by creating the Digital Archive Delivery Service.

Using the Digital Archive Delivery Service, a researcher can browse the web, identify and select the metadata record documenting the material they require, and request it using a link.

The user is then transferred to an online form that explains the process and, importantly, the obligations associated with being delivered a copy of the materials.

If happy to proceed, they complete a form establishing their identity – as a minimum a valid email address is required – and select submit.

If successful, the researcher receives an email that reiterates the conditions and obligations associated with possessing a copy of the unique URL for transfer of the materials. They then activate the URL and download the material.

Behind the scenes, there are three interlinked components to the Digital Archive Delivery Service:

1. Discovery: The front end web-discovery environment, web-indexed html documents and information discovery systems that enable researchers to find the archival materials
2. Brokering: A bridging or brokering module called the Digital Archive Delivery Enabler, which is activated by a hyperlink and negotiates the connections between the researcher, the archive and the delivery service. As soon as the researcher submits their request, the Digital Archive Delivery Enabler batches and loads the requested files into CloudStor using the FileSender API.
3. Delivery: The CloudStor FileSender delivery service: CloudStor FileSender send the notification to the researcher that reiterates the obligations and, if happy to comply, they can download the files.

The request happens in a matter of seconds because CloudStor operates on the AARNet network, which provides high-speed connectivity between the eScholarship Research Centre and all Australian universities and research institutes.

From weeks to minutes

Now at proof-of-concept stage, the Digital Archive Delivery Service has transformed the process of bringing digital archival materials to the researcher.

“The entire process, from request to delivery, takes a matter of minutes, where it could previously take days, if not weeks.

“Responding to requests also typically involves no human intervention, saving considerable time spent on administration for our staff,” A/P McCarthy said.

The source code and documentation for the proof-of-concept digital Archive Delivery Service Enabler is available on GitHub – useful for those interested in what happens technically through the use of FileSender API e.g. get/send commands.

Image credit: From ‘The Records of Thomas Hogan’ in the custody of the eScholarship Research Centre, University of Melbourne [Project Code: HOGA]. Photographer Gavan McCarthy. Page, with photograph with the inscription “the Expert” from the file ‘Arial Spraying’ [HOGA00011]; Page from the folder containing ‘T.W. Hogan’s notebooks’ [HOGA00017].