In 2003 the National Archives of Australia (NAA) established the Australian Digital Recordkeeping Initiative as part of an ongoing program which commenced back in the early 1990’s with the release of the culmination of effort by numerous parties including the NAA, the State Records Authority of New South Wales (SRANSW) plus many other government agencies, private industry, professional associations and Standards Australia in the release of the then revolutionary Records Management Standards, AS4390.
Prior to the 1990’s here had been increasing unease in the archive profession and its governing responsible bodies located in State and Nation Archive institutions and professional associations nationally and internationally with the apparent inability to capture and preserve born electronic information in an electronic archive environment. This realisation became a cry for help and a call for action in the early 1990’s and possibly before this time by historians and those persons responsible for preserving the history of nations and lesser entities which go to make up the history of the nation.
With the introduction of very primitive personal computers [PC’s] with the release of the Apple 1 back in 1976 and probably of more importance the introduction of the IBM Personal PC in 1981 a vacuum in the capture and preservation of government, corporate and personal data, information, and records created electronically occurred as the information was retained within these devices and died with them as they were recycled or sent to the tip.
Dr Andrew Wilson – Project Manager – Managing Digital Records for Access (MADIRA) of the NAA (also known as the Archives) advises that the long-term preservation of electronic information is, of course, only one component of a comprehensive approach to managing digital records, but nevertheless it is one of the central issues faced by institutions responsible for preserving access to digital objects over time. Dr Wilson believes that it is important to remember that preservation fits within a broader framework of recordkeeping. So, the Archives activities in the area of digital records preservation need to be seen in the wider context of a developing approach to managing digital records. As in any developing policy the Archives’ position has changed over time. In the mid-1990s the Archives adopted a controversial policy of distributed custody for digital records. This policy reflected the Archives’ view at the time that the best way to preserve electronic records of permanent archival value was to ensure that Australian Government agencies implemented best practice in electronic recordkeeping. The Archives’ role was to enable the adoption of best practice recordkeeping through the setting of standards and provision of high-quality advice. Throughout the mid to late 1990s the Archives’ continued to be closely and actively involved in developments in electronic recordkeeping in the Australian and international recordkeeping communities.
The initial work carried out by the Archives with AS4390 continued to provide input into the creation of the international records management standard, ISO 15489 released internationally in September 2001. By the late 1990s, beginning in 1998, the Archives embarked on an ambitious research and development program which culminated in the release, in March 2000, of the e-permanence framework. This release also signaled a change in the Archives distributed custody policy, brought about by advances in the Archives’ understanding of electronic records management issues and the increased availability of innovative technology. The new custody policy released in 2000 gave an in-principle undertaking by the Archives to accept custody of all electronic records that are appraised as having archival value. The change in custody policy initiated the development of a digital preservation program in the Archives.
The first stage of the program, which commenced in mid-2001, was the development of a conceptual understanding of electronic records using previous insights of the Archives to do with the importance of information as evidence which is significant over time, not the form of the object. This work led to the ‘essential performance’ model which is fully described in the Archives green paper: “An Approach to the Preservation of Digital Records” (2002): http://www.naa.gov.au/recordkeeping/er/digital_preservation/summary.html.
The green paper outlines the approach which has been adopted by the Archives: records are accepted into NAA custody in a variety of formats and then converted (‘normalised’) into appropriate long-term formats which can be maintained and made accessible over time. Following the publication of the green paper the Archives started work on developing tools to realise its vision and to enable it to implement the approach. The first of these, and probably the most significant, is Xena, an open-source software application for normalising and viewing digital records. The Xena source code is available from the open-source software site, SourceFORGE, at: http://sourceforge.net/projects/xena/. The digital preservation project is now in the process of being turned into an operational area within the Archives and the digital preservation process should be fully operational by the middle of 2005. Because of the very practical nature of the work being undertaken by NAA, many other archival institutions both in Australia and overseas have looked to the NAA products when developing their own approaches to digital recordkeeping and preservation. In recognition of this the Archives established in 2003 the Australian Digital Recordkeeping Initiative, a coalition of the Australian and New Zealand archival institutions which aims to consolidate and further develop a common approach to digital recordkeeping, including preservation.
The Archives have not been alone in their quest to provide a platform for the capture and preservation of electronic information. Other parties such as the Public Record Office Victoria (PROV) Victorian Electronic Records Strategy (VERS) within Australia and internationally in North America the National Archives & Records Administration (US) (NARA) with its interaction with the San Diego Supercomputer Centre’s National Partnership for Advanced Computational Infrastructure and in Europe the UK National Archives are all attempting to provide a viable solution for the preservation of electronic data, information and records. Dr Wilson advises that NARA does not yet have anything in the way of an explicit approach – that is what their two tenderers are in the process of developing. But there is nothing concrete that anyone can look at and say – this is the NARA approach. So, the Archives cannot make comments on the viability of their approach or otherwise, but the Archives would note, in passing, that the scale of their digital preservation problem is so much larger than ours at the NAA so what works for us might not work in the US context, and vice-versa. Nor is there any necessary reason to think that our different approaches (if indeed they do turn out to be different) should work in other jurisdictional contexts. So, the Archives want to make quite clear that they do not believe that it is appropriate for NAA to criticise other approaches. But the Archives are perfectly happy to say why they adopted their own approach in comparison with others.
As for the NAA and VERS approaches, they are remarkably similar in that they both focus on normalising records into XML – based archival formats. At the Archives they believe that their approach is preferable to the VERS approach for several reasons, but the Archives want to emphasise that they are not really that dissimilar. They think it’s important to retain various features of digital records that are not possible to keep with the VERS formats (PDF, TIFF, and plaintext). Our “essential performance” model allows us this flexibility and means that they do not specify a limited range of acceptable data formats for transferred records, as the VERS approach does. Also, the federal government environment is different from the Victorian government environment, and means that the Archives could not mandate particular acceptable formats as the PROV is able to, even if they wanted to adopt such a mechanism for limiting transfer formats. The VERS approach is still in the early days of implementation (only 2 departments are currently VERS compliant) and the Archives approach will not become operational until next year, so it is still a matter of seeing whether time proves that our approaches are viable.
The other approach the Archives might discuss is the one adopted by the UK National Archives. Their current approach is to migrate records from the proprietary formats in which they are created into new versions of the formats as the previous ones become obsolete. This is certainly an attractive solution in some regard, but the Archives do not believe that it would be a viable approach for them in the long term. When the Archives were developing their own conceptual approach, they did examine other approaches, including migration. They came to the conclusion that migration was not an approach that they wanted to use for various reasons: – some attributes of the original record are lost during the conversion process resulting in a different performance, while their aim is to maintain as closely as possible the original performance; – since the formats are proprietary ones they do not have any control over what is or isn’t lost in the conversion, unlike the case in the Archives approach; – migration requires significant resource commitments to a cyclical process of converting records from obsolete formats (NB: This is set out in fuller detail in the green paper referred to previously in this article). This does not mean that the migration approach won’t work in the UK context, or that the Archives do not approve of this approach. All the Archives can say is that for several reasons that seemed convincing to us, the Archives does not want to follow this particular approach.
In any case, the Archives want to insist that none of us claim that they have “the solution”. the Archives only say that they have what they think is a viable approach to the problem of preserving digital records for the long term. It might prove to be the case that they (archival institutions dealing with the problem of long-term preservation of digital objects) need to use all the approaches for various parts of their collections. It might eventually turn out that no one approach is enough to preserve all records for the long term.
Only time will tell.
Dr Wilson advises that the solutions being researched and implemented at this time are designed to address electronic records created today and into the future. At this point in time, it is not envisaged to go back to the 1970 or the 1980’s and attempt to capture and preserve all the electronic records created since that time.
Dr Wilson doesn’t believe that the approaches currently under development are being designed to address the issues of legacy electronic/digital records. These are quite a different matter from the digital records being created today. Custodial institutions will need to develop different strategies for dealing with such records. In his view the gap between today’s digital preservation capabilities and the ability to preserve legacy formats will always exist. There are a couple of reasons for this: the impermanence of the media used to carry and store electronic records, and the wide range of operating systems and software applications that were used. The first step that needs to happen in order to try to fill the gap is to transfer data from the legacy media onto modern media, such as CDs and DVDs. Not that either of these are archival storage media, but at least such a transfer will allow modern computers to access the legacy data. The second necessary step is to interpret the data and then to maintain this in an appropriate archival data format. A significant issue with legacy data is the disappearance of the hardware that is able to mount and read the legacy media – an urgent reason to attempt media conversion as soon as possible.
At the NAA they have been running a project to recover the legacy electronic records in their custody. This has involved working with a private sector data conversion company to firstly transfer data from the legacy media to high-quality CDs. The second step in this project is to attempt to read and interpret the data formats which have been recovered through the media conversion processes. The Archives have been pleasantly surprised by the very high success rates they have had with the first stage of the project – data recovery is in the order of 92%.
They are about to begin the second stage of the project, attempting to read and normalise the recovered data, so the Archives can’t yet comment on the overall outcomes of the project. They are hopeful, however, that a high proportion of the data will be intelligible and that they will eventually be able to normalise it. Their experience may not be typical, however.
The quantity of legacy records in the custody of the Archives is small, and the number of legacy media types they needed to convert was also small. The majority of the legacy media were various size floppy disks, and these did not present much of a problem to transfer to new media. They do not have the problem of facing several archival institutions and other collecting bodies, of having extremely large numbers (in the hundreds of thousands, if not millions) of magnetic tapes that need to be converted to modern media. In some cases, the sheer quantity of the legacy media involved may mean that the conversion task is not physically possible within the remaining life of the magnetic material.
The rather gloomy prediction that maybe the 1970’s to the present legacy data may be lost to future generations if the data was not printed to paper and preserved, may in fact be true in some jurisdictions.
Laurie Varendorff ARMA
The Author
Laurie Varendorff, ARMA, a former RMAA Western Australia Branch president & national director, has been involved in records management and the micrographic industry for 37 years. Laurie has his own microfilm equipment sales & support organisation – Digital Microfilm Equipment – DME – and a – records & information management – RIM – consulting & training business – The Varendorff Consultancy – TVC – located near Perth, Western Australia, & has tutored & written course material in recordkeeping & archival storage & preservation for Perth’s Edith Cowan University – ECU. mobile: +61 417 094 147; email @ Laurie Varendorff
Please Note: This article was first published in the publication – Image and Data Manager – January / February 2005 Edition on page (26) Records Management. Available @ Well Preserved
The author, Laurie Varendorff gives permission for the redistribution or republishing of this article by individuals and nonprofit professional organisations without cost based on the condition that he, as well as the URL of the article are recognised at the introduction of the article when redistributed or republished.
SPECIAL NOTE: Use of this article by publishers, commercial, government, or educational organisations requires a financial agreement to be negotiated with Laurie as the copyright holder for this work.