Towards a DAM Interoperability Standard
In the early days of the Internet, before the advent of the World Wide Web, organisations hosted repositories of information in a variety of formats. With an appropriate login and a text-only terminal emulator, you could access unstructured filesystems via FTP, or, if you were very lucky, search rudimentary indices using software such as Archie or Gopher. As we know, this all changed seemingly overnight with the advent of the web. The key driver for that change was the adoption of a set of standards which are still evolving today. At their most basic, they described common standards for content (GIF and JPEG for images for example) and context (HTML).
We are at a similar crossroads in the world of digital asset management. Organisations have derived value from organising their internal libraries and workflows using internal digital asset management systems. To unlock further value, and to utilise DAM through an entire asset’s lifecycle, the industry needs to begin adopting a standard for interoperability. In this article, I discuss the requirements for such a standard, examine existing standards and how they might be integrated, and propose a vendor-led approach for implementation.
One of the reasons why an interoperability standard is becoming so important is that during the entire lifecycle of a digital asset, it is increasingly the case that several systems and several organisations are involved. Let’s look at the example of a television ad. In reality, whilst this digital asset ends up as a file that is played out by a TV broadcaster (or, perhaps, as a file stored in your set-top box, PC or mobile device), it begins life as a brief from a brand to an advertising agency. In a more general sense, this could be described as the specification for the asset. In other industries, this could be an outline, a blueprint, a sketch or even a paragraph of text from an email. Inevitably however, this instantiation process crosses organisational boundaries. The digital asset is likely to be commissioned by one organisation, but produced by another.
Once a specification exists, work begins on production of the digital asset. In the early stages of producing a TV commercial, for example, this might consist of scripts, storyboards, mood reels, etc. Then there is the production, or shoot, followed by editing and post production. During these stages there might be rushes or dailies, rough cuts, offline and online edits. Some or all of these might be submitted for internal or external review, feedback, commentary and approval. So whilst the specification might ask for only one or two “final” digital assets, the production process yields a proliferation of temporary, related, subsidiary or component digital assets. In the case of another type of organisation, such as an architectural practice, this process might yield drawings, photography, 3D models and renders, fly-throughs, etc.
During the production process, there may be a complex hierarchy of organisations involved: authors and directors, designers and photographers, editors and finishers. Everyone needs access to different digital assets in the workflow at different times. The issue of version control is critical – time and money will be wasted if someone is working on a version which is not the latest. Managers will be very interested in the overall status of the project, which is able to be inferred from the status of the individual digital assets. Today, there is much human labour involved in maintaining that status, and handling the transition of digital assets across organisational boundaries. With the proliferation of file-sharing systems like Dropbox and YouSendIt, security is also a problem. It is virtually impossible to draw a net around all of the digital assets involved.
Once the production process is complete, and we have in hand our final digital asset, there is then the issue of “deployment”. Depending on the industry, this might mean publication, broadcast, exhibition, delivery or display; the exact nature of the deployment will vary, but there will also be many similar requirements at this stage. Once again, there will be the involvement of more than one organisation. We may need to deliver the digital asset across a network to its final resting place (or multiple places). Almost universally, we will want some sort of monitoring or measurement, in order to determine the level of activity or interest in the final digital asset. In the case of advertising, this is obvious, but it is equally important for organisations such as museums, educational institutions and government bodies to be able to put some quantitative analysis against the effort and budget expended to produce or deploy the final asset. In short, we want to know the ROI – the return on investment.
Globalisation and austerity have each had their impact on ROI and approaches to improving it. Once a final digital asset has been produced and deployed, attention turns to the derivation of further return from this investment. The asset may be localised and translated into other languages; it may be further adapted for different markets. Using our example of the TV commercial once again, it is increasingly the case that many final digital assets are produced from the one production process. Not only will there be a 30 second, a 20 second and a 10 second version of the ad, but increased image quality means there may be a cinema version from the same shoot, a web version, and perhaps some of the imagery may be repurposed for print or digital. This issue of “derived” digital assets inevitably involves complex and difficult sharing processes across international boundaries, between industries and between vertical silos within the one industry.
A digital asset that is “live” has not yet reached the end of its journey, by any means. Consider the issue of revocation. For legal or editorial reasons, or simply because it has reached the end of its publication period, access to the digital asset must be revoked. This can be achieved in a variety of ways, but once again involves crossing organisation boundaries. Consider the example of a photograph where the usage rights expire. Revoking access to any final digital assets that use this photograph is potentially a daunting exercise involving many organisations and significant labour.
Only at the end do we come to the area of digital asset management with which most of us are familiar, and where, historically, the issue of inter-organisation sharing has been limited: the archive. Knowing the value ascribed to a final digital asset, whether it be intrinsic value in the case of a museum artefact, or the value of money and time invested in its production, in the case of a creative or production asset, most organisations want to store the asset safely, and with sufficient metadata that it can later be found, retrieved and potentially re-used. However even here, in most cases, that potential for re-use is dramatically increased if there is a mechanism by which the digital asset can be queried and retrieved across organisation boundaries – as long as it is in such a way that protects the security and commercial rights of the asset owner.
Technical Aspects of a Standard
Taking into account the entirety of the aforementioned life cycle, the challenge of defining a standard for interoperability may appear daunting. However, we can break the problem down into its constituent parts. Most, if not all, of these entities will be familiar to developers of contemporary DAM systems, so the challenge really lies in exposing them securely and reliably with shared nomenclature.
Exchange of Binary Files
It has been said that the definition of a digital asset is “file plus metadata”. If we unpack that “file” part for a moment, and think about what it means to provide the ability to exchange it between organisations, we can see that it is not immediately straightforward. Firstly, the file itself may be a single contiguous binary file (as in the case of a JPEG image, for example), or it may be a package or collection of binary files with a master “control” or “layout” file (as in the case of an Adobe InDesign document with linked images, or an Apple Final Cut Pro file with many tracks of audio, video and text).
Furthermore, since some of these binary files have the potential to be quite large, it is probable that the transfer of said files may happen “out of band”, which is to say via some other network connection than the one via which the inter-system request/response is being sent. There exist content delivery networks, or CDNs, whose job it is to cache files close to the end user and maximise throughput for download. There are also proprietary acceleration technologies such as Aspera and nVerge, which are designed specifically to support the rapid transfer of large files, maximising the available bandwidth. Any putative standard should accommodate asynchronous and out-of-band fulfilment of the binary file component of the digital asset.
Exchange of Metadata
Moving on to the metadata half of the digital asset definition, consider how this might be transferred when an inter-system request is made. It may be returned inline with the response, or it may be delivered via the same method as the binary file, but as a separate, “sidecar” file. Finally, some binary file types have standardised ways of embedding metadata directly within the file – technologies such as XMP and MXF come to mind. This has the advantage that the binary file and its metadata are never separated, but it requires that the host system be able to parse the specific embedding technology.
Once the method of metadata transfer is defined, the metadata structure must, in some way, also be defined. It would be unrealistic to expect all DAM systems to share a common metadata structure, although of course industry-specific metadata standards do exist, and should be referenced within any interoperability standard. At a minimum, it should be possible either within a given request for an asset, or as a separate definition request, to obtain a schema that can be used to map the received metadata to meaningful fields within the receiving system. Such mapping is the responsibility of the receiver, in this case, but the provision of a meaningful definition is within the remit of a standard. XML and XML Schema are obvious candidates here, although the rising use of JSON as a message format should not be ignored; more of this and other existing XML-based standards anon.
The relationships between digital assets in a system are an integral part of those assets’ metadata, and are often difficult to preserve, even in simple backup and archival scenarios, let alone when transferring to and from a foreign system. Whilst it is possible for a DAM system to define any number of relationships between digital assets which have a profound business-specific meaning, most of these are derivatives of four main relationship types: elements, collections, proxies and versions. A standard for interoperability should support at least these four, and possibly a method for defining new named types based on these canonical types.
Consider an InDesign file representing a magazine advertisement. This InDesign file will most likely have links to several files required in order for it to render correctly, including perhaps images (at various resolutions), fonts, vector art, etc. When representing that file in a DAM system, a blunt approach is simply to treat the entire file set as an indivisible package, perhaps zipping it up with a file archiving tool. A more fine-grained and increasingly common approach is to maintain the hierarchy of files, such that the InDesign file is the “parent” digital asset, but each of the linked files is also a digital asset in its own right, linked to the “parent” via an element relationship.
A collection relationship serves to group together digital assets to serve some business need, rather than a structural relationship. Images may be collected into an album or lightbox, for example; film clips may be collected together into a showreel or presentation; multiple advertising digital assets may be collected together into a campaign.
Most, if not all, DAM systems support at least one type of proxy relationship: the ubiquitous thumbnail image. Wherever the master binary file is too large to use in a user interface, or to return in volume across a network, a proxy is used in its place. Simple proxies include a lower-resolution version of a large image file, or a web streaming version of a video file. More sophisticated proxies include multi-page renderings of large documents, or even, for example, complete InDesign files with the print-resolution imagery replaced with screen-resolution compositing versions. In all cases, there is a relationship of type “proxy” where some representation of the file is standing in for the original.
Finally, most digital assets are revised many times throughout their lifetime. Typically one of the benefits of a digital asset management system is that each of these revisions is stored and can be retrieved easily – a complete version history of the digital asset is available. This chain of instances of the asset, stretching back in time, relies upon a version relationship, managed in the background by the DAM system.
Authentication (establishing valid identity) and authorisation (establishing access rights) are fundamental to interoperability between systems of any type, and so perhaps they are out of scope for any discussion specifically pertaining to DAM systems. Issues of single sign-on, session management, etc. are well-covered by standards in those fields. However, within the specific domain of DAM is the issue of inherent rights. A digital asset has rights associated with it that extend beyond the boundaries of the host organisation.
Without wanting to embark on an extensive detour into digital rights management (DRM) and the problems with implementing such a scheme in anything but a consumer environment, it is nevertheless perhaps within the remit of an interoperability standard at least to permit rights to be defined and nominated as applying to given digital assets. The honouring of those rights by the foreign system and organisation must necessarily be handled by some other, probably non-software, trust mechanism.
Let’s consider some examples of interoperability where responsibility must be communicated and shared between the originating and receiving system. A common scenario is where a digital asset must be shared with someone outside the organisation for review and approval. In a non-integrated solution, the reviewer might log into the originator’s system, and then in order to share the asset with a wider audience within their own organisation, they will likely download it and email it around, or upload it to the intranet. In an integrated system observing a common standard, the originator’s DAM software may be able to talk directly to the reviewer’s intranet, such that it correctly observes the expiry of the digital asset at the appropriate time.
Another example of interoperability is when an organisation needs a third party to upload or ingest an asset. Consider a photographer, having their own DAM system, who must then upload a selection of images into a client system either for review and selection, or for final use. Some of these images may have subsidiary rights – such as location rights perhaps – which must be re-entered manually, assuming the client system supports them. In an integrated system, such rights would travel with the asset, supported by the standard.
The fallback position for any security approach is auditing. At minimum, an interoperability standard should support the ability for an asset record to be annotated with an activity log, such that systems can share (where appropriate according to business rules) a common audit trail for exchanged digital assets.
Notifications and Messaging
Of course, exchange of digital assets is only one part of the interoperability picture. To truly share the workflow around the lifecycle of a digital asset, systems must be able to notify one another of changes of state. Furthermore, there may be many potential triggers for notifications in a given system, only a subset of which will be of interest to a third party. This is an obvious candidate for a publish/subscribe design pattern, however within the standard it would also useful to be able to define callbacks, perhaps through the provision of a callback URL.
Consider the example of two digital asset management systems where a digital asset moves from system A to system B in order to be approved by the client at system B. It would be useful for system A to be able to supply a callback URL, to which system B will then send a message when the digital asset has been approved (or rejected), rather than requiring system A to poll system B constantly for changes of state.
Considerable work has been undertaken by various bodies to create standards that cover part of the challenge of DAM interoperability. For example, in print publishing, the Ghent Workgroup (GWG) has produced a set of specifications for PDF files which improve interoperability between creators and publishers. JDF (Job Definition Format) from CIP4 addresses a similar part of the workflow, with a particular focus on proofing. In the video arena, standards such as MXF (Material eXchange Format) have helped to ease interoperability between vendors of editing and broadcast systems, and have also been taken up by some broadcast DAM systems and off-line storage solutions.
In specific industries, work has been done to create standards that address the workflow of those industries. For example, in advertising, AdsML has been developed to support a standard not only for the interchange of advertising materials, but also an e-commerce specification for media buying.
Under the aegis of OASIS (Organisation for the Advancement of Structured Information Standards) – an organisation with standards such as SAML, OpenDocument and UDDI under its belt – HP, Adobe, Microsoft and OpenText, inter alia, have been working together to develop CMIS (Content Management Interoperability Services). Indeed, CMIS could be seen as a sister-standard to a putative DAM interoperability standard, dealing as it does with a subset of the same issues relating to content management systems.
Also currently being developed by OASIS (led by Oracle) is a standard called ICOM (Integrated Collaboration Object Model), whose charter is to “to specify the normative standards for collaboration objects, along with their attributes, relationships, constraints, and behaviour, in an integrated and interoperable collaboration environment”.
All of these standards – and many more that I have no doubt neglected to mention – should be able to be used within the overarching framework of a DAM interoperability standard and should inform the structure of that standard. The master standard should provide for domain-specific, industry-specific and content-specific subsidiary standards to be incorporated, and should provide a concise container architecture which is extensible to support any future standards.
A Vendor-Led Approach
Successful standards need at least two major sponsors – the very definition of “system A” and “system B”. Without the engagement of at least two of the larger DAM system vendors, it is unlikely a DAM interoperability standard would get off the ground. Fortunately, many of these vendors have already collaborated on standards before, only in other areas. For example, the aforementioned organisations behind CMIS are also DAM system vendors.
With the DAM market maturing, both vendors and customers are raising their expectations. Customers who have obtained all the value they can from implementing archive-style DAM systems, or more recently internal workflow-oriented DAM systems, are looking for the next step. Some large customers have embraced integration and have implemented bespoke middleware to enable interoperability, but the cost of such projects outweighs the return on investment in all but the largest organisations.
An interoperability standard, however, can change that. Thanks to standards, I can sit at my computer and browse from website to website without needing to be concerned with what operating system or web server software any of them uses. Imagine a future where digital assets will similarly travel from system to system, supported by an interoperability standard, enjoying end-to-end management throughout their life cycle, without ever needing to be concerned with the underlying DAM software each system uses.