The workings of big business and government have come under intense scrutiny in recent times. The Banking Royal Commission revealed multiple instances of malfeasance. The enquiry into Victoria’s hotel quarantine debacle failed to identify who authorised the use of private security firms, despite revealing details of many emails, phone conversations and text messages.
Investigations like these rely on trawling vast repositories of information — electronic and hard copy — in search of evidence. Electronic information is increasing rapidly in volume and variety. There are emails, social media feeds, messaging apps, videos, phone records, and more.
A similar information discovery challenge faces organisations responding to freedom of information (FoI) requests. The time and cost to gather the requested information is rising, along with the difficulty of finding all the relevant information.
Technology is increasingly able to streamline searches for information and even discover important details and events that were never put on record.
Public Access Deputy Commissioner in the Office of the Victorian Information Commissioner (OVIC) Joanne Kummrow knows the challenges of information discovery very well. Her role includes handling complaints in responses to any of around 40,000 FoI requests received annually by Victorian Government agencies and ministers.
She and Mimecast country manager for Australia, Nick Lennon, discussed these challenges and potential solutions to them in an InnovationAus podcast.
Kummrow warned organisations that lack the systems to record information in a way that enables full discovery and retrieval are likely to see their reputations suffer.
“It also doesn’t look good when an organisation, particularly a government organisation, is not across its information holdings and doesn’t really have an idea of what they’ve got.,” she said.
In Victoria part of the problem lies with the FoI legislation — now almost 30 years old. Ms Kummrow said it relies on a ‘pull’ model, in contrast to the newer ‘push’ model of the NSW FoI legislation.
She said that people today expect governments to be open and transparent. “Victoria’s old pull model is starting to get quite creaky around the joints after so many years. Under the NSW FoI legislation, more information has to be made available by government agencies and departments.”
Initiatives to proactively make more information available will ease the burden of information discovery and retrieval but will not alleviate that burden completely.
What’s needed, said Nick Lennon, is automation. “Organisations now have access to eDiscovery platforms and are able to aggregate those requests across different types of data, including email and messaging, to present back Freedom of Information requests quicker.
“But there is a huge discrepancy that organisations running these requests are seeing between the cost of search and the cost of discovery, and the income that may be generated from these requests. There’s a big opportunity for technology to close that gap.”
For example, Mr Lennon said, law firms are now using technology to accelerate their ability to interrogate and make decisions on the vast amounts of data involved in many legal cases.
“We see compute power and indexing playing a very significant role in accelerating workflows, and in designing workflows that allow very specific datasets to be presented so that humans can make decisions around whether that relates to a particular matter or a case.”
Also, artificial intelligence is increasingly coming to the aid of information discovery for FoI and investigative purposes.
“We’re starting to see indexing on all of the different types of data forms that are now so important: video, conversations, CCTV footage, and content coming through IoT led initiatives that government and organisations are having to build into their supply chains,” Mr Lennon said.
“We’ve seen the need for organisations to have discovery across these types of data sets as well. And that leads to AI and machine learning to accelerate the process.”
He said organisations are rapidly adopting these technologies, driven in part by the possibility of being subject to close scrutiny.
The challenges of discovering specific information in huge volumes of electronic data is enormous. And, as the Victorian quarantine enquiry showed, pinpointing a key decision or event in the absence of specific data is an even greater challenge.
However, it’s a challenge technology is starting to solve. In some industries, algorithms are being used to pick up on behavioural trends that could point to unrecorded events.
“AI and machine learning are being used to interrogate data for more than just the content,” said Mr Lennon.
“With algorithms, the way people were conversing, or the style of language being used at a point in time can indicate something could have taken place.
“I see these types of behavioural analytics components being overlaid against data to indicate behaviour that might have taken place at a point in time isn’t normal, and might need to be investigated further.
“That’s definitely taking place in sectors like finance, where they are looking to correlate language before and after transactions to try and paint a clearer picture.”
In this brave new world information that does not go on the record might reveal more than information that does.
Do you know more? Contact James Riley via Email.