THE PROS AND CONS OF USING BIG DATA IN AUDITING: A SYNTHESIS OF THE

THE PROS AND CONS OF USING BIG DATA IN AUDITING: A SYNTHESIS OF THE LITERATURE AND A RESEARCH AGENDA Michael Alles Rutgers Business School Glen Gray California State University, Northridge

The pros and cons of using Big Data in auditing Tremendous interest in Big Data by audit practitioners All Big-4 heavily investing in Big Data. For example, Deloitte Chairman and CEO Joe Ucuzoglu writes: “At Deloitte we’re investing several hundred million dollars in data analytics and artificial intelligence with some cutting-edge applications that we really believe differentiate us and our audit approach. When we use these tools, we’re able to get greater coverage. We’re able to more quickly identify risks. We’re able to complete the audit with a higher level of quality and ultimately deliver a greater level of insight to our clients.” PWC doing pilot of 10 audits with Big Data etc. 2

The pros and cons of using Big Data in auditing Academic researchers also very excited Numerous paper in the June 2015 special issue of Accounting Horizons on Big Data. 180 attendees at the September 2015 American Accounting Association conference titled “Accounting IS Big Data”. Many researchers turned away for lack of space. But Articles equally optimistic to the Accounting Horizons Big Data articles (some by the same authors) were published in the past about AI, expert systems, data mining, and continuous auditing. History of technology applied to auditing is not promising. 3

The pros and cons of using Big Data in auditing Recognize the challenges that Big Data poses in the auditing context Using a client’s Big Data as regular part of that client’s financial audits would be a paradigm shift, requiring unprecedented access to proprietary and sensitive client data that are outside tradition data requested during an audit; a significantly higher reliance on nonfinancial data (NFD), which auditors have been reluctant to use in the past because it’s not clear how validate this non-GAAP data; increased technical skills on the part of the audit team; and increased business acumen to determine what data to analyze and how to interpret the results. Just because Big Data adds value for audit clients doesn’t mean it has to do the same for auditors. 4

The pros and cons of using Big Data in auditing Apply some “professional skepticism” to the adoption of Big Data by auditors The objectives of this paper are to: (1) provide a balanced discussion of both the pros and the cons regarding incorporating Big Data into financial statement audits; and (2) present a research agenda to identify specific aspects of Big Data that could benefit auditors. Skepticism is not a negative in auditing, but the positive force from which value is created. Similarly, we bring professional skepticism to bear on the potential role of Big Data in auditing practice in order to better understand when it will add value and when it will not. 5

The pros and cons of using Big Data in auditing An illustrative example Well discussed example of where Big Data Analysis gave rise to misleading conclusion: A fascinating study combining Sandy-related Twitter and Foursquare data produced some expected findings (grocery shopping peaks the night before the storm) and some surprising ones (nightlife picked up the day after — presumably when cabin fever strikes). But these data don’t represent the whole picture. The greatest number of tweets about Sandy came from Manhattan. This makes sense given the city’s high level of smartphone ownership and Twitter use, but it creates the illusion that Manhattan was the hub of the disaster.” 6 Crawford (2013)

The pros and cons of using Big Data in auditing Response of referee to this example “some of the examples given are straw man arguments. For example, the discussion concerning Twitter and Foursquare data creating the illusion that Manhattan was the hub of the hurricane Sandy disaster is suspect—wouldn’t Big Data include an analysis of weather maps?”. How would you respond to this criticism? 7

The pros and cons of using Big Data in auditing Some possible responses to the referee We could have responded to the referee that the Big Data analysis Crawford (2013) cited was trying to assess the damage caused by the storm rather than its intensity, which is why the analysts turned to social media data. We could also have responded that the original study was conducted by a team of computer science researchers and practitioners with likely far better Big Data skills than most accounting researchers or auditors in an engagement. 8

The pros and cons of using Big Data in auditing Would our arguments have made a difference? When the referee says “wouldn’t Big Data include an analysis of weather maps?” he or she is practicing what is properly called “magical thinking”. Like many Big Data proponents who don’t know what Big Data really is, the referee has an almost supernatural belief that Big Data will miraculously include all relevant data and provide perfectly accurate results as a consequence. There is no recognition that Big Data is only a means towards an end and how well it works depends on the choices made by the analyst about what data to include and how that data is analyzed. 9

The pros and cons of using Big Data in auditing Let’s first define what Big Data is Surprisingly there is no consensus on what Big Data is other than it is big—an indication of a still developing concept. 40 different definitions in the literature. Most common definition: Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. (Wikipedia). Such a conditional definition is problematic. Ball and Brown’s (1968) data set falls under that definition. Other definitions focus on characteristics of Big Data (volume, variety, velocity, veracity etc.). 10

The pros and cons of using Big Data in auditing Big Data is much more than transactional data Auditors already have access to large volumes of transactional data from the firms that they are auditing. Hence, a particularly relevant definition of Big Data in the auditing context is that by Connolly (2012) which takes transactions as its starting point: Big Data Transactions Interactions Observations Transactions are highly structured data. Interactions are data about how people and things interact with each other or with the business. Observational data tends to come from the “Internet of Things”—RFID, GPS, CCTV, automated monitoring. 11

The pros and cons of using Big Data in auditing 12

The pros and cons of using Big Data in auditing Is there a need for more data in auditing? Connolly’s (2012) framework is useful because it puts the data currently used by auditors into perspective and shows how much scope Big Data offers to expand that input into the auditing process. What it doesn’t do is to make the case for whether increasing the magnitude of the data upon which auditing is based upon is either useful or feasible. “Extraordinary claims require extraordinary evidence.” Converse is also true: extraordinary evidence only necessary in response to extraordinary questions. Does auditing have such extraordinary questions? 13

The pros and cons of using Big Data in auditing Some caveats on the use of the term “Big Data” in auditing The use of the term “Big Data” by auditors—both on the academic and practitioner side—is not always consistent with its definition in the wider data analytics community. The long reliance of auditors on sampling means that the extension to the analysis of the whole population of accounting transactions may well be considered “Big Data” in an auditing environment. But, despite the AAA’s assertion that “Accounting IS Big Data, even the entirety of a business’s accounting transactions are smaller than what other “Big Data” users have in mind. 14

The pros and cons of using Big Data in auditing Different data or more of the same? When discussing Big Data in an audit context, there is also the need to differentiate between more of the same kind of data that auditors are already using, or more data of a different kind than what auditors have traditionally relied on to give an audit opinion. Continuous Auditing is an example of the former. True Big Data pushes the domain far outwards from financial data to non-financial data—from structured to unstructured data and from inside the organization to outside it—to an extent that may well be outside the comfort zone and technical capability of the audit profession as it is currently constituted. 15

The pros and cons of using Big Data in auditing Potential pros of using Big Data in auditing Pros Comments Strong predictive power, which is a Events/transactions included in Big Data powerful tool for setting expectations can predate accounting transactions by for financial statement auditor. days, weeks, months, and even years. For example, knowing someone is pregnant (by analyzing sales of pregnancy test kits) enables prediction of a permanent change in purchase patterns. Rich data sources to identify potential Difficult fraudulent activities. for upstream fraudster non-financial to change all transactions to cover up financial statement fraud. For example, trade based money laundering can be detected by comparing invoices with the actual weight of shipping containers. Analyzing all data increases Since fraud represents a very small probability of discovering red flags, percentage of transactions and could be “smoking outliers. guns,” and suspicious easily not included in the small samples auditors traditionally select. 16

The pros and cons of using Big Data in auditing Potential cons of using Big Data in auditing Big data problems are open-ended in as much it is hard to know what variables will prove to be correlated with the dependent variable. Type of decisions that auditors are concerned with are constrained by accounting rule and regulations and are not open-ended as the optimization of marketing strategies or the causes of heart disease might be. Even a 1% improvement in drug efficacy or a marketing campaign adds value in a profit driven business setting. Not necessarily the case in auditing. Business can use Big Data to experiment with new correlations to see if they add value. 17

The pros and cons of using Big Data in auditing Audit decision making By contrast, most auditing problems arise in a much more ordered environment, with less independent variables likely to impact upon the problem. There is limited utility to be gained from examining non-transactional data or data from outside the company itself when assessing ICFRS controls, GAAP compliance and other such typical audit problems Even fraud detection is more constrained in its likely contributing factors than a typical market segmentation problem. Need to make the case that there are audit decisions that fit Big Data scope and scale. 18

The pros and cons of using Big Data in auditing Drawbacks of Big Data output What drives Big Data is the ability in a large population of data to find patterns that are not discernible in a sample or in smaller data sets which have fewer variables to drive unexpected correlations. For Big Data to work the approach to data and how it is used needs to fundamentally change. Big data facilitates search for patterns and experimentation with unexpected correlations. Need to recognize that this mindset is easier to apply in a profit context than in compliance. 19

The pros and cons of using Big Data in auditing What Big Data requires from users “Using great volumes of information in this way requires three profound changes in how we approach data: The first is to collect and use a lot of data rather than settle for small amounts or samples, as statisticians have done for well over a century. The second is to shed our preference for highly curated and pristine data and instead accept messiness: in an increasing number of situations, a bit of inaccuracy can be tolerated, because the benefits of using vastly more data of variable quality outweigh the costs of using smaller amounts of very exact data. Third, in many instances, we will need to give up our quest to discover the cause of things, in return for accepting correlations.” Mayer-Schoenberger (2013) 20

The pros and cons of using Big Data in auditing These three effects pose increasing levels of difficulty for acceptance by auditors. Data storage may be cheap, but auditors face increasing risks of privacy breaches as data sets grow in size. Particular concern with information drawn from social networks and interactions that identify individuals outside the auditee itself. “In the past, when people collected only a little data, they often had to decide at the outset what to collect and how it would be used. Today, when we gather all the data, we do not need to know beforehand what we plan to use it for.” Cukier and Mayer-Schoenberger (2013) 21

The pros and cons of using Big Data in auditing Collecting Big Data Collecting data without a need for it may make sense in the case of scientists or internet companies. It is hard to imagine that auditors would feel the same way. Doing so goes counter to the desire of auditors to be conservative, reduce costs and to be able to clearly justify any decisions that they make. More to the point, if the data is to come from the client, then obtaining information at all, let alone with no explanation provided is difficult to attain, as the development of the Audit Data Standard indicates. 22

The pros and cons of using Big Data in auditing Messy Big Data “When we increase the scale by orders of magnitude, we might have to give up on clean, carefully curated data and tolerate some messiness. This idea runs counter to how people have tried to work with data for centuries. Yet the obsession with accuracy and precision is in some ways an artifact of an information constrained environment. When there was not that much data around, researchers had to make sure that the figures they bothered to collect were as exact as possible. Tapping vastly more data means that we can now allow some inaccuracies to slip in (provided the data set is not completely incorrect), in return for benefiting from the insights that a massive body of data provides.” Cukier and Mayer-Schoenberger (2013) 23

The pros and cons of using Big Data in auditing Auditor reaction to messy Big Data While researchers or managers in search of new marketing strategies may tolerate data ambiguity it is probable that auditors fall into that category of people for whom it runs counter to how they have worked with data for many years. Allowing “some inaccuracies to slip in” is difficult to reconcile with the focus in auditing on data integrity. But biggest mindset problem auditors face with the Big Data environment is not with securing large data sets or with the messy data that that they contain. Rather, it is with accepting the kind of results that Big Data provides. 24

The pros and cons of using Big Data in auditing From causation to correlation “These two shifts in how we think about data—from some to all and from clean to messy—give rise to a third change: from causation to correlation. This represents a move away from always trying to understand the deeper reasons behind how the world works to simply learning about an association among phenomena and using that to get things done.” Cukier and Mayer-Schoenberger (2013) Can you get an auditor to sign off on this point of view? 25

The pros and cons of using Big Data in auditing Problems with “correlation is not causation” How about this? “Correlation is enough. We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.” Anderson (2008) Researchers taught that “correlation is not causation” may find this hard to accept. Auditors may be even more reluctant to base conclusions on correlations that cannot be proven to be based on any kind of causality—increases audit and litigation risks. 26

The pros and cons of using Big Data in auditing Value of Big Data driven correlations Great Big Data success story: Google Flu Trends is popularly believed to be more timely and accurate than the CDC. “Because the relative frequency of certain queries is highly correlated with the percentage of physician visits in which a patient presents with influenza-like symptoms, we can accurately estimate the current level of weekly influenza activity in each region of the United States, with a reporting lag of about one day.” Ginsberg et al (2009) 27

The pros and cons of using Big Data in auditing Why auditors may worry about relying on correlation driven Big Data “But as we increasingly rely on Big Data’s numbers to speak for themselves, we risk misunderstanding the results and in turn misallocating important public resources. This could well have been the case had public health officials relied exclusively on Google Flu Trends, which mistakenly estimated that peak flu levels reached 11% of the US public this flu season, almost double the CDC’s estimate of about 6%. While Google will not comment on the reason for the overestimation, it seems likely that it was caused by the extensive media coverage of the flu season, creating a spike in search queries.” Crawford (2013) 28

The pros and cons of using Big Data in auditing Role of theory in interpreting Big Data results “Can numbers actually speak for themselves? Sadly, they can’t. Data and data sets are not objective; they are creations of human design. We give numbers their voice, draw inferences from them, and define their meaning through our interpretations. Hidden biases in both the collection and analysis stages present considerable risks, and are as important to the bigdata equation as the numbers themselves.” Crawford (2013) This is what leads to situations like that of the Sandy analysis. 29

The pros and cons of using Big Data in auditing Need theory to interpret Big Data results “I think what is true is that when you have large amounts of data, if you ask it the right questions, you have a greater ability to let the data speak, and so you can be much less reliant on assumptions. But you still need a strong conceptual framework to understand what’s coming out.” Athey (2013) “The dirty secret of Big Data is that no algorithm can tell you what’s significant, or what it means. Data then becomes another problem for you to solve.” Keltanen (2013) Danger for auditors is taking Big Data correlations at face value and not questioning them sufficiently. 30

The pros and cons of using Big Data in auditing Is Big Data the future of auditing? Big data is undoubtedly a fundamentally important development in the world is perceived and understood. But like with any other highly sophisticated and demanding tool, it is only as good as the way it is applied and its results understood. Cannot simply assume that if Big Data adds value in business that it will also be useful in auditing or that auditors will be compelled to adopt it. What about litigation risk? What happens when auditors are required to use Big Data continually? 31

The pros and cons of using Big Data in auditing Conclusion Point of paper is to illustrate the need to think carefully about Big Data as a means towards and end and not an end in itself. Big Data is becoming an indispensable resource to many companies and has the potential to be an extremely valuable resource to financial statement auditors. But that presumption must not be taken as a given without question, skepticism or further research. Otherwise there is the danger that Big Data will succumb to the same forces that have stalled the adoption of prior technologies equally extolled in their time for the potential to transform auditing practice. 32