At some point, human intelligence and machine intelligence will converge in a meaningful fashion. Currently it’s a moving target.

Although employees have been interacting with versions of AI through voice assistants, animation, and facial recognition software for years, AI’s progress in the second half of 2023 has been mind-blowing. With new enhanced capabilities and understanding, a meteoric rise in AI’s popularity has ensued, and the recently new generative AI services are quickly becoming essential tools for business users of all kinds.

The rapidly changing AI landscape is making the CISO role increasingly challenging, with pressure from the business to use AI to increase productivity while ensuring the company’s safety and security. Generative AI has the possibility to revolutionize our life and work, and companies are scrambling to not be left behind by their competitors in this shift. Current AI models enable companies and employees to automate tasks, generate reports and even create and modify code. These capabilities can significantly increase productivity, both individually and for the whole organization. Generative AI, including ChatGPT, is here to stay – and forcing shutdowns of this technology, like some have suggested, is not a viable option.

Risks arise from the fact that our legal and regulatory landscape surrounding generative AI is still nascent. Consequently, we may not be aware of all the legal requirements that we need to comply with when using this technology. Additionally, there are big concerns around data privacy, intellectual property rights and liability issues.

Ethical risks relate to the potential for generative AI to be used in harmful or discriminatory ways. For example, there are concerns around bias in training data sets, leading to synthetic data not representative of the real world or high moral standards. There is also a risk that generative AI could be used to create deep fakes or other forms of manipulated content that could be used to spread misinformation or harm individuals.

Data leakage risk

In recent reports, employees of a global industrial conglomerate inadvertently leaked sensitive data by using ChatGPT to check source code for errors and to summarize meeting minutes. These are the tasks that Large Language Models (LLMs) like ChatGPT absolutely rock at. While no direct public disclosure of sensitive data occurred after being entered into ChatGPT, the data could be used by ChatGPT’s creator OpenAI to train future models, which in turn could disclose it indirectly through future replies to prompts.

In the specific example of ChatGPT, the retention period for prompts is 30 days. Opt-in for training future models based on your prompts is on by default for free accounts and off for fee-based accounts. OpenAI also recently introduced a feature where you can disable chat history for specific prompts for free accounts. Conversations started when chat history is disabled won’t be used to train the models, nor will they appear in the history sidebar.

There is also an immediate risk of erroneous accidental disclosure by ChatGPT itself. For a brief period, due to a bug, ChatGPT exposed the search prompts of other users in its interface.

To mitigate the risks associated with data leakage and LLMs when dealing with critical or sensitive information, try using self-hosted copies of the models or cloud-provided ones, where the terms of use and privacy policy match your organization’s risk appetite more closely. If this is not an option, enforcing limits on the amount of data fed into public models could lower the risk of accidental data leakage. This would allow the use of LLMs like ChatGPT for most tasks while preventing a user from copying and pasting large amounts of proprietary data into the web form for summarization or review by the model.

Vulnerability exploitation

Multiple use cases now exist for using generative AI to scan source code for vulnerabilities and produce reports. This can be incredibly useful for developers and vulnerability specialists in finding and fixing issues quickly. The same capability, however, can allow malicious actors to find vulnerabilities before the defenders do. On top of that, LLMs can also generate the exploit code for the vulnerabilities discovered, effectively becoming zero-day creation machines. The result is new vulnerabilities could be exploited in seconds or minutes instead of days and weeks like before. The zero-day attack window for vulnerabilities is expected to grow for threat actors unless we can also increase the speed at which we produce and publish patches.

To lower the risk of vulnerabilities in your environment, put extra effort into keeping your systems patched and up to date. With the current advances in AI, it might be relevant to use its productivity-enhancing capabilities to lower the time to find vulnerabilities and do something about them. To keep track of your software risk exposure, consider implementing and maintaining a software bill of materials (SBoM). This should drastically reduce the time and effort to find out if you are exposed to the risk of emerging vulnerabilities and exploits. 

 Phishing risk

Phishing has been one of our most common starting points for breaches for years. Spear phishing, a subset of phishing, uses emails that require two components to be successful: advanced research on the target and a customized attack email. Together, these efforts invariably lead to higher response rates. These tasks have historically been labor-intensive for attackers, but, unfortunately for defenders, they are now very easy to automate with AI. The quality of the resulting output is remarkably high, further lowering the effort of spear phishing. Methods to reduce phishing risk include deploying anti-phishing software, educating employees and making reporting suspicious emails to “phishing ponds” a functional part of your email software.

Looking forward

While generative AI has the potential to increase productivity and competitiveness for companies, CISOs must be aware of its strategic, legal, ethical, and cybersecurity-related implications. Proper safeguards and countermeasures should be in place to mitigate the risks associated with AI use. Opting for self-hosted or cloud-based language models that support protective measures can help ensure that generative AI is used safely and securely.

Finally, learning what works for AI cybersecurity and related security tools is a must. Specifically, there are strong market incentives to try products that are convenient but that may fall short in some other area such as transparency about underlying model assumptions, training data, or system performance. Cybersecurity is a field ironically prone to path-dependent outcomes that see “insecurity generated by the ghosts of stinginess past”. Perhaps more so than with any other technological evolution in this space in the last three decades, organizations must avoid this selection of convenient over best.

Rick Rowley is a CISO advisor, an architect, and an internationally recognized speaker on innovation management. His views are his own.