‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌ ‌
 
 

National Security Up Front

New Perspectives from the RAND National Security Research Division

November 7, 2023

 

The Rise of Generative AI and the Coming Era of Social Media Manipulation 3.0

AI and social media

Illustrations by bubaone/Getty Images and jolygon/Adobe Stock. Design by Chara Williams/RAND Corporation.

 

The emergence of ubiquitous, powerful generative artificial intelligence (AI) poses a potential national security threat to the United States, expanding adversary capability for malign influence operations on social media. Of particular concern is a potentially revolutionary improvement in astroturfing: Adversaries can create a false perception of widespread grassroots support for an issue or person, manufacturing fake but seemingly human accounts on Twitter, Facebook, or other platforms to advance narratives serving the interests of those governments. China and Russia are already engaged in astroturfing, but the advent of generative AI makes it more convincing, ultimately posing a direct challenge to democratic societies.

To address this potential national security threat, the U.S. government and industry should consider pursuing technical, policy, and diplomatic mitigation strategies now. Any mitigation strategy must account for generative AI being ubiquitous and unregulated.

What is generative AI? Generative AI is an umbrella term for AI models that can produce media based on user-generated text prompts. Large language models (such as OpenAI’s GPT-3) and text-to-image models (such as Stable Diffusion) are the most mature and most deployable, but the ability to create convincing video, audio, and music is rapidly maturing.

How does generative AI change social media manipulation? Generative AI removes many of the limitations of prior generations of social media manipulation. The new technology can produce remarkably human-like interactions, with much less labor, thereby saving costs and making such manipulation scalable; its “authenticity” makes it harder to detect fake content.

Generative AI is a dream come true for China, especially the Chinese military. The People’s Liberation Army (PLA) is likely a key actor in Chinese social media manipulation, and PLA researchers have hailed generative AI as improving their capabilities for what they call “cognitive domain operations.” One PLA researcher, Li Bicheng, has worked since at least 2019 to develop an AI-run botnet that is well-positioned to harness generative AI to create its content. However, the Chinese Communist Party is also likely worried that generative AI will enable better adversary (U.S.) information operations to undermine its rule—an enduring fear.

Taiwan provides a notional illustration of how generative AI can be a game-changer for Chinese social media manipulation.

 
 

China's Tactics

 

Bots for astroturfing

Fake accounts to create appearance of broad consensus on an issue or person

  • Previous shortcomings: Relies on human-generated comments, limiting quality and scale
  • Improvement with generative AI: Bots will have written voices that are nearly indistinguishable from human-created content

Impersonation

Pretending to be another person to misrepresent their views

  • Previous shortcomings: Relies on pressuring public individuals to create misleading information
  • Improvement with generative AI: Could mass-generate text in the style of a given individual, falsify images, and mass produce those images

Swarming

Overwhelming an information space—e.g., spamming a comments section

  • Previous shortcomings: China's large-scale human effort for spreading propaganda cannot consistently avoid detection
  • Improvement with generative AI: Automates process of creating unique content at scale for spamming

Testimonials

Personal stories used to elicit emotional reactions or sway opinions

  • Previous shortcomings: Manufactured testimonials that usually appear on state media seem relatively scripted
  • Improvement with generative AI: Swiftly creates testimonials on wide-ranging content, representing various demographics for both broad and niche effects
 
 

How should the United States counter the potential security threat of generative AI? A proactive response by the U.S. government and the broader technology and policy community may begin to mitigate the threat of social media manipulation posed by generative AI. Technical, policy, and diplomatic recommendations include the following:

  • Invest in defensive technology that can detect inauthentic content; success may require long-term commitment, since current detection capability is weak.
  • Make it harder for malign actors to open new accounts on U.S. social media platforms, and redouble efforts to identify and remove existing inauthentic accounts.
  • Make a public declaration of U.S. government principles for using this emerging capability.
  • Begin monitoring China’s use of generative AI–produced content as an indicator of intent and technological progress, with special focus on Taiwan.
  • Be prepared to operate across a contested information environment in which generative AI scales propaganda and makes it ubiquitous.
  • Engage with allies and partners on these emerging risks, share evidence of China’s use of generative AI, and cooperate on potential countermeasures.
  • Consider engaging in diplomatic dialogue with China to better understand how each party views the risks of generative AI and explore the prospects of mutual commitment to limit malign use.
 
 
 

This summary is derived from the forthcoming RAND report, The Rise of Generative AI and the Coming Era of Social Media Manipulation 3.0, by William Marcellino, Nathan Beauchamp-Mustafaga, Amanda Kerrigan, Lev Navarre Chao, and Jackson Smith (PE-A2679-1).

Once published, this report will be available for download from rand.org

For more information about this study, reply to this message and we'll connect you to the authors.

Want to stop receiving these messages? Reply to this message with "Unsubscribe" in the subject line.