Over the past decade, a large number of extremist and hate groups have turned to internet platforms to inspire mass violence. Currently, there is little reliable evidence on how such campaigns radicalize targeted audiences. We provide systematic, large-scale, microevidence on the effect of Islamic State propaganda on social media. We use several machine learning algorithms to detect recruitment messages in online propaganda, identify their dissemination on Twitter, and quantify the reactions of exposed users. Analyzing content produced by the Islamic State between 2015 and 2016 shows that propaganda conveying the material, spiritual, and social benefits of joining ISIS increased online support for the group, while content displaying brutal violence decreased endorsement of ISIS across a wide range of videos. Only the group’s most extreme supporters reacted positively to violent propaganda.