Propaganda Games: Sesame Credit - The True Danger of Gamification - Extra Credits

There's an interesting video from Extra Credits about this. It's based on early announcements and repeats some of the popular points, but it addresses the gamification and social pressure side of the issue which I haven't seen discussed that much elsewhere. I recommend watching: ▶ this video

China has been working on this for a while, and barring people from transportation is not the scary bit. That's last-century scary. No, it's the emergence of a new sort of ideological warfare making use of the recent knowledge developed in gaming industry that is the scary bit. This is a new kind of threat, something we as humans haven't learned how to deal with. Here's some analysis from a game designer from 2015: ▶ this video

I believe much of this can be dealt with, if we start RIGHT NOW to address a serious issue in our systems - the lack of a way to represent Morals and Ethics (the When and the Why) in the systems we are building. This needs to provide important input to, and thus shape the DIKW(D) pyramid. I've been doing some work with the IEEE on this - and I'm looking here on ycombinator to get some real-world feeedback on what people are thinking and concerned about. I have some (personal) ideas that might work to address the concerns I'm seeing. { NOTE Some of this is taken from a previous post I wrote (but kinda missed the thread developing, I was late I don't think anyone read it). It is useful for this thread, so a modification of that post follows.} First, I think you need a way to define 'ethics' and 'morals', with a 'ontology' and a 'epistemology' to derive a metaphysic for the system (and for my $.02, aesthetics arises from here). Until we can have a bit of rigor surrounding this, it's a challenge to discuss ethics in the context of an AI, and AI in the context of the metaphysical stance it takes towards the world. This is vital, as we need to define what 'malicious use' IS . This is still an area (as the thread demonstrates) of serious contention. Take sesame credit (a great primer, and even if you know all about it, it is still great to watch: ▶ this video ). Now here is a question for you: Is it wrong for the government to create a system that uses social pressure, rather than retrib

There was a extra credit episode on that. Yes, that is purest evil. The sort that laughs alone, at night in front of a mirror, because even other evil shuns such company. ▶ this video

Extra Credits made a video about Sesame Credit a couple of years ago. It's 7.5 minutes long, and well worth the watch: ▶ this video Being a game design channel, they discuss Sesame Credit as obedience to the party, gamified.

Beyond that, consider what Tencent are doing with a "social credit score" to compete with Alibaba's Sesame Credit. It will be scary when these systems eventually migrate to or evolve in the US. Tencent social credit: * link Sesame Credit video from Extra Credits (almost a couple years old now, but most easily-engaged piece I've seen on implications of social credit): * ▶ this video

I've seen the "Social Credit System" discussed: link Video describing a similar system (linked to from the wiki page): ▶ this video Possibly one of the scariest concepts I've seen in a while.

Extra Credit's episode about Sesame Credit: ▶ this video So, where else is there a widespread system of social media scores used to enforce conformity? Right here! The only difference is that there is a greater diversity factions influencing this in the west. From what I've seen, we are all being very effectively conditioned to hate "the other." link

What if we begin to rate citizens as Apps rate users: will someone get blacklisted because he got a bad rating? This Extra Credits video is about an attempt by the Chinese government to implement this: ▶ this video

When Extra Credits "Propaganda Games" video came out about a year ago[1], I was relived when I read that "Sesame Credit" was a complete fabrication, and even questioned EC for posting such a video without source checking. Apparently is wasn't completely fabricated after all. [1] ▶ this video

I'd imagine all these pro-privacy guys cheering right now, shouting "we told you where this is going!" if only it wouldn't be so terrifying and not funny at all. First they collect your personal data to show you ads, and people don't mind, because why not? Then they monopolize the market more and more, and basically all your data ends up in their hands, making them know you literally better than yourself do (as you don't normally quantify your life and are not watching over yourself really well). And people don't mind again, because "I have nothing to hide". Then they are officially profiling their clients to use the data for political purposes. First this might be ads, then some more. And people won't mind again, because it's unacceptable for USA citizen not to hate ISIS (but it doesn't really matter what is the most iconic black sheep at the moment). Then (at the moment, actually, but just in China[1] for now) the most used social network is running "citizen loyalty bonus-points" program. And people are accepting still, because it's all so cool and nice and what not. And even though it might seem terrifying to some, I don't think "the people" will ever actually regret this: the "acceptable norm" will just become narrower and narrower, and everything else will be treated as a criminal offense, a sickness that must be cured. It makes me think that this is just how life goes, the natural way an organism evolves. But the thought of it makes me feel sick anyway. [1] - ▶ this video

Ads as content isn't that bad of a problem. Consider a review or a show that uses a product for legitimate reasons . The (far) bigger problem will be using your friends against you[1]. Advertising by making various "credit scores" dependent on your friends using the "right" products will be very hard to counter. I suggest fighting it now. [1] ▶ this video

All it will take to "imprison the mind" is convincing enough people to use a system that gives positive reinforcement when your friends act "correctly". Normal social behavior will do the rest. China (with Tencent) is already trying this, and Facebook has at least explored this area[2]. [1] ▶ this video [2] link

No, that's not quite how Sesame Credits work in China.[1][2] The system computes a social score for each citizen. But it's not run by the Government or a bank. It's run by Tencent and Alibaba. It tracks what you say online, what you buy, and who your friends are. Having a low-scoring friend brings your score down. New feature: Baihe, China's largest dating service, now uses Sesame Credit scores in matching. It's voluntary now, but will become mandatory in 2020. This is the successor to the Dang'an, a permanent record kept for each citizen of China since it went Communist. The record was typically maintained in a book by the employer, in conjunction with the Public Security Organization. The Dang'an system became less relevant as more non-state employers appeared. Now it's getting an upgrade. [1] ▶ this video [2] link

I'm sure there are more direct examples, but I watched a video about Sesame Credit recently and it paints a rather dystopian picture of what's possible when you mix surveillance and gamification. ▶ this video

Check out this nightmare of a system China is implementing to use gamification to control populations: ▶ this video

There is a huge difference between a "private" rating system and a public rating that uses your friends against you as manipulation. See Extra Credit's description[1] of how this works. We aren't there yet , but consider that Facebook wants to use your social network associations in your credit score[2]. [1] ▶ this video [2] link

> It smacks of a solution in desperate need of a problem That's because you're looking at it from the wrong perspective. This kind of junk makes sense when the problem is "how can we trick people into giving over their personal data". It doesn't have to actually work (that is, save energy or offer some significant convenience) as long as it lets you generate a database of when people spend time in their home. It's telling that insurance companies are already mentioned - as if that was a good thing. Aral Balkan's observation[1] that a lot of modern business plans are based on surveillance ("it's about the data"), Al Jazeera recent investigation[2] of this power grab is surprisingly detailed. > that cannot be allowed to be a thing I completely agree. I only hope this can be fought before we end up with one of the nastier end-games[3]. [1] link [2] link [3] ▶ this video

> anti-free-basics mindset Ahh, the usual attempt to create factions. > Facebook isn't stopping anyone from providing internet access anywhere. The very idea of "free basics" is about normalizing the idea that Facebook can prevent who people can access. Yes, other providers already exist (removing any need for "free basics"), which is why Facebook is abusing their position as a social network to manipulate people into supporting their plan. They can do this because being a gatekeeper of social interaction is one of the most powerful political tools known to man. > And they certainly don't gain this power by providing free basics. If you read any of the other threads on this topic, you will find examples Facebook's tactics such as lying to users saying their friends support "free basics". That is their existing power as a social gatekeeper. The entire point of this new plan is to extend their gatekeeper position into one of the largest untapped markets on the planet. Without even considering how this position would allow Facebook to monopolize the market, their power as a gatekeeper to monitor social interaction (why do you think they MitM encryption?) and manipulate public opinion is so powerful it should be seen as a threat to national sovereignty . You probably think this is hyperbole, but I'm deadly serious. As I mentioned in a previous thread, the end game of this power to use friends against each other already exists in China under the name Sesame Credit. I recommend wat

> convince you that all your friends are supporting their proposals This is a serious problem, because China already showed us the end game of this tactic with Sesame Credit. ▶ this video