cancel
Showing results for 
Search instead for 
Did you mean: 
Create Post

Pokemon GO - Gotta Catch 'em All: A Study in IT at Scale

Level 13

Talk about disruptive innovation. Pokemon GO, has put Nintendo back on the map as it has skyrocketed into the stratosphere of top mobile apps in just a week's time. The mobile game is a venture between Nintendo and Niantic, Inc - an independent entity in the Alphabet set of companies. It launched with much fanfare last week and generated critical mass and out-of-the-stratosphere velocity in terms of user adoption and game play. And here in lies the IT aspect.

BringIT.png

There were noticeable hiccups in Quality-of-Service (QoS) such that even though it was a Top Charts game, it had a 3.5 star rating in the Google Play Store. Essentially, the launch and overwhelming response created a situation where the elastic supply of cloud powered by Niantic's Alphabet parent company could not meet the demands of the rabidly active user base. An example was the experience of our very own James Honey, who was attempting to create an account for his youngest. He went through the web portal and created an account when it didn't timeout. It sent him an URL to verify his information to create his child's account, which also timed out. So he clicked on the customer support button, which sent an email but mean-time-to-resolution is 48-hours. Interestingly enough, he's still waiting for the verification and approval of the account for his child after 80+ hours from when he first started with the app download.

In closing, it circles back to things that IT pros already know all too well: (1) No matter how much planning and preparation goes into a production launch, it happens and IT pros have to remediate it efficiently and effectively; (2) Hybrid IT is reality as the app lifecycle is now spanning the developer's platform, running across distributed systems in the cloud(s), and is being consumed on someone's local mobile platform; and (3) the rate of change and scale of that change is ever growing over time and yet IT pros still have to deliver the CIO's SLA.

Do you think those three things are apropos? Let me know below in the comment section.

107 Comments
MVP
MVP

I believe unexpected exponential growth (which can never be fully accounted or planned for) is the nail in the coffin for many things be it retail or IT and at the same time both.  There are other disruptive activities due to this app.  People wandering around with their heads buried into the app on their phone not paying attention to what is going on around them. Then apparently fire stations are a draw as well.  That leads to a whole different level of issues  and concerns.

The cell phone companies are loving it as the Pokemons eat up peoples data...then they get more money.  Unfortunately that infrastructure doesn't scale easily.

Great post as usual kong.yang​ !

Level 14

kong.yang​ Well done... Classic tale of "get it out the door"! 

I also read where email addresses may have been leaked during signup as well... Personally, I don't do games on my phone....

Just another all consuming game for the masses.... and a large dose of crack for the instant gratification crowd.

Level 13

I have seen the same thing happen in other games.  They hold as special in-game event and suddenly all the servers are overloaded.  Did this company learn from the event and plan better for the next?  Well, not for the next several event.  Even now a 2 years later the servers have trouble during the events.  Some is code related, but much of it is load related. 

Something being unexpectedly popular is hard to plan for.  But when it happens multiple times, you begin to wonder.

Pokémon Go reaffirms the adage that has been in place for 20+ years now. The gaming industry is 1 of the 2 industries that drives the technology of the internet. Pokémon Go was unique because it was the rare occasion where players actually got up and moved around. A very interesting sociological experiment indeed.

Level 15

On the other hand, we've seen a lot of games move to cloud mode, or require constant access to servers to operate.  And then the manufacturers completely drop the ball on architecting and scaling their side of things to handle the number of users, which is even "funnier" when they insist on driving a pre-order industry where gamers have a chance to pay $50 to $100 for something that may or may not actually work. (See: Assassin's Creed Unity, Arkham Knight).

And then everyone who pirated said game laughs that their copy works on launch day while legitimate copies don't.  Games that have shown server-side issues at launch:

Your Complete Guide To The SimCity Disaster

http://www.cinemablend.com/games/Blizzard-Apologizes-Diablo-3-Server-Problems-Delays-Real-Money-Auct...

You'd think by now there'd be some formula or metric for figuring out how much rack space a new release is going to need, right?

Level 17

It's hard to scope some of these app's or what may 'catch on' next. I'm sure they scoped the initial usage and had their marks they needed to meet; but did they scope the the available resources of infrastructure? All good and fine to think you can fire up X more servers and bandwidth - but it's hard to do if the service or facility you use does not have the spare resources to allocate. At least the lack of service has not been the death of this app.. though I could live without it. At least the insane idea of random placement with no regard to actual landmarks or highly trafficked infrastructure ( GPS is highly accurate, why place lil devils on the highway) has not caused any loss of life, just a couple of accidents.

Personally, I find this hard to quantify because we don't know what monitoring suites they are using/what their architecture is! I do however feel that this is a noticeably bad execution because they had the exact same thing happen with ingress, and here we are talking google's Niantic being involved *again*. This isn't really a good sign of what they're using. It's definitely not anything cloud based/scalable enough. Maybe I'll run some packet capture from my phone to the router and see where all the pokemon go traffic goes.

This is similar to what cahunt​ says - I don't necessarily agree that the issue is the mobile architecture - the issue is the actual pokemon go/ingress servers (because it uses ingress as well).

kong.yang​ all James needs to do is keep trying to create the account and then eventually it will take. Apparently he doesn't remember what it's like to be a kid and try to log on to an online videogame

Level 13

Oh, boy.  When I was a kid, video games loaded from a 5 1/4" floppy disk.  No such thing as online then.

Level 10

I experienced the same issue as James. The only difference was I downloaded it on  July 7th to my phone. The following day I downloaded it to my daughter's iPad, but it was missing the google login option. It only had the option for the trainer club login. I was able to login and play the game using the google login on my phone and she was able to create her own account with the google login also. The issue seemed to be a problem with the authentication servers on Nintendo's Pokemon Trainer Club site rather than Niantic. Reminds me of the time the CDC posted the zombie Apocalypse survival guide and when it went viral which crashed their website, you just cannot account for how popular something will become.

Level 17

Man that was the day... I'll never forget the first job my old man had with a 3.5" - took that sucker out the PC opened a drawer and just tossed it in there like it was nothing... looked up and smiled with a ,"isn't that GREAT!"

I bet I could still find a full height 5 1/4" drive in the closet somewhere.

Level 9

I finally gave up and used the google login.  So now it is a security issue. .  I have been spoiled by Steam and Blizzard...LOL

MVP
MVP

The first games I ever played we had to load them from a cassette tape! It took like 30 mins to start up a game. And even then it was just text based.

My wife found some old card making software which came on 3.5" disks. She asks me if we can still use it. I said that computers no longer come with floppy disks

MVP
MVP

I'm blown away by the way this game has taken off. I will not be playing it but I'm astounded by the amount of people that do. And all the strange things that have occurred since this game came out.

What Pokémon GO has to do with armed robbery and a dead body - CNN.com

A work colleague said his son was lazy and used to just sit in his room playing games. Now he's out walking catching pokemons

MVP
MVP

Yeah I don't think anyone saw how popular this game was going to be. I mean how many adults were into pokemons before this game?

Ok, Now that Pokemon is out (thought it actually died), I have to say I am very unhappy. 

This means that someone is going to track/play/place/whatever this pokemon inside of a government space {those of you know what I am talking about}.  This means I am going to have to sit through 3 hours of training that covers the fact that pokemon does not live here, is not played here, and will be prohibited at the door.  But yet, some ya-who, with a private server, will violate this RULE and subject me to the training torture.

Inside of playing pokemon, please learn solarwinds software, cisco ios, juniper os, or if you have to dig deeper Microsoft......  

Level 9

All I know is that there is definately a JigglyPuff in the DataCenter.... and I will catch her...... I already caught the Squirtle in the Breakroom and the Geodude in the testing lab......  Someone just tweeted there is a Water Pokemon Gym in the Bar across the street telling me he is: 1- a bigger nerd than I am. or 2- he has a SERIOUS drinking problem at 0726 in the morning!!!

GOTTA run got a call there is a Pikachu in the server room and you know he isn't Static Grounded!!!!

Level 9

abra.jpg  ​DUDE!!!!! SERIOUSLY!!!!???!!! ITS THE ONLY STALL ON THIS FLOOR!!!!!!

Level 9

IMG_7479.png This Article apparently attracts them.....

LOL! Love both your commitment to catching them all, and your org's apparent lack of acceptable use policies

Well, all of this stemmed from the 2014 April Fools Joke right?

When they inserted Pokémon into Google Maps at static locations but because of the exponential hit it was the concept was there and took years to evolve?

Pokémon Go: The April Fools’ joke that became a global obsession - The Washington Post 

It would be really cool to see that initial data from the April Fools usage and/or what methodology they used to scale their infrastructure based on initial user response, Alpha/Beta testers or whatever else they did (Or...didn't do).

I can remember staying up till unknown-dark-thirty desperately trying to find Mewtwo, not finding, then seeing sunlight and getting ready to go to work.

Now that the Privacy concerns to Google Accounts are mostly assuaged I'm pretty happy with the experience.

Getting outside meeting fellow players or enthusiasts - but as a nerd, dudes; trying to troubleshoot what was happening where - I never, ever wanted more of a way to hack my iOS device to attach to Orion's NetPath to find out why my game locked up losing me that Charmeleon (and my wife's Pikachu - man. Did I hear it about that one, though I did genuinely feel bad for her - its the only one she wanted. )

Maybe there's hope.  Maybe the physical activity side of this is the tip of an iceberg where programmers and companies are trying to change human habits and nature, so unproductive time is at least partially spent doing something that can improve physical fitness.

On the other hand, I suspect some of the designers and programmers and developers and marketing folks responsible for this hoped or suspected (or expected) this to take off, eating massive amounts of bandwidth and expanding the non-productive consumption of time and Internet resources around the globe.

If it's unnecessary entertainment, and if it impacts corporate resources, how is it any different than offering addictive drugs to employees and their children?  The results are the same--wasted time, wasted lives, wasted resources.

We can't change human nature, and if this program has no redeeming features, then it would've been nice to not have developed it at all.  We're awfully wealthy when we can spend time gaming instead of planting food & taking care of our futures.  What a lucky (?) group of people we are, to have that luxury.

Level 11

A certain car magazine in the uk uses it's internal cloud (maas, puppet and docket I think) to spin up new web servers at peak times, I think avg time to bring one up is 150 seconds if not needing a new host up.

I could understand it if the google dc's were doing similar to cope with the loads.

But we don't know the architecture. And there's no wireshark for android

As someone else suggested we could packet capture the router and see how easy it is to filter the traffic. Then with address run netpath to provide a service to show the portals routes are up

Maybe ... But I doubt the hotel I'm in would look kindly on me doing it!

Level 11

Now I feel old, we had to type the game into the zx80!

ZX81 I think had the ram upgrade module to 16Bit memory, I don't recall a tape drive ... memory loss. But do recall getting the monthly magazines and typing  in the latest game (Grand Prix was a monster 400 liner! ) debugging was a Pain! but the satisfaction was Great when it worked!

The was an Amstrad 464 (?) colour and Chuckie Egg and learning about computers went on hold for a few years.

Now I heard the rumours of people playing Star trek MUD's on card based machines, but before my time. Any of them here?

34 years ago the BBC micro was launched, there were resource issue's back then

"The machine was released as the BBC Microcomputer on 1 December 1981, although production problems pushed delivery of the majority of the initial run into 1982"

BBC Micro - Wikipedia, the free encyclopedia

This year the BBC Micro:Bit has been launched: BBC micro:bit : home I wonder if will have similar resource issues...

Level 11

The CLI attracts them as well IMG_2016-07-11-09182090.jpg

Level 14

I just hate this Pokémon stuff.  I think it is way overboard and is already causing too many issues.  This GAME is causing serious injuries and mass stupidity.  There are people that don't realize that these stupid things don't actually exist.  What is the world coming to that this is what ALOT of people actually look forward to doing?  Just another example of 1 step forward and 2 backwards. 

I think it is especially sad from a society stand point that you get 25 or so people in the same area (25 square yards) and nobody is talking or having a conversation.  They are to busy with their technology. 

Ahem! Pokemon Go isn't causing serious injuries and mass stupidity, irresponsible people are causing serious injuries and demonstrating mass stupidity. See what I did there? 

Level 14

You are correct!

Level 14

I agree.

Level 11

God forbid people who work their butts off in an office all day get outside and have some fun playing a game.

MVP
MVP

Inconceivable !!

No one here does that

Level 11

Right! The only difference between this and a video or PC game is you're outside. As long as you're aware of your surroundings I see no harm, and don't understand the haters this game is attracting.

To each his own I suppose.

Level 14

I could care less if it is something anyone likes to do.  I have no problem with that.  Have at it!  The issue that I have is the stupidity that surrounds it with SOME people and how that stupidity affects others.  Stay off military bases and government facilities with it (nothing good will come of it), don't drive while playing it (don't hit me or anyone else), and don't break into places to find your Pokémon.

Level 13

But the biggest, best  ones have to be behind those fences, guarded gates and locked vaults like at Fort Knox and all those protected spaces.

Level 14

I just think it's crazy that people actually do think that

Level 13

I have another one.  You know that fast Road Runner looking one out on the freeway so the distracted drivers and pedestrians can compete chasing it.

Level 11

Agreed!!!

MVP
MVP

The fact that fire stations are being impacted kind of tells us that some people just don't care:

http://www.fireengineering.com/articles/2016/07/pokemon-go-public-education.html

http://www.masslive.com/news/index.ssf/2016/07/williamsburg_firefighters_ask.html

http://wjla.com/news/nation-world/everything-wrong-with-pokemon-go

these were 3 quick links google provided on the topic....

I was first made aware of this while on shift over the weekend with a number of people coming through the parking lot and spending time around the station.

Level 11

That is crazy!!!

People are inconsiderate sometimes. I also wish the creators would have been more mindful when creating the game in avoiding emergency stations.

MVP
MVP

If a fire apparatus is delayed to a scene and property lost or death or injury occurs and it is attributed by people playing the game, the company who created the game can be sued...as well as the players.  That may change things although it should never get to that point.  While the game may be an annoyance, some of the protesting we are seeing has a greater impact on responses......it's all a sad state of  affairs.

MVP
MVP

I had no idea that happened. I can't read the article you've linked but I found this one.

Google Maps is overrun with Pokémon for April Fools', and you can catch 'em all | The Verge

Interesting that it's been around this long. I still can't believe the mass interest. Now whenever I see people walking around on their phone, I automatically assume they are catching pokemons!

MVP
MVP

I remember we had the BBC computers at school. We'd buy magazines at the newsagent that had basic code to make games. So that's what we'd do at lunch time. It was a pain if you made a typo somewhere in the code! But the games were always cool to play. Well cool back then.

MVP
MVP

Yeah that's part of life now. It was even like that before this pokemon thing came along.

MVP
MVP

I don't think the creators had in mind people breaking all the rules. Did they really know how popular this was going to be? And how quick it became popular?

MVP
MVP

That's nuts. Imagine if they turn the game off. There would be mass hysteria

I remember getting a whole stack of Input magazines from a neighbour, and they had code in them for a game, with a little more each month. I always wanted to type it in, but felt I needed to have all the code before I started. BIG mistake! By the time the run had finished there were tens-of-thousands of lines to Input!

Needless to say, I never did get to see what the finished code would produce

Level 13

I remember when some World of Warcraft severs went offline unexpectedly for a while and people actually committed suicide

Level 13

I am sure they were just looking at what kind of money they can make off of this.

Level 15

funny.