Trials of Testing while Indie
We don’t have the money to hire a QA department to test Dadliest Catch for bugs and we at least didn’t think we had the resources to playtest it properly. When we started working on the game we used things we’d previously done with the first Octodad which mostly meant we reached out to friends and family to play the game. We still do this, but there’s a point where those resources either dry up in that they aren’t as interested or they aren’t as useful because at that point they’re too used to the game. At that point, like us, they’re likely unable to see the flaws we need people to point out.

There’s a time and a place for testers who are experienced and we bring them back to test new content/levels that it would make sense for them to play.  However, we also need to be consistently finding new players to make sure the game still holds up from new perspectives. The game can’t become so hard that a new player becomes lost due to not having logged the same kind of hours a more experienced player has.

Hard in this case doesn’t always mean traditional difficulty. There’s the possible situation in which we get too used to communicating to a certain subset of player, one that’s highly experienced in our chosen way of communicating to them. If we get stuck in this position then we’re alienating any one who doesn’t think in the same way. Someone needs to be able to pick-up and play the game without any interference or help from us. This is pretty obvious, but it isn’t an easy thing to achieve. This is why we need as many new testers and eyes as we can get on the game. We can’t ship ourselves digitally with each purchase to help out if and when players get stuck.

More after the break.

Locations & Audiences

Where your agile “indieness” comes in handy.

The First Gate
Pros: Gaining a more intimate understanding of how your game really does and does not work.
Cons: Takes time away from actually making things, but is unavoidable.
Reaching them: It’s you! Shake your own hand!

It may seem silly to even have to say it, but YOU should be playing your game daily. It doesn’t always work out that way, but each member of our team tries to play through the game or the level they’re working on once a day minimum. This alone takes care of a lot of the QA problems you may have because it creates an incredibly fast feedback loop for you to make quick fixes to your work. We noticed that when we ourselves played the game more it was better setup to be shown even to initial testers. You should respect the people who take the time to play your game. Make sure that you’re doing your best to present a build that isn’t constantly crashing and is beatable to the extent that you’re trying to get feedback. It can be heart-wrenching to watch the game break or crash when you’re trying to get people to play, let alone in a public arena like GDC or PAX. You can avoid this problem to an extent, but also be ready to rollback on the fly to an older build or get really good at last minute debugging.

Here we’ve held our friend Jerry captive to give the game a go.

Friends and Bribery
Pros: Good friends are always willing to be honest. Cheap.
Cons: If they don’t understand that you need to hear what’s wrong they will fluff their feedback.
Reaching them: Phone, Facebook, throwing a rock at ’em.

We may not have a lot of money, but generally college kids don’t require a ton of motivation to show up at a party. All you really need is beer and food. $5 pizzas can go a long way in getting people to come by and play your game. We use this from time to time to throw playtest parties where we invite people over to play and reward them with cheap food and beer. What’s great about these is that we encourage those people to bring friends with them who we might not otherwise know to invite. This leads to new eyes on the game and a lot of the time these players aren’t even “gamers”. Getting feedback from people who don’t play games often can be great because they have a different mindset. They have a different set of skills and haven’t been brainwashed into how games “should” work. This leads to interesting results and discoveries. We want to appeal to more than just your traditional game-playing audience.  We want to create a game you can play with your kid or just enjoy on your own. Something to laugh at and something to bring a smile to your face when you’re having a tough day.

Getting the most out of those College Loans
Pros: Students are in the thick of it and if they’re game dev students they hopefully do this sort of thing all day.
Cons: Students seem to think they know more than they do which leads to over-analyzing things instead of just giving their direct thoughts.
Reaching them: Bug professors you’ve had to ask their classes to drop by. Get in the student/school newsletter(s). Ask clubs/groups.

The nice thing about our university is that after you’ve graduated you still have access to use their labs as alumni. We take advantage of this by having playtests at the school, and that allows us to reach other aspiring developers to get their input on the game. A lot of the time this is where we get our recurring testers who we show some of the early progress stuff to. This is where we test blockouts and rough ideas because we know many of the students have already played the game or at least heard of the concept before. Their previously obtained contextual awareness allows them to look past the fact that there’s so little art in the prototypes we throw at them. At least from our perspective they’re our second gate. They’re the ones who confirm that the things we think are fun or possible indeed are.

Kevin carefully watches a tester’s every move.

Local Communities and Groups
Pros: Refreshing membership leads to new testers on a regular basis.
Cons: If you aren’t careful you can be seen as just leeching on the group for testers. Contribute and help!
Reaching them: Look to things like meetup.com or your local IGDA chapter. Also look for indie groups like Juegos Rancheros, Indie Game Collective, or Indie City Games!

Indie City Games is our local indie group where we gather together to show off what the community has been working on lately and give/get feedback on said work. Developers and normal players both give different kinds of feedback and both are important for different reasons. ICG’s meetings give us the opportunity to give short presentations on what we’ve been up to lately and to try out not just the game itself but how we talk about and present the game to others. This is always a good test on what to bring to shows like PAX, GDC, Fantastic Arcade, and other events like them. How do we structure our demo modes? What gives the player the quickest and best impression of the game that accurately represents what we want to get across? These are the sorts of questions these communities help us to answer. There are communities like this all over the place and they don’t necessarily have to be developer centric groups. Why not reach out to a local game club. You would be surprised with how excited some gamers are to try out new things that they can’t see anywhere else. People are much more interested in your work than you may think. Creating games may be a day to day thing for you, but for others it’s like magic. Use this to your advantage!

Thinking outside the box is important when you have no traditional ways of getting lots of different eyes on your game. We recently contacted a local gaming cafe called Ignite Network here in Chicago about running some playtests with them. You would be surprised at how helpful and welcoming some places can be. Sam from Ignite basically welcomed us with open arms and a few weeks ago we ran our first Ignite Network playtest. It was great to get people who we had no attachment to to play the game. Many of the players there had not even heard of the game before and so this was especially great because we got to see how players completely out of the loop would perceive the game. They have no reason to be especially nice to us and no reason to hold back on telling us what they really think of Dadliest Catch. After this playtest went so well we’re even thinking of putting together something bigger like a dedicated event.

Looking on as a Minecraft fan gives Dadliest Catch a go.

Tradeshows, Cons, and Festivals
Pros: A ton of first-time testers in a short time. Gamers who will tell you the brutal truth and give you dirty looks.
Cons: This is the one option that can be super expensive. BUT there is the potential to win passes/booth space at some.
Reaching them: Enter all the festivals and competitions you can.

Being invited to or exhibiting at a show like PAX can be really exciting and fun, but it can also be extremely tiring. A major reason behind a lot of these shows is to get the word out about your game, but don’t forget that they can be useful in other ways as well. Take a small notebook along with you. Watch people play the game in between talking to other people who are interested. These kinds of events are probably where you will get the most plays in such a short amount of time and you should utilize that as much as you can. We always are writing down notes and talking to each other about problems that pop up or trends that we’re seeing as hundreds or thousands of players try the game. This is also a great place to stress test the game for crashes. Obviously it isn’t ideal for your game to break in these situations, but it’s bound to happen at least a few times if it’s early in development or you haven’t had a ton of people go through the game this quickly. You’ll most likely run into things you would have never expected players to do and find game breaking behavior you would never see from a seasoned player.

So many possibilities!
Always be on the lookout for opportunities to test your game and be aware of what the possible benefits of various audiences can be. We take the game home with us on vacations and have our families try it out to get younger siblings, cousins, and friends to give it go. We bring it with us to normal parties like a bunch of weirdos to get inebriated people to try it. Do your best not to let unique opportunities slip through your fingers!

Equipment & Data Gathering

Pen, paper, google, and USB sticks.

Our Checklist:

  • Paper/Pen/Pencil
  • Headphones. We got 4 pair for like $40 on Newegg. Look for deals!
  • All the flash-drives
  • Newest build of the game
  • Previous build of the game
  • Laptops with your dev environment if possible for emergency’s
  • Snacks/Swag to keep the natives happy
A note Kevin left for us when he was up fixing a build the night before we left for a test.

Plan B – Z
Just about everyone these days has a flash drive or two on hand. In reality you don’t necessarily even need these to transport your game’s files to wherever you’re testing it. You could potentially use something like a private dropbox or another file serving site, but we generally like to setup our game at multiple computers at once so we can go through multiple testers without forming long annoying lines that no one actually wants to sit through to play. It’s always important to have the latest build on hand in multiple places with multiple people if possible. This way if a teammate isn’t able to show up last second, you or someone else still has the build you need. It’s also handy to have the last stable build before the one you’re testing just in case major problems arise that you didn’t expect. This way you can still continue testing something you previously did even if you aren’t able to test what you had planned to. Never pass up an opportunity for more people to play your game even if it’s been played to death. You never know when someone will come along and show you something you’ve either never seen before or never expected a player to do.

All Hail Google Drive
Surveys are nice to create in order to see trends in players behavior. For our surveys we use Google Forms which are a part of Google Drive. These provide a simple interface to poll players with general questions like, “What did you like about this level?” And another favorite, “What did you not like about this level?” The point is that this is all entirely free and it rids you of the need for something horrible like a paper survey or feedback form. Speaking of paper you should bring your own notepads to be able to jot down notes while you’re watching people play. Sometimes you need to draw things out and you’ll often see something that players will not. Don’t think you will just remember these things because you won’t. After so many people play the game you start to forget what past players have done and so you need to write that stuff down. Some things may only pop up once or twice in the whole test, but they’re important and shouldn’t be forgotten. We usually round this all out with a post-play informal interview where we ask a few of our own questions that we think up while they’re playing so we can dive a bit deeper into some of our own assumptions gained from watching them play.

If there’s anything I’ve missed, and I’m sure there is, please feel free to add your suggestions or feedback in the comments section. We’re always looking to improve our processes!

– Phil

For making it to the end of this post here is Devon dressed up as Octodad with a stormtrooper in loving embrace.