Iconic Affordance Interactions

I'm immediately noticing that many of these affordances, though differing in context, are essentially similar (pokeballs and baseballs, swinging in various media) and that others probably need to be broken down into subcategories (various kinds of "shooting", potion brewing distinct from spellcasting). Might be shifting some of these categories around at a later date to group them by general affordance characteristics.

I had a conversation with my mother about what she might be interested to see in a theme park, and although she's heard from me as I've worked on the theoretical side of the project, she doesn't seem to have really internalized what it was about, so I got an array of answers, some of which were wholly unhelpful, but also many that were intriguing in the ways they fell slightly outside the sphere of what I've been considering. In particular, among the ideas my mother suggested, she mentioned consuming various IP specific foods (such as from Harry Potter, Lord of the Rings, or Alice in Wonderland), wearing specifically iconic clothing (princess dresses, cultural fashions, etc), and some specific embodied experiences, such as falling down the rabbit hole in Alice in Wonderland. Although these might not be technologically interactive interfaces at first blush, they do all present some sense of acts (eating/drinking/dressing/falling) with sensory feedback that further reinforce immersion in the experience. I'm including them in my list because I think it provokes some interesting lines of thought for where one draws the line on what is "interactive".

For instance, we might not think putting on a shirt is interactive, but is getting kitted out in armour by a squire at a renaissance fair interactive? Is an Iron-Man helmet with automated moveable visor interactive? Does piloting a mech count as wearing it? Similarly, eating and drinking might not be "interactive" in an of themselves, per se, but bars such as Disney's Trader Sam's Enchanted Tiki Bar have special effects that trigger in the bar when you order specific drinks, which certainly feels interactive, and 4e Mur here in Montreal is a bar where you can solve a mystery based on clues that come with ordering specific drinks. In this fashion, it becomes clear that "participatory" activities do have robust design opportunities to be made fully interactive, and so I want to put some more thought into this at some point.

I initially though there'd be roughly three categories, prop-, environment-, and gesture-based, but as I looked at what I'd listed already, vehicle-like interfaces kinda feel like their own thing, so I've spun them off into a fourth category. This process also revealed a heavy bias towards prop-based interfaces, so I might want to give the other categories some concerted thought, perhaps sin conjunction with reviewing some popular franchises. I feel like I'm largely falling into what I already know (what I specifically want to avoid) so it might be beneficial to start looking at IPs instead and assess what kinds of interactions might be productive from those.

I'm also starting to think I might want to create a whole separate document for "what is shooting?" I thought I'd have to eventually elaborate it in this document, but it feels like a topic that needs its own space to breath precisely because so often it is collapsed into "guns" when mechanically it could encompass so much else.

This document is to hopefully explore some ideas around shooting as a concept, both aesthetically and mechanically. Though it is easy to criticize it, morally as well as creatively, I do think there is perhaps some greater diversity to it than I've given it credit for. I want to use this document as a way to explore and elaborate the idea, not neccesarily to use it, but to at least leave the possibility open should I come across something really novel or intriguing. Even considering the number of interactions that map onto "hitting a target at a distance" by some vague definition, i think merits closer examination.

As went about filling in some of the gaps for more abstract interactions, I realized I'd already mentioned a decent number of specific IP or listed interactions that are intrinsically tied to one IP (such as Super Mario), however I also realized I may have already made precisely the mistake I set out to avoid by listing affordances more so than specific IP or theme interactions. For example, I listed things like "flying" and then tried to reverse engineer cultural touch-points for it, when really I should have started with "dragons" or "Top Gun" and extracted flying from those popular subjects. I don't know if I have to abandon this list entirely, but I'm thinking I may actually need to take a step back from it, and perhaps reorient toward extracting affordances from a list of top 20 video games or movies.

On a separate point, I ended up with "fire breathing" on my list of interactions, and while I was initially envisioning the circus act when I added it to the list, it occurs to me it might be a kinda fun way of attacking in a 1st person dragon experience. Combining some sort of haptic flying simulation and fire breathing by blowing actuation might actually produce a somewhat compelling, if unorthodox, game where you play as a dragon. As such, while I continue to explore the blue sky domain of idea, I think I might also start some ideation for how this specific interaction might be realized, which if for no other reason will allow me to start exploring how best to document a more tangible game with this commit-oriented methodology.

Blocked out some of the themes and interactions I'm seeing in top watched video games.

Unsurprisingly, shooting continues to be a recurring theme, though once again I'm struck by the relative diversity of shooting. In particular, my eye was caught by some of the variety in gameplay modes, class systems that feature prominently in many games, and the variety of additional common features such as ults. I'm also starting to think about how, for many of these games, even healing is sometimes executed out as a "gun", such as Overwatch's Mercy, or, to pull an old reference, the Medic from TF2. I also fell down a rabbit hole examining Fortnite and it's unusual features of building, often leveraged competitively, and dancing, both as a mode of expression, and game mechanic in taco time and boogie bombs.

As much as I was initially skeptical of shooting as a mechanic, I'm increasingly wondering if part of the issue isn't so much the prevalence of shooting, so much as it's convergeance on a specific execution of shooting. I find myself wondering if it might be possible to design, say, a laser tag system that achieves some of the fantasy to which games like Fortnite, Overwatch, Valorant, or Apex Legends cater to– with team strategy, and unique abilities. I think this might be worth exploring further, in part because it's interesting, but also in the interest of just starting to actually make something.

Which brings me to my other realization of just how transformative this journaling method is for how I work. I'm so used to a slow percolation and intermittent tinkering with both materials and ideas that if left to my own devices, I'm not usually end on, or even with, a self-contained chunk of work. This has led, in a few cases, to me starting something, and then not actually journalling about it until I finished it a week or two later, or, for fear that a do the above, entirely avoiding working at all, lest I lose the freshness of the ideas when journalling. I'd initially thought I was just purely too busy and got distracted from working on this project, but I'm starting to realize that if I was not using this method of tracking progress, I probably would have been making small, incremental progress in between other work. Instead, I simply didn't do anything for three weeks, then didn't do anything for a week and a half again. I'm going to try pushing through this going forward but it's both concerning for productivity, and for the validity of how I'm using this journaling method, because it changes how I design in such a radical fashion.

I spent some time watching tutorials, tip videos, and reading wikis to get a sense of the basic mechanics of Splatoon and the general ways those mechanics tend to be used. What immediately strikes me is that a core affordance of the game, swimming, is implicated in ammo, life, visibility, and mobility in ways that are not immediately obvious in how they might be rendered tangible. I can sorta imagine a system of swapping between arbitrary states, but the fast-paced mobility options seem to be a pretty integral part of play. Maybe this could be incorporated in some way into like, variable-speed rollerskates, though that still does not address the question of visibility.

I'm also thinking about how the inking of territory might be done. Obviously this would require either projectors or screens. Projectors might be the easier option for my purposes, both for cost and safety of the tech. Targeting might be trickier, but can probably be done with some kind of machine vision and laser pattern? I'll have to look into possible tech solutions for this.

Consequently adding two items to the to-do list:

• Look into mobility tech
• Look into targeting tech

Currently focusing on the gameplay elements of painting the environment, eliminating opponents, and increasing mobility. I think the technical way to achieve the first two might actually be pretty straightforward with a modified conventional laser tag arrangement, albeit with considerable tech requirements, so that does preoccupy my mind. On some level, I'm not super concerned with the economic viability of the experience, but I do think I have to be mindful to the potential concerns of cost and maintenance if the system's demands are far too heavy, simply because a plan without practicality doesn't actually go very far.

Furthermore, I also find myself thinking about the pace of Splatoon. It seems both fast-paced and fluid. Various mechanics allowed for a game that's as much about mobility as it is about shooting, and so I feel like that pace adds to the appeal of the game. Achieving that mobility is an open question though, as conventional laser tag already typically asks its patrons to not even run to avoid injury. This is in large part because conventional laser tag uses darkened mazes, creating both low visibility and heightened obstacles, but I wonder if this potentially pushes me towards a more open-plan play field.

A few factors push me to consider an open plan:

• The aforementioned reduction of dangerous obstructions, so as to facilitate greater mobility.
• Reducing the number of projectors and cameras required, as they could number fewer and be positioned in a top-down fashion.
• Splatoon doesn't count vertical surfaces as painted territory for the purpose of the game objective, so apart from providing cover, vertical obstacles may actually be incidental to the gameplay anyways. These three factors taken together seem to provide pretty compelling and comprehensive reasoning to steer towards a more open plan, so I'll start thinking in that direction for now.

This update involved a few separate but related moving parts, all pertaining to moving the project from the bluesky brainstorming stage to the prototyping of a concrete experience.

First, previous documents were moved into a Bluesky folder and a Prototyping folder was created to hold iterations of the prototype, and the to-do list was updated to reflect the shift in project tasks. This is all pretty boilerplate.

Second, I've started working on the first prototype, leveraging the conceptual tech from The Graffiti Research Lab's blitzTag platform, which basically uses a camera, projector, and laser pointer to measure relative luminance as the input for output graphics. I was able to get my hands on a conventional laptop presentation laser point, much lower in brightness than the high-power green pointers used by the Graffiti Research Lab, but even so, I'm having some very preliminary success at tracking the dot to the exclusion of other objects in the shot. I have not yet been able to test it with a projector running, as I'm currently travelling, but I do think this is a promising tentative first step towards a playable prototype and should certainly be testable mid-next-week once I'm home. I do foresee there being issues with dialling in the exact network for extracting the laser dot, as the projector itself will produce light, of course, but so far I've been testing in a lit room and as long as the camera isn't pointed directly at an ambient light source, it does seem possible to distinguish between laser and ambient light. If this does prove to be a problem, I can always see about obtaining a more powerful laser, but that will have to be a question of safety against laser power. The other distinct question of scalability will also come into play if I can get a desktop prototype to work, but one that doesn't scale properly to the size of a play arena. In Graffiti Research Lab's blitzTag, they use a reasonably high-power laser because they're "painting" at a distance on the side of a building, so that maybe gives me a sense of the benchmark I'm designing against, but then again, their project seems to be a little bit older (references PS3 camera, page comments date from early 2010s) so I may also benefit from simply better hardware.

The final point I want to touch on with this update is returning to my proposed design principles, as the whole point of this project is to be enacting them and I feel like I've already strayed from them once previously. As I move towards a prototype, I think I'm keeping at least partially in-line with principles I and II regarding bottom-up design and iconic affordances. Principles III, IV, and V are still open questions I will have to address as I move forward with higher fidelity prototypes, but I am noticing that my previous uncertainty regarding the simulation of mobility mechanics from Splatoon will likely run directly into at least principle IV regarding feedback, and possibly principle V on dissimulation. Now that I'm actually making a thing, I'll likely have to circle back to these principles much more often to ensure I'm balancing practical considerations against the theory.

I'm probably not building out mobility for the first prototype, but I'm already bouncing ideas around. So far I've really been fixated on if there's a way to get skates of some kind (roller or ice) to work, but it's still very unclear to me these would be safe, or even effective at replicating the specific affordances of Splatoon, thereby running afoul of design principle II. "Sliding" gear borrowed from haunted attractions, whereby participants can slide on their hands and knees might prove a useful direction, and this again has some safety concerns, but perhaps comes closer to replicating the feeling of a "squid submersed in ink" that happens in an actual Splatoon game. However, sliding is kinda a specific skill, and while not entirely impossible to pick up, the learning curve might pose its own challenges against design principle IV regarding competency. Obviously I'm not going to be solving this problem tonight, and I've got plenty of time to do it, but this does feel like it's going to be a much tougher nut to crack than the shooting aspect of the experience.

Despite travelling, was able to use a basic laser pointer to get the basic functionality up and working. Should be easier to test once I'm in an environment where I can control the setup more precisely, but next step will be to test it with a projector, to see if the luminance of the projection can be balanced against the laser.

Will also need to think about how to discern between the lasers of different weapons (either for functionality or to discern team colour). Trying to identify a laser symbol might be a bit tricky, but I wonder if I could instead use frequency of flashing to do it. This would obviously be a bit more complicated and would require creating a modified programmable laser, but the hardware side shouldn't be too difficult with the help of an arduino board. Detecting frequency of flash on the laser is theoretically easy (it's how remote controls worth with IR light) but I'll have to do some research into precisely how to enact that in TouchDesigner.

Mostly just added some thoughts on using heelys with brake pads as a way to execute the mobility mechanics of Splatoon. Safety continues to feel like a big factor, but I think brake pads that can exert a carefully adjusted amount of "braking" especially on something with "rear-wheel drive" like heelys might make it sufficiently safe for use.

I was pleasantly surprised at how well it work. I will need to test again in a more controlled environment (large flat surface, adjustable lighting) but even in a noisy environment (just a wall of my bedroom) I was getting pretty good discrimination on the laser pointer's position to the exclusion of anything else. It's definitely still a low-light environment, but I think I might be able to get away with low-but-still decent lighting which improves concerns of safety by some imprecise margin.

This is an aesthetic question issue, but I think it's part of what creates the satisfying "splat" of splatoon, as well as games with similar dynamics such as Super Mario Sunshine. Right now the splat is a pretty uniform shape so I still want to experiment with alternative ways to make the splat a bit more random. Also sound effects and some real texturing for the splat would probably go some ways to adding life. Finally I want to figure out a little animation so you see the splat spread as it "hits" the surface. I think apart from the aesthetics, this would also help with making the "shooting" more intuitive since you otherwise can't see your projectile.

It's basically fully random right now so to tends to lay down a lot of superimposed splats for any given shot, obscuring the actual shape, which has somewhat more natural contours in that they're rough and messy, but is also making me realize I'm not super happy with the splat itself and I don't think this random pinwheeling will work if I want to animate the splat.

At this stage I'm actually wondering if it's entirely a mistake to be putting this time into creating the splat as a 2D graphic, rather than just exploring options for 3D fluid simulation in TouchDesigner. I'm guessing that working with a 3D simulation that I just render onto a 2D surface will probably feel more natural in all regards, rather than trying to replicate the look, feel, and dynamics by hand and math.

Was previously constructing the shape from primitives in TouchDesigner but realized you can just upload an image with transparent background. Right now I'm using a generic place holder blob shape but this will make it much easier to craft a splat shape in photoshop later, then just swap it in when I so choose. Using this blobbier and less "spidery" splat shape placeholder, I'm also realizing it doesn't feel that off-base from how ink piles up in the Splatoon game, so I may not need a "splatting" animation after all. Will re-evaluate as the splat gets closer to a "final" look.

Effects include a blobby noise texture, gold flecks, embossed edges, and specularity. Specularity and interior noise texture could be improved with embossing effects and it overall feels a bit too static, but otherwise I think this is a good start in adding dimension and texturality. Might feel even better once I get a sound effect in there.

Also starting to think about how to track score. This is getting complicated to get the texture effects across, but I'm thinking I can split channels to track the silhouette of all inked space, as well as a separate channel that tracks one team colour vs another team colour as they overlap. This would allow for both point tracking (by counting pixel colours) as well as clear delineation of ink colour.

Before I do anything else, though, I think I need to start compartmentalizing some nodes and generally cleaning up the TouchDesigner file. Everything has gotten very chaotic as I've been hastily trying various combinations and tweaking. It might be worth sketching out basic logic flow first so I don't accidentally over-compartmentalize nodes I need access to to split for different tasks, however, so i'll probably do some of that sketching next.

The touchdesigner file was getting a bit messy so I decided to go in there and organize the nodes a bit before progressing– sorta a consolidation from furiously dropping in nodes wherever. This also provided me an opportunity to remove some superfluous nodes and clarify the flow, grouping nodes into useful flows such as blob detection, texture generation, and projection construction. I think this gives me a reasonable place to start experimenting from again.

Since the last time I worked on this project, I've given some thought to how to track inking in a way that can be visually integrated for projection, but informationally kept distinct for scoring calculation. Look at the cleaned up nodes, I'm now realizing that I'm not using an approach of "printing" splats over top of each other, but rather adding a silouhette and then texturing the aggregate ink area. I think this should allow me to simply have to concurrent team nodes that track respective area inked, and each team's ink add to their own node and subtracts from the other team's. I'm going to turn my attention to that next to see if I can get that going.

Now, when a given team shoots, it adds to their aggregate coverage and subtracts from the opponents aggregate coverage, so that inking maps are always complementary. These maps, initially generated as white silhouettes, are then texture their team's respective colours and then composited to create the final projection. This allows the separate maps to be used to calculate scoring in parallel. I haven't determined the precise method for calculating scoring, but likely some function of counting what percentage of each team's coverage maps are white vs transparent vs total surface area available.

As I see it from here, the two things that would make this a self-contained playable prototype are having two separate teams, and having scoring. Scoring will be easier to test once I have two teams functioning, so I may leave that until later, but I have added both tasks to Todo.md nonetheless. As for getting two "teams" going, I've ordered some 5V lasers I can rig up with an arduino which should allow me greater control over the behaviour of my laser. Until now I have been using an off-the-shelf laser pointer but I think if I can get TouchDesigner to recognize the frequency of a flickered laser, then I should be able to use that to discern one "team" from the other.

I know I previously said I might put this off until later, but I realized I could use an analyze node to do the pixel counting I wanted, which would make implementation a snap. Now, unfortunately due to something goofy in what analyze is doing is yielding slight greater than 1 values for aggregate coverage, which of course makes no sense but is probably an artifact of aliased edges on the silhouette. Frankly it's close enough for my purposes (we're talking like 1.011 coverage instead of 1) so I went ahead and implemented scoring, albeit with a bit of extra normalization.

Since the scoring tracks proportion of area covered by each team, as well as uncovered area, the unclaimed ground is what gets cut if the total coverage is reading higher than one. That is to say, I sum both teams coverage and divide by total area. If that number is >1, then I set unclaimed area to 0 and divide the 1 proportionally to the ratio of the two teams coverage relative to each other. If the total <1, then I presume it's accurate, set (1-number) as the unclaimed territory and then divide the number proportionally to the two teams coverage. It's a bit inelegant as implemented but I'm pretty sure it will get the job done in a robust manner. I'll probably want to also implement a GUI to display scoring, as well as a timer for game play, but for now, this was the main hurdle I anticipated with regards to scoring. Everything else is largely cosmetic.

Added a bar along the bottom that shows the proportion of the screen covered by one team or the other. Unfortunately, it seems to be ticking backwards under certain circumstances. Based on how it's coded, I actually think this is not a problem with the display but rather the underlying calculation of area covered. My guess is my convoluted method of normalizing the data is probably throwing a wrench in thing somewhere so I'm going to have to revisit the flow.

I also realized I've polished the visuals a bit but haven't added any sound and I vaguely recall making a note that the sound was important to the experience. There's something about a splat or squelch that I think really makes the inking process more satisfying. In the interest of not overlooking it again, I've added a note about it to Todo.md.

Next steps will probably be to troubleshoot the scoring system, add sound effects, and add numbers to the scoring system so that (when it does work properly) there are also numerical indications of score.

I figured out where the bug was. I had reversed two nodes representing the aggregate area covered by both teams and the area uncovered by both teams. This was carried uniformly downstream, so that basic checks (ensuring all area sums to 1) were still passed, which is why I initially missed it. I had to catch it by closely tracing the values from node to node but it should be working properly now.

Pretty simple addition, but added a numerical readout to the score display. I think I'll probably want to aesthetically overhaul all of the score display in a later iteration of the prototype, since this current version is so flat and plain, but for now I think this does the job of conveying relevant information on who's in the lead.

Implementing a timer proved easier than I initial anticipated because I remembered TouchDesigner is, after all, designed for time-based work, so I simply set the built-in clock to a time limit of 30 seconds and display the clock time inverted to created a countdown. Again, I'm sure this will change, including the length of the countdown, but the current network should be easy enough to modify once I can start actually testing this prototype as a game.

As a side note for the timer, I debated if I just wanted a raw number of second or a #:## format. Since I'm not using the minutes digit, I'm just going with a raw number countdown, but at a later stage I anticipate wanting to switch over to the minutes + seconds format for longer time limits.

I purchased some cheap laser components off amazon for the purpose of getting a prototype up off the ground. They nominally operate at 5V, 20mA but on initial test they seemed a bit dull. Since I'll be looking to only flash the dot rather than ever use it continuously, I'll probably experiment with judiciously overdriving the diode to get something a little punchier, without frying it and while still keeping in mind that I don't want to blind any players.

I'll include schematics (and maybe a component list with specs for the sake of capturing non-digital elements of the project?) as I build out the circuit, but I'm thinking an arduino taking a digital input from a "trigger" microswitch, which then simply sends a few quick on-off cycles to the laser+resistor at a prescribed frequency to be identified on the TouchDesigner side. I might actually be able to execute flickering with pulse width modulation but I don't know how convenient that will be to process as a signal on the receiving end. More specifically, I don't know if I'll actually need an identifiable frequency so much as a unique singular interval between two flashes in a sufficiently localized position. Like, I'm pretty sure I'm going to have to spatially isolate laser dots in the video feed anyways, so at that point it might only take the two flashes to establish an identity. I will have to experiment more on the receiving end once I have a functional laser gun prototype.

Prototype is in accordance with circuit.png under the v001 folder, where the laser diode is nominally 5v, 20mA, but resistor A in the diagram is initially using a 120 ohm resistor, overdriving the diode during it's brief flashes because it seemed a bit dim otherwise. Shooter is running splatshooter.ino on its arduino uno board and luckily it was simple enough code that i was able to bang it out without any bugs on the first go-through. Currently the shooter flashes once, so there is no team discrimination, so I will have to experiment with adding team discrimination next.

Having tested this initial prototype, the "single shot" effect created by the flash rather than the continuous stream of the previous laser pointer definitely changes the feel of the splat. I know I abandoned animation earlier in the project, but I may want to return to animation give the splat a little more life. Right now it feels a little bit stiff and unresponsive in a way that's hard to describe other than "yes, the laser dot sure makes a static blob shape on the screen." I won't jump right into adding animation first though, since I know I want to add sound anyways. Adding the sound might make the splat feel more "splat-like" so I may not actually need anything further once that's implemented.

Finally, I hadn't initially figured I would prototype a casing for the shooter at this stage, but in it's current state, (see shooter-side and shooter-front) as essentially loose parts held together with tape, I think it's probably neccessary to give some thought to basic ergonomics, even at this stage. Not sure how I'll approach this, either modelling and 3D printing a casing, or just hacking something from recycling bin contents, but it's otherwise such a hassle to have to hold the shooter together while using it, that I think it can't help but influence perception of the affordance.

My idea here is I finally gave a voice note a try because I had a thought on the shuttle on my way to campus and couldn't archive it otherwise. I may think about using these more often in future since I have noticed I struggle with design thoughts at inopportune times. For later ease of analysis, here is the transcription of the voice note:

"Okay, trying something a little bit different for this [um] update. It's about 10:50 am on a Friday and I just got of the shuttle to the downtown campus. [um] While on the shuttle my mind just started just sorta wandering thinking about the affordances in Splatoon—[um] and this is probably getting a little bit ahead of my current prototype—but I'm thinking about the different kinds of equipment and weapons that show up in the game. In particular, [uh] I started thinking about their– their giant paint brushes or rollers [um] that are used to paint the environment and I previously had been thinking that these might end up being [uh] sort of laser brooms, if you will, with lasers projected [um] onto the ground just ahead of the broom head for the [um] for the camera to detect, but it's occurring to me that because I am actually just checking for absolute luminosity in the visible frame, I can probably just make it a self-luminous broom head by putting lights inside. [um] As I say, this is probably a little bit ahead of the curve, but I wanted to go ahead and archive this while I had the thought, because I know this is going to be useful at a later stage. "