Experiment Design

To run an experiment, I needed to define a population, as such I designed a screener survey and distributed it out amongst SCAD students and peers online to attempt to broaden the population and get an initial idea of baseline critical thinking dispositions. I already knew it would be unlikely to reach a population size that provides statistical validity, so I tried to control for as many variables as possible. Such as selecting for currently enrolled or recently graduated (within 2 years).

In addition to selecting for a specific range of enrollment, I also identified cluster patterns in responses so that I could split participants into four trial groups, 2 experiment and 2 control. I used color scales in excel to highlight patterns more easily to balance thinking dispositions across control and experiment groups. If possible, anyone indicating a strong disposition would have an oppositional or non-existent disposition added to the group to balance it out. Using the verbs included in the questions assessing each metric of the CCTDI I correlated which cognitive process it would correspond with, referenced against a large list of verbs provided in the theoretical framework.

- Specific Verbs correspond to Cognitive Processes.

- Cognitive Processes can be related to CCTDI metric via Verbs and Facione guidance

- Trial groups hand placed to balance clusters of disposition profiles.

Experiment Testing Artifact Design

During the initial study population selection, it was necessary to playtest and develop the testing artifact. Through multiple rounds of testing, various designs were explored. Initially, the design began under the working title “Neuro Net” following the influences from Charles Kadushan’s Social network theory. It attempted to create a strong network between the teacher and student, the student and the lesson, and the lesson and feedback. While much of this design from this stage was changed, it introduced many of the baseline mechanics that informed later iterations of the design artifact.

Following two rounds of usability play testing with 13 participants it became evident that the current design was not only over scope but was largely hard for all participants to understand. By attempting to address the identified gap through combining insights from concepts in Social Network Theory not only did communication between the two groups not improve, both groups left the experience actively frustrated. While this resulted in the lesson planner, a tool that would allow teachers to instance their lessons easily into the mechanics of the game, being removed— the goal of using a general visual metaphor to reinforce the process of constructing meaning stayed through every version. Initially, players would travel around a map to find different puzzle areas after completing a simple tutorial puzzle to ensure they understand the control of the game. After discarding the lesson-planning designs and focusing on the student needs, further research about CCTDI shifted the design to be entirely discipline neutral and focus on encouraging disposition towards critical thinking behaviors.

Design Revisions

Following the playtest, the final design treatment began development. In order to easily control the variables being passed between actors a structure was created to house gameplay tags that would serve to filter universal variables between interacting actors. The primary tag in use was the CSVS tag, representing the ‘resource’ that players were attempting to manage

Use on Screen Arrow Keys to interact w/ prototype. Click valid actors to make connections

Design Revisions (cont.)

The Figma prototype served as a stand-in during the development process so that puzzle ideas could be tested while the work of creating the blueprints was being completed. The figure below shows the nine puzzle areas designed in the Figma prototype that became the experiment level test.

The upper left-hand section would serve as the diegetic tutorialization, featuring a harvester actor collecting CSVS from a source node and using it to power a light on the island. Following the explanation of player controls, the subject should be able to recognize that they can do the same. Using some of their health to power the mover in the center section the player can leave the first puzzle, the island they spawned in, and progress to the rest of the level.

During this time additional functionality playtests were conducted to verify that during the re-development process the data structures worked appropriately and were legible to the testers. While this phase made extensive use of developer feedback debugging tools it became clear through tester feedback that each node should have a display over it that allowed for easier planning.

-UI Feedback

-Click connections

-Drag Clicks

Accidental connections

This was also the time to ensure that crucial to filtering of the resource variable between actors occurred to carefully ensure that when a node receives a connection it knows which specific actor it is now in network with and how much CSVS it can pass through the network at a specific rate

Data Structure

As this is a lot of information to pass back and forth, we’ll show you a peak on what makes that magic reliable. The primary variables were handled via data structures that held strings that would be compared against an actor's selected item in an enumerated list to validate results.