The Partial Truth About Virtual Objects: How Users Reason About and Explore Intentionally Simplified Environments

Juul, Jesper, Emil Hammar, Dooley Murphy, and Alessandro Canossa. 2026. “The Partial Truth about Virtual Objects: How Users Reason About and Explore Intentionally Simplified Environments.” In Proceedings of the 21st International Conference on the Foundations of Digital Games. Association for Computing Machinery. https://dl.acm.org/doi/10.1145/3815598.3815648.
https://jesperjuul.net/text/partialtruth/

Introduction

In the popular technological narrative about virtual objects and about virtual reality, virtual reality perfectly replicates the regular world – or at least will do so soon. Developers and researchers are clearly aware that current technology may not live up to promises, but the popular story seeps into even well-founded research and theory. The popular story about virtual objects and reality has four main components:

• In the popular narrative, virtual objects are—or are becoming—practically identical to their physical counterparts, often known as “digital twins,” [16] such that they have similar properties, and that they share affordances [34] with their physical counterparts; that users can do with the virtual object what they can do with the original.
• Virtual environments are often described as converging on one large world or metaverse [2].
• Virtual reality is often presented as entirely natural, (ideally) having “no interface” [23] that users need to learn [4], only “natural gestures” [24].
• The end point for virtual experiences is perceived as an idealized state of presence [26] or immersion [27].

This popular story can appear extreme and, as hype often breeds a backlash, it becomes tempting to respond by taking sides, seeing virtual objects and experiences either as practically real and as valuable as regular objects [7] or as entirely fictional [29] and hence obviously inferior to the physical world [36].

However, as we will demonstrate, the popular narrative seriously misrepresents virtual objects and virtual reality. Figure 1 shows the 2016 Uncharted 4 [31], a game developed with significant budget and technical resources, here with a boat on a beautifully rendered body of water. Yet unlike regular water, this water cannot be scooped up, it cannot evaporate or freeze, it is not wet, and it cannot partake in most of the interactions that regular water affords.

Compare this to the water in Figure 2: Stardew Valley [8] water is in many ways much simpler, lacking the complex rendering and water surface dynamics of Uncharted 4, yet with the difference that Stardew Valley water can influence plants by watering them. In Uncharted 4, water is designed for sailing on and swimming in; in Stardew Valley, water is designed to help grow plants. This difference in water implementation, with the small budget game having water features that the big budget game lacks, shows that while it is true that virtual objects are generally simpler than what they represent [1, 18], this simplification is not a scale from the detailed to the abstracted—rather the implementation of virtual objects is specific, with specific goals determining what is included and what is omitted.

Virtual Objects and the Problem of Matrix Philosophy

Are virtual objects real? In the field of philosophy, this has become a subject of countless books, conference papers, and journal articles. It is the type of question that can appear both profound and silly, because on one hand virtual objects—from on-screen buttons, to money, to video game objects—are unproblematic parts of everyday life alongside physical objects. On the other hand, virtual objects usually appear on screens, much like other kinds of represented objects (in paintings, movies, and so on) that are usually taken to not be real. This means that it is easy to use virtual objects but difficult to explain what they are.

Prominently, philosopher David Chalmers has argued that counter to common intuitions, “the entities in virtual reality really exist” [5]. This is a memorable argument, which Chalmers first outlined in a paper about The Matrix [6]. Chalmers categorizes philosophers as either irrealists, for whom virtual objects are fictional/imagined, or realists like himself, for whom virtual objects are real. To begin with Chalmers’s realist position, he argues that virtual objects are real by virtue of being grounded in computational data structures [7], such that an on-screen car really “exists” because it is defined by numbers/electrical signals in the computer. This is both interesting and ambiguous. Is it that the data structures exist, that the corresponding virtual car on screen exists, or both? [14]

The counterargument of claiming that virtual objects are fictional and not real compares video game objects to the kind of fictional objects we experience in literature or film [40], such that it is as much of a paradox that we are “Moved to Shoot Zombies” in games [29] as it is that we are moved by the fate of Anna Karenina [37]. This argument can be tied to the view that virtual objects and realities are by default less important than regular life [36].

One problem is that Chalmers is effectively performing Matrix Philosophy—a stance marked by a focus on technological promises or fictional representations of virtual reality (The Matrix, Snow Crash, or Ready Player One). As Grant Tavinor argues, Chalmers is focused on “perfect virtual realities; that is, hypothetical virtual worlds,” but ignoring actually existing virtual reality systems [40]. The defense of this philosophical discussion could be that it, like much philosophy, is about seeking conceptual clarification, but we argue that is made difficult by a confusion about what is meant by “virtual reality” – are we discussing a hypothetical technology or the technology that humans are actually creating and relating to? Janet Murray similarly identifies how a “confusion of technological prediction with magical thinking has spread to academic discourse” [30].

Focusing on the virtual objects available today, Juul (the present paper’s first author) has argued for a third way; the position that virtual objects have both statuses—they are half-real: Real digital objects governed by real rules giving rise to real social interactions and real events such as winning or losing (as applicable), yet fictional in the sense that they are generally not what they claim to be [17]. Furthermore, Juul has argued that although virtual objects appear to us as instantiations of (often) physical objects—a digital calculator, for example—they will always lack some properties that, say, a physical calculator possesses, such as the ability to remove its batteries. This holds all the way down to a quantum level, where our scientific understanding is incomplete anyway [20]. This inverts the Chalmers argument about data structures in that while data structures and corresponding programming arguably make virtual objects real in one sense, the data structures are selective approximations made for a purpose, giving all virtual objects fictional aspects.

Rather than claim that virtual objects either do or do not exist, we will here therefore explore a Partialist position from our studies of users, where virtual objects are defined by their partial status, their partial implementation, and so on. Of course, users and virtual objects have been examined before, but there still is a range of underexplored issues. On the one hand, interface design literature has generally focused on the design of 2D interface objects [32], and though it has examined how cultural backgrounds influence the perception of interface metaphors [10], it does not amount to a systematic theory of how users perceive virtual objects. The other problem is that studies of 3D environments have primarily focused on questions about the total experience of users—whether users feel immersed and transported to another location [3], or whether users feel that virtual objects are present to them [25]. To the extent that affordances have been discussed, being able to do something in virtual worlds is often seen as a mere stepping stone to total presence and immersion [15].

Goals and Methodology

What does it mean to say that virtual objects are one thing or another? We find it misleading to approach virtual objects as a kind of thought experiment based on fictional representations in novels and movies, because what then is the actual object of study? It is neither the virtual objects that we know exist, nor the experience of users, and much philosophy of virtual objects has therefore often ignored both the users and the objects.

In this paper, we build instead on the tradition of experimental philosophy [12, 21], where we take philosophical ideas that have mostly been explored theoretically and design new studies that allow us to probe these ideas practically. We have decided on four studies that approach virtual objects from four angles, with each approaching a core unresolved issue. The studies form a movement from the general to the specific, punctuated by returning to a broad perspective again. The first study concerns the biggest questions of the reality of virtual objects: How users determine their reality status and what kind of objects they are. This leads to the second study of how users make assumptions about what an object can do—what Don Norman calls its affordances [35]—based on the initial visual impression of virtual object. The third study then examines the moment-to-moment explorations that users perform in a virtual reality environment. Finally, the fourth study zooms back out and examines whether users have a broader meta or genre awareness of how virtual objects are implemented differently in different software. These are the four studies:

• Study 1: What status do users give to pictures of objects, physical objects, and virtual objects? How do users transfer knowledge from “regular” objects to virtual objects, and on what grounds? Do users sort different kinds of objects into logical categories, or do they approach them in a more improvisational and practical way? We devised a simple test where users were prompted to reason about the status of a physical ball, a ball in a picture, and a 3D virtual ball.
• Study 2: How do users use visual style to determine the affordances of a virtual object? If we then assume the possibility that users have granular ideas of how to interact with objects, we can begin to probe how users make such calls: When facing a specific virtual object, how do users determine its affordances? We showed users a similar object (a car) in three different visual styles and prompted them to identify the affordances they saw.
• Study 3: How do users initially explore the affordances of virtual objects? Having examined the initial approach of users (through visual style), we also wanted to understand the physical explorations that users perform when faced with a novel environment. We recruited novice virtual reality users and let them explore Meta’s First Steps with Hand Tracking [11] as a polished environment designed to teach users about virtual reality and hand tracking, and recorded and analyzed how users explored the environment.
• Study 4: Do users perceive that objects are implemented with goals in mind, specific to kinds (genres) of software? Finally, we wanted to explore whether users have broader “genre” conceptions of interactions: do users understand that a given type of object can be implemented very differently in different pieces of software? Do users perceive software as designed with intention? In this test, we showed users an image of a 3D car but explained to some users that it was a game, and to some that it was a simulation to assess the safety of cars during crashes.

General Method

As we were interested in identifying and exploring the reasoning and justifications that users applied for conceptualizing virtual objects, we employed qualitative interviews and analysis. By interviewing research participants (n = 47) directly, we could be flexible in asking to elucidate specific answers or reactions to our queries, as well as observing specific types of behavior when interacting with (non-)virtual objects. These studies are focused on rich qualitative data, pointing to potential future studies with larger cohorts.

In the following discussions, we have anonymized the users, numbering them consecutively per study (S1_1, S1_2, S1_3 etc.).

On all four studies, we queried participants about the amount of time spent playing digital games and on which platforms, allowing us to explore differences in approach based on game-playing experience.

Study 1: The Status of Pictured, Virtual, and Physical Objects

How do non-philosophical users categorize virtual objects in comparison to other types of objects, and on what grounds? We prompted 12 users to distinguish between a physical ball, a virtual but manipulable 3D ball, and a static digital image of a 2D ball (Figure 3). We used an existing virtual environment made in Unity3D [19], in which the user’s avatar spawns in front of a three-dimensional ball. To the left of the user’s avatar, a virtual billboard displays a two-dimensional picture of the same ball (Figure 3). Through mouse clicking or touchscreen interaction, users can move the avatar around the environment, where the avatar can collide with the objects on-screen. The 2D picture of the ball was not capable of moving or changing state in response to user input.

The study also included a physical, tangible ball placed next to the mobile device screen. This was the size of a tennis ball with silver foil around it, giving it a coarse texture. Participants were asked how they perceived and interacted with the three different instantiations of a ball (for more study details, see Appendix 1).

Study 1 Results

Unprompted, participants directed their focus to the virtual ball, as if they already knew the affordances of the pictured ball (none), and of the physical ball (many), but understood the virtual ball to be the one whose properties and affordances needed to be identified through interaction. Their identification of the affordances of the ball was mainly explorative—almost all users attempted to interact with the virtual ball before answering questions about it.

It was only through explicit calls from the experimenter to direct the attention towards the picture of the ball that users investigated it and tried to interact with the billboard before replying to the question of whether they could push, pick up, or throw this ball (they couldn’t). “That goes without saying,” stated participant S1_5.

We initially assumed that users would correspondingly find on-screen objects to be not “real” (cf. Chalmers [5]), and not real instances of a type (i.e., not a real ball). Surprisingly, three of twelve participants described the virtual ball as “real.” Of those three, S1_2 stated conditionally that, “in some way, the [3D virtual ball] is real, it is a part of the screen, so it does indeed exist,” while S1_9 stated that the virtual ball is real because “it is a physical object that interacts with the person and the world.”

Nine of the twelve participants first stated that they did not think the virtual balls were “real.”

S1_7: “They’re just programmed pixels on a screen.”
S1_5: “No, there’s nothing in this game that’s [real]. The scale ratio is not real either, because the ball is really big. It’s just as tall as the little figure.”

Three of the remaining nine participants later changed their minds and stated that as it looked and behaved like a real ball, it was genuinely real:

S1_10: “You can interact with. Because the 3D ball shares properties with a physical ball, including casting a shadow, it is a real ball.”
S1_12: “Because they look like real balls, cast shadows, and the form is the same, and the person is using the ball the same way [as a real ball], they can push it a bit like a football.”

This type of reasoning is colloquially known as “the duck test” [13], where the status of an object is derived from its observable properties. The concrete example also includes a kind of “virtual boot strapping,” where having one object influence another object makes the former be perceived as more real due to its causal powers, even if those powers are applied to something not considered real in the first place.

The way participants interacted with the screen was also dependent on their everyday experience with games and mobile devices. Some of them tapped the screen when trying to move the character, others pinched the screen, and others dragged their index finger or swiped it upwards and towards the ball. In addition, prior self-reported experience with video games seemed to significantly shape how users referred to the avatar. Participants reporting little game experience described the avatar as a step between themselves and the environment, whereas more experienced game players did not refer to the avatar, apparently considering it transparent between them and the virtual balls.

Another significant difference between novice and experienced game players appeared in their assumptions about objects: Participants with little gaming experience generally assumed the virtual ball would act similarly to its material counterpart, but those with significant gaming experience both assumed that virtual objects would have missing features (likely due to their previous experience with virtual objects) and anticipated that virtual objects could have affordances beyond what the physical type would have. For instance, three participants with prior gaming experience explicitly stated that the 3D virtual ball could “do” more than a regular ball, such as being rolled up a hill or pushed across the ground. This mirrors observations about how experienced computer users have more “abstract” interpretations of interface metaphors [32].

Interacting with virtual objects came naturally to participants, but verbalizing and philosophizing about the objects yielded pauses and furrowed eyebrows. It is likely that our questioning forced users to perform a verbalization they might not otherwise have performed [33], and our intervention likely structured the participants’ reflection in a way that unprompted gameplay would not. We see participants’ answers not as the truth about their internal thought processes, but as a documentation of their attempts at consolidating their experience of virtual objects with accepted categories and explanations. The results demonstrate exactly that – counter to what the existing literature suggests - these categories are quite brittle and easily influenced by seemingly tiny variations in setup and framing.

Study 2: Using Visual Style to Determine Affordances

Virtual objects are implemented in different ways, and in many visual styles. We wanted to explore whether participants used the visual representation of objects to approach them. For this study, we took screenshots of three representations of cars in the Unity3D engine and printed them on a one-page document. Experiencing a screenshot or recording of a game is different from experiencing running software, but images of video games are a central part of the video game ecology, used in advertising, social media posts, and reviews. It is also common to experience video games by watching others play, in-person or streamed [41]. The study’s purpose was to reveal what affordances [34] the study participants associated with each car and on what grounds (cf. Appendix 1 for questionnaire). We asked the 12 participants to ascribe affordances to each car. Like in Study 1, we recorded the answers and followed up on each question about the interviewee’s reasoning for their responses.

As Figure 4 shows, the three cars were differentiated by graphical complexity. The top car is a simple low-poly model with two simply colored surface textures and clear pixelation. The middle “police car” is a more graphically advanced car with a higher polycount, but still simplistic in some respects such as the rectangular wheels and the simplistic texturing as seen on the license plate. Lighting is also more complex than for the top car. Finally, the bottom car is the most graphically complex, bearing the most verisimilitude to an actual car (a classic Volkswagen) with a high polycount, high resolution texturing, metallic shading, and Unity’s most advanced lighting through multiple light sources in the scene. The bottom car is also presented in a more detailed test map with a sky backdrop and boxy scenery.

Learning from visual style

To explore how users saw the pictured cars in relation to regular cars, we focused our questioning on prototypical car affordances, such as driving, opening doors, opening the hood, and running out of gas. Furthermore, we asked about the size and weight of each car to gauge if users were considering them as physical objects. After each question (Appendix 1) we asked about their reasoning.

Results were both straightforward and complex: Most participants reasoned that the cars could drive due to the presence of wheels. S2_11 stated that because they thought all three cars were “in a game,” all cars could drive. S2_1 stated that they did not know if all three cars could drive because “it is a picture.” Other participants stated that only the more detailed police car and Volkswagen could drive, as the inside of the cars and their steering wheels were visible, unlike in the simple model.

Participants generally attributed more affordances to the detailed 3D car models compared to the low-polygon model. Visual cues like door handles and hinges led users to assume that doors and hoods could be opened. This meant that the low-poly car was generally assumed to have fewer affordances than the two more detailed cars had (with an important exception discussed later).

S2_5: “There are no doors on the pixelated car, therefore they cannot be opened.”

In a similar vein, participants assumed the more detailed cars had engines inside of them, because they were able to identify the car hoods.

S2_2: “I do not know if they have an engine but there is something in my head that tells me that the two below have an engine and the top one doesn’t.”

Nearly all participants thought the low-poly car had neither doors nor a hood, due to its lack of visual signifiers of door handles and hood indentations.

The role of genre expectations

For some, the affordances were based not directly on the visual style itself, but on its similarity to older games they had played, and genre experience was used as reference:

S2_8: “The top one I look at very much like some of the very first video games.”

Participants made inferences about the cars based on the imagined context they were in. As one stated about running out of gas:

S2_2: “It depends on the context, if they are in a game, then they can [run out of gas].”

One participant stated that the two more detailed cars looked like they would be in a realistic game, while the low-poly car would be in a more “playful” game (e.g., “slipping in oil”):

S2_10: “All can drive because they are programmed to drive. In a digital world, they can all drive.”

The role of environment

Counter to expectations, the 3D environment around each car was sometimes used to assess the car’s affordances. For the low-poly car, one participant said it was not a three-dimensional object due to the flat lighting and monochrome background. Two other participants stated that the background environment and platform helped them see that the highly detailed Volkswagen car was a 3D car, and not a picture of a real car.

S2_5: “Even the most realistic car, even though it looks very realistic, the 3D background shows that it’s not a real car.”

Participants assumed that the more realistic looking cars could have their tires flattened, but the low-poly car could not. Few participants differentiated between car 2 and car 3, seeing them both as highly detailed. S2_5 was one of the few who saw some differences between the middle and bottom car, because he had experience working with cars.

Simple objects with complex affordances: The influence of Minecraft

The general rule was that detailed graphics were assumed to correspond to detailed and rich affordances, but with an exception: For the simple car, those familiar with LEGO or Minecraft [28] games ascribed it more affordances. S2_6 had not played many video games themselves, but had watched their children playing Minecraft, so they believed that the low-poly car could perform many actions “like in Minecraft,” given how the graphically simple game objects in Minecraft can also perform a variety of actions. Many participants compared the low-poly car to a wooden toy car or a LEGO car, which we interpret to mean that users understood the simplification of the low-poly car as similar to the simplification exhibited by a toy or model car.

S2_8: “The top car looks more playful because it might be able to spin around in a pool of oil in some sort of game … very much like some of the very first video games” (that they played when they were younger).

Many users’ understanding of simple 3D objects thus seemed based not only on the detail of visual style, but on their personal experience with specific games. Our studies suggest that the popularity of a single popular game like Minecraft can strongly influence how some players experience visually simplified virtual objects.

Study 3: Exploring Novel Virtual Environments Hands-On

But how do users explore a novel virtual environment? To analyze this, we used Meta’s First Steps with Hand Tracking [11] introductory environment (Figure 5), which is expressly designed to introduce users to hand-tracked interaction in virtual reality. We recruited users with little to no prior VR experience for a concurrent think-aloud study (n = 10), taking first-person videos and performing analysis on the timecoded recordings and transcripts.

The environment presents users with a range of objects, from blocks to ping pong balls and paper airplanes, all of which the software via floating text labels and voiceover narration encouraged participants to inspect and manipulate. In the study, it became clear that users approached the environment holistically. They attended to individual objects not by placing them in logical categories, but as a larger exploration of the behavior of objects, their interaction, and, significantly, the simulated forces that govern them. By default, users assumed that the virtual environment mirrored the structure and rules of the physical world, and they spent significant energy verifying regular physical actions, exploring for example if it was possible to pick up objects and throw them. However, participants were especially interested in, and often verbalized, moments where the virtual environment did not behave as expected.

First Steps with Hand Tracking’s focus on hand tracking implies a recurring design conundrum in virtual reality: If the user moves their hands down to a physically impossible position, such as inside a desk, should the hands be visualized as stopping on top of the desk (leading to incongruence between physical and virtual hands), or should the hands be visualized as mysteriously penetrating solid objects? In First Steps with Hand Tracking, the developers chose the latter. This involves a range of hand-crafted special-case design decisions, where the virtual hands are sometimes solid and able to carry other objects, sometimes immaterial and able to pass through some objects while at the same time pushing others. This is of course highly unusual for physical objects, as participants spontaneously remarked on:

S3_2: "Because I can't feel the geometry, it's a bit weird. But because my hands are going into the geometry."
S3_9: "I can see the edge of the table, but I'm inside now, in the table!"
S3_9: "I'm expecting to be able to lift it, but it sort of slides through my fingers."

Another technically understandable design decision, due both to the problem of hands intersecting objects and to the imprecision of hand and finger tracking, is that picking up an object is not performed by enclosing it with your hand, but by performing a “pinch” gesture, which then snaps the nearest object into a position on the virtual hand. While controlling virtual hands was at once familiar to users, they were also the biggest source of surprises:

S3_6: “It's a bit, errr, funny to do the pinch movement when the virtual hand is doing something completely different."

Over time, users accepted these peculiarities as a property of the virtual environment, and some used regular physical concepts to explain it:

S3_8: "Okay, so now I get it that I have to squeeze my fingers entirely when I pick up the..."
S3_7: "Yeah, it's better now. So now I can start playing with it, blocks."
S3_7: "Ah, I need to close my hands to catch it."
S3_3: "At some point, it sticks to my hand, right? ... Magnet? Magnetic?”

Similarly, special-case physics and inconsistencies were noted:

S3_5: “Objects don't collide when dropped on each other but do when dropped from a height.”
S3_1: “But the bat feels really weird.” (Figure 6)

These user observations were part of a learning process, where users eventually became used to the specifics of this virtual environment. This process was apparently unremarkable for users, who all accepted that this was a specific environment with its own rules and physics to learn. This indicates that virtual objects and virtual environments should be understood not as a technological version of the regular world, or even as one alternative context to be learned [42], but as a series of unique environments, each designed with specific constraints and goals in mind, leading to specific design decisions that users must learn.

Study 4: To What Extent Do Users Understand That Implementations Differ?

For software developers, it is a truism that a given problem can be programmed in myriad ways. But are users aware that different “genres” of software [22] will implement ostensibly the same object in different ways?

In this test, we showed printed out pictures of virtual cars to participants, but in half of the 13 interviews, we primed participants to think it was part of a “car simulation safety test for a car manufacturer,” while for the second interview group, we told participants that the car was “part of a digital game.”

With no training in software development, most users readily explained that if a feature of the car served a purpose in the context, it would have to be implemented in the software. Like in previous tests, users also identified affordances based on visual identifiers (doors, engine hood, interior parts) but showed surprisingly holistic ideas about the goals of different types of software. For example, users argued that an engine was not a requirement unless it fulfilled a purpose in the game. Similarly, participants explained that in a simulation, tires would be able to be deflated, if the purpose of the simulation tests required that they could be deflated.

Simulation framing:

S4_2: Argued that things need to be simulated in the car “to prove a point.”
S4_12: If it was a simulation software, the car had to have an engine and the ability to break into pieces because “it being a simulation, it needs to crash, and I imagine it can happen.”

Game framing:

S4_4: “It probably doesn't have all the things because that would be very impractical, but if you open the hood and there's nothing there, it would be a bit strange, so if that's something you can do, I could imagine it being there.”
S4_4: “Yes, if it would make sense [to split it into pieces and see its structural weaknesses], if it was part of the game.”

While all participants distinguished between entertainment and utility software, game-experienced users had more granular ideas about games (S4_9 stated, “depends on if it’s a AAA or an indie game”), assuming for example that an object would have weight if it was relevant for the particular game genre:

S4_4: "It could [weigh something]. It depends on what kind of game it is, but I could imagine it being a game mechanic."

When framed as a simulation software, S4_13 said that the car would be able to drive when “there’s a reason to test it.” They similarly stated that tires needed to be able to deflate and the car to break apart for testing purposes.

When S4_6 was asked if they would have answered differently if they had been told that it was a game rather than simulation software, they replied that “all the engineer questions would probably answer with a no” and “you just program what it is supposed to do instead of thinking about how it technically would work.” S4_13 similarly revealed that “if it was a game, I would make fewer demands on how it should be built by different parts and the thing with gravity and stuff like that.”

Study 4 thus showed that counter to our expectations, users with no background in software development had detailed ideas about the implementation decisions that go in different kinds of software. They specifically understood virtual objects to be designed for a purpose, and that the features implemented in a virtual object reflect the purpose of the software. In Daniel Dennett’s term, users adopted a design stance and saw virtual objects as deliberately designed by humans for particular purposes [9].

Conclusion: The Partial Truth About Virtual Objects

Table 1 shows the most common misunderstandings in the simplistic technological story of virtual objects, along with what we have identified as the actual properties of virtual objects and the actual user response to them. As we’ve argued, the study of virtual objects has been skewed by “Matrix philosophy,” where virtual objects and virtual reality are discussed primarily on the basis of their representation in movies and novels, with a focus on the properties of a future virtual technology. In this narrative, a virtual object is (or will become) a twin of regular objects; all virtual environments will eventually merge into one; there is no (or will be no) need for interface design, and the biggest question is whether virtual objects are real or fictional. Focusing on the simplistic technological narrative ignores the fact that actual virtual objects come in wide variation and specificity and, as we have shown, that users are aware of this fact. We believe that thought experiments are valuable, but we find that they in this case suffer from slips between virtual objects in fictionalized representations and the properties of actually existing virtual objects. We are not the first to point out this problem [30, 40], but we hope to have contributed to the discussion by performing experimental studies of actually existing virtual objects and virtual reality. These studies have allowed us to examine virtual objects and -reality, including user responses to them, pointing to further exploration of how users respond to virtual objects, and of how implementation, user experience, and language interact. It would be interesting to expand these studies to larger cohorts and to users from more varied backgrounds.

We have in this paper focused on mundane, piecemeal, and partial questions: What status do users assign to virtual objects, and found that users are torn, sometimes switching their answers, but always assuming that exploration is needed to identify what virtual objects can do. Furthermore, we found that users use a range of cues to determine the status of an object, looking both at the visual style, their possible causal powers to influence other objects, the users’ personal associations with a visual style, as well as their genre expectations. Even novice users understood that a virtual environment could have different rules and physical laws than the regular world, accepting their task of learning the idiosyncrasies of a specific virtual environment. This exploration was also holistic, focusing not on the individual object, but on laws of the broader environment. Finally, users also understood that software and objects are designed for specific purposes, and that the nominally “same” object is implemented differently depending on the goal of the software.

This study’s results suggests that the broader HCI community should abandon hypothetical “Matrix philosophy” and binary debates about the reality status of virtual objects and focus instead on how users actually interact with the interfaces of current virtual environments. For designers of virtual experiences, this can be a liberating shift because it frees us from creating perfect “digital twins” of the regular world. Designers can instead prioritize functional design, knowing that users adapt to the specific physics and cues of a virtual world, and use genre expectations to figure out what an object can do. Ultimately, we should treat virtual objects not as flawed replicas of reality, but as deliberative, selective implementations, crafted for human experiences.

What are virtual objects in games, in VR, or in simulation software? A virtual object is an intentional technological implementation of a cultural idea for a specific purpose. A virtual object is the implementation of a human category, defined by selecting properties and intended affordances for a particular purpose—be it a game, a simulation, or any other kind of software. Virtual objects are presented to users using screens, headsets, controllers and sometimes hand tracking. Through experience, users gather a range of strategies for identifying, “what can this object do?” As opposed to the realist or irrealist view(s) of virtual environments [7], we have proposed the Partialist position, which we have now explicated through user studies: Virtual objects are always partial implementations, with partial status, approached tentatively by users. This is what virtual objects really are.

Acknowledgments

This study was supported by the EU Interreg BSG-Go project, grant COO2.

Appendices

Appendix 1: Study Setup and Questionnaires

For studies 1, 2, and 4, we targeted study and reading areas in public spaces, such as public libraries and study areas at The Royal Danish Academy in Copenhagen. The age of the participants ranged from 20 to late 50s, with a roughly equal mix of men and women. The interviews were partially transcribed through the software Whisper, which were then corrected through listening through each interview. Through an iterative, bottom-up approach driven by the qualitative transcript data, we mounted an analysis of unique and shared testimonies about how participants perceived, interacted with, and ultimately reasoned about the virtual, pictured, and material objects.

For study 3, given the use of a virtual reality headset, we judged that a quiet location was needed and only recruited from the Royal Danish Academy.

The following are the interview guides we used for each of the studies. We categorized the open-ended answers into checkboxes, appending the qualitative reasoning.

General Questions about Game Experience

Study 1: The Status of Pictured, Virtual, and Physical Objects

Study 2: Using Visual Style to Determine Affordances

Study 3: Exploring Novel Virtual Environments

Study 4: To What Extent Do Users Understand That Implementations Differ?