# Virtual Reality Virtual reality is a type of [Simulation](Simulation.md). This refers to any situation in which a person is given the experience of being in a specified environment, such as a [Flight Simulator](Flight%20Simulator.md). Virtual reality renders *logically possible* *external* experiences[^1]. ## External Experiences External experiences are those which are outside the users mind. For instance, in a flight simulator the pilot may fly through a thunderstorm. This is an external experience. However, how the pilot responds to that - be it nervous, surprised, excited - is an internal experience. Virtual reality as it will be defined here does not render internal experiences. At first this may seem too restrictive. Why can it not render internal experiences? After all we already drugs and pharmaceuticals that do that. The issue is actually a philosophical one. For a virtual reality generator to be able to specify internal experiences, it requires overriding the normal functioning of the users mind. This is effectively replacing the user with a different person. This raises quite different philosophical issues, and for that reason they are excluded from the definition of virtual reality here. ## Logically Possible Logically possible experiences are those which do not have an inherent [Contradiction](Contradiction.md) or violate the laws of logic. For example, consider trying render the experience of finding a number that is both greater than 6 and less than 6. This violates the principle of logic known as the [Law of the Excluded Middle](Law%20of%20the%20Excluded%20Middle.md). It is logically impossible to find a number to satisfies this criteria, so rendering that experience is not possible by Virtual Reality. The same goes for experiencing unconsciousness, for unconsciousness is by definition not experience anything. The law of the excluded middle applies yet again - you cannot both be experiencing and not experiencing a thing at the same time. Logical possibility is a prerequisite for [Representation](Representation.md): Any experience, whether real or simulated, must be logically possible to be represented. Simulation relies on [Computation](Computation.md), which is itself a physical process, based on *our* laws of physics. And our laws of physics do not allow for representation of logical contradictions[^2]. ## Physically Impossible Note that this does not restrict the rendering of *physically impossible* experiences - in other words, virtual reality can render experiences that violate the [The Laws of Physics](The%20Laws%20of%20Physics.md). For instance, it can render the experience of flying a spaceship and never running out of fuel. It can also render the experience of moving faster than the speed of light. At first this may seem impossible. A rendering after all is a simulation, and if we are simulating spaceship flight, then as the spaceships speed increases it would need to render the [Physical Effects of Approaching the Speed of Light](Physical%20Effects%20of%20Approaching%20the%20Speed%20of%20Light.md). The spaceship would never be able to exceed the speed of light, according to [Special Relativity](Special%20Relativity.md) an infinite amount of energy would be required to do so. But this is a mistake in reasoning! The virtual reality simulation is effectively defining its *own* [laws of physics](laws%20of%20physics.md) - in this case [Intrinsic](Intrinsic.md) rules and constraints that the program must follow. It is completely acceptable for it to define its laws of physics to allow velocities faster than light. Of course that would change the relationship between distance and time, but as long as it yields a set of rules that are [logically consistent](Logical%20Consistency.md), anything goes. ## Ultimate Limits The phenomenon of virtual reality in general occupies a central place in the scheme of things. To consider virtual reality in as general a way as possible, we can ask: * What are its ultimate limits? * What sorts of environment can [In Principle](In%20Principle.md) be artificially rendered, and with what accuracy? If we define the *repertoire* as the set of real or imaginary environments that the generator can be programmed to give the user the experience of, we can then ask: > What constraints do the laws of physics impose on the repertoires of virtual reality generators? ### The Ultimate Limits of Image Generation We can refer to the creation of artificial sense impressions as **image generation**. What constraints do the laws of physics impose on the ability of image generators to create artificial images, to render detail and to cover their respective sensory ranges? Consider the [Flight Simulator](Flight%20Simulator.md) again. Can a realistic aircraft and its surroundings be rendered, [In Principle](In%20Principle.md), with the greatest level of detail the pilot's sense can resolve? For the sense of hearing, that level of detail has already been nearly achieved with hi-fi systems. The same goes for sight. But what about the sensation of weightlessness? At first it may seem impossible to [Simulate Free Fall](Simulate%20Free%20Fall.md) and the associated sensation of weightlessness. But that is not so. Weightlessness and all other sensations can, [In Principle](In%20Principle.md), be rendered artificially. Eventually it will be possible to bypass the sense organs altogether and directly stimulate the nerves that lead from them to the brain. Thus [The Laws of Physics](The%20Laws%20of%20Physics.md) impose no limit on the range accuracy of image generators. There is no possible sensation, or sequence of sensations, that human beings are capable of experiencing that could not in principle be rendered artificially. That every possible sensation can be artificially rendered is one thing; that it will one day be possible, once and for all, to build a *single* machine that can render any possible sensation calls for something extra: [Universality](Universality.md). A machine with that capability would be a [Universal](Universality.md) *image generator*. The possibility of a universal image generator forces us to change our perspective on the question of the ultimate limits of image generation technology. At the moment progress in such technology is all about inventing more diverse and accurate ways of stimulating sense organs. But that class of problems will disappear once we have cracked the codes used by our sense organs, and developed a sufficiently delicate technique for stimulating nerves. Once we can do this with enough accuracy that our brain cannot determine the difference between those signals the ones our organs would send, the technology will have come of age, and the challenge for further improvement will not be how to render given sensations, but which sensations to render. What makes an image generator universal is that given a recording of any possible image, it can evoke the corresponding sensation in the user. This requires the image generator to be able to play recordings of unlimited duration. This requires a mechanism of swapping in and out memory, supply of an energy supply, a cooling mechanism and periodic maintenance. ### Interactivity Virtual Reality also requires interactivity. It does not play a pre specified set of images, but it composes images as it goes along, taking into account a continuous stream of information about what the user is doing. The human mind affects the body and the outside world by emitting nerve impulses. Therefore a virtual-reality generator can in principle obtain all the information it needs about what the user is doing by intercepting the nerve signals coming from the user’s brain. Those signals, which would have gone to the user’s body, can instead be transmitted to a computer and decoded to determine exactly how the user’s body would have moved. The signals sent back to the brain by the computer can be the same as those that would have been sent by the body if it were in the specified environment. It is worth noting that these ideas have been around for hundreds of years, expressed in [Descartes Evil Demon](Descartes%20Evil%20Demon.md) or the [Brain in a Vat](Brain%20in%20a%20Vat.md). ### Necessary Components of Virtual Reality We now arrive at the [Necessary but possibly not Sufficient](Necessary%20but%20not%20Sufficient.md) set of principal components for VR: 1. A set of sensors to detect what the users is doing (nerve impulse detectors) 2. A set of image generators (nerve stimulating devices) 3. A computer in control Up until now we focused almost solely on (1) and (2). However, when we look beyond transient technological limitations, we see that image generators merely provide the [Interface](Interface.md) - the "connecting cable" - between the user and true virtual reality generator: the computer. It is entirely within the computer that the specified environment is simulated. It is the computer that provides the complex and autonomous ‘kicking back’ that justifies the word ‘reality’ in ‘virtual reality’. Notice that (1) and (2) contribute nothing to the perceived environment, just as nerves are not naturally perceived as being part of our environment. Thus, virtual reality generators of the future would be better described as having only one principle component - a computer - together with some trivial peripheral devices. It is not that the practical problems of (1) and (2) are simple. It is just that they are a finite set of problems that need be solved *only once*. ### Rendering a Virtual Environment To **render** an environment means to create a simulation or representation of an environment that evokes in the user a sensory experience with some likeness to that environment. Specifying a virtual-reality environment does not mean specifying what the user will experience, but rather specifying how the environment would respond to each of the user’s possible actions. Consider a rendering of the [Center Court at Wimbledon](Center%20Court%20at%20Wimbledon.md). Its accuracy is a [Logical Property](Logical%20Property.md) of the [Program](Program.md). It can never be certified as perfectly accurate, but if one finds even one minor discrepancy, it can immediately be certified as *inaccurate*. The discussion of accuracy in virtual reality mirrors the relationship between theory and experiment in science; namely, the [Asymmetry Between Experimental Refutation and Experimental Confirmation](Asymmetry%20Between%20Experimental%20Refutation%20and%20Experimental%20Confirmation.md). There it is possible to confirm experimentally that a general theory is false, but never that it is true. And there too, a short-sighted view of science is that it is all about predicting our sense-impressions[^3]. The correct view is that, while sense-impressions always play a role, what science is about is understanding the whole of reality, of which only an infinitesimal proportion is ever experienced. The program in a virtual-reality generator embodies a general, predictive theory of the behavior of the rendered environment. Thus if the environment is physically possible, rendering it is essentially equivalent to finding rules for predicting the outcome of every experiment that could be performed in that environment. Because of the way in which scientific knowledge is created, ever more accurate predictive rules can be discovered only through ever better explanatory theories - [Predictions are Consequences of Explanations](Predictions%20are%20Consequences%20of%20Explanations.md) after all. Hence we can state: > Accurately rendering a physically possible environment depends on understanding its physics. ## Discovering Physical Laws Depends on Virtual Reality But - and this may be quite surprising - the converse is also true! > Discovering the physics of an environment depends on creating a virtual reality rendering of it. Lets break this down. We start with that claim that discovering the laws of physics of an environment depends on creating an accurate virtual reality rendering of it. Remember that a virtual reality rendering is a [Program](Program.md) that when run yields a [Simulation](Simulation.md). A program defines a set of [logical constraints or rules](Logic%20and%20Programs.md) that make up the "[laws of physics](laws%20of%20physics.md)" of the simulation (rendering). Because these are logical rules, they must satisfy constraints of logic such as [Consistency](Consistent.md). Now, normally one would say that scientific theories only describe and explain physical objects and processes, but do not render them. For example, an explanation of eclipses of the Sun can be printed in a book. But rendering the eclipse in virtual reality would require both further programming and further hardware. However, those are already present in our brains! In other words, the explanation in the book constitutes a combination of a [Program](Program.md) and *input*. Our *mind* has programs that can read this program as input, and then be executed as a [Computation](Computation.md). The result of the execution is dependent on the: 1. Program of the eclipse (it's description) 2. Program in the mind that [Interprets](Interpretation.md) the program of eclipse (the symbols on paper) The words and numbers printed in the book amount to to ‘descriptions’ of an eclipse only because someone knows the meanings of those symbols. Put another way, someone knows how to [Interpret](Interpretation.md) them. That is, the symbols evoke in the reader’s mind some sort of likeness of some predicted effect of the eclipse, against which the real appearance of that effect will be tested. Moreover, the ‘likeness’ that is evoked is interactive. When our mind runs (1) via (2), this yields a virtual reality rendering - a [Simulation](Simulation.md). This follows a set of [logical constraints or rules](Logic%20and%20Programs.md) that make up the simulations "[laws of physics](laws%20of%20physics.md)". In other words, a system with it's own [Intrinsic](Intrinsic.md) properties that govern behavior[^4]. Again, taken as a whole this constitutes a [Program](Program.md) that is run as a [Simulation](Simulation.md) in our minds. Our minds are effectively constructing a simulated experience of the eclipse based on the computers description. Our understanding of the physical world is always mediated through our internal representations and simulations. One can observe an eclipse in many ways: with the naked eye, with a camera, or using various scientific instruments, and from different places on earth. In each case an observer will experience different images, any of which can be predicted by the theory. What the printed description in the book evokes in a reader’s mind is not just a single image or sequence of images, but a general method of creating many different images, corresponding to the many ways in which the reader may make observations. In other words, it is a virtual-reality rendering that is defined by it's own set of rules ([laws of physics](laws%20of%20physics.md)). When this virtual reality rendering is of a physically possible environment, it's [laws of physics](laws%20of%20physics.md) are meant as the best guess of [The Laws of Physics](The%20Laws%20of%20Physics.md) that actually govern reality. Thus, in a broad enough sense, taking into account the processes that must take place inside the scientist’s mind, science and the virtual-reality rendering of physically possible environments are two terms denoting the same activity. ## Rendering Physically Impossible Environments At first glance it would appear that there are two distinct types of virtual reality rendering: a minority that depict physically possible environments, and a majority that depict physically impossible environments. But can this distinction survive closer examination? Consider a virtual reality generator rendering a physically impossible environment, such as a [Flight Simulator](Flight%20Simulator.md) running a program that calculates the view from the cockpit of an aircraft flying faster than light. The flight simulator is *[Rendering](Render.md)* that environment. But, the flight simulator itself is an environment that the user is experiencing, in the sense that it is a physical object surrounding the user. Clearly this is a physically possible environment. Is it a renderable environment? It is, and in fact it is exceptionally easy to render: one simply uses a second flight simulator of the same design, running the same program. Under those circumstances the second flight simulator can be thought of as rendering either the physically impossible aircraft, or a physically possible environment, the first flight simulator. Similarly, the first flight simulator could be regarded as rendering a physically possible environment, namely the second flight simulator. Thus, if we assume that any virtual-reality generator that can in principle be built, can in principle be built again, then it follows that every virtual-reality generator, running any program in its repertoire, is rendering some physically possible environment. It may be rendering other things as well, including physically impossible environments, but in particular there is always some physically possible environment that it is rendering. So, which physically impossible environments can be rendered in virtual reality? Precisely those that are not perceptibly different from physically possible environments. Therefore the connection between the physical world and the worlds that are renderable in virtual reality is far closer than it looks. We think of some virtual-reality renderings as depicting fact, and others as depicting fiction, but the fiction is always an [Interpretation](Interpretation.md) in the mind of the beholder. There is no such thing as a virtual-reality environment that the user would be compelled to interpret as physically impossible. Again, it may seem that we are making rather larges leaps of reasoning here. Let's rebuild the argument that supports the claim: "There is no such thing as a virtual-reality environment that the user would be compelled to interpret as physically impossible". Consider our beloved [Flight Simulator](Flight%20Simulator.md) that is simulating faster than light travel. This is an environment that can be [Simulated](Simulation.md). It consists of a [Program](Program.md) with it's own intrinsic rules ([laws of physics](laws%20of%20physics.md)) that it must follow. The simulation is performed via executing [Computation](Computation.md) as instructed by the program, and computation is a *physical process*[^5]. Computation and simulation are embedded within our universe and therefore must obey [The Laws of Physics](The%20Laws%20of%20Physics.md). Thus, even if the rendering is of a physically impossible process, it must be performed via physically possible computation. We might choose to render an environment as predicted by some [laws of physics](laws%20of%20physics.md) that are different from [The Laws of Physics](The%20Laws%20of%20Physics.md). We may do this as an exercise, or for fun, or as an approximation because the true rendering is too difficult or expensive. If the laws we are using are as close as we can make them to real ones, given the constraints under which we are operating, we may call these renderings ‘applied mathematics’ or ‘computing’. If the rendered objects are very different from physically possible ones, we may call the rendering ‘pure mathematics’. If a physically impossible environment is rendered for fun, we call it a ‘video game’ or ‘computer art’. All these are *interpretations*. They may be useful [Interpretations](Interpretation.md), or even essential in explaining our motives in composing a particular rendering. But as far as the rendering itself goes there is always an alternative interpretation, namely that it accurately depicts some physically possible environment. It is not customary to think of mathematics as being a form of virtual reality. We usually think of mathematics as being about abstract entities, such as numbers and sets, which do not affect the senses; and it might therefore seem that there can be no question of artificially rendering their effect on us. However, although mathematical entities do not affect the senses, the experience of doing mathematics is an external experience, no less than the experience of doing physics is. We make marks on pieces of paper and look at them, or we imagine looking at such marks — indeed, we cannot do mathematics without imagining abstract mathematical entities. But this means imagining an environment whose [laws of physics](laws%20of%20physics.md) embodies the complex and autonomous properties of those entities. For example, when we imagine the abstract concept of a line segment which has no thickness, we may imagine a line that is visible but imperceptibly wide. That much may, just about, be arranged in physical reality. But mathematically the line must continue to have no thickness when we view it under arbitrarily powerful magnification. That is not a property of any physical line, but it can easily be achieved in the virtual reality of our imagination. Imagination is a straightforward form of virtual reality. What may not be so obvious is that our ‘direct’ experience of the world through our senses is virtual reality too. For our external experience is never direct; nor do we even experience the signals in our nerves directly — we would not know what to make of the streams of electrical crackles that they carry. What we experience directly is a virtual-reality rendering, conveniently generated for us by our unconscious minds from sensory data plus complex inborn and acquired theories (i.e. programs) about how to interpret them. [All Observations are Theory Laden](All%20Observations%20are%20Theory%20Laden.md). We realists take the view that reality is out there: objective, physical and independent of what we believe about it. But we never experience that reality directly. *Every last scrap of our external experience is of virtual reality*. And every last scrap of our knowledge — including our knowledge of the non-physical worlds of logic, mathematics and philosophy, and of imagination, fiction, art and fantasy — is encoded in the form of programs for the rendering of those worlds on our brain’s own virtual-reality generator. So it is not just science — reasoning about the physical world — that involves virtual reality. All reasoning, all thinking and all external experience are forms of virtual reality. These things are physical processes which so far have been observed in only one place in the universe, namely the vicinity of the planet Earth. --- Date: 20241211 Links to: [5 - Virtual Reality](5%20-%20Virtual%20Reality.md) Tags: References: * []() [^1]: This is based on how David Deutsch defines Virtual Reality in [Fabric of Reality](Fabric%20of%20Reality.md), Chapter 5 (pg 103). I like his definition so I'll use it here. [^2]: I'm not sure if another set of laws of physics would allow for rendering of logically impossible scenarios. This would require more thought and I don't have time to go down that rabbit hole at the moment. But my quick take is that it is possible that alternate laws would allow for logically impossible scenarios to be rendered. For instance, if - Leaving off FOR pg 211 [^3]: Deutsch touches on this more in [Fabric of Reality](Fabric%20of%20Reality.md) chapter 5. But the idea is that as of today, most of VR today is focused on [components 1 and 2 (nerve impulse detectors and image generators)](Virtual%20Reality.md#Necessary%20Components%20of%20Virtual%20Reality). However, the majority of VR is actually about component 3 (the computer) [^4]: A wonderfully fleshed out example of this is the [Autoverse](Autoverse.md), explored in [Permutation City](Permutation%20City.md). [^5]: [Universal Computer](Universal%20Computer.md)