Week 8: Final Reflection

Here is the final reflection of my project. In the end, I am happy to say that I had a working final build of my virtual reality classroom to present to people on Thursday. Finishing up from last week, any type of object that exists in the room is now able to be generated back into the room with the press of a button. All of the major features that I had planned for the VR classroom have been successfully implemented.

Looking back, I remember at the beginning I was not sure I would be doing for my project. I always wanted to try doing something in VR simply because I remembered how amazed I was when I tried out the technology the first time in my life. The idea to build a classroom came to me when I remembered how much I disliked using Zoom when I had to go home during Covid-19 pandemic.

After all I had accomplished over these eight weeks, I certainly did learn a lot about different systems in game development on the Unity Engine. I especially remember back in the second week how long and complicated it was to implement the multiplayer system on my project. Originally I did not know how multiplayer was usually implemented in an application. I was surprised to see that I had to program each individual in-game object separately to be synchronized so each player would see the same thing happen with each object. Luckily, online courses and tutorials were there to help me when I was not sure how to implement some feature. I am extremely thankful for these tutorials and I would not have been able to complete my project without them. I strongly recommend to any aspiring programmers out there to always seek out help if they ever run into an obstacle that is tough to overcome.

If I had more time to work on this project, the next new features I would implement would be a user permissions system, so that there could be one user who acts as a designated teacher. So if a teacher didn’t want students to access the classroom video player, they could set the video player so that only the teacher could use it. I would also implement the ability for users to change the appearance of their in-game avatar/character model so it would be easier to identify different users in the room. Even after that, I think there is so much potential for even more features to be built into this project.

Overall I am happy with how the final build of my project turned out. Multiple users are able to work and play with each other in a virtual reality classroom environment. There are whiteboards that can be drawn on, ways to show images and videos, and a phys ed room that allows for users’ creativity and imagination to have no limits in what physically active games they wish to play. Therefore my main goal for this project has been fulfilled. I am happy that my presentation went well and my project was well received at the end. I even brought VR headsets with the application loaded on it so people from the audience had a chance to try out the classroom themselves. It looked like they were having fun testing it out.

I am happy to have been a part of the Digital Technology Summer Fellowship program. It is amazing to think that this happened just because I looked at a poster on a wall last semester. I would like to thank Eric Remy, Joshua Wagner, and my fellow DTSF peers for this wonderful experience. And thank you for reading about my DTSF project.

Week 7: Phys Ed Room and Object Deletion

I spent most of this week working on the Phys Ed room that I started near the end of last week. I wanted to create a Phys Ed room because one of my main reasons for creating this project was to allow opportunities for physical activity in a remote class medium; and a Phys Ed room would help encourage that physical activity.

This new room exists in the same 3D space as the classroom itself (the same scene in the Unity editor). To enter this room, there is a button on a wall in the classroom; and pressing this button will teleport all users to the Phys Ed room. Users return to the classroom the same way. inside the room there are goals used for sports like soccer or hockey that can be toggled on or off, basketball hoops, and two balls: a big one and small one that can be thrown more easily (the velocity it inherits from the player’s hand movements upon letting go is multiplied). It is also possible for players to spawn sticks with with which to hit the balls with like in baseball or hockey. The room also has a functioning scoreboard that automatically updates the score whenever a ball enters a goal or basket, but it is still possible for players to edit the score manually.

Players are also able to draw on the floor of the room like they do on the whiteboards. I implemented this function because I knew that fields and courts for different sports had lines and markings designating certain areas and boundaries on them. I thought making the floor act as a giant whiteboard was the simplest way to allow players to create these different sports field without putting any limit on their creativity or imagination. I wanted users to be able to use their imagination and play any game they could up with in this room.

After creating the Phys Ed room I decided to make a way for users to delete grabbable objects in the rooms so that the area would not stay cluttered and messy if users spawned in too many object. Originally my plan was to program so that players would press a button in their personal menu to delete whatever object they were currently holding in their hand. But when I tested the script I wrote, the object would not be deleted when the button was pressed while the player was holding it. Instead the object would be deleted after it was let go and the player pressed the button. But I thought this was still a good and easy way to delete objects so I decided to keep it like that. Now any player is able to delete whatever object the grabbed last, regardless of who spawned it in or if it existed in the room to start with.

Now that players are able to delete objects they currently cannot spawn themselves (balls in the phys ed room, individual markers not spawned with the marker holder, etc.), I will need to create a way for these objects to be spawned back in for next week. After that I will clean up some unused assets within the project files. And then I will be working on preparing for the presentation of this project.

Week 6: Fixed Synchronization, Slide Presentation, Beginning of Phys Ed. Area

As I mentioned last week, there were some serious bugs with the synchronization of the movable whiteboard when I was testing multiplayer with the Oculus Quest; so I started this week with fixing those bugs. When I originally created the movable whiteboard, I made the pad spawn it own whiteboard separate from the player’s spawning the pad since the whiteboard could not have nor inherit and rigidbody component (rigidbody is what gives objects physics in Unity). The problem with this approach is that the script which did this would spawn multiple whiteboards depending on player count instead of just one, and deleting the extra whiteboards caused the local code for other players to have a null reference error when updating the whiteboard’s position. Now that I have more knowledge and experience with the methods PUN 2 offers for syncing objects across different players, I decided that the best way to fix this issue was to rebuild the drawing pad prefab with a reorganized object hierarchy so that the whiteboard could be part of the object but still separated from the rigidbody component. By doing this I was also able to simplify and optimize the code to perform better. Now the is in perfect synchronization between all players across different devices.

Originally the pad did not have an attached whiteboard object but would instead spawn it separately and have it follow its position.
The new build of the drawing pad now does have an attached whiteboard, but it does not inherit the physics of the pad.

After fixing the drawing pad, I tested the synchronization of the marker holder with the Oculus device. It was also have some issues with the markers snapping back in place for some players.

While testing the marker holders, marker would snap in place properly for one player but other players would see the markers floating around like this.

I fixed this issue similar to how I fixed the whiteboard. The marker holder originally spawned its own markers that were not actually part of the holder and this caused issues similar to what the whiteboard had. Now the markers are part of the holder object prefab and the markers’ positions are now observed by the holder’s photonview (multiplayer component responsible synchronizing properties of an object it’s attached to for all players) instead of each marker having their own; this optimizes the project to run better.

The original structure of the marker holder. It did not have attached marker object but instead spawned them in separately
The new, optimized marker holder prefab

After fixing these synchronization issues. I decided that the next feature I would add would be a way for users to give PowerPoint presentations in the virtual classroom. But there is no simple way to download a PowerPoint file onto the project to display; so instead I made a way to download multiple images at once and cycle through them on a display like a slideshow. This way someone could save each slide as an image online and download these images to give a slide a slide presentation on the display.

This slide presentation tool works almost exactly as the image display tool does. The main difference is that instead of giving a URL to a single image online, the user must give a URL to a .txt file. This .txt file is expected to contain a list of URLs to different images. The script will read through this list of URLs and save each image until its own list of images to display. It will then display the first image on the screen. The user can cycle to the next or previous image using the buttons on the sides of the screen as seen above. This feature will display the same images for all players.

After creating the slide presentation tool I started making a new area where users could do more physically active activities like a physical education class. I wanted to create a phys. ed. area since one of the main reasons for my project was so students would not have to attend school sitting in a chair in front of their computer all day. I have just begun to create the gym area next to the classroom and experiment with different physics settings to make balls for sports that would act similar to their real-life counterparts. I shall continue to work on this phys. ed. room for next week.

The start of the gym room with a sphere object that bounces like a ball.

Week 5: Movable Image, Video, Oculus Build

I started off this week by creating a movable image display similar to that of the drawing pad. Since I already had scripts written from both the drawing pad and classroom video player, implementing this tool was easy. The movable image display works just like the classroom image display I made last week, and can be spawned by any player. Both types of image displays now also have a button to hide their attached keyboards.

Next I decided to make a video player. The video player looks similar to the classroom image display and works almost the same way. To play a video, a player must first type a URL that links directly to a mp4 or webm file on the internet into the text field like they do for the image display, then they hit the button that says “load video”, after the video is loaded, the user then presses the play button next to the screen. The player can pause, skip forward, or rewind the video whenever they want. Unfortunately, it is impossible to play videos from YouTube with this player; but that is YouTube’s fault: They do not like letting videos on their platform being played outside of their platform.

Next I focused on trying to get my project to run on the Oculus Quest. What makes Oculus Quest different from most other VR headsets is that it is a standalone headset that runs as an Android device (Think of it like an Android smartphone built to be a VR headset). Standalone means that the headset was built to run software without being connected to a computer. At first it was ridiculously complicated to even get the built project file onto the device. First I had to put the device into developer mode, which can only be done by logging into a smartphone app. Then I had to download some program called SideQuest which is used to properly transfer the app onto the headset. I was eventually able to get a working build of my project onto the Oculus Quest. It is even able to connect and interact with users who use a normal PC VR setup.

Unfortunately, there are some issues with the Oculus build of the project. First, the video player does not work on Oculus. I did some research to find out that Unity’s built-in video player asset cannot play videos on surfaces in a 3D space on Android devices, that only works on PC. After searching online for a solution, I seems that it is impossible to get a video playing on a surface that is synchronized for all types of connected devices, so I was unable to fix this issue. The issue involved the synchronization of the drawing pad’s whiteboard when I tried using it with another user in the room. When I spawned in the drawing pad and moved it, I saw the whiteboard snap to position as it should. But when the other user tried to move the drawing pad, the whiteboard did not move and an error showed up in the console. For next week I will devote as much time as needed to fix this issue. Then I will test the other interactable tools in the conditions to see if they encounter similar issues.

Week 4: Image Display

At the start of this week I was able to get the marker holder to behave properly for all players. Now when a player grabs a marker from the holder, other players will properly see the marker in the players hand where it should be. And now when a player lets go of a marker they took from the holder, the marker will snap right back to its place in the holder and not move from it. This prevents players from losing markers or having to bend down and and pick them back up when they drop them.

After fixing the marker holder I decided to add some instruction text to the classroom to teach users how to interact in VR and how to use the tools currently available in the classroom. It is important to provide instructions on how to use systems in your project because new users are never going to have the kind of experience that you, the creator, have with your creation.

Signs in front of the players’ spawn position teaching them how to move around, grab objects, and open their spawn menu.
This sign teaches users how to draw on the whiteboard.

After creating the instructions, I was starting to run out of ideas on what features to add next. At first I tried to make a straight edge tool to help users draw straight lines on the whiteboard. but due to sloppy physics and collisions are in the Unity engine when a player is holding an object, it was impossible for me to make a realistic ruler object. For now the straight edge idea has been put on hold.

I then got a new, more interesting idea from Mr. Remy when he talked about how much he used images from the internet when teaching classes. I decided that the classroom could use a way to display images from the internet using image URLs. So I started with a big display setup in one area of the classroom.

The beginning of the classroom image display

The idea was that a user would be able to take a known URL directly for an image on the internet, type it into the text field below the image screen, and then hit the “load image” button next to it to display the image on the wall for everyone to see. I then looked up how to script the project to download images given an image URL. I also bought a new UI keyboard asset from the Unity asset store that was simpler to implement than VRkeys. After some work, I eventually got a working UI keyboard in VR that could be used to type text into a text field.

The UI keyboard. You type on it by pointing at the keys and selecting them with the trigger button on the controllers. The text in the field is synchronized for all players.

I then scripted the display so that it takes whatever URL is in the text field, and uses it to download the image for that URL, and set it to be shown as the texture raw image acting as the “screen” for the display. It also took some time for me to synchronize it for all players, so everyone can see the image when it is loaded.

The display showing a world map image taken directly from the internet.

For next week I plan to make a movable miniature image display similar to the drawing pad for players to spawn and display their own images with. And then I hope to add a video player that works the same way as the image display.

Week 3: Voice Chat, Room Redesign, and New Features

I started off the third week by adding a voice chat system to my project. Photon, the same system that provided the PUN 2 multiplayer package and server, also provides a voice chat service called Photon Voice 2. Following a tutorial, all I had to do was drag and drop a few scripts and game components, and now players are now able to hear each others’ voices when they’re in the classroom together. The players even had tags above their avatars that show their usernames and when they are speaking.

UI text above the avatars’ heads will show whatever username is entered in the login scene.

After implementing voice chat I decided to make a new classroom model. the old SketchUp model was too big and felt dark and empty, so I made a new classroom that was smaller, more colorful, and had a brighter atmosphere overall.

The new current classroom environment.

After redesigning the classroom I wanted to create a desk with a whiteboard surface that could be written on and also physically moved around by the player. I thought this feature would be great for student participation and collaboration. The first issue I faced when making this desk was that a whiteboard would not work when it was part of a game object that had rigid body physics as a component. My solution was to make the desk such that the whiteboard was not part of it but would simple positioned wherever the surface of the desk was. With this solution I made it so that the desk moves around when the player grabs it, but then it freezes in place when the player lets go. This allows the user to position the desk however they want, no matter the height or rotation. This makes the desk more of a movable drawing pad than an actual desk, which I think is more convenient. After the player lets go of the desk, the position of the whiteboard is immediately updated to match the position and rotation of the desk surface, but the whiteboard will not move until the player lets go.

The first desk model in green to the left, to the right: the whiteboard that would go on top of the desks surface.
A whiteboard desk is frozen in place when a player is not grabbing it, so it can be positioned at any height or angle.
Two players in a room with a whiteboard desk positioned above, position and content on the whiteboard are synchronized for both players, so they see the same thing.

After creating the desk, I wanted players to be able to spawn this desk into the world themselves. I created UI menu that the players opens by pressing a button on one of the controllers (the specific button may be different for different VR headsets). This UI menu is local to the players is always positioned in front of their view so they always see it. On this menu is a button that the player points at with their left-hand remote and clicks on with the trigger. Upon clicking this button, a new desk will spawn in front of the player. Any player can now spawn as many desks as they want

Two desks spawned from the UI menu, these desks are synchronized for all players, no matter who spawns them.

Next I moved on to creating a holder to hold different colored markers to draw with. I wanted to create this so that players could easily carry multiple markers with them at once. Like the desks, this holder can be grabbed by the player and will freeze in place when the player lets go. I used a component in OpenXR called the XR Socket Interactor which allows the markers to snap into place when a player puts them onto the holder. The markers move with the holder when a player grabs it and moves it around. Players can also remove markers from the holder.

A single holder holds up to six markers. This one shows all the colors currently available.
You can draw on a whiteboard while the markers are still attached to the holder.

These marker holders are also able to be spawned from the same menu as the desks, so players can spawn their own markers.

I tried to synchronize these marker holders for all players. But currently there is an issue with the sockets on the holders. When a player tries to remove a marker from a socket, nothing wrong happens on that player’s end, but the other players do not see the marker get removed properly and instead see it shake around near the socket, so there must be a mismatch with the synchronization of the state of the markers in relation to the sockets.

What I see when I remove a marker from a socket and drop it on the ground…

…But this is what the other players see.

For nest week I hope to fix this bug and then move on to creating a way for players to type readable text that can be displayed to all players using VRKeys. And then I might create a way for users to draw perfectly straight lines on a whiteboard.

Week 2: Multiplayer Online

I dedicated this whole week to adding functioning multiplayer to my VR classroom. I wanted to make it possible for multiple users to connect to the same room and interact with each other and the classroom in near-perfect synchronization. For multiplayer, I used a system called Photon Unity Networking 2 (PUN 2). This was a free package that gave me a server on which to host the project and allow multiple players to connect to it. I carefully followed an online course, which showed that a large amount of work and setup goes into setting up a working online multiplayer. First I had to set up the project so that player could connect to the master server. I set up a scene in unity where the user could connect to the server anonymously or with a username that they type in with a virtual keyboard UI called VRKeys, an open-source unity asset where you tap on the keys like a drum set. After some coding, I was successfully able to connect to the master server as shown by the debug console message “Connected to Master Server with player name: Casey”. Seeing this message was very satisfying for me.

Confirmation of connection to the server with a chosen username.

Simply connecting to the server is not enough to get multiple user playing with each other in the same space. In PUN 2’s structure, after connecting to the server, a room must be created and players must be able to join that room. The room is the actual space where players interact together. It is possible to create multiple different rooms for players to joins; but in my case I only need one room to exist at a time. I set up the classroom scene to be a room that is created by the first player who tries to join the classroom, and the players who try to join after join that same classroom. I then had to program the room to generate a new character for each player who joins the room.

Two users in a room, each with their own avatar.

After setting up the room I had to make the room synchronize the movement of the users’ avatars and the other objects present in the room. I began with synchronize users’ avatar movement. Fortunately, the PUN 2 package comes with an asset called Photon View which can be attached to a game object to listen for and automatically listen for and synchronize changes in an objects properties so that all players would see the same changes. I just had to set up which components Photon View checked. I set it up so that when a player moves, all other players in the room will see that player’s avatar move. After synchronizing player movement, I then moved on to synchronizing objects that can be grabbed by players and are affected by physics. For testing I created two cubes in the room that players could pick up and throw. Synchronizing these cubes was more complicated than I expected due to all the different properties that go in to an object being affected physics and held by players. For example, when a player holds an object, the effect of gravity is turned off for that object. The room on the server’s end needs to know that gravity state for the object so it can make that state match for all the other players; or else the other players will see the object violently shaking up and down near the player’s hand instead of the player holding the object. This synchronization took a lot of complicated C# script programming to be done correctly. But now when a player holds one of the cubes, the other players can properly see that player holding the cube, and they can even see it fly through the air when the player throws that cube. For next week I plan to synchronize the whiteboard, implement a voice chat system, and make a built-in instruction manual to teach users how to use the objects and systems in the classroom.

Looking at a player from another players perspective.

Week 1: Introduction and Beginning

Hello. I am Casey Rhodes. My project for this fellowship is a classroom built in virtual reality that multiple people can attend and interact with remotely such that someone would be able to teach a class in it. On the first day we discussed the process of designing and planning our projects. My minimum goal for a viable product was to have a basic VR room that multiple users could connect to. The users would be able to freely move around in this room, and they would be able to draw and erase on the walls like they would on a whiteboard. I believe this function would allow for active student participation and collaboration in class. There will also be a tutorial that teaches the users how to move and interact with the room as described above. After fulfilling this minimum goal, I hope to add desks, tables, and other classroom objects that can be written on and moved around in the classroom. I also want to give users to create and remove these objects from the room at will. Then I hope give each user a surface on a graphical user interface on which to write like it’s their own notebook. I would like users to be able to save what is on their notebook as well as the set up state of the classroom as file that can be loaded. These are my main stretch goals but I am certain there is potential for more features to be added project later on.

On Wednesday I began my work on the classroom. I started by making a simple model of a room using SketchUp that I would eventually import into Unity game environment.

Outside view of the classroom after being imported into Unity

After building the model I began learning how to use Unity. I had little experience with Unity so I started simple with a tutorial on how to set up and program a basic 3D game environment so I could understand the basic tools and systems of the Unity workspace. After learning how to build and program a 3D space, I started to view tutorials on how to set up VR on Unity. Video tutorials were extremely helpful resources in teaching how to do things I formerly had little to no experience with prior to this project. I was surprised to learn that Unity had built in features that allowed for a VR environment to be set up easily by simply installing certain packages and setting up a few assets. After a few technical obstacles with the headsets’ own software I was able to enter and move around in my classroom in virtual reality.

The HTC Vive I used to test and interact with my classroom. I used a system called OpenXR that allows for compatibility with both Steam and Oculus devices.
Inside the classroom in virtual reality

For my next task I wanted to create some way of drawing on a surface in the room. I found a tutorial on how to make a functioning whiteboard in VR and decided to try that out. It took a lot of complex coding but the tutorial allowed me to create the whiteboard and markers with little to no trouble. Now it is possible for a user to pick up a marker with one of their controllers and draw on the white board similar to real life.

Whiteboard with blue and red markers and an eraser.
I have recently been trying to get another drawing system working that would allow users to draw on any surface instead of a designated whiteboard; but for next week I plan to focus on implementing multiplayer functionality so that multiple separate users can interact within the room simultaneously.