- Tom Jarvis
Using virtual reality in open source investigations
Open source investigation requires a constant willingness to adapt and seek out new methods of understanding content. One of its biggest hurdles is the distance between investigators and events preventing an immersive means of analysing an area. While this has its benefits – trauma prevention recommends limiting some types of immersion – being able to place someone on-scene can yield new insights and speed up techniques such as geolocation.
At the Centre for Information Resilience (CIR), the use of virtual reality (VR) has been tested as a means to increase immersion in analysis methodologies, such as location familiarisation, geolocation, and geoverification.
Virtual reality headsets enable users to engage in both virtual and virtualised environments. One of the key aspects of this is the ability to view Google Earth in VR. Viewing Google Earth with a VR headset has its advantages in a way that is difficult to explain to someone who has never donned a headset.
It’s very hard to represent the immersion one has with a 2D screenshot. Still, this screenshot shows the street where bodies were found during the Bucha massacre which was then investigated by looking around in VR to obtain matches of buildings.
While viewing something on-screen approximates 3D spatial views, viewing in VR fills your whole vision with a true 3D representation. When using Google Earth 3D, your perception changes from someone looking at a photo from above to that of a giant standing between mountains.
When zoomed into the ground level, you see the terrain as a person on the ground would, caveated with the fact that the overlays are as seen from above. But the real power is when investigating in locations with street views or 360° panoramas.
These perspectives provide an additional layer of immersion and exploration which can be especially beneficial to an investigator. For example, with the latest Bucha massacre footage, watching the video can be enhanced with the virtualisation of driving down the street in Google Street View, where turning your head allows you to explore different angles and perspectives of the scene.
Google Earth VR is a very easy-to-use interface designed for casual exploration, but with major benefits for investigation.
Likewise, with 360° panorama photos in areas without Street View, an investigator could be placed in a market or field of interest and examine a very similar scene to that of an incident.
Perhaps the most useful feature is the potential for horizon analysis. Horizon analysis is an essential technique for geolocation and verification of imagery. By flying over a VR representation of the planet in 3D, horizon matches can be obtained much faster and allow much better spatial perspectives than what you could obtain with Google Earth on a screen.
The following image is from a graveyard in Myanmar where horizon analysis was crucial for identifying the location in order to establish a verification of claims. Marked on the image are the horizon features needing to be identified.
An image needing geolocated with very limited info. This was a graveyard in Myanmar with the only useful information being that it was in a graveyard and the distinctive horizon.
An investigation of all graveyards in the area allowed for near-ground-level vantage points to look around and identify the key horizon features for a match. Seen below is the match found rapidly by using Google Earth VR.
The four notable hills seen in the background of the image are easy to spot in 3D compared to on a screen.
Fortunately this area also offered a 360° panorama taken from the ground. These are especially useful in areas such as Myanmar where Street View is unavailable. This is normally accessible via the globe over one of your “hands”.
Entering this is groundbreaking as it allows you to stand in the graveyard and view it as a person, rather than as a satellite overlaying on a 3D model. While it doesn’t show the exact location of interest within the graveyard, it confirms that the location is indeed a graveyard, something that was very difficult to prove with just satellite imagery.
Another opportunity with 360° cameras arises when you consider their growing sales. As shown above, even in a relatively poor, conflict-ridden part of the world, we were able to get lucky and have full VR-ready viewpoints of a critical location because someone took a 360° photograph.
These may present greater perspectives in conflicts and protests offering a broader depiction of events. While standard video analysis is limited by the choice of camera direction, often missing key details, the adoption of 360° cameras could allow for a much better panorama of a situation.
Consider the challenges of verifying footage of protests in Myanmar. There have been plenty of cases where an investigation into a protestor being shot has been limited by them being hit just off-screen. Likewise, unless the victim was between the shooter and the camera operator, having both in view is rare, meaning attribution of the shooting is extremely difficult to verify. What may be deemed relevant to show in a moment of adrenaline is useful to an investigator, however, smartphones generally have a field of view under 90°. This limitation results in more than three-quarters of horizontal-plane evidence being omitted at any given moment.
With 360° images and footage, this may be an issue that is all but eliminated. In order to keep up with – and perhaps spur on – this emerging opportunity, the adoption of VR is a consideration that shouldn’t be ignored.