Table of Contents

UnrealCV

Join the chat at https://gitter.im/unrealcv/unrealcv Docs Status

UnrealCV is a project to help computer vision researchers build virtual worlds using Unreal Engine 4 (UE4). It extends UE4 with a plugin by providing:

  1. A set of UnrealCV commands to interact with the virtual world.
  2. Communication between UE4 and an external program, such as Caffe.

UnrealCV can be used in two ways. The first one is using a compiled game binary with UnrealCV embedded. This is as simple as running a game, no knowledge of Unreal Engine is required. The second is installing UnrealCV plugin to Unreal Engine 4 (UE4) and use the editor of UE4 to build a new virtual world.

Please read Tutorial: Getting Started to learn using UnrealCV.

annotation Images generated from the technical demo RealisticRendering

Citation

If you found this project useful, please consider citing our paper

@article{qiu2017unrealcv,
  Author = {Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao,Zihao Xiao, Tae Soo Kim, Yizhou Wang, Alan Yuille},
  Journal = {ACM Multimedia Open Source Software Competition},
  Title = {UnrealCV: Virtual Worlds for Computer Vision},
  Year = {2017}
}

Contact

If you have any suggestion or interested in using UnrealCV, please contact us.

Getting started

This page introduces UnrealCV commands and how to use them to perform basic tasks. We also show how to use a python script to control an UnrealCV embedded game through these commands.

Download a game binary

This tutorial will use a game binary to demonstrate UnrealCV commands. You can also create your own game using UnrealCV plugin.

First you need to download a game binary from our model zoo. For this tutorial, please download RealisticRendering. After unzip and run the binary, you are expected to see a screen like this. The game will be started in a window mode with resolution 640x480, you can change the resolution by changing the configuration file of UnrealCV.

_images/rr_init.png

Initial screen of the game

Use mouse to look around and use keys w a s d to navigate, use q e to level the camera up and down. If you want to release mouse cursor from the game, press ` (the key on top of tab).

UnrealCV commands

UnrealCV provides a set of commands for computer vision research. These commands are used to perform various tasks, such as control camera and get ground truth. The table below summaries commands used in this tutorial. The complete list can be found in the command list in the reference section.

Command Help
vset /viewmode [viewmode_name] Set ViewMode to (lit, normal, depth, object_mask)
vget /camera/0/lit Save image to disk and return filename
vset /camera/0/location [x] [y] [z] Set camera location

Try UnrealCV commands

Unreal Engine provides a built-in console to help developers to debug games. This built-in console is a convenient way of trying UnrealCV commands. To open the console, press ` (the key on top of tab) twice, a console will pop out, as shown in Fig. 2. Type in vset /viewmode object_mask you are expected to see the object instance mask. Use vset /viewmode lit to switch back to normal rendering setting.

_images/console.png

Use console to try UnrealCV commands

Use python client to execute commands

If we want to generate a large-scale synthetic dataset, or do active tasks, such as reinforcement learning, in this virtual world. We need to allow an intelligent agent to perceive, navigate and interact in the scene. We provide UnrealCV client to enable other programs to communicate with this virtual world. The client will use a plain-text protocol to exchange information with the game.

Here we use the python client for illustration. If you are looking for a MATLAB client, please see the MATLAB client.

First, we need to install the python client library.

Install UnrealCV python library

pip install unrealcv

Generate some images from the scene

from unrealcv import client
client.connect() # Connect to the game
if not client.isconnected(): # Check if the connection is successfully established
  print 'UnrealCV server is not running. Run the game from http://unrealcv.github.io first.'
else:
  filename = client.request('vget /camera/0/lit')
  filename = client.request('vget /camera/0/depth depth.exr')

You can find this example in examples/10lines.py.

If you encountered any errors following this tutorial, please see the diagnosis page to find a solution.

Next: Use UnrealCV in the game mode or plugin mode?

For the game mode, you can use a compiled game binary. You can freely control the camera in this game and generate images and ground truth from it. But it is not easy to change the scene, such as add more objects or change the material properties. If you have access to an UE4 project and know how to use the UE4Editor, you can install the plugin to UE4Editor, so that you can combine the power of UE4Editor and UnrealCV to create new virtual worlds for research.

Articles

  • To fully understand how does UnrealCV work and the technical details, please read its architecture or our paper. For a complete list of available commands, please read the command list in the reference section.
In [1]:
%matplotlib inline

Generate Images

This ipython notebook demonstrates how to generate an image dataset with rich ground truth from a virtual environment.

In [2]:
import time; print(time.strftime("The last update of this file: %Y-%m-%d %H:%M:%S", time.gmtime()))
The last update of this file: 2017-10-21 21:26:41

Load some python libraries The dependencies for this tutorials are PIL, Numpy, Matplotlib

In [3]:
from __future__ import division, absolute_import, print_function
import os, sys, time, re, json
import numpy as np
import matplotlib.pyplot as plt

imread = plt.imread
def imread8(im_file):
    ''' Read image as a 8-bit numpy array '''
    im = np.asarray(Image.open(im_file))
    return im

def read_png(res):
    import StringIO, PIL.Image
    img = PIL.Image.open(StringIO.StringIO(res))
    return np.asarray(img)

def read_npy(res):
    import StringIO
    return np.load(StringIO.StringIO(res))

Connect to the game

Load unrealcv python client, do pip install unrealcv first.

In [4]:
from unrealcv import client
client.connect()
if not client.isconnected():
    print('UnrealCV server is not running. Run the game downloaded from http://unrealcv.github.io first.')
    sys.exit(-1)
INFO:__init__:211:Got connection confirm: 'connected to RealisticRendering'

Make sure the connection works well

In [5]:
res = client.request('vget /unrealcv/status')
# The image resolution and port is configured in the config file.
print(res)
Is Listening
Client Connected
9000
Configuration
Config file: /Users/qiuwch/unrealcv/UE4Binaries/RealisticRendering/MacNoEditor/RealisticRendering.app/Contents/UE4/Engine/Binaries/Mac/unrealcv.ini
Port: 9000
Width: 640
Height: 480
FOV: 90.000000
EnableInput: true
EnableRightEye: false

Load a camera trajectory

In [7]:
traj_file = './camera_traj.json' # Relative to this python script
import json; camera_trajectory = json.load(open(traj_file))
# We will show how to record a camera trajectory in another tutorial

Render an image

In [8]:
idx = 1
loc, rot = camera_trajectory[idx]
# Set position of the first camera
client.request('vset /camera/0/location {x} {y} {z}'.format(**loc))
client.request('vset /camera/0/rotation {pitch} {yaw} {roll}'.format(**rot))

# Get image
res = client.request('vget /camera/0/lit lit.png')
print('The image is saved to %s' % res)

# It is also possible to get the png directly without saving to a file
res = client.request('vget /camera/0/lit png')
im = read_png(res)
print(im.shape)

# Visualize the image we just captured
plt.imshow(im)
The image is saved to /Users/qiuwch/unrealcv/UE4Binaries/RealisticRendering/MacNoEditor/RealisticRendering.app/Contents/UE4/Engine/Binaries/Mac/lit.png
(480, 640, 4)
Out[8]:
<matplotlib.image.AxesImage at 0x108dd8310>
_images/tutorials_generate_images_tutorial_12_2.png

Ground truth generation

Generate ground truth from this virtual scene

In [9]:
res = client.request('vget /camera/0/object_mask png')
object_mask = read_png(res)
res = client.request('vget /camera/0/normal png')
normal = read_png(res)

# Visualize the captured ground truth
plt.imshow(object_mask)
plt.figure()
plt.imshow(normal)
Out[9]:
<matplotlib.image.AxesImage at 0x1097db8d0>
_images/tutorials_generate_images_tutorial_14_1.png
_images/tutorials_generate_images_tutorial_14_2.png

Depth is retrieved as a numpy array For UnrealCV < v0.3.8, the depth is saved as an exr file, but this has two issues. 1. Exr is not well supported in Linux 2. It depends on OpenCV to read exr file, which is hard to install

In [10]:
res = client.request('vget /camera/0/depth npy')
depth = read_npy(res)
plt.imshow(depth)
Out[10]:
<matplotlib.image.AxesImage at 0x10905d090>
_images/tutorials_generate_images_tutorial_16_1.png

Get object information

List all the objects of this virtual scene

In [11]:
scene_objects = client.request('vget /objects').split(' ')
print('Number of objects in this scene:', len(scene_objects))

# TODO: replace this with a better implementation
class Color(object):
    ''' A utility class to parse color value '''
    regexp = re.compile('\(R=(.*),G=(.*),B=(.*),A=(.*)\)')
    def __init__(self, color_str):
        self.color_str = color_str
        match = self.regexp.match(color_str)
        (self.R, self.G, self.B, self.A) = [int(match.group(i)) for i in range(1,5)]

    def __repr__(self):
        return self.color_str

id2color = {} # Map from object id to the labeling color
for obj_id in scene_objects:
    color = Color(client.request('vget /object/%s/color' % obj_id))
    id2color[obj_id] = color
    # print('%s : %s' % (obj_id, str(color)))
Number of objects in this scene: 296

Parse the segmentation mask

In [12]:
def match_color(object_mask, target_color, tolerance=3):
    match_region = np.ones(object_mask.shape[0:2], dtype=bool)
    for c in range(3): # r,g,b
        min_val = target_color[c] - tolerance
        max_val = target_color[c] + tolerance
        channel_region = (object_mask[:,:,c] >= min_val) & (object_mask[:,:,c] <= max_val)
        match_region &= channel_region

    if match_region.sum() != 0:
        return match_region
    else:
        return None

id2mask = {}
for obj_id in scene_objects:
    color = id2color[obj_id]
    mask = match_color(object_mask, [color.R, color.G, color.B], tolerance = 3)
    if mask is not None:
        id2mask[obj_id] = mask
# This may take a while
# TODO: Need to find a faster implementation for this

Clean up resources

In [19]:
client.disconnect()

More Examples

More examples can be found in the examples folder of the repo.

Install UnrealCV Plugin

This page briefly describes how to install UnrealCV as a UE4 plugin. Make sure you read getting started before trying to use the plugin.

Use compiled plugin binary

You can download compiled UnrealCV binaries from our github release page. Then copy the compiled binaries to the plugins folder to install it. Build it yourself by following the Compile from source code. You can install the plugin to either a game project or to UE4 engine.

  • Install to project
    • Go to project folder which contains [ProjectName].uproject
    • Create a folder called Plugins
    • Put UnrealCV folder into the Plugins folder.
  • Install to Unreal Engine
    • Go to the plugin folder of Unreal Engine which is Engine/Plugins
    • Put UnrealCV folder into the Plugins folder.

Open Menu -> Edit -> Plugins, make sure UnrealCV is installed and enabled. You have to be in play mode before you type the commands.

_images/plugin.png

Install from UE4 marketplace (coming)

For Windows and Mac user, UnrealCV will be released to the UE4 marketplace. We are still finalizing the submission to the UE4 marketplace and it will be available soon.

Compile from source code

If you want to try a version of UnrealCV not provided in our github release page, for example, you want to try some experimental features not released yet. Compiling the plugin code from source code is the only choice.

To compile UnrealCV plugin, use

pip install -U unrealcv
    # Install the latest version of unrealcv, the build.py depends on unrealcv.automation module
python build.py
# This script will search common Unreal Engine folders of Windows and Mac
# If this script fails to find UE4 installation path, you can also manually specify the engine path
python build.py --UE4 {UE4}
# For example
# python build.py --UE4 "/Users/Shared/Epic Games/UE_4.16"

After running this command you should see Automation.Execute: BUILD SUCCESSFUL and the plugin binaries will be produced in the Plugins/UnrealCV folder. Then you can copy the compiled plugin to Plugins folder.

If you want to modify UnrealCV code and add new features. Please refer to the development setup. Setting up a dev environment takes a bit more time but will make it much easier to debug and modify code.

Note

When using the plugin in the editor, it is strongly recommend to turn off the setting Editor Preference -> General -> Misc. -> Use Less CPU when in Background.

Special tips for Linux

In Linux, the Unreal Engine needs to be built from source code. How to compile from source code can be found in this official document Building On Linux.

Previous versions of unrealcv depends on using OpenEXR module to generate depth, this requirement has been removed.

Basic usage in a new project

ue4: 4.14, unrealcv: v0.3.9, level: basic, updated: 06/27/2017

This tutorial assumes the basic knowledge of using a command line. If you are a windows user and do not have much knowledge of using command line and find it difficult to understand, please let us know.

1. Install the plugin (build from source code in this tutorial)

In this tutorial the project and plugin folders are refered as the console varialbe plugin_folder and project_folder

export plugin_folder=$HOME/workspace/unrealcv

# Clone the UnrealCV repository
git clone https://github.com/unrealcv/unrealcv ${plugin_folder}

# Switch to version `v0.3.10`
cd {plugin_folder}
git checkout v0.3.10

# Build the plugin binary
export UE4="/Users/Shared/Epic Games/UE_4.14/"
python build.py --UE4 ${UE4}

More details about the plugin installation can be found in Install UnrealCV Plugin.

2. Create a new project and copy the plugin to the project folder

Creat a ‘Blueprint - First Person’ project.

http://i.imgur.com/pAbXXXi.png

Refer the project folder with a project_folder variable

export project_folder="$HOME/Documents/Unreal Projects/MyProject"

# Copy the compiled plugin to the project folder to install it.
cp -r "${plugin_folder}"/Plugins/ "${project_folder}"/Plugins/

Restart the UE4 project and make sure the plugin is successfully loaded

3. Open the Unreal project and make sure the plugin is installed

http://i.imgur.com/hAWJHqt.png

4. Try the command in UE4Editor

http://i.imgur.com/AcONETx.png

5. Try the command is the python console

http://i.imgur.com/er986gI.png

Usage in the Editor

In UE4 editor, you can run UnrealCV command, edit the scene and change the material properties. We show a few examples here using the scene RealisticRendering.

Run UnrealCV Command

While playing the level in the editor, press ` to open the built-in console and run UnrealCV like in the game binary.

https://image.ibb.co/jog255/command1.png

Edit Object Specularity

Select an object, e.g. the wooden floor, you want to edit in UE4 editor, and double click the image of Element 0 at Details -> Materials tab to edit its property.

https://image.ibb.co/eQDuTF/editor1.png

Edit SpecularWood property, e.g. increase the value to make it more specular.

https://image.ibb.co/fsTYNa/material1.png

The results are as follows,

https://image.ibb.co/mhoNYF/specular.png

Edit Object Color

Following similar steps, you can edit the color of an object.

https://image.ibb.co/cmW9vv/color1.png

The results are as follows,

https://image.ibb.co/mq1RtF/color2.png

Use UnrealCV commands in the editor

https://preview.ibb.co/mavL8Q/editor_mask.png

You can use UnrealCV commands the same way as in a standalone binary.

Reference

This page is contributed by the UnrealStereo project and the tool and images will be released soon.

The configuration file

Start from UnrealCV v0.3.1, the config of UnrealCV can be configured in a configuration file. Use vget /unrealcv/status to see current configuration.

Change game resolution

The output resolution of UnrealCV is independent of the window size.

If you want to change the display resolution. In game mode, use console command r.setres 320x240

When use play -> selected viewport the resolution can not be changed, use play -> new window editor instead.

Change the server port

Use vget /unrealcv/status to get the directory of the configuration file. Then open the configuration file and modify the server port.

Package a game binary

In some scenarios you will want to package your game projects into a binary, instead of using it in the editor, for example you want to deploy the game to a deep learning server without UE4 installed or share your game project with other researchers

This page briefly describes how to create a game binary with UnrealCV embedded. Some game binaries can be found in the model zoo

Guide to submit a binary

  1. Modify UE4 config file
  2. Package your game project into a binary
  3. Make a pull request to modify the Model Zoo
  4. We will review your pull request and merge the changes to the UnrealCV website

1. Modify an UE4 config file

First, you need to add a line to UE4 engine config file Engine\Config\ConsoleVariables.ini by adding this line to the end.

r.ForceDebugViewModes = 1

If this line is missing, UnrealCV commands of the packaged game will not work correctly.

2. Package your game project

UE4 makes it easy to release your project as a game binary. You can use the editor menu to package a game. Many related blog posts can be found online.

  • Use UE4 Editor to package a game
  • Use script to package a game
python build.py --UE4 {UE4 instal path} {uproject path}

For example, python build.py --UE4 "C:\Program Files\Epic Games\UE_4.16" C:\qiuwch\workspace\uprojects\UE4ArchinteriorsVol2Scene1\ArchinteriorsVol2Scene1.uproject

3. Make a pull request

The last step is making a pull request to modify the model zoo page and add your content. We list some information that should be provided in the pull request. These information can help others better utilize the game binary. This is an example.

Binary name (required):
An easy to remember name that people can refer to the binary you created
Author information (required):
Author name and contact information
Binary description (required):
What this virtual world is designed for? Generating dataset, reinforcement learning, etc.??
UnrealCV version (required):
It can be a release version such as v0.3, a git short sha version, or a pointer to a commit of your fork. This information is to help others find which API is available and the corresponding documentation.
Download link (required):
Please host binaries in your website, if you have difficulties finding a place to host content, we can help you find some free solutions. Please also include which platform (win,mac,linux) your binaries are built for.
Related paper or project page (optional):
If this game binary is related to a research project, make a link to here.

Packaging binaries automatically

In the UnrealCV team, we use a set of packaging scripts to automate the packaging and testing. These scripts are hosted in <qiuwch/unrealcv-packaging>.

The architecture of UnrealCV

_images/pipeline.png

Fig.1: Architecture of UnrealCV

The focus of UnrealCV is to do IPC (Inter Process Communication) between a game and a computer vision algorithm. The communication can be summarized by Fig.1. A game created by Unreal Engine 4 will be extended by loading UnrealCV server as its module. When the game launches, UnrealCV will start a TCP server and wait for commands. Any program can use UnrealCV client code to send plain text UnrealCV command to control the scene or retrieve information. The design of UnrealCV makes it cross-platform and support multiple programming languages. The command system is designed in a modular way and can be easily extended.

Unrealcv consists two parts:

  1. unrealcv server which is embedded into a video game.
  2. unrealcv client uses commands to control the server.

The initialization procedure of UnrealCV. The module loaded and start a TCP server waiting for connection. Commands will be parsed by regexp.

Project Layout

client/            # Client code for Python and MATLAB
    examples/      # Examples showing how to use client code to do tasks
    matlab/        # MATLAB client
    python/        # Python client
    scripts/       # Scripts for tasks, such as building and packaging
Content/           # Plugin data
docs/              # Documentation of UnrealCV
Resources/         # Plugin resource
Source/            # Plugin C++ source code
test/              # Test code
UnrealCV.uplugin   # Descriptor file for an UE4 plugin
README.md

Command System

Unreal Engine 4 has some built-in commands to help game development. These commands can be typed into a built-in console. Using these commands, a developer can profile the game performance and view debug information. To invoke the built-in console of a game, type the ` key (the key above tab).

UnrealCV provides commands useful for computer vision researchers. What is more, these commands can be used by an external program. A built-in command can also be used using the special command vrun.

Command cheatsheet

See this ipython notebook to see an imcomplete demo of available commands.

1. Camera operation

See Source/UnrealCV/Private/Commands/CameraHandler.h(.cpp) for more details.

vget /camera/[id]/location
(v0.2) Get camera location [x, y, z]
vget /camera/[id]/rotation
(v0.2) Get camera rotation [pitch, yaw, roll]
vset /camera/[id]/location [x] [y] [z]
(v0.2) Set camera location [x, y, z]
vset /camera/[id]/rotation [pitch] [yaw] [roll]
(v0.2) Set camera rotation [pitch, yaw, roll]
vget /camera/[id]/[viewmode]
(v0.2) Get [viewmode] from the [id] camera, for example: vget /camera/0/depth
vget /camera/[id]/[viewmode] [filename]

(v0.2) Same as the above, with an extra parameter for filename

filename:Filename is where the file will be stored.
example:vget /camera/0/lit lit.png
vget /camera/[id]/[viewmode] [format]

(v0.3.7) Support binary data format

format:If only file format is specified, the binary data will be returned through socket instead of being saved as a file.
example:vget /camera/0/lit png
vget /camera/[id]/object_mask
(v0.2) The object mask is captured by first switching the viewmode to object_mask mode, then take a screenshot
vset /viewmode [viewmode]
(v0.2) Set ViewMode to (lit, normal, depth, object_mask)
vget /viewmode
(v0.2) Get current ViewMode
vget /camera/[id]/pose
(v0.3.10) Get camera location [x, y, z] and rotation [pitch, yaw, roll]
vset /camera/[id]/pose [x] [y] [z] [pitch] [yaw] [roll]
(v0.3.10) Teleport camera to location [x, y, z] and rotation [pitch, yaw, roll]
vget /camera/[uint]/horizontal_fieldofview
(v0.3.10) Get camera horizontal field of view
vset /camera/[uint]/horizontal_fieldofview [FOV]
(v0.3.10) Set camera horizontal field of view
vget /camera/[uint]/vis_depth npy
(v0.3.10)
vget /camera/[uint]/plane_depth npy
(v0.3.10)

2. Object interaction

See Source/UnrealCV/Private/Commands/ObjectHandler.h(.cpp) for more details

vget /objects
(v0.2) Get the name of all objects
vget /object/[obj_name]/color
(v0.2) Get the labeling color of an object (used in object instance mask)
vget /object/[obj_name]/location
Get object location [x, y, z]
vget /object/[obj_name]/rotation
Get object rotation [pitch, yaw, roll]
vset /object/[obj_name]/location [x] [y] [z]
Set object location [x, y, z]
vset /object/[obj_name]/rotation [pitch] [yaw] [roll]
Set object rotation [pitch, yaw, roll]
vset /object/[obj_name]/color [r] [g] [b]
(v0.2) Set the labeling color of an object
vset /object/[str]/show
(v0.3.10) Show object
vset /object/[str]/hide
(v0.3.10) Hide object

3. Plugin commands

See Source/UnrealCV/Private/Commands/PluginHandler.h(.cpp) for more details.

vget /unrealcv/status
(v0.2) Get the status of UnrealCV plugin
vget /unrealcv/help
(v0.2) List all available commands and their help message

4. Action commands

See Source/UnrealCV/Private/Commands/ActionHandler.h(.cpp)

vset /action/keyboard [key_name] [delta]
(v0.3.6) Valid key_name can be found in here
vset /action/game/pause
(v0.3.10) Pause the game
vset /action/game/level [level_name]
(v0.3.10) Open a new level
vset /action/input/enable
(v0.3.10) Enable input
vset /action/input/disable
(v0.3.10) Disable input
vset /action/eyes_distance [eye_distance]
(v0.3.10) Set the eye distance between left eye and right eye (camera 1). This command might be marked as deprecated when we finish multiple camera support.

Run UE4 built-in commands

vrun [cmd]
(v0.3) This is a special command used to execute Unreal Engine built-in commands. UE4 provides some built-in commands for development and debug. They are not very well documented, but very useful.

A few examples are:

  • stat FPS - show current frame rate
  • shot - take a screenshot
  • show Material - toggle the display of Material

These commands can be executed in the UE4 console. If you want to use them in UnrealCV, you can prefix these commands with vrun stat FPS.

Run Blueprint commands

vexec [cmd]
TODO

Model Zoo

We provide compiled virtual worlds to play with. All the digital contents belong to the original author. If you want to use UnrealCV plugin in the editor, you can find UE4 projects in UE4 Resources

Hint

Model Zoo is still experimental. If you are a beginner, we highly recommend you start with RealisitcRendering to avoid unknown bugs. If you found any bugs, you can report an issue. We will fix them as soon as possible.

Run the downloaded binary

  • In Mac: Run [ProjectName].app
  • In Linux: Run ./[ProjectName]/Binaries/Linux/[ProjectName]
  • In Windows: Run [ProjectName]/[ProjectName].exe

Read Package a game binary, if you are interested in sumbitting a binary to the model zoo.

RealisticRendering

Source:https://docs.unrealengine.com/latest/INT/Resources/Showcases/RealisticRendering/
Preview:
_images/realistic_rendering.png
Description:Realistic Rendering is a demo created by Epic Games to show the rendering power of Unreal Engine 4. Here we provide an extended version of this demo with UnrealCV embedded.
Plugin:0.3.10
Download:Windows, Linux, Mac
  • Docker
`nvidia-docker run --name rr --rm -p 9000:9000 --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" qiuwch/rr:0.3.8`

ArchinteriorsVol2Scene1

Source:https://www.unrealengine.com/marketplace/archinteriors-vol-2-scene-01
Preview:
https://image.ibb.co/jb2QCa/arch1.png
Description:ArchinteriorsVol2Scene1 is a scene of a 2 floors apartment.
Plugin:0.3.10
Download:Windows, Linux, Mac

ArchinteriorsVol2Scene2

Source:https://www.unrealengine.com/marketplace/archinteriors-vol-2-scene-02
Preview:
https://image.ibb.co/hwmkCa/arch2.png
Description:ArchinteriorsVol2Scene2 is a scene of a house with 1 bedroom and 1 bathroom.
Plugin:0.3.10
Download:Windows, Linux, Mac

ArchinteriorsVol2Scene3

Source:https://www.unrealengine.com/marketplace/archinteriors-vol-2-scene-03
Preview:
https://image.ibb.co/nyC3yF/arch3.png
Description:ArchinteriorsVol2Scene3 is a scene of an office.
Plugin:0.3.10
Download:Windows, Linux, Mac,

UrbanCity

Source:https://www.unrealengine.com/marketplace/urban-city
Preview:
https://image.ibb.co/kgrJXa/urbancity.png
Description:UrbanCity is a scene of a block of street.
Plugin:0.3.10
Download:Windows, Linux, Mac

Trouble Shooting

Issues and workarounds

Issues and workaround can be found in issue tracker. Please use the search function before posting your issue, your problem might have already been answered.

If the plugin crashed the editor, please send us your crash log to help us figure out what is going wrong. The crash log can be found in Saved/CrashReport. If you can not find the crash report, you can also send us the core dump file.

Supported Unreal Engine Version

  4.12 4.14 4.16
Windows      
Linux      
Mac      

Client Support

Python support for python 2 and 3 are verified with tox.

Verified projects

Unreal Engine projects are of dramatic different scales and complexity. It can be as simple as just a single room, or be a large city or outdoor scene. UnrealCV is far from perfect and it has compatible issues with some Unreal projects. Here are a few projects we are currently using and have verified that UnrealCV can work well with. If you want us to test a map (project), please let us know.

Here are a list of Unreal projects that we tried and verified.

Known issues and solutions

We tried our best to make the software stable and easy to use, but accidents sometimes happen. Here are a list of issues that you might find. Use the search function ctrl+f to search your error message. If you can not find useful information here, post a new issue. Subscribe to an issue if you want to get future notification.

  • python3 support. See issue #49, Thanks to @jskinn for the idea!

  • The screen resolution is not what I want

    • In editor, change Editor preference -> Level Editor -> Play
    • In the game mode, use console command setres 640x480
    • Change the config file shown in vget /unrealcv/status
  • The speed of UnrealCV

  • The OpenEXR requirement

  • The Unreal Engine config file not changed

  • The image and ground truth not aligned

  • Can not connect to the binary

Use vget /unrealcv/status to make sure the server is listening and no client is connected.

Platform specific issues

Mac

When in mac, the server can not detect the socket disconnection. If the first time connection is successful and the second time is faile. Then please close and re-open it again.

Native error= Cannot find the specified file

https://answers.unrealengine.com/questions/574562/cannot-package-a-plugin-in-415mac.html

Invalid SDK MacOSX.sdk, not found in /Library/Developer/CommandLineTools/Platforms/MacOSX.platform/Developer/SDKs

https://answers.unrealengine.com/questions/316117/missing-project-modules-1.html https://github.com/nodejs/node-gyp/issues/569#issuecomment-255589932

Linux

  • The binary can not run

    For example an error like this.

    [2017.05.25-04.14.33:476][  0]LogLinux:Error: appError called: Assertion failed: Assertion failed:  [File:/UE4/Engine/Source/Runtime/OpenGLDrv/Private/Linux/OpenGLLinux.cpp] [Line: 842]
    
    Unable to dynamically load libGL: Could not retrieve EGL extension function eglQueryDevicesEXT
    

    It is very likely an issue with the OpenGL of the system.

    sudo apt-get install mesa-utils and run glxgears. Make sure you can see a window with gears running in it.

CHANGELOG

Development branch

  • v0.3.10
    • Commands contributed in pull request #91, authored by @bennihepp
      • Add vget /camera/[id]/pose, vset /camera/[id]/pose
      • Add vget/vset /camera/[id]/horizontal_fieldofview
      • Add vget /camera/[id]/vis_depth npy and vget /camera/[id]/plane_depth npy
      • Add vset /object/[id]/show, vset /object/[id]/hide
      • Add vset /action/input/enable, vset /action/input/disable
    • Add more commands
      • Add vget /object/[id]/mobility, vget /object/[id]/location, vget /object/[id]/rotation
      • Add vget /camera/[id]/normal npy
      • Add vset /action/eyes_distance [eye_distance]
      • Add vset /action/game/pause
    • Update the python client to support python3
    • Improve documentation
  • v0.3.9
    • Fix a bug that prevents object mask generation, which is introduced in v0.3.7
    • Fix #53 that the painting of object does not work
    • Fix #49 python3 support, thanks to @jskinn and @befelix
  • v0.3.8 :
    • Integrate cnpy into unrealcv
    • Add vget /camera/depth npy, which can return tensor as a numpy binary.
  • v0.3.7 :
    • Add vget /camera/lit png to retrieve binary data without saving it.
  • v0.3.6 :
    • Change docs from markdown to reStructuredText
    • Add docker to automate tests
    • Add vset /action/keyboard [key_name] [delta]
  • v0.3.5 : Add vexec to support the invocation of blueprint functions, Add GetWorld() in FCommandHandler.
  • v0.3.4 : Delay the object mask painting from initialization code
  • v0.3.3 : Add vget /scene/name
  • v0.3.2 :
    • Add vget /unrealcv/version
    • Add vset /action/eyes_distance
    • Fix vget /camera/[id]/location to support multiple cameras
    • Update test code
  • v0.3.1 : Fix GWorld issue

v0.3.0 - Stability improvement

  • Add support for Unreal 4.13, 4.14
  • Stability improvement, fix crash caused by the usage of GWorld
  • Fix some incorrect ground truth, blueprint actor not correctly displayed.
  • Add playground project
  • Add docs to docs.unrealcv.org
  • Add API documentation created by doxygen
  • Fix an issue that prevents the packaging of games.
  • Add vrun command to exec UE4 built-in command

API update:

  • vrun [built-in command]
  • vset /camera/[id]/moveto [x] [y] [z] # With collision enabled

v0.2.0 - First public release

Features

  • Add communication to UE4Editor and a compiled game
  • Add Python and MATLAB client to communicate with UnrealCV server
  • Add ground truth extraction, include: depth, object-mask, surface normal
  • Add support for Linux, Win and Mac

Initial API, see more details in the command list

  • vget /objects
  • vget /object/[obj_name]/color
  • vset /object/[obj_name]/color [r] [g] [b]
  • vget /camera/[id]/location
  • vget /camera/[id]/rotation
  • vset /camera/[id]/location [x] [y] [z]
  • vset /camera/[id]/rotation [pitch] [yaw] [roll]
  • vget /camera/[id]/[viewmode]
  • vget /camera/[id]/[viewmode] [filename]
  • vset /viewmode [viewmode]
  • vget /viewmode
  • vget /unrealcv/status
  • vget /unrealcv/help

The upcoming release will follow the concept of Semantic Versioning

Contact

UnrealCV is an open source project created by Weichao Qiu. It is hosted in Github and everyone is welcomed to contribute.

  • If you want a new feature for your research or found any bug, please submit an issue in the issue tracker.
  • Want to provide a compiled binary for the community? Please read the guidance in Model Zoo.
  • Want to learn how to create a game using Unreal Engine 4? Please check the UE4 official documentation.

Team Members

UnrealCV is developed and maintained by a group of students working on computer vision. The list of all contributors can be found here.

Here is a brief introduction of team members. If you want to be added to here, click the Edit on Github button on the top-right of this page.

  • Weichao Qiu: Constructing virtual worlds to train and diagnose computer vision algorithms
  • Fangwei Zhong: Reinforce learning in virtual environments
  • Siyuan Qiao: Stochastic virtual scene generation
  • Tae Soo Kim: Deep learning for transferable concepts between the real and the virtual environment
  • Yi Zhang: Algorithm diagnosis with synthetic data
  • Zihao Xiao

UE4 Resources

If you are interested in using synthetic data in your research projects, please see the synthetic-computer-vision repository for some related works.

UE4 is a good choice for research for various reasons:

  • Rich 3D resources
  • Open source
  • Cross-platform
  • Physically based material system

Here we collect some resources to help you use UE4 and UnrealCV in your research.

3D Resources (models, scenes)

C++ Development

Development

UnrealCV can be compiled as a plugin as shown in the Compile from source code, but it you want to modify the plugin source code, compiling it together with an UE4 C++ project will make debug much easier. For development, we need to

  • Create a C++ game project
  • Get the corresponding plugin
  • Compile the C++ project
  • Add a new command

Create a C++ game project

UE4 has two types of game projects. Blueprint project and C++ project. We need a C++ project.

In a C++ project, the plugin code will be compiled together with the game project.

The simplest way to start is using the playground project. Playground project is a simple UE4 project to show basic features of UE4 and UnrealCV. It serves as a development base and test platform for UnrealCV team.

We use git-lfs to manage large binary files. Please make sure you have installed git-lfs on your computer. Then you can get the playground project by

git lfs clone https://github.com/unrealcv/playground.git

Get the corresponding plugin

Now the folder Plugins/UnrealCV in the repository is empty. Please refer to Install UnrealCV Plugin to get the corresponding plugin and put it in the folder.

Compile the C++ project

Windows

  • Install visual studio.
  • To generate visual studio solution, right click playground.uproject and choose Generate Visual Studio project files.
  • Open the *.sln solution file and build the solution. The plugin code will be compiled together with the game project.

Linux

# Modify to a path that specified in step 1
export UE4=/home/qiuwch/UnrealEngine/4.13
UE4Editor() { bin=${UE4}/Engine/Binaries/Linux/UE4Editor; file=`pwd`/$1; $bin $file; }
UE4GenerateProjectFiles() {
  bin=${UE4}/GenerateProjectFiles.sh; file=`pwd`/$1;
  $bin -project=${file} -game -engine;
}
  • Generate project file and use Makefile
UE4GenerateProjectFiles playground.uproject
make playgroundEditor
# or make playgroundEditor-Linux-Debug
UE4Editor playground.uproject

Mac

  • Install Xcode.
  • To generate Xcode Project, right click playground.uproject and choose Service->Generate Xcode Project.
  • Open the *.xcworkspace file and build. The plugin code will be compiled together with the game project.

Useful resources for development, UnrealCV architecture

Add a new command

UnrealCV provides a set of commands for accomplishing tasks and the list is growing. But it might not be sufficient for your task. If you need any function that is missing, you can try to implement it yourself.

The benefits of implementing an UnrealCV command are:

  1. You can use the communication protocol provided by UnrealCV to exchange data between your program and UE4.
  2. You can share your code with other researchers, so that it can be used by others.

Note

You are supposed to edit your code in playground->Plugins->UnrealCV instead of UE4->Plugins->UnrealCV.

First we go through a very simple example which prints a message. Assume that we want to add a commamd vget /object/helloworld to print “Hello World!”. We need to modify two files: ObjectHandler.h and ObjectHandler.cpp.

In ObjectHandler.h, we need to add a member function:

FExecStatus HelloWorld(const TArray<FString>& Args);

In ObjectHandler.cpp, we define this member function:

FExecStatus FObjectCommandHandler::HelloWorld(const TArray<FString>& Args)
{
        FString Msg;
        Msg += "Hello World!";
        return FExecStatus::OK(Msg);
}

Then we need to bind the command with the function:

void FObjectCommandHandler::RegisterCommands()
{
        ...

        Cmd = FDispatcherDelegate::CreateRaw(this, &FObjectCommandHandler::HelloWorld);
        Help = "Print Hello World";
        CommandDispatcher->BindCommand(TEXT("vget /object/helloworld"), Cmd, Help);

        ...
}

After the modification, we can compile and use the new command.

Here we will walk you through how to implement a command vset /object/[id]/rotation to enable you set the rotation of an object.

FExecStatus return the exec result of this command. The result will be returned as a text string.

Available variables for a command are GetWorld(), GetActor(), , GetLevel().

A new function will be implemented in a CommandHandler. CommandDispatcher will use CommandHandler.

Python API

Some useful utility functions for reading data

The unrealcv.automation module to help building the plugin, packaging model zoo binaries, etc.

Contribute

We will be grateful if you help us mature UnrealCV.

Bug Reporting

If you find a bug in UnrealCV, you can search for similar issues in the Github first. If you still cannot solve the problem, please open an issue and describe the bug you encountered.

Requesting A New Feature

UnrealCV is still developing and we really want to make it more helpful to computer vision researchers. Please tell us your expectation about what can be done with UnrealCV. You are welcomed to open an issue and tell us what you want.

Pull Request

If you make improvements to UnrealCV,like fixing a bug or adding a new feature, please submit a pull request. Please make sure that you follow the syntax of UnrealCV. You will be greatly appreciated if you can fully describe the changes you have made.