Difference between revisions of "Comparison of X3D AR Proposals"
| Line 9: | Line 9: | ||
== 1. Introduction == | == 1. Introduction == | ||
| − | This document compares the existing proposals for extending X3D to support augmented and mixed reality visualization. Three | + | This document compares the existing proposals for extending X3D to support augmented and mixed reality visualization. Three main proposals are compared in terms of requirements – two from Korean Chapter (KC1 and KC2) and one from Instant Reality (IR). |
| Line 15: | Line 15: | ||
== 2. Using Live Video stream as a texture == | == 2. Using Live Video stream as a texture == | ||
| − | === 2.1 Proposal | + | === 2.1 Proposal KC1 === |
This proposal proposed a new sensor node, CameraSensor (previously named LiveCamera node), for retrieving live video data from a camera device, and then routing the video stream to a PixelTexture node. The X3D browser is in charge of implementing and handling devices and mapping the video data to the CameraSensor node inside the X3D scene. The video stream itself is provided as a value (SFImage) field of the node which is updated every frame by the browser implementation according to the camera data. | This proposal proposed a new sensor node, CameraSensor (previously named LiveCamera node), for retrieving live video data from a camera device, and then routing the video stream to a PixelTexture node. The X3D browser is in charge of implementing and handling devices and mapping the video data to the CameraSensor node inside the X3D scene. The video stream itself is provided as a value (SFImage) field of the node which is updated every frame by the browser implementation according to the camera data. | ||
| Line 41: | Line 41: | ||
While this approach avoids performance problems by not exposing SFImage fields updated in real-time, it lacks of supports for using live video stream data for other purposes, such as background. This is to be solved partially by adding a new node MovieBackground, which behaves similarly to the MovieTexture but uses the user selected movie file or live video stream from a camera for filling the background of the 3D scene. | While this approach avoids performance problems by not exposing SFImage fields updated in real-time, it lacks of supports for using live video stream data for other purposes, such as background. This is to be solved partially by adding a new node MovieBackground, which behaves similarly to the MovieTexture but uses the user selected movie file or live video stream from a camera for filling the background of the 3D scene. | ||
| − | === 2.2 Proposal | + | === 2.2 Proposal KC2 === |
| − | This proposal uses similar approach to Proposal | + | This proposal uses similar approach to Proposal KC1 in terms of explicitly defining a sensor node that represents a camera device. The video stream on the image field of the LiveCamera node is then routed to a texture node. |
<pre> | <pre> | ||
| Line 57: | Line 57: | ||
| − | === 2.3 Proposal | + | === 2.3 Proposal IR === |
This proposal proposes a general purpose IOSensor node, which allows to access external devices (e.g., joysticks and cameras) inside the X3D scene. Note: Due to technical reasons the input and output fields of a device can only be determined at runtime. Hence, most in-/out-slots are dynamically generated based on the device one wants to access. | This proposal proposes a general purpose IOSensor node, which allows to access external devices (e.g., joysticks and cameras) inside the X3D scene. Note: Due to technical reasons the input and output fields of a device can only be determined at runtime. Hence, most in-/out-slots are dynamically generated based on the device one wants to access. | ||
| Line 82: | Line 82: | ||
=== 2.4 Discussion === | === 2.4 Discussion === | ||
| − | Proposal | + | Proposal KC1 and KC2 proposes a new node, specific for a camera, while proposal IR proposes a more generic type of node to be applied for variety of sensors. The trade off between simplicity and flexibility/extensibility needs further discussion. |
| Line 89: | Line 89: | ||
== 3. Using Live Video stream as a background == | == 3. Using Live Video stream as a background == | ||
| − | === 3.1 Proposal | + | === 3.1 Proposal KC1 === |
The proposal proposed a MovieBackground node, extended from Background node to support ‘liveSource’ field which is assigned with a CameraSensor node (as described in 2.1) from which the Background node receives the live video stream data. Once the ‘liveSource’ field is assigned with a validate CameraSensor node, the background image is updated according to the live video stream from the CameraSensor node, assigned. For other purpose of use, it could also have a url field on which general source of movie clip could be assigned an used as a background. | The proposal proposed a MovieBackground node, extended from Background node to support ‘liveSource’ field which is assigned with a CameraSensor node (as described in 2.1) from which the Background node receives the live video stream data. Once the ‘liveSource’ field is assigned with a validate CameraSensor node, the background image is updated according to the live video stream from the CameraSensor node, assigned. For other purpose of use, it could also have a url field on which general source of movie clip could be assigned an used as a background. | ||
| Line 103: | Line 103: | ||
| − | === 3.2 Proposal | + | === 3.2 Proposal KC2 === |
This proposal proposes to extend the TextureBackground node to support live video background. The video stream image is routed from the LiveCamera node to the frontTexture field. However, since TextureBackground node acts as an environment map there is a problem with the orientation of the TextureBackground, which is world registered and is not fixed to the viewpoint movement. To solve this problem, this proposal proposes to add a Boolean field called the ARmode. When the ARmode flag is true, the orientation of TextureBackground is fixed to the viewpoint, resulting the front side texture remains as a background. | This proposal proposes to extend the TextureBackground node to support live video background. The video stream image is routed from the LiveCamera node to the frontTexture field. However, since TextureBackground node acts as an environment map there is a problem with the orientation of the TextureBackground, which is world registered and is not fixed to the viewpoint movement. To solve this problem, this proposal proposes to add a Boolean field called the ARmode. When the ARmode flag is true, the orientation of TextureBackground is fixed to the viewpoint, resulting the front side texture remains as a background. | ||
| Line 129: | Line 129: | ||
| − | === 3.3 Proposal | + | === 3.3 Proposal IR === |
This proposal deals with the problem similar to the case for using the camera image for texture. It proposes a PolygonBackground node, which represents a background that renders a single polygon using the specified appearance. It allows for defining an aspect ratio of the background image that is independent of the actual window size. Different modes are possible to fit the image in the window (vertical or horizontal). | This proposal deals with the problem similar to the case for using the camera image for texture. It proposes a PolygonBackground node, which represents a background that renders a single polygon using the specified appearance. It allows for defining an aspect ratio of the background image that is independent of the actual window size. Different modes are possible to fit the image in the window (vertical or horizontal). | ||
| Line 156: | Line 156: | ||
=== 3.4 Discussion === | === 3.4 Discussion === | ||
| − | Proposal | + | Proposal KC1 proposes a dedicated node for movie backgrounds. Proposal KC2 proposes to extend TextureBackground node and repurpose it for fixed textured background. Proposal IR takes more general approach, and proposes a multi-purpose PolygonBackground node, which can contain any type of appearance including shaders. While the latter gives more flexibility, it requires details to be elaborated, compared to the former which is more simple. Again, the trade off between simplicity and flexibility/extensibility needs further discussion. |
== 4. Supporting color keying in texture == | == 4. Supporting color keying in texture == | ||
| − | === 4.1 Proposal | + | === 4.1 Proposal KC1 === |
This proposal proposed to add a ‘keyColor’ field to the MovieTexture node, which indicates the color expected to be rendered as transparent, in order to provide chroma key effect on the movie texture. The browser will be in charge of rendering the parts of the MovieTexture with as transparent, and those browser that does not support this feature could simply fall back with rendering the MovieTexture in a normal way (i.e. showing the texture as is). | This proposal proposed to add a ‘keyColor’ field to the MovieTexture node, which indicates the color expected to be rendered as transparent, in order to provide chroma key effect on the movie texture. The browser will be in charge of rendering the parts of the MovieTexture with as transparent, and those browser that does not support this feature could simply fall back with rendering the MovieTexture in a normal way (i.e. showing the texture as is). | ||
| Line 171: | Line 171: | ||
</pre> | </pre> | ||
| − | === 4.2 Proposal | + | === 4.2 Proposal KC2 === |
This proposal does not include this feature. | This proposal does not include this feature. | ||
| − | === 4.3 Proposal | + | === 4.3 Proposal IR === |
This proposal doesn't include a direct solution to this case, since it is not straightforward related to AR applications. Closely related functions in this proposal would be the ColorMaskMode node, the BlendMode, StencilMode and DepthMode as a child of the Appearance node. | This proposal doesn't include a direct solution to this case, since it is not straightforward related to AR applications. Closely related functions in this proposal would be the ColorMaskMode node, the BlendMode, StencilMode and DepthMode as a child of the Appearance node. | ||
| Line 195: | Line 195: | ||
=== 4.4 Discussion === | === 4.4 Discussion === | ||
| − | Proposal | + | Proposal KC1 suggests simpler way to provide a specific color keying function for textures, while proposal IR suggests a more generic functions that can achieve required function. Although, the corresponding nodes in proposal IR misses certain features to fulfill color keying out-of-the-box, this can be achieved via shaders. Again, the trade off between simplicity and flexibility/extensibility needs further discussion. |
| Line 201: | Line 201: | ||
== 5. Retrieving tracking information == | == 5. Retrieving tracking information == | ||
| − | === 5.1 Proposal | + | === 5.1 Proposal KC1 === |
This proposal suggests using the same CameraSensor node, used for retrieving live video stream, for retrieving tracking information. As described in 2.1, the proposed CameraSensor node includes ‘position’ and ‘orientation’ fields that represent the tracking information of the camera motion. | This proposal suggests using the same CameraSensor node, used for retrieving live video stream, for retrieving tracking information. As described in 2.1, the proposed CameraSensor node includes ‘position’ and ‘orientation’ fields that represent the tracking information of the camera motion. | ||
| Line 217: | Line 217: | ||
The method has its limitations with not supporting tracking information of general objects other than the camera sensor. | The method has its limitations with not supporting tracking information of general objects other than the camera sensor. | ||
| − | === 5.2 Proposal | + | === 5.2 Proposal KC2 === |
| − | This proposal proposes a new node named "ImagePatch" which provides tracking information of a visual marker. In comparison with Proposal | + | This proposal proposes a new node named "ImagePatch" which provides tracking information of a visual marker. In comparison with Proposal KC1, this is a separate node from a node that represents camera sensor. This allows using multiple visual markers for tracking. |
<pre> | <pre> | ||
| Line 239: | Line 239: | ||
| − | === 5.3 Proposal | + | === 5.3 Proposal IR === |
For retrieving tracking information, this proposal uses the same IOSensor node as used for retrieving camera image. In this example, the TrackedObject1Camera_ModelView field of the IOSensor node represents the transformation matrix of the tracked position/orientation of the tracked object (visual marker). However, these are all dynamic fields and depend on the configuration as defined in the pm file. | For retrieving tracking information, this proposal uses the same IOSensor node as used for retrieving camera image. In this example, the TrackedObject1Camera_ModelView field of the IOSensor node represents the transformation matrix of the tracked position/orientation of the tracked object (visual marker). However, these are all dynamic fields and depend on the configuration as defined in the pm file. | ||
| Line 257: | Line 257: | ||
=== 5.4 Discussion === | === 5.4 Discussion === | ||
| − | While both proposes to retrieve tracking information from a node that represents a camera sensor, proposal | + | While both proposal KC1 and IR proposes to retrieve tracking information from a node that represents a camera sensor, proposal KC1 gives the tracking information of the camera, while proposal IR deals with the tracking information of tracked object. This makes proposal IR to be more extensible in terms of supporting multiple tracking objects. However, the method used for defining tracking objects and markers through proprietary configuration file needs to be revised for standardization. On the other hand, proposal KC2 proposes a dedicated node for tracking, separated from a camera sensor node. As a result, multiple tracking objects are easily supported by creating multiple instances of this tracking node. Proposal KC2 also has a GPS tracking node, besides computer vision based tracking. GPS-based tracking should be investigated and compared to another proposal by Myeongwon Lee, which was originally discussed at X3D Earth working group [http://www.web3d.org/membership/login/memberwiki/index.php/X3D_v3.3_Specification_Changes#GpsSensor_node]. |
| Line 263: | Line 263: | ||
== 6. Using tracking information to change 3D scene == | == 6. Using tracking information to change 3D scene == | ||
| − | === 6.1 Proposal | + | === 6.1 Proposal KC1 === |
This proposal proposes to use routing method to link tracking information from the CameraSensor node to a Viewpoint node’s position and orientation, in general. This could be also extended by a MatrixViewpoint node (to be described in 8.1) which could have a field to identify the corresponding CameraSensor node, causing the same results without explicitly routing the corresponding fields. | This proposal proposes to use routing method to link tracking information from the CameraSensor node to a Viewpoint node’s position and orientation, in general. This could be also extended by a MatrixViewpoint node (to be described in 8.1) which could have a field to identify the corresponding CameraSensor node, causing the same results without explicitly routing the corresponding fields. | ||
| − | === 6.2 Proposal | + | === 6.2 Proposal KC2 === |
This proposal also uses routing method for using tracking information in the 3D scene, routing tracking results (position and orientation) to transform nodes. | This proposal also uses routing method for using tracking information in the 3D scene, routing tracking results (position and orientation) to transform nodes. | ||
| Line 294: | Line 294: | ||
| − | === 6.3 Proposal | + | === 6.3 Proposal IR === |
This proposal proposes to use the routing method to link tracking information from the IOSensor node to a Transform node of a corresponding virtual object or viewpoint. Example: | This proposal proposes to use the routing method to link tracking information from the IOSensor node to a Transform node of a corresponding virtual object or viewpoint. Example: | ||
| Line 338: | Line 338: | ||
=== 6.4 Discussion === | === 6.4 Discussion === | ||
| − | While all of the proposals relies on routing for applying tracking results for updating the 3D scene, as discussed in 5.4, proposal | + | While all of the proposals relies on routing for applying tracking results for updating the 3D scene, as discussed in 5.4, proposal KC1 focuses on updating the Viewpoint node, while proposal KC2 and IR allows updating both, the camera as well as a virtual object (or scene). Proposal IR also proposes a new type of transformation node for dealing with transformation matrices, too, while proposal KC1 sticks to traditional position and orientation vectors. |
| − | In addition, proposal | + | In addition, proposal KC2 proposes higher level event generation nodes that triggers tracking based events such as proximity and visibility. |
| Line 345: | Line 345: | ||
== 7. Retrieving camera calibration (internal parameters) information == | == 7. Retrieving camera calibration (internal parameters) information == | ||
| − | === 7.1 Proposal | + | === 7.1 Proposal KC1 === |
This proposal suggests using the same CameraSensor node, used for retrieving live video stream, for retrieving camera calibration information. As described in 2.1, the proposed CameraSensor node includes a ‘projmat’ field which represents the calibration information of the CameraSensor. | This proposal suggests using the same CameraSensor node, used for retrieving live video stream, for retrieving camera calibration information. As described in 2.1, the proposed CameraSensor node includes a ‘projmat’ field which represents the calibration information of the CameraSensor. | ||
| Line 360: | Line 360: | ||
| − | === 7.2 Proposal | + | === 7.2 Proposal KC2 === |
| − | This proposal takes similar approach to proposal | + | This proposal takes similar approach to proposal KC1, providing a field representing camera calibration information in the node for live video camera. |
| − | === 7.3 Proposal | + | === 7.3 Proposal IR === |
This proposal suggests using the same IOSensor node, used for retrieving images from camera sensor. Several fields (in this example they are called e.g. TrackedObject1Camera_PrincipalPoint, TrackedObject1Camera_FOV_horizontal, TrackedObject1Camera_FOV_vertical, TrackedObject1Camera_CAM_aspect) in this node provide the calibration information. | This proposal suggests using the same IOSensor node, used for retrieving images from camera sensor. Several fields (in this example they are called e.g. TrackedObject1Camera_PrincipalPoint, TrackedObject1Camera_FOV_horizontal, TrackedObject1Camera_FOV_vertical, TrackedObject1Camera_CAM_aspect) in this node provide the calibration information. | ||
| Line 379: | Line 379: | ||
=== 7.4 Discussion === | === 7.4 Discussion === | ||
| − | All three proposals suggest reusing the node that is used for accessing a camera sensor, and using a dedicated field of that node for providing camera calibration information. While proposal | + | All three proposals suggest reusing the node that is used for accessing a camera sensor, and using a dedicated field of that node for providing camera calibration information. While proposal KC1 and KC2 suggest using projection matrix as a calibration information, proposal IR suggests using a set of parameters. The latter approach could be safer in terms of encapsulating the projection matrix of a viewpoint which could be implementation dependent based on what graphics API it is using. |
| Line 385: | Line 385: | ||
== 8. Using calibration information to set properties of (virtual) camera == | == 8. Using calibration information to set properties of (virtual) camera == | ||
| − | === 8.1 Proposal | + | === 8.1 Proposal KC1 === |
This proposal suggests a MatrixViewpoint node, which is a child of a scene node which represents a virtual viewpoint calibrated according to the corresponding physical live video camera (on the user's computer). The 'projmat' field represents the internal parameters (or projection matrix) of the MatrixViewpoint. The ‘position' and ‘orientation’ fields represent three dimensional position and orientation of the viewpoint within the virtual space. The ‘cameraSensor’ field represents a CameraSensor node, from which the viewpoint parameters (including projmat, position and orientation fields) of the MatrixViewpoint are updated according to. Once the ‘cameraSensor’ field is assigned with a validate CameraSensor node, the viewpoint parameters are updated according to the values from the CameraSensor node, assigned. Otherwise, it could be also used with routing each parameter of the MatrixViewpoint node from corresponding source of calibrated values. | This proposal suggests a MatrixViewpoint node, which is a child of a scene node which represents a virtual viewpoint calibrated according to the corresponding physical live video camera (on the user's computer). The 'projmat' field represents the internal parameters (or projection matrix) of the MatrixViewpoint. The ‘position' and ‘orientation’ fields represent three dimensional position and orientation of the viewpoint within the virtual space. The ‘cameraSensor’ field represents a CameraSensor node, from which the viewpoint parameters (including projmat, position and orientation fields) of the MatrixViewpoint are updated according to. Once the ‘cameraSensor’ field is assigned with a validate CameraSensor node, the viewpoint parameters are updated according to the values from the CameraSensor node, assigned. Otherwise, it could be also used with routing each parameter of the MatrixViewpoint node from corresponding source of calibrated values. | ||
| Line 398: | Line 398: | ||
| − | === 8.2 Proposal | + | === 8.2 Proposal KC2 === |
| − | This proposal suggests similar approach with proposal | + | This proposal suggests similar approach with proposal KC1, which uses a viewpoint node that accepts camera calibration information in matrix form. |
| − | === 8.3 Proposal | + | === 8.3 Proposal IR === |
<pre> | <pre> | ||
| Line 438: | Line 438: | ||
== 9. Specifying nodes as physical object representatives == | == 9. Specifying nodes as physical object representatives == | ||
| − | === 9.1 Proposal | + | === 9.1 Proposal KC1 === |
This proposal suggests a GhostGroup node for indicating its child nodes being representatives of physical objects for visualizing correct occlusion. The proposed node is extended from Group node to support those geometries of its child nodes are rendered as ghost objects. The browser should render the child nodes only into the depth buffer and not into the color buffer. As a result, the portion of the live video image corresponding to the ghost object is visualized with correct depth value, forming correct occlusion with other virtual objects. | This proposal suggests a GhostGroup node for indicating its child nodes being representatives of physical objects for visualizing correct occlusion. The proposed node is extended from Group node to support those geometries of its child nodes are rendered as ghost objects. The browser should render the child nodes only into the depth buffer and not into the color buffer. As a result, the portion of the live video image corresponding to the ghost object is visualized with correct depth value, forming correct occlusion with other virtual objects. | ||
| Line 448: | Line 448: | ||
| − | === 9.2 Proposal | + | === 9.2 Proposal KC2 === |
This proposal does not include this feature. | This proposal does not include this feature. | ||
| − | === 9.3 Proposal | + | === 9.3 Proposal IR === |
This proposal proposes using a ColorMaskMode node to render the geometry not into color buffer, and only to the depth buffer. In addition, a new field "sortKey" is proposed for the Appearance node for making sure the ghost objects are rendered prior to other geometries. | This proposal proposes using a ColorMaskMode node to render the geometry not into color buffer, and only to the depth buffer. In addition, a new field "sortKey" is proposed for the Appearance node for making sure the ghost objects are rendered prior to other geometries. | ||
| Line 466: | Line 466: | ||
=== 9.4 Discussion === | === 9.4 Discussion === | ||
| − | While proposal | + | While proposal KC1 suggests a high level, simple to use approach for a specific application in AR/MR, proposal IR provides more detail controllable, general purpose approach. |
== 10. Conclusion == | == 10. Conclusion == | ||
| − | While all of the proposals suggest and cover similar set of functionalities required for supporting AR and MR visualization in X3D, Proposal | + | While all of the proposals suggest and cover similar set of functionalities required for supporting AR and MR visualization in X3D, Proposal KC1 and KC2 take the path of relatively higher level control, that provides simpler syntax that could be applied for specific cases for AR and MR. On the contrary, proposal IR introduces more generic purpose nodes and suggests to combine them to implement required functions, making AR and MR visualization a special use case of proposed extension. Considering the difference between the proposals, trade off between simplicity and flexibility/extensibility leads to a need for further discussion. |
In the content authors' point of view, providing higher-level abstracted control gives more simpler and easy to use syntax. However, detail control might be missing which could be necessary for applications other than AR/MR. | In the content authors' point of view, providing higher-level abstracted control gives more simpler and easy to use syntax. However, detail control might be missing which could be necessary for applications other than AR/MR. | ||
Revision as of 21:42, 19 March 2012
Contents
- 1 Comparison between existing proposals - Working Draft
- 1.1 1. Introduction
- 1.2 2. Using Live Video stream as a texture
- 1.3 3. Using Live Video stream as a background
- 1.4 4. Supporting color keying in texture
- 1.5 5. Retrieving tracking information
- 1.6 6. Using tracking information to change 3D scene
- 1.7 7. Retrieving camera calibration (internal parameters) information
- 1.8 8. Using calibration information to set properties of (virtual) camera
- 1.9 9. Specifying nodes as physical object representatives
- 1.10 10. Conclusion
Comparison between existing proposals - Working Draft
Augmented Reality Working Group, Web3D Consortium
Jan 25, 2012
1. Introduction
This document compares the existing proposals for extending X3D to support augmented and mixed reality visualization. Three main proposals are compared in terms of requirements – two from Korean Chapter (KC1 and KC2) and one from Instant Reality (IR).
2. Using Live Video stream as a texture
2.1 Proposal KC1
This proposal proposed a new sensor node, CameraSensor (previously named LiveCamera node), for retrieving live video data from a camera device, and then routing the video stream to a PixelTexture node. The X3D browser is in charge of implementing and handling devices and mapping the video data to the CameraSensor node inside the X3D scene. The video stream itself is provided as a value (SFImage) field of the node which is updated every frame by the browser implementation according to the camera data.
CameraSensor:X3DDirectSensorNode {
SFImage [out] value
SFBool [out] on FALSE
SFMatrix4f [out] projmat "1 0 0 0 … “
SFBool [out] tracking FALSE
SFVec3f [out] position
SFRotation [out] orientation
}
While this straight forward, routing SFImage values might lead to performance and implementation problem. As an alternative, the same proposal also proposed to extend the behavior of the existing MovieTexture node to support live video stream within the node. The proposed behavior X3D browser is to allow users to select a file or a camera device for the MovieTexture node in the scene, if the url field of the node is empty (or filled with special token values, such as ‘USER_CUSTOMIZED’).
<Appearance>
<MovieTexture loop='true' url=''/>
</Appearance>
While this approach avoids performance problems by not exposing SFImage fields updated in real-time, it lacks of supports for using live video stream data for other purposes, such as background. This is to be solved partially by adding a new node MovieBackground, which behaves similarly to the MovieTexture but uses the user selected movie file or live video stream from a camera for filling the background of the 3D scene.
2.2 Proposal KC2
This proposal uses similar approach to Proposal KC1 in terms of explicitly defining a sensor node that represents a camera device. The video stream on the image field of the LiveCamera node is then routed to a texture node.
LiveCamera {
SFString [in, out] source “default”
SFImage [out] image
SFMatrix4f [out] projmat “1 0 0 …”
SFBool [out] on FALSE
SFBool [out] tracking FALSE
SFVec3f [out] position
SFRotation [out] orientation
}
2.3 Proposal IR
This proposal proposes a general purpose IOSensor node, which allows to access external devices (e.g., joysticks and cameras) inside the X3D scene. Note: Due to technical reasons the input and output fields of a device can only be determined at runtime. Hence, most in-/out-slots are dynamically generated based on the device one wants to access.
<IOSensor type='' name='' description='' enabled='TRUE' />
The camera sensor (including marker/poster/... tracking) is loaded through an instance of IOSensor, by defining the type of the sensor and it's fields as specified in the configFile (*.pm). Here is an example:
<IOSensor DEF='VisionLib' type='VisionLib' configFile='TutorialMarkerTracking_OneMarker.pm'>
<field accessType='outputOnly' name='VideoSourceImage' type='SFImage'/>
<field accessType='outputOnly' name='TrackedObject1Camera_ModelView' type='SFMatrix4f'/>
<field accessType='outputOnly' name='TrackedObject1Camera_PrincipalPoint' type='SFVec2f'/>
<field accessType='outputOnly' name='TrackedObject1Camera_FOV_horizontal' type='SFFloat'/>
<field accessType='outputOnly' name='TrackedObject1Camera_FOV_vertical' type='SFFloat'/>
<field accessType='outputOnly' name='TrackedObject1Camera_CAM_aspect' type='SFFloat'/>
</IOSensor>
Using the camera image for texture is nothing more than routing the VideoSourceImage field of the IOSensor node to a PixelTexture node, which can also be part of a Background or Foreground appearance.
2.4 Discussion
Proposal KC1 and KC2 proposes a new node, specific for a camera, while proposal IR proposes a more generic type of node to be applied for variety of sensors. The trade off between simplicity and flexibility/extensibility needs further discussion.
3. Using Live Video stream as a background
3.1 Proposal KC1
The proposal proposed a MovieBackground node, extended from Background node to support ‘liveSource’ field which is assigned with a CameraSensor node (as described in 2.1) from which the Background node receives the live video stream data. Once the ‘liveSource’ field is assigned with a validate CameraSensor node, the background image is updated according to the live video stream from the CameraSensor node, assigned. For other purpose of use, it could also have a url field on which general source of movie clip could be assigned an used as a background.
MovieBackground:X3DBackgroundNode {
... // same to the original Background node
SFString [in] url
SFNode [in] liveSource
}
Similar to the case in 2.1, the proposal also suggests a different approach where the MovieBackground node doesn’t explicitly need a CameraSensor node, but to let the browser to ask the user to choose the movie source (including camera device) when the url field is left empty (or filled with special token values, such as ‘USER_CUSTOMIZED’).
3.2 Proposal KC2
This proposal proposes to extend the TextureBackground node to support live video background. The video stream image is routed from the LiveCamera node to the frontTexture field. However, since TextureBackground node acts as an environment map there is a problem with the orientation of the TextureBackground, which is world registered and is not fixed to the viewpoint movement. To solve this problem, this proposal proposes to add a Boolean field called the ARmode. When the ARmode flag is true, the orientation of TextureBackground is fixed to the viewpoint, resulting the front side texture remains as a background.
TextureBackground : X3DBackgroundNode
{
SFBool [in] set_bind
SFBool [in] ARmode
MFFloat [in,out] groundAngle [] [0,π/2]
MFColor [in,out] groundColor [] [0,1]
SFNode [in,out] backTexture NULL [X3DTextureNode]
SFNode [in,out] bottomTexture NULL [X3DTextureNode]
SFNode [in,out] frontTexture NULL [X3DTextureNode]
SFNode [in,out] leftTexture NULL [X3DTextureNode]
SFNode [in,out] metadata NULL [X3DMetadataObject]
SFNode [in,out] rightTexture NULL [X3DTextureNode]
SFNode [in,out] topTexture NULL [X3DTextureNode]
MFFloat [in,out] skyAngle [] [0,π]
MFColor [in,out] skyColor 0 0 0 [0,1]
SFFloat [in,out] transparency 0 [0,1]
SFTime [out] bindTime
SFBool [out] isBound
}
3.3 Proposal IR
This proposal deals with the problem similar to the case for using the camera image for texture. It proposes a PolygonBackground node, which represents a background that renders a single polygon using the specified appearance. It allows for defining an aspect ratio of the background image that is independent of the actual window size. Different modes are possible to fit the image in the window (vertical or horizontal).
<PolygonBackground positions='0 0, 1 0, 1 1, 0 1' texCoords='0 0 0, 1 0 0, 1 1 0, 0 1 0' normalizedX='TRUE' normalizedY='TRUE' fixedImageSize='0,0' zoomFactor='1.0' tile='TRUE' doCleanup='TRUE' mode='VERTICAL' clearStencilBitplanes='-1' description='' />
Using the proposed PolygonBackground node, the image from the camera is simply routed to the texture used for the PolygonBackground node. The image assigned to the image outslot of the IOSensor is routed to the texture in the appearance of the PolygonBackground node.
<PolygonBackground fixedImageSize='640,480' mode='VERTICAL'>
<Appearance>
<PixelTexture DEF='tex' />
<TextureTransform scale='1 -1'/>
</Appearance>
</PolygonBackground>
<ROUTE fromNode='VisionLib' fromField='VideoSourceImage' toNode='tex' toField='image'/>
To make the polygon for the background fill the viewport, the PolygonBackground's field fixedImageSize is used for describing the aspect ratio of the image, and the mode field is set to "VERTICAL" or "HORIZONTAL" which describes the way the polygon fits the viewport.
3.4 Discussion
Proposal KC1 proposes a dedicated node for movie backgrounds. Proposal KC2 proposes to extend TextureBackground node and repurpose it for fixed textured background. Proposal IR takes more general approach, and proposes a multi-purpose PolygonBackground node, which can contain any type of appearance including shaders. While the latter gives more flexibility, it requires details to be elaborated, compared to the former which is more simple. Again, the trade off between simplicity and flexibility/extensibility needs further discussion.
4. Supporting color keying in texture
4.1 Proposal KC1
This proposal proposed to add a ‘keyColor’ field to the MovieTexture node, which indicates the color expected to be rendered as transparent, in order to provide chroma key effect on the movie texture. The browser will be in charge of rendering the parts of the MovieTexture with as transparent, and those browser that does not support this feature could simply fall back with rendering the MovieTexture in a normal way (i.e. showing the texture as is).
MovieTexture:X3DTexture2DNode {
... // same to the MovieTexture node described in 2.1
SFColor [in] keyColor
}
4.2 Proposal KC2
This proposal does not include this feature.
4.3 Proposal IR
This proposal doesn't include a direct solution to this case, since it is not straightforward related to AR applications. Closely related functions in this proposal would be the ColorMaskMode node, the BlendMode, StencilMode and DepthMode as a child of the Appearance node.
The ColorMaskMode masks a specific color channel, and this results in color changes in the global image. Rather than resulting pixels in key color to appear transparent, the ColorMaskMode makes color changes in every pixel. The ColorMaskMode together with the Appearance node's sortKey field (default sortKey is 0, a sortKey smaller than that is rendered first, and greater than another one is rendered last) can also be used to create invisible ghosting objects.
<ColorMaskMode maskR='TRUE' maskG='TRUE' maskB='TRUE' maskA='TRUE' />
The BlendMode gives general control over alpha blending function. However, there is no such function that compares the source images with a given key color, which is necessary to have proper result for color keying.
<BlendMode srcFactor='src_alpha' destFactor='one_minus_src_alpha' color='1 1 1' colorTransparency='0' alphaFunc='none' alphaFuncValue='0' equation='none' />
To achieve chroma keying for an arbitrary color, you can e.g. use a user defined shader that discards all fragments whose color is equal to the given one.
4.4 Discussion
Proposal KC1 suggests simpler way to provide a specific color keying function for textures, while proposal IR suggests a more generic functions that can achieve required function. Although, the corresponding nodes in proposal IR misses certain features to fulfill color keying out-of-the-box, this can be achieved via shaders. Again, the trade off between simplicity and flexibility/extensibility needs further discussion.
5. Retrieving tracking information
5.1 Proposal KC1
This proposal suggests using the same CameraSensor node, used for retrieving live video stream, for retrieving tracking information. As described in 2.1, the proposed CameraSensor node includes ‘position’ and ‘orientation’ fields that represent the tracking information of the camera motion.
CameraSensor:X3DDirectSensorNode {
SFImage [out] value
SFBool [out] on FALSE
SFMatrix4f [out] projmat "1 0 0 0 … “
SFBool [out] tracking FALSE
SFVec3f [out] position
SFRotation [out] orientation
}
The method has its limitations with not supporting tracking information of general objects other than the camera sensor.
5.2 Proposal KC2
This proposal proposes a new node named "ImagePatch" which provides tracking information of a visual marker. In comparison with Proposal KC1, this is a separate node from a node that represents camera sensor. This allows using multiple visual markers for tracking.
ImagePatch : X3DARNode
{
MFString [in, out] filename
SFVec3f [in, out] position
SFRotation [in, out] orientation
}
This proposal also proposes nodes for retrieving tracking information from sensors, other than vision based tracking. For instance, GPSLocation node provides tracking information from a GPS sensor.
GPSLocation : X3DSensorNode
{
SFBool [in, out] status
MFString [in, out] values
}
5.3 Proposal IR
For retrieving tracking information, this proposal uses the same IOSensor node as used for retrieving camera image. In this example, the TrackedObject1Camera_ModelView field of the IOSensor node represents the transformation matrix of the tracked position/orientation of the tracked object (visual marker). However, these are all dynamic fields and depend on the configuration as defined in the pm file.
<IOSensor DEF='VisionLib' type='VisionLib' configFile='TutorialMarkerTracking_OneMarker.pm'>
<field accessType='outputOnly' name='VideoSourceImage' type='SFImage'/>
<field accessType='outputOnly' name='TrackedObject1Camera_ModelView' type='SFMatrix4f'/>
<field accessType='outputOnly' name='TrackedObject1Camera_PrincipalPoint' type='SFVec2f'/>
<field accessType='outputOnly' name='TrackedObject1Camera_FOV_horizontal' type='SFFloat'/>
<field accessType='outputOnly' name='TrackedObject1Camera_FOV_vertical' type='SFFloat'/>
<field accessType='outputOnly' name='TrackedObject1Camera_CAM_aspect' type='SFFloat'/>
</IOSensor>
The node could support multiple tracking objects by changing the configFile (TutorialMarkerTracking_OneMarker.pm file in the sample code), and defining additional Modelview, Projection, etc. fields for tracked objects and/or the camera pose.
5.4 Discussion
While both proposal KC1 and IR proposes to retrieve tracking information from a node that represents a camera sensor, proposal KC1 gives the tracking information of the camera, while proposal IR deals with the tracking information of tracked object. This makes proposal IR to be more extensible in terms of supporting multiple tracking objects. However, the method used for defining tracking objects and markers through proprietary configuration file needs to be revised for standardization. On the other hand, proposal KC2 proposes a dedicated node for tracking, separated from a camera sensor node. As a result, multiple tracking objects are easily supported by creating multiple instances of this tracking node. Proposal KC2 also has a GPS tracking node, besides computer vision based tracking. GPS-based tracking should be investigated and compared to another proposal by Myeongwon Lee, which was originally discussed at X3D Earth working group [1].
6. Using tracking information to change 3D scene
6.1 Proposal KC1
This proposal proposes to use routing method to link tracking information from the CameraSensor node to a Viewpoint node’s position and orientation, in general. This could be also extended by a MatrixViewpoint node (to be described in 8.1) which could have a field to identify the corresponding CameraSensor node, causing the same results without explicitly routing the corresponding fields.
6.2 Proposal KC2
This proposal also uses routing method for using tracking information in the 3D scene, routing tracking results (position and orientation) to transform nodes.
Besides this basic method for using raw tracking information, this proposal also proposes higher level event nodes, such as VisibilitySensor and RangeSensor. VisibilitySensor node triggers events when whether tracking visual marker is detected or lost, while the RangeSensor node triggers event when a tracking object gets close enough within a certain range.
VisibilitySensor : X3DEnvironmentalSensorNode
{
SFBool [in, out] enabled
SFTime [out] enterTime
SFTime [out] exitTime
SFBool [out] isActive
}
RangeSensor : X3DEnvironmentalSensorNode
{
SFBool [in, out] enabled
SFTime [out] enterTime
SFTime [out] exitTime
SFBool [out] isActive
SFInt32 [in, out] sequence
SFString [in, out] lBound
SFString [in, out] uBound
}
6.3 Proposal IR
This proposal proposes to use the routing method to link tracking information from the IOSensor node to a Transform node of a corresponding virtual object or viewpoint. Example:
<MatrixTransform DEF='TransformRelativeToCam'>
<Shape>
<Appearance>
<Material diffuseColor='1 0.5 0' />
</Appearance>
<Teapot size='5 5 5' />
</Shape>
</MatrixTransform>
<ROUTE fromNode='VisionLib' fromField='Camera_ModelView' toNode='TransformRelativeToCam' toField='set_matrix'/>
For routing a transform matrix to a transform node, this proposal also proposes a MatrixTransform node that takes a transform matrix directly, rather than using position and orientation fields. The render field allows determining visibility.
MatrixTransform : X3DGroupingNode {
...
SFBool [in,out] render TRUE
SFMatrix4f [in,out] matrix identity
}
It is of course also possible to route the tracked camera pose (also in orientation/position notation) to the bound Viewpoint node. There are different field-of-view modes: vertical, horizontal, and smaller. The field-of-view and principal point delivered by the IOSensor can be routed to the viewpoint; example below.
<Viewpoint principalPoint='0 0' fieldOfView='0.785398' fovMode='SMALLER' aspect='1.0' retainUserOffsets='FALSE' zFar='-1' jump='TRUE' zNear='-1' description='' position='0 0 10' orientation='0 0 1 0' centerOfRotation='0 0 0' />
<Viewpoint DEF='vp' position='0 0 0' fovMode='VERTICAL‘ /> <ROUTE fromNode='VisionLib' fromField='Camera_PrincipalPoint' toNode='vp' toField='principalPoint'/> <ROUTE fromNode='VisionLib' fromField='Camera_FOV_vertical' toNode='vp' toField='fieldOfView'/> <ROUTE fromNode='VisionLib' fromField='Camera_CAM_aspect' toNode='vp' toField='aspect'/>
6.4 Discussion
While all of the proposals relies on routing for applying tracking results for updating the 3D scene, as discussed in 5.4, proposal KC1 focuses on updating the Viewpoint node, while proposal KC2 and IR allows updating both, the camera as well as a virtual object (or scene). Proposal IR also proposes a new type of transformation node for dealing with transformation matrices, too, while proposal KC1 sticks to traditional position and orientation vectors. In addition, proposal KC2 proposes higher level event generation nodes that triggers tracking based events such as proximity and visibility.
7. Retrieving camera calibration (internal parameters) information
7.1 Proposal KC1
This proposal suggests using the same CameraSensor node, used for retrieving live video stream, for retrieving camera calibration information. As described in 2.1, the proposed CameraSensor node includes a ‘projmat’ field which represents the calibration information of the CameraSensor.
CameraSensor:X3DDirectSensorNode {
SFImage [out] value
SFBool [out] on FALSE
SFMatrix4f [out] projmat "1 0 0 0 … “
SFBool [out] tracking FALSE
SFVec3f [out] position
SFRotation [out] orientation
}
7.2 Proposal KC2
This proposal takes similar approach to proposal KC1, providing a field representing camera calibration information in the node for live video camera.
7.3 Proposal IR
This proposal suggests using the same IOSensor node, used for retrieving images from camera sensor. Several fields (in this example they are called e.g. TrackedObject1Camera_PrincipalPoint, TrackedObject1Camera_FOV_horizontal, TrackedObject1Camera_FOV_vertical, TrackedObject1Camera_CAM_aspect) in this node provide the calibration information.
<IOSensor DEF='VisionLib' type='VisionLib' configFile='TutorialMarkerTracking_OneMarker.pm'>
<field accessType='outputOnly' name='VideoSourceImage' type='SFImage'/>
<field accessType='outputOnly' name='TrackedObject1Camera_ModelView' type='SFMatrix4f'/>
<field accessType='outputOnly' name='TrackedObject1Camera_PrincipalPoint' type='SFVec2f'/>
<field accessType='outputOnly' name='TrackedObject1Camera_FOV_horizontal' type='SFFloat'/>
<field accessType='outputOnly' name='TrackedObject1Camera_FOV_vertical' type='SFFloat'/>
<field accessType='outputOnly' name='TrackedObject1Camera_CAM_aspect' type='SFFloat'/>
</IOSensor>
7.4 Discussion
All three proposals suggest reusing the node that is used for accessing a camera sensor, and using a dedicated field of that node for providing camera calibration information. While proposal KC1 and KC2 suggest using projection matrix as a calibration information, proposal IR suggests using a set of parameters. The latter approach could be safer in terms of encapsulating the projection matrix of a viewpoint which could be implementation dependent based on what graphics API it is using.
8. Using calibration information to set properties of (virtual) camera
8.1 Proposal KC1
This proposal suggests a MatrixViewpoint node, which is a child of a scene node which represents a virtual viewpoint calibrated according to the corresponding physical live video camera (on the user's computer). The 'projmat' field represents the internal parameters (or projection matrix) of the MatrixViewpoint. The ‘position' and ‘orientation’ fields represent three dimensional position and orientation of the viewpoint within the virtual space. The ‘cameraSensor’ field represents a CameraSensor node, from which the viewpoint parameters (including projmat, position and orientation fields) of the MatrixViewpoint are updated according to. Once the ‘cameraSensor’ field is assigned with a validate CameraSensor node, the viewpoint parameters are updated according to the values from the CameraSensor node, assigned. Otherwise, it could be also used with routing each parameter of the MatrixViewpoint node from corresponding source of calibrated values.
MatrixViewpoint : X3DViewpointNode{
SFMatrix4f [in,out] projmat
SFVec3f [in,out] position
SFRotation [in,out] orientation
SFNode [in,out] cameraSensor
}
8.2 Proposal KC2
This proposal suggests similar approach with proposal KC1, which uses a viewpoint node that accepts camera calibration information in matrix form.
8.3 Proposal IR
Viewpoint : X3DViewpointNode {
...
SFString [in,out] fovMode VERTICAL
SFVec2f [in,out] principalPoint 0 0
SFFloat [in,out] aspect 1.0
SFFloat [in,out] zNear -1
SFFloat [in,out] zFar -1
}
The new fields provide a more general camera model than the standard Viewpoint. The "principalPoint" field defines the relative position of the principal point. If the principal point is not equal to zero, the viewing frustum parameters (left, right, top, bottom) are simply shifted in the camera's image plane. A value of x = 2 means the left value is equal to the default right value. A value of x = -2 means the right value is equal to default. If the principal point is not equal to zero, the "fieldOfView" value is not equal to the real field of view of the camera, otherwise it complies with the default settings.
To extend this idea, the "fovMode" defines whether the field of view is measured vertically, horizontally or in the smaller direction, which is important for correctly parameterizing the aforementioned cinematographic camera. The field ``aspect defines the aspect ratio for the viewing angle defined by the "fieldOfView" range. This setting is independent of the current aspect ratio of the window, but reflects the aspect ratio of the actual capturing device. This extension allows us to model cameras with a non-quadratic pixel format, i.e. it defines (width / height) of a pixel.
In addition to the Viewpoint extension we include a new camera node named Viewfrustum. This node has the two input/output fields "modelview" and "projection" of type SFMatrix4f. With the Viewfrustum node we are able to define a camera position and projection utilizing a standard projection/ modelview matrix pair.
Viewfrustum : X3DViewpointNode {
...
SFMatrix4f [in,out] modelview (identity)
SFMatrix4f [in,out] projection (identity)
}
8.4 Discussion
All of the proposals propose a new type of Viewpoint nodes to support camera calibration information described in section 7. While they use different type and number of fields for representing the camera calibration information, they all use same routing method to apply these value to a Viewpoint node. As discussed in 7.4, assigning a projection matrix directly to a viewpoint may result in defects, such as incorrect projections or near-far clipping planes.
9. Specifying nodes as physical object representatives
9.1 Proposal KC1
This proposal suggests a GhostGroup node for indicating its child nodes being representatives of physical objects for visualizing correct occlusion. The proposed node is extended from Group node to support those geometries of its child nodes are rendered as ghost objects. The browser should render the child nodes only into the depth buffer and not into the color buffer. As a result, the portion of the live video image corresponding to the ghost object is visualized with correct depth value, forming correct occlusion with other virtual objects.
GhostGroup: X3DGroupingNode{
... // same to the original Group node
}
9.2 Proposal KC2
This proposal does not include this feature.
9.3 Proposal IR
This proposal proposes using a ColorMaskMode node to render the geometry not into color buffer, and only to the depth buffer. In addition, a new field "sortKey" is proposed for the Appearance node for making sure the ghost objects are rendered prior to other geometries.
<Shape>
<Appearance sortKey='-1'>
<ColorMaskMode maskR='false' maskG='false' maskB='false' maskA='false'/>
</Appearance>
...
</Shape>
9.4 Discussion
While proposal KC1 suggests a high level, simple to use approach for a specific application in AR/MR, proposal IR provides more detail controllable, general purpose approach.
10. Conclusion
While all of the proposals suggest and cover similar set of functionalities required for supporting AR and MR visualization in X3D, Proposal KC1 and KC2 take the path of relatively higher level control, that provides simpler syntax that could be applied for specific cases for AR and MR. On the contrary, proposal IR introduces more generic purpose nodes and suggests to combine them to implement required functions, making AR and MR visualization a special use case of proposed extension. Considering the difference between the proposals, trade off between simplicity and flexibility/extensibility leads to a need for further discussion.
In the content authors' point of view, providing higher-level abstracted control gives more simpler and easy to use syntax. However, detail control might be missing which could be necessary for applications other than AR/MR.
In the browser implementor point of view, encapsulating the functions into higher level apis gives more room to choose their own way to implement the given function. On the other hand, if further detail control is required and added later for other applications, this could affect the ways how previous higher level components are implemented and may result in need for change in implementation level. Testing each function would be more complicated if low level details are accessible to scene authors, since there are more cases to test in order to make sure it works in general case.
Providing both in parallel could be an alternative, however, this would give more burdens to the browser implementors.
