Difference between revisions of "X3D JSON Encoding"
From Web3D.org
(→Standardization) |
m |
||
| (47 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
| − | These are X3D Working Group notes on the creation of an X3D JSON encoding and | + | These are X3D Working Group notes on the creation of an X3D JSON encoding and corresponding conversion processes. |
| + | |||
| + | Overall status: encoding is essentially complete, as demonstrated by | ||
| + | [http://www.web3d.org/x3d/stylesheets/X3dToJson.html X3D to JSON Stylesheet Converter]. | ||
| + | |||
| + | Further work awaits completed formalization of | ||
| + | [https://json-schema.org JSON Schema specifications]. Current assets include preliminary interoperability implementations at | ||
| + | [http://www.web3d.org/specifications/x3d-3.3-JSONSchema.json JSON schema] | ||
| + | ([http://www.web3d.org/specifications/X3dJsonSchemaDocumentation3.3/x3d-3.3-JSONSchema.html documentation]), | ||
| + | [https://github.com/coderextreme/X3DJSONLD X3DJSONLD JavaScript conversion library] and | ||
| + | [https://github.com/Web3dConsortium draft X3D JSON specification]. | ||
== Design Requirements, Goals and Use Cases == | == Design Requirements, Goals and Use Cases == | ||
| Line 6: | Line 16: | ||
* Round-trippable encoding supporting X3D abstract specification | * Round-trippable encoding supporting X3D abstract specification | ||
| − | Design goals: | + | Design goals and primary use cases: |
* Exchange format for a variety of 3D geometry and scene graphs | * Exchange format for a variety of 3D geometry and scene graphs | ||
* Loader for various JavaScript-controlled renderers | * Loader for various JavaScript-controlled renderers | ||
* Manipulate a scene graph using JavaScript | * Manipulate a scene graph using JavaScript | ||
| − | + | Design considerations: are there any special use cases for having X3D JSON available in JavaScript? | |
| − | * Are there any use cases that might modify how X3D is represented in JSON? | + | * Are there any use cases that might modify how X3D is represented in JSON? Yes. |
| − | * If so, it would be good to spell them out and understand them well. | + | * If so, it would be good to spell them out and understand them well... Accomplished. |
| − | * We want conversion rules to permit implementations that can achieve user goals. | + | * We want conversion rules to permit implementations that can achieve user goals... Optimization efforts successful. |
| + | |||
| + | Observation: the more that this work has progressed, the more the JSON output resembles VRML/ClassicVrml structure with slight differences in syntax. This approach has a valuable extra benefit in that results are broadly understandable by many object-oriented programmers. | ||
== Initial implementations == | == Initial implementations == | ||
| − | + | [http://www.web3d.org/x3d/stylesheets/X3dToJson.html X3D to JSON Stylesheet Converter], an experimental effort to help create a JSON encoding for X3D Graphics. | |
| − | * | + | * JSON translations for all of the 3900+ scenes in [http://www.web3d.org/x3d/content/examples/X3dResources.html#Examples X3D Example Archives] have been deployed. |
| − | + | ** Production build scripts autoconvert .x3d to .json, test with [http://www.jslint.com JSLint] validation, and publish JSON results | |
| − | * | + | ** In version control at [https://sourceforge.net/p/x3d/code/HEAD/tree/www.web3d.org/x3d/stylesheets/X3dToJson.xslt sourceforge] |
| − | + | * Conversion support included in [https://savage.nps.edu/X3D-Edit/ X3D-Edit] for testing of new examples by authors | |
| − | + | * Conversion syntax discussions continue on [http://www.web3d.org/community/public-mailing-lists x3d-public mail list] and [http://web3d.org/pipermail/x3d-public_web3d.org/2014-October/002981.html x3dom-developers mail list] (with [http://www.x3dom.org/x3dom/test/functional/flipper.html "dump scene as json" test] by Dr. Yvonne Jung) | |
| + | |||
| + | Related work in progress: X3D JSON loader to test round-trippability works satisfactorily | ||
| + | |||
| + | This X3D JSON Encoding page is periodically refreshed based on implementation/evaluation results. | ||
== Conversion Considerations == | == Conversion Considerations == | ||
| − | Primary design criterion: | + | Primary design criterion: round-trippable lossless representation of X3D scene. |
| − | Conversion approach of greatest practical interest: | + | Conversion approach of greatest practical interest: XML to/from JSON. Issues: |
| − | # How to convert attribute names to distinguish them from child elements. | + | # How to convert attribute names to distinguish them from child elements. Resolution: "@attributeName". |
| − | # JSON handling of container elements to preserve parent/child relationships, distinguishing child elements from attributes | + | # JSON handling of container elements to preserve parent/child relationships, distinguishing child elements from attributes Resolution: use SFNode/MFNode field names as unique keys. |
| − | # Creation of JSON elements with datatypes appropriate to content (e.g., integer, float, strings, etc.). | + | # Creation of JSON elements with datatypes appropriate to content (e.g., integer, float, strings, etc.). Note special JSON rules for floats (not equivalent to IEEE floats). |
| − | # Both X3D and | + | # Both X3D and JavaScript can include comments, although JSON does not. Need an option for inclusion (by default) or removal (optional) of comments in order to ensure 100% round-trip conversion capabilities. |
# jslint-validatable field/fieldValue representations within Scripts and prototypes | # jslint-validatable field/fieldValue representations within Scripts and prototypes | ||
# Support for singleton (self-closing) XML tags also needs to be considered, without loss of generality. | # Support for singleton (self-closing) XML tags also needs to be considered, without loss of generality. | ||
# Inclusion and preservation of embedded XML namespace information in an XML (.x3d) document: might not be necessary or possible. | # Inclusion and preservation of embedded XML namespace information in an XML (.x3d) document: might not be necessary or possible. | ||
| + | |||
| + | The | ||
| + | [http://www.web3d.org/x3d/stylesheets/X3dToJson.html X3D to JSON Stylesheet Converter] | ||
| + | considers these and other interoperability issues in great detail. | ||
== Standardization == | == Standardization == | ||
| Line 43: | Line 63: | ||
Probably smartest to first start out defining an X3D best practice. | Probably smartest to first start out defining an X3D best practice. | ||
| − | This capability likely needs to be defined as one of the [http://www.web3d.org/ | + | This capability likely needs to be defined as one of the [http://www.web3d.org/standards X3D standards] as shown in the [http://www.web3d.org/specifications/X3dSpecificationRelationships.png X3D Specification Relationships] diagram. |
The most probable place to put it is as a new Part 5 to ISO/IEC 19776. In this manner, it would correspond to the XML, Classic VRML, and Compressed Binary encodings. | The most probable place to put it is as a new Part 5 to ISO/IEC 19776. In this manner, it would correspond to the XML, Classic VRML, and Compressed Binary encodings. | ||
| Line 51: | Line 71: | ||
Here are suggested discussion points for X3D teleconferences and future followups. | Here are suggested discussion points for X3D teleconferences and future followups. | ||
| − | # Is there a good/consistent way for X3DOM to utilize such capabilities? | + | # Is there a good/consistent way for X3DOM, Cobweb or other tools to utilize such capabilities? |
| − | # Is there a way for [http://threejs.org Three.js] X3D loader, or other javascript libraries, to utilize such capabilities? | + | # Is there a way for [http://threejs.org Three.js] X3D loader, [http://d3js.org D3.js] X3D loader, or other javascript libraries, to utilize such capabilities? |
# Is there a single authoritative reference for JSON itself? and for JSON-XML conversions? See [6] for the JSON Data Interchange Format, need to confirm no others. | # Is there a single authoritative reference for JSON itself? and for JSON-XML conversions? See [6] for the JSON Data Interchange Format, need to confirm no others. | ||
| − | # What is the right file extension? | + | # What is the right file extension? .json is distinguishable from .js used in plain Javascript in Script code; .x3dj distinguishes X3D JSON |
| − | # Can a file reader distinguish the incoming encodings (.x3d .x3dv .x3db .x3de .json) independent of file extension or MIME media type? | + | ## [https://tools.ietf.org/html/rfc7159 IETF] (referencing ECMA) states that <code>.json</code> is the standard extension |
| − | + | ## Internet MIME-type is <code>application/json</code> | |
| + | # Can a file reader distinguish the incoming encodings (.x3d .x3dv .x3db .x3de .json/.x3dj) independent of file extension or MIME media type? | ||
# Compare compression size and decompression speed of a TestMesh.x3d.json.gz to TestMesh.x3db and TestMesh.x3d.exi (EXI will likely win because it includes data typing) | # Compare compression size and decompression speed of a TestMesh.x3d.json.gz to TestMesh.x3db and TestMesh.x3d.exi (EXI will likely win because it includes data typing) | ||
| + | # Show .dot notation in JavaScript code accessing elements and attributes, see if it is sensible | ||
| − | + | Lots of progress continues. | |
== References == | == References == | ||
| + | * Roy Walmsley, Don Brutzman and John Carlson, "[http://www.web3d.org/specifications/java/references/BrutzmanWalmsleyCarlsonJsonEncodingForX3dJuly2016.pdf A JSON Encoding for X3D]", [http://web3d2016.web3d.org Web3D 2016 Conference], Anaheim California, 22-24 July 2016. (also [https://dl.acm.org/citation.cfm?id=2945304&dl=ACM&coll=DL ACM Digital Library]) | ||
| + | * [http://www.jslint.com JSLint] validation | ||
| + | * X3D JSON loader, https://github.com/coderextreme/X3DJSONLD | ||
| + | * X3D to JSON Stylesheet Converter, http://www.web3d.org/x3d/stylesheets/X3dToJson.html | ||
* XML to JSON Converter (provides option to assign a prefix to JSON attributes, default is @ character) http://www.freeformatter.com/xml-to-json-converter.html | * XML to JSON Converter (provides option to assign a prefix to JSON attributes, default is @ character) http://www.freeformatter.com/xml-to-json-converter.html | ||
* Apache Camel, XML JSON Data Format (camel-xmljson) [http://camel.apache.org/xmljson.html | * Apache Camel, XML JSON Data Format (camel-xmljson) [http://camel.apache.org/xmljson.html | ||
* JavaScript Object Notation (JSON) Definition http://json.org | * JavaScript Object Notation (JSON) Definition http://json.org | ||
* ECMA 404: JSON Data Interchange Format (Final Draft) http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf | * ECMA 404: JSON Data Interchange Format (Final Draft) http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf | ||
| − | * JSON Markup Language (JsonML) http://www.jsonml.org | + | * JSON Markup Language (JsonML) http://www.jsonml.org |
* XSLTJSON: Transforming XML to JSON using XSLT http://www.bramstein.com/projects/xsltjson | * XSLTJSON: Transforming XML to JSON using XSLT http://www.bramstein.com/projects/xsltjson | ||
* Converting Between XML and JSON http://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html | * Converting Between XML and JSON http://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html | ||
| Line 78: | Line 104: | ||
* Initial message [http://web3d.org/pipermail/x3d-public_web3d.org/2014-July/002854.html] | * Initial message [http://web3d.org/pipermail/x3d-public_web3d.org/2014-July/002854.html] | ||
* July's thread listing [http://web3d.org/pipermail/x3d-public_web3d.org/2014-July/thread.html] | * July's thread listing [http://web3d.org/pipermail/x3d-public_web3d.org/2014-July/thread.html] | ||
| + | * Ongoing on [http://www.web3d.org/mailman/listinfo/x3d-public_web3d.org x3d-public mailing list], occasionally on [http://www.x3dom.org/?page_id=3 x3dom mailing list] | ||
Latest revision as of 07:24, 1 February 2023
These are X3D Working Group notes on the creation of an X3D JSON encoding and corresponding conversion processes.
Overall status: encoding is essentially complete, as demonstrated by X3D to JSON Stylesheet Converter.
Further work awaits completed formalization of JSON Schema specifications. Current assets include preliminary interoperability implementations at JSON schema (documentation), X3DJSONLD JavaScript conversion library and draft X3D JSON specification.
Contents
Design Requirements, Goals and Use Cases
Design requirement:
- Round-trippable encoding supporting X3D abstract specification
Design goals and primary use cases:
- Exchange format for a variety of 3D geometry and scene graphs
- Loader for various JavaScript-controlled renderers
- Manipulate a scene graph using JavaScript
Design considerations: are there any special use cases for having X3D JSON available in JavaScript?
- Are there any use cases that might modify how X3D is represented in JSON? Yes.
- If so, it would be good to spell them out and understand them well... Accomplished.
- We want conversion rules to permit implementations that can achieve user goals... Optimization efforts successful.
Observation: the more that this work has progressed, the more the JSON output resembles VRML/ClassicVrml structure with slight differences in syntax. This approach has a valuable extra benefit in that results are broadly understandable by many object-oriented programmers.
Initial implementations
X3D to JSON Stylesheet Converter, an experimental effort to help create a JSON encoding for X3D Graphics.
- JSON translations for all of the 3900+ scenes in X3D Example Archives have been deployed.
- Production build scripts autoconvert .x3d to .json, test with JSLint validation, and publish JSON results
- In version control at sourceforge
- Conversion support included in X3D-Edit for testing of new examples by authors
- Conversion syntax discussions continue on x3d-public mail list and x3dom-developers mail list (with "dump scene as json" test by Dr. Yvonne Jung)
Related work in progress: X3D JSON loader to test round-trippability works satisfactorily
This X3D JSON Encoding page is periodically refreshed based on implementation/evaluation results.
Conversion Considerations
Primary design criterion: round-trippable lossless representation of X3D scene.
Conversion approach of greatest practical interest: XML to/from JSON. Issues:
- How to convert attribute names to distinguish them from child elements. Resolution: "@attributeName".
- JSON handling of container elements to preserve parent/child relationships, distinguishing child elements from attributes Resolution: use SFNode/MFNode field names as unique keys.
- Creation of JSON elements with datatypes appropriate to content (e.g., integer, float, strings, etc.). Note special JSON rules for floats (not equivalent to IEEE floats).
- Both X3D and JavaScript can include comments, although JSON does not. Need an option for inclusion (by default) or removal (optional) of comments in order to ensure 100% round-trip conversion capabilities.
- jslint-validatable field/fieldValue representations within Scripts and prototypes
- Support for singleton (self-closing) XML tags also needs to be considered, without loss of generality.
- Inclusion and preservation of embedded XML namespace information in an XML (.x3d) document: might not be necessary or possible.
The X3D to JSON Stylesheet Converter considers these and other interoperability issues in great detail.
Standardization
Probably smartest to first start out defining an X3D best practice.
This capability likely needs to be defined as one of the X3D standards as shown in the X3D Specification Relationships diagram.
The most probable place to put it is as a new Part 5 to ISO/IEC 19776. In this manner, it would correspond to the XML, Classic VRML, and Compressed Binary encodings.
Discussion points
Here are suggested discussion points for X3D teleconferences and future followups.
- Is there a good/consistent way for X3DOM, Cobweb or other tools to utilize such capabilities?
- Is there a way for Three.js X3D loader, D3.js X3D loader, or other javascript libraries, to utilize such capabilities?
- Is there a single authoritative reference for JSON itself? and for JSON-XML conversions? See [6] for the JSON Data Interchange Format, need to confirm no others.
- What is the right file extension? .json is distinguishable from .js used in plain Javascript in Script code; .x3dj distinguishes X3D JSON
- IETF (referencing ECMA) states that
.jsonis the standard extension - Internet MIME-type is
application/json
- IETF (referencing ECMA) states that
- Can a file reader distinguish the incoming encodings (.x3d .x3dv .x3db .x3de .json/.x3dj) independent of file extension or MIME media type?
- Compare compression size and decompression speed of a TestMesh.x3d.json.gz to TestMesh.x3db and TestMesh.x3d.exi (EXI will likely win because it includes data typing)
- Show .dot notation in JavaScript code accessing elements and attributes, see if it is sensible
Lots of progress continues.
References
- Roy Walmsley, Don Brutzman and John Carlson, "A JSON Encoding for X3D", Web3D 2016 Conference, Anaheim California, 22-24 July 2016. (also ACM Digital Library)
- JSLint validation
- X3D JSON loader, https://github.com/coderextreme/X3DJSONLD
- X3D to JSON Stylesheet Converter, http://www.web3d.org/x3d/stylesheets/X3dToJson.html
- XML to JSON Converter (provides option to assign a prefix to JSON attributes, default is @ character) http://www.freeformatter.com/xml-to-json-converter.html
- Apache Camel, XML JSON Data Format (camel-xmljson) [http://camel.apache.org/xmljson.html
- JavaScript Object Notation (JSON) Definition http://json.org
- ECMA 404: JSON Data Interchange Format (Final Draft) http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf
- JSON Markup Language (JsonML) http://www.jsonml.org
- XSLTJSON: Transforming XML to JSON using XSLT http://www.bramstein.com/projects/xsltjson
- Converting Between XML and JSON http://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html
- XML/JSON Perl Converter http://search.cpan.org/~ken/XML-XML2JSON-0.06/lib/XML/XML2JSON.pm
- IBM's PHP converter http://www.ibm.com/developerworks/library/x-xml2jsonphp
- Java Converter http://www.json.org/javadoc/org/json/XML.html
- Google Code Library https://code.google.com/p/x2js
Mailing list discussions
- Initial message [1]
- July's thread listing [2]
- Ongoing on x3d-public mailing list, occasionally on x3dom mailing list
